Everything is a (Python) module

# Everything is a (Python) module

About me

me: Salvador de la Puente González
github: http://github.com/delapuente
twitter: @salvadelapuente
site: http://salvadelapuente.com

Dicebamus hesterna die...

Por Salva de la Puente, PyConES 2016

Transparencia sobre la maquinaria de import durante la charla de PyConES 2015, a modo de broma, no es relevante.

Por Raul Cumplido, PyConES 2016

[Syntactic macros in Python](../syntactic-macros-python/) [delapuente/mcpy](https://github.com/delapuente/mcpy)

Index 1. Foreign language integration 2. Feasability 3. The import machinery 3. `abm`: abstract modules

## Foreging language integration

I've joined IBM Research to work for **Emerging Techonology Experiences**. I work on [Qiskit](), a Python framework for designing and running quantum computing experiments.

Designing cuantum circuits ![Bell state circuit](./imgs/bell-state.png) This circuit makes reading the register to output `00` or `11` with a 50% chance each.

This is how a quantum experiment looks alike in Qiskit:

from qiskit import *
def bell():
  qr = QuantumRegister(2)
  cr = ClassicalRegister(2)
  qc = QuantumCircuit(qr, cr)
  qc.h(qr[1])
  qc.cnot(qr[1], qr[0])
  qc.measure(qr, cr)
  return qc

result = execute(qc, backend='local_qasm_simulator').result()
result.get_counts()
# will print {'counts': {'00': 534, '11': 490}}

Clear but verbose.

How about a specific language for creating circuits?


# in circuits.crc
crc bell q[2] -> c[2]:
  - :h: q1
  - q1 :cx: q0
end

More succint and funnier to write...

But also easy to integrate with Python ```python import crc from circuits import bell bell() # will print {'counts': {'00': 534, '11': 490}} ```

## Feasability

Python can load from `.zip` modules: ```bash $ tree . └── foo.zip ``` ```python import os, sys sys.path.insert(0, os.getcwd() + '/foo.zip') import bar bar.__file__ # will print '/Users/salva/test/foo.zip/bar.py' ```

Python can load non `.py` modules such as `.pyc` or `.so`. ```bash $ tree . └── bar.pyc ``` ```python import bar bar.__file__ # will print 'bar.pyc' ```

How does Python **distinguish** between loading a `.pyc` or a `.py` file?

## The import machinery

Let's say we execute: ```python import utils ``` ```bash $ tree . └── utils.py 0 directories, 1 file ```

It first check the **`sys.modules` cache**, which is a mapping of (fully qualified) module names and module objects. If not in the cache...

The **finders** in `sys.meta_path` are used to find the modules.

```bash import utils ``` ```bash sys.meta_path: +------------------+-----------------+-------------+ | builtin_importer | frozen_importer | path_finder | +------------------+-----------------+-------------+ ```

```bash import utils ^^^^^ ``` ```bash sys.meta_path: +------------------+-----------------+-------------+ | builtin_importer | frozen_importer | path_finder | +------------------+-----------------+-------------+ ^^^^^^^^^^^^^^^^^^ ``` ```python builtin_importer.find_spec('utils', None) ``` Returns `None`

```bash import utils ^^^^^ ``` ```bash sys.meta_path: +------------------+-----------------+-------------+ | builtin_importer | frozen_importer | path_finder | +------------------+-----------------+-------------+ ^^^^^^^^^^^^^^^^^ ``` ```python frozen_importer.find_spec('utils', None) ``` Returns `None`

```bash import utils ^^^^^ ``` ```bash sys.meta_path: +------------------+-----------------+-------------+ | builtin_importer | frozen_importer | path_finder | +------------------+-----------------+-------------+ ^^^^^^^^^^^^^ ``` ```python path_finder.find_spec('utils', None) ``` Returns a **`module_spec`** and use it to load the module!

![Step into](imgs/step-into.png) ```python path_finder.find_spec('utils', None) ```

The `PathFinder` instance in the `sys.meta_path` is in charge of searching for modules inside _sorts of paths_. It turns out, **it is extensible**.

```bash module_name = 'utils' ``` ```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ```

The `sys.path` and `sys.path_hooks` lists are used to determine **where to search** the module.

The module's name is used to determine **what to search**.

```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ^^^^^^^^^^^ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ^^^^^^^^^^^^^^ ``` ```python entry_finder = zip_importer('foo.zip') ``` Returns an **entry finder**. Meaning: "The `zip_importer` can search inside `'foo.zip'`...".

```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ^^^^^^^^^^^ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ^^^^^^^^^^^^^^ ``` ```python entry_finder = zip_importer('foo.zip') entry_finder.find_spec(module_name, parent_path) ``` Returns `None`. Meaning: "...but could not find `module_name`."

```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ^^^^ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ^^^^^^^^^^^^^^ ``` ```python entry_finder = zip_importer('') # '' means the current working directory ``` Returns a `None`. Meaning: "`zip_importer` cannot search inside the `CWD`"

```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ^^^^ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ^^^^^^^^^^^^^ ``` ```python entry_finder = file_finder('') # returns a file entry finder ``` Returns an **entry finder**. Meaning: "`file_finder` can search inside the `CWD`..."

```bash sys.path: +-----------+----+---------------------------------------------------+ | 'foo.zip' | '' | '/Users/salva/.venv/lib/python3.6/site-packages/' | +-----------+----+---------------------------------------------------+ ^^^^ ``` ```bash sys.path_hooks: +--------------+-------------+ | zip_importer | file_finder | +--------------+-------------+ ^^^^^^^^^^^^^ ``` ```python entry_finder = file_finder('') # returns a file entry finder entry_finder.find_spec(module_name, parent_path) ``` Returns a **`module_spec`**. Meaning: "...and found `module_name`."

![Step into](imgs/step-into.png) ```python entry_finder.find_spec(module_name, parent_path) ```

The `file_finder` uses a mapping of extensions and `Loader` classes to determine which loader bind to `module_spec.loader`.

```bash parent_path = None module_name = 'utils' # just the last component ``` ```bash file_finder._loaders: +--------------------+-----------------------+----------------------------+ | ('.so', ExtLoader) | ('.py', SourceLoader) | ('.pyc', SourcelessLoader) | +--------------------+-----------------------+----------------------------+ ```

```bash parent_path = None module_name = 'utils' ``` ```bash file_finder._loaders: +--------------------+-----------------------+----------------------------+ | ('.so', ExtLoader) | ('.py', SourceLoader) | ('.pyc', SourcelessLoader) | +--------------------+-----------------------+----------------------------+ ^^^^^^^^^^^^^^^^^^^^ ``` ```python if Path(module_name + '.so').exists(): module_spec.loader = ExtLoader() ``` Does not exist, continue searching...

```bash parent_path = None module_name = 'utils' ``` ```bash file_finder._loaders: +--------------------+-----------------------+----------------------------+ | ('.so', ExtLoader) | ('.py', SourceLoader) | ('.pyc', SourcelessLoader) | +--------------------+-----------------------+----------------------------+ ^^^^^^^^^^^^^^^^^^^^^^^ ``` ```python if Path(module_name + '.py').exists(): module_spec.loader = SourceLoader() ``` Exists! The `module_spec` has the proper loader in place and it is returned to the import machinery.

The **`loader` attribute** of the `module_spec` is used to create and execute the module. Something like: ```python module = module_spec.loader.create_module(module_spec) module_spec.loader.exec_module(module) ```

The module is finally cached in **`sys.modules`**. And now the module is imported and everything is fine!

So, can we add new pairs extension/loader class? ```bash file_finder._loaders: +--------------------+-----------------------+----------------------------+ | ('.so', ExtLoader) | ('.py', SourceLoader) | ('.pyc', SourcelessLoader) | +--------------------+-----------------------+----------------------------+ ```

Nope. Loaders are passed during initialization of the import machinery module in a function marked as private and ~~cannot~~ should not be altered.

From the Zen of Python

> Special cases aren't special enough to break the rules.

> Although **practicality beats purity**.

## `abm`: abstract modules [delapuente/abm](https://github.com/delapuente/abm)

You can install it from `pip`: ```bash $ pip install abm ``` Or GitHub: ```bash $ pip install -e git+https://github.com/delapuente/abm.git#egg=abm ```

Activate it with the following code: ```python import abm.activate ``` Now you can register a new loader for an extension with: ```python import sys sys.abm_hooks['.jpg'] = JpgLoader ```

### How does it work?

`_loaders` setter and getter, and `activate` function: ```python def activate(): if not hasattr(sys, 'abm_hooks'): sys.abm_hooks = {} FileFinder._loaders = property(_get_loaders, _set_loaders) def _set_loaders(self, loaders): self._builtin_loaders = loaders def _get_loaders(self): return chain( self._builtin_loaders, getattr(sys, 'abm_hooks').items()) ```

The `activate` module creates **`sys.abm_hooks`** and changes the `FileFinder` class to convert the internal `_loaders` attribute into a class property. The getter and setter for this property behave as follows:

1. The setter **diverts the value to a new instance attribute** to retain the original initialization. 2. The getter **combines the new attribute with the hooks from `sys.abm_hooks`**.

This way, when reading `_loaders`... ```bash file_finder._loaders: ----------------------+----------------------------+ +---------------------+ ('.py', SourceLoader) | ('.pyc', SourcelessLoader) | + | ('.jpg', JpgLoader) | ----------------------+----------------------------+ +---------------------+ file_finder._builtin_loaders + sys.abm_hooks.items() ```

### Writing loaders

The `.ini` file: ```ini ; config.ini [section] key = value ``` ```bash $ tree . └── config.ini 0 directories, 1 file ```

The module class: ```python from types import ModuleType from configparser import ConfigParser class IniModule(ModuleType, ConfigParser): def __init__(self, spec_name): ModuleType.__init__(self, spec_name) ConfigParser.__init__(self) ```

The loader: ```python from abm.loaders import AbmLoader class IniLoader(AbmLoader): extensions = ('.ini',) def create_module(self, spec): module = IniModule(spec.name) self.init_module_attrs(module) return module def exec_module(self, module): module.read(self.path) ```

Usage: ```python import abm.activate IniLoader.register() import config assert(config['section']['key'] == 'value') ```

Remember: 1. `create_module` is in charge of creating the module instance. 2. `exec_module` is in charge of populating the module.

## See also * [The import system](https://docs.python.org/3/reference/import.html) * [The `importlib` reference](https://docs.python.org/3/library/importlib.html) * [`crc`: a language for writing quantum circuits](https://github.com/delapuente/crc)