Syntactic macros in Python

# Syntactic macros in Python

Colorless green ideas sleep furiously

— Noam Chomsky, American linguistic

## What are macros?

Traditionally, macros are **substitutions** of fragments of the source code by some transformation of themselves.

This substitution is called **_macro-expansion_** and its performed by the compiler in a previous pass before compiling the actual code.

There are several forms of macros, probably most famous are _text substitution macros_ in which a preprocessor **search and replace** specific text sequences.

**Basic C macro** ```cpp #define TRUE 1 int isamacro = TRUE; // becomes isamacro = 1 ```

**Parametrized C macro** ```cpp #define max(x, y) ((x) > (y) ? (x) : (y)) int maximum = max(5+1, v); // becomes ((5+1) > (v) ? (5+1) : (v)) ```

**For-in iteration protocol** ```cpp #define for_in(T, v, c) \ for (iter<T> (*v) = c->iter(); v; v = v->next()) for_in(int, n, intarray) { printf("Double of %d is %d", *n, *n * 2); } /* becomes for (iter<int> (*n) = intarray->iter(); n; n = n->next()) { printf("Double of %d is %d", *n, *n * 2); } */ ```

Text preprocessors knows nothing about the structure of the source code is replacing.

But syntactic macros does...

## What are syntactic macros?

They are transformations of the **syntactic tree**. The macro is actually a function taking an **AST as input** and returning another **AST as expansion**.

**Basic LISP macro** ```lisp (defmacro when (test exp . rest) `(if ,test (progn ,exp . ,rest))) (when nil (display "Launching missiles!\n")) ;; Expand to ;; (if nil ;; (progn (display "Launching missiles!\n"))) ```

**Proposal for _Python_ `log` macro** ```python log[people[7].name] # Expands to print('people[7].name:', people[7].name) ```

With sintactic macros we can **abuse** the language syntax and provide new pragmatics. I.e. **create new meaning**.

**Proposal for _Python_ `customliterals` macro** ```python # Runtime error: `AttributeError: __exit__` with customliterals: tuple is point print((0,0).distance((1,1))) '''Expands to: print( point( (0,0) ).distance( point( (1,1) ) ) ) ''' ```

But syntactic macros _per se_ does not allow to extend the language. The source code must be recognized as a valid AST before expansion.

**The `d` (_dice roll_) operator** ```python roll = 5 d 6 # would expand in (randint(1, 6+1) for n in range(5)) # Pre-runtime error: `SyntaxError: invalid syntax` ```

Would not be cool to **use Python to expand Python**?

## macropy [lihaoyi/macropy](https://github.com/lihaoyi/macropy)

* A macro expander in **_import-time_**. * A complete library with lots of useful macros. * An authoring framework for creating new macros. * Works with **CPtyhon 2.7.2**, **PyPy 2.0** * Partial support in Python 3.x

**Install & basic setup** ``` # pip2 install macropy ``` ```python # run.py import macropy.activate # important! import myprogram.py # myprogram.py from mymacros import macros, ... '''Do something with macros...''' # mymacros.py from macropy.core.macros import * macros = Macros() # important! '''Define macros here''' ``` ```python # Or in the Python console, instead of `activate` import macropy.console ```

**The Case macro** ```python from macropy.case_classes import macros, case @case class Point(x, y): pass p = Point(1, 2) print str(p) # Point(1, 2) print p.x # 1 print p.y # 2 print Point(1, 2) == Point(1, 2) # True x, y = p print x, y # 1 2 ``` Advanced topics about [case classes](https://github.com/lihaoyi/macropy#case-classes) in the docs.

**The Quick Lambda macro** ```python from macropy.quick_lambda import macros, f, _ print map(f[_ + 1], [1, 2, 3]) # [2, 3, 4] print reduce(f[_ * _], [1, 2, 3]) # 6 ``` More about [quick lambdas](https://github.com/lihaoyi/macropy#quick-lambdas) in the documentation.

**The show_expanded macro** ```python from macropy.case_classes import macros, case from macropy.tracing import macros, show_expanded with show_expanded: @case class Point(x, y): pass ``` More introspection utilities as [show_expanded](https://github.com/lihaoyi/macropy#show_expanded) in the docs.

And tons of more features: * [Lazy](https://github.com/lihaoyi/macropy#lazy), [String Interpolation](https://github.com/lihaoyi/macropy#string-interpolation) & [Tracing](https://github.com/lihaoyi/macropy#tracing) macros. * MacroPEG [Parser Combinator](https://github.com/lihaoyi/macropy#macropeg-parser-combinators). * Experimental [pattern matching](https://github.com/lihaoyi/macropy#pattern-matching) & [tail-call optimization](https://github.com/lihaoyi/macropy#tail-call-optimization). * PINQ, [SQL integration](https://github.com/lihaoyi/macropy#pinq-to-sqlalchemy) in Python. * [Pyxl](https://github.com/lihaoyi/macropy#pyxl-snippets) & [JS](https://github.com/lihaoyi/macropy#js-snippets) snippets. * And [even more](https://github.com/lihaoyi/macropy#macropy-103)...

### Writing macros

**The [log](https://github.com/lihaoyi/macropy#tracing) macro** It's quite similar to write [LISP macros](). ```python character = { 'name': 'Iñigo Montoya' } # We want this: log[character['name']] # ...to expand into: print 'character[\'name\'] ->', character['name'] ```

Mark the module as a macro container ```python from macropy.core.macros import * macros = Macros() ```

Use a decorator to specify what kind of use you want for your macro ```python from macropy.core.macros import * macros = Macros() @macros.expr def log(tree, **kw): return tree ```

Use [hygienic quasiquotes](https://github.com/lihaoyi/macropy#quasiquotes) to build new ASTs avoiding the ugly [AST API](https://docs.python.org/3.5/library/ast.html) ```python from macropy.core.macros import * from macropy.core.hquotes import macros, hq, ast, u macros = Macros() @macros.expr def log(tree, **kw): label = unparse(tree) + ' ->' return hq[eprint(u[label], ast[tree])] def eprint(label, target): print label, target ```

* `unparse(tree)` is a **function** returning the Python code for _tree_. * `hq[tree]` is a **macro** that returns the AST for the Python code needed to build _tree_ but preserving the macro context. * `ast[tree]` is a **macro** used only inside _hq_ to insert the AST in _tree_ as part of the expression in which the _ast_ macro is found. * `u[tree]` is a **macro** used only inside _hq_ to insert the AST of the result of evaluating _tree_ in the macro context in the expression where the _u_ macro is found. Only [built-in types](https://docs.python.org/3.5/library/stdtypes.html) are supported.

More in the [tutorials section](https://github.com/lihaoyi/macropy#tutorials).

## How does it work?

![macropy working schema](./imgs/macropy-schema.png) macropy **intercepts the module when importing it**, expand the AST, and executes the new AST.

#### importing * **New Import Hooks** ([PEP 0302](https://www.python.org/dev/peps/pep-0302/)) allows to customize _import system_. * [Import system](https://docs.python.org/3/reference/import.html) relies on finders and loaders. * A [finder](https://docs.python.org/3/glossary.html#term-finder) searches a module and return a loader for it. * A [loader](https://docs.python.org/3/glossary.html#term-loader) reads and [executes](https://docs.python.org/3/library/functions.html#exec) the module. ```python import macropy.activate ``` * That line adds a [custom finder](https://github.com/lihaoyi/macropy/blob/13993ccb08df21a0d63b091dbaae50b9dbb3fe3e/macropy/core/import_hooks.py#L22) in charge of expanding the AST before executing it.

#### expansion I [ast._parse()_](https://docs.python.org/3.5/library/ast.html#ast.parse) function returns the AST for source code. ![ast expansion](./imgs/ast-expansion-1.png) macropy [looks for nodes representing macros](https://github.com/lihaoyi/macropy/blob/13993ccb08df21a0d63b091dbaae50b9dbb3fe3e/macropy/core/import_hooks.py#L42-L60).

#### expansion II Found nodes are split into macro name and wrapped tree. ![new ast computation](./imgs/ast-expansion-2.png) The macro function is executed passing the wrapped tree as parameter.

#### execution Now the AST has been expanded, the custom loader executes the new AST in the module context.

## mcpy [delapuente/mcpy](https://github.com/delapuente/mcpy)

* Focus on **expanding** macros. * Developed as an study case for **learning**. * Very **small library** compared with macropy. * No utilities for authoring.

Show me [da code](https://github.com/delapuente/mcpy/blob/master/demo/run.py)!

## See also * [Wikipedia article about macros](https://en.wikipedia.org/wiki/Macro_%28computer_science%29) * [Macros: Defining Your Own](http://www.gigamonkeys.com/book/macros-defining-your-own.html) * The [expansion code for macropy](https://github.com/lihaoyi/macropy/blob/13993ccb08df21a0d63b091dbaae50b9dbb3fe3e/macropy/core/macros.py#L100).

About me

me: Salvador de la Puente González
twitter: @salvadelapuente
My web sites: http://unoyunodiez.com
http://github.com/delapuente