brandtbucher / hax

Write compiled bytecode inline with pure Python. 🤖

I'm a computer-engineer-turned-software-engineer, heavily involved in CPython development.

PEP 744 – JIT Compilation
Python Enhancement Proposals
April 11th, 2024

A tour of CPython's runtime
University of California, Irvine
March 11th, 2024

2023

A JIT compiler for CPython
CPython Core Dev Sprint 2023
October 10th, 2023

Inside CPython 3.11's new specializing, adaptive interpreter
PyCon US 2023
April 21st, 2023

A tour of CPython's bytecode compiler
University of California, Irvine
March 14th, 2023

2022

Python 3.11 Release Episode
Talk Python
October 28th, 2022

A Team at Microsoft is Helping Make Python Faster
Microsoft Python DevBlog
October 26th, 2022

Making CPython 3.11 Especially Fast
Python 3.11 Release
October 24th, 2022

Python Perf: Specializing, Adaptive Interpreter
Talk Python
September 15th, 2022

A Perfect match
PyCon US 2022
April 29th, 2022

2021

A Perfect match
PyCon JP 2021
October 16th, 2021

Structural Pattern Matching
Python 3.10 Release Stream
October 4th, 2021

What's New in Python 3.10
PyCharm by JetBrains
October 4th, 2021

2020

Dynamic Pattern Matching with Python
Dynamic Languages Symposium 2020
November 15th, 2020

PEP 634 – Structural Pattern Matching: Specification
Python Enhancement Proposals
September 12th, 2020

PEP 622 – Structural Pattern Matching
Python Enhancement Proposals
June 23rd, 2020

PEP 618 – Add Optional Length-Checking To zip
Python Enhancement Proposals
May 1st, 2020

PEP 614 – Relaxing Grammar Restrictions On Decorators
Python Enhancement Proposals
February 10th, 2020

2019

PEP 584 – Add Union Operators To dict
Python Enhancement Proposals
March 1st, 2019

hax's Issues

Automatically set the correct flags on the resulting code object.

So YIELD_* automatically makes it a generator, etc.

Allow qualified name access.

We should allow things like hax.LOAD_CONST(...) in addition to just LOAD_CONST(...).

Segmentation fault with HAX_LABEL/LABEL

I have no clue why this happens but here's what's going on:

In [1]: from hax import *
   ...: import dis

In [2]: @hax
   ...: def test():
   ...:     HAX_LABEL("l1")
   ...:     print(0)
   ...:     JUMP_ABSOLUTE("l1")
   ...:

In [3]: dis.dis(test)
  2     >>    0 LOAD_GLOBAL              0 (print)
              2 LOAD_CONST               1 (0)
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 JUMP_ABSOLUTE            0 (to 0)
             10 LOAD_CONST               0 (None)
             12 RETURN_VALUE

In [4]: @hax
   ...: def test():
   ...:     flag = True
   ...:     HAX_LABEL("l1")
   ...:     print(0)
   ...:     JUMP_ABSOLUTE("l1")
   ...:

In [5]: dis.dis(test)
  3           0 LOAD_CONST               1 (True)
              2 STORE_FAST               0 (flag)

  4           4 LOAD_GLOBAL              0 (print)
              6 LOAD_CONST               2 (0)
        >>    8 CALL_FUNCTION            1
             10 POP_TOP
             12 JUMP_ABSOLUTE            4 (to 8)
             14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

In [6]: test()
0
Segmentation fault

So if there's a variable defined before it, it for some reason makes the jump absolute go to CALL_FUNCTION instead of LOAD_GLOBAL

Use labels to specify jump targets.

The current jump situation requires the user to reverse-engineer our complicated transformations. We should provide some sort of labeling mechanism, and have the jumping instructions take a string argument. Ideally, jumps should be tempting to write, not daunting to write.

This should also have the benefit of increasing readability!

-

Optimize the resulting code object.

Currently, the code object we generate has too big of a stack, extra names/constants, and a ton of NOPs. We should take a page out of peephole.c’s playbook and optimize this stuff. We do need to be careful, though, to not change any of the legitimate generated instructions (including user-specified NOPs, which could be tricky).

Because this would make jumps basically impossible to write, #1 would most likely need to happen first.

Remove deprecated LABEL instruction.

Prefer HAX_LABEL instead.

Support user-supplied EXTENDED_ARG ops.

Currently, the behavior for user-specified EXTENDED_ARG ops is pretty undefined.

Should we support them? What would that even look like?

Clean things up.

A lot of this stuff can be simplified. Lets try to do that before going stable.

Use bytecode package for final assembly.

We could drastically simplify (and improve) things by using bytecode as a backend, rather than our hand-rolled, hacky, one-pass "recompiler". I think this should fix things like line numbers, too!

Allow more complicated constants.

I'm thinking code objects, frozensets, etc. We'd probably need an artificial HAX_* code to make this work, just not sure what the best design is yet.

Get to 100% coverage.