naleraphael / bytefall

Another Python bytecode interpreter implemented in Python for version >= py34, with a few new features

bytefall

This is a Python virtual machine implemented in pure Python and targeting Python version >= 3.4. It mainly derives from the following great works: nedbat/byterun and darius/tailbiter.

In this project, complete bytecode operations are implemented. And the structure of implementation is modified to make it more easily be extended for multiple versions of Python.

Besides, operations related to asyncio are also supported.

More features for debugging bytecode are going to be implemented, because I'm also trying to use this tool to fix some bugs in my own project bytejection.

Installation

$ pip install git+https://github.com/NaleRaphael/bytefall.git

Usage

Currently, version of virtual machine is automatically chosen according to your base Python runtime. Because it requires a base runtime to compile Python code to bytecode.

Therefore, you may need to create different virtual environment with the version of Python runtime you want to run this virtual machine.

$ python -m bytefall [YOUR_SCRIPT.py]

Run tests

$ python -m pytest ./tests/

# run tests including cases that usually take long time
$ python -m pytest ./tests/ --runslow

Features

• You can run bytefall with an argument --debug to get detailed traceback of an unexpected error

  $ python -m bytefall --debug [YOUR_SCRIPT.py]

• To show detailed information of each bytecode instruction which is going to be executed, you can run bytefall with --show_oparg.

  $ python -m bytefall --show_oparg [YOUR_SCRIPT.py]

  Then you can get something like this:

  # Format:
  # Instruction name | Arguments of instruction | Stack content on current frame
  
  LOAD_CONST (<code object main at 0x0000021583CAC150, file "foo.py", line 2>,) []
  LOAD_CONST ('main',) [<code object main at 0x0000021583CAC150, file "foo.py", line 2>]
  MAKE_FUNCTION (0,) [<code object main at 0x0000021583CAC150, file "foo.py", line 2>, 'main']
  STORE_NAME ('main',) [<Function main at 0x0000021583DDAA58>]
  # ...
  

• To trace execution of each bytecode instruction, you can run bytefall with --trace_opcode, and use pdb.set_trace() to determine the entry.

  Try it online (repl.it)

  $ python -m bytefall --trace_opcode [YOUR_SCRIPT.py]

  # YOUR_SCRIPT.py
  def main():
      foo()
      import pdb; pdb.set_trace()
      bar()   # <- entry
      buzz()

• To explore the internal execution of virtual machine with pdb, you can run it with an environment variable DEBUG_INTERNAL

  $ DEBUG_INTERNAL=1 python -m bytefall [YOUR_SCRIPT.py]

bytefall's Issues

Failed to build a class from a custom metaclass

Just a memo for this issue.

Take the following code as an example.

class MyMeta(type):
    def __new__(cls, name, bases, dct):
        x = super().__new__(cls, name, bases, dct)
        x.name = 'Foo'
        return x

class Foo(metaclass=MyMeta):
    def greet(self):
        print('hello from {}!'.format(self.name))

if __name__ == '__main__':
    Foo().greet()

After running this snippet, we will get an error: TypeError: __classcell__ must be a nonlocal cell, not <class 'bytefall.objects.cellobject.Cell'>.

It is because __classcell__ in namespace should be a built-in cell, not the one implemented in Python, while we are going to make a new class from metaclass. (see also cpython/typeobject.c::type_new and the following code)

# @bytefall/ops.py
def build_class(func, name, *bases, **kwds):
    # ... omitted code
    cls = metaclass(name, bases, namespace)
    # ... omitted code

To fix it, we can replace the cell by the one actually created from internal. Like the solution taken by byterun:

def make_cell(value):
    fn = (lambda x: lambda: x)(value)
    return fn.__closure__[0]

def build_class(func, name, *bases, **kwds):
    # ... omitted code
    if '__classcell__' in namespace:
        real_cell = make_cell(None)
        real_cell.cell_contents = namespace['__classcell__'].contents
        namespace['__classcell__'] = real_cell

    cls = metaclass(name, bases, namespace)
    # ... omitted code

But here comes another problem. super() in MyMeta.__new__() is failed to be executed, we got an error RuntimeError: super(): __class__ cell not found.

After looking into CPython source code, that message is written in the function typeobject.c::super_init().

// @cpython/Objects/typeobject.c::super_init()
static int
super_init(PyObject *self, PyObject *args, PyObject *kwds)
{
    // ... omitted code
    if (type == NULL) {
        /* Call super(), without args -- fill in from __class__
           and first local variable on the stack. */
        PyFrameObject *f;
        PyCodeObject *co;
        Py_ssize_t i, n;
        f = PyThreadState_GET()->frame;     // 1.

        // ... omitted code

        for (i = 0; i < n; i++) {
            PyObject *name = PyTuple_GET_ITEM(co->co_freevars, i);  // 2.
            assert(PyUnicode_Check(name));
            if (_PyUnicode_EqualToASCIIId(name, &PyId___class__)) {
                // code for finding cell, and cast cell to a type object
                Py_ssize_t index = co->co_nlocals +
                    PyTuple_GET_SIZE(co->co_cellvars) + i;
                PyObject *cell = f->f_localsplus[index];
                if (cell == NULL || !PyCell_Check(cell)) {
                    PyErr_SetString(PyExc_RuntimeError,
                      "super(): bad __class__ cell");
                    return -1;
                }
                type = (PyTypeObject *) PyCell_GET(cell);
                // ... omitted code
            }
        }
        if (type == NULL) {     // 3.
            PyErr_SetString(PyExc_RuntimeError,
                            "super(): __class__ cell not found");
            return -1;
        }
    }
    // ... omitted code

In the simplified code above:

1. f is a frame hold by interpreter.
2. In that loop, we are going to find __class__ in co.co_freevars. Once we found that, we can know the index of cell in f.f_localsplus. Then we can get type object from it.
3. If no __class__ in co.co_freevars, cell won't be availabe. Hence that we cannot get type.

However, in our implementation, all information is stored in Frame implemented by us instead of built-in frame. Therefore, frame f retrieved by PyThreadState_GET()->frame in step 1 is not the one with correct data for execution, and that's why step 2 and 3 failed.

A possible solution for this is re-implementing super_init and type_new, but we still need to go deeper into it.