rocky / python-xasm

Python cross version bytecode/wordcode assembler

xasm

NOTE: this is in beta

A Cross-Python bytecode Assembler

Introduction

The Python xasm module has routines for assembly, and has a command to assemble bytecode for several different versions of Python.

Here are some potential uses:

• Make small patches to existing Python bytecode when you don’t have source
• Craft custom and efficient bytecode
• Write an instruction-level optimizing compiler
• Experiment with and learn about Python bytecode
• Foil uncompyle6 so that it can’t disassemble bytecode (at least for now)

This will support bytecodes from Python version 1.0 to 3.8 or so.

The code requires Python 2.7 or later.

Assembly files

More detail will be filled in, but some principles:

• Preferred extension for Python assembly is .pyasm
• assembly is designed to work with the output of pydisasm --asm
• Assembly file labels are at the beginning of the line and end in a colon, e.g. END_IF
• instruction offsets in the assembly file are ignored and don't need to be entered
• in those instructions that refer to offsets, if the if the operand is an int, exactly that value will be used for the operand. Otherwise we will look for labels and match up with that

Installation

The standard Python routine:

pip install -e .
pip install -r requirements-dev.txt

A GNU makefile is also provided so make install (possibly as root or sudo) will do the steps above.

Testing

make check

A GNU makefile has been added to smooth over setting running the right command, and running tests from fastest to slowest.

If you have remake installed, you can see the list of all tasks including tests via remake --tasks.

Example Assembly File

For this Python source code:

def five():
    return 5

print(five())

Here is an assembly for the above:

# Python bytecode 3.6 (3379)

# Method Name:       five
# Filename:          /tmp/five.pl
# Argument count:    0
# Kw-only arguments: 0
# Number of locals:  0
# Stack size:        1
# Flags:             0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# First Line:        1
# Constants:
#    0: None
#    1: 5
  2:
            LOAD_CONST           (5)
            RETURN_VALUE


# Method Name:       <module>
# Filename:          /tmp/five.pl
# Argument count:    0
# Kw-only arguments: 0
# Number of locals:  0
# Stack size:        2
# Flags:             0x00000040 (NOFREE)
# First Line:        1
# Constants:
#    0: <code object five at 0x0000>
#    1: 'five'
#    2: None
# Names:
#    0: five
#    1: print
  1:
            LOAD_CONST           0 (<code object five at 0x0000>)
            LOAD_CONST           ('five')
            MAKE_FUNCTION        0
            STORE_NAME           (five)

  3:
            LOAD_NAME            (print)
            LOAD_NAME            (five)
            CALL_FUNCTION        0
            CALL_FUNCTION        1
            POP_TOP
            LOAD_CONST           (None)
            RETURN_VALUE

The above can be created automatically from Python source code using the pydisasm command from `xdis`:

pydisasm --format xasm /tmp/five.pyc

In the example above though, I have shortend and simplified the result.

Usage

To create a python bytecode file from an assemble file, run:

pyc-xasm [OPTIONS] ASM_PATH

For usage help, type pyc-xasm --help.

To convert a python bytecode from one bytecode to another, run:

pyc-convert [OPTIONS] INPUT_PYC [OUTPUT_PYC]

For usage help, type pyc-convert --help.

See Also

• https://github.com/rocky/python-xdis : Cross Python version disassemble
• https://github.com/rocky/x-python : Cross Python version interpreter
• https://github.com/rocky/python-xasm/blob/master/HOW-TO-USE.rst : How to write an assembler file
• https://rocky.github.io/pycon2018-light.co/ : Pycolumbia 2018 Lightning talk showing how to use the assembler

python-xasm's Issues

Error in python 3.8.3 ,how to fix?

why do i get an error like this when trying it in python 3.8.3? Is this a bug? The following is the error code

$ pydisasm --version
pydisasm, version 4.6.1

$ pydisasm --asm tes.pyc
# pydisasm version 4.6.1
# Python bytecode 3.8 (3413)
# Disassembled from Python 3.8.3 (default, May 14 2020, 14:40:57)
# [Clang 8.0.7 (https://android.googlesource.com/toolchain/clang b55f2d4ebfd35bf6
# Timestamp in code: 1591520548 (2020-06-07 16:02:28)
# Source code size mod 2**32: 101 bytes

# Method Name:       helo
# Filename:          tes.py
# Argument count:    1
# Position-only argument count: 0
# Keyword-only arguments: 0
# Number of locals:  1
# Stack size:        6
# Flags:             0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# First Line:        1
# Constants:
#    0: None
# Names:
#    0: print
# Varnames:
#       teks
# Positional arguments:
#       teks
  2:
            SETUP_FINALLY        L14 (to 14)

  3:
            LOAD_GLOBAL          (print)
            LOAD_FAST            (teks)
            CALL_FUNCTION        1
            POP_TOP
            POP_BLOCK
            JUMP_FORWARD         L26 (to 26)

L14:
   4:
            POP_TOP
            POP_TOP
            POP_TOP

  5:
            POP_EXCEPT
            JUMP_FORWARD         L26 (to 26)
            END_FINALLY
L26:
            LOAD_CONST           (None)
            RETURN_VALUE


# Method Name:       <module>
# Filename:          tes.py
# Argument count:    0
# Position-only argument count: 0
# Keyword-only arguments: 0
# Number of locals:  0
# Stack size:        2
# Flags:             0x00000040 (NOFREE)
# First Line:        1
# Constants:
#    0: <Code38 code object helo at 0xae631688, file tes.py>, line 1
#    1: 'helo'
#    2: '__main__'
#    3: 'Hello ,World!'
#    4: None
# Names:
#    0: helo
#    1: __name__
  1:
            LOAD_CONST           0 (<Code38 code object helo at 0xae631688, file tes.py>, line 1)
            LOAD_CONST           ('helo')
            MAKE_FUNCTION        0 (Neither defaults, keyword-only args, annotations, nor closures)
            STORE_NAME           (helo)

  7:
            LOAD_NAME            (__name__)
            LOAD_CONST           ('__main__')
            COMPARE_OP           (==)
            POP_JUMP_IF_FALSE    L24 (to 24)

  8:
            LOAD_NAME            (helo)
            LOAD_CONST           3 ('Hello ,World!')
            CALL_FUNCTION        1
            POP_TOP
L24:
            LOAD_CONST           (None)
            RETURN_VALUE

then I save it in the test.pyasm file
and when i run python xasm an error pops up like this

$ pyc-xasm tes.pyasm
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/pyc-xasm", line 11, in <module>
    load_entry_point('xasm==1.1.0', 'console_scripts', 'pyc-xasm')()
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/xasm/xasm_cli.py", line 28, in main
    asm = asm_file(asm_path)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/xasm/assemble.py", line 124, in asm_file
    asm = Assembler(python_version)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/xasm/assemble.py", line 52, in __init__
    self.opc = get_opcode(python_version)
TypeError: get_opcode() missing 1 required positional argument: 'is_pypy'

it looks like the real error is in the main.py file in the xdis package in this get_opcode function

def get_opcode(version, is_pypy):
 49     # Set up disassembler with the right opcodes
 50     if type(version) in (list, tuple):
 51         version = ".".join([str(x) for x in version])
 52     lookup = str(version)
 53     if is_pypy:
 54         lookup += "pypy"
 55     if lookup in op_imports.keys():
 56         return op_imports[lookup]
 57     if is_pypy:
 58         pypy_str = " for pypy"
 59     else:
 60         pypy_str = ""
 61     raise TypeError("%s is not a Python version%s I know about" % (version, pypy_str))

if you have a little time please fix it right away, because this tool is really really good sir, I've tried it in python2.7.18 and didn't get any errors, but I'm a little disappointed because I thought this tool could only be used in python2

if you are too busy, just tell me what to fix!
Thanks

AssertionError

hi
i ran this command on a pyc file

pydysasm -F xasm mifile.pyc > myfile.pyasm

when i ran this

pyc-xasm myfile.pyasm i get this error

Traceback (most recent call last):
File "/usr/local/bin/pyc-xasm", line 33, in
sys.exit(load_entry_point('xasm==1.2.0', 'console_scripts', 'pyc-xasm')())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/xasm-1.2.0-py3.10.egg/xasm/xasm_cli.py", line 28, in main
File "/usr/local/lib/python3.10/site-packages/xasm-1.2.0-py3.10.egg/xasm/assemble.py", line 290, in asm_file
AssertionError: File needs to start out with: # Python bytecode

RuntimeError in pyc-xasm

I want to compile a .pyasm file to .pyc via using pyc-xasm

This is my main.py

print("Hello, world")

I compiled this as main.pyc

And

pydisasm main.pyc > main.pyasm

Now I want to convert the file back

pyc-xasm main.pyasm

RuntimeError: Don't understand operand ('Hello, expecting int or (..). Instruction 1: LOAD_CONST ('Hello,

Python API ?

I could not find it on the doc.

Is is possible to dynamically assemble/disassemble from within a program ? Something like:

import dis


instructions = dis.get_instructions(fn.__code__)
...  # Apply modifications to the bytecode
new_fn = assemble(instructions)

Context: I'm trying to run a function with closure on eval, which isn't supported. So I'm trying to replace those closure by globals.

Error with Python 3.8.10 assemble file

pydisasm version 5.0.10
Python bytecode 3.8 (3413)
Disassembled from Python 3.8.10 (default, May 5 2021, 03:01:07)

i use pydisasm --format xasm but i got error when i try to compile back pyc

Traceback (most recent call last):
  File "c:\users\m0hamed\desktop\python-xasm-master\python-xasm-master\xasm\assemble.py", line 419, in create_code
    inst.arg = label[target] - offset
KeyError: '(to'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\m0hamed\AppData\Local\Programs\Python\Python38-32\Scripts\pyc-xasm-script.py", line 11, in <module>
    load_entry_point('xasm', 'console_scripts', 'pyc-xasm')()
  File "c:\users\m0hamed\appdata\local\programs\python\python38-32\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\m0hamed\appdata\local\programs\python\python38-32\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\m0hamed\appdata\local\programs\python\python38-32\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\m0hamed\appdata\local\programs\python\python38-32\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\m0hamed\desktop\python-xasm-master\python-xasm-master\xasm\xasm_cli.py", line 28, in main
    asm = asm_file(asm_path)
  File "c:\users\m0hamed\desktop\python-xasm-master\python-xasm-master\xasm\assemble.py", line 150, in asm_file
    co = create_code(asm, label, backpatch_inst)
  File "c:\users\m0hamed\desktop\python-xasm-master\python-xasm-master\xasm\assemble.py", line 425, in create_code
    err("Label %s not found" % target, inst, i)
  File "c:\users\m0hamed\desktop\python-xasm-master\python-xasm-master\xasm\assemble.py", line 364, in err
    raise RuntimeError(msg)
RuntimeError: Label (to not found. Instruction 0:
 188: SETUP_FINALLY     (to

pyasm file here
https://we.tl/t-ntUBq3RuHD

Any idea ???

Error while 'make check'

make: py.test: No such file or directory
make: *** [Makefile:14: test] Error 127

How do i fix it?

Output from xdis doesn't work when assembling with xasm

Hello,

I ran this command:
pydisasm --format xasm myfile.pyc > test.pyasm

then the next command:
py run.py (I made this file because of some weirdness with my python versions, just assume it executes the same way as pyc-xasm test.pyasm)

The problem is with the jumps:
RuntimeError: Label (to 16) not found. I know about { POP_JUMP_IF_FALSE (to 16), JUMP_ABSOLUTE (to 16)}. Instruction 24: POP_JUMP_IF_FALSE (to 16)

The bytecode looks like this
POP_JUMP_IF_FALSE (to 16)

For some reason it thinks (to 16) is a label, while it's an offset. In your HOW-TO-USE.rst you specify that offsets can just be written like this: POP_JUMP_IF_FALSE 16

I could just make a regex and replace it like that to make it work but that feels a bit hacky. Is there any proper solution?

Convenient API to use it from python is strongly needed.

can't find method

I got this problem: can't find method greet
appending bogus greet to list of constants

when running pyc-xasm --pyc-file test .\compiled_code.pyasm

this is the code that generated compiled_code.pyasm

source_code = """
def greet():
    print("Hello, world!")
"""

compiled_code = compile(source_code, "<string>", "exec")
pyc_file = "compiled_code.pyasm"
with open(pyc_file, "w") as file:
    xdis.std.disassemble(compiled_code, file=file)

and this is the compiled_code:

# pydisasm version 6.0.5
# Python bytecode 3.9 ()
# Disassembled from Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)]
# Method Name:       <module>
# Filename:          <string>
# Argument count:    0
# Position-only argument count: 0
# Keyword-only arguments: 0
# Number of locals:  0
# Stack size:        2
# Flags:             0x00000040 (NOFREE)
# First Line:        2
# Constants:
#    0: <code object greet at 0x000002028CC845B0, file "<string>", line 2>
#    1: 'greet'
#    2: None
# Names:
#    0: greet
  2:           0 LOAD_CONST           (<code object greet at 0x000002028CC845B0, file "<string>", line 2>)
               2 LOAD_CONST           ('greet')
               4 MAKE_FUNCTION        (Neither defaults, keyword-only args, annotations, nor closures)
               6 STORE_NAME           (greet)
               8 LOAD_CONST           (None)
              10 RETURN_VALUE


# Method Name:       greet
# Filename:          <string>
# Argument count:    0
# Position-only argument count: 0
# Keyword-only arguments: 0
# Number of locals:  0
# Stack size:        2
# Flags:             0x00000043 (NOFREE | NEWLOCALS | OPTIMIZED)
# First Line:        2
# Constants:
#    0: None
#    1: 'Hello, world!'
# Names:
#    0: print
  3:           0 LOAD_GLOBAL          (print)
               2 LOAD_CONST           ('Hello, world!')
               4 CALL_FUNCTION        1
               6 POP_TOP
               8 LOAD_CONST           (None)
              10 RETURN_VALUE

python_version is None

pyc-xasm.exe "tasm-3.5.pyasm"

 File "d:\program files (x86)\python\lib\site-packages\xasm\xasm_cli.py", line 28, in main
    asm = asm_file(asm_path)
  File "d:\program files (x86)\python\lib\site-packages\xasm\assemble.py", line 158, in asm_file
    python_version_pair = version_str_to_tuple(python_version, len=2)
  File "d:\program files (x86)\python\lib\site-packages\xdis\version_info.py", line 54, in version_str_to_tuple
    return tuple([int(v) for v in python_version.split(".")[:len]])
AttributeError: 'NoneType' object has no attribute 'split'

in file "assemble" line 158, python_version = version that I added one line, and then worked normally.

Incorrect version check in `create_code`?

I'm not sure whether this is a bug, or intended functionality, but I figured I would post it anyway.

In xasm/assembly.py's create_code function it uses the assembler's bytecode version, instead of the current interpreter version, to determine whether co_code should be a bytes-like object or a str.

if asm.opc.version >= 3.0:
    co_code = bytearray()
    for j in bcode:
        co_code.append(j)
    asm.code.co_code = bytes(co_code)
else:
    asm.code.co_code = ''.join([chr(j) for j in bcode])

In my specific case, I'm assembling Python 2.7 instructions from a Python 3.7 interpreter, and passing the resulting code object to uncompyle6, which expects a bytes-like co_code, so I have to convert it first:

code = create_code(asm, [], [])

# HACK/FIX: xasm's `create_code` makes `co_code` a str on opcode version < 3,
# and uncompyle6 requires this to be bytes-like, so we convert it.
if sys.version_info > (3, 0, 0):
    code.co_code = bytes([ord(c) for c in code.co_code])

uncompyle6.main.decompile(version, code, sys.stdout)

If this is intended functionally, feel free to close the issue or tell me and I'll close it. :)

pyc-xasm KeyError: '0'

In the python interactive command line, I try to output bytecode via dis.dis(my_function_bytecode)

>>> sys.version
'2.6.2 (r262:71600, Dec 16 2020, 13:15:02) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-1)]'
>>> import aaa.bbb.ccc as ttt
>>> import dis
>>> dis.dis(ttt.handler.__code__.co_code)
          0 LOAD_GLOBAL         0 (0)
          3 LOAD_CONST          1 (1)
          6 LOAD_CONST          2 (2)
          9 CALL_FUNCTION       2
         12 STORE_FAST          1 (1)
         15 LOAD_GLOBAL         0 (0)

Copy the output to the test.pyasm file and add the 'Python bytecode 2.6 (62131)' header

# Python bytecode 2.6 (62131)
          0 LOAD_GLOBAL         0 (0)
          3 LOAD_CONST          1 (1)
          6 LOAD_CONST          2 (2)
          9 CALL_FUNCTION       2
         12 STORE_FAST          1 (1)
         15 LOAD_GLOBAL         0 (0)
         18 LOAD_CONST          1 (1)
         21 LOAD_CONST          2 (2)
         24 CALL_FUNCTION       2
         27 STORE_FAST          2 (2)
         30 LOAD_GLOBAL         0 (0)
         33 LOAD_CONST          1 (1)
         36 LOAD_CONST          2 (2)
         39 CALL_FUNCTION       2
        ......
       1908 BINARY_MODULO  
       1909 LOAD_CONST         26 (26)
       1912 LOAD_ATTR          81 (81)
       1915 LOAD_FAST          26 (26)
       1918 LOAD_CONST         52 (52)
       1921 LOAD_CONST          4 (4)
       1924 SLICE+3        
       1925 CALL_FUNCTION       1
       1928 CALL_FUNCTION       2
       1931 POP_TOP        

I tried to convert 'test.pyasm' to pyc by using pyc-xasm, but the following error appears. How should I fix it

➜  python-xasm git:(master) ✗ ./pyc-xasm ibapapachemod_1.pyasm
Traceback (most recent call last):
  File "./pyc-xasm", line 33, in <module>
    sys.exit(load_entry_point('xasm', 'console_scripts', 'pyc-xasm')())
  File "/opt/homebrew/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/poom/Downloads/python-xasm/xasm/xasm_cli.py", line 28, in main
    asm = asm_file(asm_path)
  File "/Users/poom/Downloads/python-xasm/xasm/assemble.py", line 326, in asm_file
    opname, operand = get_opname_operand(asm.opc, fields[1:])
  File "/Users/poom/Downloads/python-xasm/xasm/assemble.py", line 39, in get_opname_operand
    if opc.opmap[opname] < opc.HAVE_ARGUMENT:
KeyError: 'SLICE+3'
➜  python-xasm git:(master) ✗

PEP 552 Header for Python >= 3.7

.pyc files from Python versions 3.7 and above have a 16 byte header (PEP 552) as opposed to 12 bytes in Python 3.3-3.6.

The write_pyc file function doesn't take this into account.

python-xasm/xasm/write_pyc.py

Lines 8 to 26 in f88191b

	def write_pycfile(fp, code_list, timestamp=None,
	version=xdis.PYTHON_VERSION):
	magic_bytes = magics[version]
	magic_int = magic2int(magic_bytes)
	fp.write(magic_bytes)
	if timestamp is None:
	timestamp = int(time.time())
	if magic_int == 3393:
	fp.write(pack('I', timestamp))
	fp.write(pack('I', 0))
	elif magic_int in (3394, 3401, 3412, 3413, 3422):
	fp.write(pack('<I', 0)) # pep552_bits
	else:
	fp.write(pack('<I', timestamp))
	if version > 3.2:
	fp.write(pack('<I', 0)) # size mod 2**32

How To Convert Bytes to __code__

Let's say I had the following python bytes:

def func1():
    print(1)

print(func1.__code__.co_code)
b't\x00d\x01\x83\x01\x01\x00d\x00S\x00'

So let's assume all I had/knew is b't\x00d\x01\x83\x01\x01\x00d\x00S\x00'

How would I (using only those bytes) be able to assemble those bytes into a code object <code object <module> at 0x7fa973e55d40, file "", line 1> and assign it to another functions __code__. For example, func2.__code__ = assembled_code_object_from_bytes?

Can't decomplie

link.zip
Can't decomplie with this pyc.
Thanks!

AttributeError: 'NoneType' object has no attribute 'split'

anyone know what i did wrong?

.pyasm
of.zip

Traceback (most recent call last):
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\D-Com\AppData\Local\Programs\Python\Python37\Scripts\pyc-xasm.exe\__main__.py", line 7, in <module>
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\xasm\xasm_cli.py", line 28, in main
    asm = asm_file(asm_path)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\xasm\assemble.py", line 158, in asm_file
    python_version_pair = version_str_to_tuple(python_version, len=2)
  File "c:\users\d-com\appdata\local\programs\python\python37\lib\site-packages\xdis\version_info.py", line 54, in version_str_to_tuple
    return tuple([int(v) for v in python_version.split(".")[:len]])
AttributeError: 'NoneType' object has no attribute 'split'

Look over and fix up SIP hash in 3.7+