gpdaniels / spike-prime Goto Github PK

The ideal end goal for the decompiler would be to make it produce micropython code that could be run in the place of the decompiled mpy file.

Hi!

I've been working on understanding Spike Prime Hub internals as well.
Were you able to decompile the .mpy files or reverse engineer the ujson protocol?

I theorycrafted a way to dump the contents of the built-in hub module to hopefully be able to implement a full-featured open-source firmware for the LEGO SPIKE Hub.
Here is how it works (on paper):

# The hub module
import hub
# For memory access
import machine
module_ptr = id(hub)
# module->globals.map.table[0]
name_ptr = machine.mem32[machine.mem32[machine.mem32[module_ptr + 4] + 12] + 0 * 8]
data_ptr = machine.mem32[machine.mem32[machine.mem32[module_ptr + 4] + 12] + 0 * 8 + 4]
def read_str(ptr):
    len = machine.mem32[ptr + 8]
    addr = machine.mem32[ptr + 12]
    return bytes_at(addr, len).decode()
print(read_str(name_ptr))

Known sensors

• Large angular motor
• Medium angular motor
• Colour sensor
• Depth sensor
• Force sensor

Will need tiles for the motors, colour sensor, and depth sensor as they are outputs.

Using the dump-api.py script from the simulator it's possible to get what I hope is all the functions available on the hub (along with lots of other information)

What is needed:

• See if it possible to determine all the built in modules, help("modules") prints to stdout, but on the hub this cannot be redirected because the micropython version is too old.
• Extract recursively hub modules/classes/functions/variables.
• Search the hub filesystem for any further modules/classes/functions.
• Determine the arguments to all these modules/classes/functions (I think the mpy tool might be able to help with some).
• Extract the values of all variables discovered.
• Generate stub (or filled if possible) modules/classes/functions/variables for all functions on the hub.
• Link remaining functions to the simulator virtual hub API.

These installers don't seem to be available on the Lego education webiste: https://education.lego.com/en-us/downloads/spike-prime/software

I have no idea what firmwares were included within these installers.

Hi everyone,
I have a problem with rshell. Though I do know that you most certainly won't be able to solve it, I am hoping you might have an idea who I could contact to solve this, as you seem far more experienced hacking around with mindstorms.

I just got the new mindstorms and started poking around with it. I have no experience with micropython and read about rshell here for the first time.
Unfortunately I have a pretty fundamental problem. Whenever I try to list the /pyboard directory using rshell, an error is thrown, as the result cannot be parsed. It seems like the result from the helper functions print is incomplete and the beginning is missing.
I already noticed this behavior a couple of days before when I was just using screen to connect to the mindstorms and started printing the content of files out of curiosity whats going on on the mindstorms: In my helper functions I had to read the files line by line and use separate print commands for each line to get the complete file content. Otherwise I had the same behavior that I was just seeing the end of the files. It seems like the mindstorms print command has some hiccups, too small buffer, I don't know..
Do you have any idea who to contact to somehow solve this?

In case you are interested, this is how the debug output of rshell looks like

/Users/mschulte> ls /pyboard/
Executing "ls /pyboard/"
----- About to send 634 bytes of code to the pyboard -----
def stat(filename):
    """Returns os.stat for a given file, adjusting the timestamps as appropriate."""
    import os
    try:
        # on the host, lstat won't try to follow symlinks
        rstat = os.lstat(filename)
    except:
        rstat = os.stat(filename)
    return rstat[:7] + tuple(tim + 946681200 for tim in rstat[7:])


def get_stat(filename):
    """Returns the stat array for a given file. Returns all 0's if the file
       doesn't exist.
    """
    try:
        return stat(filename)
    except OSError:
        return (0,) * 10
output = get_stat('/')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b'(16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)\r\n'
-----
----- About to send 1207 bytes of code to the pyboard -----
def is_visible(filename):
    """Determines if the file should be considered to be a non-hidden file."""
    return filename[0] != '.' and filename[-1] != '~'


def stat(filename):
    """Returns os.stat for a given file, adjusting the timestamps as appropriate."""
    import os
    try:
        # on the host, lstat won't try to follow symlinks
        rstat = os.lstat(filename)
    except:
        rstat = os.stat(filename)
    return rstat[:7] + tuple(tim + 946681200 for tim in rstat[7:])


def listdir_stat(dirname, show_hidden=True):
    """Returns a list of tuples for each file contained in the named
       directory, or None if the directory does not exist. Each tuple
       contains the filename, followed by the tuple returned by
       calling os.stat on the filename.
    """
    import os
    try:
        files = os.listdir(dirname)
    except OSError:
        return None
    if dirname == '/':
        return list((file, stat('/' + file)) for file in files if is_visible(file) or show_hidden)
    return list((file, stat(dirname + '/' + file)) for file in files if is_visible(file) or show_hidden)
output = listdir_stat('/')
if output is None:
    print("None")
else:
    print(output)

-----
-----Response-----
b"00, 946681200)), ('sounds', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('spike', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('ui', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('hub_runtime.mpy', (32768, 0, 0, 0, 0, 0, 2015, 946681200, 946681200, 946681200)), ('version.py', (32768, 0, 0, 0, 0, 0, 54, 946681200, 946681200, 946681200)), ('.runtime_hash', (32768, 0, 0, 0, 0, 0, 953, 946681200, 946681200, 946681200)), ('extra_files', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('.extra_files_hash', (32768, 0, 0, 0, 0, 0, 539, 946681200, 946681200, 946681200))]\r\n"
-----
Traceback (most recent call last):
  File "/usr/local/bin/rshell", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/rshell/command_line.py", line 4, in main
    rshell.main.main()
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 2966, in main
    real_main()
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 2949, in real_main
    shell.cmdloop(cmd_line)
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 1755, in cmdloop
    cmd.Cmd.cmdloop(self)
  File "/usr/local/Cellar/[email protected]/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 1785, in onecmd
    self.onecmd_exec("".join(group))
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 1796, in onecmd_exec
    return cmd.Cmd.onecmd(self, line)
  File "/usr/local/Cellar/[email protected]/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/cmd.py", line 217, in onecmd
    return func(arg)
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 2423, in do_ls
    ldir_stat = auto(listdir_stat, filename)
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 588, in auto
    return dev.remote_eval(func, dev_filename, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rshell/main.py", line 1515, in remote_eval
    return eval(self.remote(func, *args, **kwargs))
  File "<string>", line 1
    00, 946681200)), ('sounds', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('spike', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('ui', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('hub_runtime.mpy', (32768, 0, 0, 0, 0, 0, 2015, 946681200, 946681200, 946681200)), ('version.py', (32768, 0, 0, 0, 0, 0, 54, 946681200, 946681200, 946681200)), ('.runtime_hash', (32768, 0, 0, 0, 0, 0, 953, 946681200, 946681200, 946681200)), ('extra_files', (16384, 0, 0, 0, 0, 0, 0, 946681200, 946681200, 946681200)), ('.extra_files_hash', (32768, 0, 0, 0, 0, 0, 539, 946681200, 946681200, 946681200))]
                 ^
SyntaxError: unmatched ')'

I was hoping you would know what firmwares include the ubluetooth library (e.g.import ubluetooth). I’ve read that it is included in recent Spike firmwares, but I can’t import it on the new 51515 controller I own. https://www.eurobricks.com/forum/index.php?%2Fforums%2Ftopic%2F180029-spike-prime-powerup-remote-in-python-noob-needs-help%2F

• Small hub
• Small motor
• 3x3 display

Hi @gpdaniels ,
Can you explain to me how to dump the Spike Prime Hub firmware?

When I execute the decompile the decompiled .py files contain less than expected code. Many of the files when I decompile them result in .py files containing output similar to the following event_loop.py where the file has imports, a declaration, then return none.

import utimeq
import collections
EventLoop = __build_class__(0, "EventLoop")
return None

Other files have lines like the following

$push(3)
BROKEN_OP`eq`
BROKEN_OP`btrue 147`
$push(hasattr(hub, "display"))

I was expecting to see more information like class, methods, variables, conditional statements, loops, etc.

Is the decompiler working as expected with the output I am getting?

Hi, I'm having a problem with the container.sh script that I didn't have before and a friend of mine is encountering the same problem. In the past the script used to work correctly but now it looks like is having some problems while initializing the builda.

I made the mistake of updating the firmware using the new SPIKE 3 software ... now I can't run my Python scripts, AND I can't reprogram back to the previous firmware! Yikes! Is there any way to revert to a previous firmware using a hardware reset?

Desktop app:

• GUI
• BT connection
• BLE connection

Android app:

• GUI
• BT connection
• BLE connection
• Permissions
• Running in the background

  I found the following in my notes (it has been a while), but it would be good if someone wants to confirm:

• Hold left during boot: Do not mount file system, so skip executing boot.py and main.py.
• Hold right during boot: Mount file system, run boot.py but skip main.py.
• Hold left + right: Mount, but then reset the file system, creating a new boot.py and main.py.

I don't know if left+right will work if the file system is already corrupted. If it's the same as the Pyboard reset then it should probably work.

And if not, you should still be able to hold only left, and then use the REPL to reset the filesystem instead.

Originally posted by @laurensvalk in #41 (reply in thread)

Is it still possible to write uncompiled .py files like in the 1.0 firmware?
I'd like to write a program in micro-python and I want it to be launchable from the default menu. (Adding a program-picture etc.)

my spike prime keeps corrupting the file and showing heads up could not open file

The new firmware is supposed to have some significant changes at MicroPython level.

I only have the Robot Inventor hub, but the Spike app 3.0 won't update it. I've heard the Spike firmware does run on Robot Inventor.

Do you know of any problems flashing a Spike firmware onto a 51515 hub? Is this thought to be a bad idea? (I haven’t tried it yet)

Current issues:

• Some operations are not handled correctly by the disassembler.
• Not all required information (for decompilation) is output by the disassembler.
• The decompiler doesn't handle all operations.

If you have a brand new SPIKE Prime hub running the below script will allow you to copy the firmware from the device.
This requires a linux machine (I used ubuntu 18.04)

# Install the dfu-util program if you do no already have it.
# sudo apt-get install dfu-util

# List the connected devices, this should show the hub is connected.
sudo dfu-util --list

# Read the firmware.
sudo dfu-util --alt 0 --dfuse-address 0x08000000:1048576 --upload ~/firmware.bin

# Take ownership of the file created by the dfu-util program.
sudo chown $(whoami) ~/firmware.bin

# Remove the first 0x8000 bytes, as these are before the firmware starts.
fallocate -c -o 0 -l 32768 ~/firmware.bin

# Remove the trailing bytes that have the value 0xFF, as these are unnecessary.
sed -i '$ s/\xFF*$//' ~/firmware.bin

# Rename the firmware to the md5 hash of itself.
mv ~/firmware.bin ~/$(md5sum ~/firmware.bin | awk '{ print $1 }').bin

I foolishly updated my hub before ripping the firmware.

Do you have the complete product code of the STM32F413 chip used for the Spike Hub?
STM32F413?? ?
Have you also a picture of the main board?

• 45609 LEGO Technic Small Hub
• 45607 LEGO Technic Small Angular Motor
• 45608 LEGO Technic 3×3 Color Light Matrix
• 45612 LEGO Technic Small Hub Battery

There's now a small version of the SPIKE prime hub which probably runs a different firmware and has a different filesystem.
See: https://education.lego.com/en-us/products/lego-technic-small-hub/45609

It would be good to keep a record of these.

If you have one please get in touch!

Previously: #14

Is this still possible, if so, how? I looked around but couldn't get the usual (official) suspects to work.
I even tried flashing it with the pybricks web flasher, didn't seem to work.

Python lego mindstorms has some official docs. Could they be added here? (just pull them out of the apk)

Hello, First of all, thank you so much for this great plugin. I am trying to connect lego minestrone inventor with this plugin via VS code. Unfortunately, I am getting this Connecting to Hub Failed! Opening COM8: File not found.

Can you please look into this issue?

Hey I just found this repo, and it seems like you've been doing really amazing work.

I've been working on writing an upload script that works on Linux.
I documented the effort in an issue on this other repo I found that was documenting the hub internals:
azzieg/mindstorms-inventor#2

I've just successfully uploaded my first program:
https://gist.github.com/pepijndevos/9d55dd4b53f630186545fa6361fc229a

Feel free to integrate this code in any way you like.

Are you aware of where the AFOL Mindstorms community hangs out? Would be great to share info as we learn more.

Is there a way that I can flash firmware on the LEGO Spike Prime over serial communication? The idea is to update the firmware without the LEGO desktop app.
And the next question would be how I can get the updated firmware.