xboxdev / xboxpy

Python module to interface with original Xbox hard- and software

XboxDev

XboxDev is the place for all original Xbox projects to come together.

XboxDev is an effort to provide tools and documentation for the original Microsoft Xbox and Xbox-based SEGA Chihiro. We not only care about technical details about those platforms, but also how to develop homebrew and emulate them.

We are not interested in documenting the Xbox 360, Xbox One or any other console at this point.

You can create an issue to contact XboxDev maintainers.

Ecosystem

We currently provide the following services:

• Discord for communication
• XboxDevWiki to document research
• GitHub organization to host tools and files

Want to help? Contact us!

Please come chat with us:

• XboxDev on Discord

and for closely related projects:

• XQEMU: #xqemu on Freenode IRC (Webchat)
• Cxbx-Reloaded: Discord

You'll also get the details for XboxDevWiki account creation on any of these channels.

Coding style

The usual rule is: follow the existing code / commit style of whatever you are working on. If the previous commits use prefix style: use prefix style. If the surrounding code respects 80 column limit: respect it. If the surrounding code uses an unusual C style: mimic it.

Disclaimer

XboxDev is not affiliated with, endorsed by, or sponsored by Microsoft Corporation, Nvidia Corporation, licensed Xbox developers, publishers, manufacturers or any of their affiliates or subsidiaries. All product and company names are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Reverse engineering of the original software and hardware is done to achieve interoperability with other computing platforms. In the process, excerpts of the reverse engineered source code will be shown for educational purposes.

No copyright infringement is intended at any stage during creation of XboxDev software, hardware or documentation.

xboxpy's Issues

XBDM hack is slow

Current code (eb69c7fba9cdd1bb1e81eec60f4fed4394a42c67):

def xbdm_hack(address, operation, data=0):
  SetMem(hack_bank, struct.pack("<III", address, operation, data))
  xbdm_command("resume thread=0x" + format(hack_bank, 'X'))
  return GetMem(hack_bank + 8, 4)

(Also see xbdm-hack.md for more information)

Here is what the hack does for reads and writes:

Reads

def xbdm_read_8(address):
  return xbdm_hack(address, 1)

1. xbdm_hack will call SetMem, which is the first communication with Xbox, to setup the next step
2. xbdm_hack will call xbdm_command which is another communiation with Xbox
3. xbdm_hack will readback the result which is another communication with Xbox

That is 3 back-and-forth transfers for a single read.

Writes

def xbdm_write_8(address, data):
  xbdm_hack(address, 4, int.from_bytes(data, byteorder='little', signed=False))

1. xbdm_hack will call SetMem, which is the first communication with Xbox, to setup the next step
2. xbdm_hack will call xbdm_command which is another communiation with Xbox
3. xbdm_hack will readback the result which is another communication with Xbox, even if it isn't returned from xbdm_write.

That is 3 back-and-forth transfers for a single read, with one of them being completly useless.
There's various ways to optimize it.

Calls

def xbdm_call(address, stack):
  assert(len(stack) < 64)
  SetMem(hack_bank + 12, stack)
  return xbdm_hack(address, 7, len(stack))

1. xbdm_call will call SetMem, which is the first communication with Xbox
2. xbdm_hack will call SetMem, which is the more communication with Xbox, to setup the next step
3. xbdm_hack will call xbdm_command which is another communiation with Xbox
4. xbdm_hack will readback the result which is another communication with Xbox

---

Ideally we'd pass all optional input data in the xbdm_hack request, and return all optional output data in the response.
Maybe we should also allow packing of commands, simply to avoid XBDM command processing overhead.

Refactor interface selection

I don't remember how the interface function overloads / replacing original functions work, but it's quite hacky.

There should be an object oriented interface to each backend, and we should create an object to access the xbox with that specific interface.
I have a fake interface in nv2a-trace already

There should also be a helper interface to auto-decide the interface for the user (this is the primary job of xboxpy.py: convience loader for gdb sessions, as xboxpy is not in module path).

call() in XBDM interface supports limited stack size of 64 bytes

Due to #15, there is a limitation of 64 bytes per call: assert(len(stack) < 64)

This issue will be closed when #15 closes; it's just here for visibility (as users might not find the more technical one)

xboxpy is hard to test

As xboxpy is meant as a library, it's currently not easy to test.

For now, I'd recommend pushing unit tests to my python-scripts collection until we have a dedicated place for this.

The gdb interface is hard to use and quite limited in terms of performance and features.
For running on a real Xbox you'll also need a modified fork of nxdk-rdt or you need xbdm (both not easy to obtain or install).

Add support for Name Answering Protocol in XBDM interface

See this article for more information.

We should implement this independently of the rest of xboxpy.

We'll probably want a better URL scheme or other interface to xboxpy for the following, so that will be a seperate issue:
If a user wants to connect to an Xbox target, we could re-use that NAP implementation.
Discovery could be used to see if only one Xbox responds, so we can just use it as target.
If a name is supplied we could also use that name (although this might conflict with DNS).

SetMem in XBDM interface can cause stack overflow

Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007f4ceb3d7600 (most recent call first):
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 20 in xbdm_read_line
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 31 in xbdm_parse_response2
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 67 in xbdm_parse_response
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 74 in xbdm_command
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 138 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 132 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
  File "/home/fox/Data/Projects/XboxDev/xboxpy/xboxpy/interface/if_xbdm.py", line 133 in SetMem
[...]

Should be rewritten iteratively.

Refactor how imports work

Example:

from . import *
from . import memory
from .apu_regs import *
from .aci import export_wav

I'm not a python coder, but I assume this is quite horrible..

Provide documentation

The code using this module was left at https://github.com/JayFoxRox/xbox-tools/tree/master/python-scripts. We should clean up these samples and add them to the xboxpy ecosystem.

Alternatively, we should just document all functions properly, generate documentation and just refer to bigger projects using xboxpy (such as an NV2A or APU debugger).

The later option is actually what I'd prefer, to avoid having the problem of "is this an example or a seperate tool"

XBDM interface does not support `call`

I don't see any code which implements call for XBDM (which surprises me). We should implement this.

Add pixel conversion code

There should be routines which convert NV2A pixel data to a more usable format. Possibly PIL images or A8R8G8B8 bytes(). I prefer the PIL solution because almost everyone will have it anyway, and it supports DXT.
PIL could be an optional dependency.

There's work in nv2a-trace which will eventually move here. See Texture.py in nv2a-trace (current version linked, please look where file has moved in master branch)
Please work on nv2a-trace if you want to solve this issue.

There is also some dirty unswizzling code in nv2a.py which must be looked at, see #6

call() does not preserve float registers

If you intend to use floats in your calls, you have to manually clean-up and restore.

It has not been decided what the final calling convention for our call() is.
For now, assume that you always have to clean up the FPU, and this is a documentation issue.

There sould be a comment in the code, or some docs which make this clear. The GitHub issue is a temporary measure.
Alternatively, if you feel strongly about this, you can implement float cleanup. But I personally believe it would overcomplicate an already complicated assembler function.

XBDM hack corrupts xbdm.dll

Our current hack uses the xbdm.dll PE header for it's communication needs.

@Ernegien suggested to use the xbdm.dll relocation section for our scratch space instead.
According to him we'll have at least 4kb across any version.
I think it's a great idea as the PE header might be re-used, but it's unlikely the .reloc will be re-used (or does XBDM support relocation?).
Alternatively we could look for unused padding space.

We can probably use the xbdm modsection command to locate both of these - it's also supported in every XBDM version.

---

We also overwrite the code of the thread resume function DmResumeThread. Instead, we should just add a lightweight hook which re-enters the original function, so we are less invasive.
Our code should use the same memory area as the communication space, and ensure that we never consume more memory than available.
We should also switch to another function which is available in every XBDM version.

Many kernel exports missing

xboxpy/ke.py can be used to retrieve pointers to kernel variables, or call kernel functions.
However, a lot of those are not implemented yet.

For an overview of all kerne exports, see our article in XboxDevWiki.

This issue can be implemented in smaller steps. It's an ideal beginner task as it's mostly copy/paste.

The conversion / interface creation could also be automated. @mborgerson has made a tool for OpenXBOX which can parse kernel functions (including type information).

Swizzling in NV2A is broken

There are 2 unswizzling routines, one is severly limited and was taken from screenshot code by @Ernegien . It only supports 640x480 with pitch of 2560 and 32 bpp.

I don't remember where I took the other algorithm from, but it doesn't seem to work for the surfaces.

I wonder if there are different swizzling modes or surface/textures differences, such as tiling.
Overall, the code should be rewritten and get more testing.

XBDM interface incompatible with older versions of XBDM

Installing the XBDM hack requires GetMem2 (called from read_u32 in resolve_export).

This was only added in version 5028.

A fallback in xboxpy to GetMem exists, but it's disabled.
We should also warn the user because this has severe performance implications.

xboxpy/interface/dbg_pb2.py ought to be generated

The current supplied version of dbg_pb2.py did not work (at least for me), but supplying my own generated one from running make in nxdk-rdt works fine.

JayFoxRox: so the proposed fix / test is to generate dbg_pb2.py using your protobuf [as was done during make for nxdk-rdt] and then copy it into your xboxpy installation

XBDM hack corrupts xboxkrnl.exe

Our call command from #12 uses the xboxkrnl.exe header at 0x80010000. We should instead improve the assembly to use code-relative addressing. The easiest is probably by writing a pointer to the communication space to the code and just loading the pointer from that. Alternatively, the old call f; f: pop eax trick.

Release as pip package

We should work towards shipping this as a pip module, so installation is a simple one-liner.