openxc / openxc-python

OpenXC Python library

OpenXC for Python

Version:	2.2.0
Web:	http://openxcplatform.com
Download:	http://pypi.python.org/pypi/openxc/
Documentation:	http://python.openxcplatform.com
Source:	http://github.com/openxc/openxc-python/
Keywords:	vehicle, openxc, python

The OpenXC Python library (for Python 3.6.7) provides an interface to vehicle data from the OpenXC Platform. The primary platform for OpenXC applications is Android, but for prototyping and testing, often it is preferrable to use a low-overhead environment like Python when developing.

In addition to a port of the Android library API, the package also contains a number of command-line tools for connecting to the CAN translator and manipulating previously recorded vehicle data.

If you are getting the error "ValueError: No backend available" on windows please reinstall your libusb0 driver using https://github.com/openxc/vi-windows-driver if you are in a envirment where you can not use an unsigned driver please use https://zadig.akeo.ie/

Due to changes in signals.cpp openxc-python Version 2.0.0 must be used with vi-firmware 8.0.0 or greater. Due to changes with large diagnostic responses Version 2.1.0 must be used with vi-firmware 8.1.0 or greater.

To package run "setup.py sdist bdist_wheel" to push to pypi run "python -m twine upload dist/*" Version files:

• CHANGELOG.rst
• README.rst
• openxc/version.py
• docs/index.rst

License

Copyright (c) 2012-2020 Ford Motor Company

Licensed under the BSD license.

openxc-python's Issues

Can't control endianness from JSON mapping

A report from Gabriel:

And one note regarding the signals.json documentation: you describe the bit order differences and offer a new setting for it. It would be great if we also could set the byte order. For my bus I had to switch to little Endian in canread.cpp:

float openxc::can::read::decodeSignal(CanSignal* signal, uint64_t data) {
    uint64_t rawValue = getBitField(data, signal->bitPosition,
            signal->bitSize, false);
    return rawValue * signal->factor + signal->offset;
}

Print log messages to stderr by default

The log mode is defaulting to off, but since you have to compile the VI firmware with DEBUG=1 to get log messages anything, we may as well default it to stderr on the Python side.

Want to Control the Order of Extra Sources

When using the "parent" structure in message set files, it has become necessary to control the order in which the extra_sources are copied into the signals.cpp file.

Odometer measurement showing up at "m" instead of "km" in openxc-dashboard

Add a 'raw' option to the dashboard to display CAN messages from the 'passthrough' format

The format right now: openxc/openxc-message-format#3

Message IDs in configuration files are case sensitive

They have to match the mapped database IDs exactly - we should do case insensitive comparisons or actually parse them as numbers and not strings.

Bit position calculation may be incorrect in code generation

A report from Gabriel:

there seems to be a bug in the signals.cpp generation. When I execute the new generator with my signals.json, the bit_position in SIGNALS is wrong:

CanSignal SIGNALS[][MAX_SIGNAL_COUNT] = {
{ // message set: post_messages

    {&CAN_MESSAGES[0][0], "some_value", -8, 16, 0.100000, 0.000000, 0.000000, 0.000000, 1, true, false, NULL, 0, false, NULL}, // some_value

...

When I manually replace the bit_position with the correct values, everything works fine.

Gracefully exit if the USB/BT connection is severed

Right now if you detach or reflash the VI while reading data from the Python library, an unhandled exception kills the program. This isn't a big deal, but if you're using the openxc-dashboard tool ncurses doesn't exit cleanly and your terminal is all screwed up.

Code generation of raw filtered CAN message

When configure a raw filtered CAN message and then generate signals.cpp the message Skipping disabled message None(message's hex value) is showed.
See below:

{   "buses": {
      "hs": {
            "controller": 1,
            "speed": 500000,
            "raw_can_mode": "filtered"
        }
    },
    "messages": {
      "0xCF00400": {
        "bus": "hs"
      }
    }
}

$ openxc-generate-firmware-code --message-set ./raw_filtered_can_eec1.txt > ./src/signals.cpp
Skipping disabled message None (0xcf00400)

Python 3.3 and curses.wrapper

The curses.wrapper module was folded back in to the main curses module in Python 3.3. It still exists as curses.wrapper in Python 3.2.

This issue causes openxc-dashboard to crash at line 11 of dashboard.py (import curses.wrapper)

Commenting out line 11 works as a (temporary) fix for the issue.

--corrupted flag has no effect

It should either be removed or made to do something - the previous behavior was to print out all data, even if it wasn't a valid JSON message.

Print a more helpful error message if there is no USB backend installed

Point to the documentation on how to install one.

Add examples of using the API to the docs

This lib is mostly focused on the command line tools, but underneath is a capable API that developers can use to build their own apps. As Paul pointed out in the Google Group, we could use some good starter examples in the docs.

Can't use pyserial as dep in setup.py using Python 3.2

Tracking upstream: https://sourceforge.net/tracker/?func=detail&aid=3574160&group_id=46487&atid=446302

I'm working around this by added a direct URL to the .tar.gz at PyPi in the dependency_links section. Nobody seems to be paying attention to the bug tracker for pyserial.

Support binary protobuf decoding from VI

This is working in the binary-encoding branch, but it needs cleanup and in the process I borked JSON decoding.

We also need to support recording and playing back protobuf trace files.

It'd be great if it could auto-detect the stream format (i.e. if it's all ASCII, it's probably JSON) but otherwise we'll need a flag.

When many messages are corrupted, dashboard is very slow

The buffer is probably growing very large because many messages are coming in corrupted, so each line takes a much longer time to process. I'm noticing this on the LPC1769 board, which is having some corruption issues on USB.

Automatically determine raw/translated type when writing messages

Support displaying diagnostic responses in the dashboard

Bit numbering inversion setting can't be overridden in a signal anymore

See https://groups.google.com/forum/#!topic/openxc/oh_1Z5qGqD4

For CLI commands with the --trace switch, specify if the tracefile is being written or if it's being read

Use struct field labels during initialization in generated firmware code

Particularly for all of the fields in each CanSignal object.

Extra Sources Not in Correct Location in Code Generator

In version 0.10.4, lines.extend(self._build_extra_sources()) was moved to later in the signals.cpp code. Needs to be bumped up before lines.extend(self._build_signals()) in order for vi-firmware to compile.

vi-firmware compile issue: error: 'customDecoder_in_extra_sources' was not declared in this scope

commit where extra_sources location was changed: 9b867af

fixed in https://github.com/openxc/openxc-python/tree/extra-sources-fix

Add a bit of filtering to dashboard display to make identifying changed signals easier

Because the dashboard updates so quickly, it can be easy to miss when a signal changes if it goes right back to the previous value. For example, when trying to identify a CAN message for a button press, the values over time might be 0 0 0 0 0 0 0 0 1 0 0 0 0 0 where it's marked as 'pressed' for only a split second. We'll have to think a bit about how to do this filtering without making it inaccurate or "floaty" feeling, but I imagine this is something others have worked on before.

Support sending protobuf encoded commands to VI

When openxc/vi-firmware#223 is implemented, we need to able to specify the outgoing format for commands sent from the Python library. The library already supports reading protobuf, and it automatically determines the format based on the data stream. Since we won't have a command stream to inspect, we need to add a flag to set the format.

log-mode "off" not working

I still get log output when using the flag --log-mode off with the CLI tools in the latest code.

Latest version of pip can't install because of pyusb versioning

The latest version of pip will not install "alpha" versions of packages (or dependencies) unless you use the --pre flag. pyusb is only available in an alpha version right now, so it will refuse to install the openxc package.

Ideally pyusb could release a stable version, otherwise we should pin this package to a specific version of pyusb and see if that helps.

Sort measurements in dashboard by name

Every time you start it they are in a potentially different order, which is kind of annoying.

Simplify JSON input files by reading states, types from XML

Improve Bluetooth device detection

A good example by Paul Craven: https://bitbucket.org/pcraven/openxc-monitor

Use specific name for a mapped signal if generic_name is not defined

Support for network socket connections.

There are a few things that stream data through network sockets. The Python libraries currently can not connect to those.

This should be a matter of adding a new source within the existing framework.

Only default to inverted bit mapping if using a database

Generally I've found when someone is adding bit position info manually, they're referring to an Excel spreadsheet they got from someone, and those have always had the non-inverted bit position.

USB error: Windows openxc testing

Getting an error on running openxc-dump on windows.

agonz122@QPL1W78Q4XVM1 /cygdrive/c/users/agonz122/Documents/OpenXC
$ openxc-dump
Traceback (most recent call last):
File "/usr/bin/openxc-dump", line 8, in
load_entry_point('openxc==0.9.1', 'console_scripts', 'openxc-dump')()
File "build/bdist.cygwin-1.7.17-i686/egg/openxc/tools/dump.py", line 38, in main
File "build/bdist.cygwin-1.7.17-i686/egg/openxc/sources/usb.py", line 53, in init
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 554, in set_configuration
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 91, in managed_set_configuration
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 70, in managed_open
File "build/bdist.cygwin-1.7.17-i686/egg/usb/backend/libusb1.py", line 494, in open_device
File "build/bdist.cygwin-1.7.17-i686/egg/usb/backend/libusb1.py", line 403, in _check
usb.core.USBError: [Errno 88] Operation not supported or unimplemented on this platform

agonz122@QPL1W78Q4XVM1 /cygdrive/c/users/agonz122/Documents/OpenXC
$ openxc-dashboard
Traceback (most recent call last):
File "/usr/bin/openxc-dashboard", line 8, in
load_entry_point('openxc==0.9.1', 'console_scripts', 'openxc-dashboard')()
File "build/bdist.cygwin-1.7.17-i686/egg/openxc/tools/dashboard.py", line 215, in main
File "/usr/lib/python2.6/curses/wrapper.py", line 43, in wrapper
return func(stdscr, _args, *_kwds)
File "build/bdist.cygwin-1.7.17-i686/egg/openxc/tools/dashboard.py", line 195, in run_dashboard
File "build/bdist.cygwin-1.7.17-i686/egg/openxc/sources/usb.py", line 53, in init
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 554, in set_configuration
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 91, in managed_set_configuration
File "build/bdist.cygwin-1.7.17-i686/egg/usb/core.py", line 70, in managed_open
File "build/bdist.cygwin-1.7.17-i686/egg/usb/backend/libusb1.py", line 494, in open_device
File "build/bdist.cygwin-1.7.17-i686/egg/usb/backend/libusb1.py", line 403, in _check
usb.core.USBError: [Errno 88] Operation not supported or unimplemented on this platform

Environment:

Windows 7 64bit
Cygwin with pythong, libusb, libusb-win32, easy_install, pip installed
openxc package installed by pip install https://github.com/openxc/openxc-python
libusb-win32 driver installed as well from INF file that was generated from the INF wizard.

Testing:

Load up cygwin
Connect kit to laptop using micro-usb
Run openxc-dump

requests package is required but not in dependencies

Just make it a dependency, it won't really hurt anything and it's easier than trying to document all of the possible permutations of deps you might need.

Dashboard is comparatively much slower in reading from USB vs. openxc-dump

When reading from a CAN translator programmed with the emulator firmware, the openxc-dump can read continuously and get no to very few dropped CAN messages reported by the firmware.

With the dashboard, we get a steady stream of dropped messages - albeit fewer messages that if it were totally unplugged from USB.

I don't think this is much of an issue in an actual vehicle (the emulator data right is artificially high) but it should be possible to construct the dashboard so it reads as fast as openxc-dump, even if it doesn't display every single update to the screen.

Show arbitrary measurements in dashboard

Add a "type" field to signal definition that sets both send/receive handlers

we must also still allow total overriding of the handlers in both directions

ASCII Dashboard percentage graph can get too large

I think if the value jumps too high (this is with a bad build) it can throw off the percentage calculation, and the ---------------=--------- is way too long.

Data from trace source is string, data from USB is bytes

The message parsing functions are expecting bytes right now, so this doesn't work in Python 3.

Dashboard app crashes in 2012 Focus when user pushes the 4 directional buttons on the steering wheel

Here's what happens:

1. Start the car
2. Run the dashboard app inside Linux terminal (everything looks good at this point)
3. Press button on 4-directional pad on steering wheel
4. Dashboard stops with the following error:

Exception in thread Thread-1:

                                             Traceback (most recent call last):

                                                                                 File "/usr/lib/python2.7/threading.py", line 551, in 
__bootstrap_inner


         self.run()

          File "/usr/lib/python2.7/threading.py", line 504, in run

                                                                      self.__target(*self.__args, **self.__kwargs)

                                                                                                                    File "/usr/local/lib/python2.7
/dist-packages/openxc-0.9.0-py2.7.egg/openxc/sources/base.py", line 29, in run

                                                                       data_remaining=len(message_buffer) > 0)

                                                                                                                File "/usr/local/lib/python2.7
/dist-packages/openxc-0.9.0-py2.7.egg/openxc/vehicle.py", line 38, in _receive

                                                                   sink.receive(measurement, **kwargs)

                                                                                                        File "/usr/local/lib/python2.7/dist-
packages/openxc-0.9.0-py2.7.egg/openxc/tools/dashboard.py", line 164, in receive

                                                                   self.elements[measurement.name].update(measurement)

                                                                                                                        File "/usr/local
/lib/python2.7/dist-packages/openxc-0.9.0-py2.7.egg/openxc/tools/dashboard.py", line 83, in update

                                                                                 if measurement.valid_state():

                                                                                                                File "/usr/local/lib/python2.7
/dist-packages/openxc-0.9.0-py2.7.egg/openxc/measurements.py", line 67, in valid_state

                                                                           return self.value in self.states

                                                                                                           TypeError: argument of type 'NoneType' 
is not iterable

Use new ReadTheDocs theme

http://ericholscher.com/blog/2013/nov/4/new-theme-read-the-docs/

Add Dashboard option to show only changed messages over a time window

To facilitate reverse engineering stuff on CAN, it'd be useful to automatically filter out CAN messages or signals that aren't changing. For example if a message hasn't changed in 2 seconds, remove it from the list. At some point you'll be left with a much smaller set of messages and it should be more obvious which changes when you hit a button, for example.

Diagnostic Messages Not Coding Correctly in 'addRecurringRequest(...)'

In structures.py, addRecurringRequest(...) function is given 10 inputs but the max allowable inputs for the function in vi-firmware is 8. structures.py includes factor and offset inputs which are not in the vi-firmware function declaration.

Should we plan to support factor and offset in vi-firmware? or should we remove these two inputs in the code generation and rely solely on custom decoders?

Create a tool to obscure specific GPS zones in vehicle traces

Provide a list of locations with a radius that you wish to obscure in a vehicle trace, and it will go through and change anything with that radius to just the center point. Ideal if you want to publish a trace but not give away your home address, but instead just leave it as "ann arbor".

Add a deprecation warning for using 'handler' on a message in code generation

Create web interface tool to replace desktop ncurses programs

A signals with states in a JSON mapping cannot be ignored

no matter what, the stateHandler is set automatically and it cannot be ignored. I thought 37dfac8 would fix it, but it didn't have any effect.

Document all valid attributes for buses, messages and signals and warn if invalid is used

The attributes are documented in the sphinx docs right now but if you typo in the JSON mapping file, it's easy for that to get missed. To save on future debugging, we should just spit a warning out if you use an attribute that's not one of the recognized ones.

obd2scanner can get confused by translated output format

Original report

It looks like the openxc-obd2scanner tool may get confused if the VI is also outputting data in the translated message format. We should able to reproduce this by running the scanner against the pre-built automatic OBD-II query build.

Support recording and playing back protobuf trace files.