swift-nav / libsbp

Swift Binary Protocol client libraries

Specification and Bindings for Swift Binary Protocol

• Installing sbp2json, json2sbp, json2json and related tools
• Building / installing
  • Using Docker
    • Fetching the prebuilt image from DockerHub
    • Creating your own image
    • Using the docker image
  • Installing from package managers
  • Installing development Python versions
  • Adding development version as a pip dependency
  • Installing from source
• SBP Development Procedures
• SBP Protocol Specification
• JSON Schema Definitions
• Kaitai Struct Format Descriptions
• LICENSE

The Swift Navigation Binary Protocol (SBP) is a fast, simple, and minimal binary protocol for communicating with Swift devices. It is the native binary protocol used by the Piksi GPS receiver to transmit solutions, observations, status and debugging messages, as well as receive messages from the host operating system, such as differential corrections and the almanac.

This project provides language-agnostic specification and documentation for messages used with SBP, a compiler for generating message bindings, and client libraries in a variety of languages. This repository is organized into the following directory structure:

• docs: Protocol documentation and message definitions.
• spec: Machine readable protocol specification in YAML.
• generator: Simple, template-based generator for different languages.
• python: Python client and examples.
• c: C client library and examples.
• haskell: Haskell client and examples.
• java: Java client library and examples.
• javascript: JavaScript client library and examples.
• rust: Rust client library and examples.
• sbpjson: Tools for parsing SBP-JSON.
• kaitai: Kaitai Struct Format Description and generated code.

Except for the generator, all of the above are generated and should not be modified directly.

Installing sbp2json, json2sbp, json2json and related tools

This repository also provides the following utilities for comprehending and inspecting SBP data:

• sbp2json: converts SBP binary into a JSON representation, in which field names and values are expanded into JSON objects, common fields such as "message type" and "payload" are included as well.

• json2sbp: uses the "message type", "payload" and related fields from an SBP JSON stream to reconstruct the binary representation.

• json2json: some tools (notably the Swift GUI Console) produce abbreviated JSON logs with only common fields such as "message type" and "payload", the json2json tool expands these JSON objects to include fields specific the individual message.

To install a released version of these tools, visit the releases page and download an archive for your platform.

To install from source, you can use Rust's cargo tool (first install Rust), then run:

cargo install --git https://github.com/swift-nav/libsbp.git --bins

There's also a Haskell version available which can be installed by running stack install in the ./haskell directory of a checkout of this repo (after installing stack) or by visiting the releases by and downloading an sbp_tools_haskell archive. This variant of the tools predate the Rust and Python versions, and also includes an sbp2yaml tool as well as a sbp2prettyjson tool.

Finally, a Python version of the sbp2json tool exists, which is installable on any platform that supports Python via pip, e.g.:

pip3 install sbp

The tool can then be invoked as follows:

python3 -m sbp2json <sbp.bin

The performance of the Python version is significantly slower than Rust and Haskell, but works on all platforms that Python itself supports.

Building / installing

Before you start, run

git pull --tags

in you local libsbp repository to pull the tags. This will ensure the correct version number is generated.

Using Docker

Before you begin, make sure you have Docker installed. Start Docker desktop.

Fetching the prebuilt image from DockerHub

The quickest method to get going is to just pull a prebuilt copy from DockerHub (no guarantees on freshness) by running the following on your command line:

docker run --rm -v $PWD:/mnt/workspace -i -t swiftnav/libsbp-build:2023-12-19 /bin/bash

This will mount your local copy of the libsbp repository onto the image.

Check this link for newer tags. Alternatively, you could run

docker run --rm -v $PWD:/mnt/workspace -i -t swiftnav/libsbp-build:latest-master /bin/bash

if you are facing issues with compilation and the tags are out of date as well.

Creating your own image

Otherwise, the Dockerfile will create a docker image that contains all the necessary dependencies to build libsbp. You can make a local image fresh from this file by running docker build as such:

docker build -t libsbp-build - <Dockerfile

Reading the Dockerfile from STDIN prevents docker from pulling in the whole repostory into the build context (which is then immediately discarded anyway). You can customize the UID of the user that's created with the docker image by passing the desired UID value to the build:

docker build -t libsbp-build --build-arg UID=1234 - <Dockerfile

You can then make this image operate on your local workspace like this:

docker run --rm -v $PWD:/mnt/workspace -i -t libsbp-build:latest /bin/bash

Using the docker image

Once in the image, simply type make all to generate all the libsbp bindings. This could take several hours to run. Alternately, the docker image will run the make all command by default, so you can kick off the make all process by simply running the following command:

docker run --rm -v $PWD:/mnt/workspace -i -t libsbp-build:2023-12-19

To speed up this process you can attempt to run Python environment tests in paralell with:

docker run --rm -v $PWD:/mnt/workspace -i -t -e SBP_TOX_PARALLEL=auto libsbp-build:2023-12-19

When you are finished, quit Docker so that it would not unnecessarily use up resources on your machine.

If you run into issues during the generation process, try running make clean. Alternatively, you could recompile from a clean, newly-cloned libsbp repository on your machine, which would minimize the chance of running into compilation issues from an old build.

Installing from package managers

Some bindings are available on package managers:

• python: available on pip
• haskell: available on Hackage
• javascript: available on NPM

Installing development Python versions

To install the Python binding from source (using pip) run the following command:

pip install 'file:///path/to/libsbp#subdirectory=python'

Or via setuptools directly:

cd /path/to/libsbp
cd python
python setup.py

Adding development version as a pip dependency

Run the following command:

pip install "git+https://github.com/swift-nav/libsbp@<GIT_REVISION>#egg=sbp&subdirectory=python"

Or add this to requirements.txt:

git+https://github.com/swift-nav/libsbp@<GIT_REVISION>#egg=sbp&subdirectory=python

Installing from source

You can build one binding at a time or update all at once:

make python

or

make all

are both valid. To see a list of all valid targets, run make help.

Python version notes:

1. By default the Python targets make python and make test-python (as well as make all) run tests on all Python versions officially supported by the libsbp Python bindings, currently 3.6-3.9, skipping any versions not installed. To run tests on just specific Python version(s), specify the TOXENV environment variable, e.g., TOXENV=py37 make python. Travis runs Python tests on all supported versions.
2. By default the code generators are run on the system's (or virtual env's) default Python interpreter. Currently Python versions 2.7, 3.5, and 3.7 are officially supported, other versions may or may not work. The generated libsbp bindings should be the same on all supported Python versions. To use a different version than your default Python interpreter, specify the GENENV environment variable, e.g., GENENV=py27 make all (you must have that version of Python installed beforehand).
3. To run both the generator and the Python tests on specific Python versions, specify both envs, e.g., GENENV=py37 TOXENV=py37 make python

SBP Development Procedures

See the "how to" page for instructions on adding new messages, updating the documentation, and releasing new versions of this library.

SBP Protocol Specification

SBP consists of two pieces: (i) an over-the-wire message framing format and (ii) structured payload definitions. As of Version 1.0, the packet consists of a 6-byte binary header section, a variable-sized payload field, and a 16-bit CRC value. SBP uses the CCITT CRC16 (XMODEM implementation) for error detection.

Please see the docs for a full description of the packet structure and the message types. Developer documentation for the language-specific sbp libraries is here. Please refer to the changelog for more information about the evolution of the library and its messages.

JSON Schema Definitions

For web clients processing SBP in JSON form, JSON schema definitions are provided. Libraries for JavaScript, TypeScript, and Elm generated by the QuickType tool are provided. See the HOWTO for instructions on updating these schemas.

Kaitai Struct Format Descriptions

Kaitai Struct is a declarative language used to describe binary data structures. KSY files containing format specifications for all supported SBP messages can be generated using the kaitai target. The resulting files can be found in the kaitai/ksy directory. This target also runs the Kaitai Struct compiler to generate bindings for perl and python. These bindings (along with test cases) can be found in the kaitai/perl and kaitai/python subdirectories respectively.

The Kaitai Struct format description files can also potentially be used to generate bindings for the following targets:

• graphviz
• csharp
• rust
• java
• go
• cpp_stl
• php
• lua
• nim
• html
• ruby
• construct
• javascript

Notes on python bindings

The python bindings generated by the Kaitai Struct compiler allow for significantly faster parsing in comparison to the construct-based bindings included in libsbp. However, due to the nature of Kaitai Struct, these bindings do not allow for serialisation (i.e. construction of SBP objects). Serialisation for java and python targets is supported as an experimental feature in the development version of Kaitai Struct (> 0.10) so this restriction should hopefully be removed in the future.

It should also be noted that the names used by the KSY bindings are not 1:1 identical to the construct bindings due constraints which Kaitai Struct places on various token types (e.g. it is forbidden to use upper case characters in field names). The test cases in the kaitai/ksy/tests directory (particularly those in test_parsers.py) can be used as a starting point for working with these bindings, as well as demonstrating the minor differences between the output generated by each set of bindings.

LICENSE

Copyright © 2015-2023 Swift Navigation

Distributed under the MIT open source license.

libsbp's Issues

Regenerating Doxygen, sphinx for C and Python

API libraries generated via Doxygen should be regenerated and linked into the wiki.

Integrate libsbp python serial_link.py into piksi_firmware scripts via git submodule

Definition and wrappers down to the nibble

There are a few places in LibSBP where information is encoded down to the nibble (for instance the n_obs word in the observation header). It would be nice if a consumer of the python, java, or c , etc wrappers didn't have to do any bitmasking to get the information.

SBP object implementation can't use slots

http://bugs.python.org/issue15535

As far as I can tell, if you implement a class with slots, the python pickling library will have difficulties, looking for an implicit __dict__ attribute when none exists. Incidentally, you can add a __dict__ attribute try and fool python, but then it'll go ahead and call dict() anyway (which sucks). In any case, whenever we've moved off using picking serialization for anything, we ought to use __slots__ for the memory help, since it may actually be useful here for reducing memory usage of serial_link and the testing stuff.

For an implementation example, see: https://github.com/construct/construct/blob/master/construct/core.py#L100

Integrate libsbp c bindings with libswiftnav

Remove init known baseline feature from console

For the example of the headaches it causes:
https://groups.google.com/forum/#!topic/swiftnav-discuss/UpXC7QTub34

Closes swift-nav/piksi_firmware#370

Change MSG_EPHEMERIS id to be in "over the radio" group

Should be 0x4?

Purely functional libsbp generator

The libsbp generator is a good candidate for an easy GHC rewrite (possibly using ede templating: https://github.com/brendanhay/ede). Let's burn that bridge later.

node.js client for serial/tcp message deframing.

have to install python sbp to generate python docs

'make html' uses some information from source tree and some from the installed sbp package

Python API doc index should be autogenerated

Currently you have to manually add new SBP message packages to python/docs/source/index.rst

Generated C headers don't close Doxygen brace

e.g. https://github.com/henryhallam/libsbp/blob/ext_events/c/include/libsbp/ext_events.h line 22 opens a Doxygen defgroup but it's never closed.

'make html' doesn't generate C docs

Create Jenkins pull request builder

Integrate SBP, language-specific developer documentation

Add to developer wiki.
High-level explanation for non-developer audience.
Language-specific tutorials embedded in Doxygen/sphinx consumed from the serial (1).

See also: http://docs.swift-nav.com/wiki/Piksi_Integration_Tutorial

Yaml spec: audit message definitions

Update documentation descriptions and payload info for various piksi.yaml messages.

A bunch of these messages, which are not intended for public consumption, still ought to be documented since we're generating code for them.

Add tests to generator.py, update setup.py to call coverage, py.test, etc.

Standardize SBP parsing

Parsing and generation are currently a bit adhoc, which makes it difficult to reference binary messages defined elsewhere in the spec (for example). It may be useful to (i) use a standard format for specifying messages and (ii) provide an easy way of referencing other defined structures. This would be similar to how Protobuf's are specified, but targeted for our SBP layout.

As of March 2015: task for merging generator.py

Rename uarts in MSG_UART_STATE

These should be renamed:

uarts0 -> uarta
uarts1 -> uartb
uarts2 -> uart_ftdi

0/1/2 just came from the fact these were sent as an array.

Port sbp_log_analysis to use libsbp v2 (via pip install, probably)

Write GPSD driver for SBP

Generate C bitmask #defines for bit fields

• A single #define for single-bit fields
• Mask and offset #define for multi-bit fields
  e.g. for MSG_HEARTBEAT,

#define SBP_MSG_HEARTBEAT_FLAGS_EXT_ANT (1<<31)
#define SBP_MSG_HEARTBEAT_FLAGS_NAP_ERR (1<<2)
#define SBP_MSG_HEARTBEAT_FLAGS_IO_ERR (1<<1)
#define SBP_MSG_HEARTBEAT_FLAGS_SYS_ERR (1<<0)

Need some way to specify the short name for the bit in the yaml.

Comments only sometimes line up in generated C headers

e.g.

#define SBP_MSG_UART_STATE    0x0018
typedef struct __attribute__((packed)) {
  uart_channel_t uart_a;       /**< State of UART A */
  uart_channel_t uart_b;       /**< State of UART B */
  uart_channel_t uart_ftdi;    /**< State of UART FTDI (USB logger) */
  latency_t latency;      /**< UART communication latency */
} msg_uart_state_t;

Clearly it's trying to align the comments since it uses 3 more spaces after uart_b than uart_ftdi, but it's failing to account for the different lengths of the member types.

C lib: audit C imports for libswiftnav

Generate Python bindings for libsbp specifications

Fork existing generator to use Construct for variable length fields.

March 2015 update:
Python template: Change templated python generation to use construct for nested message definitions and variable length fields (1 - 2 day, Buro). Cleanup for broader use.

Update latex templating for protocol definitions

Command line option and generator submodule for Latex templating.
Adding command line option for generating Latex documentation.
Did this generate compilable Latex?
Tests

Note that Latex documentation should probably clearly distinguish between stable and unstable messages.

Document process for adding new messages

• Policy on packages
• Policy on message ID assignment
• How to regenerate / commit / ship the changes

MsgPrint should have at least one field.

See: https://github.com/swift-nav/libsbp/pull/52/files#diff-acda77cabc422238f7faaf9a311e6290R7

Expand JSON Serialization to entire object

We need full JSON serialization of the SBP messages. Presently, just the SBP base message is being JSON serialized. Some binary fields will need to be base64 encoded.

Python client: Port serial_link.py and deframing callbacks to existing libsbp master

Porting from piksi_firmware/scripts.
Adding test coverage
JSON output for logging?

'make latex' fails with 'too many floats'

python/README.md refers to nonexisting requirements.txt

Install dependencies

pip install -r requirements.txt

does not exist.

MSG_ALMANAC could actually be documented.

See contents here:

https://github.com/mfine/piksi_tools/blob/serial-leaning/piksi_tools/almanac.py

Templating and code for javascript

Update sbp integration tutorial?

@fnoble @denniszollo, as part of getting libsbp out for v1 documentation, I've noticed that we have an existing integration tutorial for libswiftnav's SBP message definitions: https://github.com/swift-nav/sbp_tutorial. Perhaps we'll have to update this as well?

Update bitfield definitions to be defined via yaml

Right now all the bit logic is hardcoded into the message packing code. It should be defined by yaml.

Protocol version number in docs is hard-coded, should be same as sbp package version?

Cleanup hacks and technical debt

1. Documentation for python bindings.
2. Generalize generator to handle more complex structs.
3. Add tests for generator.

Tests for message deserialization.

Tests are needed, at least for the python client, to check for proper message handling and dispatch.

Sync repo README and gh-pages landing page.

Currently, the README for the repo landing page and the documentation landing page at http://swift-nav.github.io/libsbp/ are a bit different (perhaps not surprising, given that one talks about the repo and the other talks about documentation). These though, should have a happy middle ground where the gh-page landing page has the same content as the README.

Deserializing empty payloads on MSG_SETTINGS_READ_BY_INDEX doesn't work

>>> import sbp
>>> s = sbp.SBP(msg_type=0xA2, sender=1781, length=0, payload="", crc=0xCCE3)
>>> import sbp.table
>>> sbp.table.dispatch(s)
sbp/table.py:76: RuntimeWarning: SBP payload deserialization error! 0xa2
  RuntimeWarning)
<SBP (preamble=0x55, msg_type=0xA2, sender=1781, length=0, payload=, crc=0xCCE3)>
>>>

Not sure where or if the serialization is failing here, but came across this failure from a live log.

Improve landing SBP documentation.

See:
http://docs.swiftnav.com/wiki/SwiftNav_Binary_Protocol

Specifically, protocol documentation:
0. Upload API docs to developer wiki (user-facing PDF specification)

1. Message framing and parsing (written)
2. Message definitions from payload contents (generated)

Missing newlines at end of many autogenerated files

(and some non-auto-generated files)

henry@hank:~/swift/libsbp$ for x in $(find * -type f); do if [ -n "$(tail -c 1 <"$x")" ]; then echo "$x missing newline"; fi; done
c/test/check_settings.c missing newline
c/test/check_bootload.c missing newline
c/test/check_system.c missing newline
c/test/check_navigation.c missing newline
c/test/check_piksi.c missing newline
c/test/check_flash.c missing newline
c/test/check_tracking.c missing newline
c/test/check_acquisition.c missing newline
c/test/check_standard.c missing newline
c/test/check_file_io.c missing newline
c/test/check_observation.c missing newline
c/include/libsbp/settings.h missing newline
c/include/libsbp/acquisition.h missing newline
c/include/libsbp/bootload.h missing newline
c/include/libsbp/tracking.h missing newline
c/include/libsbp/system.h missing newline
c/include/libsbp/observation.h missing newline
c/include/libsbp/ext_events.h missing newline
c/include/libsbp/piksi.h missing newline
c/include/libsbp/navigation.h missing newline
c/include/libsbp/file_io.h missing newline
c/include/libsbp/logging.h missing newline
c/include/libsbp/flash.h missing newline
generator/.gitignore missing newline
generator/MANIFEST.in missing newline
generator/.coveragerc missing newline
python/sbp/observation.py missing newline
python/sbp/acquisition.py missing newline
python/sbp/system.py missing newline
python/sbp/bootload.py missing newline
python/sbp/ext_events.py missing newline
python/sbp/logging.py missing newline
python/sbp/tracking.py missing newline
python/sbp/flash.py missing newline
python/sbp/piksi.py missing newline
python/sbp/file_io.py missing newline
python/sbp/navigation.py missing newline
python/sbp/settings.py missing newline
python/data/serial_link_log_20141125-150750_test2.log.dat missing newline
python/tox.ini missing newline
python/pytest.ini missing newline
python/.coveragerc missing newline

Regenerate everything script

Would be nice if there was one script you could run at the top level after changing the YAML that built / generated all the things.

SBP pickle framer might be stripping

I noticed this earlier with construct's CString object, where the payload string parsed from SBP seems to be missing the null termination character in the handshake value for the MsgBootloaderHandshake. As such, the parsing of this field can fail if it's not actually NULL terminated.

Release libsbp python bindings to the Python package index.

Cleanup setup.py and upload egg(?) to pypi.

Testing locally via pip install.
Update Jenkins provisioning in sbp_log_analysis project to use pip.

`make` should just do `make all`

(by convention)

Defined SBP protocol version number

• Semantic versioning
• Send version number in HEARTBEAT?

@henryhallam

Integrate libsbp c bindings with piksi_firmware