sco1 / xbmini-py

Python Toolkit for the GCDC HAM

xbmini-py

Python Toolkit for the GCDC HAM

Known Firmware Compatibility

This package is currently tested against firmware versions 1379, 2108, and 2570, compatibility with other firmware versions is not guaranteed.

Installation

Install from PyPi with your favorite pip invocation:

$ pip install xbmini-py

You can confirm proper installation via the xbmini CLI:

$ xbmini --help
Usage: xbmini [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  batch-combine  Batch combine XBM files for each logger and dump a...
  dash           Dash UI launchers

CLI Usage

xbmini batch-combine

Batch combine XBM files for each logger and dump a serialized XBMLog instance to a CSV in its respective logger's directory.

Input Parameters

Parameter	Description	Type	Default
--top-dir	Path to top-level log directory to search.1	Path|None	GUI Prompt
--log-pattern	XBMini log file glob pattern.2	str	"*.CSV"
--dry-run	Show processing pipeline without processing any files.	bool	False
--skip-strs	Skip files containing any of the provided substrings.	list[str]	["processed", "trimmed", "combined"]

1. Log searching will be executed recursively starting from the top directory
2. Case sensitivity is deferred to the host OS

xbmini dash

A series of helper UIs are provided by Dash. Running the CLI commands will start a local server for the user to interact with.

WARNING: These apps are intended for use on a development server only. Do not use them in a production environment.

A list of available UIs can be accessed via the command line:

$ xbmini dash --help
Usage: xbmini dash [OPTIONS] COMMAND [ARGS]...

  Dash UI launchers

Options:
  --help  Show this message and exit.

Commands:
  trim  Helper UI for trimming serialized XBMLog CSVs.

xbmini-py's Issues

Add sensor full scale lookup for legacy sensor header

For older firmware versions (1379 currently known), the sensor full scale value isn't provided in the log file's header information so a dummy value is used. e.g.:

xbmini-py/xbmini/parser.py

Lines 121 to 127 in 0b11d58

	sensor_info["Mag"] = cls(
	name="Mag",
	sample_rate=mag_sample_rate,
	sensitivity=sensitivity,
	full_scale=-1, # This needs to be mapped from the sensitivity, check the docs
	units=units,
	)

There is a lookup table in the documentation that maps counts to the respective sensor's full scale rate, so this can be transferred over. This isn't currently used in any of the existing MATLAB code but it would still be good to have to match completeness with newer firmware outputs.

Refactor Sensor Parsing

As noted in #13, the header information has changed significantly for the IMU-GPS device. The current parsing code is pretty specific to the last encountered HAM-IMU firmware version, so I think it should really be dispatched based on device type rather than trying to branch within for the new device.

I think we should drop the legacy device support completely & move forward with just the IMU-GPS and HAM-IMU+alt devices only. We don't use the legacy devices anymore and it just complicates the branching.

Add device configuration helper

• Device configuration generator
• Helper for writing config to device
  • Poll for USB devices & identify connected GCDC logger(s)
  • Optionally allow relabeling of drive
  • Copy existing logs to target & optionally clear

Utilize `iloc` rather than integer keys

Starting with Pandas v2.1.0 we start to get deprecation warnings on some of our data access, e.g.:

FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
    assert log_obj.press_temp["press_alt_m"][0] == pytest.approx(136.20, abs=1e-2)

This may only be in the test code but should check over the rest of the codebase

Gracefully Handle Sensor Faults

During the device's self-test on startup it may encounter a fault with one or more sensors, e.g.:

;Title, http://www.gcdataconcepts.com, HAM-IMU+alt, MPU9250 BMP280
;Version, 2108, Build date, Mar  5 2021,  SN:abc123
;Start_time, 2022-01-01, 04:20:34.497
;Uptime, 5,sec,  Vbat, 3754, mv, EOL, 3500, mv
MPU Fault
;BMP280 SI, 0.050,s
;Deadband, 0, counts
;DeadbandTimeout, 5.000,sec
;Time, Ax, Ay, Az, Gx, Gy, Gz, Qw, Qx, Qy, Qz, Mx, My, Mz, P, T
; 0.000 stopping logging: shutdown: switched off

While the logic for catching missing sensor information during parsing of the header does raise on this as a side-effect, this should have an explicit check.

We have an example of this for firmware version 2108, but nothing for older firmware versions so I'm not sure if it looks the same.

Make trim app an optional dependency

Align Sensitivity Override Acceptance

With the sensor parsing refactor merged in e0ae707 came an escape hatch for logs with incomplete sensor data in the header, allowing parsing to continue under the assumption that the user will use sensitivity_override to fill in the missing information when instantiating a log object. This wasn't fully percolated to the batch utilities, so we could end up with objects in serialized or attempting to be serialized states without a means to set sensor configurations.

Loading of processed logs should also allow for the specification of a new sensitivity override, which would rewind the previous configuration (no access to original data) & apply the new one. Note that this is only valid for HAM-IMU devices; IMU-GPS logs don't report raw counts.

We should end up with the following flows:

• xbmini.log_parser.XBMLog.from_multi_raw_log
  • Accept a sensor override and raise_on_missing_sensor flag
  • Override logic already handled by xbmini.log_parser.load_log
• xbmini.log_parser.XBMLog.from_raw_log_file
  • Accept a sensor override and raise_on_missing_sensor flag
  • Loading pipeline is then handled by from_multi_raw_log
• xbmini.log_parser.XBMLog.from_processed_csv
  • Accept a sensor override
    • If an override is provided and it's a HAM-IMU device, apply the new override
    • If an override is provided and it's an IMU-GPS device, ignore the override and print a warning
• xbmini.log_parser.batch_combine
  • Accept a sensor override
  • If a sensor override is provided, set raise_on_missing_sensor flag to False and continue downstream
  • Loading pipeline is then handled by from_multi_raw_log
  • If there is a sensor error, the logger is skipped & a warning is printed to the user so they can retry with an override

Investigate alternatives to kaleido

Kaleido is currently used to export the plots made by plotly to disk. Including kaleido makes for a really heavy install so it would be worth investigating alternative packages, or an alternative to the plotly stack altogether.

Add Support For IMU-GPS

The new XBM adds GPS support and has some differences to the log structure. Sample header structure pasted at the end.

• Add new parsing dispatch
  • Header no longer contains an explicit product name, just sensor information. Since this is the only one with GPS, dispatch can probably just be based on that.
• Parse sensor information
  • Sensor information looks to be a combination of the previous XBM and what was termed "legacy"; Sensors may be labeled by their sensor ID (e.g. BMP384 for baro), and may have common names for other sensors on the same chip (e.g. LSM6DSMfor the IMU line, with fullscale gyro on the same line).
  • GPS satellite in view information provided in header
• Parse log data
  • Quaternions no longer present
  • GPS data likely samples at a lower rate than the IMU, data is interleaved like pressure/temperature has been previously

Sample new header:

;Title, http://www.gcdataconcepts.com, LSM6DSM, BMP384, GPS
;Version, 2570, Build date, Sep 13 2023,  SN:ABCD12342069
;Start_time, 2023-11-01, 12:40:00.104
;Uptime, 6,sec,  Vbat, 4198, mv, EOL, 3500, mv
;Deadband, 0, counts
;DeadbandTimeout, 0.000,sec
;BMP384, SI, 0.100,sec, Units, Pa, mdegC
;Alt Trigger disabled
;LSM6DSM, SR,104,Hz, Units, mG, mdps, fullscale gyro 250dps, accel 4g
;Magnetometer, SR,10,Hz, Units, nT, Temperature, 19,degC
;CAM_M8 Gps, SR,1,Hz
;Gps Sats, TOW, 312085000, ver, 1, numSat, 14
;, gnssId, svId, cno, elev, azmith, prRes, flags,inUse
;, GPS , 002, 00, 34, 313, 0, 0x00001211
;, GPS , 008, 00, 19, 261, 0, 0x00001211
;, GPS , 010, 00, 51, 067, 0, 0x00001a11
;, GPS , 018, 00, 01, 148, 0, 0x00001a11
;, GPS , 021, 00, 44, 307, 0, 0x00001211
;, GPS , 023, 00, 20, 095, 0, 0x00001911
;, GPS , 025, 00, 03, 100, 0, 0x00001a11
;, GPS , 027, 00, 15, 220, 0, 0x00001211
;Time, Ax, Ay, Az, Gx, Gy, Gz, Mx, My, Mz, P, T, TOW, Lat,Lon, Height(m), MSL(m), hdop(m), vdop(m)