stmicroelectronics / fp-sns-datalog2

The FP-SNS-DATALOG2 function pack represents an evolution of FP-SNS-DATALOG1 and provides a comprehensive solution for saving data from any combination of sensors and microphones configured up to the maximum sampling rate. Please check st.com where a more recent version of the software might be available.

FP-SNS-DATALOG2 Firmware Package

The FP-SNS-DATALOG2 function pack represents an evolution of FP-SNS-DATALOG1 and provides a comprehensive solution for saving data from any combination of sensors and microphones configured up to the maximum sampling rate.

FP-SNS-DATALOG2 is based on application-level modules ((Sensor Manager, EMData, …) that a final user can reuse and extend easily to build up its custom application. These application modules adopt state-of-the-art design patterns, and they support natively low-power modes. To enable this solution, the function pack has been built on top of eLooM, an embedded Light object oriented fraMework for STM32 applications properly designed for embedded low power applications powered by STM32.

The FP-SNS-DATALOG2 application allows to store sensor data onto a microSD™ card (secure digital High capacity - SDHC) formatted with the FAT32 file system, or stream to a PC via USB (WinUSB class) using the companion host software (cli_example) provided for Windows and Linux. It can also configure an ISM330DHCX and LSM6DSV16X Machine Learning Core unit and the ISM330IS Intelligent Sensor Processing Unit, so to read the output of the selected algorithm.

This application allows downloading the selected configuration from a JSON file to the board as well as starting and controlling data acquisition. The same set of commands can be operated from a host via Command Line Interface or via Bluetooth using the STBLESensor app which lets you manage the board and sensor configurations, start/stop data acquisition on SD card and control data labeling. To read and process sensor data acquired using FP-SNS-DATALOG2, a few easy-to-use scripts in Python are provided within the software package. The scripts have been successfully tested with Python 3.10.

The package also includes an example of ultrasound condition monitoring (UltrasoundFFT) for STEVAL-STWINBX1 that calculates the FFT of the IMP23ABSU analog microphone signal and streams the result to a PC GUI via USB. The microphone sampling rate is set by default to 192 kHz whereas the microphone bandwidth is up to 80 kHz.

A new PDetect example for STEVAL-STWINBX1 + STEVAL-PDETECT1 for presence and motion detection has been added in FP-SNS-DATALOG2 v2.1.0.

The FP-SNS-DATALOG2 software has been developed for the B-U585I-IOT02A, STEVAL-STWINBX1, STEVAL-STWINKT1B and STEVAL-MKBOXPRO development kit and X-NUCLEO-IKS02A1 with NUCLEO-U575ZI-Q or with NUCLEO-H7A3ZI-Q.

Here is the list of references to user documents:

• DB4865 : STM32Cube function pack for high speed datalogging and ultrasound processing
• UM3106 : Getting started with the STM32Cube function pack for high speed datalogging and ultrasound processing
• Doxygen: Doxygen documentation
• STM32Cube: STM32Cube

Known Limitations

• To run properly DATALOG2 example on STWIN.box, before using those applications, WIFI module firwmare must be upgraded. The binary is available in Utilities/WiFi_module_upgrade folder.

Development Toolchains and Compilers

• IAR Embedded Workbench for ARM (EWARM) toolchain V9.20.1
• RealView Microcontroller Development Kit (MDK-ARM) toolchain V5.37
• STM32CubeIDE v1.14.0

Supported Devices and Boards

• STEVAL-MKBOXPRO
• STEVAL-PDETECT1
• STEVAL-STWINBX1
• STEVAL-STWINKT1B
• B-U585I-IOT02A
• X-NUCLEO-IKS02A1 with NUCLEO-U575ZI-Q
• X-NUCLEO-IKS02A1 with NUCLEO-H7A3ZI-Q

Backward Compatibility

• None

Dependencies

• None

fp-sns-datalog2's Issues

Corrupted data in long measurements

I don`t understand how this issue can be closed as completed. Yes the explanation is that there is some USB interference or something similar but why is all data from that moment on corrupted? This basically makes it impossible to take measurements for more than a few minutes. I would see it as a bug that the software cannot recover from a transmission problem that probably only lasts a few milliseconds.

#3

Originally posted by @nicolasj92 in #3 (comment)

Ultrasound FFT does not start as expected

I cloned the repository and set up Python (V3.10.11) to uses the HSDPyhtonSDK together with STEVAL-STWINBX1. The application hsdatalog_GUI works well. But when I start the ultrasound_fft_app, The application shows the GUI as expected, but when I connect, the GUI view is not the expected one, but the same as hsdatalog_GUI, see here below.

Here the logfile: 20230829_07_54_37_app_debug.log

Here the terminal output:

No HSDatalog_v1 devices connected!
Trying to open communication using libhs_datalog_v2...
64bit
HSD_DLLv2: DeviceManager created
HSD_DLLv2: libusb_close! 
HSD_DLLv2: libusb_close! 
HSD_DLLv2: libusb_close! 
HSD_DLLv2: libusb_close! 
HSD_DLLv2: libusb_close! 


HSD_DLLv2: CONNECTED!    
Commmunication Opened correctly
qt.pysideplugin: Environment variable PYSIDE_DESIGNER_PLUGINS is not set, bailing out.
dtmi found in locally in base supported models
dtmi: dtmi/appconfig/steval_stwinbx1/fpSnsDatalog2_ultrasoundFft-3.json
dtmi found in locally in base supported models
dtmi: dtmi/appconfig/steval_stwinbx1/fpSnsDatalog2_datalog2-3.json     
2023-08-29 07:37:33,574 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: iis3dwb_acc - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component iis3dwb_acc Updated correctly
2023-08-29 07:37:33,685 - HSDatalogApp.st_hsdatalog.HSD_link.communication.PnPL_HSD.PnPLHSD_com_manager - ERROR - No Component Status[d_id:0,iis3dwb_ext_acc] returned. (PnPLHSD_com_manager.py:234)
2023-08-29 07:37:33,711 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_Controller - WARNING - The component [iis3dwb_ext_acc] defined in DeviceTemplate has not a Twin in Device Status from the FW (HSD_Controller.py:440)
ComponentWidget - WARNING - No status to update for iis3dwb_ext_acc Component
2023-08-29 07:37:33,879 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: iis2mdc_mag - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component iis2mdc_mag Updated correctly
2023-08-29 07:37:34,041 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: imp23absu_mic - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component imp23absu_mic Updated correctly
2023-08-29 07:37:34,209 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: ism330dhcx_acc - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component ism330dhcx_acc Updated correctly
2023-08-29 07:37:34,365 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: ism330dhcx_gyro - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component ism330dhcx_gyro Updated correctly
2023-08-29 07:37:34,548 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: ism330dhcx_mlc - status: False (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component ism330dhcx_mlc Updated correctly
2023-08-29 07:37:34,745 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: iis2dlpc_acc - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component iis2dlpc_acc Updated correctly
2023-08-29 07:37:34,904 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: stts22h_temp - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component stts22h_temp Updated correctly
2023-08-29 07:37:35,014 - HSDatalogApp.st_hsdatalog.HSD_link.communication.PnPL_HSD.PnPLHSD_com_manager - ERROR - Component Status[d_id:0,stts22h_ext_temp] parsing error. (PnPLHSD_com_manager.py:232)
2023-08-29 07:37:35,029 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - WARNING - It is impossible to know the Sensor [stts22h_ext_temp] enabling status from the FW device status (HSD_DeviceConfigPage.py:162)
2023-08-29 07:37:35,042 - HSDatalogApp.st_hsdatalog.HSD_link.communication.PnPL_HSD.PnPLHSD_com_manager - ERROR - Component Status[d_id:0,stts22h_ext_temp] parsing error. (PnPLHSD_com_manager.py:232)
2023-08-29 07:37:35,042 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_Controller - WARNING - The component [stts22h_ext_temp] defined in DeviceTemplate has not a Twin in Device Status from the FW (HSD_Controller.py:440)
ComponentWidget - WARNING - No status to update for stts22h_ext_temp Component
2023-08-29 07:37:35,186 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: ilps22qs_press - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component ilps22qs_press Updated correctly
2023-08-29 07:37:35,350 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: imp34dt05_mic - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component imp34dt05_mic Updated correctly
2023-08-29 07:37:35,520 - HSDatalogApp.st_hsdatalog.HSD_GUI.HSD_DeviceConfigPage - DEBUG - comp_name: iis2iclx_acc - status: True (HSD_DeviceConfigPage.py:157)
ComponentWidget - INFO - Component iis2iclx_acc Updated correctly
ComponentWidget - INFO - Component automode Updated correctly
ComponentWidget - INFO - Component log_controller Updated correctly
2023-08-29 07:37:35,829 - HSDatalogApp.st_hsdatalog.HSD_GUI.Widgets.TagsInfoWidget - DEBUG - Component: tags_info (TagsInfoWidget.py:186)
TagsInfoWidget - INFO - Component tags_info Updated correctly
ComponentWidget - WARNING - Property type not supported. (comp: acquisition_info, cont:tags) status not updated
ComponentWidget - INFO - Component acquisition_info Updated correctly
ComponentWidget - INFO - Component firmware_info Updated correctly
ComponentWidget - INFO - Component DeviceInformation Updated correctly

Corrupted data in long measurements

We are using the STEVAL-STWINBX1 at a CNC machine to gather vibration data. We are using the fp-sns-datalog2 HSDatalog Python framework to record the data over USB. Unfortunately, the data seems to get corrupted in longer measurements (>5 min).

In these cases, the dataframes get corrupted and it seems as though the axis of the accelerometer are switched around

The following code is used to read the raw measurement files:

from st_hsdatalog.HSD.HSDatalog import HSDatalog
hsd_factory = HSDatalog()
hsd = hsd_factory.create_hsd(acq_folder)

sensor = HSDatalog.get_sensor(hsd, sensor_name)
stwin_data = HSDatalog.get_dataframe(hsd, sensor)[0]

and the resulting dataframe shows the corrupt data - in the beginning, everything is ok but in the end, the timestamps are invalid and it seems as though the axis are switched:

The HSDatalog call gives the following warnings:
dtmi found in locally in base supported models dtmi: dtmi/appconfig/steval_stwinbx1/fpSnsDatalog2_datalog2-2.json [c:\Users\N.Jourdan_Lokal\anaconda3\lib\site-packages\numpy\core\function_base.py:128](file:///C:/Users/N.Jourdan_Lokal/anaconda3/lib/site-packages/numpy/core/function_base.py:128): RuntimeWarning: invalid value encountered in multiply stop = asanyarray(stop) * 1.0 [c:\Users\N.Jourdan_Lokal\anaconda3\lib\site-packages\numpy\core\function_base.py:127](file:///C:/Users/N.Jourdan_Lokal/anaconda3/lib/site-packages/numpy/core/function_base.py:127): RuntimeWarning: invalid value encountered in multiply start = asanyarray(start) * 1.0
`

Support for an optical position sensor.

Dear ST Team

I am currently working on implementing a 2 dimensional position tracking sensor PAT9136 from PixArt in the FP-SNS-DATALOG2 firmware and I am asking me if this could be interresting for you to have this implementation merged. This would still have to be authorized by PixArt, as their technical data is confidential. I'll have PixArt visit us and ask for authorization if you're interested.

Regards,
Ludwig

Data recording stops for a second

Hi

As I already mentioned in the issue #11 (comment) there are sporadic recording gaps occuring.

This only applies to sensors that are read periodically using an OS timer, i.e. in the case of the basic FW implementation, the temperature sensor STTS22H and it seems to depend on the device configuration. When I define the configuration to use only the STTS22H, no gaps appear, but using the configuration as below, the time between each execution of the timer function may randomly exceed one second.

I modified the STTS22HTaskTimerCallbackFunction to detect this condition as follows:

static void STTS22HTaskTimerCallbackFunction(ULONG param)
{
  STTS22HTask *p_obj = (STTS22HTask *) param;
  SMMessage report;
  report.sensorDataReadyMessage.messageId = SM_MESSAGE_ID_DATA_READY;
  report.sensorDataReadyMessage.fTimestamp = SysTsGetTimestampF(SysGetTimestampSrv());

  double delta_timestamp = report.sensorDataReadyMessage.fTimestamp - p_obj->prev_timer_timestamp;
  
  if (delta_timestamp > 1.5f * (p_obj->task_delay / 1000.0f))
  {
    SYS_DEBUGF(SYS_DBG_LEVEL_WARNING, ("STTS22H: timer interval exceeded %.2f\r\n", delta_timestamp));
  }
  p_obj->prev_timer_timestamp = report.sensorDataReadyMessage.fTimestamp;

  if (TX_SUCCESS != tx_queue_send(&p_obj->in_queue, &report, TX_NO_WAIT))
  {
    /* unable to send the report. Signal the error */
    sys_error_handler();
  }
}

As the following log file shows, during recodring the console output the message STTS22H: timer interval exceeded 1.02.

STWIN.box.log

Here is the recorded data: DL2_00001.zip

Remark

• When the problem is detected, LED_1 (green) also stops flashing for a while
• No difference in Release or Debug targets

---

Setup

Device : STWIN.box (USB or battery powered)
Device Config: device_config.json
IDE: µVision V5.38.0.0
Compiler: V6.19

HSDatalog and Python SDK installation not working on 64-bit ARM (Raspberry Pi 5)

Describe the set-up
STEVAL-STWINBX1 + FP-SNS-DATALOG2 2.0.1
Raspberry Pi 5 8GB with Raspberry Pi OS 64-bit (latest)
Linux 6.1.65-v8-16k+ #1703 SMP PREEMPT Tue Dec 5 16:28:46 GMT 2023 aarch64

Describe the bug
Python installation and the underlying HSDatalog library do not work on 64-bit ARM. Using 64-bit is necessary to make use of all memory on Raspberry Pi 5 models (for single process, which we need).

How To Reproduce the bug
Try to compile https://github.com/STMicroelectronics/fp-sns-datalog2/blob/main/Utilities/cli_example/CMakeLists.txt (after fixing paths)

/usr/bin/c++ -std=c++11 -rdynamic CMakeFiles/cli_example.dir/main.cpp.o -o cli_example -Wl,-rpath,<>/STM32CubeFunctionPack_DATALOG2_V2.0.1/Utilities/cli_example/lib/libhs_datalog/raspberryPi <>/STM32CubeFunctionPack_DATALOG2_V2.0.1/Utilities/cli_example/lib/libhs_datalog/raspberryPi/libhs_datalog_v2.so -lpthread
/usr/bin/ld: <>/STM32CubeFunctionPack_DATALOG2_V2.0.1/Utilities/cli_example/lib/libhs_datalog/raspberryPi/libhs_datalog_v2.so: error adding symbols: file in wrong format collect2: error: ld returned 1 exit status

Workarounds
For Python, problem is that PySide 6.4.3 aarch64 is not on pip. Removing that requirement from the wheel allows installation to succeed.

For HSDatalog, no workaround. Either release the source code for libhs_datalog so that we can recompile ourselves, or provide aarch64 version

Python hsdatalog_cli.py does not work with provided configs

When trying to start the hsdatalog_cli.py in Utilities\HSDPython_SDK\examples\hsdatalog_cli.py with any of the provided example configs in Utilities\STWIN.box_config_examples the script crashes with an error on startup.

Steps to reproduce:
python fp-sns-datalog2\Utilities\HSDPython_SDK\examples\hsdatalog_cli.py -f fp-sns-datalog2\Utilities\STWIN.box_config_examples\all_sensors.json

The problem seems to be that dictionaries are used for the "samples_per_ts" and this is not accepted by PnPLHSD_com_manager.py as it only accepts bool int float or str properties:

File "C:\Users\N.Jourdan_Lokal\projects\fp-sns-datalog2\Utilities\HSDPython_SDK\examples\Views\cli_views.py", line 409, in _update
self._hsd_info_model.upload_device_conf_file()
File "C:\Users\N.Jourdan_Lokal\projects\fp-sns-datalog2\Utilities\HSDPython_SDK\examples\hsdatalog_cli.py", line 143, in upload_device_conf_file
res = HSDLink.update_device(self.hsd_link, self.selected_device_id, self.cli_flags.file_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\N.Jourdan_Lokal\anaconda3\Lib\site-packages\st_hsdatalog\HSD_link\HSDLink.py", line 422, in update_device
return hsd_link.update_device(device_id, device_config_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\N.Jourdan_Lokal\anaconda3\Lib\site-packages\st_hsdatalog\HSD_link\HSDLink_v2.py", line 407, in update_device
return self.__com_manager.update_device(d_id, device_json_file_path, self.__dt_manager.get_components())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\N.Jourdan_Lokal\anaconda3\Lib\site-packages\st_hsdatalog\HSD_link\communication\PnPL_HSD\PnPLHSD_com_manager.py", line 389, in update_device
self.__set_component_values(d_id, component, dtdl_components[key])
File "C:\Users\N.Jourdan_Lokal\anaconda3\Lib\site-packages\st_hsdatalog\HSD_link\communication\PnPL_HSD\PnPLHSD_com_manager.py", line 371, in __set_component_values
self.set_property(d_id, comp[comp_name][content.name], comp_name, content.name)
File "C:\Users\N.Jourdan_Lokal\anaconda3\Lib\site-packages\st_hsdatalog\HSD_link\communication\PnPL_HSD\PnPLHSD_com_manager.py", line 344, in set_property
print(res)
^^^
UnboundLocalError: cannot access local variable 'res' where it is not associated with a value

Is it possible to create a real-time application that uses the hsdlink python SDK?

I want to build an application that uses the sensor data over usb in near-real time. E.g. always perform some operations on the data of the last second. Is this possible with the SDK? So far I can only see that the data is stored in .dat files and needs to be converted post-acqusition before it can be used.
I tried to dig through the GUI application code to find out how its handling the real time plotting but so far I don't fully understand whats going on there.

USB_DataLog_Run script paths

Caution
The Issues are strictly limited for the reporting of problem encountered with the software provided in this project.

Describe the set-up
Install the fp-sns-datalog2 repository on a linux system using git clone

Describe the bug
Utilities/cli_example/USB_DataLog_Run.sh won't run. The file has incorrect paths for command and config file.

How To Reproduce the bug
Change to the directory with the USB_DataLog_Run script.
chmod 777 to make the script executable
Run the script in a terminal.
File not found errors result

Additional context
Target folder name is ./bin_linux, not ./bin_unix
The default config file has a lower-case name with an underscore

Screenshots

Battery measurement return voltage = 0

Hi all

I am currently using the fp-sns-datalog2 battery powered and I'm faced with a random effect.

While a SD card data recording is running, the recording stops to works after an irregular time interval. I go trough the steps which lead to this behaviour and I found out, that sometime the battery measurement returned by the function UtilTask_GetBatteryStatus, resp BCPAcquireState is equal to zero. This lead sending a message of type DT_USER_BUTTON in the function DatalogAppTaskAdvOBTimerCallbackFunction and stop data recording.

I use fp-sns-datalog2 version 2.0.1 on the STWIN.box wiht the implementation of a custom sensor type whose data is read periodically (20 ms intervals) via the timer implemented in the corresponding task.

My IDE is Keil MDK uVision 5.38.0.0

I do not have taken a closer look to the battery management and my workaround is to discard battery values of zero in the DatalogAppTaskAdvOBTimerCallbackFunction

Do you have a suspected reason for this behavior and a method to correct it?

Kind resgards,
Ludwig

Does not run on STEVAL-STWINKT1B

Setup

• STEVAL-STWINKT1B
• CubeIDE 1.13.2

Describe the bug

Cloning the project and compiling the respective project (Projects/Applications/STM32L4R9ZI-STWIN) using CubeIDE works.
Flashing the project (ST Link V3) on to the board works, however the project does not appear to be running.

• "dmesg" shows a "STM32 BOOTLOADER" attached instead of the HSD device
• LED not blinking
• No BLE service discoverable

Flashing the datalog1 firmware project using the same method also works and the firmware runs. dmesg shows the HSD device.

How To Reproduce the bug

1. Flash this repository's Projects/Applications/STM32L4R9ZI-STWIN project on a freshly bought STEVAL-STWINKT1B

Does a scaling take place when retriving data from file?

I have recorded data to a SD card using the STWIN.box and I want to do more data analysis (e.g. filtering) on it.

The sensor I uses is "iis3dwb_acc", when I retrive the data using HSDatalog.get_dataframe it seems that the data are not the same as when they are plotted with HSDatalog.plot does a scaling take place? Do I have to uses HSDatalog.get_data_and_timestamps to get same data?

DTDL2PnPL_cGen

I'm adding a sensor and I wonder if it would be possible to get the DTDL2PnPL_cGen generator instead of having to implement a PnPL module by hand.