Exception on entry about garmindb

[tcgoetz]

That's expected. "Exception on entry" is saying that a monitoring entry has undocumented fields 37 and 38. The other log line is saying that the file as undocumented messages with Ids 24 and 211.

If I can get the field and message formats, I can add them and those messages with be decoded. Garmin uses a bunch of messages and fields that are not detailed int he SDK.

[umaar]

Thank you for explaining!

[benbowen]

Yesterday, I pulled and rebuilt my database. I also get the 24 and 211 "unknown_msg_X" INFO printed hundreds of times.

Is there something I can do to help pinpoint the missing attributes and values in the fit files? If yes, I will need some help getting started with that. The log tells me the fit files which contain the unexpected field names.

[tcgoetz]

@benbowen There are make targets test_import_activities and test_monitoring_file that allow you to specify a single FIT file with TEST_FIT_FILE or TEST_ACTIVITY_ID. They dump the corresponding FIT file with debugging turned up. You can file a bug with the log line for the unknown message. If there are multiple samples of that message with different values, please include a few.

[benbowen]

Sorry @tcgoetz, but I can't figure out the syntax to specify a fit file. I tried setting environment variables of "TEST_FILE_FILE" and "TEST_ACTIVITY_ID" to a filename, but when I execute "make test_import_activities" or "make test_monitoring_file" a different fit file is used. I'm doing this wrong....

Here is terminal output:

[bpb@Benjamins-MacBook-Pro-4 ~ 11:49 AM] ~/repos/garmin_activity/GarminDB (master) > export TEST_ACTIVITY_ID=24955272738
[bpb@Benjamins-MacBook-Pro-4 ~ 11:54 AM] ~/repos/garmin_activity/GarminDB (master) > make test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
DEBUG:HealthDB.DB:DB {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug 0 
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 1 English units: True
Traceback (most recent call last):
  File "import_garmin_activities.py", line 401, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 397, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 43, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/Fit/File.py", line 30, in __init__
    self.file = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit'
make: *** [test_import_activities] Error 1
[bpb@Benjamins-MacBook-Pro-4 ~ 11:54 AM] ~/repos/garmin_activity/GarminDB (master) > export TEST_FIT_FILE=24955272738
[bpb@Benjamins-MacBook-Pro-4 ~ 11:56 AM] ~/repos/garmin_activity/GarminDB (master) > make test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
DEBUG:HealthDB.DB:DB {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug 0 
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 1 English units: True
Traceback (most recent call last):
  File "import_garmin_activities.py", line 401, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 397, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 43, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/Fit/File.py", line 30, in __init__
    self.file = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit'
make: *** [test_import_activities] Error 1

[tcgoetz]

@benbowen because they are Make variables not env variables.

Try:
make TEST_ACTIVITY_ID=24955272738 test_import_activities

I will push a commit to clean up running a single file in debug to get this info soon.

[benbowen]

Still not getting it. Here is info log when scraping new data:

INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:/Users/bpb/HealthData/FitFiles/2018_Monitoring//24955272738.fit (2018-09-19 23:59:59) [monitoring_b] message types: ['file_id', 'device_info', 'software', 'monitoring_info', 'monitoring', 'unknown_msg_24', 'event', 'unknown_msg_211']

Here is my attempt to test import it

[bpb@Benjamins-MacBook-Pro-4 ~ 12:54 PM] ~/repos/garmin_activity/GarminDB (master) > make TEST_FIT_FILE=24955272738 test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs

I omit the rest since you can see its not the specified file or path. It jumps out with an IOError: no such file.

The same thing happens no matter which combination of make targets and variable names I pick.

The only obvious thing I see is 2017 is hard-coded into path here:
https://github.com/tcgoetz/GarminDB/blob/master/Makefile#L148

[tcgoetz]

I'm not sure which version you have, but if you update, the latest works for me like:

orion-dock:GarminDB_master tgoetz$ make TEST_GC_ID=1654887240 test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/tgoetz/HealthData/FitFiles/Activities/1654887240.fit" --sqlite /Users/tgoetz/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/tgoetz/Projects/GarminDB_master/FileProcessor.pyc:Reading file: /Users/tgoetz/HealthData/FitFiles/Activities/1654887240.fit

The makefile line uses a ?= so it sets if not set.

[tcgoetz]

After which you get output like:

DEBUG:Fit.File: Message [14]: DataMessage: <UnknownMessageType.unknown_233: 233>: unknown_2([5, 1, 0, 0] ([5, 1, 0, 0])),

which means the file contained an unknown message #233 with one unknown filed #2 consisting of an array of four numbers.

[benbowen]

My bad. I swear I did a git pull a couple days ago, but I just pulled again and a dozen files were updated. I'll reinvistigate now that I'm "more" certain that I'm on current master.

[tcgoetz]

I've been pushing a lot of code over the past few days.

Here's an example of a known message:

DEBUG:Fit.File: Message [3]: DataMessage: <MessageType.device_info: 23>: battery_voltage(None v (65535) [invalid]), device_type(None (255) [invalid]), unknown_8(None (4294967295) [invalid]), unknown_9(None (255) [invalid]), unknown_13(None (65535) [invalid]), unknown_15(None (4294967295) [invalid]), unknown_16(None (4294967295) [invalid]), unknown_17([None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None] ([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])), unknown_30(None (255) [invalid]), sensor_position(None (255) [invalid]), product(GarminProduct.Fenix5_Sapphire (2697)), cum_operating_time(None (4294967295) [invalid]), device_index(0 (0)), ant_transmission_type(None (0) [invalid]), serial_number(3943966595 (3943966595)), product(GarminProduct.Fenix5_Sapphire (2697)), software_version(2.4 (240)), timestamp(2017-04-02 13:45:24 (860089525)), source_type(SourceType.local (5)), unknown_29([None, None, None, None, None, None] ([255, 255, 255, 255, 255, 255])), ant_device_number(None (0) [invalid]), unknown_24(None (0) [invalid]), unknown_23(None (255) [invalid]), manufacturer(Manufacturer.Garmin (1)), hardware_version(None (255) [invalid]), battery_status(None (255) [invalid]), ant_network(None (255) [invalid]),

Note that even in a known message, there are unknown fields and unused fields ('invalid').