recent change on download_garmin.py? about garmindb

[benbowen]

Any ideas @tcgoetz ? I did a conda update --all for better or worse (no effect for this bug). My computer also did an auto-restart and installed latest Mac OS updates. These updates made me change the way my VPN software worked. Looking at the header for login def, I can imagine the problem is Garmin changed something there.

[tcgoetz]

Sorry. I've been away. I hit this last week and made some changes to login. It's working for me now. Get the latest and see if your still having issues.

[benbowen]

No worries! You are amazing for putting this thing together!

I'm worried that I should not have done "update -all"

It looks like my version of sqlalchemy is freaking out.

INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 0 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Importing /Users/bpb/HealthData/FitFiles/2018_Monitoring//21685089764.fit (2018-06-16 19:04:59) [FileType.monitoring_b] with message types: [<MessageType.file_id: 0>, <MessageType.device_info: 23>, <MessageType.software: 35>, <MessageType.monitoring_info: 103>, <MessageType.monitoring: 55>, <UnknownMessageType.unknown_211: 211>]

Traceback (most recent call last):
  File "import_garmin.py", line 301, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 287, in main
    gfd.process_files(db_params_dict)
  File "import_garmin.py", line 79, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 67, in write_file
    self.write_message_types(fit_file, fit_file.message_types())
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 59, in write_message_types
    self.write_message_type(fit_file, message_type)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 46, in write_message_type
    function(fit_file, message_type, messages)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 38, in write_generic
    function(fit_file, message)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 339, in write_monitoring_info_entry
    GarminDB.MonitoringInfo.find_or_create(self.garmin_mon_db, entry)
  File "/Users/bpb/repos/garmin_activity/GarminDB/HealthDB/DB.py", line 231, in find_or_create
    DB.commit(session)
  File "/Users/bpb/repos/garmin_activity/GarminDB/HealthDB/DB.py", line 58, in commit
    session.commit()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
    self._flush(objects)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
    flush_context.execute()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 830, in _emit_insert_statements
    execute(statement, multiparams)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: monitoring_info.file_id [SQL: u'INSERT INTO monitoring_info (timestamp, file_id, activity_type, resting_metabolic_rate, cycles_to_distance, cycles_to_calories) VALUES (?, ?, ?, ?, ?, ?)'] [parameters: ('2018-06-16 19:05:00.000000', None, 'walking', 2784, 1.829, 0.09183998724998008)] (Background on this error at: http://sqlalche.me/e/gkpj)
make: *** [import_monitoring] Error 1

[tcgoetz]

file_id is None which shouldn't happen. If you run:

make TEST_GC_ID=21685089764 test_import_fit_activities

what do the logs just before the exception say? Can you paste them in here? You must be getting an exception in write_file_id_entry.

[tcgoetz]

Probably looking for the output of htis line:

                logger.debug("No entry handler %s for message type %s (%d) from %s: %s",
                    handler_name, repr(message_type), len(messages), fit_file.filename, str(messages[0]))

[benbowen]

Here it is. Whenever I go to my kick-boxing class, the Garmin watch often automatically starts either swimming or an indoor cycling activity; so likely the activity being parsed is one of those two activity types.

There is a lengthy log of battery, system settings, user-profile, position, heart-rate, cadence messages. The log ends with a message and traceback like you say above.

DEBUG:Fit.File:  Message [6]: DataMessage: <UnknownMessageType.unknown_113: 113>: unknown_4(893258310 (893258310)), unknown_5(0 (0)), unknown_2(100000 (100000)), unknown_3(1343662 (1343662)), unknown_0(0 (0)), unknown_1(1 (1)), timestamp(2018-04-21 08:41:00 (893259661)), 
DEBUG:Fit.File:File /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: 2018-04-21 08:01:13 -> 2018-04-21 08:41:00
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Importing /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit (2018-04-21 08:01:13) [FileType.activity] with message types: [<MessageType.file_id: 0>, <MessageType.file_creator: 49>, <MessageType.event: 21>, <MessageType.device_info: 23>, <MessageType.source: 22>, <MessageType.unknown_141: 141>, <MessageType.device_settings: 2>, <MessageType.user_profile: 3>, <UnknownMessageType.unknown_79: 79>, <MessageType.sport: 12>, <MessageType.unknown_13: 13>, <MessageType.zones_target: 7>, <MessageType.record: 20>, <MessageType.lap: 19>, <UnknownMessageType.unknown_216: 216>, <MessageType.unknown_140: 140>, <MessageType.session: 18>, <MessageType.activity: 34>, <UnknownMessageType.unknown_113: 113>]
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:No entry handler write_file_id_entry for message type <MessageType.file_id: 0> (1) from /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: DataMessage: <MessageType.file_id: 0>: product(GarminProduct.VivoActive_HR (2337)), product(GarminProduct.VivoActive_HR (2337)), time_created(2018-04-21 08:01:13 (893257274)), number(None (65535) [invalid]), serial_number(3935780983 (3935780983)), type(FileType.activity (4)), product_name(4294967295 (4294967295) [invalid]), manufacturer(Manufacturer.Garmin (1)), 
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Processed 1 <MessageType.file_id: 0> entries for /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:No entry handler write_device_info_entry for message type <MessageType.device_info: 23> (12) from /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: DataMessage: <MessageType.device_info: 23>: battery_voltage(None v (65535) [invalid]), device_type(None (255) [invalid]), unknown_8(None (4294967295) [invalid]), unknown_9(None (255) [invalid]), unknown_13(None (65535) [invalid]), unknown_15(None (4294967295) [invalid]), unknown_16(None (4294967295) [invalid]), unknown_17([None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None] ([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])), product(GarminProduct.VivoActive_HR (2337)), cum_operating_time(None (4294967295) [invalid]), device_index(0 (0)), ant_transmission_type(None (0) [invalid]), serial_number(3935780983 (3935780983)), product(GarminProduct.VivoActive_HR (2337)), software_version(4.3 (430)), timestamp(2018-04-21 08:01:13 (893257274)), source_type(SourceType.local (5)), ant_device_number(None (0) [invalid]), unknown_24(None (0) [invalid]), unknown_23(None (255) [invalid]), manufacturer(Manufacturer.Garmin (1)), hardware_version(None (255) [invalid]), battery_status(None (255) [invalid]), ant_network(None (255) [invalid]), 
WARNING:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:device_info: {'battery_voltage': None, 'device_type': 4, 'unknown_8': None, 'unknown_9': None, 'unknown_13': None, 'unknown_15': None, 'unknown_16': None, 'unknown_17': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], 'Product': <GarminProduct.VivoActive_HR: 2337>, 'cum_operating_time': None, 'device_index': 1, 'ant_transmission_type': None, 'serial_number': None, 'product': <GarminProduct.VivoActive_HR: 2337>, 'software_version': 4.3, 'timestamp': datetime.datetime(2018, 4, 21, 8, 1, 13), 'source_type': <SourceType.local: 5>, 'ant_device_number': None, 'unknown_24': None, 'unknown_23': None, 'manufacturer': <Manufacturer.Garmin: 1>, 'hardware_version': None, 'battery_status': None, 'ant_network': None} - 'int' object has no attribute 'value'
ERROR:import_garmin_activities.py:Failed to parse /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: Failed to translate device_info
Traceback (most recent call last):
  File "import_garmin_activities.py", line 399, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 395, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 45, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 67, in write_file
    self.write_message_types(fit_file, fit_file.message_types())
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 56, in write_message_types
    self.write_message_type(fit_file, message_type)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 46, in write_message_type
    function(fit_file, message_type, messages)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 38, in write_generic
    function(fit_file, message)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 376, in write_device_info_entry
    raise ValueError('Failed to translate device_info')
ValueError: Failed to translate device_info
make: *** [test_import_fit_activities] Error 1

[benbowen]

I'm fairly confident that this error is because I was out of sync between my Fit rep and GarminDB repo. I did a git pull on Fit and was able to get through a ton of activity files without error.