tcgoetz / garmindb

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.

---

---

GarminDB

Python scripts for parsing health data into and manipulating data in a SQLite database. SQLite is a light weight database that doesn't require a server.

What they can do:

• Automatically download and import Garmin daily monitoring files (all day heart rate, activity, climb/descend, stress, and intensity minutes) from the user's Garmin Connect "Daily Summary" page.
• Extract sleep, weight, and resting heart rate data from Garmin Connect, store it as JSON files, and import it into the DB.
• Download and import activity files from Garmin Connect. A summary table for all activities and more detailed data for some activity types. Lap and record entries for activities.
• Summarizing data into a DB with tables containing daily, weekly, monthly, and yearly summaries.
• Graph your data from the commandline or with Jupyter notebooks.
• Retain downloaded JSON and FIT files so that the DB can be regenerated without connecting to or redownloading data from Garmin Connect.
• Export activities as TCX files.

Once you have your data in the DB, I recommend using a supplied Jupyter notebooks, third party Jupyter notebooks, and/or SQLite browser like SQLite Studio, HeidiSQL, or DB Browser for SQLite for browsing and working with the data. The scripts create some default views in the DBs that make browsing the data easier.

Using It

Releases

GarminDb releases are hosted on PyPI. GarminDb requires Python 3.x. With Python installed, install the latest release with pip by running pip install garmindb in a terminal.

• Copy GarminConnectConfig.json.example to ~/.GarminDb/GarminConnectConfig.json, edit it, and add your Garmin Connect username and password and adjust the start dates to match the dates of your data in Garmin Connect.
• Starting out: download all of your data and create your db by running garmindb_cli.py --all --download --import --analyze in a terminal.
• Incrementally update your db by downloading the latest data and importing it by running garmindb_cli.py --all --download --import --analyze --latest in a terminal.
• Ocassionally run garmindb_cli.py --backup to backup your DB files.

Update to the latest release with pip install --upgrade garmindb.

From Source

The scripts are automated with Make. Run the Make commands in a terminal window.

• Git clone GarminDB repo using the SSH clone method. The submodules require you to use SSH and not HTTPS. Get the command from the green button on the project home page.
• Run make setup in the cloned tree to get the scripts ready to process data.
• Copy GarminConnectConfig.json.example to ~/.GarminDb/GarminConnectConfig.json, edit it, and add your Garmin Connect username and password and adjust the start dates to match the dates of your data in Garmin Connect.
• Run make create_dbs once to fetch and process for you data.
• Keep all of your local data up to date by periodically running only one command: make.

There is more help on using the program in the wiki.

Jupyter Notebooks

Jupyter notebooks for analzing data from the database can be found in the 'Jupyter' directory in the source tree. Links to user submitted notebooks can be found in the wiki.

Plugins

Plugins allow the user to expand the types of data that are processed and stored in the database. GarminDb already has a number of plugins for handling data from third-party Connect IQ apps and data fields. Read more about plugins here.

Support This Project

Do you find this project useful?

Success Stories

Find out who's using GarminDb on what platforms, OSes, and python versions here. If you're using GarminDB and your scenario isn't listed send me a message or file an issue with your success case.

Notes

• You may get a DB version exception after updating the code, this means that the DB schema was updated and you need to rebuild your DBs by running garmindb_cli.py --rebuild_db. Your DBs will be regenerated from the previously downloaded data files. All of your data will not be redownloaded from Garmin.
• The scripts were developed on MacOS. Information or patches on using these scripts on other platforms are welcome.
• When a database update finishes, a summary of the data in the DB will be saved to stats.txt. The output includes the date ranges included in the downloaded daily monitoring files and activities. It includes the number of records for daily monitoring, activities, sleep, resting heart rate, weight, etc. Use the summary information to determine if all of your data has been downloaded from Garmin Connect. If not, adjust the dates in GarminConnectConfig.json and runt he download again.
• In GarminConnectConfig.json the "steps" element of the "course_views" is list of course ids that per course database views will be generated for. The database view allows you to compare all activities from that course.

Bugs and Debugging

• If you have issues, file a bug here on the project. See the Issues tab at the top of the project page. Run make bugreport or garmindb_bug_report.py and include bugreport.txt in your bug report.
• Besides errors that appear on the screen, one of the first places to look for more information is the log files (garmindb.log).
• If you're having issues with a particular data files, please considering sharing so I can debug it and add support.

Contributing

Please submit a pull request targeting the develop branch and add your self to the contributors file. Run make flake8 at the top level and fix all errors before submitting your pull request.

garmindb's Issues

Error after new skiing activity

After I have completed two "Skiing" activities, I get the following error message:
(At least I think it has something to do with those activities)

I'm on the newest 1.2.1 release.

Processing FileType.activity FIT data from /home/christian/HealthData/FitFiles/Activities
Traceback (most recent call last):
  File "garmin.py", line 388, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 381, in main
    analyze_data(debug)
  File "garmin.py", line 224, in analyze_data
    analyze.get_stats()
  File "/home/christian/Projekte/GarminDB/analyze_garmin.py", line 173, in get_stats
    self.__get_activities_stats()
  File "/home/christian/Projekte/GarminDB/analyze_garmin.py", line 94, in __get_activities_stats
    stat_logger.info("Sports: %s", ', '.join(sports))
TypeError: sequence item 6: expected str instance, NoneType found
Command exited with non-zero status 1
/usr/bin/time
Makefile:121: recipe for target 'update_garmin' failed
make: *** [update_garmin] Error 1

Crashes when no data is available for date

I believe the current code is crashing when no data exists on server for a given date. For example, today's date (I haven't synced my watch today).

INFO:root:get_monitoring_day: 2018-10-09
Traceback (most recent call last):
  File "download_garmin.py", line 445, in <module>
    main(sys.argv[1:])
  File "download_garmin.py", line 428, in main
    download.get_monitoring(date, days)
  File "download_garmin.py", line 176, in get_monitoring
    self.get_monitoring_day(day_date)
  File "download_garmin.py", line 168, in get_monitoring_day
    response = self.get(self.garmin_connect_download_daily_url + '/' + date.strftime("%Y-%m-%d"))
  File "download_garmin.py", line 76, in get
    response.raise_for_status()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://connect.garmin.com/modern/proxy/download-service/files/wellness/2018-10-09
make: *** [download_monitoring] Error 1

DBs folder no longer created

I set up a new instance, ran the initial command to get data from GC and then ran the make import_new_monitoring command and it gave me an error about the DBs folder being missing. I created the DBs folder manually and then ran the make import_new_monitoring command and everything went as planned.

Running make on linux complains about pip

I'm trying to install and run the package on linux in a Python 2.7 conda environment, and I'm getting stuck at make setup. I see an error for pip (copied below), but I'm not sure what's going on because pip is definitely installed in this environment. I also manually installed sqlalchemy with conda to see if this would skip that part of make (it doesn't). Can you suggest any debugging tips?

...
Installed /usr/lib/python2.7/site-packages/python_tcxparser-0.7.2-py2.7.egg
Processing dependencies for python-tcxparser==0.7.2
Searching for lxml==3.2.1
Best match: lxml 3.2.1
Adding lxml 3.2.1 to easy-install.pth file

Using /usr/lib64/python2.7/site-packages
Finished processing dependencies for python-tcxparser==0.7.2
writing list of installed files to 'files.txt'
sudo pip install --upgrade sqlalchemy
sudo: pip: command not found
make: *** [deps] Error 1

...
(py27) Tue Sep 11 07:52:31 warthog:~/data/explorations/GarminDB
$ pip --version
pip 18.0 from /home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/site-packages/pip (python 2.7)

numerical error when generating graphs

Hi,

love the possibility to generate graphs. I had the following error:

$ make graphs
/usr/local/bin/python3 graphs.py --all
(...)
  File "/Users/laurentperrinet/quantic/Documents/outdoor/GarminDB/HealthDB/summary_base.py", line 82, in intensity_time_goal_percent
    return (conversions.time_to_secs(self.intensity_time) * 100) / conversions.time_to_secs(self.intensity_time_goal)
ZeroDivisionError: division by zero
make: *** [graphs] Error 1

any idea? would be glad to test.
cheers,
Laurent

create_dbs issue - Ubuntu 18.04

Whacked the local HealthData files and rebuilt the environment twice now but it keeps dying when I unzipping the files after I download the 742 activities from Garmin. Here's a sample of the output of the error:
Probably need to install/update a package.
Thoughts?

Processing daily FIT data
0% (9 of 1170) | | Elapsed Time: 0:00:10 ETA: 1:33:11Traceback (most recent call last):
File "garmin.py", line 344, in
main(sys.argv[1:])
File "garmin.py", line 337, in main
import_data(debug, test, latest, weight, monitoring, sleep, rhr, activities)
File "garmin.py", line 162, in import_data
gfd.process_files(db_params_dict)
File "/home/mo/Projects/GarminDB/import_garmin.py", line 91, in process_files
fp.write_file(Fit.file.File(file_name, self.measurement_system))
File "/home/mo/Projects/GarminDB/Fit/file.py", line 42, in init
self.__parse()
File "/home/mo/Projects/GarminDB/Fit/file.py", line 72, in __parse
data_message = DataMessage(definition_message, self.file, self.measurement_system)
File "/home/mo/Projects/GarminDB/Fit/data_message.py", line 31, in init
data_field = DataField(fit_file, definition_message, definition_message.field_definitions[index], measurement_system)
File "/home/mo/Projects/GarminDB/Fit/data_field.py", line 23, in init
super(DataField, self).init(file, schema, None, definition_message.endian)
File "/home/mo/Projects/GarminDB/Fit/data.py", line 103, in init
self._convert()
File "/home/mo/Projects/GarminDB/Fit/data_field.py", line 26, in _convert
self.value_obj = self.field.convert(self.dict[self.field.name], self.field_definition.invalid(), self.measurement_system)
File "/home/mo/Projects/GarminDB/Fit/fields.py", line 124, in convert
value_obj = self.obj_func(value / self.scale, invalid)
TypeError: 'int' object is not callable
Makefile:125: recipe for target 'garmin' failed
make: *** [garmin] Error 1

Exception on entry

After running this command: make GC_USER='username' GC_PASSWORD='password'

I can see it starts populating the database, but I see a lot of these errors in the terminal during execution, any ideas?

INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 2, 16, 14, 41, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:/Users/umarhansa/HealthData/FitFiles/2018_Monitoring//21375899400.fit [monitoring_b] message types: ['file_id', 'device_info', 'software', 'monitoring_info', 'monitoring', 'unknown_msg_211', 'event', 'unknown_msg_24']
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 6, 7, 0, 52, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 6, 7, 16, 52, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'

Really cool project by the way!

Unable to enter date

The command I try;
make GC='01/01/2018' GC_DAYS=35 GC_USER=[email protected] GC_PASSWORD=gcpassword scrape_monitoring

The error I get;
python scrape_garmin.py -d -n 35 -u [email protected] -p gcpassword -m "/home/pi/HealthData/FitFiles/Incoming"
Traceback (most recent call last):
File "scrape_garmin.py", line 297, in
main(sys.argv[1:])
File "scrape_garmin.py", line 225, in main
date = datetime.datetime.strptime(arg, "%m/%d/%Y").date()
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '-n' does not match format '%m/%d/%Y'
Makefile:90: recipe for target 'scrape_monitoring' failed
make: *** [scrape_monitoring] Error 1

We have tried the following date code combinations;
1/1/18, 01/01/18, 1/1/2018, 01/01/2018, 1-1-18, 01-01-18, 01-10-2018
'1/1/18', '01/01/18', '1/1/2018', '01/01/2018', '1-1-18', '01-01-18', '01-10-2018'
"1/1/18", "01/01/18", "1/1/2018", "01/01/2018", "1-1-18", "01-01-18", "01-10-2018"
All of them return the same error code.

Running on a Raspberry Pi;
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 8.0 (jessie)
Release: 8.0
Codename: jessie
I am on make version 4.0.
Python 2.7.9

sqlite3.IntegrityError during rebuild_db

Hi,
I'm using this on bash for windows, and the scrape of fit files works perfectly, but when importing/rebuilding the below error is generated. this happens for all .fit files:

INFO:import_garmin.py:Debug: False English units: True
INFO:/home/dale/GarminDB/FileProcessor.pyc:Reading directory: /root/HealthData/FitFiles/2018_Monitoring/
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/root/HealthData/DBs'} debug: -1
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/root/HealthData/DBs'} debug: -1
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/root/HealthData/DBs'} debug: -1
INFO:/home/dale/GarminDB/FitFileProcessor.pyc:Debug: False English units: True
INFO:/home/dale/GarminDB/FitFileProcessor.pyc:Importing /root/HealthData/FitFiles/2018_Monitoring//24287393166.fit (2018-09-01 06:39:59) [FileType.monitoring_b] with message types: [<MessageType.file_id: 0>, <MessageType.device_info: 23>, <MessageType.software: 35>, <MessageType.monitoring_info: 103>, <MessageType.monitoring: 55>, <UnknownMessageType.unknown_211: 211>]
Traceback (most recent call last):
File "import_garmin.py", line 301, in
main(sys.argv[1:])
File "import_garmin.py", line 287, in main
gfd.process_files(db_params_dict)
File "import_garmin.py", line 79, in process_files
fp.write_file(Fit.File(file_name, self.english_units))
File "/home/dale/GarminDB/FitFileProcessor.py", line 62, in write_file
self.write_message_types(fit_file, fit_file.message_types())
File "/home/dale/GarminDB/FitFileProcessor.py", line 57, in write_message_types
self.write_message_type(fit_file, message_type)
File "/home/dale/GarminDB/FitFileProcessor.py", line 44, in write_message_type
function(fit_file, message_type, messages)
File "/home/dale/GarminDB/FitFileProcessor.py", line 37, in write_generic
function(fit_file, message)
File "/home/dale/GarminDB/FitFileProcessor.py", line 332, in write_monitoring_info_entry
GarminDB.MonitoringInfo.find_or_create(self.garmin_mon_db, entry)
File "/home/dale/GarminDB/HealthDB/DB.py", line 231, in find_or_create
DB.commit(session)
File "/home/dale/GarminDB/HealthDB/DB.py", line 58, in commit
session.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 943, in commit
self.transaction.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 467, in commit
self._prepare_impl()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2254, in flush
self._flush(objects)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2380, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2344, in _flush
flush_context.execute()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
rec.execute(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
uow
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
mapper, table, insert)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 830, in _emit_insert_statements
execute(statement, multiparams)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: monitoring_info.file_id [SQL: u'INSERT INTO monitoring_info (timestamp, file_id, activity_type, resting_metabolic_rate, cycles_to_distance, cycles_to_calories) VALUES (?, ?, ?, ?, ?, ?)'] [parameters: ('2018-09-01 06:40:00.000000', None, 'walking', 2447, 1.729, 0.0677344808351263)] (Background on this error at: http://sqlalche.me/e/gkpj)
make: *** [import_monitoring] Error 1

Make Setup does not download Fit Module

Running make setup on a new install gives the following error:

GarminDB Havens$ make setup
git submodule init
Submodule 'Fit' (https://github.com/tcgoetz/Fit.git) registered for path 'Fit'
Submodule 'garmin-connect-export' (https://github.com/tcgoetz/garmin-connect-export.git) registered for path 'garmin-connect-export'
Submodule 'python-tcxparser' (https://github.com/tcgoetz/python-tcxparser.git) registered for path 'python-tcxparser'
git submodule update
Cloning into '/Users/Havens/Documents/GitHub/GarminDB/Fit'...
Cloning into '/Users/Havens/Documents/GitHub/GarminDB/garmin-connect-export'...
Cloning into '/Users/Havens/Documents/GitHub/GarminDB/python-tcxparser'...
error: Server does not allow request for unadvertised object 5b58d4f5b7e59f859a69200fc51819ecc748d0ef
Fetched in submodule path 'Fit', but it did not contain 5b58d4f5b7e59f859a69200fc51819ecc748d0ef. Direct fetching of that commit failed.
make: *** [submodules_update] Error 1

Issue when creating a new DB using the Mac Binaries

I am trying to create a new DB using the Mac binaries. After ruuning the download_create_dbs.sh script, I got the following error:

Creating new DBs by downloading from Garmin Connect
Geting activities: '/Users/u1560476/HealthData/FitFiles/Activities' (10000)
100% (71 of 71) |#######################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
unzip_files: /Users/u1560476/HealthData/FitFiles/Activities
100% (161 of 161) |#####################################################################################################| Elapsed Time: 0:04:04 Time: 0:04:04
unzip_files: /Users/u1560476/HealthData/FitFiles/Monitoring/2019
Traceback (most recent call last):
File "garmin.py", line 339, in
File "garmin.py", line 329, in main
File "garmin.py", line 107, in download_data
File "download_garmin.py", line 174, in unzip_files
File "zipfile.py", line 1225, in init
File "zipfile.py", line 1292, in _RealGetContents
zipfile.BadZipFile: File is not a zip file
[10328] Failed to execute script garmin

Any thoughts on that?

Request for daily hydration data.

Would be nice to have hydration data in the daily summary.

"There is no data to export for this date"

When scraping the Garmin website i've seen this notice periodically pop up. Generally the screen is fully white and this popup comes up in the firefox window. It seems like this may happen when the page is not fully loaded and the script attempts to export data that is not yet available.

It does not always happen on the same date. For example I was updating the database and the script errored on 8/27/17. I removed the temp files and re-ran the script and it passed 8/27/17 without any issues.

The script then errors out completely and if you re-start - the program will begin again where the database has left off. This causes the script to re-download the zip files between the start date and the date where the error occured. It may be a good idea to have the script scan the temp folder where the zip files are downloaded first to see if there is anything left over from a previous scan? Otherwise you are forced to hope that the script completes before erroring so that the zip files may be moved to the incoming folder.

You can get around this by unzipping/moving the fit files to the incoming folder by hand, running the

make import_monitoring

command and then removing the incoming fit files after they have been imported

can not change HEALTH_DATA_DIR

I have been changing HEALTH_DATA_DIR in defines.mk to another directory, but is stills saves data in ~/HealthData/. I have tried many quirks but to no avail.

And .. Thanks a lot for your great library!

recent change on download_garmin.py?

The past two days, my history failed to download. Did Garmin change their API? Are other people able to download their monitoring files? I checked through a browser and I can login fine with the credentials I'm using.

The command (called from make):

python download_garmin.py -l --sqlite /Users/bpb/HealthData/DBs -u <name> -p <pass> -m "/Users/bpb/HealthData/FitFiles/2018_Monitoring"

The traceback:

Traceback (most recent call last):
  File "download_garmin.py", line 441, in <module>
    main(sys.argv[1:])
  File "download_garmin.py", line 389, in main
    download.login(username, password)
  File "download_garmin.py", line 132, in login
    response = self.get(self.garmin_connect_modern_url, params)
  File "download_garmin.py", line 72, in get
    response.raise_for_status()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://connect.garmin.com/modern?ticket=ST-2475442-cyLN5naZfOHIzOXA2gMa-cas
make: *** [download_new_monitoring] Error 1

no module GarminConnectEnums

Is this missing from source or separate repo?

python import_garmin_activities.py -e -l --input_dir "/Users/bpb/HealthData/FitFiles/Activities" --sqlite /Users/bpb/HealthData/DBs
Traceback (most recent call last):
  File "import_garmin_activities.py", line 13, in <module>
    import GarminConnectEnums
ImportError: No module named GarminConnectEnums

Issue when creating a new DB

After cloning the repo, and run make setup, I tried to create my first DB by running make create_dbs, but the following error appeared:

Traceback (most recent call last):
File "garmin.py", line 18, in
from version import print_version, python_version_check, log_version
File "/Users/u1560476/Documents/GarminDB/version.py", line 7, in
import utilities.version as uv
File "/Users/u1560476/Documents/GarminDB/utilities/init.py", line 11, in
from utilities.db import DbParams, DB, DBObject
File "/Users/u1560476/Documents/GarminDB/utilities/db.py", line 35
return f'<{self.class.name}() {repr(self.dict)}'
^
SyntaxError: invalid syntax
Traceback (most recent call last):
File "garmin.py", line 18, in
from version import print_version, python_version_check, log_version
File "/Users/u1560476/Documents/GarminDB/version.py", line 7, in
import utilities.version as uv
File "/Users/u1560476/Documents/GarminDB/utilities/init.py", line 11, in
from utilities.db import DbParams, DB, DBObject
File "/Users/u1560476/Documents/GarminDB/utilities/db.py", line 35
return f'<{self.class.name}() {repr(self.dict)}'
^
SyntaxError: invalid syntax
time /usr/local/bin/python3 garmin.py --all --download --import --analyze
Traceback (most recent call last):
File "garmin.py", line 18, in
from version import print_version, python_version_check, log_version
File "/Users/u1560476/Documents/GarminDB/version.py", line 7, in
import utilities.version as uv
File "/Users/u1560476/Documents/GarminDB/utilities/init.py", line 11, in
from utilities.db import DbParams, DB, DBObject
File "/Users/u1560476/Documents/GarminDB/utilities/db.py", line 35
return f'<{self.class.name}() {repr(self.dict)}'
^
SyntaxError: invalid syntax
0.07 real 0.05 user 0.01 sys
make: *** [garmin] Error 1

Any suggestions? Of course, I replaced the example .json file with the one having the information I want.

Unable to start program

When using the expected input

make GC_DATE=01/01/2018 GC_DAYS=10 GC_USER=username GC_PASSWORD=password scrape_monitoring

I get the response:

python scrape_garmin.py -d 01/01/2018 -n 10 -u username -p password  -m "/Users/USERNAME/HealthData/FitFiles/Incoming"
Missing arguments:
scrape_garmin.py -d [<date> -n <days> | -l <path to dbs>] -u <username> -p <password> [-m <outdir> | -w <outdir>]
  -d <date ex: 01/21/2018> -n <days> fetch n days of monitoring data starting at date
  -l check the garmin DB and find out what the most recent date is and fetch monitoring data from that date on
  -m <outdir> fetches the daily monitoring FIT files for each day specified, unzips them, and puts them in outdit
  -w  fetches the daily weight data for each day specified and puts them in the DB

I can't seem to get around this, all of the arguments are filled out as far as I can tell. Thanks

Database does not retain data

Looking at this from the README:

Keep all of your local data up to date by running only one command: make GC_USER={username} GC_PASSWORD={password}

When I do this, the program fails because it tries to get information from 2 years ago (which does not exist)

I have since solved this issue, it looks like the script by default starts to grab data from 2 years ago if no date is given. This caused an issue for me because I only have data going back to 6/7/17. I changed to manually specify this date on line 276 of your scrape_garmin.py file

#date = datetime.datetime.now().date() - datetime.timedelta(days)
date = datetime.datetime(2017, 06, 07)

However, after this finishes and reaches the current day, it continues to go PAST the current day and triers tomorrow. The garmin window in firefox just returns the current day and the script continues to move forward over and over again.

Also no data is actually entered into the database - but the database file is created.

garmin_activities.db empty

I'm not sure if this is yet supported or not as it doesn't specify in the readme file, but the garmin_activities.db is created but nothing gets imported into it. I can see the tables and the columns, but no actual data.

Just curious if this should be populated with data or if it is part of a future plan.

Thanks

Initial Scrape Issue

When running

make GC_DATE=02/01/2018 GC_DAYS=25 GC_USER=USER GC_PASSWORD=PWscrape_monitoring

It begins to scrape but fails with error

No handlers could be found for logger "scrape_garmin.py"
Traceback (most recent call last):
File "scrape_garmin.py", line 330, in
main(sys.argv[1:])
File "scrape_garmin.py", line 307, in main
scrape.get_monitoring(date, days)
File "scrape_garmin.py", line 147, in get_monitoring
self.browse_daily_page(profile_name, day_date)
File "scrape_garmin.py", line 135, in browse_daily_page
page_container = self.wait_for_pagecontainer(self.browser, 10)
File "scrape_garmin.py", line 103, in wait_for_pagecontainer
return self.wait_for_xpath(driver, time_s, "//div[@id='pageContainer']")
File "scrape_garmin.py", line 99, in wait_for_xpath
return WebDriverWait(driver, time_s).until(EC.presence_of_element_located((By.XPATH, xpath)))
File "/Library/Python/2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Unrecognized device when running create_db.sh (BinaryRelease release V 1.01)

I do work with: MacOS Sierry (10.12.6)
would like to use GarminDB in combination with Garmin Instinct watch
I just downloaded the version 1.01 of the binary release.
After unpacking in a directory I called GarminDB_Darwin_101
here I edited the GarminConnectConfig.json
content (I did replace my id and password with XXX and YYYY:
cat GarminConnectConfig.json
{
"credentials" : {
"user" : "XXXX",
"password" : "YYYY"
},
"data" : {
"weight_start_date" : "06/01/2019",
"sleep_start_date" : "06/01/2019",
"rhr_start_date" : "06/01/2019",
"monitoring_start_date" : "06/01/2019",
"download_days" : 31,
"download_latest_activities" : 10,
"download_all_activities" : 1000,
"download_days_overlap" : 3
},
"copy" : {
"mount_dir" : "/Volumes/GARMIN"
}
}

Forgot to connect my Garmin Instict to the Mac
ran:
$ ./create_dbs.sh
output in 2019_0610_create_db_log.txt:

than connected the Garmin Instinct watch to the Mac and ran
$ ./update_dbs.sh
output in 2019_06_10_update_db_log.txt

Here the message Unrecognized device: {'sensor_id': None, 'device_type': <LocalDeviceType.accelerometer: 3>, 'device_index': 3, 'unknown_8': None, 'unknown_9': None, 'unknown_13': None, 'unknown_16': None, 'unknown_17': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], 'unknown_30': None, 'sensor_position': <BodyLocation.invalid: 255>, 'cum_operating_time': None, 'battery_voltage': None, 'ant_transmission_type': None, 'serial_number': None, 'unknown_31': None, 'product': <UnknownProduct.UnknownProduct_16427: 16427>, 'software_version': '0.00', 'timestamp': datetime.datetime(2019, 6, 8, 21, 53, 1), 'source_type': <SourceType.local: 5>, 'unknown_29': [None, None, None, None, None, None], 'ant_device_number': None, 'unknown_23': None, 'manufacturer': 64442, 'ant_related': None, 'hardware_version': None, 'battery_status': <BatteryStatus.invalid: 255>, 'ant_network': <AntNetwork.invalid: 255>} - 'int' object has no attribute 'name'

From what I understand: the Garmin Instinct is not recognized.

Recalling form our chat discussion, I am attaching the garmn.log
and a fit file of todays run (that I manually copied from the watch, from the activities folder and than zipped it so it can be uploaded to github)
.
Hope I did upload the files you need to directly use the file from the watch.
Best
Claus

2019_0610_create_db_log.txt
2019_06_10_update_db_log.txt
garmin.log
2019-06-10-12-16-41.zip

No rule to make target error

I am trying to get older data for weight and trying to use make download_weight and get the following?
pi@raspberrypi:/GarminDB $ make download_weight
make: *** No rule to make target 'download_weight'. Stop.
pi@raspberrypi:/GarminDB $

I was able to get data by entering
pi@raspberrypi:~/GarminDB $ /usr/bin/python3 garmin.py --weight --download --import --analyze --latest

Cannot get to run

I tried starting over from scratch and cannot get your program to run. When I run it the first time - I get this result:

python analyze_garmin.py -S22:00,06:00 --sqlite /Users/tgoetz/HealthData/DBs
analyze_garmin.py -s -m ...
python import_garmin.py -e --sleep_input_dir "/Users/USER/HealthData/Sleep" --sqlite /Users/USER/HealthData/DBs
INFO:import_garmin.py:Debug: False
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.py:Reading directory: /Users/USER/HealthData/Sleep
python import_garmin.py -e --weight_input_dir "/Users/USER/HealthData/Weight" --sqlite /Users/USER/HealthData/DBs
INFO:import_garmin.py:Debug: False English units: True
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/Weight
python import_garmin.py -e --rhr_input_dir "/Users/USER/HealthData/RHR" --sqlite /Users/USER/HealthData/DBs
INFO:import_garmin.py:Debug: False
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/RHR
for dir in /Users/USER/HealthData/FitFiles/2018_Monitoring/; do
python import_garmin.py -e --fit_input_dir "$dir" --sqlite /Users/USER/HealthData/DBs;
done
INFO:import_garmin.py:Debug: False English units: True
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/FitFiles/2018_Monitoring/
python import_garmin_activities.py -e --input_dir "/Users/USER/HealthData/FitFiles/Activities" --sqlite /Users/USER/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 0
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/FitFiles/Activities
INFO:import_garmin_activities.py:Debug: 0 English units: True
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/FitFiles/Activities
INFO:import_garmin_activities.py:Debug: 0 English units: True
INFO:/Users/USER/Documents/GitHub/GarminDB/FileProcessor.pyc:Reading directory: /Users/USER/HealthData/FitFiles/Activities
python analyze_garmin.py --analyze --dates --sqlite /Users/USER/HealthData/DBs
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/Users/USER/HealthData/DBs'} debug: -1
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/Users/USER/HealthData/DBs'} debug: -1
INFO:GarminDB.GarminSummaryDB:GarminSummaryDB: {'db_type': 'sqlite', 'db_path': '/Users/USER/HealthData/DBs'} debug: -1
INFO:HealthDB.SummaryDB:SummaryDB: {'db_type': 'sqlite', 'db_path': '/Users/USER/HealthData/DBs'} debug: -1
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/Users/USER/HealthData/DBs'} debug: -1
INFO:analyze_garmin.py:File records: 0
INFO:analyze_garmin.py:tcx files: 0
INFO:analyze_garmin.py:activity files: 0
INFO:analyze_garmin.py:monitoring_b files: 0
INFO:analyze_garmin.py:Weight records: 0
INFO:analyze_garmin.py:Max Weight: None
INFO:analyze_garmin.py:Min Weight: None
INFO:analyze_garmin.py:Avg Weight: None
INFO:analyze_garmin.py:Latest Weight: None
INFO:analyze_garmin.py:Stress records: 0
INFO:analyze_garmin.py:Max Stress: None
INFO:analyze_garmin.py:Min Stress: None
INFO:analyze_garmin.py:Avg Stress: None
INFO:analyze_garmin.py:Latest Stress: None
INFO:analyze_garmin.py:RHR records: 0
INFO:analyze_garmin.py:Max RHR: None
INFO:analyze_garmin.py:Min RHR: None
INFO:analyze_garmin.py:Avg RHR: None
INFO:analyze_garmin.py:Latest RHR: None
INFO:analyze_garmin.py:Sleep records: 0
INFO:analyze_garmin.py:Max Sleep: None
INFO:analyze_garmin.py:Min Sleep: None
INFO:analyze_garmin.py:Avg Sleep: None
INFO:analyze_garmin.py:Latest Sleep: None
INFO:analyze_garmin.py:Activity summary records: 0
INFO:analyze_garmin.py:Activities lap records: 0
INFO:analyze_garmin.py:Activity records: 0
INFO:analyze_garmin.py:Activities years: 0 ([])
INFO:analyze_garmin.py:Running activities: 0
INFO:analyze_garmin.py:Walking activities: 0
INFO:analyze_garmin.py:Cycling activities: 0
INFO:analyze_garmin.py:Mountain_Biking activities: 0
INFO:analyze_garmin.py:Hiking activities: 0
INFO:analyze_garmin.py:Elliptical activities: 0
INFO:analyze_garmin.py:Treadmill_Running activities: 0
INFO:analyze_garmin.py:Paddling activities: 0
INFO:analyze_garmin.py:Resort_Skiing_Snowboarding activities: 0
INFO:analyze_garmin.py:Monitoring records: 0
INFO:analyze_garmin.py:Monitoring Years (0): []
Traceback (most recent call last):
File "analyze_garmin.py", line 283, in
main(sys.argv[1:])
File "analyze_garmin.py", line 280, in main
analyze.summary()
File "analyze_garmin.py", line 198, in summary
sleep_period_start = GarminDB.Attributes.get_time(self.garmindb, 'sleep_time')
File "/Users/USER/Documents/GitHub/GarminDB/HealthDB/DB.py", line 472, in get_time
return datetime.datetime.strptime(cls.get(db, key), "%H:%M:%S").time()
TypeError: must be string, not None
make: *** [garmin_summary] Error 1

I'm not sure if Garmin changed something or the scripts no longer work. Thanks!

how do I use the db files e.g. for comparing workouts?

Sorry but I do not understand how to use the db files.
Is the an example case i could use as a guid?

ERROR:root:Login failed: <!DOCTYPE html>

Hi there,
first of all thanks for this great piece of Phyton scripts!
I used it for 11 days on Xubuntu Bionic LTS without any flaw. Yesterday I used without any issues, today I tried it again and now it throws errors. So I cloned the most recent version from github and started over but unfortuantely it throws the same errors. It starts with:
`make GC_DATE=2019-04-05 GC_DAYS=9 GC_USER=MyUserName GC_PASSWORD=MyPassWord create_dbs
mkdir -p /home/wechsler/HealthData/FitFiles/2019_Monitoring
python download_garmin.py -d 2019-04-05 -n 9 -u MyUserName -p MyPassWord -P "/home/wechsler/HealthData/FitFiles" -m "/home/wechsler/HealthData/FitFiles/2019_Monitoring"
INFO:root:Profile: /home/wechsler/HealthData/FitFiles
ERROR:root:Login failed:

`

and ends with:
INFO:root:Fetching 1000 activities INFO:root:get_activity_types: '/home/wechsler/HealthData/FitFiles/Activities' INFO:root:get_activities: '/home/wechsler/HealthData/FitFiles/Activities' (1000) INFO:root:get_activity_summaries Traceback (most recent call last): File "download_garmin.py", line 467, in <module> main(sys.argv[1:]) File "download_garmin.py", line 419, in main download.get_activities(activities, activity_count, overwite) File "download_garmin.py", line 236, in get_activities for activity in activities: TypeError: 'NoneType' object is not iterable Makefile:247: recipe for target 'download_all_activities' failed make: *** [download_all_activities] Error 1
However, I doublechecked my Username and Password and it works on Garmin Connnect without any issues. Perhaps you may have a hint for me how to get it working again? Thanks in advance,
wechsler42

Mac binary 1.2.4 error when processing Tcx?

I got this message after initial creating database from Connect:

Processing activities tcx data
 14% (10 of 71) |###                     | Elapsed Time: 0:00:19 ETA:   0:02:40Traceback (most recent call last):
  File "garmin.py", line 389, in <module>
  File "garmin.py", line 379, in main
  File "garmin.py", line 202, in import_data
  File "import_garmin_activities.py", line 165, in process_files
  File "import_garmin_activities.py", line 112, in __process_file
  File "Tcx/tcx.py", line 195, in read
  File "Tcx/tcx.py", line 177, in update
  File "Tcx/tcx.py", line 148, in __tag_values
  File "Tcx/tcx.py", line 148, in <listcomp>
AttributeError: 'NoneType' object has no attribute 'strip'
[8538] Failed to execute script garmin

Tried different timeframe (changed the start date) on config but it also failed at 14%.

Failed to translate device info

Any ideas on the below error?

Failed to parse /Users/Gabe/HealthData/FitFiles/Activities/3228757138.fit: Failed to translate device_info
Traceback (most recent call last):
File "garmin.py", line 289, in
main(sys.argv[1:])
File "garmin.py", line 282, in main
import_data(debug, test, latest, weight, monitoring, sleep, rhr, activities)
File "garmin.py", line 173, in import_data
gfd.process_files(db_params_dict)
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/import_garmin_activities.py", line 48, in process_files
fp.write_file(Fit.File(file_name, self.english_units))
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/FitFileProcessor.py", line 64, in write_file
self.write_message_types(fit_file, fit_file.message_types())
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/FitFileProcessor.py", line 50, in write_message_types
self.write_message_type(fit_file, message_type)
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/FitFileProcessor.py", line 40, in write_message_type
function(fit_file, message_type, messages)
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/FitFileProcessor.py", line 32, in write_generic
function(fit_file, message)
File "/Volumes/MainExternal/Gabe/OneDrive/Coding/GarminDB/FitFileProcessor.py", line 386, in write_device_info_entry
raise ValueError('Failed to translate device_info')
ValueError: Failed to translate device_info
make: *** [update_garmin] Error 1

make setup problem

Hello,
I have problems when it comes to run the make setup on Ubuntu 18 and Virtualbox.
Here is the problem:

yann@yann-VirtualBox:~$ cd GarminDB/
yann@yann-VirtualBox:~/GarminDB$ make setup
make: /usr/local/bin/python3: Command not found
make: /usr/local/bin/python3: Command not found
git submodule init
git submodule update
Cloning into '/home/yann/GarminDB/Tcx'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:tcgoetz/Tcx.git' into submodule path '/home/yann/GarminDB/Tcx' failed
Failed to clone 'Tcx'. Retry scheduled
Cloning into '/home/yann/GarminDB/utilities'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:tcgoetz/utilities.git' into submodule path '/home/yann/GarminDB/utilities' failed
Failed to clone 'utilities'. Retry scheduled
Cloning into '/home/yann/GarminDB/Tcx'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:tcgoetz/Tcx.git' into submodule path '/home/yann/GarminDB/Tcx' failed
Failed to clone 'Tcx' a second time, aborting
Makefile:43: recipe for target 'submodules_update' failed
make: *** [submodules_update] Error 1
yann@yann-VirtualBox:~/GarminDB$ 

ImportError: No module named request

Hi,
I've been trying to extract the Garmin data and here's the error I received.
My command

make GC_DATE='01/10/2018' GC_DAYS=150 GC_USER=saral.karki GC_PASSWORD=password create_dbs

And the error I get

Traceback (most recent call last): File "download_garmin.py", line 8, in <module> import requests ImportError: No module named requests make: *** [download_monitoring] Error 1

What am I doing wrong?

I tried running pip install requests but I get a message saying requirement already satisfied.

no module named Fit

I used your most recent commit to download my fit files into HealthData/FitFiles/Incoming, but get an error when going to next steps in the instructions:

> python import_garmin_fit.py -e --input_dir "HealthData/FitFiles/2017_Monitoring" --sqlite HealthData/DBs
Traceback (most recent call last):
  File "import_garmin_fit.py", line 10, in <module>
    import Fit
ImportError: No module named Fit

When I look at GarminDB/Fit on my computer, its empty. On github, Fit in GarminDB is linked to a standalone repo.

Should I be doing something special, other than git clone, to force GarminDB to pull in the Fit module? I can manually git clone Fit. Is there a better way?

No Sleep, Weight, RHR data

I have a problem with Sleep, RHR, Weight and Calories Data.

The data is only created for the first day specified in die GarminConnectConfig.json

There is also just 1 json file in the respective directories (/healthdata/RHR ...) and only 1 daily summary json.

hrmin, hrmax, steps ... on the other hand, work perfectly.

Parsing error

With the latest 1.3.0 version I'm getting a parsing error when trying to do make rebuild_dbs

Failed to parse /home/christian/HealthData/FitFiles/Activities/3574713631.fit: 'int' object has no attribute '_name'

Create_dbs error - Win10

Hi,
Getting this error trying to make create_dbs:

:~/GarminDB$ make create_dbs
python garmin.py --all --download --import --analyze
Geting activities: '/home/slacker/HealthData/FitFiles/Activities' (1000)
N/A% (0 of 1000) | | Elapsed Time: 0:00:00 ETA: --:--:--Traceback (most recent call last):
File "garmin.py", line 344, in
main(sys.argv[1:])
File "garmin.py", line 334, in main
download_data(overwite, latest, weight, monitoring, sleep, rhr, activities)
File "garmin.py", line 97, in download_data
download.get_activities(activities_dir, activity_count, overwite)
File "/home/slacker/GarminDB/download_garmin.py", line 269, in get_activities
activity_name_str = conversions.printable(activity['activityName'])
File "/home/slacker/GarminDB/Fit/conversions.py", line 114, in printable
return filter(lambda x: x in string.printable, string_in)
TypeError: 'NoneType' object is not iterable
Makefile:125: recipe for target 'garmin' failed
make: *** [garmin] Error 1

The first time I tried it successfully downloading ~1200 json files to /FitFiles/Activities but bailed around 20 minutes / 60% complete with the above message.
This is on a fresh install of the Windows Ubuntu Linux subsystem to which I've apt'd the following additional packages:
make python3 python3-pip ipython3 python-simplejson python-setuptools python-dateutil python-pip python-tk

Thanks

make import_monitoring errors when run

I set up a fresh version of this on my machine to test and this is what I found:

GarminDB Havens$ make GC_DATE=06/07/2017 GC_DAYS=5 GC_USER=USER GC_PASSWORD=PW scrape_monitoring
mkdir -p /Users/Havens/HealthData/FitFiles/Incoming
python scrape_garmin.py -d 06/07/2017 -n 5 -u [email protected] -p NJKLIOJN0l*  -m "/Users/Havens/HealthData/FitFiles/Incoming"
No handlers could be found for logger "scrape_garmin.py"

GarminDB Havens$ make import_monitoring
mkdir -p /Users/Havens/HealthData/DBs
ls: /Users/Havens/HealthData/FitFiles/*Monitoring*/: No such file or directory
for dir in ; do \
		python import_garmin_fit.py -e --input_dir "/Users/Havens/HealthData/FitFiles/2018_Monitoring" --sqlite /Users/Havens/HealthData/DBs; \
	done

The first command now runs without issue, but when running the make import_monitoring function, the program searches the monitoring folder but all of the fit files are actually placed in the "incoming" folder instead of the 2017_Monitoring or 2018_Monitoring folders

Issue with build

Hi. Getting the following error on latest Mac release. BTW, on the notes, it's copy_create_dbs.sh vs. create_dbs.sh listed in readme. (I'll do a PR if you want). Let me know.

The path on my Garmin 530 is "Activities" not Activity.

GarminDb_Darwin_1.0.5 ./copy_create_dbs.sh
Creating new DBs by copying from a USB mounted Garmin device
Copying activities files from /Volumes/GARMIN/GARMIN/ACTIVITY to /Users/juliemorris/HealthData/FitFiles/Activities
Traceback (most recent call last):
File "garmin.py", line 344, in
File "garmin.py", line 331, in main
File "garmin.py", line 72, in copy_data
File "copy_garmin.py", line 37, in copy_activities
File "file_processor.py", line 41, in dir_to_files
OSError: [Errno 2] No such file or directory: '/Volumes/GARMIN/GARMIN/ACTIVITY'
[21571] Failed to execute script garmin

Thanks.

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Hello. Got an error during make create_dbs

Running make on windows 10 Makefile:216: recipe for target 'import_activities' failed

Hi tcgoetz

First of all, thanks a lot for this code! I think it's a shame that Garmin doesn't allow to download csv files...
I managed to get the scripts running on windows 10 by installing the Ubuntu subsystem just like you recommended.

At first I got this error:

I handled the error by rewriting the exception like this:
except:
logger.error("Failed to parse %s:" % (file_name))

But now I run into a new error when the make file tried to build the database. I have data in the garmin, garmin_monitoring and even the activities database seems complete. But there is no data in the summary databases yet. Maybe this error stops the make file before these databases can be built? Do you have an idea how I could fix this error?

Errors when running make setup

I don't get very far with make setup. I kept getting errors on make setup so I had to clone utilities and Tcx separately. After than when I tried to make create_db I get the following

pi@raspberrypi:~/garmin/GarminDB $ make create_db
Traceback (most recent call last):
File "garmin.py", line 18, in
from version import print_version, python_version_check, log_version
File "/home/pi/garmin/GarminDB/version.py", line 7, in
import utilities.version as uv
File "/home/pi/garmin/GarminDB/utilities/init.py", line 11, in
from utilities.db import DbParams, DB, DBObject
File "/home/pi/garmin/GarminDB/utilities/db.py", line 13, in
from sqlalchemy import create_engine, func, desc, extract, and_, literal_column
ModuleNotFoundError: No module named 'sqlalchemy'
Traceback (most recent call last):
File "garmin.py", line 18, in
from version import print_version, python_version_check, log_version
File "/home/pi/garmin/GarminDB/version.py", line 7, in
import utilities.version as uv
File "/home/pi/garmin/GarminDB/utilities/init.py", line 11, in
from utilities.db import DbParams, DB, DBObject
File "/home/pi/garmin/GarminDB/utilities/db.py", line 13, in
from sqlalchemy import create_engine, func, desc, extract, and_, literal_column
ModuleNotFoundError: No module named 'sqlalchemy'
make: *** No rule to make target 'create_db'. Stop.

make: *** [import_weight] Error 1

Sorry tcgoetz, it is me again ...
With the latest built from today I face an error message I did not have before. I just post the very end of it, just let me know if you need more information:
python import_garmin.py --weight_input_dir "/home/wechsler/HealthData/Weight" --sqlite /home/wechsler/HealthData/DBs INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/home/wechsler/HealthData/DBs'} debug: False INFO:import_garmin.py:Debug: 0 English units: False INFO:/home/wechsler/Downloads/FitMaster2/GarminDB/FileProcessor.pyc:Reading directory: /home/wechsler/HealthData/Weight INFO:import_garmin.py:Processing 38 weight files INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/home/wechsler/HealthData/DBs'} debug: False Traceback (most recent call last): File "import_garmin.py", line 344, in <module> main(sys.argv[1:]) File "import_garmin.py", line 325, in main gwd.process_files(db_params_dict) File "import_garmin.py", line 49, in process_files date_str = json_data['startDate'] TypeError: list indices must be integers, not str Makefile:282: recipe for target 'import_weight' failed make: *** [import_weight] Error 1
Thanks in advance for helping this error disappear.

KeyError: 'WELLNESS_RESTING_HEART_RATE'

Looks like Garmin might not store RHR the same way in the past

INFO:root:get_rhr
INFO:root:get_rhr_chunk: 2018-08-22 - 2018-09-21
INFO:root:get_rhr_chunk: 2018-07-23 - 2018-08-22
INFO:root:get_rhr_chunk: 2018-06-23 - 2018-07-23
INFO:root:get_rhr_chunk: 2018-05-24 - 2018-06-23
INFO:root:get_rhr_chunk: 2018-04-24 - 2018-05-24
INFO:root:get_rhr_chunk: 2018-03-25 - 2018-04-24
INFO:root:get_rhr_chunk: 2018-02-23 - 2018-03-25
INFO:root:get_rhr_chunk: 2018-01-24 - 2018-02-23
INFO:root:get_rhr_chunk: 2017-12-25 - 2018-01-24
INFO:root:get_rhr_chunk: 2017-11-25 - 2017-12-25
INFO:root:get_rhr_chunk: 2017-10-26 - 2017-11-25
INFO:root:get_rhr_chunk: 2017-09-26 - 2017-10-26
INFO:root:get_rhr_chunk: 2017-08-27 - 2017-09-26
INFO:root:get_rhr_chunk: 2017-07-28 - 2017-08-27
INFO:root:get_rhr_chunk: 2017-06-28 - 2017-07-28
INFO:root:get_rhr_chunk: 2017-05-29 - 2017-06-28
INFO:root:get_rhr_chunk: 2017-04-29 - 2017-05-29
INFO:root:get_rhr_chunk: 2017-03-30 - 2017-04-29
INFO:root:get_rhr_chunk: 2017-02-28 - 2017-03-30
INFO:root:get_rhr_chunk: 2017-01-29 - 2017-02-28
INFO:root:get_rhr_chunk: 2016-12-30 - 2017-01-29
INFO:root:get_rhr_chunk: 2016-11-30 - 2016-12-30
INFO:root:get_rhr_chunk: 2016-10-31 - 2016-11-30
Traceback (most recent call last):
  File "download_garmin.py", line 428, in <module>
    main(sys.argv[1:])
  File "download_garmin.py", line 424, in main
    download.save_json_file(rhr + '/rhr_' + str(int(time.time())), download.get_rhr())
  File "download_garmin.py", line 266, in get_rhr
    chunk_data = self.get_rhr_chunk(start, end)
  File "download_garmin.py", line 256, in get_rhr_chunk
    rhr_data = response.json()['allMetrics']['metricsMap']['WELLNESS_RESTING_HEART_RATE']
KeyError: 'WELLNESS_RESTING_HEART_RATE'
make: *** [download_rhr] Error 1

input Error during make create_dbs

Hi.!

Devices: Garmin Fexi3HR, Garmin HRM4

(builtins.LookupError) "164" is not among the defined enum values
[SQL: UPDATE devices SET serial_number=?, timestamp=?, manufacturer=?, product=? WHERE devices.serial_number = ?]
[parameters: [{'product': 'UnknownProduct_0', 'manufacturer': 164, 'serial_number': '3923282663000003', 'timestamp': datetime.datetime(2017, 2, 27, 16, 19, 47), 'devices_serial_number': 3923282663000003}]]

witch aditional info do you need for debugging?

How can i findout, with file produces this error?

bugreport.txt

make: *** No rule to make target `download_garmin', needed by `create_dbs'. Stop.

Any idea what I'm doing wrong, ran make setup, created/edited GarminConnectConfig.py and I get this when I run make create_dbs.
Thanks!

Import error

After the latest update i'm getting the following error message:

Traceback (most recent call last):
File "garmin.py", line 23, in
from import_garmin_activities import GarminJsonSummaryData, GarminJsonDetailsData, GarminActivitiesExtraData, GarminTcxData, GarminActivitiesFitData
File "/home/christian/Projekte/GarminDB/import_garmin_activities.py", line 18, in
from garmin_db_tcx import GarminDbTcx
File "/home/christian/Projekte/GarminDB/garmin_db_tcx.py", line 9, in
from Tcx import Tcx
ImportError: cannot import name 'Tcx'

garmin table devices version mismatch. The DB schema has been updated. Please rebuild the garmin DB.

Hello, got an error again when trying to do make create_dbs. Is it something wrong with my databse in garmin connect or what?

Another question: what should be the right file pathway in defines for the the db files? (windows)

HEALTH_DATA_DIR=$/mnt/c/Users/Simon/Desktop/Ny/GarminDB/HealthData/
DB_DIR=$(HEALTH_DATA_DIR)/DBs
BACKUP_DIR=$(HEALTH_DATA_DIR)/Backups

Import Error

Happens when trying to build the database for the first time:

Processing FileType.activity FIT data from /Users//HealthData/FitFiles/Activities
 18% (45 of 242) |####                   | Elapsed Time: 0:01:04 ETA:   0:07:07Traceback (most recent call last):
  File "sqlalchemy/sql/sqltypes.py", line 1479, in _db_value_for_elem
KeyError: 213

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sqlalchemy/engine/base.py", line 1173, in _execute_context
  File "sqlalchemy/engine/default.py", line 799, in _init_compiled
  File "sqlalchemy/sql/sqltypes.py", line 1584, in process
  File "sqlalchemy/sql/sqltypes.py", line 1495, in _db_value_for_elem
LookupError: "213" is not among the defined enum values

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "garmin.py", line 389, in <module>
  File "garmin.py", line 379, in main
  File "garmin.py", line 218, in import_data
  File "fit_data.py", line 54, in process_files
  File "fit_file_processor.py", line 83, in write_file
  File "fit_file_processor.py", line 68, in __write_message_types
  File "fit_file_processor.py", line 57, in __write_message_type
  File "fit_file_processor.py", line 46, in __write_generic
  File "fit_file_processor.py", line 478, in _write_device_info_entry
...
sqlalchemy.exc.StatementError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(builtins.LookupError) "213" is not among the defined enum values
[SQL: UPDATE devices SET serial_number=?, timestamp=?, manufacturer=? WHERE devices.serial_number = ?]
[parameters: [{'manufacturer': 213, 'timestamp': datetime.datetime(2017, 5, 14, 15, 20, 29), 'serial_number': '3927983834000008', 'devices_serial_number': 3927983834000008}]]
[11988] Failed to execute script garmin

Wrong sleepdata when no Data at Garmin Server

Hi.!

when I synchronize sleep data shortly after midnight, 7 hours is entered in GarminDB, although no sleep data is displayed via the web interface. If the sleep data are on the GaminServer synchroniced, the sleep totals still remain at 7 hours after rerun make.

garmin-connect-export submodule not pushed to github

I guess the submodule was not pushed to github on the latest commit.

Update error

Just did an make update and got the following error:

error: Der Server lehnt Anfrage nach nicht angebotenem Objekt 80247380a47021ce37838251e704f75b48774d78 ab.
"fetch" in Submodul-Pfad 'Fit' ausgeführt, aber 80247380a47021ce37838251e704f75b48774d78 nicht enthalten. Direktes Anfordern dieses Commits ist fehlgeschlagen.
Makefile:45: recipe for target 'submodules_update' failed
make: *** [submodules_update] Error 1

When running make i get the following:

Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 218, in import_data
    gfd.process_files(db_params_dict)
  File "/home/christian/Projekte/GarminDB/fit_data.py", line 54, in process_files
    fp.write_file(fit_file)
  File "/home/christian/Projekte/GarminDB/fit_file_processor.py", line 83, in write_file
    self.__write_message_types(fit_file, fit_file.message_types())
  File "/home/christian/Projekte/GarminDB/fit_file_processor.py", line 71, in __write_message_types
    self.__write_message_type(fit_file, message_type)
  File "/home/christian/Projekte/GarminDB/fit_file_processor.py", line 57, in __write_message_type
    function(fit_file, message_type, messages)
  File "/home/christian/Projekte/GarminDB/fit_file_processor.py", line 46, in __write_generic
    function(fit_file, message_dict)
  File "/home/christian/Projekte/GarminDB/fit_file_processor.py", line 299, in _write_session_entry
    current.update_from_dict(activity, ignore_none=True, ignore_zero=True)
TypeError: update_from_dict() got an unexpected keyword argument 'ignore_zero'

global name 'Exceptions' is not defined

Maybe download_garmin.py is missing a commit?

INFO:root:get_rhr
INFO:root:get_rhr_chunk: 2018-08-25 - 2018-09-24
INFO:root:get_rhr_chunk: 2018-07-26 - 2018-08-25
INFO:root:get_rhr_chunk: 2018-06-26 - 2018-07-26
INFO:root:get_rhr_chunk: 2018-05-27 - 2018-06-26
INFO:root:get_rhr_chunk: 2018-04-27 - 2018-05-27
INFO:root:get_rhr_chunk: 2018-03-28 - 2018-04-27
INFO:root:get_rhr_chunk: 2018-02-26 - 2018-03-28
INFO:root:get_rhr_chunk: 2018-01-27 - 2018-02-26
INFO:root:get_rhr_chunk: 2017-12-28 - 2018-01-27
INFO:root:get_rhr_chunk: 2017-11-28 - 2017-12-28
INFO:root:get_rhr_chunk: 2017-10-29 - 2017-11-28
INFO:root:get_rhr_chunk: 2017-09-29 - 2017-10-29
INFO:root:get_rhr_chunk: 2017-08-30 - 2017-09-29
INFO:root:get_rhr_chunk: 2017-07-31 - 2017-08-30
INFO:root:get_rhr_chunk: 2017-07-01 - 2017-07-31
INFO:root:get_rhr_chunk: 2017-06-01 - 2017-07-01
INFO:root:get_rhr_chunk: 2017-05-02 - 2017-06-01
INFO:root:get_rhr_chunk: 2017-04-02 - 2017-05-02
INFO:root:get_rhr_chunk: 2017-03-03 - 2017-04-02
INFO:root:get_rhr_chunk: 2017-02-01 - 2017-03-03
INFO:root:get_rhr_chunk: 2017-01-02 - 2017-02-01
INFO:root:get_rhr_chunk: 2016-12-03 - 2017-01-02
INFO:root:get_rhr_chunk: 2016-11-03 - 2016-12-03
Traceback (most recent call last):
  File "download_garmin.py", line 432, in <module>
    main(sys.argv[1:])
  File "download_garmin.py", line 428, in main
    download.save_json_file(rhr + '/rhr_' + str(int(time.time())), download.get_rhr())
  File "download_garmin.py", line 270, in get_rhr
    chunk_data = self.get_rhr_chunk(start, end)
  File "download_garmin.py", line 260, in get_rhr_chunk
    except Exceptions:
NameError: global name 'Exceptions' is not defined
make: *** [download_rhr] Error 1