Running make on linux complains about pip about garmindb

[tcgoetz]

pip is installed for your user. sudo is using user 'root''s environment and I bet pip is not in root's path. I assume sudo pip --version also fails.

Sicne you have python installed for your user only, you don't want sudo pip, just pip. I can look later at making it configurable to install python packages for all users for the user. In the meantime you can remove 'sudo' form in front of pip.

[slochower]

Oh, good catch. Yes, that's correct. I'm running inside a conda environment because my system Python is 3.6. I removed the sudo lines in the Makefile, and then had to do a git add . and git commit to get the subsequent git pull for a submodule to work.

[slochower]

It appears that the HealthData/Sleep directory isn't created. After I successfully get through make setup, when I try make GC_DATE=<date to start downloading data from> GC_DAYS={number of days of data to download} GC_USER={username} GC_PASSWORD={password} build_dbs, I get:

python analyze_garmin.py -S22:00,06:00  --sqlite /Users/tgoetz/HealthData/DBs
analyze_garmin.py -s <sqlite db path> -m ...
python import_garmin.py -e --sleep_input_dir "/home/XXX/HealthData/Sleep" --sqlite /home/XXX/HealthData/DBs
INFO:import_garmin.py:Debug: False
INFO:/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py:Reading directory: /home/XXX/HealthData/Sleep
Traceback (most recent call last):
  File "import_garmin.py", line 307, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 296, in main
    gsd = GarminSleepData(sleep_input_file, sleep_input_dir, latest, debug)
  File "import_garmin.py", line 98, in __init__
    self.file_names = FileProcessor.FileProcessor.dir_to_files(input_dir, 'sleep_.*\.json', latest)
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py", line 28, in dir_to_files
    for file in os.listdir(input_dir):
OSError: [Errno 2] No such file or directory: '/home/XXX/HealthData/Sleep'
make: *** [import_sleep] Error 1

(py27) Tue Sep 11 08:12:47 warthog:~/data/explorations/GarminDB
$ ls ~/HealthData/
DBs  FitFiles

[tcgoetz]

commit 95a1857 (HEAD -> master, origin/master, origin/HEAD)
Author: Tom Goetz <******>
Date: Thu Sep 13 07:38:55 2018 -0400

Issue #11: make installing python deps as root optional; fix make dep for creating sleep dir

[slochower]

Thanks for the update! This gets me past the initial hurdle and it started to download a bunch of sleep data. However, when trying to import it (I think), I think epoc_ms is set to None.'

INFO:import_garmin.py:Importing /home/XXX/HealthData/Sleep/sleep_2018-07-10.json (2018-07-10) without REM data
INFO:import_garmin.py:DB updated 2018-07-10 with 0 sleep level entries
Traceback (most recent call last):
  File "import_garmin.py", line 307, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 298, in main
    gsd.process_files(db_params_dict)
  File "import_garmin.py", line 126, in process_files
    json_data = json.load(open(file_name), object_hook=json_parser)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "import_garmin.py", line 111, in json_parser
    entry['sleepStartTimestampGMT'] = Fit.Conversions.epoch_ms_to_dt(entry['sleepStartTimestampGMT'])
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/Fit/Conversions.py", line 62, in epoch_ms_to_dt
    return datetime.datetime.fromtimestamp(epoch_ms / 1000.0)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'
make: *** [import_sleep] Error 1

After doing a git pull and stashing my local changes, I just tried running make GC_USER={} GC_PASSWORD={} build_dbs. Is that correct?

[tcgoetz]

If you look at 2018-07-10 on the Garmin site, do you have sleep data? Should there be 0 entries for that date?

INFO:import_garmin.py:DB updated 2018-07-10 with 0 sleep level entries

[slochower]

Nope -- no sleep data for that day (which is correct).

[tcgoetz]

Is it working without issues now?

[slochower]

Getting a little farther. I just did a pull and re-ran make. I think now it didn't make the directory MSHealth.

INFO:analyze_garmin.py:Monitoring Years (0): []
python import_mshealth_csv.py -e --input_dir "/home/XXX/HealthData/MSHealth" --sqlite /home/XXX/HealthData/DBs
Traceback (most recent call last):
  File "import_mshealth_csv.py", line 180, in <module>
    main(sys.argv[1:])
  File "import_mshealth_csv.py", line 170, in main
    msd = MSHealthData(input_file, input_dir, db_params_dict, english_units, debug)
  File "import_mshealth_csv.py", line 65, in __init__
    self.file_names = FileProcessor.FileProcessor.dir_to_files(input_dir, 'Daily_Summary_.*.csv')
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py", line 28, in dir_to_files
    for file in os.listdir(input_dir):
OSError: [Errno 2] No such file or directory: '/home/XXX/HealthData/MSHealth'
make: *** [import_mshealth] Error 1

I certainly appreciate the support, but don't feel any time pressure to work on this for me. I'm just interested in playing with the data when I have it.

[tcgoetz]

commit e0f7314 (HEAD -> master, origin/master, origin/HEAD)

Author: Tom Goetz [email protected]
Date: Sun Sep 16 13:54:24 2018 -0400

Issues: 14: create mshealth and fitbit dirs if needed

NP. Your debugging it for me on a platform I'm not using, so your helping me.

[slochower]

It appears to finish make!

Although I'm not sure what things are in which database. If I open garmin.db in SQLiteStudio I can see "resting_hr" under Tables and the Structure looks good (index 1 is "day" and index 2 is "resting_heart_rate") but there is nothing under the Data tab. garmin.db is 164 kb; I'm not sure if that is too small.

[tcgoetz]

Is there data in the JSON files in HealthData/RHR/ ?

[slochower]

Nope -- that directory is empty. And now that I look back at the output from `make`, I can see that it has 0 or None for a lot of things that should have data. ``` INFO:analyze_garmin.py:File records: 0 INFO:analyze_garmin.py:tcx files: 0 INFO:analyze_garmin.py:activity files: 0 INFO:analyze_garmin.py:monitoring_b files: 0 INFO:analyze_garmin.py:Weight records: 0 INFO:analyze_garmin.py:Max Weight: None INFO:analyze_garmin.py:Min Weight: None INFO:analyze_garmin.py:Avg Weight: None INFO:analyze_garmin.py:Latest Weight: None INFO:analyze_garmin.py:Stress records: 0 INFO:analyze_garmin.py:Max Stress: None INFO:analyze_garmin.py:Min Stress: None INFO:analyze_garmin.py:Avg Stress: None INFO:analyze_garmin.py:Latest Stress: None INFO:analyze_garmin.py:RHR records: 0 INFO:analyze_garmin.py:Max RHR: None INFO:analyze_garmin.py:Min RHR: None INFO:analyze_garmin.py:Avg RHR: None INFO:analyze_garmin.py:Latest RHR: None INFO:analyze_garmin.py:Sleep records: 342 INFO:analyze_garmin.py:Max Sleep: 13:24:00 INFO:analyze_garmin.py:Min Sleep: 00:08:00 INFO:analyze_garmin.py:Avg Sleep: 9.0 INFO:analyze_garmin.py:Latest Sleep: 09:46:00 INFO:analyze_garmin.py:Activity summary records: 0 INFO:analyze_garmin.py:Activities lap records: 0 INFO:analyze_garmin.py:Activity records: 0 INFO:analyze_garmin.py:Activities years: 0 [] INFO:analyze_garmin.py:Running activities: 0 INFO:analyze_garmin.py:Walking activities: 0 INFO:analyze_garmin.py:Cycling activities: 0 INFO:analyze_garmin.py:Mountain_Biking activities: 0 INFO:analyze_garmin.py:Hiking activities: 0 INFO:analyze_garmin.py:Elliptical activities: 0 INFO:analyze_garmin.py:Treadmill_Running activities: 0 INFO:analyze_garmin.py:Paddling activities: 0 INFO:analyze_garmin.py:Resort_Skiing_Snowboarding activities: 0 INFO:analyze_garmin.py:Monitoring records: 0 INFO:analyze_garmin.py:Monitoring Years (0): [] ```

…

On Sun, Sep 16, 2018 at 3:33 PM Tom Goetz ***@***.***> wrote: Is there data in the JSON files in HealthData/RHR/ ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFJOS70unA97ELmwmVvfOkbx_42JRAw9ks5ubtHJgaJpZM4WjfG6> .

[tcgoetz]

What types of data do you have on Connect? Do the dates you have data for fall in the values you used for GC_DATE and GC_DAYS?

You can try just downloading particular types of data to see if you get what you expect:

make GC_USER={username} GC_PASSWORD={password} download_monitoring
make GC_USER={username} GC_PASSWORD={password} download_all_activities
make GC_USER={username} GC_PASSWORD={password} download_sleep
make GC_USER={username} GC_PASSWORD={password} download_rhr

You can also try running

make GC_USER={username} GC_PASSWORD={password}

and it should download data for the previous year.

[slochower]

I tried your last command to get all the data for the previous year. STill gives None for some things that have recorded in Connect. Ohhhhhh. I tried to login to the website -- "Your account has been locked. Please reset your password." I did reset the password now, but I'm wary of running make, in case it gets locked again. Any suggestions?

[tcgoetz]

Actually, no, I've never had my account get locked, even in the early development when I was working out the login.

[tcgoetz]

Thinking about this more, I have a guess. The sleep, RHR, and a few other types of data can get a whole year+ of data from one graph. That's one page access. Monitoring data is retrieved one page access per day. Accessing 365 monitoring pages in a row probably set off a threshold alarm. I've chnaged the default to get one month of data. Hopefully that's small enough. To get your data set you can get it one month at a time with sequential fetches spaced out over time using GC_DATE and GC_DAYS to cover the period you need. I'll look at adding some pauses into the web browsing.

[tcgoetz]

Have you tried this again? Does this work for you?

[slochower]

Sorry, not yet. Will get back to this next week.

…

On Sep 25, 2018, at 7:33 AM, Tom Goetz ***@***.***> wrote: Have you tried this again? Does this work for you? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFJOSxcOeKMUtPFnEVlwfBtNGA1R6eu5ks5uej7VgaJpZM4WjfG6>.

[slochower]

Hm, just did a make pull and then make GC_USER=xxxxxxxxx GC_password=xxxxxx build_dbs and I see

python analyze_garmin.py -S22:00,06:00 --sqlite /Users/tgoetz/HealthData/DBs
Traceback (most recent call last):
  File "analyze_garmin.py", line 10, in <module>
    import GarminDB
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/__init__.py", line 1, in <module>
    from GarminDB import *
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/GarminDB.py", line 8, in <module>
    from Fit import FieldEnums
ImportError: cannot import name FieldEnums
make: *** [garmin_config] Error 1

[tcgoetz]

There is no "make pull". Did you mean "git pull" or "make update"? Either way it looks like your Fit submodule didn't get updated.

[slochower]

Oops, git pull. But you're right -- doing git submodule update pulls in a new Fit submodule.

Now seeing:

Traceback (most recent call last):
  File "import_garmin.py", line 301, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 292, in main
    gsd.process_files(db_params_dict)
  File "import_garmin.py", line 110, in process_files
    garmindb = GarminDB.GarminDB(db_params_dict)
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/GarminDB.py", line 27, in __init__
    self.version.version_check(self, self.db_version)
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/HealthDB/DB.py", line 533, in version_check
    raise RuntimeError("DB %s version mismatch. Please rebuild the DB. (%s vs %s)" % (db.db_name, self.version, version_number))
RuntimeError: DB garmin version mismatch. Please rebuild the DB. (1 vs 3)
make: *** [import_sleep] Error 1

[tcgoetz]

Yep. See the first bullet in the "Notes" section of the front page.

[slochower]

Awesome! I opened garmin.db and I see data in sleep and sleep_events but not the other categories.

[tcgoetz]

Sleep data comes from json files downloaded into ~/HealthData/Sleep/. The other columns come from daily monitoring fit files downloaded into ~/HealthData/FitFiles/_Monitoring. Do you have that directory and are there files in it? On the Garmin Connect website what date ranges do you have "Daily Summary" pages for? Are those same date ranges represented in ~/HealthData/FitFiles/_Monitoring? If not then you need to download them via "make GC_DATE=[date to start downloading] GC_DAYS=[number of days to download] download_monitoring"

[slochower]

Ah. I do have FitFiles and 2018_Monitoring but nothing inside of them. On Garmin connect, under "Daily Summary" I don't have anything for today, but I do have yesterday, and the day before that, etc. Not sure how to find the overall date range.

[tcgoetz]

Actually, there is an easier way to tell how much and what kind of data you have. When you run "make" and you DB is updated, the last thing that does it to run "make garmin_summary" which displays stats about whats in the DB.

Here's the summary Re monitoring data I get:

INFO:analyze_garmin.py:Monitoring records: 224701
INFO:analyze_garmin.py:Monitoring Years (2): [2017, 2018]
INFO:analyze_garmin.py:2017 Months (11): ['Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
INFO:analyze_garmin.py:2017 Days (309 count vs 310 span): [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365]
INFO:analyze_garmin.py:Days gap between 78 (2017-03-19 00:00:00) and 80 (2017-03-21 00:00:00)
INFO:analyze_garmin.py:2018 Months (9): ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep']
INFO:analyze_garmin.py:2018 Days (273 count vs 273 span): [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273]

[tcgoetz]

Why don't you start with downloading the last months data? What happens when you run:
make download_monitoring

[slochower]

Weird. Getting a login error even though I just confirmed via connect.garmin.com it works. I'll look into it further tonight or tomorrow.

One thought: I don't see anything after -p here. Is it possible that it is not reading GC_password environmental variable?

python download_garmin.py -d 09/01/2018 -n 31 -u [email protected] -p  -m "/home/dslochower/HealthData/FitFiles/2018_Monitoring"
ERROR:root:Login failed: <!DOCTYPE html>

[tcgoetz]

It's GC_PASSWORD. The case matters.