Fit file import error with empty date. about choochoo

[andrewcooke]

I'm sorry - I caught pneumonia while abroad and am basically a zombie right now. I will look at this when I recover / get back home. Looks like a typical error when the data aren't as expected and my code is not that great. Sorry. If you can attach example data it will help when I can look. Will be at eat next week as don't fly til Monday. Sorry again Andrew

…

On Tue, Aug 20, 2019 at 12:35:11PM -0700, Ryan wrote: I was trying to import Garmin fit files from a Vivoactive 3. I think one of the items must not have an date? Stack trace below. Any ideas what might be going on or how to debug? CRITICAL: 'NoneType' object has no attribute 'replace' DEBUG: Exception: 'NoneType' object has no attribute 'replace' DEBUG: Type: <class 'AttributeError'> DEBUG: Traceback: File ***@***.***/Health/choochoo/ch2/__init__.py", line 80, in main command(args, db) File ***@***.***/Health/choochoo/ch2/commands/monitor.py", line 24, in monitor run_pipeline(db, PipelineType.STATISTIC) File ***@***.***/Health/choochoo/ch2/stoats/pipeline.py", line 29, in run_pipeline pipeline.cls(db, *pipeline.args, id=pipeline.id, **kargs).run() File ***@***.***/Health/choochoo/ch2/stoats/pipeline.py", line 71, in run missing = self._missing(s) File ***@***.***/Health/choochoo/ch2/stoats/calculate/response.py", line 82, in _missing start = round_hour(self.__full_range(s, True), up=False) File ***@***.***/Health/choochoo/ch2/lib/date.py", line 167, in round_hour -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #6

[ryancornia]

I just sent an example to your e-mail address. Let me know if you have any problem reproducing.

[andrewcooke]

hi,

as far as i can tell you didn't actually attach any data? i think maybe the problem is that you replied to github. instead, can you send the data directly to me at [email protected] ?

thanks. i am feeling better and will look at this, but really need the data.

cheers,
andrew

[andrewcooke]

actually, looking at the error, maybe it's not clear what data caused the problem. if you imported multiple files when you saw the problem then can you either send me just one of the files (any one) or (better, but more work), delete the database and restart importing just one file and send me whichever fails. hope that's clear.

[andrewcooke]

[a bunch of people are watching this issue. if anyone else has example data or a recipe for triggering this, please chip in.]

[andrewcooke]

thanks - have data now. will look at this asap.

[andrewcooke]

ok, so looking at the data it contains no activity. it's just monitoring data. so i guess the error is coming the code not finding any activity to process. i'll improve the logic so that it doesn't fail in that case.

[andrewcooke]

ok, so it works for me. famous last words. a test works using the data provided and this script works, running the commands from the command line.

can you run the commands in that script (modifying them appropriately for your case) and see if it works for you?

[ryancornia]

Worked for me as well!

[andrewcooke]

ok, so do you still have the problem reported originally? that script was a completely new install and database. you may need to wipe what you had and start over (delete the .ch2 directory in your home directory)

[ryancornia]

It fixed the original issue for me.

[ryancornia]

I was still having problems with other files, so completely wiped everything, including ~/.ch2.

Re-tested after cleaning out everything, and the error is still present. This command:

ch2 --dev -v5 -f pwd/ch2.sql monitor /home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit -Kn_cpu=1

Yielded:

(env) ryan@ryan-HP-ENVY-Notebook:~/GoogleDrive/[email protected]/Health/ch2/choochoo$ ch2 --dev -v5 -f `pwd`/ch2.sql monitor /home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit -Kn_cpu=1
    INFO: Version 0.23.4
    INFO: Using database at /home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/ch2.sql
    INFO: Running MonitorReader([], {'force': False, 'n_cpu': 1, 'paths': [...], 'worker': False}
   DEBUG: Running <class 'ch2.stoats.read.monitor.MonitorReader'>([], {'paths': ['/home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit'], 'force': False, 'worker': False, 'n_cpu': 1})
   DEBUG: Have 1 missing ranges
   DEBUG: Batching for n_cpu=1, overhead=1, cost_writes=1, cost_calc=20
   DEBUG: Limit on parallel workers from database contention is 21.0
   DEBUG: Limit on parallel workers from CPU count is 1
   DEBUG: Limit on total workers from work available is 1
   DEBUG: Limit on total workers from overhead is 21.0
   DEBUG: Limit on total workers to boost batch size is 3
    INFO: Threads: 1/1
   DEBUG: Run /home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit
   DEBUG: Reading fit file from /home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit
   DEBUG: Unpickling profile
 WARNING: There was a problem reading the pickled profile.
 WARNING: If you installed via pip then please create an issue at
 WARNING: https://github.com/andrewcooke/choochoo for support.
 WARNING: If you installed via git please see `ch2 help package-fit-profile`
 WARNING: Could not process /home/ryan/GoogleDrive/[email protected]/Health/fit-files/[email protected]_37140810636.fit (ignored)
   DEBUG: Exception: Could not read global-profile.pkl (see log for more details)
   DEBUG: Type: <class 'Exception'>
   DEBUG: Traceback:
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/read/__init__.py", line 42, in _run_one
    self._read(s, path)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/read/__init__.py", line 53, in _read
    key, data = self._read_data(s, path)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/read/monitor.py", line 131, in _read_data
    records = self._read_fit_file(path, merge_duplicates, fix_degrees, unpack_single_bytes)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/read/__init__.py", line 61, in _read_fit_file
    types, messages, records = filtered_records(read_fit(path))
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/fit/format/read.py", line 78, in filtered_records
    types, messages = read_profile(warn=warn, profile_path=profile_path)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/fit/profile/profile.py", line 49, in read_profile
    types, messages = read_internal_profile()
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/fit/profile/profile.py", line 39, in read_internal_profile
    raise Exception('Could not read %s (see log for more details)' % PROFILE_NAME)

    INFO: Ran MonitorReader in 0s
    INFO: Running ElevationCalculator([], {'owner_in': '[unused - data via activity_statistics]'}
   DEBUG: Running <class 'ch2.stoats.calculate.elevation.ElevationCalculator'>([], {'owner_in': '[unused - data via activity_statistics]'})
   DEBUG: Delimit times: None - None
   DEBUG: Have 0 missing ranges
    INFO: No missing data for ElevationCalculator
    INFO: Ran ElevationCalculator in 0s
    INFO: Running ExtendedPowerCalculator([], {'owner_in': '[unuse...PowerEstimate.Bike'}
   DEBUG: Running <class 'ch2.stoats.calculate.power.ExtendedPowerCalculator'>([], {'owner_in': '[unused - data via activity_statistics]', 'power': 'PowerEstimate.Bike'})
   DEBUG: Delimit times: None - None
   DEBUG: Have 0 missing ranges
    INFO: No missing data for ExtendedPowerCalculator
    INFO: Ran ExtendedPowerCalculator in 0s
    INFO: Running ImpulseCalculator([], {'impulse_ref': 'HRImpulse.Bike', 'owner_in': 'SegmentReader'}
   DEBUG: Running <class 'ch2.stoats.calculate.impulse.ImpulseCalculator'>([], {'owner_in': 'SegmentReader', 'impulse_ref': 'HRImpulse.Bike'})
   DEBUG: HRImpulse.Bike: HRImpulse(dest_name='HR Impulse / 10s', gamma=2.0, zero=1, one=6, max_secs=60)
   DEBUG: Delimit times: None - None
   DEBUG: Have 0 missing ranges
    INFO: No missing data for ImpulseCalculator
    INFO: Ran ImpulseCalculator in 0s
    INFO: Running ResponseCalculator([], {'impulse_ref': 'HRI...sponses_ref': [...]}
   DEBUG: Running <class 'ch2.stoats.calculate.response.ResponseCalculator'>([], {'owner_in': 'ImpulseCalculator', 'responses_ref': ['Fitness42d.Bike', 'Fitness84d.Bike', 'Fatigue7d.Bike'], 'impulse_ref': 'HRImpulse.Bike'})
CRITICAL: 'NoneType' object has no attribute 'replace'
   DEBUG: Exception: 'NoneType' object has no attribute 'replace'
   DEBUG: Type: <class 'AttributeError'>
   DEBUG: Traceback:
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/__init__.py", line 80, in main
    command(args, db)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/commands/monitor.py", line 24, in monitor
    run_pipeline(db, PipelineType.STATISTIC)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/pipeline.py", line 29, in run_pipeline
    pipeline.cls(db, *pipeline.args, id=pipeline.id, **kargs).run()
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/pipeline.py", line 71, in run
    missing = self._missing(s)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/calculate/response.py", line 82, in _missing
    start = round_hour(self.__full_range(s, True), up=False)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/lib/date.py", line 167, in round_hour
    down = time.replace(second=0, microsecond=0, minute=0)

    INFO: See `ch2 help` for available commands.
    INFO: Docs at http://andrewcooke.github.io/choochoo
Traceback (most recent call last):
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/bin/ch2", line 10, in <module>
    sys.exit(main())
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/__init__.py", line 80, in main
    command(args, db)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/commands/monitor.py", line 24, in monitor
    run_pipeline(db, PipelineType.STATISTIC)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/pipeline.py", line 29, in run_pipeline
    pipeline.cls(db, *pipeline.args, id=pipeline.id, **kargs).run()
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/pipeline.py", line 71, in run
    missing = self._missing(s)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/stoats/calculate/response.py", line 82, in _missing
    start = round_hour(self.__full_range(s, True), up=False)
  File "/home/ryan/GoogleDrive/[email protected]/Health/ch2/choochoo/env/lib/python3.7/site-packages/ch2/lib/date.py", line 167, in round_hour
    down = time.replace(second=0, microsecond=0, minute=0)
AttributeError: 'NoneType' object has no attribute 'replace'

[andrewcooke]

you need to do the equivalent of the line in the script that says

ch2 --dev -v5 package-fit-profile /home/andrew/project/ch2/choochoo/data/sdk/Profile.xlsx

to rebuild the profile.

this is explained in some detail in the warning messages, but for some reason the exception was swallowed and the program continued running. i'll look into making it fail earlier.

[andrewcooke]

a new release - 0.23.5 - will now fail earlier (hopefully) in this case, giving a clearer error.

[ryancornia]

I had done that via this command from choochoo:

ch2 --dev -v5 package-fit-profile ./data/sdk/Profile.xlsx

Is that Profile not the one to use? I just tried it again to double check, and get the same error.

[andrewcooke]

oh, ok :( please can you copy_paste the output from that command (or email me if it's large - [email protected]). that should read the file to check, in which case you'd expect the other command to also be able to read the file. it's all a bit worrying...

[andrewcooke]

thanks for the logs. it looks like the package-fit-profile isn't actually running because the system wasn't configured. if you try to run it after the config command, but before the monitor command, does everything work? if not, can i see the log to the package command in that case?

i'll change things so that package-fit-profile is allowed. thanks.

[andrewcooke]

version 0.23.6 allows package-fit-profile before config.

[andrewcooke]

i don't understand how my script worked before!

[ryancornia]

That worked, and processed my files, but immediately after I run the command:

dev/ch2 --dev -v5 -f pwd/ch2.sql monitor ../../fit-files/*

I try:

dev/ch2 diary

I get the following:

 Welcome to Choochoo.

 Before using the ch2 command you must configure the system.

 Please see the documentation at http://andrewcooke.github.io/choochoo

 To generate a default configuration use the command

     ch2 config

 NOTE: The default configuration is only an example.  Please see the docs for more details.

even though I have already created the config before.

[andrewcooke]

ok that's easy. the -f pwd/ch2.sql is setting teh directory for where your data are stored. when you run the diary command you're not specifying that, so it's using teh default, which is elsewhere (and so not configured).

use dev/ch2 -f pwd/ch2.sql diary

(with backquotes around pwd which i can't work out how to add).