andrewcooke / choochoo

Training Diary

Choochoo (ch2)

An open, hackable and free training diary.

Originally I hoped that anyone could run this code by installing a Python package. When it became clear that was too complex and error-prone I moved to a Docker image. However, the Docker approach is also complex (and growing in complexity - currently it requires three Docker images running in parallel) so I am now looking at web hosting (for paid use to cover costs).

As a consequence of all this, the install and startup instructions are limited and, for a time (until plans change or I get web hosting working), this project will be more difficult to use. I'm sorry about this, but it's clearly not sustainable in its current form. Hopefully it will continue to evolve until it reaches a more useful state.

(c) 2018-2020 Andrew Cooke + contributors

Contact [email protected].

choochoo's Issues

New Users! Arrrgh

I suddenly have some people trying the code, which is great, but showing a pile of errors. I am working on fixing them now. If you have problems please tell me and I'll add them to the list. Also, if you have FIT files that can be included in public tests, please send them to me at [email protected] (along with a bug report). I'll try get a new release out soon.

Diary hanging on Jupyter display

If the database has an existing, stale entry for the Jupyter process then display from the diary will stall.

This is because I'm accessing sqlite from two connections (diary and controller) and the diary connection has a lock that blocks the Jupyter controller connection from deleting the stale process.

I need to pass the diary session into the controller. This really messes up encpsulation so I need to think if there's a better way.

Add kit to web interface

Before #34 since we need to select kit on upload?

Error loading activities in 0.29.0

Sorry, just found this error in the new release. I will fix ASAP but am very busy today.

Add uploading data to web interface

Feature request: add activity download from garmin

Hi,

It took some time from me to realize why my recent activities weren't visible in the system, despite downloading "all " data from Garmin: the download fetches only the health monitor data and not activities. Since I sync the watch to Garmin regularly and plug it to the computer only for FW updates, it would be a valuable feature (at least for me) if the "garmin" command would download also the activity data.

Best,
Jouni

Unable to Initially Configure

Running initial setup with a base directory specified, I am unable to complete the initial configure with the configure button on the website.

When I click the configure button, it shows as "Please Wait Writing data." and does not change from this screen. If the page is refreshed, it returns to the initial config page.

Unable to locate ch2 on install

I've installed as instructed with Python3.7 - it shows all dependencies are installed. When I run the start command "ch2" the system does not recognize the command. Where is ch2 located so i can run manually or create a symlink?

Add more info to kit output?

Seems like we could show dates.

ModuleNotFoundError: No module named 'ch2' (in Jupyter)

It seems that Jupyter / ipython is only finding ch2 because I have a path hard-coded in .ipython/profile_default/ipython_config.py:

c.InteractiveShellApp.exec_lines = [
    'import sys',
    'sys.path.append("/home/andrew/project/ch2/choochoo")'
]

This should not be necessary!

OperationalError: (sqlite3.OperationalError) locking protocol

Recieving the OperatinError when trying to import .the downloaded .fit files for the first time. Running Ubuntu through WSL, and using own venv by following the steps in the documentation.

Problem starting 'diary'

Hi,

I installed ch2 from github and apparently managed to import my activity and monitoring .fit files. However, starting 'diary' without further arguments* fails. The problem seems to trigger an error in file diary/model.py, function from_field() in the return statement. The problem is that the field 'type' in kargs is 'text' and the dictionary specified in the return statement doesn't contain that as a key. More specifically, the contents of kargs given to the function:
{'type': 'text'}
and when provided to the return statement:
{'value': None, 'label': 'Notes', 'db': <ch2.sql.tables.statistic.StatisticJournalText object at 0x12dedcfd0>}

I tried adding a handle to the text() function into the dictionary, but has then problems handling the keyword arguments.

It is possible that this is a configuration issue, since I'm using the default one without any insight how to modify it.

*Starting it with the argument '--month' doesn't crash. But when I enter 'All activities', the execution crashes with:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/90934/cmdline'

As I'm running this on macos, such path doesn't exist. Is ch2 only for Linux?

Best,
Jouni

Add 'sanity check' to web interface

Check that the database is well-configured - parameters defined for various activities, etc.

Specify bind address for Jupyter

See #47 - we need Jupyter server to track the web server!

Web interface

Working on this now.

Add kit to diary?

Currently kit usage is not displayed in the diary.

Fit file import error with empty date.

I was trying to import Garmin fit files from a Vivoactive 3. I think one of the items must not have an date? Stack trace below. Any ideas what might be going on or how to debug?

CRITICAL: 'NoneType' object has no attribute 'replace'
DEBUG: Exception: 'NoneType' object has no attribute 'replace'
DEBUG: Type: <class 'AttributeError'>
DEBUG: Traceback:
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/init.py", line 80, in main
command(args, db)
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/commands/monitor.py", line 24, in monitor
run_pipeline(db, PipelineType.STATISTIC)
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/stoats/pipeline.py", line 29, in run_pipeline
pipeline.cls(db, *pipeline.args, id=pipeline.id, **kargs).run()
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/stoats/pipeline.py", line 71, in run
missing = self._missing(s)
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/stoats/calculate/response.py", line 82, in _missing
start = round_hour(self.__full_range(s, True), up=False)
File "/home/ryan/GoogleDrive/[email protected]/Health/choochoo/ch2/lib/date.py", line 167, in round_hour

Base ignored on web upload

Need to work so can't fix this immediately. Means that using --base appears to not work (is actually uploading to default location).

Problems with Date16 class

Hi, there are some issues with the Date16 class.

I'm trying to parse a fit file with 802 heart rate measurements from the same day (2019-07-25). Here is a snippet of some records with the token, parsed time, and heart rate value. As you can see, it suddenly rolls over to the 26th between 15:06 and 15:34. 53 of the 803 values rolls over to the next day like this.

EDIT: Sorry the formatting is not good. Here is the file if you want to verify:
38014592427.zip

DTA 022c7845 2019-07-25 15:01:00+00:00 69 DTA 02687849 2019-07-25 15:02:00+00:00 73 DTA 02a47860 2019-07-25 15:03:00+00:00 96 DTA 02e07870 2019-07-25 15:04:00+00:00 112 DTA 02587971 2019-07-25 15:06:00+00:00 113 DTA 03d07978 2019-07-26 09:20:16+00:00 120 DTA 030c7a82 2019-07-26 09:21:16+00:00 130 DTA 03487a61 2019-07-26 09:22:16+00:00 97 DTA 03847a51 2019-07-26 09:23:16+00:00 81 DTA 03c07a55 2019-07-26 09:24:16+00:00 85 DTA 03fc7a6c 2019-07-26 09:25:16+00:00 108 DTA 03387b84 2019-07-26 09:26:16+00:00 132 DTA 03747b89 2019-07-26 09:27:16+00:00 137 DTA 03b07b8c 2019-07-26 09:28:16+00:00 140 DTA 03ec7b81 2019-07-26 09:29:16+00:00 129 DTA 03287c7c 2019-07-26 09:30:16+00:00 124 DTA 03647c86 2019-07-26 09:31:16+00:00 134 DTA 03a07c76 2019-07-26 09:32:16+00:00 118 DTA 03dc7c6f 2019-07-26 09:33:16+00:00 111 DTA 03187d69 2019-07-26 09:34:16+00:00 105 DTA 03547d6c 2019-07-26 09:35:16+00:00 108 DTA 03907d6e 2019-07-26 09:36:16+00:00 110 DTA 03087e69 2019-07-26 09:38:16+00:00 105 DTA 03447e70 2019-07-26 09:39:16+00:00 112 DTA 03807e73 2019-07-26 09:40:16+00:00 115 DTA 03bc7e74 2019-07-26 09:41:16+00:00 116 DTA 03f87e70 2019-07-26 09:42:16+00:00 112 DTA 03347f6b 2019-07-26 09:43:16+00:00 107 DTA 03707f6e 2019-07-26 09:44:16+00:00 110 DTA 03ac7f67 2019-07-26 09:45:16+00:00 103 DTA 03e87f64 2019-07-25 15:34:00+00:00 100 DTA 03248056 2019-07-25 15:35:00+00:00 86 DTA 03608053 2019-07-25 15:36:00+00:00 83

for offset, token in tokens: #print(offset, token) record = token.parse_token() data = record.as_dict(no_bad_values).data if 'heart_rate' in data: #values, units = data['heart_rate'] print(token, data['timestamp_16'][0][0], data['heart_rate'][0][0])

Add back template for plotting groups?

Or remove outdated docs at https://andrewcooke.github.io/choochoo/nearby

HR Impulse Not Available For Group

Causes problems with std_activity_statistics for example. This is a simple consequence of previous changes. Fix coming in a moment.

Add support for google maps?

Using gmap library.

Document web interface

Diary month / year displays don't handle multiple activities well

Missed this in previous work on multiple activities. Crashing assuming that statistics are unique.

Display 'all time' statistics in web interface

These are calculated by the Summary calculated and some related values displayed in the summaries, but we also have things like total active distance that are not displayed anywhere.

Swimming support

Thanks for the great package. Do you plan to add support for swimming fit files? Currently if I try to load a swimming activity with ch2 activities C:\Users\...\SportsData\4017751481.fit I get...

    INFO: Reading activity data from C:\Users\...\SportsData\4017751481.fit
 WARNING: Unrecognised sport: "swimming" in C:\Users\...\SportsData\4017751481.fit
 WARNING: Could not process C:\Users\...\SportsData\4017751481.fit (ignored)

crashes when DB less complete than expected

eg summary plot when no elevation or step data. probably other cases too.

Duplicate Rest HR Statistics (and resulting errors in health display)

I'm seeing a duplicate value for Rest HR from the MonitorCalculator. This is causing standard_health_statistics() to fail.

Latest release (0.25.3). Working on it now. Don't want to fix the standard_health_statistics() error because the underlying issue is the duplicate. Unfortunately it uses composite sources so the database is having a hard time failing earlier.

No module named 'ch2.migrate' in version 0.31.0

Hi Andrew,

just a quick info for you. If I use the latest release 0.31.0 I get this error:

Traceback (most recent call last):
  File "/Users/soerenkroell/Development/Temp/choochoo3/env/bin/ch2", line 5, in <module>
    from ch2 import main
  File "/Users/soerenkroell/Development/Temp/choochoo3/env/lib/python3.7/site-packages/ch2/__init__.py", line 34, in <module>
    from .commands.upgrade import upgrade
  File "/Users/soerenkroell/Development/Temp/choochoo3/env/lib/python3.7/site-packages/ch2/commands/upgrade.py", line 7, in <module>
    from ..migrate.import_.activity import upgrade_activity
ModuleNotFoundError: No module named 'ch2.migrate'

If I use 0.30.1 everything works fine.

Also it looks like you have changed the commands in 0.31.0. The "Getting Started" docs still says ch2 config default, but I guess now it is ch2 configure load default. Correct me if I´m wrong.

Best wishes,
Soeren

Add user / auth to web interface

Possible to treat choochoo as a library?

Hello, I've been thinking for a long time about the need to create an open alternative to Strava. Other such projects already exist (e.g. https://gitlab.com/pikatrack/pikatrack) with varying degrees of functionality. It appears to me that choochoo already contains much of the analysis and features that would be required to implement a Strava clone with even more added functionality.

I was wondering about your thoughts on how this project could be used as a library rather than through Jupyter notebooks. I think it has a lot of promise. This is a project I could see myself contributing too and utilizing as a library for a Strava clone.

Anyway, just wanted to start a conversation. I realize this is a monumental task, but maybe if the stars align it could be something we could collaborate on.

Multi-sport Statistics

FF statistics (probably) want to respond to all heart rate measurements, across multiple activity groups. In the future we may want to have the ability to enable additional FF stats for single activities. We certainly don't want the current situations where FF statistics fail once there are multiple activity groups defined.

So, at a minimum, FF should work across multiple groups.

Latest docker image fails to start

Hi --

I downloaded the latest (as of 28 July 2020) and tried running it as per the instructions. The image fails to run because docker-start.sh is missing.

silverscreen:~ $ docker image pull andrewcooke/choochoo:latest
latest: Pulling from andrewcooke/choochoo
8559a31e96f4: Pull complete
62e60f3ef11e: Pull complete
93c8ae153782: Pull complete
ea222f757df7: Pull complete
e97d3933bbbe: Pull complete
4f4fb700ef54: Pull complete
c8f5d1cc3ca4: Pull complete
d1315e51ef1c: Pull complete
0769753c23bd: Pull complete
20ef4b553d44: Pull complete
5a3ca383baed: Pull complete
2e7ba621efb9: Pull complete
b0c3f3f5471e: Pull complete
a0b6b1795955: Pull complete
ac14f0078c4a: Pull complete
3bbb3f9babb7: Pull complete
224f91130d2c: Pull complete
301d95506e33: Pull complete
Digest: sha256:faa0ba50d5be44cfb1eeb111f4d707376bd6ca66b23801aee2f2d5304e87923f
Status: Downloaded newer image for andrewcooke/choochoo:latest
docker.io/andrewcooke/choochoo:latest
silverscreen:~ $ docker run -p 8000:8000 -p 8001:8001 andrewcooke/choochoo:latest
/bin/sh: 1: ./docker-start.sh: not found
silverscreen:~ $ docker run -it  -p 8000:8000 -p 8001:8001 andrewcooke/choochoo:latest /bin/bash
root@c03273dabe5e:/# pwd
/
root@c03273dabe5e:/# ls
app  boot  etc	 lib	media  opt   root  sbin  sys  usr
bin  dev   home  lib64	mnt    proc  run   srv	 tmp  var

I manually copied over docker-star.sh and it's up and running.

Update documentation and/or automate dependency installation on initial setup

1. scripts must be run from inside dkr folder

dkr/make-choochoo-image.sh: line 47: ./make-choochoo-dockerfile.sh: No such file or directory fixed by cd dkr/ and then running ./make-choochoo-image.sh

2. webpack is required

sh: webpack: command not found resolved by npm install webpack; npm link webpack; npm install webpack-cli

3. docker needs to be running

failed to dial gRPC: cannot connect to the Docker daemon fixed by (obviously?) starting docker.

4. python env looks to be required?

./make-choochoo-dockerfile.sh: line 46: ../py/env/bin/activate: No such file or directory Not sure what to do about this one! Maybe it is fine?

5. Missing dep?

#12 3.626 ERROR: Could not find a version that satisfies the requirement bonjour-py==0.3 (from -r requirements.txt (line 5)) (from versions: none) #12 3.626 ERROR: No matching distribution found for bonjour-py==0.3 (from -r requirements.txt (line 5)). Not sure what to do about this either.

Monitor Import Currently Broken

I am working on a fix.

I don't think anyone actually uses this (the joys of having no users). But if you do, my apologies - this is was introduced in 0.18.0 and I hope it will be fixed in 0.19.0.

Change 'owner' format?

Currently this is a short class name, which takes up a fair amount of space in the database. Maybe we could reduce it to some function of that? Like initials? Or a hash? Or something indexed?

UFuncTypeError in Jupyter

Hi, I wanted to test your project but I can't get it to work. Loaded activity data and monitor data from my Fenix5 into the database and tried to visualize it with jupyter template by running:
ch2 jupyter show health
But the notebook crashes with

---

UFuncTypeError Traceback (most recent call last)
in
1 s = session('-v2 -f ~/.ch2/database-0-28.sql')
----> 2 health = std_health_statistics(s)

~/projects/virtualenv/lib/python3.8/site-packages/ch2/data/frame.py in std_health_statistics(s, start, finish, *extra)
342 check=False)
343 if present(stats_2, REST_HR):
--> 344 stats = merge_to_hour(stats, stats_2)
345 stats_3 = statistics(s, DAILY_STEPS, ACTIVE_TIME, ACTIVE_DISTANCE, _d(FITNESS_D_ANY), _d(FATIGUE_D_ANY), *extra,
346 start=start, finish=finish)

~/projects/virtualenv/lib/python3.8/site-packages/ch2/data/frame.py in merge_to_hour(stats, extra)
327
328 def merge_to_hour(stats, extra):
--> 329 return stats.merge(extra.reindex(stats.index, method='nearest', tolerance=dt.timedelta(minutes=30)),
330 how='outer', left_index=True, right_index=True)
331

~/projects/virtualenv/lib/python3.8/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
225 @wraps(func)
226 def wrapper(*args, **kwargs) -> Callable[..., Any]:
--> 227 return func(*args, **kwargs)
228
229 kind = inspect.Parameter.POSITIONAL_OR_KEYWORD

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/frame.py in reindex(self, *args, **kwargs)
3854 kwargs.pop("axis", None)
3855 kwargs.pop("labels", None)
-> 3856 return self._ensure_type(super().reindex(**kwargs))
3857
3858 def drop(

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/generic.py in reindex(self, *args, **kwargs)
4541
4542 # perform the reindex on the axes
-> 4543 return self._reindex_axes(
4544 axes, level, limit, tolerance, method, fill_value, copy
4545 ).finalize(self)

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/frame.py in _reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
3741 index = axes["index"]
3742 if index is not None:
-> 3743 frame = frame._reindex_index(
3744 index, method, copy, level, fill_value, limit, tolerance
3745 )

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/frame.py in _reindex_index(self, new_index, method, copy, level, fill_value, limit, tolerance)
3757 tolerance=None,
3758 ):
-> 3759 new_index, indexer = self.index.reindex(
3760 new_index, method=method, level=level, limit=limit, tolerance=tolerance
3761 )

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/indexes/base.py in reindex(self, target, method, level, limit, tolerance)
3142 # check is_overlapping for IntervalIndex compat
3143 if self.is_unique and not getattr(self, "is_overlapping", False):
-> 3144 indexer = self.get_indexer(
3145 target, method=method, limit=limit, tolerance=tolerance
3146 )

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/indexes/base.py in get_indexer(self, target, method, limit, tolerance)
2738 indexer = self._get_fill_indexer(target, method, limit, tolerance)
2739 elif method == "nearest":
-> 2740 indexer = self._get_nearest_indexer(target, limit, tolerance)
2741 else:
2742 if tolerance is not None:

~/projects/virtualenv/lib/python3.8/site-packages/pandas/core/indexes/base.py in _get_nearest_indexer(self, target, limit, tolerance)
2819
2820 target = np.asarray(target)
-> 2821 left_distances = abs(self.values[left_indexer] - target)
2822 right_distances = abs(self.values[right_indexer] - target)
2823

UFuncTypeError: ufunc 'subtract' cannot use operands with types dtype('<M8[ns]') and dtype('O')

Zero Avg Active Speed

seen while developing monthly summary on web.

Drop activity timespan?

One of the larger tables and not used, afaict.

Power Broken

Power calculations are currently broken. They will be fixed as part of integrating the power configuration with kit in the next release (along with better docs on how to configure this).

Add topics (diary entries) tied to activities?

Currently there is no way to associated diary data directly with an activity. The underlying problem is how to re-associate data when the activity is reloaded. This could be solved somehow and would allow things like titles for activities that persist across version updates.

Easy deploy to cloud

Docker image? Something so people can use the web interface remotely.

Add non-zero initial fitness when fitting parameters?

There's no reason fitness should be zero at the start of measuring, so we could try fitting the initial value. This is in the fit parameters template.

More efficient NearbyCalculator

This is running for every group even if I force a range that includes just one group. Why?

Add gear tracking?

With cadence and speed we may be able to figure out wheel size and gears. Or request a cassette and chainring data and calculate gears. Would it be useful? How would is be displayed?

Add 'motivational' statistics

Things like congratulating people on best time for X distance etc etc.

Add support for OpenStreetMap?

See #15.

Month entry for diary missing activity info when using default config

Add shift when comparing activities

We could cross-correlate elevation to find the best fit between two activities so that small differences in approach don't introduce noise in the differences.

Are Segment Sources Composite?

Seems like these should be composite with activities? (added while documenting as a remindme).