annnic / cichlid-analysis

cichlid behaviour analysis

cichlid-analysis

cichlid behaviour analysis code

General pipeline Get units: Run measure_units.py (assumes you're measuring the width of the divider which is divider_base_mm = 15) on each camera folder's background. Run measure_fish.py (allows you to select a frame to measure the standard length of the fish, and update the fish sex)

Run run_fish_als.py (creates initial plots to allow you to see how well the fish is tracked, generates als.csv files which have the full track for the fish)

Quality control: Run tracker_checker.py (a GUI for looking at the tracking of videos you think might not have tracked well - generally if thresholded plot has a lot of black)

Ways to improve tracking quality:

1. divide_tracking.py (allows you to retrack movies with different background images created from that video with the inputted time bins)
2. split_tracking.py (allows you to exclude a period of the tracking, this is often necessary if there was a bump to the camera, water level got too low and we can see when it was added etc).
3. retracking.py (can change the ROI, generate new backgrounds etc).

After you have gone through the videos, you now only need the _als.csv and _meta.csv files. We move these to an analysis folder which have these files for each fish of that species.

Now you can run run_combine_fish.als (creates combined data and bins data for that species, generates the _als_30m.csv and _als_fv2.csv files)

You can move the _als_30m.csv and _als_fv2.csv files to a folder which has these files for all species. This will allow you to run: run_binned.py (for plots and analysis across fish species using the _als_30m.csv data) and run_feature_vector.py (for plots and analysis across fish species using the _als_fv2.csv data)

cichlid-analysis's Issues

Plot speed vs y-position (and x-position)

Adding Quantitative measurements to run_combine_als.

Plot speed vs position as scatter plot. Can be either all smoothed data points, or data points from the average day.

bug in run_combine_als

On Neohel

...
loaded file FISH20210526_c2_r0_Neolamprologus-helianthus_su_meta.csv
Traceback (most recent call last):
File "/Users/annikanichols/Documents/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py", line 69, in
fish_tracks['time_of_day_m'] = fish_tracks.ts.apply(lambda row: int(str(row)[11:16][:-3]) * 60 + int(str(row)[11:16][-2:]))
File "/Users/annikanichols/Documents/cichlid-analysis/.venv/lib/python3.7/site-packages/pandas/core/series.py", line 4213, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas/_libs/lib.pyx", line 2403, in pandas._libs.lib.map_infer
File "/Users/annikanichols/Documents/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py", line 69, in
fish_tracks['time_of_day_m'] = fish_tracks.ts.apply(lambda row: int(str(row)[11:16][:-3]) * 60 + int(str(row)[11:16][-2:]))
ValueError: invalid literal for int() with base 10: ''

Process finished with exit code 1

Issue with 'retracking.py' tracking final short videos

Seems that 'retracking.py' retracks the final almost always short video and creates a .csv file. This then causes an error with 'run_fish_als.py' when loading in the tracks .csv's.

To reproduce, make sure the file "20201007-083658_165_roi-0_tracks_20210121_Thresh_35_Area_100_roi-0.csv" is in the ROI folder and run 'run_fish_als.py' for fish/ROI "FISH20200930_c2_r0_Cyprichromis-leptosoma_sm"

Add current ROI to 'tracking_checker'

Can we add a feature to 'tracking_checker' where it displays the ROI on the video? Particularily useful for videos that have been retracked with a smaller ROI but still need to be QC'd

'retracking' doesn't work if split tracking files exist in directory

Running 'retracking' on all videos (track all videos "y") errors out if run on a folder where 'split_tracking' has already been run (removing the new background images, and the 'cleaned' csvs solves this issue).

This behaviour makes sense, 'split_tracking' would have to be re-done on the retracked videos (if necessary). Solution might be for manual deletion of these files, or 'exclude' annotation to be added as a first step of 'retracking' if they exist? I am manually moving them to another location now.

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py
Update background? y/n:
n
Track videos? y/n:
y
Track all videos? y/n:
y
objc[7265]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff9fcb03c8) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x129f47f50). One of the two will be used. Which one is undefined.
Track with another roi? y/n:
y
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c1_Cyprichromis-leptosoma/roi_file.yaml
{'cam_ID': '19463369', 'roi_0': (118, 114, 576, 816), 'roi_1': (688, 97, 544, 848)}
Traceback (most recent call last):
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py", line 115, in
curr_roi[2]]
TypeError: 'NoneType' object is not subscriptable

Process finished with exit code 1'

'run_fish_als' errors out after 'split_tracking' after 'retracking' fish

I was confusing myself, so I'm making this a new issue:

'split_tracking' now works for retracked videos, but 'run_fish_als' errors out when running on fish that were retracked, then split_tracked:

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_fish_als.py
Analyse multiple folders (ROIs) (y) or only one ROI (n)?: 
n
objc[10546]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff9fcb03c8) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x124067f50). One of the two will be used. Which one is undefined.
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c2_Cyprichromis-leptosoma/config.yaml
{'cam_ID': '19463146', 'filechunk': 3600, 'fps': 10, 'mm_per_pixel': 0.49956460050324186, 'tot_hours': 166}
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c2_Cyprichromis-leptosoma/FISH20200930_c2_r0_Cyprichromis-leptosoma_sm/meta_data.yaml
{'fish_length_mm': 50.0, 'sex': 'm', 'species': 'Cyprichromis leptosoma'}
two  options for replacement for split movie... exiting
Traceback (most recent call last):
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_fish_als.py", line 358, in <module>
    full_analysis(rootdir)
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_fish_als.py", line 38, in full_analysis
    track_full, speed_full = extract_tracks_from_fld(rootdir, file_ending)
TypeError: cannot unpack non-iterable bool object

Process finished with exit code 1

Lilkely due to having both "20201005-164315_125_roi-0_tracks_20210121_Thresh_35_Area_100_roi-0.csv" and "20201005-164315_125_roi-0_tracks_20210128_Thresh_35_Area_100_Range00000-03790_.csv" in the folder for the split tracked video.

Originally posted by @maxshafer in #7 (comment)

tracking_checker.py doesn't use recalculated backgrounds

Change tracking_checker.py to first check if there is a new background for the video in question, and to load that one instead.

Also need to allow it to find multiple backgrounds (if divide_tracking.py was used with anything other then 60). This would be in the GUI (should read out frames from background names).

'retracking' doesn't work on Cyplep c4 r0

'retracking' is working for other folders (including c4 r1) but for some reason errors out for this folder?

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py
Update background? y/n: 
n
Track videos? y/n: 
y
Track all videos? y/n: 
y
objc[16560]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff9fcb03c8) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x12da74f50). One of the two will be used. Which one is undefined.
Didn't find remade background, will use original background in camera folder
Track with another roi? y/n: 
y
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c4_Cyprichromis-leptosoma/roi_file.yaml
{'cam_ID': '19463356', 'roi_0': (22, 81, 640, 880), 'roi_1': (642, 67, 608, 864)}
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c4_Cyprichromis-leptosoma/FISH20200930_c4_r0_Cyprichromis-leptosoma_sm/roi_file.yaml
{'cam_ID': 'na', 'roi_0': (6, 81, 615, 794)}
tracking /Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c4_Cyprichromis-leptosoma/FISH20200930_c4_r0_Cyprichromis-leptosoma_sm/20200930-120320_0_roi-0.mp4
Traceback (most recent call last):
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py", line 138, in <module>
    area_size=100)
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/offline_tracker.py", line 54, in tracker
    frameDelta_full = cv2.absdiff(background_full, gray)
cv2.error: OpenCV(4.4.0) /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/pip-req-build-4jcifzim/opencv/modules/core/src/arithm.cpp:669: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'arithm_op'


Process finished with exit code 1

Change sex in name?

'split_tracking.py' broken for retracked videos

Running 'split_tracking.py' on retracks causes an error:

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/quality_control/split_tracking.py
objc[2617]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff9fcb03c8) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x121600f50). One of the two will be used. Which one is undefined.
Traceback (most recent call last):
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/quality_control/split_tracking.py", line 175, in
displacement_internal, track_single = load_track(track_path)
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/io/tracks.py", line 11, in load_track
track_internal = np.genfromtxt(csv_file_path, delimiter=',')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1749, in genfromtxt
fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 195, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 535, in open
raise IOError("%s not found." % path)
OSError: /Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c2_Cyprichromis-leptosoma/FISH20200930_c2_r0_Cyprichromis-leptosoma_sm/20201005-114400_120_roi-0.csv not found.

Process finished with exit code 1

Reproducible on "/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200930_LT_Cyplep/20200930_c2_Cyprichromis-leptosoma/FISH20200930_c2_r0_Cyprichromis-leptosoma_sm/20201005-114400_120_roi-0.mp4"

Plot out daily average for movement

Retracking with new background

/Users/annikanichols/Documents/cichlid-analysis/.venv/bin/python3 /Users/annikanichols/Documents/cichlid-analysis/cichlidanalysis/tracking/retracking.py
Update background? y/n:
n
Track videos? y/n:
y
Track all videos? y/n:
y
Track with another roi? y/n:
y
/Volumes/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200727_LT_Neopul-daffodil/20200727_cam_1/roi_file.yaml
{'cam_ID': '19463146', 'roi_0': (605, 185, 480, 672)}
/Volumes/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200727_LT_Neopul-daffodil/20200727_cam_1/FISH20200727_c1_r0_Neolamprologus-pulcher-daffodil_sf/roi_file.yaml
{'cam_ID': 'na', 'roi_0': (12, 91, 438, 566)}
tracking with background ['20200727-110529_0_roi-0_per90_background.png']
tracking /Volumes/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200727_LT_Neopul-daffodil/20200727_cam_1/FISH20200727_c1_r0_Neolamprologus-pulcher-daffodil_sf/20200727-110529_0_roi-0.mp4
Traceback (most recent call last):
File "/Users/annikanichols/Documents/cichlid-analysis/cichlidanalysis/tracking/retracking.py", line 158, in
area_size=100)
File "/Users/annikanichols/Documents/cichlid-analysis/cichlidanalysis/tracking/offline_tracker.py", line 54, in tracker
frameDelta_full = cv2.absdiff(background_full, gray)
cv2.error: OpenCV(4.4.0) /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/pip-req-build-ucld1hvm/opencv/modules/core/src/arithm.cpp:669: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'arithm_op'

Process finished with exit code 1

In feature vectors sometimes y position is empty - Cygnus data

Split_tracking.py errors if S + E are too close to end of video

I've seen this behaviour a few times, always fixed by making 'e' early in the video (to leave more time after the split).

Don't have error message, but should be easy to reproduce. Putting it here mostly for posterity.

'run_combine_als' error 'Dataframe' object has no attribute 'y_nt'

Here's the error that 'run_combine_als' spits out, I couldn't make sense of the file's it references, but it seems to be trying to access 'y_nt' after it has already been removed?

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py
objc[12034]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff9fcb03c8) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x132de1f50). One of the two will be used. Which one is undefined.
loaded file FISH20210108_c1_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210108_c2_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210108_c3_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210108_c4_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210108_c5_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210108_c6_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c1_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c2_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c3_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c4_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c5_r0_Telmatochromis-vittatus_su_als.csv
loaded file FISH20210115_c6_r0_Telmatochromis-vittatus_su_als.csv
All als.csv files loaded
time to load tracks 341.2032651901245
loaded file FISH20210108_c1_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210108_c2_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210108_c3_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210108_c4_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210108_c5_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210108_c6_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c1_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c2_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c3_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c4_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c5_r0_Telmatochromis-vittatus_su_meta.csv
loaded file FISH20210115_c6_r0_Telmatochromis-vittatus_su_meta.csv
time to load tracks 525.7627079486847
<class 'pandas.core.frame.DataFrame'>
Int64Index: 66363874 entries, 0 to 5527185
Data columns (total 9 columns):
 #   Column         Dtype         
---  ------         -----         
 0   tv_ns          float64       
 1   speed_mm       float64       
 2   x_nt           float64       
 3   y_nt           float64       
 4   FishID         object        
 5   ts             datetime64[ns]
 6   time_of_day_m  int64         
 7   daynight       object        
 8   movement       float64       
dtypes: datetime64[ns](1), float64(5), int64(1), object(2)
memory usage: 4.9+ GB
removed y_nt
removed x_nt
removed tv_ns
Traceback (most recent call last):
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py", line 346, in <module>
    spd_vs_y(meta, fish_tracks, fish_tracks_30m, fish_IDs, rootdir)
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/plotting/position_plots.py", line 49, in spd_vs_y
    vertical_pos_30m = fish_tracks_30m[fish_tracks_30m.FishID == fish].y_nt
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5139, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'y_nt'

Process finished with exit code 1

Issue with fish that was tracked with previous version of the retracking software

This applies to:

/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20200907_LT_Neogra/20200907_c6_Neolamprologus-gracilis/FISH20200907_c6_r0_Neolamprologus-gracilis_sm

And the following error when running 'helpers.py':

`/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py
Correct new tracks? y/n:
y
What is the date tag for the new tracks? YYYYMMDD:
20201030
2022-01-24 14:18:54.511 Python[15945:1150266] WARNING: <NSOpenPanel: 0x7f7f6997d0b0> running implicitly; please run panels using NSSavePanel rather than NSApplication.
updating timestamps of 20200907-172825_0_roi-0_tracks_20201030_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20200907-172825_0_roi-0.csv
updating timestamps of 20200907-182820_1_roi-0_tracks_20201030_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20200907-182820_1_roi-0.csv
Traceback (most recent call last):
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 230, in
correct_tags(date_tag, rootdir)
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 206, in correct_tags
update_csvs(os.path.join(vid_directory, orig_file), os.path.join(vid_directory, new_f))
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 157, in update_csvs
copy_timestamps(orig_csv_path, new_csv_path)
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 130, in copy_timestamps
_, track_single_orig = load_track(orig_csv_path_i)
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/io/tracks.py", line 12, in load_track
track_internal = np.genfromtxt(csv_file_path, delimiter=',')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1749, in genfromtxt
fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 194, in open
ds = DataSource(destpath)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 251, in init
self._destpath = os.path.abspath(destpath)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/posixpath.py", line 383, in abspath
cwd = os.getcwd()
OSError: [Errno 45] Operation not supported

Process finished with exit code 1`

Deal with different timings of the fish data

Earlier assays were done from 7.30am - 7.30pm daylight, but now 7.00am - 7.00pm daylight. As this should be consistent from now on, we'll convert old data 7.00am - 7.00pm daylight by adding 30min to the timestamps

Bug with retracking.py

Seems like there might be a bug in the retracking.py code. I get the same error message when trying this on various videos (new and old). Its not isolated to the N. marunguensis videos.

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py
Update background? y/n:
n
Track videos? y/n:
y
Track all videos? y/n:
y
Didn't find remade background, will use original background in camera folder
Track with another roi? y/n:
y
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20210505_LT_Neolon/20210505_c3_Neolamprologus-longicaudatus/roi_file.yaml
{'cam_ID': '19503389', 'roi_0': (130, 195, 528, 752), 'roi_1': (656, 193, 544, 752)}
/Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20210505_LT_Neolon/20210505_c3_Neolamprologus-longicaudatus/FISH20210505_c3_r0_Neolamprologus-longicaudatus_su/roi_file.yaml
couldn't find roi_file.yaml in folder: /Volumes/BZ/Scientific Data/RG-AS04-Data01/Cichlid_sleep_videos/FISH20210505_LT_Neolon/20210505_c3_Neolamprologus-longicaudatus/FISH20210505_c3_r0_Neolamprologus-longicaudatus_su
called Tcl_FindHashEntry on deleted table

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Change documentation of sex to 'measure_fish'

Since we always sex the fish after the experiment, it no longer makes sense to record the sex in the meta data at the start of the experiment.

Solution: during 'measure_fish', have a prompt which asks for the sex - which can then be retrieved from the cichlid sexing google sheet.

Another bug with 'run_combine_als'

Another bug - running combine als on Altshe data that has been retracked then divide_tracking. Also tried re-making als files (running fish_als again) to no avail.

/usr/local/bin/python3.7 /Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py
2021-06-10 12:58:45.398 Python[27248:1217134] WARNING: <NSOpenPanel: 0x7f8552f71cf0> running implicitly; please run panels using NSSavePanel rather than NSApplication.
Traceback (most recent call last):
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/analysis/run_combine_als.py", line 47, in
fish_tracks = load_als_files(rootdir)
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/io/tracks.py", line 180, in load_als_files
data = pd.read_csv(os.path.join(folder, file), sep=',')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/io/parsers.py", line 458, in _read
data = parser.read(nrows)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/io/parsers.py", line 1196, in read
ret = self._engine.read(nrows)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/io/parsers.py", line 2155, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 862, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 4683840, saw 8

Process finished with exit code 1

Calculate total rest/wake activity

Add to run_combine_als.

Calculate total rest/wake (avg of 24hrs), plus rest/wake for day, night, and twilight.

Spilt_tracking will always use old roi

Need to update it so if the folder is retracked then it will use the roi in the folder

Save out 30min data for speed, movement, y position etc

In order to be able to combine data across fish need to save out these smaller files

Issue with split_tracking.py not being able to save out .csv file

Our best guess is that it has to do with not being able to access the mounted drive (where the .csv file should be saved), but the same operation works ~50% of the time in our hands.

Here is the error message, not always reproducible:

reached end of video
Saving data output
Traceback (most recent call last):
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/quality_control/split_tracking.py", line 231, in
split_range=split_range[idx])
File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/offline_tracker.py", line 139, in tracker
np.savetxt(filename, datanp, delimiter=",")
File "<array_function internals>", line 6, in savetxt
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1358, in savetxt
fh = np.lib._datasource.open(fname, 'wt', encoding=encoding)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 194, in open
ds = DataSource(destpath)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 251, in init
self._destpath = os.path.abspath(destpath)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/posixpath.py", line 383, in abspath
cwd = os.getcwd()
OSError: [Errno 45] Operation not supported

Process finished with exit code 1

in retracking, if there's a incomplete video it crashes the script

'retracking' bug when split tracks in folder

Small bug when updating timestamps during 'retracking'

updating timestamps of 20201005-144324_123_roi-1_tracks_20210130_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20201005-144324_123_roi-1.csv
updating timestamps of 20201005-154314_124_roi-1_tracks_20210130_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20201005-154314_124_roi-1.csv
updating timestamps of 20201005-164308_125_roi-1_tracks_20210130_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20201005-164308_125_roi-1.csv
updating timestamps of 20201005-164308_125_roi-1_tracks_20210130_Thresh_35_Area_100_roi-0.csv and adding exclude tag to 20201005-164308_125_roi-1_tracks_20210121_Thresh_35_Area_100_Range00000-03868_.csv
Traceback (most recent call last):
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/retracking.py", line 187, in <module>
    update_csvs(os.path.join(vid_dir, retracked_file), os.path.join(vid_dir, new_f))
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 145, in update_csvs
    copy_timestamps(orig_csv_path, new_csv_path)
  File "/Users/maxwellshafer/cichlid-analysis/cichlidanalysis/tracking/helpers.py", line 124, in copy_timestamps
    track_single_retracked[:, 0] = track_single_orig[:, 0]
ValueError: could not broadcast input array from shape (31600) into shape (36000)

Process finished with exit code 1

I moved the files out of the folder, and just ran part of the retracking code to finish this one off, but may need a bit of tweaking (it only error after retracking everything)

Pick background to select roi for retracking

Since the water level changes, sometime when you retrack you want to be able to choose a level that is. always below the water. Allowing the user to select the image to set the roi on can help this.

Problem caculating total rest

For 'Aaltolamprologus-calvus'
total rest =
0 31.32
1 30.31
2 18.70
3 12.97
4 24.17
5 33.69
6 21.18
7 31.60
8 37.27
9 29.25
10 31.49
11 32.70
Name: total_rest, dtype: float64

shouldn't bee able to be more than 24

Retracking takes backgrounds in order

Test to see if video matches background

Add fish size to feature vector

Add diel patterns

Improve how nocturnal, crepuscular and dirunal are called