strawlab / flydra

live, low-latency markerless multi-camera 3D animal tracking system

flydra - multi-camera tracking system

⚠️⚠️Please consider our updated software, called Braid.⚠️⚠️ Braid is a modern and maintained derivative of flydra. Braid performs the same tasks as flydra, only better. Flydra is not very actively maintained and substantial updates from the Straw Lab are not anticipated.

Flydra is a markerless, multi-camera tracking system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. Fundamentally, the multi-target tracking algorithm is based on an extended Kalman filter and the nearest neighbour standard filter data association algorithm.

Discussion

For questions or discussion, please use the "multicams" Google Group.

Installation

For installation, we recommend using our Ansible playbooks. In particular, the ros-kinetic-flydra role or the ros-kinetic-freemovr install on Ubuntu 16.04 with ROS Kinetic, either flydra alone or within a full FreemoVR system.

History

This software was originally develped by Andrew Straw in the Dickinson Lab at Caltech from 2004-2010. Ongoing development continued, coordinated by the Straw Lab, from 2010. The software was open sourced in 2017.

Funding

We gratefully acknowledge support from from the Packard Foundation, AFOSR (FA9550-06-1-0079), ARO (DAAD 19-03-D-0004), NIH (R01 DA022777), NSF (0923802), and Caltech to Michael H. Dickinson, and AFOSR (FA9550-10-1-0086), ERC (Starting Grant 281884 FlyVisualCircuits), WWTF (CS11-029), Boehringer Ingelheim, Research Institute of Molecular Pathology (IMP), and the University of Freiburg to Andrew D. Straw.

Directory organization

• docs - documentation
• flydra_analysis - calibration and analysis routines
• flydra_core - realtime 3D tracking
• packaging - making Debian/Ubuntu packages

Publications

Flydra is described in the following papers:

Straw AD✎, Branson K, Neumann TR, Dickinson MH. Multicamera Realtime 3D Tracking of Multiple Flying Animals. Journal of The Royal Society Interface 8(11), 395-409 (2011) doi:10.1098/rsif.2010.0230.

Stowers JR*, Hofbauer M*, Bastien R, Griessner J⁑, Higgins P⁑, Farooqui S⁑, Fischer RM, Nowikovsky K, Haubensak W, Couzin ID, Tessmar-Raible K✎, Straw AD✎. Virtual Reality for Freely Moving Animals. Nature Methods (2017) doi:10.1038/nmeth.4399.

Please cite these if you use Flydra.

License

With the exception of third party software, the software, documentation and other resouces are licensed under either of

• Apache License, Version 2.0, (./LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (./LICENSE-MIT or http://opensource.org/licenses/MIT) at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Code of conduct

Anyone who interacts with flydra in any space including but not limited to this GitHub repository is expected to follow our code of conduct.

flydra's Issues

flydra-sphinx-docs math markup broken

See for example

https://code.astraw.com/flydra-doc/orientation_ekf_fitter.html#process-model

bug in data2smoothed

Hi Andrew,

This isn't a real show-stopper, but I figured you'd like to know:

If one of the 4 cameras is missing when running Data2smoothed with
--time-data, this is the error:

cheers,
paolo

flydra_analysis_data2smoothed
3-12/90_0/DATA20100312_140356.3D_orientation.h5
--min-ori-quality-required=0 --time-data=3-12/DATA20100312_140356.h5
--dest-file=3-12/90_0/DATA20100312_140356.mat

traceback (most recent call last):
File "/usr/bin/flydra_analysis_data2smoothed", line 9, in
load_entry_point('flydra==0.4.69', 'console_scripts','flydra_analysis_data2smoothed')()
File "/usr/lib/pymodules/python2.6/flydra/a2/data2smoothed.py", line 250, in main
*_kwargs)
File "/usr/lib/pymodules/python2.6/flydra/a2/data2smoothed.py", line 125, in convert
mainbrain_timestamp = remote_timestamp_gain[remote_hostname] + offset[remote_hostname] # find mainbrain timestamp
KeyError: 'cam3'
Closing remaining open files:
3-12/90_0/DATA20100312_140356.3D_orientation.h5... done
3-12/DATA20100312_140356.h5... done

flightlab@desktop1:~/Desktop/repeatability2$ flydra_analysis_data2smoothed
3-12/90_0/DATA20100312_140356.3D_orientation.h5
--min-ori-quality-required=0 --time-data=3-12/DATA20100312_140356.h5
--dest-file=3-12/90_0/DATA20100312_140356.mat

roslib.names does not exist

This might be a issue because I am using ros hydro, changing:

nodename = nodename[1:].split(roslib.names.SEP)[-1]

to:

nodename = nodename[1:].split(rospy.names.SEP)[-1]

Seems to fix the problem.

timestamps off by more than 5 msec --synchronization error

Hi flydra users,

I am facing a problem after running the "roslaunch ros_flydra main_brain.launch" and then "rosrun ros_flydra camnode". In the main_brain terminal it shows a warning "[WARN] [1594270086.445616]: timestamps off by more than 5 msec -- synchronization error". Is that just a warning or creating any issues?

I need this help. Thanks all.

Make the data more accessible to other users

This would be beneficial for all flydra/humdra users and seems like it would help get Steve up and running with the various data formats the flydra currently produces. Furthermore, it would support the goal of allowing us (the biologists) to focus on gathering the data and doing the experiments, and quite possibly encourage computer vision and machine learning people, for example, to help us analyze our data. Currently flydra emits realtime output in .h5 files with both the 2D data and rough 3D data and .ufmf movie files, which are then analyzed and cleaned up through a series of further scripts to produce more final data including things like Kalman smoothed 3D position and orientation trajectories. Documenting the intermediate and final file formats (and potentially what the various processing steps do) would be a really useful goal.

AVT camera not found

I am trying to run a camnode launch file. The launch looks like so:

<launch>
  <node name="Cam1" pkg="camiface_ros" type="camnode">
    <param name="device_guid" type="str" value="AVT-02-2020C-06767" />
  </node>
  <node name="Cam2" pkg="camiface_ros" type="camnode">
    <param name="device_guid" type="str" value="AVT-02-2020C-07056" />
  </node>
</launch>

I also tried using underscores.

And this is what the log looks like:

^[[0m[ INFO] [1531769131.179236474]: host timestamps OFF^[[0m
^[[33m[ WARN] [1531769132.255169176]: Selected camera AVT-02-2020C-07056 (-1) not found^[[0m

When running flydra in self-discovery mode, the cameras are found and detected as:

beelab@beelab:~$ rosrun ros_flydra camnode
[INFO] [1531768934.973439]: ROS name: /flydra_camnode
[INFO] [1531768935.003848]: choosing camera guid: Allied Vision Technologies-02-2020C-07056 (cam_id: 0)
[INFO] [1531768935.004234]: choosing camera guid: Allied Vision Technologies-02-2020C-07058 (cam_id: 1)
[INFO] [1531768935.004501]: choosing camera guid: Prosilica-02-2020C-06767 (cam_id: 2)

This Flydra instance is a fresh installation on July 16th.

Thank you!

make ufmf view into a datanode

ufmf documents (the originals) are not viewed as a datanode and thus there is no auto-linking to the child nodes. For example, see this document in the Altshuler lab database.

calibration description in docs uses old MATLAB stuff

The docs at https://code.astraw.com/flydra-doc/calibration.html are out of date: the calibration code at https://github.com/strawlab/MultiCamSelfCal can now run just fine using Octave. Also, the configuration is much simplified and can reside in simple files alongside the calibration data.

flydra_kalmanize: undefined symbol: IsZero

When running flydra_analysis versions 0.7.6 through 0.7.9 (from source) or 0.7.10 (from pypi) on python 2.7 all give me the following error when running flydra_kalmanize:

(test_flydra_py27) buzz@tornado:~/src$ flydra_kalmanize -r /home/buzz/DATA/20201123_braid_calibration/data/20201130_132722.braidz.h5.recal/ /home/buzz/DATA/20201123_braid_calibration/data/20201130_132722.braidz.h5
Traceback (most recent call last):
File "/home/buzz/test_flydra_py27/bin/flydra_kalmanize", line 5, in
from flydra_analysis.kalmanize import main
File "/home/buzz/test_flydra_py27/local/lib/python2.7/site-packages/flydra_analysis/kalmanize.py", line 5, in
import flydra_core.reconstruct
File "/home/buzz/test_flydra_py27/local/lib/python2.7/site-packages/flydra_core/reconstruct.py", line 18, in
import flydra_core._pmat_jacobian_water as _pmat_jacobian_water # in pyrex/C for speed
File "flydra_core/_pmat_jacobian_water.pyx", line 1, in init flydra_core._pmat_jacobian_water
File "flydra_core/_refraction.pyx", line 1, in init flydra_core._refraction
ImportError: /home/buzz/test_flydra_py27/local/lib/python2.7/site-packages/flydra_core/_Roots3And4.so: undefined symbol: IsZero

When attempting to run on python 3.6 (using either the pypi version, or the dev branch of flydra), all give me this python 3 related error:

(test_flydra_py36) buzz@tornado:~/src$ flydra_kalmanize -r /home/buzz/DATA/20201123_braid_calibration/data/20201130_132722.braidz.h5.recal/ /home/buzz/DATA/20201123_braid_calibration/data/20201130_132722.braidz.h5
/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/kalmanize.py:432: UserWarning: dynamic model not specified. using "EKF mamarama, units: mm"
"dynamic model not specified. " 'using "%s"' % dynamic_model_name
/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_core/reconstruct.py:1205: UserWarning: No minimum eccentricity specified, using default
warnings.warn("No minimum eccentricity specified, using default")
/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/kalmanize.py:473: UserWarning: unable to determine reconstructor source filename for 'normal files'
"filename for %r" % reconstructor.cal_source_type
Traceback (most recent call last):
File "/home/buzz/test_flydra_py36/bin/flydra_kalmanize", line 8, in
sys.exit(main())
File "/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/kalmanize.py", line 1161, in main
kalmanize(*args, **kwargs)
File "/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/kalmanize.py", line 478, in kalmanize
frames_per_second = get_fps(results)
File "/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/analysis/result_utils.py", line 515, in get_fps
parsed = read_textlog_header(results, fail_on_error=fail_on_error)
File "/home/buzz/test_flydra_py36/lib/python3.6/site-packages/flydra_analysis/analysis/result_utils.py", line 438, in read_textlog_header
infostr = textlog1["message"].tostring().strip("\x00")
TypeError: a bytes-like object is required, not 'str'

Is there a way to get the python 2.7 version working?

flydra 0.7.3 camera synchronization fails

Starting main_brain.launch works, camnode runs and detects four cameras, but camera synchronization fails in the following way:

roscd ros_flydra; python scripts/simple-console.py --sync
sync cameras
Traceback (most recent call last):
  File "scripts/simple-console.py", line 86, in <module>
    sync_cameras()
  File "scripts/simple-console.py", line 16, in sync_cameras
    xx.call()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 495, in call
    service_uri = self._get_service_uri(request)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 463, in _get_service_uri
    raise ServiceException("service [%s] unavailable"%self.resolved_name)
rospy.service.ServiceException: service [/flydra_mainbrain/do_synchronization] unavailable

refraction bug

I just hit this:

Exception in thread CoordinateProcessor:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/dist-packages/flydra/main_brain/coordinate_receiver.py", line 418, in run
self.process_data( incoming_2d_data )
File "/usr/lib/python2.7/dist-packages/flydra/main_brain/coordinate_receiver.py", line 555, in process_data
deferred_2d_data,
File "/usr/lib/python2.7/dist-packages/flydra/main_brain/coordinate_receiver.py", line 706, in _process_parsed_data
debug2=self.debug_level.isSet())
File "/usr/lib/python2.7/dist-packages/flydra/kalman/flydra_tracker.py", line 117, in calculate_a_posteriori_estimates
debug1=debug2) for tro in self.live_tracked_objects ]
File "_flydra_tracked_object.pyx", line 365, in _flydra_tracked_object.TrackedObject.calculate_a_posteriori_estimate (src/_flydra_tracked_object.c:5965)
File "_flydra_tracked_object.pyx", line 472, in _flydra_tracked_object.TrackedObject._filter_data (src/_flydra_tracked_object.c:10111)
File "_flydra_tracked_object.pyx", line 515, in _flydra_tracked_object.TrackedObject._filter_data (src/_flydra_tracked_object.c:8011)
File "/usr/lib/python2.7/dist-packages/flydra/reconstruct.py", line 1646, in find2d
distorted=distorted )
File "/usr/lib/python2.7/dist-packages/flydra/water.py", line 48, in view_points_in_water
height, r[i], depth[i]))
File "_refraction.pyx", line 41, in _refraction.find_fastest_path_fermat (src/_refraction.c:746)
File "_refraction.pyx", line 63, in _refraction.find_fastest_path_fermat (src/_refraction.c:618)
File "_Roots3And4.pyx", line 24, in _Roots3And4.real_root_less_than (src/_Roots3And4.c:651)
ValueError: more than one valid root found

flydra-sphinx-docs does not build on recent sphinx version 1.0.5

This error mistifies me:

 File "/data/work/scm/snp_env/src/flydra/flydra-sphinx-docs/ext/numpydoc.py", line 137, in   get_directive
    raise RuntimeError("No directive named '%s' found" % name)
 RuntimeError: No directive named 'function' found

complete log from sphinx:

# Sphinx version: 1.0.5
# Python version: 2.6.1
# Docutils version: 0.7 release
# Jinja2 version: 2.5.5
Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/sphinx/cmdline.py", line 172, in main
    warningiserror, tags)
  File "/Library/Python/2.6/site-packages/sphinx/application.py", line 116, in __init__
    self.setup_extension(extension)
  File "/Library/Python/2.6/site-packages/sphinx/application.py", line 259, in setup_extension
    mod.setup(self)
  File "/data/work/scm/snp_env/src/flydra/flydra-sphinx-docs/ext/numpydoc.py", line 120, in setup
    app.add_directive('np-' + name, wrap_mangling_directive(name, objtype))
  File "/data/work/scm/snp_env/src/flydra/flydra-sphinx-docs/ext/numpydoc.py", line 140, in wrap_mangling_directive
    base_directive = get_directive(base_directive_name)
  File "/data/work/scm/snp_env/src/flydra/flydra-sphinx-docs/ext/numpydoc.py", line 137, in get_directive
    raise RuntimeError("No directive named '%s' found" % name)
RuntimeError: No directive named 'function' found

'PST' timezone not recognized by pytz

The h5 file seems to have have 'PST' as the timezone for me, but this is not recognized by pytz. Instead, use 'US/Pacific', which does work? This is more of a bug with pytz ('EST' and 'MST' both work, so 'PST' should), but perhaps we should be robust to that?

Crash in camnode.py while calibrating

@etiennecampione writes: "Hereafter is the error message I almost always get (let's say 95% of the time) while calibrating a Flycube. Can you indicate me what is going on? It's a really annoying and time-consuming problem".

[strawlab@flycube8 flycube8](master=)$ roslaunch ./flycube_only_flydra_calib.launch --screen
... logging to /home/strawlab/.ros/log/7433d1b0-cf1a-11e5-b183-bcee7bdac428/roslaunch-flycube8-5300.log
[...]

Exception in thread LogMessageSender:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/dist-packages/flydra/camnode.py", line 1041, in run
    self._main_brain.log_message(*log_message)
  File "/usr/lib/python2.7/dist-packages/flydra/camnode.py", line 197, in log_message
    self._log_message(req)
  File "/opt/ros/ros.electric.boost1.46/ros_comm/clients/rospy/src/rospy/impl/tcpros_service.py", line 430, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/ros.electric.boost1.46/ros_comm/clients/rospy/src/rospy/impl/tcpros_service.py", line 506, in call
    transport.send_message(request, self.seq)
  File "/opt/ros/ros.electric.boost1.46/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 590, in send_message
    self.write_buff.truncate(0)
AttributeError: 'NoneType' object has no attribute 'truncate'
[INFO] [WallTime: 1455014842.446372] setting camera property expected_trigger_framerate=100.0
[FATAL] [WallTime: 1455014842.446997] ERROR: thread LogMessageSender died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.497085] ERROR: thread save_cam.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.537913] ERROR: thread save_small.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.578685] ERROR: thread cam_processor.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.619367] ERROR: thread save_cam.mainloop died unexpectedly. Quitting
[INFO] [WallTime: 1455014842.628120] triggerbox_client: waiting for clockmodel estimate
[FATAL] [WallTime: 1455014842.629951] ERROR: thread save_small.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.630143] ERROR: thread cam_processor.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.630313] ERROR: thread LogMessageSender died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.630480] ERROR: thread save_cam.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.630644] ERROR: thread save_small.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.630809] ERROR: thread cam_processor.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.631007] ERROR: thread LogMessageSender died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.631231] ERROR: thread save_cam.mainloop died unexpectedly. Quitting
[FATAL] [WallTime: 1455014842.631436] ERROR: thread save_small.mainloop died unexpectedly. Quitting
[INFO] [WallTime: 1455014843.128506] triggerbox_client: waiting for clockmodel estimate
[flydra_camera_node-4] process has died [pid 5341, exit code 1].
log files: /home/strawlab/.ros/log/7433d1b0-cf1a-11e5-b183-bcee7bdac428/flydra_camera_node-4*.log

How does the transport die? Is it possible that the logging service (and therefore the transport) is closed in one camera LogMessageThread thread while there are still messages to log in some other camera thread, and so we would need to add sync code? Or could it just be a network problem? Surely @astraw will know better about this.

I'm saving the full logs of that run here; they include a few other stacktraces.

  /mnt/strawscience/santi/strawlab/calibration-transport-not-open-logs

registration of cameras fails occasionally (address in use)

There is a race when multiple camnodes try to register at once and the port increment counter in mainbrain gets confused.

trace from camnode:
Traceback (most recent call last):
File "/home/strawlab/ros-flydra.electric.boost1.46/bin/flydra_camera_node", line 9, in
load_entry_point('flydra==0.4.69-git', 'console_scripts', 'flydra_camera_node')()
File "/home/strawlab/flydra.git/flydra/camnode.py", line 2870, in main
parse_args_and_run()
File "/home/strawlab/flydra.git/flydra/camnode.py", line 2953, in parse_args_and_run
benchmark=benchmark,
File "/home/strawlab/flydra.git/flydra/camnode.py", line 2315, in init
camnode_ros_name = rospy.get_name())
File "/usr/lib/pymodules/python2.7/Pyro/core.py", line 381, in call
return self.__send(self.__name, args, kwargs)
File "/usr/lib/pymodules/python2.7/Pyro/core.py", line 456, in _invokePYRO
return self.adapter.remoteInvocation(name, Pyro.constants.RIF_VarargsAndKeywords, vargs, kargs)
File "/usr/lib/pymodules/python2.7/Pyro/protocol.py", line 457, in remoteInvocation
return self._remoteInvocation(method, flags, *args)
File "/usr/lib/pymodules/python2.7/Pyro/protocol.py", line 532, in _remoteInvocation
answer.raiseEx()
File "/usr/lib/pymodules/python2.7/Pyro/errors.py", line 72, in raiseEx
raise self.excObj
socket.error: [Errno 98] Address already in use

trace from mainbrain
REGISTER NEW CAMERA Basler_21029386 on noise.flycave @ ros node /flydra_camera_node_noise
Exception in thread CoordinateProcessor thread:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/home/strawlab/flydra.git/flydra/MainBrain.py", line 884, in run
predicted_framenumber = n_frames_skipped + self.last_framenumbers_skip[cam_idx] + 1
IndexError: list index out of range

image-based-orientations in webapp sometimes don't save movies even when requested

This document (in the Altshuler lab database) shows a case where no images were saved despite requesting it. Also, no error messages were saved.

Flydra fails to smooth after updating to patched-pytables 3.2.0

@astraw @nzjrs Let's comment here on these errors.

check 2D orientations in pseudo real time

It would be good to have a way to check 2D orientations in pseudo real time. By pseudo I mean that while the program is running you can run an analysis script and generate a graph of the 2D position tracking of what has happened up to that point in time. If you want to take this a step further, it would be great to try to implement a script that has a graphical output of what is going on in actual real time. This is a humdra-specific project to the extent that 2D orientation is estimated differently for humdra (sampling rate 5x that of wingbeat rate) versus flydra (sampling rate 0.5x that of wingbeat rate), but a majority of the legwork should be the shared.

start saving hdf5 produces error

There is an error when attempting to save hdf5 files.

Here are the steps:

Flydra launches with 8 cameras fine, except for the following warning:

/home/beelab/ros/flydra-kinetic/src/ros_flydra/nodes/main_brain:54: SyntaxWarning: The publisher should be created with an explicit keyword argument 'queue_size'. Please see http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers for more information.
  self.pub = rospy.Publisher("%s/tracking" % cam_id, sensor_msgs.msg.Image, subscriber_listener=self)

Cameras syncs okay:

beelab@beelab:~/ros/flydra-kinetic/src/ros_flydra$ roscd ros_flydra; python scripts/simple-console.py --sync
sync cameras
done

But when trying to start saving hdf5:
beelab@beelab:~/ros/flydra-kinetic/src/ros_flydra$ roscd ros_flydra; python scripts/simple-console.py --start-saving-hdf5

the following error is produced:

[ERROR] [1537378936.478427]: Error processing request: image cannot be None
['Traceback (most recent call last):\n', '  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 625, in _handle_request\n    response = convert_return_to_response(self.handler(request), self.response_class)\n', '  File "/usr/lib/python2.7/dist-packages/flydra_core/MainBrain.py", line 676, in _ros_generic_service_dispatch\n    result = getattr(self, calledfunction)(**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/flydra_core/MainBrain.py", line 1226, in start_saving_data\n    raise ValueError(\'image cannot be None\')\n', 'ValueError: image cannot be None\n']

and in the other command line window:

beelab@beelab:~/ros/flydra-kinetic/src/ros_flydra$ roscd ros_flydra; python scripts/simple-console.py --start-saving-hdf5
start saving HDF5
Traceback (most recent call last):
  File "scripts/simple-console.py", line 100, in <module>
    start_saving_hdf5()
  File "scripts/simple-console.py", line 34, in start_saving_hdf5
    xx.call()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call
    responses = transport.receive_once()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 727, in receive_once
    p.read_messages(b, msg_queue, sock) 
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages
    self._read_ok_byte(b, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte
    raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str))
rospy.service.ServiceException: service [/flydra_mainbrain/start_saving_data] responded with an error: error processing request: image cannot be None

Missing cameras dataset in reprojection error files

Hi,

@etiennecampione is having some troubles when trying to generate the reprojection error plots.

Traceback (most recent call last):
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/bin/plot-historical-reproj", line 5, in <module>
pkg_resources.run_script('ros-electric-flycave==3.3', 'plot-historical-reproj')
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
execfile(script_filename, namespace, namespace)
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/lib/python2.7/site-packages/ros_electric_flycave-3.3-py2.7.egg/EGG-INFO/scripts/plot-historical-reproj", line 190, in <module>
data,limit = get_data(args.assay,start_date,args.uuid,args.limit)
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/lib/python2.7/site-packages/ros_electric_flycave-3.3-py2.7.egg/EGG-INFO/scripts/plot-historical-reproj", line 130, in get_data
show_progress=False).get_reprojection_distance()
File "/home/auto-data-webserver/.virtualenvs/flycave_webserver/local/lib/python2.7/site-packages/ros_electric_flycave-3.3-py2.7.egg/autodata/files.py", line 294, in get_reprojection_distance
for i,s in store['cameras'].iterrows():
File "/usr/lib/python2.7/dist-packages/pandas/io/pytables.py", line 416, in __getitem__
return self.get(key)
File "/usr/lib/python2.7/dist-packages/pandas/io/pytables.py", line 619, in get
raise KeyError('No object named %s in the file' % key)
KeyError: 'No object named cameras in the file'
Closing remaining open files:/mnt/strawscience/data/auto_pipeline/raw_archive/by_uuid/6c8accbc347311e58634d850e6c4a608/20150727_172257.repro_errors.h5...done

The cameras dataset is not there anymore (I guess we could just fake it, if necessary). It seems to be happening in all recent experiments. Are we missing something?

camnode launch file fails

I created the following launch file:

<launch>
<rosparam command="load" file="$(find freemoovr_arena)/launch/flydra.yaml" />
<node name="AVT-02-2020C-07056" pkg="ros_flydra" type="camnode2-pylon" args="--camera-name=AVT-02-2020C-07056 --http-server-addr=127.0.0.1:4011 --ros-tracking-start-delay=30"> </node>
<node name="AVT-02-2020C-07058" pkg="ros_flydra" type="camnode2-pylon" args="--camera-name=AVT-02-2020C-07058 --http-server-addr=127.0.0.1:4011 --ros-tracking-start-delay=30">
</node>
</launch>

which results in the following error:

flightlab@ubuntu:~$ roslaunch /home/flightlab/ros/flydra-kinetic/src/ros_flydra/launch/prosilica1.launch
... logging to /home/flightlab/.ros/log/f1867084-65f2-11e8-b48b-000c2967883e/roslaunch-ubuntu-95368.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

Traceback (most recent call last):
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/__init__.py", line 306, in main p.start()
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/parent.py", line 268, in start self._start_infrastructure()
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/parent.py", line 217, in _start_infrastructure self._load_config()
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/parent.py", line 132, in _load_config roslaunch_strs=self.roslaunch_strs, verbose=self.verbose)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/config.py", line 451, in load_config_default loader.load(f, config, verbose=verbose)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 748, in load self._load_launch(launch, ros_config, is_core=core, filename=filename, argv=argv, verbose=verbose)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 720, in _load_launch self._recurse_load(ros_config, launch.childNodes, self.root_context, None, is_core, verbose)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 680, in _recurse_load self._rosparam_tag(tag, context, ros_config, verbose=verbose)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 95, in call return f(*args, **kwds)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 228, in _rosparam_tag cmd, ns, file, param, subst_value = self.opt_attrs(tag, context, (XmlLoader.ROSPARAM_OPT_ATTRS))
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 202, in opt_attrs return [self.resolve_args(tag_value(tag,a), context) for a in attrs]
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/xmlloader.py", line 183, in resolve_args return substitution_args.resolve_args(args, context=context.resolve_dict, resolve_anon=self.resolve_anon)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/substitution_args.py", line 370, in resolve_args resolved = _resolve_args(resolved, context, resolve_anon, commands)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/substitution_args.py", line 383, in _resolve_args resolved = commands[command](resolved, a, args, context)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/substitution_args.py", line 151, in _find source_path_to_packages=source_path_to_packages)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/substitution_args.py", line 197, in _find_executable full_path = _get_executable_path(rp.get_path(args[0]), path)
File "/usr/lib/python2.7/dist-packages/rospkg/rospack.py", line 203, in get_path raise ResourceNotFound(name, ros_paths=self._ros_paths)
ResourceNotFound: freemoovr_arena
ROS path [0]=/opt/ros/kinetic/share/ros
ROS path [1]=/home/flightlab/ros/flydra-kinetic/src
ROS path [2]=/opt/ros/kinetic/share

write unit tests for scale and units of dynamic models and camera calibrations

I think there may be bugs with the scale and the units of the various dynamic models and calibrations. We should really write some unit tests to find them and eliminate them.

add ability to replay .ufmf and .h5 file to be re-tracked

Implementing low-level 2D tracking of near-raw video (.ufmf) for flydra

Background

For the humdra project, we've been saving out .ufmf ("micro .fmf") videos as standard operating procedure -- while we can do online 3D position tracking, the orientation tracking depends on image analysis that is not implemented for online usage and consequently we need to save image regions around the birds to disk. (Therefore, the hard requirement for realtime analysis in this case is limited to identifying the appropriate 2D image regions that are then saved to disk.)

The immediate problem

The present issue is that we've discovered that in some cases, the low level online realtime 2D feature extraction is falling victim to an assumption made for Drosophila that does not hold true for hummingbirds. Specifically, we track only a single 2D feature coming from a given NxN pixel square of image space. With flies, multiple animals only transiently co-localize in a given small patch, and when considering views from multiple angles, this is almost never a problem in practice for estimating 3D positions, even of multiple flies. With hummingbirds, however, we are discovering that this is a real issue (exacerbated by other factors that have pushed us to increase the size of the NxN pixel local region).

The solution

The good news is that we do have all relevant information on disk - even though we didn't do the low-level 2D feature extraction correctly the first time around, the .ufmf videos (and the .h5 file with time sync information, amongst other things) have all the required information to completely re-extract this information. What is required is to develop the capability the re-play these videos to the relevant flydra realtime image processing code (with new parameters that minimize or eliminate the hummingbird-related issues). I've made one or two attempts in the past to get this working, but never managed to drive anything to completion. I anticipate that the effort will require two steps. The first is to get the appropriately synchronized data from the .ufmf files saved from multiple cameras available from processing. This is more-or-less already implemented in the flydra.a2.ufmf_tools.iterate_frames() function. The second step is to feed these data into a suitably modified program that runs the same image analysis as camnode. Because the full-frame images that contribute to making the full background model are also saved to the .ufmf file, but not every full-size frame that arrives, there will be differences from the realtime system in terms of how to deal with updating the background model. I think the most useful way to implement this would be to extend the camnode program to handle this scenario rather than to re-implement a purely-offline low-level 2D tracker program. Nevertheless, the camnode program is quite ugly and I would be sympathetic to the creation of a new program for doing the job. (There are remnants of my attempts to do this still within the camnode program, which may help or hinder the do-it-in-camnode option.)

Other benefits to the solution

A) We can use this to add automated testing of the basic fly & bird 2D tracking code. This would go into the automated test suite that gets run nightly (see http://flydra.bb.astraw.com/waterfall ) and would ensure that new errors don't creep unnoticed into the code we all depend on. This would especially be facilitated if we did implement the replay solution within camnode itself.
B) It would allow us to trivially save out the relevant video data in a raw form that wold be suitable for sharing with others (e.g. computer vision people) who might want to develop new tracking algorithms and not depend on the whole creaking mass of crufty software we call flydra.

And I hesitate to suggest a more-work option with more long-term benefits, but for the sake of completeness

One further suggestion: a re-write of camnode to A) make it more ROS-centric and B) easier to extend (as suggested above) and maintain, would be to re-write the whole thing. I would suggest using ROS nodelets to handle the sharing of the realtime image acquisition, processing, saving, and streaming across different threads, and this would mean switching to C++ as the main implementation language. That would quite likely also mean a serious reduction in the complexity of the Python->C->Python conversions that happen in order to call the image acquisition and processing libraries mostly providing C interfaces.

image_based_orientation 'morphed' image same as 'abs-diff'

When using image_based_orientation to save movies it looks like the 'morphed' image (row 1, column 4) is the same as the 'abs_diff' image (row 1, column 3).

Also, when using the parameter --final-thresh (at least for the hummingbirds) this doesn't have an affect unless it's cranked up to >100. The default value is 7, which doesn't seem to have much of an effect.

unable to calibrate with 0.6.8

@etiennecampione was unable to calibrate flycube8 using 0.6.8 as there were no points present in the h5 file, despite them being clearly visible in the image view. rolling back to 0.6.7 and not changing any other settings fixed the issue.

I don't think this is related to #30 as commenting out the logging code (both the service proxy on the camnode side, and the service on the mainbrain side) does not fix the bug.

this is in contrast with @maxlklaxl who has made several calibrations with 0.6.8 (although that was on ros-hydro)

flydra_analysis_flip_calibration

NotImplementedError: no mirroring implemented for skewed cameras

MCSC often produces calibration with the cameras pointed outwards, solving this would speed up the calibration process

mainbrain.h5 files causing such inverted calibrations and right calibrations are stored in: strawscience/Max/forAndrew/camera_orientation/

flydra_camera_node should publish /cam_id/image_raw ROS topics

To enable use with ROS systems, the flydra_camera_nodes should publish their images to the ROS network.

use 3D distrance to set ROI radius in image_based_orientation

There is a warning 'roiradius hard-coded to %d: could be set from 3D tracking' which is exactly true. This looks like an extra 2 or 3 lines of code could really help here.

draw detected points in /tracking view even if no calibration is loaded

no reason not to

plot_summary only plots first flight episode when multiple single episodes are in fanout file

The following command only plots the first episode:

flydra_analysis_plot_summary --stim-xml=fanout.xml --interactive -k DATA20101130_233000.kalmanized.h5

(This fanout.xml file has two <single_episode> elements for this .h5 file.) Of course, plotting without the fanout.xml file shows all the data.

tracked object data should be periodically flushed to disk

Currently, to save realtime 3D data to disk, we rely on either A) a call to stop saving data or B) a tracked object being killed. However if the mainbrain crashes hard, this can mean long trajectories that are currently being tracked get completely lost. Therefore, we should implement periodically flushing the currently tracked objects to disk without requiring them to be killed.

make camera node publish ROS /camera_node/camera_info topic

The flydra camera node class should support the normal ROS camera info stuff. See this http://wiki.ros.org/camera_info_manager_py

Frame discontinuities in mainbrain files

Continuation of #24, to update on the issue of trials in mainbrain warping in time, possible fixes at analysis level for old files and possible fixes when writing.

This code is a mixture of ad-hoc script and reusable module wannabe to check for "continuity" problems and generate reports to assess the extent of the problem in our files and suggest test cases. Some of it parses outputs from flydra's check continuity script. Most important is this function, which we could adapt to check stuff at least in flydra, if not also during the auto-pipeline, combine and others. We also generate a couple of handy dataframes.

Current logs live here:
/mnt/strawscience/santi/mainbrain-holy-logs
There there are some h5 files with data aggregated in pandas dataframes.

I think we have at least two kinds of discontinuities:

• Old (fixed around September 2014): many trials with many duplicated frames (but different data), "jumps" of between -3 and 0 frames (0 = frame repeated). A complete summary for 20131205_174729, one of Lisa's DCN experiments, is in /mnt/strawscience/santi/mainbrain-holy-logs/mainbrain-full-logs. If we need a fix, we could just drop repeated frames when resmoothing (if we agree that won't hurt much the estimates). These are some summary stats:

Max number of jumps in an obj_id:  196
Max number of missing frames in an obj_id:  0
Max number of exceeding obs in an obj_id:  413
Seen jump sizes:  [-3.0, -2.0, -1.0, 0.0]

• New: a few trials at the beginning; these seem to just have two blocks of observations (like 3,4,5,0,1,2), so we could simply merge them in the proper order or just drop them.

I haven't really made sure that there are not other types of errors.

flush currently tracked object to disk when stopping ongoing save

As observed in 20150522_191410.mainbrain.h5, there seems to be a bug in which mainbrain can stop saving a file without flushing all (presumably currently tracked) 3D objects.

montage-ufmf non-fatal error message

When montage-ufmf finishes running it exits with this error message:

Closing remaining open files: /home/flightlab/Desktop/1/1/DATA20100218_142912.3D_orientation.kh5-smoothcache... done 1/DATA20100218_142912.3D_orientation.h5... done
Exception AttributeError: AttributeError("'NoneType' object has no attribute 'ref'",) in <bound method CachingAnalyzer.del of <flydra.a2.core_analysis.CachingAnalyzer instance at 0x4adcb48>> ignored
Exception TypeError: 'must be type, not None' in ignored

But the video turns out fine so this is not really an urgent issue.

MemoryError -- flydra_analysis_export_flydra_hdf5

tired it 2 times on the fishtrax machine, both times the same error
the .h5 file is 13G

(PY)strawlab@fishtrax:~/FLYDRA$ flydra_analysis_export_flydra_hdf5 20140719_152830.mainbrain.h5
STAGE 1: finding timestamps
opening file 20140719_152830.mainbrain.h5...
caching raw 2D data... done
(cached index of 145054462 frame values of dtype int64)
hostname time_gain time_offset

---

'localhost' 1.0 -0.00697165842925

caching Kalman obj_ids...
finding unique obj_ids...
(found 57384)
(will export 57384)
finding 2d data for each obj_id...
skipping frame 403 (obj 1): no data2d_distorted data
skipping frame 404 (obj 2): no data2d_distorted data
skipping frame 405 (obj 3): no data2d_distorted data
skipping frame 4439 (obj 47): no data2d_distorted data
skipping frame 4621 (obj 52): no data2d_distorted data
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:168: UserWarning: no host fishtrax in timestamp data. making up data.
'data.'%remote_hostname)
STAGE 2: running Kalman smoothing operation
detected file loaded with dynamic model "EKF fishbowl40"
for smoothing, will use dynamic model "fishbowl40"
/home/strawlab/src/flydra/flydra/a2/core_analysis.py:1467: UserWarning: passing data_file as string to core_analysis.CachingAnalyzer.load_data()
warnings.warn('passing data_file as string to '
/home/strawlab/src/flydra/flydra/a2/core_analysis.py:555: RuntimeWarning: invalid value encountered in true_divide
velocity_direction = velocity/speed[:,np.newaxis]
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:243: UserWarning: algebra error smoothing obj_id 17414, skipping.
warnings.warn('algebra error smoothing obj_id %d, skipping.'%(obj_id,))
/home/strawlab/src/flydra/flydra/a2/data2smoothed.py:243: UserWarning: algebra error smoothing obj_id 47242, skipping.
warnings.warn('algebra error smoothing obj_id %d, skipping.'%(obj_id,))
[obj_id: 57432] 2.07 obj_ids /s100%|##########################################################|Time: 7:41:40
Traceback (most recent call last):
File "/home/strawlab/PY/bin/flydra_analysis_export_flydra_hdf5", line 9, in
load_entry_point('flydra==0.5.9', 'console_scripts', 'flydra_analysis_export_flydra_hdf5')()
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 280, in export_flydra_hdf5
main(hdf5_only=True)
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 377, in main
**kwargs)
File "/home/strawlab/src/flydra/flydra/a2/data2smoothed.py", line 262, in convert
recarray = numpy.rec.array(allrows)
File "/usr/lib/python2.7/dist-packages/numpy/core/records.py", line 790, in array
new = new.copy()
MemoryError
Closing remaining open files: /home/strawlab/FLYDRA/20140719_152830.mainbrain.kh5-smoothcache... done 20140719_152830.mainbrain.h5... done

montage_ufmfs: missing --obj-only parameter

The previous analysis steps have --obj-only parameters but montage-ufmfs doesn't. If you try to input a set of frames into montage-ufmfs that have objects that to not have 3-D orientations it doesn't work. To get around this you can analyze the entire h5 file, but this is very time consuming if you only want to get a video of a short sequence.