openworm / open-worm-analysis-toolbox

A testing pipeline that allows us to run a behavioural phenotyping of our virtual worm running the same test statistics the Schafer Lab used on their worm data.

About OpenWorm

OpenWorm aims to build the first comprehensive computational model of Caenorhabditis elegans (C. elegans), a microscopic roundworm. With only a thousand cells, it solves basic problems such as feeding, mate-finding and predator avoidance. Despite being extremely well-studied in biology, a deep, principled understanding of the biology of this organism remains elusive.

We are using a bottom-up approach, aimed at observing the worm behaviour emerge from a simulation of data derived from scientific experiments carried out over the past decade. To do so, we are incorporating the data available from the scientific community into software models. We are also forging new collaborations with universities and research institutes to collect data that fill in the gaps.

You can earn a badge with us simply by trying out this package! Click on the image below to get started.

Quickstart

We have put together a Docker container that pulls together the major components of our simulation and runs it on your machine. When you get it all running it does the following:

1. Run our nervous system model, known as c302, on your computer.
2. In parallel, run our 3D worm body model, known as Sibernetic, on your computer, using the output of the nervous system model.
3. Produce graphs from the nervous system and body model that demonstrate its behavior on your computer for you to inspect.
4. Produce a movie showing the output of the body model.

Example Output

NOTE: Running the simulation for the full amount of time would produce content like the above. However, in order to run in a reasonable amount of time, the default run time for the simulation is limited. As such, you will see only a partial output, equivalent to about 5% of run time, compared to the examples above. To extend the run time, use the -d argument as described below.

Installation

Pre-requisites:

1. You should have at least 60 GB of free space on your machine and at least 2GB of RAM
2. You should be able to clone git repositories on your machine. Install git, or this GUI may be useful.

To Install:

1. Install Docker on your system.
2. If your system does not have enough free space, you can use an external hard disk. On MacOS X, the location for image storage can be specified in the Advanced Tab in Preferences. See this thread in addition for Linux instructions.

Running

1. Ensure the Docker daemon is running in the background (on MacOS/Windows there should be an icon with the Docker whale logo showing in the menu bar/system tray).
2. Open a terminal and run: git clone http://github.com/openworm/openworm; cd openworm
3. Optional: Run ./build.sh (or build.cmd on Windows). If you skip this step, it will download the latest released Docker image from the OpenWorm Docker hub.
4. Run ./run.sh (or run.cmd on Windows).
5. About 5-10 minutes of output will display on the screen as the steps run.
6. The simulation will end. Run stop.sh (stop.cmd on Windows) on your system to clean up the running container.
7. Inspect the output in the output directory on your local machine.

Advanced

Arguments

• -d [num] : Use to modify the duration of the simulation in milliseconds. Default is 15. Use 5000 to run for time to make the full movie above (i.e. 5 seconds).

Other things to try

• Open a terminal and run ./run-shell-only.sh (or run-shell-only.cmd on Windows). This will let you log into the container before it has run master_openworm.py. From here you can inspect the internals of the various checked out code bases and installed systems and modify things. Afterwards you'll still need to run ./stop.sh to clean up.
• If you wish to modify what gets installed, you should modify Dockerfile. If you want to modify what runs, you should modify master_openworm.py. Either way you will need to run build.sh in order to rebuild the image locally. Afterwards you can run normally.

FAQ

What is the Docker container?

The Docker container is a self-contained environment in which you can run OpenWorm simulations. It's fully set up to get you started by following the steps above. At the moment, it runs simulations and produces visualizations for you, but these visualizations must be viewed outside of the Docker container. While you do not need to know much about Docker to use OpenWorm, if you are planning on working extensively with the platform, you may benefit from understanding some basics. Docker Curriculum is an excellent tutorial for beginners that is straightforward to work through (Sections 1 - 2.5 are plenty sufficient).

Is it possible to modify the simulation without having to run build.sh?

Yes, but it is marginally more complex. The easiest way is to modify anything in the Docker container once you are inside of it - it will work just like a bash shell. If you want to modify any code in the container, you'll need to use an editor that runs in the terminal, like nano. Once you've modified something in the container, you don't need to re-build. However, if you run stop.sh once you exit, those changes will be gone.

How do I access more data than what is already output?

The simulation by default outputs only a few figures and movies to your home system (that is, outside of the Docker container). If you want to access the entire output of the simulation, you will need to copy it from the Docker container.

For example, say you want to extract the worm motion data. This is contained in the file worm_motion_log.txt, which is found in the /home/ow/sibernetic/simulations/[SPECIFIC_TIMESTAMPED_DIRECTORY]/worm_motion_log.txt. The directory [SPECIFIC_TIMESTAMPED_DIRECTORY] will have a name like C2_FW_2018_02-12_18-36-32, and its name can be found by checking the output directory. This is actually the main output directory for the simulation, and contains all output, including cell modelling and worm movement.

Once the simulation ends and you exit the container with exit, but before you run stop.sh, run the following command from the openworm-docker-master folder:

docker cp openworm:/home/ow/sibernetic/simulations/[SPECIFIC_TIMESTAMPED_DIRECTORY]/worm_motion_log.txt ./worm_motion_log.txt

This will copy the file from the Docker container, whose default name is openworm. It is crucial that you do not run stop.sh before trying to get your data out (see below)

What is the difference between exit and stop.sh?

When you are in the Docker Container openworm, and are done interacting with it, you type exit to return to your system's shell. This stops execution of anything in the container, and that container's status is now Exited. If you try to re-start the process using run-shell-only.sh, you will get an error saying that the container already exists. You can choose, at this point, to run stop.sh. Doing so will remove the container and any files associated with it, allowing you to run a new simulation. However, if you don't want to remove that container, you will instead want to re-enter it.

How do I enter a container I just exited?

If you run stop.sh you'll delete your data and reset the container for a new run. If, however, you don't want to do that, you can re-enter the Docker container like this:

docker start openworm                 # Restarts the container
docker exec -it openworm /bin/bash    # Runs bash inside the container

This tells Docker to start the container, to execute commands (exec) with an interactive, tty (-it) bash (bash) shell in the container openworm.

You'll be able to interact with the container as before.

Documentation

to find out more about OpenWorm, please see the documentation at http://docs.openworm.org or join us on Slack.

This repository also contains project-wide tracking via high-level issues and milestones.

open-worm-analysis-toolbox's Issues

Attempt to integrate Peter McCluskey's code gist into this repo

Look at Peter McCluskey’s code and see if it could be used for our purposes as well.

https://gist.github.com/PeterMcCluskey/6418155

Create pie chart of video segmentation

Like this, from p131 of Ev Yemini's dissertation

Have WormExperimentFile.create_animation(self) take a num_frames parameter

So that the user can specify what length of animation to create. Have the parameter be optional and default to all frames.

Convert all repository documentation from MD to RST for compatibility with Sphynx

http://rst.ninjs.org/

Add the information @slarson had in the inital comment on overall openworm issue 82 to the documentation

E.g. a link to https://groups.google.com/forum/?fromgroups#!searchin/openworm-discuss/worm$20behavior/openworm-discuss/Fhbnag_FHsY/s18gywbjOHcJ

Also plenty of other good information.

Try replicating some of the Schafer Worm Tracker 2.0 interface in the matplotlib worm visualizer.

Try replicating some of this function in the worm visualizer:
http://www.mrc-lmb.cam.ac.uk/wormtracker/index.php?action=analysis

move pipeline md to documentation, and move all data files out of root path

and document where they are (in worm_data shared dropbox folder)

Consider representing the Schafer FTP server as a Git-Annex

http://git-annex.branchable.com/

(A suggestion from David Dalrymple)

AndrewPa had created a "File Hoover" to grab files from the Schafer FTP server.

https://gist.github.com/AndrewPa/5855669

There is concern that this FTP server is not a long-term stable source for these files, so it may not be a good idea to pursue this.

Clean up and organize issues onto Waffle board

https://waffle.io/openworm/movement_validation

Display features in a coloured grid-chart just like in the Yemini paper

viz. page 7 of the supplementary data, or on page 2 of the advance online paper:

A database of C. elegans behavrioral phenotypes, by Yemini et. al, in the journal Nature Methods

Translate morphology features

Have GitHub ignore updates to __pycache__ when syncing the repo

Here is an explanation of how to do this: https://help.github.com/articles/ignoring-files

Examine the "Track-A-Worm" CV Algorithm

Our repo is based on the Schafer Lab code. Their machine vision algorithm will return as an array of NaNs any frame where the worm is touching itself (e.g. creating omega turns)

Another machine algorithm is also available (Track-A-Worm), and it is supposed to be able to segment video frames with omega turns.

To complete this issue, try to run both algorithms and see how they differ and report back on differences.

Understand how to read ASCII-encoded parts of the .MAT experiment files

Make sure code follows PEP-8

In particular, PEP-8 recommends:

1. Use four-character indendation (@MichaelCurrie breaks this rule, please fix!)
2. Limit all lines to a maximum of 79 characters. (@JimHokanson sometimes breaks this rule, please fix :)

Change FPS from a global setting in config.py to a parameter of the file

The frames per second can differ between experiment files, so this must be made into a parameter rather than a constant in config.py.

Make the config local

The configuration options are currently global in nature. It would be better if these were passed in locally so that changes to these parameters could be made at run time rather than manually editing the config file.

Also, the options should be cleaned during this process. Each parameter in the config should be identified by its section as well as accompanied by a definition of what it impacts as well as the units of the parameter.

Provide @vellamike with an example simulation output file

As per this discussion

Fix the flicker in the render

Work on fixing the flicker that is currently appearing in the rendering, by using an animation module accounting for dropped frames (these show up as NaNs in the worm skeleton data).

Understand degree of convertibility between contour and skeleton

If we take submissions of skeleton movements, how will we handle the lack of a contour? To what extent can we determine the contour from some default worm shape?
-Jim

Add data from MRC supplemental information to feature spreadsheet

2013 - Yemini et al. - Supplementary Data - nmeth.2560-S1.pdf contains a description in words for each feature.

Consider adding a matplotlib illustration for each feature as well.

Name the test harness / open source WT2 project

This project deserves a name, that captures the three goals of the project to

1. create an open source version of WT2
2. create a real/virtual worm statistical test harness
3. track of the statistics for multiple worms in a database

Change "\\" to r"\"

Document all of the structure of the worm experiment files

Add to the existing documentation that Jim has done on the structure of the worm data in these files, including perhaps the fields not in the “worm” structure

Normalized worm - data dict

Why does the normalized worm have an attribute, data_dict, that contains attributes?

Wouldn't it be better to place those attributes directly in the object?

Try a scatterplot of worm position over time.

Time is given by info.video.length.time in the experiment file

Create a generalized interpolation function

Right now * SchaferExperimentFile.interpolate_dropped_frames * returns a new SchaferExperimentFile instance with all frames of skeletons interpolated linearly.

Instead, let's have a standalone function that will take any three-dimensional numpy array and interpolate linearly along the desired axis.

Furthermore, let's have a parameter that can be passed to specify the threshold, e.g. you could pass 2 and only contiguous NaN frames of length 2 or less would be interpolated. This could be advantageous for avoid the interpolation of omega turns.

Finally, we can add an interpolate method to our NormalizedWorm and other classes, which uses this standalone function to interpolate ALL of its data, not just the skeleton as is the case now.

Render more worm attributes in matplotlib

Play with matplotlib to try rendering the worm in different ways, such as rendering information about ventral/dorsal orientation and the body contour of the worm.

Translate path features

Add "struct" dereferencing

Incorporate this structure:

wormFile["worm"]["locomotion"]["motion"]["backward"]["frames"]["distance"].value

from https://gist.github.com/JimHokanson/6420348

Currently the only structure explored is this one:

skeletonX = wormFile["worm"]["posture"]["skeleton"]["x"].value
skeletonY = wormFile["worm"]["posture"]["skeleton"]["y"].value

Calculate morphology.length as a feature

Right now it is loaded directly from the normalized worm file, but we won't have that luxury when it comes to the input of the artificial worm.

Instead it should be natively "computed from the segmented skeleton by converting the chain-code pixel length to microns."

Design decision: locate special helper functions in NormalizedWorm class?

design principle: should be able to call special functions of NormalizedWorm?

question: do i pass the minimum to the feature helpers functions? because if so, I can't have special helper functions in NormalizedWorm.??

Translate posture features

Display time information about the worm in the plot

Fix dependency: getCoils requires locomotion.velocity

Awkward dependencies:

• coiling - seg_worm.feature_helpers.posture.getCoils - requires
  worm.locomotion.velocity.midbody.speed

Edit the pipeline documentation to show graphics at each step

E.g. show the raw worm, then the contour, then measurements like width, etc., then features, then stats.

Use subclassing in animation code

http://matplotlib.org/1.3.0/examples/animation/subplots.html

Move feature calculation code into classes

Make README simpler and clearer with more usage examples

Follow PyOpenWorm's README.md for a good example of how to do it right.

Move any content that is not appropriate for the README into linked documents off the main page.

Put hardcoded configuration settings in a config.py file

Move N_ECCENTRICITY and any other hardcoded values to a config.py file.

http://stackoverflow.com/questions/6198372/most-pythonic-way-to-provide-global-configuration-variables-in-config-py

http://docs.python.org/2/library/configparser.html

Dependent attributes (properties) should use @property decorator

Please add @Property to all methods which are actually attributes in order to avoid needing to use parentheses to unbind the methods.

For example

nw.num_frames returns a method object

nw.num_frames() executes the method and returns a value

Changing this to:

@Property
def num_frames

now we can do:
nw.num_frames and get the value

Add Sphynx for documentation

Sphinx creates lovely and pythonic documentation: e.g. Pyvot or Beautiful Soup

Translate locomotion features

Organize the config

The config is currently a mess. Each parameter in the config should be identified by its section as well as accompanied by a definition of what it impacts

Test features against SegwormMatLabClasses code

Optimize eccentricity code

The eccentricity code is painfully slow to run. The Python code currently uses a naive approach. The Matlab code was written to take advantage of certain knowledge that we have to make the execution much faster.

Matlab code:
https://github.com/JimHokanson/SegwormMatlabClasses/blob/master/%2Bseg_worm/%2Bfeatures/%40posture/getEccentricity.m

Add full DropBox integration

https://www.dropbox.com/developers/core/start/python

Change user data path in wormpy_example

Change user data path to come from a file, which is set in .gitignore to be ignored, so it can be set differently for each fork.

Interpolate a specific number of contiguous dropped frames

add an option to interpolate only X number of contiguous dropped frames (e.g. if 1 is specified, only single dropped frames are fixed, which are virtually guaranteed to be correctly interpolated given the slow speed of the deformation and movement of the worm relative to the frame rate)

Error: Non-negative bend stddevs are zero

sign_array = - np.ones(np.shape(mask)) * mask

must be changed to add ~mask * np.ones(np.shape(mask))