yyan-neuro / ncams

Toolbox for using multiple cameras from intrinsic calculations to reconstructing kinematics

NCams

NCams is a toolbox for intrinsic and extrinsic calibration for multiple cameras, extracting 3D marker trajectories and estimate skeletal kinematics. It also supports recording with FLIR cameras.

NCams is installed as a Python module with several submodules that include camera calibration, estimation of relative camera positions, triangulation of the marker information from multiple cameras, preparing inverse kinematics, filtering in 2D or 3D. The module can run on Windows or Unix/Linux.

Getting Started

This project's code is available on GitHub. The example raw data is available on Dropbox and raw images on Box.

Prerequisites

Software:

• DeepLabCut - if using it for labeling markers. See below for installation recommendations.
• NVIDIA drivers
• CUDA 10.0
• Spinnaker Python module if using FLIR cameras for capture.
• OpenSim 4.0 to calculate inverse kinematics.
• Module dependencies are listed in the setup file.

If you have problems with installations, check out our installation tips.

There are no hardware prerequisites for the core functionality of the toolbox. If using DeepLabCut or the spinnaker_tools, however, then an NVIDIA videocard that supports CUDA or FLIR cameras are required resspectively.

Installation

1. Download the repository or clone it using git: git clone https://github.com/CMGreenspon/NCams.git.
2. Open Terminal, or Command Line or the desired Anaconda environment (e.g. the one with DeepLabCut installed) in the project folder.
3. Run python setup.py install.

Examples of use

All examples have a corresponding 'run_' script to show examples of executing the scripts in the command line, avoiding the IDE.

Calibration and pose estimation

Example with capturing data contains example code for setting up multiple FLIR cameras, calibration of lenses, estimation of their poses, and creation of videos from captured images.

Example with supplied data can help you calibrate the lenses on the cameras, estimate their poses, and create of videos. The data to run the code on is stored on Dropbox and Box.

Labeling and 3D marker reconstruction

The analysis goes over marking images with DeepLabCut, training a network and triangulation of the marker data.

Analysis of multiple sessions follows the analysis example, but is specifically designed to handle multiple sessions of recordings from the same cameras.

Tips have suggestions on NCams/DeepLabCut use that can be useful.

Inverse kinematics

The guide describes the necessary tools and steps to obtain joint angles of a skeletal model that follow the measured markers using OpenSim (SimTK). We use a [publically available skeletal model](??? not yet) of the human right arm and hand that we modified and markered to fit our purposes.

Structure of the repository

• ncams -- imported module
• Examples -- examples and guides on using NCams
• documentation -- various documentation on NCams
• opensim_models -- OpenSim skeletal models that we use for inverse kinematics
• dlc_markers -- images with location of markers relatively to bone segments and a DLC config with marker names and skeleton
• images -- demonstrational images and gifs

Authors

• Charles M. Greenspon
• Anton Sobinov
• Developed in Bensmaia Lab.