thladnik / vxpy

Multiprocess based software for vision experiments in Python

vxPy

vxPy is a multiprocessing-based software for vision experiments in Python.

It leverages OpenGL-based 3D graphics rendering, using the Python visualization library VisPy, for dynamic generation and realtime updating of visual stimuli. vxPy utilizes multicore hardware for fast online aquisition and analysis of behavioral and other sensor data, as well as control of external devices, such as actuators or LEDs, via configurable microcontroller interfaces.

This is the core package for vxPy. The accompanying application to configure and run the UI is hosted at vxPy_app.

Requirements

vxPy has been tested on Windows 11 and Ubuntu 20.04 and 22.04 LTS. It requires Python 3.8 or higher. For best performance Ubuntu is recommended.

Installation

Installing Python

Windows

Download and install the Python 3.8+ binaries if not already installed from https://www.python.org/downloads/

Ubuntu

Install Python

user@machine: ~$ sudo apt-get install python3.x 

Installing vxPy

Linux

Create a new folder where you'd like to install the vxPy application (herevxPy_app). Using a terminal, create a virtual environment inside the empty folder, install vxPy and set up the application folder

user@machine: ~/vxPy_app$ python3.x -m venv venv
user@machine: ~/vxPy_app$ ./venv/bin/activate
(venv) user@machine: ~/vxPy_app$ pip install vxpy
(venv) user@machine: ~/vxPy_app$ vxpy setup

You can then run the default demo configuration with

(venv) user@machine: ~/vxPy_app$ vxpy -c configurations/example.yaml run

Upon first start, vxpy will download demo data files, which may take some additional time.

Compatible devices

Cameras

TheImagingSource (TIS) cameras

Under Windows TIS cameras are supported out of the box, using the TIS' original tisgrabber DLLs and their ctype bindings included in vxPy.

In order to use TIS cameras under Linux, you need to install tiscamera (Github repository) by following the instructions there.

Within the Python environment, you then need to install pycairo and PyGObject with

(venv) user@machine: ~/vxPy_app$ pip install pycairo PyGObject

WARNING: starting with version 1.0.0, tiscamera no longer supports older camera models (see table supported devices). If you're using one of those, instead install the latest pre-1.0.0 stable release (0.14.0) by checking it out with

user@machine: ~/tiscamera$ git checkout tags/v-tiscamera-0.14.0

directly after cloning the repository and before installing the dependencies or building the binaries.

Basler cameras

Basler cameras are supported for Windows and Linux. Just download the pylon installer for your plattform from the Basler website.

Then install the respective Python pypylon package into your environment with

(venv) user@machine: ~/vxPy_app$ pip install gst PyGObject pypylon

Adding a camera

Cameras can be added via the configuration manager

(venv) user@machine: ~/vxPy_app$ vxpy -c path/to/config.yaml configure

or directly by adding them to the configuration file

CAMERA_DEVICES:
  camera01_behavior:
    api: vxpy.devices.camera.basler_pylon.BaslerCamera
    serial: 12345
    model: a2A1920-160umBAS
    width: 1936
    height: 1216
    frame_rate: 80
    basler_props:
      BinningHorizontalMode: Average
      BinningHorizontal: 1
      BinningVerticalMode: Average
      BinningVertical: 1
      GainAuto: false
      Gain: 20
      ExposureAuto: false
      ExposureTime: 10000

DAQs

Internally, vxPy uses the standard Firmata protocol for synchronization and communication with external application and devices via Arduino (and Arduino-compatible) devices.

To the user, aquisition or writing of analog and digital signals is easily accessible through the configurable IO interface by adding devices and PINs to the configuration YAML file:

IO_DEVICES:
  Dev1_microscope:
    api: vxpy.devices.arduino_daq.ArduinoDaq
    model: Arduino
    port: /dev/ttyACM1
    pins:
      y_mirror_in:
        type: analog
        direction: input
        map: a:1:i # PIN 1 analog input
      frame_sync:
        type: digital
        direction: input
        map: d:2:i # PIN 2 digital input
      frame_trigger_out:
        type: digital
        direction: output
        map: d:3:o  # PIN 3 digital output

For more advanced use-cases (e.g. when programming routines for realtime analysis in vxPy), all configured IO devices and their PINs are also directly available in the Python script.

Configured devices can either be directly accessed through their SerialDevice instance

import vxpy.core.devices.serial as vxserial

device = vxserial.get_serial_device_by_id('Dev1_microscope')
device.board.get_pin('d:3:o').write(1)

and individual, configured PINs can be written to automatically from attributes that are created in a vxPy analysis routine

import vxpy.core.io as vxio

vxio.set_digital_output('frame_trigger_out', 'frame_trigger_attribute') 

vxpy's Issues

Add end time to hdf5 attribute

@thladnik could you add a end_time attribute for each phase stored in the Display.hdf5 file please, it will be useful for aligning calcium data with the stimulus

Architecture problem for class "Stimulus"

In current Stimulus class, one stimulus only allows to have 1 base vertex/fragment shader+vertex/fragment shader, 1 program and 1 variable buffer. In many case, e.g stimuli requires stencil test, the draw method needs to use at least two programs using different/same shader and variable buffer for rendering.

A quick way to solve this issue is to change the vertex/fragment shaders and programs to a list or a dictionary (preferable for documentation and logging). The draw method is allowed to use any shader/program in the list, and the program can therefore use all variable in the shared variable buffer

Problems encountered

Format:

# Problems Index - State
* **Description**: ...
* **Solution**: ....
* **Explanation**: ...

• Problem Index: P1, P2 ...
• State: Unsolved / Solved

telemetrix and tifffile packages not included in installation via 'vxpy setup' command

If these two packages are not installed vxpy will start and shutt down.
Maybe they could be included in the setup process.

Restarting display process cause error in remote setting of visual uniforms

To reproduce the problem:

1. Do scene registration and background calculation
2. The camera/core.py will emit p_fish_position (line 485).
3. Start the visual closeloop_dotchasing under motion_rivalry. The position emitted can be checked at line 181 of ``motion_rivalry.py`
4. Restart the display and start the visual closeloop_dotchasing again, should see this error in the console

Traceback (most recent call last):
File "...\MappApp\mappapp\core\visual.py", line 491, in draw
   self.render(frame_time)
File "...\MappApp\mappapp\visuals\planar\motion_rivalry.py", line 188, in render
   target_pos = fish_pos + stalk_vec
TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'

Function request for Display window and `Stimulus` class

1. The program now cannot be terminated properly by closing the GUI window, @thladnik could you fix it?
2. Would it be possible to initiate the display process by clicking "start protocol"? So if user close the display window or the openGL crashes, you can always restart it
3. Should be possible to allow users to create more than one stencil mask right? I'm working on it anyhow

ROI activity tracker improvements

1. Set a threshold for each ROI individually
2. Save ROI activity traces in camera file
3. Color or text label for each ROI within the Ca image (so that you can match the ROI with the activity trace)