metacell / minian

miniscope analysis pipeline with interactive visualizations

About MiniAn

MiniAn is an analysis pipeline and visualization tool inspired by both CaImAn and MIN1PIPE package specifically for Miniscope data.

Quick Start Guide

Option #1: conda

1. Download and install conda.
2. Make sure your conda is up to date: conda update -n base -c default conda
3. Download the MiniAn package: git clone https://github.com/DeniseCaiLab/minian.git
4. Go into MiniAn folder you just cloned: cd minian/
5. Create an conda environment: conda env create -n minian -f environment.yml
6. Activate the conda enviornment you created during minian installation: source activate minian
7. Fire up jupyter: jupyter notebook and open the notebook "pipeline_noted.ipynb"

Option #2: docker

1. Download and install docker.
2. Read the notes below first then run: docker run -p 8888:8888 -v MY_DATA_PATH:/media denisecailab/minian:latest
3. Once you read the following, open a browser and navigate to localhost:8888:

The Jupyter Notebook is running at:
http://blahblah:8888
Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Note: this approach runs everything in a more isolated docker container, which makes customizing/extending the code harder and may affect performance. It is recommended to use this approach only when you encounter errors during installation or running demo data with the first conda option, in which case it is highly recommended to submit an issue first as well.

As a result of isolation, you have to explicitly expose resources from within the container to your machine. In our case we want to expose two things: the network port that jupyter is listening to and a file system shared across your machine and the container. The -p 8888:8888 argument in the docker command expose port 8888 from within the container to port 8888 on your machine. If you prefer another port, for example in case another jupyter application is already running and occupies port 8888, change the number after the colon. The -v MY_DATA_PATH:/media argument creates a bind mount from MY_DATA_PATH on your machine to /media within the container, so that files under this two paths are synced and act as one, and you can refer to anything under MY_DATA_PATH by substituting that with /media from the notebook within the container. See here for more details and change the path to suit your needs.

As a final note, everything within the container does not persist across sessions except those in the bind mount, which means all the modifications you made to the default notebook under /minian will be reset to the state in the original image. Thus it is advised to keep a working copy of minian on your local machine and mount them in the container if you are using this in production mode.

License

This project is licensed under GNU GPLv3.

minian's Issues

Lock versions of dependencies

With a requirements file (or similar)

Update parameter dicts rather than reassigning values

One of the first cells sets default parameters, which are dicts. As a user runs through the pipeline, they will typically change parameters by reassigning the value to the one inside the dict and key that they want. For example, param_first_spatial['sparse_penal'] = 0.05. There is a risk of human error that isn't currently caught by the pipeline. For example, if I make a typo: param_first_spatial['sparse_pena'] = 0.05, and then run update_spatial(required_args, sparse_penal=param_first_spatial['sparse_penal'], I will end up accidentally using the default.

There should be a way to catch invalid dict keys. One possibility is to use **param in each function so that the function will throw an error when it encounters an invalid argument name in the dict key that's being passed. Another is to use classes rather than dicts, using __slots__: https://docs.python.org/2.5/ref/slots.html

draft wiki/FAQ

Create PRs for issues completed issues in the last sprint

Define documentation improvements

Unit testing for critical path Pt.4

Based on test plan

Improve documentation at the function level (docstrings)

Simplify visualization code blocks and improve code comments on the notebooks

Setup shared drive folder for dataset

Verify package dependencies (dask version is important)

This issue seems to be related to having an incompatible version of dask.

Estimate importing cell reg from matlab algo and port it to Python (with no dependency on matlab version)

Link --> https://github.com/zivlab/CellReg

Test shared data sets Pt. 2

Add configuration flag for interactive / programmatic notebook

Failed to load frame during glow removal step

Details here.
Data that produces this error under the folder pp2_TraumaEnd in the Google drive.

Improve visualization of motion correction

The current visualization in the pipeline is not very helpful to validate if motion correction worked

Fix black errors in travis build

Currently the travis build would fail, and it seems black is behind this. See logs here: https://travis-ci.org/github/DeniseCaiLab/minian/builds/697624646

Can we ask black to spit out something like a "format quality score" instead of complaining and wanting to re-format all the scripts?

Optimize memory usage / profiling investigation

Some parts of the pipeline consume a lot of RAM (memory) causing the Jupyter notebook kernel to crash.

for example this step:
visualization
generate_videos(minian, varr, dpath, param_save_minian['fname'] + ".mp4", scale='auto')

Define adoption strategy

Unit testing for critical path Pt.6

Based on test plan

Unit testing for critical path Pt.5

Based on test plan

Review / try out test data sets Pt.1

Continue adding unit tests for critical path

Based on test plan

Add Continuous Integration (Travis) for master, development and feature branches on PRs

Hanging during visualizations, possibly due to dask/holoviews interaction

See this issue.

Anecdotally from my experience, this has resolved the issue on Linux, but not on Windows. Further testing needed.

The current hacky fix is this, which works for all cases from what I can tell, but a patch has not been rolled out containing that fix, leaving most users to implement it themselves.

rework datastructure for seeds

currently seeds is a pandas dataframe and result of refinement steps are stored as boolean mask columns. This doesn't play well if the users run refinement steps in a non-linear order.

Freeze environment.yml

People are still running into environment issues like here. We need to lock package versions in the environment.yml.

Bug introduced in cell registration due to incompatibilities / missed refactoring

See issue here.

cross registration ValueError

step 1
...
id_dims = {'animal','session'}
...

step 4
...
minian_ds = open_minian_mf(
dpath, id_dims, pattern=f_pattern, backend='zarr')
...

ValueError                                Traceback (most recent call last)
<ipython-input-4-810a24a5dde9> in <module>
----> 1 minian_ds = open_minian_mf(
      2     dpath, id_dims, pattern=f_pattern, backend='zarr')

/opt/projects/metacell/minian/minian/minian/utilities.py in open_minian_mf(dpath, index_dims, result_format, pattern, sub_dirs, exclude, **kwargs)
    711             print(["{}: {}".format(d, v) for d, v in zip(index_dims, key)])
    712     if result_format is 'xarray':
--> 713         return xrconcat_recursive(minian_dict, index_dims)
    714     elif result_format is 'pandas':
    715         minian_df = pd.Series(minian_dict).rename('minian')

/opt/projects/metacell/minian/minian/minian/utilities.py in xrconcat_recursive(var, dims)
    767             pass
    768         var_ps = pd.Series(var_dict)
--> 769         var_ps.index.set_names(dims, inplace=True)
    770         xr_ls = []
    771         for idx, v in var_ps.groupby(level=dims[0]):

~/miniconda3/envs/minian/lib/python3.8/site-packages/pandas/core/indexes/base.py in set_names(self, names, level, inplace)
   1323         else:
   1324             idx = self._shallow_copy()
-> 1325         idx._set_names(names, level=level)
   1326         if not inplace:
   1327             return idx

~/miniconda3/envs/minian/lib/python3.8/site-packages/pandas/core/indexes/base.py in _set_names(self, values, level)
   1237             raise ValueError("Names must be a list-like")
   1238         if len(values) != 1:
-> 1239             raise ValueError(f"Length of new names must be 1, got {len(values)}")
   1240 
   1241         # GH 20527

ValueError: Length of new names must be 1, got 2

Error regarding generating videos in dev branch

See issue here.

Add comments to the pipeline to make it more maintainable / readable to developers

Define visualization improvements

draft api documentations

Test/fix co-registration on pp8

Unit testing for critical path Pt.3

Based on test plan

batch-processing support

there use to be a notebook to carry out batch-processing using papermill. The two biggest question is:

1. how should be parameters be encoded in a json-friendly way?
2. what features are important for this other than a big for loop?

better handling of metadata

Currently the metadata are recorded as coordinates of xarray and pulled from folder names with the meta_dict. This is often overlooked by users. The result is that people often ended up with same/messed up coordinates with session, which in turn will mess up cross-session registration. A few thoughts on how to make this better:

1. write a function to quickly fix the metadata and resave all the datasets (I have a half-working one already somewhere in the code).
2. find alternative way to record metadata and inform the cross-session registration process.
3. forget about metadata and carry out cross-session registration on unlabeled arrays (register across certain axis).

Draft test plan

Define robustness improvements

Library code walkthrough(s)

Can we improve the notebook duplication?

Right now we have 2 notebook versions, one is noted (documentation) and the other has just the code

Generate read the docs or similar from docstrings

Performance improvements EPIC

progressbar not defined in cross-registration notebook

step 3

NameError                                 Traceback (most recent call last)
<ipython-input-3-cb61fb1036bb> in <module>
     17 from minian.visualization import AlignViewer
     18 hv.notebook_extension('bokeh', width=100)
---> 19 pbar = ProgressBar(minimum=2)
     20 pbar.register()

NameError: name 'ProgressBar' is not defined

Unit testing for critical path Pt.2

Based on test plan

strange usage boolean use_spatial

see:

minian/minian/cnmf.py

Line 499 in f3501c8

if use_spatial == 'full':

at this line use_spatial is tested against a string ( == 'full') however use_spatial is a boolean

check what the correct usage/implementation needs to be

cross registration SyntaxWarning messages

step 3 (load modules) of the cross registration notebook produces SyntaxWarning messages:

/opt/projects/metacell/minian/minian/minian/utilities.py:667: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if backend is 'netcdf':
/opt/projects/metacell/minian/minian/minian/utilities.py:669: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if chunks is 'auto':
/opt/projects/metacell/minian/minian/minian/utilities.py:679: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif backend is 'zarr':
/opt/projects/metacell/minian/minian/minian/utilities.py:683: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if chunks is 'auto':
/opt/projects/metacell/minian/minian/minian/utilities.py:712: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if result_format is 'xarray':
/opt/projects/metacell/minian/minian/minian/utilities.py:714: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif result_format is 'pandas':
/opt/projects/metacell/minian/minian/minian/utilities.py:730: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if backend is 'netcdf':
/opt/projects/metacell/minian/minian/minian/utilities.py:737: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif backend is 'zarr':

replace replace the if a is b: syntax with if a == b:

Define code quality improvements / identify refactorings

Share test data sets

Publish minian to pypi

Hex issue from original repo