neurodata / brainlit Goto Github PK
View Code? Open in Web Editor NEWMethod container for computational neuroscience on brains.
Home Page: http://brainlit.neurodata.io/
License: Other
Method container for computational neuroscience on brains.
Home Page: http://brainlit.neurodata.io/
License: Other
Neuron traces are commonly stored in the SWC format. Networkx is a popular python library for manipulating graph data structures. It will be useful to have standardized methods to read in SWCs as networkx graphs and, similarly, to export networkx graphs as SWCs.
check his demo
scripts in the mouselight_code
repo
Currently the requirements.txt
file in the root directory and the docs
branch have not been updated to the most current version of brainlit and possibly contain unnecessary packages, causing minor slowdowns.
Test:
swc.py
(already done, just add tests for using read_s3
with other methods)visualize.py
(requires adapting the code)Update notebooks for
swc.py
visualize.py
Build docs in sphinx
write a somewhat optimized method (not a for loop) which:
Currently a segmentation volume/url is required to interact with a NeuroglancerSession
object, when it should be possible to interact without out (to view image layers).
self.url_segments = None
This would be a script that gets run on the cis
server.
/cis/local/jacs/...
It would be a great QA script to instill confidence that uploads are valid.
This way to code style with match uploading_brains
Tasks:
[Done] 1. Merge segmentation algorithm pipeline and tutorial notebook.
[Done] 2. Generate framework/pipeline tutorial to display automated segmentation into Napari, to enable easier manual segmentation ("semi-augmented" segmentation). Ensure manual segmentations can be uploaded to AWS via this protocol.
[Done] 3. Write documentation (and (if useful) a manuscript - (not doing this, sticking with documentation)) describing the process and pipeline for a user to fix errors with a paint tool and save a binary segmentation mask.
[Done] 4. Ensure algorithm can be run on consecutive points and the results be combined.
Submit a pull request to this repository which does the following things:
brainlit/preprocessing/features/nonlinear_features.py
which:These features would then be able to be used with a classifier (such as brainlit/algorithms/regression/log_regression.py
) to predict axon vs not axon
Write tests for brainlit/preprocessing/features/nonlinear_features.py
and document each method.
create a jupyter notebook in the documentation which demonstrates functionality visually and serves as a tutorial for other developers to trust and use the code
Tasks for this semester: Automate process of uploading raw octree data and pulling back as a neuroglancer precomputed format to be segmented.
Create temporary s3 bucket for storing precomputed data format.
Pull both image and fragment data in an octree format(most likely from a cis computer) and upload as both neuroglancer precomputed onto the s3 bucket.
Leverage code already in mouselight_code repo (swc2neuroglancer, upload_to_neuroglancer) for this task
Pull neuroglancer data from bucket to view and segment in napari.
Final DoD is a script for processing raw swc, image data into ngl_pipeline.py for visualization
Use session for the download
df_to_graph method in brainlit/utils/swc.py
makes a graph without including metadata for structure and radius
The method needs to add node attributes in addition to x, y, and z to reconstruct a swc from the graph
The features folder computes statistics on a volume to reduce dimensionality of the data our algorithms run on.
brainlit/preprocessing/linear_features.py
converts a volume into a set of features based on different filter activations.
brainlit/preprocessing/neighborhood.py
converts a volume into a set of features based on neighborhood-based algorithms.
Submit a Pull Request to the brainlit repository which makes the following changes:
modify linear_features.py
to function on precomputed volumes using brainlit
methods.
test linear_features.py
to validate output - including jupyter notebook tutorials to verify visually.
write docstrings for each method in linear_features.py
which allow others to easily use the methods.
modify neighborhood.py
to function on precomputed volumes using brainlit
methods.
test neighborhood.py
to validate output - including jupyter notebook tutorials to verify visually.
write docstrings for each method in neighborhood.py
which allow others to easily use the methods.
After recent development, many methods are not documented properly
Add clear docstrings to
Due to changes in the core brainlit code, many tests were disabled.
Slowly working on uncommenting and moving tests out from the tests/archive
folder will improve code coverage as functionality is restored.
Additionally, many old tests were not designed as true unit tests and stand to be improved.
I suggest taking a look at a general unit testing guide and a pytest-specific guide.
The tests to be revived include everything in tests/archive/
:
test_adaptive_thresh.py
test_image_process.py
test_linear_features.py
test_log_regression.py
test_visualize.py
test_preprocess.py
test_subneighborhood.py
Code coverage can be tracked via https://codecov.io/gh/neurodata/brainlit
I imagine this will take multiple PRs
Brainlit uses sphinx-autodoc
which makes documentation for the repo directly from method docstrings.
registration
folder needs to be added to the list of subpackages hereregistration.rst
needs to be created in the same directly mimicking files like algs.rst such that
.. autoapifunction::
and .. autoapiclass::
, respectively.sphinx-autodoc
which parses file docstrings as documentation #42just uncomment the lines in the __init__.py
file and debug
centering and resolution validated
allow for creation of new channels
Make a notebook to (approximately) replicate this poster.
In particular, I want to see a comparison between a logistic classifier that only looks at voxel intensity, and an MLP that looks at a 7x7x7 neighborhood. The task is to classify class 0 (background) vs class 1 (foreground).
You can generate training data by using tube_seg on the swc points to identify examples of foreground, and you can brainstorm your own way to identify examples of background.
The final comparison should be ROC curves on some test data.
The current introductory tutorial notebook does not work on Windows OS, likely because of path issues in the dependency, CloudVolume. First, can someone else replicate this issue? Can we update CloudVolume for find a work around?
This means
Currently this issue is focused on:
brainlit/utils/ngl_pipeline.py
brainlit/utils/swc.py
brainlit/utils/upload.py
brainlit/utils/util.py
There are several previously published neuron segmentation algorithms, we will be comparing our methods to those at some point, does anyone want to get started implementing them?
Or, find an alternative way to give collaborators read access to an s3 bucket with brain data.
Or, allow the case that each collaborator should manually request the private key.
Or, expose the private key allowing anyone to pull data from our s3 bucket (probably not this one).
CloudVolume will sometimes hang randomly, interrupting large uploads.
Currently, if an error happens, we only print a short error message, and the upload must be restarted from scratch.
It would be far superior to print the mip and chunk the error was on along with the error, enabling continuation of a large, multi-day upload.
The previous data upload looked like the following, using code from the old-upload
branch.
Current progress to make this feature usable to any user is in the continue_upload
branch.
Specifically,
upload.py
upload_volumes
method on line 258upload_volumes
on lines [309-315]Adding tests and debugging still needs to be done. Tests would be added in tests/test_upload.py
, adding the continue_upload
parameter to the type checking test for upload_volumes
, as well as adding a test exclusively checking validity of continuing uploads from the middle and end indices.
from @jovo
(full disclosure not currently aware of what this means)
https://brainlight.readthedocs.io/en/latest/reference/preprocessing.html#features
you can only see the class name, but not the parameters to fit, for example.
The test files make development via docker slow.
Solution is to move test files to a separate testing branch and leave master clean (at most, just the lowest resolution image).
Follow procedure in this poster to fit splines to neurons.
While neuron traces are often stored in the .swc format, there are certain limitations to neuron structure within .swc files. A feature is needed to convert .swc files to .mat files, which store neurons as vertex and edge lists.
Additionally, the feature should allow for the merging of multiple individual neurons by creating random edges between two neurons - this will be used to evaluate the performance of G-Cut, a program from the following paper designed to segment individual neurons from neuron clusters: https://www.nature.com/articles/s41467-019-09515-0
brainlit/algorithms/regression/log_regression.py
tests/test_log_regression.py
good luck!
How to replicate:
delete .cloudvolume from the home directory and run .aws.sh
run docs/utils/downloading_brains.ipynb
And you get a SignatureDoesNotMatch
error when creating a cloudvolume object with an s3 url.
Scripts in the above folders are built on top of utils, so when utils changed some broke. This is mainly fixing imports and modifying parameter inputs.
As the repo is refactored (see the Reformatting project), pep484 type hints are being added (example here).
Typeguard allows for type checking via pep484 type hints, and using them would streamline the codebase and reduce some clutter.
Doing this instead of multiple lines of check_type(<input>, <type>)
is far cleaner.
Lines to be changed include every type-checking line after a method (for example, upload.py
lines 48-51). You can identify the line if it begins with check_
.
Scripts with type hints and checking so far include:
Implement a segmentation algorithm that takes a swc file as input. Create a binary mask of cylinders of a specified radius to connect the appropriate vertices of the swc.
Test and document these functions.
Similar to test_upload.py
, using type checking and much more thorough validation.
test_session.py
test_swc.py
Curvature and torsion are computed via https://en.wikipedia.org/wiki/Torsion_of_a_curve
The derivatives of b-splines can be computed in closed form using B-splines (see e.g. Kunoth's book)
scipy represents splines with knots and orders, make a method to compute curvature and torsion of these splines.
Basic functionality (tested with documentation)
Parallelizing the upload of the higher resolutions has failed thus far.
Offset in current upload of the data
alternatively, figure out how to give travis credentials (seems very sketch)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.