This working Repo contains our notes / utilities / info for our cloud DeepCell imaging project.
Here is the high level workflow for using DeepCell:
Note that DeepCell itself does not process TIFF files. The TIFF channels must be extracted into Numpy arrays first.
Also note that DeepCell performs its own pre- and post-processing around the TensorFlow prediction. In particular, DeepCell divides the input into 512x512 tiles which it predicts in batches, then reconstructs the overall image.
- GOAL: Understand and optimize using DeepCell to perform cellular image segmentation on GCP at scale.
- KEY LINK #1: our benchmarking process.
- KEY LINK #2: our support/testing notebooks.
- KEY LINK #3: our project board & work areas for this project.
GPU makes a dramatic difference in model inference time.
Memory usage increases linearly with number of pixels.
Here are some areas we've identified:
- Preprocessing
- DeepCell converts everything to 64bit float. That's memory intensive. Do we actually need to?
- Prediction
- Benchmark results (GPU, batch size, resolution)
- Investigate Google TensorFlow optimizations
- Postprocessing
- h_maxima: need to ship a ~15x speedup optimization
Nothing special. You just need Python 3.10 at the latest.
python3.10 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Some incantations are needed to work on Apple silicon computers. You also need Python 3.9.
DeepCell depends on tensorflow
, not tensorflow-macos
. Unfortunately we need tensorflow-macos
specifically to provide TF2.8 on arm64 chips.
The solution is to install the packages one at a time so that the DeepCell failure doesn't impact the other packages.
python3.9 -m venv venv
source venv/bin/activate
pip install -r requirements-mac-arm64.txt
cat requirements.txt | xargs -n 1 pip install
# Let it fail to install DeepCell, then:
pip install -r requirements.txt --no-deps
# Lastly install our own library. Note --no-deps
pip install --editable . --no-deps
I think but am not sure that the first --no-deps
invocation is unnecessary as pip install installs dependencies.