bodenmillergroup / imcsegmentationpipeline

A pixel classification based multiplexed image segmentation pipeline

A flexible multiplexed image segmentation pipeline based on pixel classification

Introduction

The pipeline is based on CellProfiler (tested v4.2.1) for segmentation and Ilastik (tested v1.3.3post3) for pixel classification. It is streamlined by using the imcsegpipe python package available via this repository as well as custom CellProfiler modules (ImcPluginsCP, release v4.2.1).

This repository showcases the basis of the workflow with step-by-step instructions. As an alternative and dockerized version of the pipeline, check out steinbock.

This pipeline was developed in the Bodenmiller laboratory at the University of Zurich (www.bodenmillerlab.com) to segment hundreds of highly multiplexed imaging mass cytometry (IMC) images. The concepts applied here to IMC data can also be transfered to data generated by other highly multiplexed imaging modalities.

For a general overview on IMC as technology and data processing tasks, please refer to bodenmillergroup.github.io/IMCWorkflow.

Usage

For the main part of the analysis, you will need to install Ilastik and CellProfiler.

Before being able to pre-process the data, you will need to setup the environment:

1. Install conda

2. Clone the repository:

git clone --recursive https://github.com/BodenmillerGroup/ImcSegmentationPipeline.git

3. Setup the conda environment:

cd ImcSegmentationPipeline
conda env create -f environment.yml

4. Configure CellProfiler to use the plugins by opening the CellProfiler GUI, selecting Preferences and setting the CellProfiler plugins directory to path/to/ImcSegmentationPipeline/resources/ImcPluginsCP/plugins and restart CellProfiler. Alternatively you can clone the ImcPluginsCP repository individually and set the path correctly in CellProfiler.

5. Activate the environment created in 3. and start a jupyter instance

conda activate imcsegpipe
jupyter lab

This will automatically open a jupyter instance at http://localhost:8888/lab in your browser. From there, you can open the scripts/imc_preprocessing.ipynb notebook and start the data pre-processing.

In brief, the main analysis steps include:

1. Pre-processing of the raw images to create .ome.tiffs and .tiff stacks for ilastik training and measurement (python).
2. Ilastik pixel classification based on random crops of the images (CellProfiler, Ilastik).
3. Image segmentation based on the classification probabilities (CellProfiler).
4. Measurement and export of cell-specific features, such as marker expression (CellProfiler).

Example data

To test these pipelines on example data, please run the scripts/download_examples.ipynb script.

Documentation

For a more detailed overview on the individual analysis steps, please visit https://bodenmillergroup.github.io/ImcSegmentationPipeline/.

This pipeline was presented at the 2019 Imaging Mass Cytometry User Group Meeting. The slides can be downloaded here. The slides briefly explain why we chose this approach to image segmentation and provide help to run the pipeline.

Changelog

For changes in specific releases, please refer to the CHANGELOG.

License

We freely share this pipeline in the hope that it will be useful for others to perform high quality image segmentation and serve as a basis to develop more complicated open source IMC image processing workflows. In return we would like you to be considerate and give us and others feedback if you find a bug/issue and raise a GitHub Issue on the affected projects or on this page.

Contributing

To contribute to this work, please fork the repository, make changes to it and open a pull request.

Contributors

Creator: Vito Zanotelli

Contributor: Jonas Windhager, Nils Eling, Milad Adibi

Maintainer: Milad Adibi

Citation

Please cite the following paper when using this workflow in your research:

Windhager, J., Zanotelli, V.R.T., Schulz, D. et al. An end-to-end workflow for multiplexed image processing and analysis. Nat Protoc (2023). https://doi.org/10.1038/s41596-023-00881-0

@article{Windhager2023,
    author = {Windhager, Jonas and Zanotelli, Vito R.T. and Schulz, Daniel and Meyer, Lasse and Daniel, Michelle and Bodenmiller, Bernd and Eling, Nils},
    title = {An end-to-end workflow for multiplexed image processing and analysis},
    year = {2023},
    doi = {10.1038/s41596-023-00881-0},
    URL = {https://www.nature.com/articles/s41596-023-00881-0},
    journal = {Nature Protocols}
}

imcsegmentationpipeline's Issues

Histocat Inquiry

Hello,

Is it possible to input the created .tiff files in the "Histocat" folder into Histocat to view various channels, and export the files as .tiff to later put into Ilastik to make classifying easier? I am trying to classify images in Ilastik based on several different channels and I am currently using Histocat as a guide, however a quicker method would be to directly use the images in Histocat into Ilastik and later skip the "batch processing" step.
Is this feasible? Thank you

Issue Generating the analysis stacks

I am running the pipeline, and appears an error generating the analysis stacks.

ERROR:root:Error in 20200924_Disease_s0_p5_r1_a1_ac.ome.tiff
Traceback (most recent call last):
File "pandas_libs\parsers.pyx", line 1119, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas_libs\parsers.pyx", line 1244, in pandas._libs.parsers.TextReader._convert_with_dtype
File "pandas_libs\parsers.pyx", line 1259, in pandas._libs.parsers.TextReader._string_convert
File "pandas_libs\parsers.pyx", line 1450, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 1: invalid continuation byte

I Imagine that maybe it is some issue with the csv file.
My .zip file consist in 1 .mcd file and 4.txt files. All 4 samples where measured during the same experiment, but each of them has a panel slightly different.

Thanks,
Aleix

Looking for 201805_cp_segmentation_example imc_full_sm.tiff

Hi I am looking for the imc_full_sm.tiff which is the spill over matrix in a TIFF format. I would like to use your spillover matrix to correct spill over in my data set.

example data

In [6] says: "## This will download the example data - remove if you work with your own data!"

I inquired when it would download and how ... but it appears that it was successful in downloading a few hours later (3). Is this the typical turn-around time for using python to download a file?

Add a `How to adapt workflow to new data` section

I think that this is quite crucial for new users.

Citation

Hi, thanks for putting this out there.
What would be the most appropriate way to cite the workflow in publications?

How to adapt the pipeline for 'tissue level' maks

From issue #40:
"I found in your type II diabetes manuscript, you have did another segmentation to mask the blood vessel and islets in ilastik, and get the distance from cells to the rim. How to achieve this ? can you just briefly explain the steps?"

Issue converting mcd containing folders into imc zip folders

Hello,

I've been getting the following error when running the step to convert mcd containing folders into imc zip folders.

ERROR:root:Error in ../example_data/SP11_1305A4.zip
Traceback (most recent call last):
File "", line 11, in
File "/opt/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/scripts/convertfolder2imcfolder.py", line 35, in convert_folder2imcfolder
assert(len(mcd_files) == 1)
AssertionError

Would you recommend any way to troubleshoot this in Jupiter notebook, or is this an issue related to the zipped files itself?

Really appreciate your help,
Cassandra

Problem Generating Analysis Stack Tiffs

Hello,

I've recently been re-running the pipeline and have come across an error with the step "Generate the analysis stacks" to generate the initial tiff folder images. They appear to be only cropped sections of the full (square) image. I've included a screenshot of the tiff folder preview and an opened image file:


I have run the pipeline from the beginning several times to confirm, but this is reoccurring, either affecting all the tiffs or only a portion of them.
I have tried to continue with the pipeline, but when I get to Ilastik it is clear that I cannot properly capture the cells due to the small section.

I'd really appreciate any help in troubleshooting this error!
Thanks,
Cassandra

CellProfiler "Load Pipeline" Error

Hello! I am experiencing an error while importing the 2_Segment_Ilastik.cppipe file. The error message reads:
" Error while loading RescaleObjects: Could not find the RescaleObjects module. Do you want to stop processing "

Could this be an issue with CellProfiler?

The ordinal 191 could not be located in the dynamic link library imctools/Library/bin/mkl_intel_thread.dll

Hey @votti ,

I was up and running, training the segmentation on the 500x500 crops, exported my probabilities, was trying to convert them into uncertainties when the jupyter notebook stopped working for me,

Im getting the The ordinal 191 could not be located in the dynamic link library imctools/Library/bin/mkl_intel_thread.dll error in the title, when trying to run the first cell in the .ipynb

I cloned the new commit after you added logging, and tried a clean install of anaconda but I keep getting the same errors,

Thoughts?

Im in Wondows10 using Anaconda2 64-bit

ImportError: DLL load failed: The specified module could not be found.

I have been attempting to get the pipeline to run on the example data as we wish to use this pipeline for our own data. However, numerous attempts to run the script has resulted in the error indicated in the title of this post.

Working on 64-bit Windows PC.
Have downloaded all the programs, downloaded the .zip file containing all of the repositories and unpackaged the zip and placed the folder here: C:\2019_IMCWorkshop

I am unsure of how to proceed and appreciate any feedback and help.

Script: Rename Input folders -> Configurations

To make it more obivious.

NaNs not allowed with masked objects

Dear Votti,
I have been using your CP3 compensation module with enthusiasm these past weeks. To my amazement, the module recently produced an error complaining that no NaNs or Infs were allowed in the array or something in that vein. I believe this may be due to my passing masked objects to the module, some of which may be completely masked. Would you be good enough to update your otherwise superb module to account for this case? I would be grateful.
Raza

Images of type float must be between -1 and 1

When I use the V2, Cellprofiler: 1_prepare_ilastik,
an error message pops up “Images of type float must be between -1 and 1” and this happens with only some of my images

`probability2uncertainty` script `cannot compress LZW`

The probability2uncertainty script does not work with the current tifffile version, as this requires imagecodecs.

This was solved by installing setting the numpy version to 1.16.4 (conda install numpy=1.16.4) and installing imagecodecs-lite (pip install imagecodecs-lite).

Thus these versions should be added/fixed in the conda environment.

(reported by @toobiwankenobi )

Wrong command to start notebook

Currently it says conda jupyter notebook in the documentation which is wrong and should be jupyter notebook

Error loading zipped mcd/txt file

I was able to run the pipeline without any major issues with the example data. However, I am getting an assertion error. The folder contains files from each run "filename.mcd" and "filename_R01_001_1.txt" in a zipped file titled filename.zip. I zipped using the compress function on my mac. We don't have multiple regions (just one per mcd). I am not sure why I keep getting this error:

ERROR:root:Error in ../data/1155NEC.zip
Traceback (most recent call last):
File "", line 11, in
File "/Users/Bunmi/.conda/envs/imctools/lib/python3.6/site-packages/imctools/scripts/convertfolder2imcfolder.py", line 45, in convert_folder2imcfolder
assert len(mcd_files) == 1
AssertionError

Thanks,

ilastik project

hi,
because I have now had the issue myself, that if the data used for ilastik training changed directory, or the training itself moved to another directory, the project is not usable any more if it was setup with absolute paths. Since it appears to me that absolute paths are the default input for ilastik I recommend that we add a section to the desription and recommend storing "relative paths". then one can simply always copy data and ilastik project to other locations and they will still work.

histocat doesn't open output histocat file

Hi,

I run your pipeline and complete the segmentation part. Everything works like charm. Until I try to visualize the output of this function ome2micat.omefolder2micatfolder with histocat. I am not too sure if the problem is histocat or there is something missing in the /output/histocat.
A quick look at the folder I try to open, it contains all the tiff file of different channel listed in acquisition.csv file. ~metal channel + antibody clone. tiff and a _mask.tiff with the same name as the parent folder name.
After I opened it with Load Sample, directed to the directory with histcat, nothing happened at all.
Thanks

Computation of cell morphological features

Hi,

I am currently using the ImcSegmentationPipeline on the test data provided and I just realised that the last step of the pipeline does not export/produce by default classical cell morphological features such as the size of the cell, the eccentricity, granularity etc.. Is there currently anyway to easily get such features with the pipeline ?

Thank you in advance for your help :)

Best

Pierre

Nothing happens when opening 'ImcSegmentationPipeling/scripts/imc_preprocessig.ipynb'

I'm having an issue with launching the pipeline. I was able to open it last week and I don't think anything has changed yet when I try to open 'ImcSegmentationPipeling/scripts/imc_preprocessig.ipynb' nothing happens. Do you have suggestions?

'No objects to concatenate' error when generating a csv with all the acquisition metadata

Hi all!
I am getting a 'No objects to concatenate' error when trying to use the pipeline. I have already checked the zipped .mcd-files and the panel-.csv-file multiple times and also the directories but I just cannot resolve the issue...
I would really appreciate the help! Thank you very much!

Below I have attached the respective screenshots. Following is the error text:

Generate a csv with all the acquisition metadata

The csv file can be found in the data/cpout folder

exportacquisitioncsv.export_acquisition_csv(folder_ome, fol_out=folder_cp)

---

ValueError Traceback (most recent call last)
in
----> 1 exportacquisitioncsv.export_acquisition_csv(folder_ome, fol_out=folder_cp)

~\anaconda3\envs\imctools\lib\site-packages\imctools\scripts\exportacquisitioncsv.py in export_acquisition_csv(fol_ome, fol_out, outname)
34 if outname is None:
35 outname=AC_META
---> 36 dat_meta = read_acmeta(fol_ome)
37 dat_meta.to_csv(os.path.join(fol_out, outname+SUF_CSV), index=False)
38

~\anaconda3\envs\imctools\lib\site-packages\imctools\scripts\exportacquisitioncsv.py in read_acmeta(fol_ome)
28
29 def read_acmeta(fol_ome):
---> 30 dat_acmeta = _read_and_concat(fol_ome, SUFFIX_ACMETA, COL_ACID)
31 return dat_acmeta
32

~\anaconda3\envs\imctools\lib\site-packages\imctools\scripts\exportacquisitioncsv.py in _read_and_concat(fol_ome, suffix, idname)
21 ac_names = os.listdir(fol_ome)
22 dat = pd.concat([pd.read_csv(os.path.join(fol_ome, a, a+suffix)) for a in ac_names],
---> 23 keys=ac_names, names=[COL_ACSESSION, VAL_TEMP])
24 dat = dat.reset_index(VAL_TEMP, drop=True)
25 dat = dat.reset_index()

~\anaconda3\envs\imctools\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
282 verify_integrity=verify_integrity,
283 copy=copy,
--> 284 sort=sort,
285 )
286

~\anaconda3\envs\imctools\lib\site-packages\pandas\core\reshape\concat.py in init(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
329
330 if len(objs) == 0:
--> 331 raise ValueError("No objects to concatenate")
332
333 if keys is None:

ValueError: No objects to concatenate

zipfile error

The pipeline works wonderfully with the example data, but when I put my own data it cannot unzip the folder containing the mcd file and the text files. It gives the following error :
zipfile.BadZipFile: zipfiles that span multiple disks are not supported
is there a way to solve this? I used the standard windows zipping tool to create my zip file. My zip files are around 9-11GB.
Many thanks.

Pipeline not compatible with CellProfiler V4

When I try to open the first CellProfiler pipeline, I get the following error:

Error while loading Smooth Multichannel: Could not find the Smooth Multichannel module
Do you want to stop processing?

I have changed the plugin folder path and restarted CellProfiler, but it still doesn't work.

How to generate .txt files form acquisition of .mcd files?

Index Error in 3_measure_mask.cppipe

Encounter an error like this:
Error while processing MeasureObjectNeighbors Multichannel: index 11 is out of bounds for axis 2 with size 11

seems like something wrong with the markers? cause I have only selected 11 markers in pannel.csv
Any advices to solve this problems might be aprreciate.
Thank you

basic question: how to add imctools to conda using a PC

I am using conda from Anaconda3 installation and want to add imctools to my environments. I tried "conda install -c bioconda imctools" but recieved a number of python =2.7 conflicts. Sorry for the novice question but can someone help point me in the proper direction to get this accomplished?

Error converting mcd containing folders into imc zip folders

I am trying to run the pipeline with an .mcd file that is 19.7GB in size on Windows 10, version 1803. The processor is intel(R) Xenon(R) Silver 4112 CPU @2.60GHz 2.59. The operating type is a 64-bit operating system x64-based processor. the installed RAM is 48.0GB with 46.6GB usable. There is 1.17TB of free storage on the data drive.

when trying to execute the "convert zipped IMC acquisitions to input format" step I get the following error:

ERROR:root:Error in D:/cHL_PreProcessing/input/TMA_A1_1.zip Traceback (most recent call last): File "<timed exec>", line 11, in <module> File "C:\Users\GamboaA1\Documents\imctools\lib\site-packages\imctools\scripts\convertfolder2imcfolder.py", line 31, in convert_folder2imcfolder zipf.extractall(tmpdir.name) File "C:\Users\GamboaA1\Documents\imctools\lib\zipfile.py", line 1501, in extractall self._extract_member(zipinfo, path, pwd) File "C:\Users\GamboaA1\Documents\imctools\lib\zipfile.py", line 1556, in _extract_member shutil.copyfileobj(source, target) File "C:\Users\GamboaA1\Documents\imctools\lib\shutil.py", line 82, in copyfileobj fdst.write(buf) OSError: [Errno 28] No space left on device
I have no issues running the provided example data set and also was able to run a smaller recently acquired mcd file (124.7 MB).

1TB of free space seems like it should be enough to handle the file, so I would like to know if you have any idea as to the memory RAM requirements for very large MCD files.

Error in generating the histocat folder with masks

Dear Votti,

I'm having trouble with the step generating histocat folder with masks using the example data. The following error is returned.

NotADirectoryError Traceback (most recent call last)
in

/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/scripts/ome2micat.py in omefolder2micatfolder(fol_ome, outfolder, fol_masks, mask_suffix, dtype)
58 mask_suffix = '_mask.tiff'
59
---> 60 ome_files = [fn for fn in os.listdir(fol_ome) if fn.endswith('.ome.tiff')]
61 if fol_masks is not None:
62 fn_masks = [fn for fn in os.listdir(fol_masks) if fn.endswith(mask_suffix)]

NotADirectoryError: [Errno 20] Not a directory: '../output/ometiff/.DS_Store'

---

While files are created within the histocat folder in output, they all appear to be empty black images. I really appreciate your help!

Question about H5 output in the imcSegmentation Pipeline

Hello.

So the instructions below mention that the input to ilastik should be h5s (B.2). However I am not sure how to output h5s from cell profiler. There is a setting in cellProfiler under "output settings", where I can save the entire project as H5 or in matlab output format. However the instructions (B2 below) indicate that the images output from CellProfiler should all be in h5 format.
After using the ImcSegmentationPipeline Jupyter notebook the folders are created and the ome.tiffs are converted correctly. After inputing all the tiffs under the folder titled 'tiffs', the output generated in the folder 'ilastik', are only tiff files. there are no h5s generated using the instructions from part A. should there be h5s generated?
The modules I am using in CellProfiler are Smooth multichannel, Resize, and Crop bb. Should I also be using the CellProfiler module 'ExportCellToH5'?

Thank you very much.

A) Cellprofiler: 1_prepare_ilastik
In this module we prepare the data for Ilastik pixel classification, by first removing strong outlier pixels,
then scaling the images 2x and then taking random 500x500 crops to do the train the pixel classifier.
The following parts of this module need to be adapted:

1. File list: choose all files in the tiff subfolder
2. Default Output Folder: Choose the ilastik subfolder
   No further parts need to be adapted. In our 16 core computer this step takes ca 5 min for the example dataset.
   B) Ilatik: Train a pixel classifier
   This uses the random crops generated in the last step.
3. Make a new pixel classification project. Save the project file in the ilastik subfolder.
4. Add the .h5 random crops: Raw data -> Add Seperate Images -> Select all .h5 images in the ilastik subfolder.

Dimesion mismatch between 'masks' & 'image'-tiffs after 2_segment_ilastik

Dear Votti,

I have an issue at the end of the '2_segment_ilastik'-step of the pipeline, which I just cannot seem to get around....

The step itself works fine and the segmentation seems to be successful, however the generated masks are being mostly saved as 250x250pixel -tiff files, while my image-tiff-files in the histocat-folder from earlier are usually larger (e.g. 607x453 in one ROI).

This mismatch in dimensions seems to be causing an error when I try using histoCat or ImaCytE afterwards. Both programs fail to load the tiff-files and give following error instead:

C:\Program Files\BerndBodenmillerGroup\histoCAT\application>Samples arranged..Loading Masks,Tiffs of all Samples..
Warning: File: C:\Users\Maxim\AppData\Local\Temp\Maxim\mcrCache8.4\histoC0\histoCAT\Loading_New\Load_Functions\Load_mask.m Line: 162 Column: 5
This try-catch syntax will continue to work in R2007b, but may be illegal or may mean something different in future releases of MATLAB.
See MATLAB R2007a Release Notes, "Warning Generated by try-catch" for details.

In Load_MatrixDB at 29
In Master_LoadSamples at 41
In gui_mainfcn at 95
In histoCAT at 42
In @(hObject,eventdata)histoCAT('Master_LoadSamples',hObject,eventdata,guidata(hObject))
Error using iptassert (line 19)
Size of I doesn't match size information found in the first input argument.

Error in regionprops>ParseInputs (line 1168)

Error in regionprops (line 154)

Error in Process_SingleCell_Tiff_Mask>@(chan)struct2array(regionprops(Current_Mask,chandat{chan},'MeanIntensity'))' (line 80)

Error in Process_SingleCell_Tiff_Mask (line 81)

Error in DataProcessing_Master (line 29)

Error in Master_LoadSamples (line 50)

Error in gui_mainfcn (line 95)

Error in histoCAT (line 42)

Error in @(hObject,eventdata)histoCAT('Master_LoadSamples',hObject,eventdata,guidata(hObject))

Error while evaluating Menu Callback

I was following your guide for segmentation and have tried playing around with the settings as well, but I cannot get around this error and histoCat refuses to work.
Do you know how to fix this issue?

It would be splendid! Thank You for the help beforehand!
Best, Max

Memory issue in CellProfiler when running Measure_mask pipeline

Hi Vito,

I am having some issues segmenting an image due to memory, so I was wondering if you had seen this in the past and could suggest any solution.

I am having the following error when doing the last step of the segmentation pipeline (Measure_mask_basic):
"Error while processing MeasureObjectNeighbors:
(Worker) MemoryError: "

Sizes for the inputs are:
Full stack: 680MB
Probabilities: 226MB
Mask: 35MB

The system I'm using has 620GB of available physical memory.

Thanks!

Best,
Cristina

comments on readme and example run through

Hi,

1. When I run the imc_preprocessing ipynb, it all works fine on the example dataset. I had thought that this dataset included the Ilastik training, but it transpires it doesn't. However, there is no message telling me that, instead it suggests at the end that it has finished. Reading through, I am now going through the manual stages of the cppipe pipelines and Ilastik classification.

2. It would be worth adding a note that CP needs the plugins folder from ImCPluginsCP adding as the plugins folder in CP-Preferences, as it pops up an error for finding the smotthing script if you don't do this before loading the 1st pipeline.

3. Another suggestion would be a note to say that the imc_preprocessing file will run on any data that are in the input folder if the example data download section ([6]) is commented out, and that for ease, users should place their mcd zip files in there.

Jupyter notebook won't launch

I was at the workshop on Oct 2nd and was not able to launch Jupyter on my laptop. I tried again back in the office today- I have installed the IMCtools, and choose that as the applications for launching Jupyter Notebook 6.0.1. Nothing happens when I click Launch. When I launch using base (root), it opens a new window in my browser as expected, but not sure why it doesn't work with using the IMCtools. Has anyone else had this issue? I am using a Mac.

Issue running 1_prepare_ilastik_edit

Hi,

I ran your 1_prepare_ilastik project in the cell profiler, but the output did not preserve all the channels I inputted. I inputted a 3-stack tiff into this workflow, but only the 1st stack is processed and preserved in the output. I did not follow your steps for converting IMC output into a tiff, I just have a tiff with 3 channels that I wish to perform pixel classification. I'm not exactly sure how my tiff might be formatted differently from yours, but I'd greatly appreciate it if you could provide any insight into how this might be happening. Thanks a lot for making this pipeline and all the related scripts, I really look forward to using them!

Best regards,
Michelle

Ilastik error 'failed to generate export files'

Hi there,

With the example data I get an error saying ilastik failed to generate export files,
would anyone be able to help me solve this issue

kind regards
Rachel

Error with load zip file

I can't load the zip file (each zip have a mcd file, not txt files) successfully.

The error I encountered was:
ERROR:root:Error in /Users/fhe/Downloads/seg_test/zip/1.mcd.zip
Traceback (most recent call last):
  File "<timed exec>", line 12, in <module>
  File "/Users/f/opt/anaconda3/lib/python3.7/site-packages/imctools/scripts/convertfolder2imcfolder.py", line 45, in convert_folder2imcfolder
    assert len(mcd_files) == 1
AssertionError
ERROR:root:Error in /Users/fhe/Downloads/seg_test/zip/2.mcd.zip
Traceback (most recent call last):
  File "<timed exec>", line 12, in <module>
  File "/Users/f/opt/anaconda3/lib/python3.7/site-packages/imctools/scripts/convertfolder2imcfolder.py", line 45, in convert_folder2imcfolder
    assert len(mcd_files) == 1
AssertionError

Any advices might be appreciated, Thank you

Jupyter notebook error: no module named 'imctools'

Hi everyone,
I am working on a windows machine and I am having an error right at the start of the imc_preprocessing jupyter notebook.

I downloaded the segmentation pipeline folder from here:
https://github.com/BodenmillerGroup/ImcSegmentationPipeline

Then I extracted the folder & started Anaconda Navigator GUI. To install the imctools environment I did the following:
Environments -> Import -> selected the 'conda_imctools.yml' file from the downloaded folder.
The environment imports successfully.
Then in the anaconda prompt I did:
activate imctools
jupyter notebook

Navigated to the imc_preprocessing jupyter notebook. And in the first part of the code I receive the following error "ModuleNotFoundError: No module named 'imctools'".

If anyone can help me out trying to troubleshoot this error it would be greatly appreciated!
Thanks,
Steve

Better error handling

Currently, the error capturing is implemented to catch all errors and notify the user that an error happened, but the actual error is not displayed, which makes it difficult to debug user errors.

Adding a proper error logging mechanism would be helpful.

Error in generating ome.tiff from mcd

Hi,
Recently I got an error here. seems that something wrong in decoding mcd xml file? Any idea to fix this?

ERROR INFOMATION:

../zip/20200814I90R2h_Jd.zip
ERROR:root:Error in ../zip//20200814I90R2h_Jd.zip
Traceback (most recent call last):
  File "<timed exec>", line 12, in <module>
  File "/opt/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/scripts/convertfolder2imcfolder.py", line 52, in convert_folder2imcfolder
    mcd = McdParser(mcd_files[0])
  File "opt/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/io/mcdparser.py", line 34, in __init__
    McdParserBase.__init__(self, filename, filehandle, metafilename)
  File "opt/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/io/mcdparserbase.py", line 49, in __init__
    self.retrieve_mcd_xml()
  File "opt/anaconda3/envs/imctools/lib/python3.6/site-packages/imctools/io/mcdparser.py", line 97, in retrieve_mcd_xml
    xml = mm[xml_start:xml_stop].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 1452: invalid start byte

Thank you,
Funan

Add output folder to save filtered and compensated tiff files

Hi Vito,
here's an issue to track what we talked about.
It would be good to write out filtered and compensated .tiff files into the tiffs folder as default step in the pipeline.
Let's chat more about this at some point.

no session JSON files available.

Hi there,
I've tried running a test dataset through the ipynb script, and when I get to the part where you generate a csv I get the error no session JSON files available, would anyone be able to help me solve this problem? Is it something to do with where I've put my data files?

Github does not render ipynb

I noticed that Github does sometimes fail to render the ipython notebook.
Sorry, something went wrong. Reload?
Somethings hitting reload helps, sometimes not. Does anybody know a solution for this one?

A workaround is using the `Jupyter notebook viewer instead:
https://nbviewer.jupyter.org/github/BodenmillerGroup/ImcSegmentationPipeline/blob/development/scripts/imc_preprocessing.ipynb

How

Add a well trained example ilastik classifier to the example data

So far there is none and it would really help people to understand the concept/run the example, if there was one.

Moving the configurations in a configuration file

Currently, the configurations for the pipeline are handled at the beginning of a jupyter notebook. In the spirit of also allowing the pipeline to be reproducibly run from the command line, using a configuration file would be more appropriate.

Conda doesnt install imctools in Mac

Using the environment file found in setup conda does not install imctools when run in mac (without giving any warning).

The workaround was to install it manually by opening a console from the imctools environment and running:

pip install git+https://github.com/BodenmillerGroup/imctools.git@develop-test

Error Processing imctools in Conda

Hi guys,

When trying to load the IMCTools environment in Conda the following error message appears without loading the environment. Any suggestions would be greatly appreciated! Thank you for your help.

ScannerError(None, None, 'mapping values are not allowed here', <ruamel_yaml.error.StringMark object at 0x000001F8FA0782A0>)

Running everything fully automated

Question by email:

I was also wondering if there is any way to code everything on a notebook rather than being back and forward with Cell Profiler and Ilastik. I am sure there is and I was wondering if you have already tried to do so and, if yes, if you can share it 😊

In-house we have a development setup where we can directly orchestrate everything from a jupyter notebook and even deploy it on our cloud infrastructure, essentially by wrapping around the command line interfaces of cellprofiler/ilastik.

Here some pointers how this can be done:
CP: https://github.com/CellProfiler/CellProfiler/wiki/Adapting-CellProfiler-to-a-LIMS-environment#cmd
Ilastik: https://www.ilastik.org/documentation/basics/headless.html
Wrapping command lines interfaces from jupyter notebooks: http://mmcdan.github.io/posts/interacting-with-the-shell-via-jupyter-notebook/
Gc3pie: https://gc3pie.readthedocs.io/en/master/ (a package to orchestrate batch execution of command line programs on clusters)

However such implementations are quite tailored to the local infrastructure, operating system, how programs are installed etc, so I am a bit hesitant to share it before we do not have it well streamlined and documented.
I also used to have such sections in the tutorial but based on feedback it seemed to confuse more than actually help :/

Also: While it is possible to make a rough, completely automatized segmentation just based on nuclear and total intensity stain alone, whenever you want leverage pixel classification, a manual step jumping to ilastik is not really avoidable except if very standardized pannels/samples are used such that the training can allways be reused.