bodenmillergroup / readimc

Python package for reading imaging mass cytometry (IMC) files

readimc

Python package for reading imaging mass cytometry (IMC) files

Documentation is available at https://bodenmillergroup.github.io/readimc

Citation

Please cite the following paper when using readimc in your work:

Windhager, J., Zanotelli, V.R.T., Schulz, D. et al. An end-to-end workflow for multiplexed image processing and analysis. Nat Protoc (2023). https://doi.org/10.1038/s41596-023-00881-0

@article{Windhager2023,
    author = {Windhager, Jonas and Zanotelli, Vito R.T. and Schulz, Daniel and Meyer, Lasse and Daniel, Michelle and Bodenmiller, Bernd and Eling, Nils},
    title = {An end-to-end workflow for multiplexed image processing and analysis},
    year = {2023},
    doi = {10.1038/s41596-023-00881-0},
    URL = {https://www.nature.com/articles/s41596-023-00881-0},
    journal = {Nature Protocols}
}

readimc's Issues

options for OSError: MCD file corrupted

Our group has had several instances where .mcd files are parsed and experience the following types of mcd file corruption errors:

OSError: inconsistent acquisition image data size

OSError: mcd file corrupted: invalid acquisition image data offsets

Are there options in readimc for how to handle or generalize the errors for these types of mcds? In many cases the source of the error needs to be investigated by the individual who ran the experiment, and because the MCDFile parser seems to be quite restrictive with these error warnings, these data cannot be read into our programs when these errors arise.

Error when loading IMC txt file with TXTFile

An error orrcured when I ran the fowllowing script:

with TXTFile('0419_ROI_007_7.txt') as f:
img=f.read_acquisition()

While I could successfully open this txt file in MCD Viwer.
What's the problem when I read it in Python?

Memory mapping

Allow for memory mapping as alternative to reading full images into memory

Support for reading directly from zip files

Does the readimc API for MCDFile allow for reading mcd files directly from a zipped state? Our group often compresses these files to save space, and this functionality would be great for faster data processing.

large panorama image reading

I have a mcd file with 3 panoramas. While trying to read the panoramas I get an error :

---------------------------------------------------------------------------
DecompressionBombError                    Traceback (most recent call last)
File [~/.conda/envs/tnbc/lib/python3.9/site-packages/readimc/mcd_file.py:211](https://vscode-remote+ssh-002dremote-002b132-002e72-002e65-002e230.vscode-resource.vscode-cdn.net/sise/assafzar-group/assafzar/leor/chapter_two/TNBC/imc/notebooks/~/.conda/envs/tnbc/lib/python3.9/site-packages/readimc/mcd_file.py:211), in MCDFile.read_panorama(self, panorama)
    210 try:
--> 211     return self._read_image(
    212         data_start_offset, data_end_offset - data_start_offset
    213     )
    214 except Exception as e:

File [~/.conda/envs/tnbc/lib/python3.9/site-packages/readimc/mcd_file.py:339](https://vscode-remote+ssh-002dremote-002b132-002e72-002e65-002e230.vscode-resource.vscode-cdn.net/sise/assafzar-group/assafzar/leor/chapter_two/TNBC/imc/notebooks/~/.conda/envs/tnbc/lib/python3.9/site-packages/readimc/mcd_file.py:339), in MCDFile._read_image(self, data_offset, data_size)
    338 data = self._fh.read(data_size)
--> 339 return imread(data)

File [~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/v2.py:227](https://vscode-remote+ssh-002dremote-002b132-002e72-002e65-002e230.vscode-resource.vscode-cdn.net/sise/assafzar-group/assafzar/leor/chapter_two/TNBC/imc/notebooks/~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/v2.py:227), in imread(uri, format, **kwargs)
    226 with imopen(uri, "ri", **imopen_args) as file:
--> 227     result = file.read(index=0, **kwargs)
    229 return result

File [~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py:147](https://vscode-remote+ssh-002dremote-002b132-002e72-002e65-002e230.vscode-resource.vscode-cdn.net/sise/assafzar-group/assafzar/leor/chapter_two/TNBC/imc/notebooks/~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py:147), in LegacyPlugin.read(self, index, **kwargs)
    145     return img
--> 147 reader = self.legacy_get_reader(**kwargs)
    148 return reader.get_data(index)

File [~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py:116](https://vscode-remote+ssh-002dremote-002b132-002e72-002e65-002e230.vscode-resource.vscode-cdn.net/sise/assafzar-group/assafzar/leor/chapter_two/TNBC/imc/notebooks/~/.conda/envs/tnbc/lib/python3.9/site-packages/imageio/core/legacy_plugin_wrapper.py:116), in LegacyPlugin.legacy_get_reader(self, **kwargs)
    115 self._request.get_file().seek(0)
...
    217         f"MCD file '{self.path.name}' corrupted: "
    218         f"cannot read image for panorama {panorama.id}"
    219     ) from e

OSError: MCD file 'Leap004.mcd' corrupted: cannot read image for panorama 2

This is not an expected behavior because if I use MCD Viewer I'm able to read everything an nothing is corrupted.

Any suggestion what can cause this? (unfortunately I cannot share the data itself)

Check for overlapping raw data blocks in MCD file metadata

We recently had the case that the (otherwise valid) metadata of two acquisitions pointed to the identical raw data block. readimc should check for corruptions of this type.

>>> a12.metadata["DataStartOffset"]
'776817931'
>>> a13.metadata["DataStartOffset"]
'776817931'

Slow restoring from txt files

Takes multiple minutes for a 450MB text file

Likely due to readimc's strategy to read with numpy rather than pandas (?)