Coder Social home page Coder Social logo

insightsoftwareconsortium / itkioomezarrngff Goto Github PK

View Code? Open in Web Editor NEW
9.0 5.0 4.0 130 KB

ITK IO for images stored in OME-Zarr format.

License: Apache License 2.0

CMake 10.04% C++ 83.83% Python 6.13%
imaging insight-toolkit itk ome-zarr ome-zarr-converter

itkioomezarrngff's Introduction

ITKIOOMEZarrNGFF

Build Status

PyPI Version

License

Overview

This is an ITK external module for IO of images stored in Zarr-backed OME-NGFF file format.

Installation

The itk-ioomezarrngff Python package is available on the Python Package Index.

> python -m pip install itk-ioomezarrngff

Example Usage

C++

Usage from C++ should not require any special action. In special situations, you might need to invoke:

itk::OMEZarrNGFFImageIOFactory::RegisterOneFactory();

Python

In Python, we need to explicitly specify the IO, otherwise DICOM IO will be invoked because it is the built-in default for directories. Example:

import sys
import itk
imageio = itk.OMEZarrNGFFImageIO.New()
image = itk.imread(sys.argv[1], imageio=imageio)
itk.imwrite(image, sys.argv[2], imageio=imageio, compression=False)

Build Instructions

ITKIOOMEZarrNGFF is an ITK C++ external module. It may be built with CMake and build tools such as Ninja, gcc, or MSVC.

In the future ITKIOOMEZarrNGFF may be made available as an ITK remote module for direct inclusion in the ITK build process.

Prerequisites

Building

ITKIOOMEZarrNGFF uses CMake for its build process.

# Create the build directory
> mkdir path/to/ITKIOOMEZarrNGFF-build
> cd path/to/ITKIOOMEZarrNGFF-build

# Configure the project
path/to/ITKIOOMEZarrNGFF-build > cmake -DITK_DIR:PATH="path/to/ITK-build" "path/to/ITKIOOMEZarrNGFF"

# Build the project
path/to/ITKIOOMEZarrNGFF-build > cmake --build . --config "Release"

Testing

ITKIOOMEZarrNGFF tests may be run with CTest:

path/to/ITKIOOMEZarrNGFF-build > ctest -C "Release"

Wrapping

See the ITK Software Guide for information on wrapping ITK external modules for Python.

Additional Notes

ITKIOOMEZarrNGFF depends on a fork of Google's Tensorstore library for Zarr interoperation. The InsightSoftwareConsortium/Tensorstore fork implements additional zip support, both for filesystem and in memory zip reading and writing.


Acknowledgements

ITKIOOMEZarrNGFF was developed in part with support from:

itkioomezarrngff's People

Contributors

dzenanz avatar jhlegarreta avatar tbirdso avatar thewtex avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

itkioomezarrngff's Issues

seg fault instantiating image io

I started with a fresh ubuntu 22.04 system and installed python 3.10.12 and then ran pip install itk-ioomezarrngff but I get a seg fault when I try to use it.

$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itk
>>> iio = itk.OMEZarrNGFFImageIO.New()
Segmentation fault

I also tried in google colab, and there it crashes the notebook kernel:

https://colab.research.google.com/drive/11B3OUETuDJ_HmFlYqOc8LBMLPpKScTwN?usp=sharing

How to use ITKIOOMEZarrNGFF in C++?

Can you please elaborate a little bit more on how to use ITKIOOMEZarrNGFF in C++? I see, https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/blob/fe5bc3caecfe774e64acabfd75e9d74e22eacfb5/README.md says:

Usage from C++ should not require any special action. In special situations, you might need to invoke:

itk::OMEZarrNGFFImageIOFactory::RegisterOneFactory();

Looks fine, but then, what steps are there to take to get it compiled? Is the idea for the client application to use something like ExternalProject_Add(ITKIOOMEZarrNGFF ...) in their CMakeLists.txt?

Is the intention to (eventually?) allow ITKIOOMEZarrNGFF to be selected as an optional module when CMake-configuring ITK?

Update Python wrapping

I ran into: itkOMEZarrNGFFImageIO: warning(4): ITK type not wrapped, or currently not known: itk::OMEZarrNGFFAxis

Investigate thread safety

From #44 (comment):

When I run tests in parallel with ctest -j8, I get:

  9 - IOOMEZarrNGFF_readSubregion (Subprocess aborted)

Serially, they pass.

I have been able to reproduce these errors with the -j<n> flag with different tests failing. In particular I've noticed errors where Tensorstore appears to fail to read different image dimensions. There may be an issue with store contexts not being unique among threads or processes.

Tasks

  • Add a test to verify that using two OMEZarrNGFFImageIO instances with two ImageReaders of different dimensions in sequence does not result in an error + metadata is read correctly
  • Investigate failures in running tests in parallel
  • Add a test to verify that running multiple threads with one OMEZarrNGFFImageIO instance and ImageReader per thread with different data sources / regions does not result in failures
  • If thread/process/context safety issues are identified, start by documenting the issue ("multiple instances are not allowed") or seeing if we can enforce one context at a time.
  • If safety issues are identified, revisit and determine how/whether we can refactor for thread/process/context safety.

Document how to use via Python packages

Current behavior:

>>> image = itk.imread(r"C:\Dev\ITKIOOMEZarrNGFF\v0.4\cyx.ome.zarr")
WARNING: In C:\P\IPP\ITK-source\ITK\Modules\IO\GDCM\src\itkGDCMSeriesFileNames.cxx, line 100
GDCMSeriesFileNames (000002D3727E1340): No Series were found

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Dzenan\miniconda3\envs\deep_learning\lib\site-packages\itk\support\extras.py", line 1251, in imread
    raise FileNotFoundError(f"no DICOMs in: {filename}.")
FileNotFoundError: no DICOMs in: C:\Dev\ITKIOOMEZarrNGFF\v0.4\cyx.ome.zarr.

should have been fixed by InsightSoftwareConsortium/ITK@55bdee9 which made it into v5.3.0.

Fix Python build

dzenan@corista:~/ITKIOOMEZarrNGFF-rwd$ ninja
[2/2] Generating .pyi files for Python wrapper interface
FAILED: Wrapping/CMakeFiles/itk-stub-files /home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk/_proxies.pyi /home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk/__init__.pyi /home/dzenan/ITKIOOMEZarrNGFF-rwd/Wrapping/CMakeFiles/itk-stub-files 
cd /home/dzenan/ITKIOOMEZarrNGFF-rwd/Wrapping && /usr/bin/python3.10 /home/dzenan/ITK-git/Wrapping/Generators/Python/itk/pyi_generator.py --pyi_dir /home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk --pkl_dir /home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk-pkl --index_list_file /home/dzenan/ITKIOOMEZarrNGFF-rwd/Wrapping/GlobalIdxFilesList.txt
WARNING: index file  is missing,  Python stub hints will not be generated for this file. 
Exception: No index files were found in directory '/home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk-pkl'
Traceback (most recent call last):
  File "/home/dzenan/ITK-git/Wrapping/Generators/Python/itk/pyi_generator.py", line 569, in <module>
    raise Exception(except_comment)
Exception: No index files were found in directory '/home/dzenan/ITK-git-rel/Wrapping/Generators/Python/itk-pkl'
ninja: build stopped: subcommand failed.
dzenan@corista:~/ITKIOOMEZarrNGFF-rwd$

This seems like a duplicate of InsightSoftwareConsortium/ITK#3595.

Dependency simplification

ENH: Support different axis orders

Current Behavior

After #43 ITKIOOMEZarrNGFF will support reading from OME-Zarr stores where spatial axes appear in the order z,y,x. For instance, the following are supported:

  • y,x
  • z,y,x
  • t,y,x
  • t,c,z,y,x
  • t,z,y,x,c
    etc.

OME-Zarr and ITK axis access conventions are reversed. A 3D ITK image represents data with x,y,z axes, where "x" is the fastest moving and "z" is the slowest moving image axis. A 3D OME-Zarr store with axes "z", "y", "x" equivalently represents data where "z" is the slowest moving image axis and 'x" is the fastest moving image axis.

EDIT: Note that the order of spatial axes is not explicitly enforced. In theory images with out-of-order spatial axes can be read out of order. However, spatial metadata in the resulting ITK image may similarly be out of order.

Requested Behavior

Allow the user to specify the order in which OME-Zarr axes should permute to map to ITK axes. Handle spatial metadata (direction, origin, spacing) accordingly.

For instance:

imageio = itk.OMEZarrNGFFImageIO.New()
imageio.SetNamedAxes(['x','y','z']) # equivalent to ITK default
# read, etc...

or:

imageio = itk.OMEZarrNGFFImageIO.New()
imageio.SetNamedAxes(['x','z','y'])
# read, etc...

NamedAxes should default to the expected ITK spatial axes, i.e. [x,y,z].

Enhancement 1

Allow the user to query what named axes are available and reconfigure accordingly.

image_reader.SetImageIO(imageio)
image_reader.SetFileName(filename)
image_reader.UpdateOutputInformation()
print(imageio.GetNamedAxes())    # ['x','t','angle',etc]

Enhancement 2

Consider mapping arbitrary axes to ITK dimensions:

imageio.SetNamedAxes(['t','angle','z'])
imageio.SetNamedIndex('x', 5)
imageio.SetNamedIndex('y',10)
# read, etc...

Metadata may get tricky under this general approach.

Support axis units

Current behavior

Image metadata is always read in regardless of axis unit metadata. For instance, a 3D image with spacing of 1.0 millimeters along each axis and another 3D image with spacing of 1.0 micrometers along each axis are both read in with spacing "[1.0,1.0,1.0]".

Requested behavior

My understanding is that ITK image metadata is "assumed" to have units in millimeters for historical reasons. We should parse the units attached to each physical axis and convert the resulting ITK image to have spacing in "millimeters" as standard practice. For instance, the image described above with "micrometer" spacing should be read in with spacing "[0.001, 0.001, 0.001]".

Additional Notes

The ngff-zarr CLI tool accepts unit inputs for OME-Zarr generation and can be used for testing.

Add support for vector/RGB/RGBA images

OME-Zarr channel axis should be represented using ITK's vectors, with the usual exception of 3-channel images using RGBPixel and 4-channel ones using RGBAPixel as component type. This will necessitate changing memory layout if the zarr store is not already in an order where channel is fastest-changing index.

Couldn't opn a zarr example

First off, thanks for doing this awesome work ๐Ÿ‘

But I did run into some trouble trying to load some data that does work when I use the ome_zarr package.

There's a gist code that works with the same data source here.

But when I use the itk.OMEZarrNGFFImageIO I get this error:

>>> zarrEndpoint = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4007801.zarr"
>>> image = itk.imread(zarrEndpoint, imageio=imageio)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/sr/python-install/lib/python3.9/site-packages/itk/support/extras.py", line 1314, in imread
    reader = template_reader_type.New(**kwargs)
  File "/opt/sr/python-install/lib/python3.9/site-packages/itk/support/template_class.py", line 661, in New
    return self._NewImageReader(
  File "/opt/sr/python-install/lib/python3.9/site-packages/itk/support/template_class.py", line 165, in _NewImageReader
    imageIO.ReadImageInformation()
RuntimeError: /Users/runner/work/ITKIOOMEZarrNGFF/ITKIOOMEZarrNGFF/src/itkOMEZarrNGFFImageIO.cxx:383:
ITK ERROR: OMEZarrNGFFImageIO(0x7fc84001ae60): tensorstore error: FAILED_PRECONDITION: Error opening "zarr" driver: Error reading https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/4007801.zarr/0/.zarray: Error parsing object member "fill_value": Expected integer in the range [0, 65535], but received: "0" [source locations='tensorstore-src/tensorstore/internal/json_binding/json_binding.h:859,tensorstore-src/tensorstore/driver/kvs_backed_chunk_driver.cc:1055,tensorstore-src/tensorstore/kvstore/kvstore.cc:252,tensorstore-src/tensorstore/driver/driver.cc:112'] [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"http_request_concurrency\":{},\"http_request_retries\":{}},\"driver\":\"zarr\",\"kvstore\":{\"base_url\":\"https://uk1s3.embassy.ebi.ac.uk\",\"driver\":\"http\",\"path\":\"/idr/zarr/v0.1/4007801.zarr/0/\"},\"recheck_cached_data\":false,\"recheck_cached_metadata\":false}']

CMake warning due to unescaped description

a7fe056 changed readme format. Now I get a CMake warning when running locally, complaining about syntax from the readme.md document:

CMake Warning (dev) at C:/Dev/ITK-git/CMake/ITKModuleMacros.cmake:98 (message):
  Unknown argument [

  ``

  In Python, we need to explicitly specify the IO, otherwise DICOM IO will be
  invoked because it is the built-in default for directories.  Example:
...

(It is hard to put the whole output here as it gets formatted by GitHub)

[Bug] Pip install fails

Hi,

With a conda environment with python 3.9 and itk 5.3.0 installing through pip fails. Specifically, this error gets thrown:

ERROR: Could not find a version that satisfies the requirement itk-ioomezarrngff (from versions: none)ERROR: No matching distribution found for itk-ioomezarrngff

When trying to install from source it gives:

ร— python setup.py egg_info did not run successfully.
  โ”‚ exit code: 1
  โ•ฐโ”€> [8 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info
      writing /tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-x5670qbg/itk_ioomezarrngff.egg-info/SOURCES.txt'
      error: package directory 'itk' does not exist
      [end of output]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.