Coder Social home page Coder Social logo

tlnagy / ometiff.jl Goto Github PK

View Code? Open in Web Editor NEW
25.0 5.0 7.0 80.78 MB

I/O operations for OME-TIFF files in Julia

License: Other

Julia 100.00%
image-processing ome-tiff julia-language julia ome-files micromanager image-analysis image-analysis-workflows

ometiff.jl's Introduction

OMETIFF.jl

Stable Release Documentation Build Status

Adds support for reading OME-TIFF files to the Images.jl platform. Allows fast and easy interfacing with high-dimensional data with nice labeled axes provided by AxisArrays.jl.

Features

  • Can open a wide-range of OMETIFF files with a special focus on correctness
  • Supports memory-mapping to open large TIFF files quickly even on memory-constrained machines
  • Spatial and temporal axes are annotated with units if available (like μm, s, etc)
  • Channel and position axes use their original names
  • Elapsed times are extracted and returned using the same labeled axes
  • Important metadata is extracted and included in an easy to access format

Installation

OMETIFF.jl will be automatically installed when you use FileIO to open an OME-TIFF file. You can also install it by running the following in the Julia REPL:

] add OMETIFF

Usage

julia> using FileIO, Images

julia> img = load("/Users/tamasnagy/Downloads/66perc-h2o-vs-iso_1_MMStack.ome.tif")
Gray ImageMeta with:
  data: 4-dimensional AxisArray{Gray{N0f16},4,...} with axes:
    :y, 0.0 μm:0.6518 μm:666.7914000000001 μm
    :x, 0.0 μm:0.6518 μm:666.7914000000001 μm
    :time, 0.0 ms:15000.0 ms:405000.0 ms
    :position, Symbol[:A5_Site_0, :A5_Site_1, :B5_Site_0, :B5_Site_1]
And data, a 1024×1024×28×4 reshape(reinterpret(Gray{N0f16}, ::Array{UInt16,6}), 1024, 1024, 28, 4) with eltype Gray{Normed{UInt16,16}}
  properties:
    Elapsed_Times: Unitful.Quantity{Float64,𝐓,Unitful.FreeUnits{(s,),𝐓,nothing}}[2.525 s 3.35 s 5.638 s 6.534 s; 15.398 s 16.195 s 18.743 s 19.506 s;  ; 390.389 s 391.154 s 393.282 s 393.984 s; 405.391 s 406.13 s 408.316 s 409.101 s]
    Description: nd4 + nd8 in

julia> size(img) # lets get the dimensions
(1024, 1024, 28, 4)

julia> axisnames(img) # wait, but what do they correspond to?
(:y, :x, :time, :position)

julia> img[Axis{:position}(:A5_Site_1), Axis{:time}(2)]; # get the 2nd time point in position A5

julia> img["Elapsed_Times"][Axis{:position}(:A5_Site_1), Axis{:time}(2)] # get exact time when that slice was taken
16.195 s

julia> img["Description"] # get any notes embedded in the image
"nd4 + nd8 in"

More advanced usage

The image updates all the axes as we subset it. Observe that since we're grabbing 5x5x1x1 subset of the image, all the axes update to reflect this slice.

julia> using Unitful

julia> img[Axis{:y}(1:5), Axis{:x}(1:5), Axis{:time}(15000u"ms"), Axis{:position}(1)]
Gray ImageMeta with:
  data: 2-dimensional AxisArray{Gray{N0f16},2,...} with axes:
    :y, 0.0 μm:0.6518 μm:2.6072 μm
    :x, 0.0 μm:0.6518 μm:2.6072 μm
And data, a 5×5 Array{Gray{N0f16},2} with eltype Gray{Normed{UInt16,16}}
  properties:
    Elapsed_Times: Quantity{Float64,𝐓,Unitful.FreeUnits{(s,),𝐓,nothing}}[2.525 s 3.35 s 5.638 s 6.534 s; 15.398 s 16.195 s 18.743 s 19.506 s;  ; 390.389 s 391.154 s 393.282 s 393.984 s; 405.391 s 406.13 s 408.316 s 409.101 s]
    Description: nd4 + nd8 in

Dumping embedded OME-XML

To quickly access the OME-XML stored inside a TIFF file, use the OMETIFF.dump_omexml function:

julia> using OMETIFF

julia> println(OMETIFF.dump_omexml("test/testdata/singles/single-channel.ome.tif"))
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Creator="OME Bio-Formats 5.2.2" UUID="urn:uuid:2bc2aa39-30d2-44ee-8399-c513492dd5de" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd">
  <Image ID="Image:0" Name="single-channel.ome.tif">
    <Pixels BigEndian="true" DimensionOrder="XYZCT" ID="Pixels:0" SizeC="1" SizeT="1" SizeX="439" SizeY="167" SizeZ="1" Type="int8">
      <Channel ID="Channel:0:0" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
        <UUID FileName="single-channel.ome.tif">urn:uuid:2bc2aa39-30d2-44ee-8399-c513492dd5de</UUID>
      </TiffData>
    </Pixels>
  </Image>
</OME>

ometiff.jl's People

Contributors

github-actions[bot] avatar hsupoleng avatar johnnychen94 avatar rsrock avatar timholy avatar tkelman avatar tlnagy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

ometiff.jl's Issues

Border slice lost when loading OOM multi-file image

julia> img = FileIO.load("dish2_ctrl_v_bix_1_MMStack.ome.tif", inmemory = false)^C

julia> size(img)
(1024, 1024, 3, 361, 2)

julia> for i in 1:3, j in 1:361, k in 1:2
           try
               img[:, :, i, j, k].data
           catch
               println(i, " ", j, " ", k)
           end
       end
1 338 2

Appears as though the slice isn't added to the IFD list so it fails on access:

julia> img[:, :, 1, 338, 2]
ERROR: KeyError: key (1, 1, 338, 2) not found
Stacktrace:
  [1] getindex
    @ ~/.julia/packages/OrderedCollections/PRayh/src/ordered_dict.jl:380 [inlined]
  [2] getindex(::OMETIFF.DiskOMETaggedImage{ColorTypes.Gray{FixedPointNumbers.N0f16}, 4, 6, UInt32, Matrix{ColorTypes.Gray{FixedPointNumbers.N0f16}}}, ::Int64, ::Int64, ::Int64, ::Int64, ::Int64, ::Int64)
    @ OMETIFF ~/.julia/dev/OMETIFF/src/mmap.jl:54
  [3] _unsafe_getindex_rs
    @ ./reshapedarray.jl:250 [inlined]

Update strip read code on Julia 1.4

We won't need to change the shape of the temporary buffer on Julia 1.4 since the read! functions signature was widened in JuliaLang/julia@1dbccbd so the following code can be simplified:

OMETIFF.jl/src/loader.jl

Lines 146 to 150 in 02517da

# if the data is stripped and we haven't fix tmp's layout then lets make
# tmp equal to one strip. This'll be fixed in Julia 1.4
if n_strips > 1 && size(tmp) != (strip_len, )
tmp = Array{rawtype}(undef, strip_len)
end

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

document and add more examples

The README only has the most barebones example and really doesn't show everything that you can do once you load an annotated image. It might be best to structure this as a blog post.

reduce package size

It turns out that test images take a lot of space and make a lot of bandwidth spent when ]add OMETIFF for normal users.

$ du -sh *
 16K	docs
4.0K	LICENSE.md
4.0K	Project.toml
8.0K	README.md
 44K	src
132M	test

Maybe upload them to https://github.com/tlnagy/exampletiffs?

Tracking of IFD offset across separate files is very brittle

The current strategy for tracking IFD offsets across separate files is brittle. I need to store the offset the first time I encounter a new file and then add this offset to each IFD in that file. Currently, this causes multiposition files that are larger than 4gigs to fail silently.

This logic needs to be replaced:

# if this file isn't one we've observed before, increment the offset
if !in(filepath, obs_filepaths)
ifd = file_ifd_offset
file_ifd_offset += 1
push!(obs_filepaths, filepath)
end

Ideally, going forward obs_filepaths should also store the offset so the lookup is quick.

Broken JSON metadata blocks file loading

I need to thoroughly diagnose this, but I have multiple MicroManager images that have broken embedded JSON comments that cause the loading to come to a screeching halt at

metadata = JSON.parse(String(metadata_bytes))

Examples (for later reference) include

/mnt/eurekapool/data/2022/220720_fxm_collagen_minimalprocess/exp2_collagen_fxm_chemotaxis_1/exp2_collagen_fxm_chemotaxis_1_MMStack.ome.tif

While the longer term fix is clearly to identify why this is happening, the shorter term fix is to abort and throw a warning. This is solely for nice-to-have metadata and is fine if it can't be read.

Extract and report actual elapsed time information

The whole image TimeIncrement value often doesn't line up with the actual time of when each plane was taken. It would be helpful to report the actual DeltaT for each plane in an image, if available. Unlike with TimeIncrement, the elapsed times don't line up nicely along different dimensions so the same Z or C slice might have the same "timepoint", but will have different DeltaT values. Thus, it would probably best to report them as additional information instead of in the time axis values.

Add option for preserving dimensions of length 1

Currently, we drop all dimensions that have a length of 1:

OMETIFF.jl/src/loader.jl

Lines 150 to 152 in 4adee1a

unused_dims = Tuple(idx for (idx, key) in enumerate(keys(dims)) if dims[idx] == 1)
squeezed_data = dropdims(data; dims=unused_dims)
used_axes = [axes[i] for i in 1:length(axes) if !(i in unused_dims)]

since I originally considered them superfluous, but there could be circumstances when having that information would actually be beneficial, e.g. for saving (see #30) or for verification.

Incomplete micromanager acquisitions cause out of bounds error

Aborted micromanager acquisitions can cause BoundsErrors:

BoundsError: attempt to access 260-element Array{Union{Nothing, Tuple{String,String}},1} at index [261]

Stacktrace:
 [1] setindex! at ./array.jl:766 [inlined]
 [2] ifdindex!(::Array{Union{Nothing, NTuple{4,Int64}},1}, ::Array{Union{Nothing, Tuple{String,String}},1}, ::Set{String}, ::EzXML.Node, ::NamedTuple{(:Y, :X, :Z, :C, :T, :P),NTuple{6,Int64}}, ::String, ::Int64) at /home/tlnagy/.julia/dev/OMETIFF/src/parsing.jl:61
 [3] load(::Stream{DataFormat{:OMETIFF},IOStream}) at /home/tlnagy/.julia/dev/OMETIFF/src/loader.jl:51
 [4] (::getfield(Main, Symbol("##3#4")))(::IOStream) at ./In[6]:3
 [5] #open#312(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::getfield(Main, Symbol("##3#4")), ::String) at ./iostream.jl:375
 [6] open(::Function, ::String) at ./iostream.jl:373
 [7] top-level scope at In[6]:1

This appears to happen due to there being extra TiffData elements versus the dimensions reported by Pixels.

Suitable for general use on TIFFs?

I've been trying out putting together an ImageIO package, to collect some lower-level, hopefully faster, image IO approaches to potentially allow moving away from ImageMagick.

Would using this package be appropriate for general TIFFs?

I've currently clang-wrapped LibTIFF (and LibPNG, LibTurboJpeg) and was planning on going that route, unless you think this is well suited and should just be a dep?

I noted this has been developed for accuracy (which I think is great for TIFFs in particular), but it is also faster than say, ImageMagick? (Package and image load times?)

Multiple position images should have position as an extra axis

Currently, I return a list of images, but it would be nice to return an image with the position as another dimension.

I might have to be clever with extracting a name for each element in that dimension as many OME-TIFFs don't have correct StageLabels, etc

Error in loading some images from TestImages.jl

I have some errors when I try to load some images from package TestImages.jl (I have the same output with ImageMagick.jl). It works with TiffImages.jl.
Any idea why this happens?

julia> using TestImages

julia> using OMETIFF

julia> testimage("cameraman.tif")
Error encountered while loading "/home/afertin/.julia/artifacts/e752bdc739f34d02e79c7fa834bc2f2e0d71c7e0/cameraman.tif".

Fatal error:
ERROR: /home/afertin/.julia/artifacts/e752bdc739f34d02e79c7fa834bc2f2e0d71c7e0/cameraman.tif: Null count for "Tag 34022" (type 1, writecount -3, passcount 1). `_TIFFVSetField' @ error/tiff.c/TIFFErrors/540
Stacktrace:
 [1] handle_error(e::ErrorException, q::FileIO.File{FileIO.DataFormat{:TIFF}})
   @ FileIO ~/.julia/packages/FileIO/ZknoK/src/error_handling.jl:82
 [2] handle_exceptions(exceptions::Vector{Any}, action::String)
   @ FileIO ~/.julia/packages/FileIO/ZknoK/src/error_handling.jl:77
 [3] load(::FileIO.Formatted; options::Any)
   @ FileIO ~/.julia/packages/FileIO/ZknoK/src/loadsave.jl:186
 [4] load
   @ ~/.julia/packages/FileIO/ZknoK/src/loadsave.jl:166 [inlined]
 [5] #load#13
   @ ~/.julia/packages/FileIO/ZknoK/src/loadsave.jl:118 [inlined]
 [6] load
   @ ~/.julia/packages/FileIO/ZknoK/src/loadsave.jl:118 [inlined]
 [7] testimage(filename::String; download_only::Bool, ops::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ TestImages ~/.julia/packages/TestImages/C9XQF/src/TestImages.jl:100
 [8] testimage(filename::String)
   @ TestImages ~/.julia/packages/TestImages/C9XQF/src/TestImages.jl:92
 [9] top-level scope
   @ REPL[3]:1

Support for Big-TIFF

I find that FileIO can load Big-TIFF directly without OME-TIFF. But OMETIFF.jl throw error ERROR: Big-TIFF files aren't supported yet. Could you add the support for Big-TIFF? Thanks!
(Now, I just change .ome.tif to .btf.tiff or others, which avoid load OMETIFF.jl)

Imspectorpro ome.tif file: ERROR: KeyError: key "Type" not found

I have .ome.tif files exported from lavision biotec's imspectorpro.

using FileIO
tf = load("pathto.ome.tif")

Top part of stack-trace

Fatal error:
ERROR: KeyError: key "Type" not found
Stacktrace:
  [1] getindex(node::EzXML.Node, attr::String)
    @ EzXML ~/.julia/packages/EzXML/ZNwhK/src/node.jl:1200
  [2] load(io::Stream{DataFormat{:OMETIFF}, IOStream, String}; dropunused::Bool, inmemory::Bool)
    @ OMETIFF ~/.julia/packages/OMETIFF/xqGQs/src/loader.jl:57
  [3] #30
    @ ~/.julia/packages/OMETIFF/xqGQs/src/loader.jl:3 [inlined]
  [4] open(f::OMETIFF.var"#30#31"{Bool, Bool}, args::File{DataFormat{:OMETIFF}, String}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:330
  [5] open
    @ ./io.jl:328 [inlined]
  [6] #load#29
    @ ~/.julia/packages/OMETIFF/xqGQs/src/loader.jl:2 [inlined]
  [7] load(f::File{DataFormat{:OMETIFF}, String})
    @ OMETIFF ~/.julia/packages/OMETIFF/xqGQs/src/loader.jl:2
  [8] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
  [9] invokelatest
    @ ./essentials.jl:706 [inlined]
 [10] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})

Is there something I'm doing wrong or is this kind of tif not supported in ometiff? I can load with no problem in python tiffile.

Something up with MicroManager OMEs?

I'm debugging some code involving reading MicroManager OME files, and I think I stumbled on something weird here. If I open the test image singles/background_1_MMStack.ome.tif using this package, you'll see that it's transposed. However, other ome.tiff images are not transposed. For example, the neutrophil image is correct.

Do you know if there is something funny about MicroManager files?

[edit] Sure enough, if I add that file to the ""TIFF value verifications" test block, that test fails.

BoundsError when accessing Tag

Hi, I'm getting the following error when trying to load a .ome.tif image with coords (X,Y,C,Z,T): LoadError: BoundsError: attempt to access 0-element Vector{TiffImages.Tag} at index [1]. I am able to open this image in Fiji/ImageJ without an issues. How can I go about figuring out what is wrong? Thanks!

Profile loading code

I've never properly profiled my code to find performance bottlenecks, but it takes 13 seconds and 3.7GiB to load a 1.7GB image on my workstation, which seems excessive. I should be more aggressive on assuming internal consistency and only fallback when it fails:

OMETIFF.jl/src/loader.jl

Lines 98 to 99 in d1e0989

# TODO: This shouldn't be allocated for each ifd
tmp = Array{master_rawtype}(undef, read_dims...)

DimensionMismatch("array could not be broadcast to match destination")

Load Micromanager .ome.tif1x8x2560x2160x1(CTXYZ), but fail to match dimension. Do wrong .ome.tif tag trigger this error. More, Julia can load image to 2160x2560x8 array when I change .ome.tif to .tif to by pass OMETIFF

FYI

img_test = load("/RawData/20200102/leaking chip/MMStack_Pos0.ome.tif")
┌ Warning: ignored the empty prefix for 'http://www.openmicroscopy.org/Schemas/OME/2015-01'; expected to be non-empty
└ @ EzXML /home/hf/.julia/packages/EzXML/QtGgF/src/xpath.jl:85
Error encountered while loading "/RawData/20200102/leaking chip/MMStack_Pos0.ome.tif".

Fatal error:
DimensionMismatch("array could not be broadcast to match destination")

Stacktrace:
 [1] handle_error(::DimensionMismatch, ::File{DataFormat{:OMETIFF}}) at /home/hf/.julia/packages/FileIO/I1ONY/src/error_handling.jl:82
 [2] handle_exceptions(::Array{Any,1}, ::String) at /home/hf/.julia/packages/FileIO/I1ONY/src/error_handling.jl:77
 [3] #load#27(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(load), ::FileIO.Formatted) at /home/hf/.julia/packages/FileIO/I1ONY/src/loadsave.jl:189
 [4] load(::FileIO.Formatted) at /home/hf/.julia/packages/FileIO/I1ONY/src/loadsave.jl:169
 [5] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(load), ::String) at /home/hf/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118
 [6] load(::String) at /home/hf/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118
 [7] top-level scope at In[5]:1
println(OMETIFF.dump_omexml("/RawData/20200102/leaking chip/MMStack_Pos0.ome.tif"))
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2015-01" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2015-01 http://www.openmicroscopy.org/Schemas/OME/2015-01/ome.xsd">
  <Instrument ID="Instrument:0">
    <Detector ID="Detector:Andor sCMOS Camera" Model="Zyla 5.5 CL 3 Tap" SerialNumber="VSC-00788"/>
  </Instrument>
  <Image ID="Image:0" Name="MMStack_Pos0">
    <InstrumentRef ID="Instrument:0"/>
    <StageLabel Name="pos0" X="0.0" XUnit="µm" Y="0.0" YUnit="µm"/>
    <Pixels BigEndian="false" DimensionOrder="XYCZT" ID="Pixels:0" Interleaved="true" SizeC="1" SizeT="8" SizeX="2560" SizeY="2160" SizeZ="1" Type="uint16">
      <Channel ID="Channel:0:0" Name="" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="1" FirstZ="0" IFD="1" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="2" FirstZ="0" IFD="2" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="3" FirstZ="0" IFD="3" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="4" FirstZ="0" IFD="4" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="5" FirstZ="0" IFD="5" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="6" FirstZ="0" IFD="6" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <TiffData FirstC="0" FirstT="7" FirstZ="0" IFD="7" PlaneCount="1">
        <UUID FileName="MMStack_Pos0.ome.tif">urn:uuid:5a8594cd-4c87-4319-accb-79c766dd7b21</UUID>
      </TiffData>
      <Plane DeltaT="6.3129650892522E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="0" TheZ="0"/>
      <Plane DeltaT="6.31296604946E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="1" TheZ="0"/>
      <Plane DeltaT="6.312966129446E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="2" TheZ="0"/>
      <Plane DeltaT="6.3129661469683E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="3" TheZ="0"/>
      <Plane DeltaT="6.3129667945224E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="4" TheZ="0"/>
      <Plane DeltaT="6.312968457888E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="5" TheZ="0"/>
      <Plane DeltaT="6.3129684689447E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="6" TheZ="0"/>
      <Plane DeltaT="6.3129684846153E11" DeltaTUnit="ms" PositionX="0.0" PositionXUnit="µm" PositionY="0.0" PositionYUnit="µm" PositionZ="0.0" PositionZUnit="µm" TheC="0" TheT="7" TheZ="0"/>
    </Pixels>
  </Image>
</OME>

Add setters to memory-mapped arrays

While #51 added support for read-only disk arrays, it would be nice to also be able to write data in a similar memory-efficient way. This will likely take some coordination with #30.

Switch to new Julian iteration protocol.

Currently, I use a custom iteration protocol to iteratively load IFDs in a given file, but I should switch OMETIFF over to the new standard Julian protocol for better organization and maintainability. On each iteration, the file can either load the next IFD or return the next already loaded IFD. Additionally, I can provide the state which could simplify the temporary fix in #20 and I can eliminate the file.loc pointer.

Large micromanager stack gives ReadOnlyMemoryError

Hello, I have some large micromanager tiff stacks (50000+ images) that I would like to load as a memory-mapped stack. I get a ReadOnlyMemoryError when I run the following code.

julia> stack = OMETIFF.load(File{format"OMETIFF"}(stack_path); inmemory=false)
ERROR: LoadError: ReadOnlyMemoryError()

Additionally, when I use FileIO directly, I get

julia> stack = load(stack_path; inmemory=false)
Error encountered while load File{DataFormat{:OMETIFF}, String}("path/to/MMStack_Default.ome.tif").

Fatal error:
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] handle_error(e::ReadOnlyMemoryError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:61
 [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/error_handling.jl:56
 [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Formatted; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:228
 [4] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:185
 [5] load(::String; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:inmemory,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/DNKwN/src/loadsave.jl:113
 [6] top-level scope
   @ REPL[6]:1

I would provide an example file but it is quite large. Perhaps there is a good way to deal with this? I can help investigate this as well.

Any pointers would be much appreciated.

PS: This image stack was also from an aborted acquisition, but I saw an earlier closed issue that dealt with that so its probably not related (?).

add lazy loading of larger files/split files

Like NRRD.jl this package should really support mmapping because then it could open files much greater in size than the available memory.

This is going to require something more complex than NRRD.jl since the information will likely be split across multiple files and places inside each file. Therefore I'll need to do this lazily when actually requested because otherwise it is infeasible.

Prior art:

Re: error with opening a ome.tiff file.

Hi,

My images was acquired by a Zeiss confocal microscope and saved images as ome.tiff. When I tried to load the file in julia, I got this error:

julia> load(movingfns[10])                                                                                                                                                             
Error encountered while load FileIO.File{FileIO.DataFormat{:OMETIFF}, String}("/media/donghoon_001/imaging_dat/birc_nikon_confocal/230808_and_230725_adlib_fasted_refed/tif/230725_2_3_
3_2_refed_round3_dapi_hcrt_gal_cartpt_Stitch.ome.tiff").                                                                                                                               
                                                                                                                                                                                       
Fatal error:                                                                                                                                                                           
ERROR: ArgumentError: collection must be non-empty 
Stacktrace:                                                                                                                                                                  [194/1469]
  [1] first                                                                                                                                                                            
    @ ./abstractarray.jl:466 [inlined]                                                                                                                                                 
  [2] inmemoryarray(ifds::OrderedCollections.OrderedDict{NTuple{4, Int64}, Tuple{TiffImages.TiffFile{UInt32}, TiffImages.IFD{UInt32}}}, dims::NamedTuple{(:Y, :X, :Z, :C, :T, :P), NTup
le{6, Int64}}; verbose::Bool)                                                                                                                                                          
    @ OMETIFF ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:122                                                                                                                        
  [3] load(io::FileIO.Stream{FileIO.DataFormat{:OMETIFF}, IOStream, String}; dropunused::Bool, verbose::Bool, inmemory::Bool)                                                          
    @ OMETIFF ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:90
  [4] load                                   
    @ ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:29 [inlined]
  [5] #33                                    
    @ ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:3 [inlined]
  [6] open(f::OMETIFF.var"#33#34"{Bool, Bool}, args::FileIO.File{FileIO.DataFormat{:OMETIFF}, String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:395                       
  [7] open                                   
    @ ./io.jl:392 [inlined]                  
  [8] #load#32                               
    @ ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:2 [inlined]
  [9] load(f::FileIO.File{FileIO.DataFormat{:OMETIFF}, String})
    @ OMETIFF ~/.julia/packages/OMETIFF/3RcZb/src/loader.jl:1
 [10] #invokelatest#2
    @ ./essentials.jl:819 [inlined]                                                                                                                                          [173/1469]
 [11] invokelatest                           
    @ ./essentials.jl:816 [inlined]          
 [12] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:219
 [13] action                                 
    @ ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:196 [inlined]
 [14] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185
 [15] action                                 
    @ ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 [inlined]
 [16] load(::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:113
 [17] load(::String)                         
    @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:109
 [18] top-level scope                        
    @ REPL[266]:1                            
Stacktrace:                                  
 [1] handle_error(e::ArgumentError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/error_handling.jl:61
 [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector}}, action::String)
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/error_handling.jl:56                                                                                                          [152/1469]
 [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:228
 [4] action                                  
   @ ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:196 [inlined]
 [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185
 [6] action                                  
   @ ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 [inlined]
 [7] load(::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:113
 [8] load(::String)                          
   @ FileIO ~/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:109
 [9] top-level scope                         
   @ REPL[266]:1

Interestingly, I was able to load the same files in Fiji (but the pixel parameters were were not correct). I also checked the header file with tiffcomment command:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!--Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: http://loci.wisc.edu/ome/ome-tiff.html.-->
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2010-04" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2010-04/ome.xsd http://www.openmicroscopy.org/Schemas/OME/2010-04/ome.xsd">
  <Instrument ID="Instrument:0">
    <Detector Gain="1" Voltage="650" Zoom="1" AmplificationGain="1" ID="Detector:0:0" Type="PMT" />
    <Detector Gain="1" Voltage="650" Zoom="1" AmplificationGain="1" ID="Detector:1:0" Type="PMT" />
    <Detector Gain="1" Voltage="650" Zoom="1" AmplificationGain="1" ID="Detector:1:1" Type="PMT" />
    <Detector Gain="1" Voltage="649.981" Zoom="1" AmplificationGain="1" ID="Detector:2:0" Type="PMT" />
    <Objective ID="Objective:0" Immersion="Air" LensNA="0.45" NominalMagnification="10" />
  </Instrument>
  <Image ID="Image:1" Name="230725_2_3_2_6_fasted_round3_dapi_hcrt_gal_cartpt_Stitch.ome">
    <AcquiredDate>2023-08-30T00:43:52</AcquiredDate>
    <Description></Description>
    <InstrumentRef ID="Instrument:0" />
    <ObjectiveSettings ID="Objective:0" />
    <Pixels ID="Pixels:Uint12" DimensionOrder="XYZCT" Type="uint12" SizeX="2867" SizeY="4712" SizeZ="5" SizeT="1" SizeC="4" PhysicalSizeX="0.830266" PhysicalSizeY="0.830266" PhysicalSizeZ="6">
      <Channel ID="Channel:0" Name="Ch1-T1" PinholeSize="25.835" Fluor="DAPI" SamplesPerPixel="1">
        <DetectorSettings ID="Detector:0:0" Binning="1x1" />
      </Channel>
      <Channel ID="Channel:1" Name="Ch1-T2" PinholeSize="28.9734" Fluor="Alexa Fluor 488" SamplesPerPixel="1">
        <DetectorSettings ID="Detector:1:0" Binning="1x1" />
      </Channel>
      <Channel ID="Channel:2" Name="Ch2 GaAsP-T2" PinholeSize="28.9734" Fluor="Alexa Fluor 647" SamplesPerPixel="1">
        <DetectorSettings ID="Detector:1:1" Binning="1x1" />
      </Channel>
      <Channel ID="Channel:3" Name="Ch2 GaAsP-T3" PinholeSize="33.3092" Fluor="Alexa Fluor 546" SamplesPerPixel="1">
        <DetectorSettings ID="Detector:2:0" Binning="1x1" />
      </Channel>
      <TiffData />
      <Plane TheZ="0" TheT="0" TheC="0" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="0" TheT="0" TheC="1" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="0" TheT="0" TheC="2" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="0" TheT="0" TheC="3" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="1" TheT="0" TheC="0" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="1" TheT="0" TheC="1" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="1" TheT="0" TheC="2" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="1" TheT="0" TheC="3" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="2" TheT="0" TheC="0" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="2" TheT="0" TheC="1" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="2" TheT="0" TheC="2" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="2" TheT="0" TheC="3" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="3" TheT="0" TheC="0" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="3" TheT="0" TheC="1" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="3" TheT="0" TheC="2" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="3" TheT="0" TheC="3" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="4" TheT="0" TheC="0" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="4" TheT="0" TheC="1" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="4" TheT="0" TheC="2" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
      <Plane TheZ="4" TheT="0" TheC="3" DeltaT="0" PositionX="0" PositionY="0" PositionZ="0" />
    </Pixels>
  </Image>
</OME>

Also I have the "original" czi file format. If I open it in fiji and store it as ome.tif, Julia could load the resulting ome.tif file. While I can convert all files in Fiji, I wanted to check if these is better ways to solve this issue.

Handle imaging positions

Currently, the code only handles XYZCT information. It looks like OME-XML doesn't have natural support for multiple positions, but I'll need to handle them regardless since the extra TiffData nodes are in XML. So in order:

  1. The design I'm developing in #1 should degrade gracefully (i.e. only load the TiffDatas corresponding to the current file) in the case of multiple positions. This should be a simple check: if number of TiffDatas is longer than the product of ZCT dimensions then this is likely a multiple position file and then I would match TiffData uuids with the uuid of the current file.

  2. In the future it might be nice to have positions as another dimension. It looks like Micromanager at least seems to embed StageLabel nodes in the XML under Image that gives me a name and the order of positions so that be a way to get this to work.

Use proper mapping for OME-XML time units to Unitful time units

Currently, I directly cast the OME-XML time units to Unitful ones:

# This is an ugly hack to convert the unit string into Unitful.Unit till
# https://github.com/PainterQubits/Unitful.jl/issues/214 gets fixed
unitstr = replace(image[units], "u" => "μ")

This is really fragile and there should be a proper 1 to 1 mapping between the OME-XML time units (see https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#UnitsTime for the possible values) and https://github.com/PainterQubits/Unitful.jl/blob/be0190219e60f9eac5a5b60f1011e942a865aad9/src/pkgdefaults.jl#L246-L247

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.