marksgraham / oct-converter Goto Github PK

Hello,
thanks for this nice repo ! I'm facing some troubles reading an E2E file. I followed the instructions by doing :

from  file_io.e2e import E2E 
filepath = "pathToImage/test_file.E2E"
file = E2E(filepath)
oct_volumes = file.read_oct_volume() 

An error is raised line 148 in e2e.py though, because one of the volume_string key is not part of volume_array_dict.

Here is the full error :

KeyError                 Traceback (most recent call last)
<ipython-input-20-3b5e0ad42b68> in <module>
----> 1 oct_volumes = file.read_oct_volume()

~/3rdparty/OCT-Converter/file_io/e2e.py in read_oct_volume(self)
    146                         raw_volume = list(map(self.read_custom_float, all_bits))
    147                         image = np.array(raw_volume).reshape(image_data.width, image_data.height)
--> 148                         image = 256 * pow(image, 1.0 / 2.4)
    149                         volume_string = '{}_{}_{}'.format(chunk.patient_id, chunk.study_id, chunk.series_id)
    150                         volume_array_dict[volume_string][int(chunk.slice_id / 2) - 1] = image

KeyError: '2031_3756_24070'

I'm not super familiar with E2E format, but I can share you the test image (it's too big to pass it via the present issue, even zipped).

Best reagrds

Could I please ask what is the "IMG_OBS" in the fds.py? I am trying to access the fundus image from the topcon fda file.

Hi, Mark,
This is a really useful tool for imaging research. I am wondering if you could add the conversion from FDS to DCM in the future.

Hi,

Thank you for the great tool! We are using it to convert all the formats however we are noticing that when a dicom file contains only a single image, the tool splits it into many single pixel slices. Do you know how we can prevent this or determine the if dicom is only a single image?

Thank you!

For numpy 1.24 and above, the error module 'numpy' has no attribute 'int' is thrown for oct_volume.peek() function.

The reason according to this SO answer is that numpy has removed the aliases np.int and np.float as of 1.24.

I can confirm the same by installing numpy ver 1.23 and below (which work fine) and 1.24.1 (which gives the error mentioned above).

Possible easy fix is to add a requirement for a numpy version lower than 1.24 i guess.

Documenting the roadmap here, feel free to pick up any issues or suggest more:

Easy first issues

• Add type hints throughout. This reader has an example. Types should also be removed from all the docstrings.
• Move construct definitions out of reader classes and into their own subfolder (I think @Dbrown411 is already working on this)
• The fda reader has a lot more functionality implemented than the fds reader, such as support for more metadata and for reading contours. These should be added to the fds reader, too.

Harder

• Improve the .e2e reader. Have had recent reports of users unable to read .e2e, (e.g. see here and here). I think the reader could be improved by learning from this C++ based reader, LibOCT.
• Improve .e2e reading of acquisition date/birthday acquisition. The code for acquisition date just uses a heuristic at the moment, and the birthdate is plain wrong, but we now have example code for how they should be properly extracted here.
• Add testing. I've wanted to do this a while, but I'm a bit stumped. While we could unit test some of the codebase with synthetic data, a lot of the functionality would ideally be tested on real data (i.e. checking a change to a reader does not change the output). However, most data I have is private, so we need a way of running tests on data whilst ensuring that data cannot be accessed by anyone but me. Alternatively, we need public data from more manufactuers - currently we only have for .fda and .fds. Any ideas welcome!
• Add proper documentation. The codebase is simple but I think we have enough non-technical users that would appreciate some proper documentation hosted on read the docs or similar.
• Dicom export for OCT and fundus
• Add a Factory for creating file readers.

Hi marksgraham, I encountered an error when trying to open a .oct image, can you give me some clue? thank you!

Below is the error message:

from oct_converter.readers import BOCT
fn = r'../raw-test/oct-raw/{8968AB72-5A10-46CD-99D4-50CF5F5F8974}.oct'
oct = BOCT(fn)
oct.read_oct_volume()

UnicodeDecodeError Traceback (most recent call last)
Cell In [21], line 1
----> 1 oct.read_oct_volume()

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/oct_converter/readers/boct.py:170, in BOCT.read_oct_volume(self, diskbuffered)
167 self.patient_id = self.filepath.stem
169 # Lazily parse the file without loading frame pixels
--> 170 oct = self.file_structure.parse_file(self.filepath)
171 header = oct.header
172 self.frames = FrameGenerator(oct.data)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:309, in Construct.parse_file(self, filename, **contextkw)
305 r"""
306 Parse a closed binary file. See parse().
307 """
308 with open(filename, 'rb') as f:
--> 309 return self.parse_stream(f, **contextkw)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:300, in Construct.parse_stream(self, stream, **contextkw)
298 context._params = context
299 try:
--> 300 return self._parsereport(stream, context, "(parsing)")
301 except CancelParsing:
302 pass

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2120, in Struct._parse(self, stream, context, path)
2118 for sc in self.subcons:
2119 try:
-> 2120 subobj = sc._parsereport(stream, context, path)
2121 if sc.name:
2122 obj[sc.name] = subobj

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2653, in Renamed._parse(self, stream, context, path)
2651 def _parse(self, stream, context, path):
2652 path += " -> %s" % (self.name,)
-> 2653 return self.subcon._parsereport(stream, context, path)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2120, in Struct._parse(self, stream, context, path)
2118 for sc in self.subcons:
2119 try:
-> 2120 subobj = sc._parsereport(stream, context, path)
2121 if sc.name:
2122 obj[sc.name] = subobj

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2653, in Renamed._parse(self, stream, context, path)
2651 def _parse(self, stream, context, path):
2652 path += " -> %s" % (self.name,)
-> 2653 return self.subcon._parsereport(stream, context, path)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2120, in Struct._parse(self, stream, context, path)
2118 for sc in self.subcons:
2119 try:
-> 2120 subobj = sc._parsereport(stream, context, path)
2121 if sc.name:
2122 obj[sc.name] = subobj

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:2653, in Renamed._parse(self, stream, context, path)
2651 def _parse(self, stream, context, path):
2652 path += " -> %s" % (self.name,)
-> 2653 return self.subcon._parsereport(stream, context, path)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
311 def _parsereport(self, stream, context, path):
--> 312 obj = self._parse(stream, context, path)
313 if self.parsed is not None:
314 self.parsed(obj, context)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:704, in Adapter._parse(self, stream, context, path)
702 def _parse(self, stream, context, path):
703 obj = self.subcon._parsereport(stream, context, path)
--> 704 return self._decode(obj, context, path)

File ~/miniconda3/envs/unet/lib/python3.9/site-packages/construct/core.py:1610, in StringEncoded._decode(self, obj, context, path)
1609 def _decode(self, obj, context, path):
-> 1610 return obj.decode(self.encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 66: invalid continuation byte

Upon trying to open an E2E file, I get the following error:

'ascii' codec can't decode byte 0xff in position 8: ordinal not in range(128)

What is the most likely cause of this? Thanks!

Hi Mark

Hope you're doing well.
I've been using your code to open fda files from Topcon's OCT, but lately I'm having problems to open them as I usually did.
When I use the fda.read_oct_volume() I have the following issue: "ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject"

I really cannot see what is the problem cause the code is the same as always was but now it doesn't work ...
If I use the fda.read_oct_volume_2 I can open the file but only extracts 9 slices from the whole cube instead of 128 or 256...

By he way, I have some code pending to pull request to you. If you remember, we talk several times before. I have some code to extract from fda the eye, date of capture, and patient data (the latter if present in the file). Since 2016, patient data is not present in the file so I have to extract them from a file that is created at the time of the scan which is named "FILELIST".

Hope you can help me with that issue,
Thanks

Antonio

Hi
I have some folders of pdb, sdb, and edb files which are outputs of the Heidelberg system. Each folder is related to a patient and may contain different types of retinal images such as fundus or fluorescein angiography or OCT.
I want to convert these files to images.
I wonder if your code can do this for me?

Discussed in #97

I am trying to run examples/demo_e2e_extraction.py but it fails on the import with following below.

Can you please guide me how to run examples? My python version is 3.6.10.

$ python examples/demo_e2e_extraction.py

Traceback (most recent call last):
  File "demo_e2e_extraction.py", line 1, in <module>
    from oct_converter.readers import E2E
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/oct_converter/readers/__init__.py", line 5, in <module>
    from .fda import FDA
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/oct_converter/readers/fda.py", line 4, in <module>
    from pylibjpeg import decode
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/pylibjpeg/__init__.py", line 29, in <module>
    import pydicom
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/pydicom/__init__.py", line 32, in <module>
    from pydicom.dataelem import DataElement
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/pydicom/dataelem.py", line 18, in <module>
    from pydicom import config  # don't import datetime_conversion directly
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/pydicom/config.py", line 238, in <module>
    import pydicom.pixel_data_handlers.pylibjpeg_handler as pylibjpeg_handler  # noqa
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/pydicom/pixel_data_handlers/pylibjpeg_handler.py", line 66, in <module>
    import openjpeg
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/openjpeg/__init__.py", line 3, in <module>
    from ._config import DICOM_ENCODERS, DICOM_DECODERS
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/openjpeg/_config.py", line 2, in <module>
    from openjpeg.utils import decode_pixel_data
  File "/home/karakays/.pyenv/versions/3.6.10/lib/python3.6/site-packages/openjpeg/utils.py", line 6, in <module>
    import _openjpeg
  File "openjpeg/_openjpeg.pyx", line 1, in init _openjpeg
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

Hi Mark,
I am trying to modify your fds code for reading my .fda file as below:
.fda file
I can see the '@IMG_FUNDUS' from the chunkdict and I think it should contain the fundus image, however, I got the error "cannot reshape array of size 954595 into shape (3,2048,1536)", could you please help to get the fundus image from the .fda file, thank you very much!

Hi Mark,

Thanks a lot for the library. I will be using it a lot for my Ph D. Thesis

I am trying to read UKBiobank file in fda format. I got the following error:
"No decoders are available error"

I nhave debuf-gged the code
in function read_oct_volume (self) there is a function called
slice = decode (raw_slice) which calls from pylibjpeg
in the decode function decoders = get_getcoders() returns empty set. I have installed all the files.

Am I missing something?
Thanks again for your great support for the library
Best Yasemin

Hi Mark,

Great work on this project. Let me preface this by saying I don't have an issue with the OCT-Converter project, I am currently working on a project to extract some patient data from FDA files and I'm having a hard time finding the structure of the data.

I am currently trying to extract some patient data (name, eye side etc) from FDA files. The data seems to be in a chunk with the tag PATIENT_INFO_03, the uocte page (https://bitbucket.org/uocte/uocte/wiki/Topcon%20File%20Format) doesn't have any documentation on this chunk (only PATIENT_INFO_02).

I have some FDA files and some exported data (using Topcon's OCTDataCollector.exe) and doing a brute force search doesn't yield any matches either. I feel the data is encrypted but I can't be too sure.

My reason for posting here is that I'm hoping you might have come across this and know something about it.

Jabez

Hi，
I use the FDA format file to export png images, and then found the width and height are not the same as the one exported with OCT software。
Where are the width and height of the exported image format set?
Or does the source file have original information about width and height?

Looking forward to your help， thanks.

Hi everyone,

First of all, thanks @marksgraham for taking the time to build this amazing OCT-converter.

My question is related to the acquisition date for .e2e OCTs. Neither your OCT-converter nor the original website from the uocte project seems to provide a hint to obtain it: https://bitbucket.org/uocte/uocte/wiki/Heidelberg%20File%20Format

Is there anyone who has been able to extract the acquisition date for the .e2e format? Or at least has a hint at where it could be?

Thanks

Awesome work,
Could you please handle the following issue too:
It seems it misses one Bscan for conversion. For example, an E2E file with 31 b-scans is converted to 30 images.

Hi Mark!

First of all, congrats for developing code to read .fda files. You made a great job!
I previously used your tool to extract .fds files to png but now it would be even more helpful to me to work directly with .fda (as it is the default file for storage in my hospital) without intermediate transformations.
I have downloaded your code but I'm having some problems to read the images. Applying your example codes to one of my .fda files it outputs the following error:

"""""
2
----> 3 from oct_converter.readers import FDA
4
5 # a sample .fda file can be downloaded from the Biobank resource here:

ImportError: cannot import name 'FDA' from 'oct_converter.readers' (/Users/antonio_mac/OCT-Converter/oct_converter/readers/init.py)
"""""""
I have reinstalled all the code but it didn't help..
I hope you can help me cause making this code work would be great for me..

Thanks in advance!

I really like this tool, and I'm using it a lot. One thing I'm confused about is the order of the extracted OCT volumes. If we know that the volumes were acquired in a particular order, does the tool extract those volumes in the same acquired order? Also, is there any metadata we can extract about eye laterality of the volume? Ideally, I'd like to be able to know which volume corresponds to which eye. Thanks.

Hi I would be grateful for some advice regarding this error please:
(base) [xyz@spectre13 examples]$ python demo_e2e_extraction.py
Traceback (most recent call last):
File "/lustre/xyz/examples/demo_e2e_extraction.py", line 9, in
volume.peek(show_contours=True) # plots a montage of the volume
TypeError: peek() got an unexpected keyword argument 'show_contours'

Many thanks!

Some users have requested the ability to write in the .vol format, for compatability with some ImageJ plugins which expect this. I can't find any Python code that writes to this format, but there is a reader here and it should be possible to create the writer by reversing the read code.

Hello there,

I have various fda files. While running the script I am getting an error in some files. Below are oct_header information and error statuses.

The error received and the place where the error was received are given below.

What can be the problem? Do you have a solution suggestion?

oct_volume = fds.read_oct_volume()
Traceback (most recent call last):
File "", line 1, in
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/oct_converter/readers/fds.py", line 86, in read_oct_volume
raise ValueError('Could not find OCT header @IMG_SCAN_03 in chunk list')
ValueError: Could not find OCT header @IMG_SCAN_03 in chunk list

I am not sure why I am getting this error. Any help would be very appreciated!

This is what I get when I do fds = FDS(filepath)
b'@FDA_FILE_INFO'
b'@HW_INFO_03'
b'@PATIENT_INFO_03'
b'@CAPTURE_INFO_02'
b'@IMG_JPEG'
b'@PARAM_SCAN_04'
b'@IMG_TRC_02'
b'@PARAM_TRC_02'
b'@IMG_MOT_COMP_03'
b'@IMG_PROJECTION'
b'@REGIST_INFO'
b'@ALIGN_INFO'
b'@CONTOUR_INFO'
b'@FAST_Q2_INFO'
b'@thumbnail'
b'@GLA_LITTMANN_01'
b'@PATIENTEXT_INFO'
b'@EFFECTIVE_SCAN_RANGE'
b'@MAIN_MODULE_INFO'
b'@REPORT_INFO'
b'@CONTOUR_MASK_INFO'
b'@TOPQEXT_INFO'
b'@PARAM_ANGIOGRAPHY'
b'@REF_IMG_SCAN'
b'@FOCUS_MOTOR_INFO'
b'@ANGIO_TRACKING_INFO'
b'@IMG_ANGIOGRAPHY_02'
b'@TABIL_REGAVG_INFO'

The files look to contain angiography to me. They get picked up as fundus images, but the image_header produced here doesn't seem right - the values of height and width are huge, causing a memory error when the reader tries to read a chunk of size width*height:

Discussed in #90

Traceback (most recent call last):
  File "X:\m1\my-e2e.py", line 13, in <module>
    file.read_fundus_image()
  File "C:\prg\dev\Python\Python311\Lib\site-packages\oct_converter\readers\e2e.py", line 342, in read_fundus_image
    raw_volume = np.frombuffer(f.read(count), dtype=np.uint8)
                               ^^^^^^^^^^^^^

from oct_converter.readers import E2E

filepath = "x:\m1\EMRorder20230217.e2e"
file = E2E(filepath)
oct_volumes = (
    file.read_oct_volume()
)  # returns a list of all OCT volumes with additional metadata if available
for volume in oct_volumes:
    volume.peek(show_contours=True)  # plots a montage of the volume
    volume.save("{}_{}.avi".format(volume.volume_id, volume.laterality))

fundus_images = (
    file.read_fundus_image()
)  # returns a list of all fundus images with additional metadata if available
for image in fundus_images:
    image.save("{}+{}.png".format(image.image_id, image.laterality))

43 4D 44 62 00 00 00 00 00 00 00 00 64 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 4D 44 62 4D 44 69 72 00 00 00 00 00
64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 02 00 00 58 00 00 00
C8 00 00 00 DC 9D 78 56 4D 44 62 44 69 72 00 00
00 00 00 00 64 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00
58 00 00 00 00 00 00 00 F2 7A 56 34 8C 00 00 00
8C 58 00 00 83 00 00 00 00 00 00 00 11 00 00 00
FF FF FF FF FF FF FF FF FF FF FF FF FF FF 00 00
09 00 00 00 D2 9C 65 87 B8 00 00 00 4B 59 00 00

Hi There!
First of all, thank you Mark for your work!

I was trying out your implementation for converting Heidelberg's E2E format to images and videos, respectively.
I can execute your example file with my exported E2E-file from HEYEX (v1) but there seem to be 2 problems:

1. The resulting .avi file seems to consist only of one image, although in the step where all slides get displayed through the volume.peek() code line, the images of the slides are all different.

2. The converted images of the slides seem to be a bit darker than what I see in the HEYEX software. Therefore, some areas of the slides are not displayed very detailed. However, I'm not sure how significant this problem is as I'm not an ophthalmologist nor do I have any kind of medical background.

Thanks for your help!

Hi Mark

Thank you so much for the great tool. I am trying to extract data from E2E file but I get an error when I read the file:

from oct_converter.readers import E2E
filepath = '76543017.E2E'
file = E2E(filepath)
oct_volumes =file.read_oct_volume() 

then I get the following response:

---------------------------------------------------------------------------
StreamError                               Traceback (most recent call last)
Cell In[13], line 1
----> 1 oct_volumes =file.read_oct_volume() 

File /usr/local/lib/python3.8/site-packages/oct_converter/readers/e2e.py:219, in E2E.read_oct_volume(self)
    217 f.seek(start)
    218 raw = f.read(60)
--> 219 chunk = self.chunk_structure.parse(raw)
    221 if chunk.type == 9:  # patient data
    222     raw = f.read(127)

File /usr/local/lib/python3.8/site-packages/construct/core.py:288, in Construct.parse(self, data, **contextkw)
    274 def parse(self, data, **contextkw):
    275     r"""
    276     Parse an in-memory buffer (often bytes object). Strings, buffers, memoryviews, and other complete buffers can be parsed with this method.
    277 
   (...)
    286     :raises ConstructError: raised for any reason
    287     """
--> 288     return self.parse_stream(io.BytesIO(data), **contextkw)

File /usr/local/lib/python3.8/site-packages/construct/core.py:300, in Construct.parse_stream(self, stream, **contextkw)
    298 context._params = context
    299 try:
--> 300     return self._parsereport(stream, context, "(parsing)")
    301 except CancelParsing:
    302     pass

File /usr/local/lib/python3.8/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
    311 def _parsereport(self, stream, context, path):
--> 312     obj = self._parse(stream, context, path)
    313     if self.parsed is not None:
    314         self.parsed(obj, context)

File /usr/local/lib/python3.8/site-packages/construct/core.py:2120, in Struct._parse(self, stream, context, path)
   2118 for sc in self.subcons:
   2119     try:
-> 2120         subobj = sc._parsereport(stream, context, path)
   2121         if sc.name:
   2122             obj[sc.name] = subobj

File /usr/local/lib/python3.8/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
    311 def _parsereport(self, stream, context, path):
--> 312     obj = self._parse(stream, context, path)
    313     if self.parsed is not None:
    314         self.parsed(obj, context)

File /usr/local/lib/python3.8/site-packages/construct/core.py:2653, in Renamed._parse(self, stream, context, path)
   2651 def _parse(self, stream, context, path):
   2652     path += " -> %s" % (self.name,)
-> 2653     return self.subcon._parsereport(stream, context, path)

File /usr/local/lib/python3.8/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
    311 def _parsereport(self, stream, context, path):
--> 312     obj = self._parse(stream, context, path)
    313     if self.parsed is not None:
    314         self.parsed(obj, context)

File /usr/local/lib/python3.8/site-packages/construct/core.py:703, in Adapter._parse(self, stream, context, path)
    702 def _parse(self, stream, context, path):
--> 703     obj = self.subcon._parsereport(stream, context, path)
    704     return self._decode(obj, context, path)

File /usr/local/lib/python3.8/site-packages/construct/core.py:312, in Construct._parsereport(self, stream, context, path)
    311 def _parsereport(self, stream, context, path):
--> 312     obj = self._parse(stream, context, path)
    313     if self.parsed is not None:
    314         self.parsed(obj, context)

File /usr/local/lib/python3.8/site-packages/construct/core.py:4832, in FixedSized._parse(self, stream, context, path)
   4830 if length < 0:
   4831     raise PaddingError("length cannot be negative", path=path)
-> 4832 data = stream_read(stream, length, path)
   4833 if self.subcon is GreedyBytes:
   4834     return data

File /usr/local/lib/python3.8/site-packages/construct/core.py:91, in stream_read(stream, length, path)
     89     raise StreamError("stream.read() failed, requested %s bytes" % (length,), path=path)
     90 if len(data) != length:
---> 91     raise StreamError("stream read less than specified amount, expected %d, found %d" % (length, len(data)), path=path)
     92 return data

StreamError: Error in path (parsing) -> magic3
stream read less than specified amount, expected 12, found 0

What might be the problem??

Dear Authors,

demo_fda_extraction.py cannot run.
"No decoders are available"

Could you fix this? Thanks a lot!

Hi there,

I am using your lovely e2e reader with Python 3.9. Some of the arrays I am trying to load are ragged (and raise a warning) so I also used the following to inspect these ragged arrays:
import warnings
warnings.filterwarnings("error")

That made me come across another warning:
The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead

generated by line 252 in the e2e.py file:
raw_volume = np.fromstring(f.read(image_data.height * image_data.width), dtype=np.uint8)

According to https://stackoverflow.com/questions/60320840/numpy-fromstring-deprecated-use-frombuffer-instead it seems like the loading should be done by np.frombuffer (instead of np.fromstring). I edited the source code locally and it seems to indeed prevent this warning. I don't know if there are consequences for these fromstring warnings but it might be something you want to check on the next versions.

Regards,
Oren

Hello, thank you for the great package. When trying to extract the laterality info using:

for volume in oct_volumes:
volume.save('{}.png'.format(volume.laterality))

I get files saved as None.png. I'm sure I'm doing something obvious wrong here but I'd appreciate a pointer.

Thanks

Originally posted by @amanasj in #28 (comment)

Hi there,

Thanks for providing this codebase!

I was wondering if there were any plans to add functionality that could extract segmentation contours from E2E files? There is some example code for doing this (in C++) in uocte.

All the best,
Oisin

Hi Mark
I'm afraid I'm completely new to python. I have V3 Shell installed on a windows 10 pc. If I copy and paste :

git clone https://github.com/marksgraham/OCT-Converter.git
pip install -e OCT-Converter

I get a syntax error with the cursor over the c in clone.

I have a Topcon 1000 mk2 which is getting long in the tooth and will have to replace it soon. I would like to export the data from it so that I can view it on a DICOM viewer.

Can your program do bulk conversion? How can the files be linked to the patient after conversion?

Any help much appreciated.

Mark

Mark Jones

Hello,
I was working on with this repo for fda files and I can not extract the images with decode function after that I add a try except and with except part I decode the data with bytes io. And I add some functions to get the other informations in fda file and wrote another .py file with a function that extract bscans images, video, fundus image, gray scale fundus image and a json which contains other informations in fda file.

Hi Mark

Hope you're doing well.
These days I've been extracting some fda files to png and I've notice that the program extracts all the images with the same distribution of pixels so seems an artifact when compared to the images extracted directly from the device.
Do you know if there is some parameter in the code which I could tune to modify that distribution?

Here's a link where you can see the comparison of the distribution of pixel from oct-converter (at left) and extracted from the device.

https://drive.google.com/file/d/17xFnlKed8gyWeeFzJvqknOkXIXVM3_hT/view?usp=sharing

Thanks in advance
Regards!
A

Hi Mark,

You added bscan_metadata to the e2e reader in the latest commits. I think each slice of layer segment has one corresponding bscan_metadata. I was wondering how can I use posX1, posY1, posX2, posY2 etc to find the locations of these layers on the fundus image?

Thank you,
Hua

Thanks for your work on the Zeiss Dicom format, now I can read the dicom files from Zeiss SD-OCT Macular Cube structural scan (512x128, 200x200 on 6x6mm).
However, I cannot read the Zeiss SD-OCTA scans, which size 350x350x1024 for 6x6mm, and size 245x245x1024 for 3x3mm. Looks like no JP2 header "\x00\x00\x00\x0C" in the PixelData.
I guess they might change the way to obfuscate the data but still use the JPEG2000.

Have you tried this on the SD-OCTA dicom file?

C:\ProgramData\Anaconda3\lib\site-packages\pydicom\charset.py:471: UserWarning: Found unknown escape sequence in encoded string value - using encoding iso8859
warnings.warn(msg + f" - using encoding {encodings[0]}")

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3418, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 4, in
oct_volumes, fundus_volumes = img.read_data() # returns a list OCT and fudus images
File "C:\ProgramData\Anaconda3\lib\site-packages\oct_converter\readers\zeissdicom.py", line 93, in read_data
unscrambled_frame = self.unscramble_frame(scrambled_frame)
File "C:\ProgramData\Anaconda3\lib\site-packages\oct_converter\readers\zeissdicom.py", line 146, in unscramble_frame
raise ValueError("No JP2 header found in the scrambled pixel data")
ValueError: No JP2 header found in the scrambled pixel data

Hello all,
I am working with file types of:
.pdb
.edb
.sdb
.mdb

Is there anyone working on the conversion/reading of these file types? If not I can get started with it and would welcome pointers.
Thank you,
Jason

Dear Mark,
Thanks a lot for developing OCT-Converter.
I wish to extract the photo from some .oct files (an example attached), but met the StreamError. I wonder if there is any suggestion on solving the problem? Many thanks!

Best regards,
Xiaogang

Default_0002_Mode2D.zip

Thank you for developing this tool. I am using it to convert .e2e format to extract oct images. I have almost 1000 files in .e2e format and trying to extract the oct images from it. However for some of the files it correctly extract the oct images but for few files it gives the following two different errors while extracting oct images. These both errors are for two different .e2e files.

1. cannot reshape array of size 366807 into shape (768,768)
2. 'int' object has no attribute 'shape'
   I have also attached both the errors.
   Thanks you!.

Hi Mark,
Hope you are doing well.
Recently I have been collecting some fda files which include OCTA images. I guess the useful information should be stored in these two chunks '@PARAM_ANGIOGRAPHY' and 'IMG_ANGIOGRAPHY_02'.

Have you tried extracting the OCTA directly from the fda file before, just like extracting OCT and fundus?

Thanks in advance
Regards!

Hi and thanks for the tool.

I am trying to parse segmentation data from fda files to extend eyepy.

I believe the segmentation is stored in @CONTOUR_INFO chunks (one per layer).

I've noticed that the read_any_info_and_make_dict cannot be used to retrieve the segmentation of all layers as they share the same name and key in the dictionary.

What do you think about adding a read_segmentation() method to FDA class ? I can do it myself if it sounds good.

Hi,
for me version 0.1 installs whenever I try to install the package via pip. If sepcifying oct-converter==0.2 I get the follwoing Error:
ERROR: Could not find a version that satisfies the requirement oct-converter==0.2 (from versions: 0.1)
My pip is up to date. Any idea why that is and how to install the current version?

Thanks a lot!

Sorry, had the impression I was using python 3.7 which was wrong..

First, your work is wonderful but when I try to convert.oct to .img I meet errors.
I used the program in the samples , demo_boct_extraction.py
but when I run it, it logs like this. If you have any idea, I will be grateful for your help.

D:\Project\a.zip
Traceback (most recent call last):

File "D:\Project\demo_boct_extraction.py", line 9, in
boct = BOCT(filepath)

File "E:\anaconda\anaconda\lib\site-packages\oct_converter\readers\boct.py", line 64, in init
"magicNumber" / Hex(Int32un),

File "E:\anaconda\anaconda\lib\site-packages\construct\core.py", line 616, in rtruediv
return Renamed(self, newname=name)

File "E:\anaconda\anaconda\lib\site-packages\construct\core.py", line 2642, in init
super().init(subcon)

File "E:\anaconda\anaconda\lib\site-packages\construct\core.py", line 675, in init
raise TypeError("subcon should be a Construct field")

TypeError: subcon should be a Construct field

hi
I don't seem to be able to understand what the license is.
could you please clarify in the Readme, or add a header to all .py files?

thanks
mic

Hey,
your converter is a long needed tool. Somehow I have problems to open E2E files using your example E2E code. Everytime, I got this error message:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\IPython\core\interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('C:/Users/josef/Desktop/Nocturne/Development_Projects/Nocturne_Software/OCT-Converter/examples/demo_e2e_extraction.py', wdir='C:/Users/josef/Desktop/Nocturne/Development_Projects/Nocturne_Software/OCT-Converter/examples')
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.2.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.2.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/josef/Desktop/Nocturne/Development_Projects/Nocturne_Software/OCT-Converter/examples/demo_e2e_extraction.py", line 11, in
fundus_images = file.read_fundus_image() # returns a list of all fundus images with additional metadata if available
File "C:\Users\josef\Desktop\Nocturne\Development_Projects\Nocturne_Software\OCT-Converter\oct_converter\readers\e2e.py", line 225, in read_fundus_image
chunk = self.chunk_structure.parse(raw)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 288, in parse
return self.parse_stream(io.BytesIO(data), **contextkw)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 300, in parse_stream
return self._parsereport(stream, context, "(parsing)")
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 2120, in _parse
subobj = sc._parsereport(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 2653, in _parse
return self.subcon._parsereport(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 703, in _parse
obj = self.subcon._parsereport(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 4832, in _parse
data = stream_read(stream, length, path)
File "C:\ProgramData\Anaconda3\envs\noct\lib\site-packages\construct\core.py", line 91, in stream_read
raise StreamError("stream read less than specified amount, expected %d, found %d" % (length, len(data)), path=path)
construct.core.StreamError: Error in path (parsing) -> magic
stream read less than specified amount, expected 12, found

Hi everyone,
Is there anyone who ever work with .opt files.

Why this now in 0.3.0?

In [1]: from oct_converter.readers import E2E
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from oct_converter.readers import E2E

File ~/.virtualenvs/peye2/lib/python3.8/site-packages/oct_converter/readers/__init__.py:8, in <module>
      6 from .img import IMG
      7 from .poct import POCT
----> 8 from .zeissdcm import ZEISSDICOM

ModuleNotFoundError: No module named 'oct_converter.readers.zeissdcm'

When working with either dicom (preferred) or *.img files (hopefully won't have to) from a "Carl Zeiss Cirrus 6000" machine, the code here is unable to parse the image, and instead throws errors. I have two different formats (*.dcm, *.img) available and also two different TransferSyntaxUIDs, but can't get any options to work.

with dicom:

from oct_converter.readers import Dicom

for i, filepath in fpaths.items():
    try:
        file = Dicom(filepath)
        oct_volume = file.read_oct_volume()  # returns an OCT volume with additional metadata if available
        print(i)
    except Exception as e:
        print(f"#{i} failed: {e}")

0 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset
1 failed: No matching JPEG 2000 format found
2 failed: Unexpected tag '(ab8f, 7b89)' when parsing the Basic Table Offset item.
3 failed: Unexpected tag '(1eab, 3f83)' when parsing the Basic Table Offset item.
4 failed: Unexpected tag '(ab3a, b796)' when parsing the Basic Table Offset item.
5 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset
6 failed: No matching JPEG 2000 format found
7 failed: Unexpected tag '(4a27, c7c7)' when parsing the Basic Table Offset item.
8 failed: Unexpected tag '(9bea, 539b)' when parsing the Basic Table Offset item.
9 failed: Unexpected tag '(9d70, b305)' when parsing the Basic Table Offset item.
10 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset
11 failed: No matching JPEG 2000 format found
12 failed: Unexpected tag '(87ad, ae02)' when parsing the Basic Table Offset item.
13 failed: Unexpected tag '(3ef9, c56e)' when parsing the Basic Table Offset item.
14 failed: Unexpected tag '(93ca, 9a6a)' when parsing the Basic Table Offset item.
15 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset
16 failed: No matching JPEG 2000 format found
17 failed: Unexpected tag '(e6b5, 94bc)' when parsing the Basic Table Offset item.
18 failed: Unexpected tag '(55a2, 7aa4)' when parsing the Basic Table Offset item.
19 failed: Unexpected tag '(108a, 99c8)' when parsing the Basic Table Offset item.
20 failed: unsupported operand type(s) for //: 'str' and 'str'
21 failed: No matching JPEG 2000 format found
22 failed: No matching JPEG 2000 format found
23 failed: Unexpected tag '(d484, c455)' when parsing the Basic Table Offset item.
24 failed: Unexpected tag '(9ab2, 6eb9)' when parsing the Basic Table Offset item.
25 failed: Unexpected tag '(f54c, 1a7f)' when parsing the Basic Table Offset item.
26 failed: Unexpected tag '(71a0, 48eb)' when parsing the Basic Table Offset item.
27 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset
28 failed: No matching JPEG 2000 format found
29 failed: No matching JPEG 2000 format found
30 failed: No matching JPEG 2000 format found
31 failed: Unexpected tag '(f24f, eb93)' when parsing the Basic Table Offset item.
32 failed: Unexpected tag '(e60c, 05ba)' when parsing the Basic Table Offset item.
33 failed: Unexpected tag '(0253, 5d4a)' when parsing the Basic Table Offset item.
34 failed: Unexpected tag '(88ba, f184)' when parsing the Basic Table Offset item.
35 failed: Unable to convert the pixel data: one of Pixel Data, Float Pixel Data or Double Float Pixel Data must be present in the dataset

In case it's helpful here are the TransferSyntaxUID specifications for these .dcm files (metadata reads fine in PyDicom, but not the image)

00 -> 1.2.840.10008.1.2.1
01 -> 1.2.840.10008.1.2.4.90
02 -> 1.2.840.10008.1.2.4.90
03 -> 1.2.840.10008.1.2.4.90
04 -> 1.2.840.10008.1.2.4.90
05 -> 1.2.840.10008.1.2.1
06 -> 1.2.840.10008.1.2.4.90
07 -> 1.2.840.10008.1.2.4.90
08 -> 1.2.840.10008.1.2.4.90
09 -> 1.2.840.10008.1.2.4.90
10 -> 1.2.840.10008.1.2.1
11 -> 1.2.840.10008.1.2.4.90
12 -> 1.2.840.10008.1.2.4.90
13 -> 1.2.840.10008.1.2.4.90
14 -> 1.2.840.10008.1.2.4.90
15 -> 1.2.840.10008.1.2.1
16 -> 1.2.840.10008.1.2.4.90
17 -> 1.2.840.10008.1.2.4.90
18 -> 1.2.840.10008.1.2.4.90
19 -> 1.2.840.10008.1.2.4.90
20 -> 1.2.840.10008.1.2.4.90
21 -> 1.2.840.10008.1.2.4.90
22 -> 1.2.840.10008.1.2.4.90
23 -> 1.2.840.10008.1.2.4.90
24 -> 1.2.840.10008.1.2.4.90
25 -> 1.2.840.10008.1.2.4.90
26 -> 1.2.840.10008.1.2.4.90
27 -> 1.2.840.10008.1.2.1
28 -> 1.2.840.10008.1.2.4.90
29 -> 1.2.840.10008.1.2.4.90
30 -> 1.2.840.10008.1.2.4.90
31 -> 1.2.840.10008.1.2.4.90
32 -> 1.2.840.10008.1.2.4.90
33 -> 1.2.840.10008.1.2.4.90
34 -> 1.2.840.10008.1.2.4.90
35 -> 1.2.840.10008.1.2.1

for the IMG version, all of the errors fall into some form of cannot reshape array of size _____ into shape (1024,512,128) where the _____ is either 5242880 or 2097152

Made Improvments On fda_testing.py, oct.py, fda.py

• There is a warning when writing video from bscans says:
  "Lossy conversion from float64 to uint8. Range [0.0, 255.0]. Convert image to uint8 prior to saving to suppress this warning."

• When testing the files with fda_testing.py if an image chunk is not in fda file methods raise Value Error and
  the code terminating for preventing that the if condition in image methods updated as other methods. Like if an
  image chunk is not in fda file the method print the info and return None. But this cause another error which is terminate
  the code on testing side the error come from save function and says "None type object has no attribute save" for
  prevent that an if statement added on fda_testing.py checking if image data exist or None.

• I made the changes you want on the fda.py now there is only one method for making the json from other chunks datas