kaiko-ai / spark-dicom Goto Github PK

The de-identifier should use a hash function to generate pseudonyms for tags with the U action (from the table here).

To make the private and sequence tags more usable, the JSON should use the keyword of the tag as keys instead of the tag itself

This will allow the deidentification to be incorporated in an end to end workflow

Running

display(spark.read.format("dicomFile")
  .option("recursiveFileLookup", true)
  .option("includeContent", true)
  .load("/mnt/tcia/gcs-public-data--healthcare-tcia-apollo/"))

results in NoSuchMethodError: org.apache.hadoop.fs.FSDataInputStream.readAllBytes()[B

likely due to dependency mismatch.

As discussed in: #48 (comment)

Following comment from @robopoc

Currently, a parsing error stops the whole DataSource read.

Because of the nature of DICOM files, failure is bound to happen.

The goal would be to allow read failure, and report them to a column.

Example:

• An element of Keyword "A" of VR TM has an invalid time written 12:10:55.0 (with : which does not comply to standard)
• In a column "errors", we get [{ column: "A", message: "DateTimeParseException: ..." }]

Would load the raw binary and drop it into a column.

Need optionality just like PixelData

Currently, sequence tags are ingested into BinaryType columns. In order to better work with the data in these tags, we should read them in as json formatted StringType

Currently, when the data source reads a file which is not a DICOM file, the entire job fails.

The data source should just go to the next file.

An idea would be to add a boolean column isDicom. Then, parsing a non-DICOM file would still give you an entry, but without any data but isDicom = false. This would increase ease of use as it would allow to ensure file detection.

I'm getting the below error message when trying to read DICOM files from the TCIA gs bucket.

java.time.format.DateTimeParseException: Text '122734.625000' could not be parsed, unparsed text found at index 10

Code to reproduce (pyspark):

APOLLO_FILE_PATH = "gs://gcs-public-data--healthcare-tcia-apollo/dicom/"
apollo_df = spark.read.format("dicomFile").option("recursiveFileLookup", "true").load(APOLLO_FILE_PATH)
apollo_df.collect()

Currently, private tags are not read at all.

We could have a column "private data" in StringType in which we dump private tags as a JSON string.