kaiko-ai / spark-dicom Goto Github PK
View Code? Open in Web Editor NEWSpark dicom (streaming) connector
License: Apache License 2.0
Spark dicom (streaming) connector
License: Apache License 2.0
The de-identifier should use a hash function to generate pseudonyms for tags with the U
action (from the table here).
To make the private and sequence tags more usable, the JSON should use the keyword of the tag as keys instead of the tag itself
This will allow the deidentification to be incorporated in an end to end workflow
Running
display(spark.read.format("dicomFile")
.option("recursiveFileLookup", true)
.option("includeContent", true)
.load("/mnt/tcia/gcs-public-data--healthcare-tcia-apollo/"))
results in NoSuchMethodError: org.apache.hadoop.fs.FSDataInputStream.readAllBytes()[B
likely due to dependency mismatch.
As discussed in: #48 (comment)
Following comment from @robopoc
Currently, a parsing error stops the whole DataSource read.
Because of the nature of DICOM files, failure is bound to happen.
The goal would be to allow read failure, and report them to a column.
Example:
12:10:55.0
(with :
which does not comply to standard)[{ column: "A", message: "DateTimeParseException: ..." }]
Would load the raw binary and drop it into a column.
Need optionality just like PixelData
Currently, sequence tags are ingested into BinaryType columns. In order to better work with the data in these tags, we should read them in as json formatted StringType
Currently, when the data source reads a file which is not a DICOM file, the entire job fails.
The data source should just go to the next file.
An idea would be to add a boolean column isDicom
. Then, parsing a non-DICOM file would still give you an entry, but without any data but isDicom = false
. This would increase ease of use as it would allow to ensure file detection.
I'm getting the below error message when trying to read DICOM files from the TCIA gs bucket.
java.time.format.DateTimeParseException: Text '122734.625000' could not be parsed, unparsed text found at index 10
Code to reproduce (pyspark):
APOLLO_FILE_PATH = "gs://gcs-public-data--healthcare-tcia-apollo/dicom/"
apollo_df = spark.read.format("dicomFile").option("recursiveFileLookup", "true").load(APOLLO_FILE_PATH)
apollo_df.collect()
Currently, private tags are not read at all.
We could have a column "private data" in StringType in which we dump private tags as a JSON string.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.