nsrr / edfize

Ruby gem used to load, validate, and parse European Data Format files. Used for batch testing EDFs for errors.

National Sleep Research Resource

Public documentation for the National Sleep Research Resource. The private repository is currently available at github.com/sleepepi/nsrr.

Goal

Enhance clinical and translational work in human sleep medicine and physiology by creating the NHLBI National Sleep Research Resource (NSRR), a comprehensive, easily accessible, well-annotated national repository of sleep data, including bio-physiological signals, which are linked to risk factor and outcome data for participants in major NIH studies.

Teams

Sleep Terms

• Matt Kim Team Leader
• Kevin Gleason
• Dan Mobley
• Michael Rueschman
• Satya Sahoo

Sleep Arch

• Michael Rueschman Team Leader
• Madalena Costa
• Kevin Gleason
• Ary Goldberger
• Dan Mobley
• Naresh Punjabi
• Research Assistants

Sleep Port

• Remo Mueller Team Leader
• Licong Cui
• Matt Kim
• Dan Mobley
• GQ Zhang

Sleep Tools

• Dan Mobley Team Leader
• Madalena Costa
• Dennis Dean
• Ary Goldberger
• Catherine Jayapandian
• Ken Loparo

Regulatory and Community Interface

• Susan Redline Team Leader
• Dan Mobley
• Susan Surovec
• BWH Reg.

License

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

edfize's Issues

Add test that date and time are formatted properly at beginning of header

Add documentation to explain how to use this gem to load and read EDFs as Ruby objects from a simple ruby file

While the main purpose of edfize is to validate the format of a folder of EDFs, the underlying data model to read in EDFs could be used to do further calculations on the signals themselves as they are presented as Ruby arrays of Fixnum.

This could also lead to more adventurous models to modify certain components of the model, and then write the model back to a new EDF file. (ex: scrubbing identifiable information, etc).

Provide test to check that EDF header and signal header reserved areas are blank

Having information in the reserved area other than blank characters is a common sign of corruption with the EDF header. Characters from these locations are often leaked from other the data signal itself and indicate that the EDF was not properly formatted when it was created.

Add test to validate date in header

The Date in the EDF header can be easily parsed using ruby Date class.

Allow edfize to read EDFs across many subfolders

Currently, edfize does not look across many subfolders to build the list of EDFs to check. Some of our projects have single EDF files stored within single folders (e.g. /participant12345edf/), so being able to run edfize at the root of such a collection of folders would be helpful.

Incorporate tutorial for writing EDFs

# Tutorial 02 - Write an EDF
#
#   gem install edfize
#
#   ruby tutorial_02_write_an_edf.rb
#
# Generates a small EDF with 10 data records and 3 signals (each signal with 1 or 2 samples per data record)

require 'rubygems'
require 'edfize'
require 'fileutils'
require 'date'

filename = 'simulated-01.edf'

FileUtils.touch filename

NUMBER_OF_SIGNALS = 3

# Loads the file and reads the EDF Header
edf = Edfize::Edf.create(filename) do |t|
  t.version                        = 0
  t.local_patient_identification   = "TEST001 M #{Date.parse("1984-05-07").strftime("%d-%^b-%Y")} Joe_Schmoe"
  t.local_recording_identification = ""
  t.start_date_of_recording        = Date.today.strftime("%d.%m.%y")
  t.start_time_of_recording        = Time.now.strftime("%H.%M.%S")
  t.number_of_bytes_in_header      = Edfize::Edf::HEADER_OFFSET + (NUMBER_OF_SIGNALS * Edfize::Signal::SIGNAL_CONFIG.collect{|k,h| h[:size]}.inject(:+))
  t.reserved                       = ""
  t.number_of_data_records         = 10
  t.duration_of_a_data_record      = 1
  t.number_of_signals              = NUMBER_OF_SIGNALS
end

signals = []

signals[0] = Edfize::Signal.create do |t|
  t.label                    = "SaO2"
  t.transducer_type          = ""
  t.physical_dimension       = ""
  t.physical_minimum         = 0.0
  t.physical_maximum         = 100.0
  t.digital_minimum          = -32768
  t.digital_maximum          = 32767
  t.prefiltering             = ""
  t.samples_per_data_record  = 1
  t.reserved_area            = ""
  t.digital_values           = (0..9).to_a
end

signals[1] = Edfize::Signal.create do |t|
  t.label                    = "H.R."
  t.transducer_type          = ""
  t.physical_dimension       = ""
  t.physical_minimum         = 0.0
  t.physical_maximum         = 250.0
  t.digital_minimum          = -32768
  t.digital_maximum          = 32767
  t.prefiltering             = ""
  t.samples_per_data_record  = 1
  t.reserved_area            = ""
  t.digital_values           = (0..9).to_a
end

signals[2] = Edfize::Signal.create do |t|
  t.label                    = "EEG(sec)"
  t.transducer_type          = ""
  t.physical_dimension       = "uV"
  t.physical_minimum         = -125.0
  t.physical_maximum         = 125.0
  t.digital_minimum          = -128
  t.digital_maximum          = 127
  t.prefiltering             = ""
  t.samples_per_data_record  = 2
  t.reserved_area            = ""
  t.digital_values           = (0..19).to_a
end



def save(edf_self, edf_signals)
  edf_self
  # puts edf_self.print_header

  Edfize::Edf::HEADER_CONFIG.keys.each do |section|
    size = Edfize::Edf::HEADER_CONFIG[section][:size]
    string = ("%-#{size}.#{size}s" % edf_self.send(section).to_s)
    IO.binwrite(edf_self.filename, string, edf_self.send(:compute_offset, section) )
  end


  puts "Signals #{edf_signals.inspect}"

  Edfize::Signal::SIGNAL_CONFIG.keys.each do |section|
    offset = Edfize::Edf::HEADER_OFFSET + NUMBER_OF_SIGNALS * edf_self.send(:compute_signal_offset, section)
    edf_signals.each_with_index do |signal, signal_number|
      section_size = Edfize::Signal::SIGNAL_CONFIG[section][:size]
      string = ("%-#{section_size}.#{section_size}s" % signal.send(section).to_s)
      IO.binwrite(edf_self.filename, string, offset+(signal_number*section_size))
    end
  end

  all_samples_per_data_record = edf_signals.collect{|s| s.samples_per_data_record}
  total_samples_per_data_record = all_samples_per_data_record.inject(:+).to_i

  offset = 0
  offsets = []
  all_samples_per_data_record.each do |samples_per_data_record|
    offsets << offset
    offset += samples_per_data_record
  end

  signal_values = []

  (0..edf_self.number_of_data_records - 1).each do |data_record_index|
    edf_signals.each_with_index do |signal, signal_index|
      # read_start = data_record_index * total_samples_per_data_record + offsets[signal_index]
      # string = signal.digital_values[0..1].pack('s<*')
      read_start = data_record_index * signal.samples_per_data_record
      read_end   = read_start + signal.samples_per_data_record - 1
      puts "Signal: #{signal.label} #{signal.digital_values[read_start..read_end]}"
      signal_values += signal.digital_values[read_start..read_end]
    end
  end

  data_string = signal_values.pack('s<*')

  IO.binwrite(edf_self.filename, data_string, edf_self.number_of_bytes_in_header)

end

save(edf, signals)

Add test that looks at checksums across all EDF files checked

.. in order to possibly identify duplicate files in the batch of EDFs that were just checked. We have recently run into an issue where a student converted the same study twice yet saved it with different filenames. It would be nice to throw out a list of "potential duplicate" studies in edfize.

Provide option to limit the verbosity of tests

Currently running edfize test will create pretty verbose tests.

Perhaps add an option to limit the verbosity to squelch passing tests, and only show failing tests. Also perhaps make failing tests either show the full error, or just show that they are failing.

Perhaps edfize test --squelch-passing or edfize test --failing or edfize test --failing-verbose

This is important when running against results for around 6,000 EDFs.

Provide test to check if expected EDF size matches the actual EDF file size

This should be run against a the current directory of EDFs when running spout t

Documentation on Warnings

Looking through the EDF specification, it looks like all the EDF headers contain a value 0 for the edf_data_format_version. While this makes sense and could have been increased to 1 for an "EDF+" version, I was surprised to find that the EDF+ standard also has a "0" as the edf_data_format_version.

See more details on why the 0 was kept here: edfplus.info/specs/edfplus.html#header The argument that is provided that this was done to provide forwards-compatibility with EDF-only viewers for EDF+ files.

Since I haven't seen any other format for EDFs it appears that any number could be placed there, and still have the rest of the EDF intact.

This brings up the idea that an EDF model should perhaps indicate it as a "warning" to a user that the number is expected to be 0 but isn't in the case that this is wrong. The viewer could still attempt to load the EDF however.

These types of errors may go into more of a warning category for edfizer since they don't affect the integrity of the EDF itself.

These warnings would then be listed in a more verbose version of edfize test, perhaps using a flag such as edfize test --include-warnings.

Create HTML view of passing errors and failures.

Similar to how spout has a coverage report, perhaps make a clever html report for EDFs to find errors.

This would address the issue of the Windows command box limiting the amount of output that could be seen for large EDF folders (even with non-verbose and failing tests only).

Deprecate `edfs_in_current_directory_and_subdirectories`

Add test to assure that example file works correctly

Updates to the gem shouldn't break the example ruby code in the README

Cache test results so that interrupting the tests in the middle could resume correctly

This would provide the ability to test a folder and interrupt in the middle without losing the first half of the results. Important when testing a large volume of EDFs in batch.

Cached results would also potentially have to be deleted to restart the test edfize test --clean as the test specification could potentially change (in development) or the EDF files themselves could change (staging into production).

Add check to assure that digital maximum is within expected bounds

Read more here: https://emotiv.com/forum/forum14/topic2242/messages/

Because the resolution of the EDF-format is 16-bit and encoding is 2's complement, the maximum value for the digital maximum field can not exceed 32767. Kass

edfize t should provide an update of which EDF it is currently checking

when using the edfize t --failing, it appears to not be running due to lack of output. Add something in similar to: "checking X of Y EDFs"