xdf-modules / xdf-matlab Goto Github PK

This line is wrong: https://github.com/xdf-modules/xdf-Matlab/blob/master/load_xdf.m#L530

What can happen is that in the case of a clock reset, the timestamps can go from a very high to a very small range of values. In this case, this line of code will fail to spot the large negative spike in the diff(timestamps) array and the result will be completely wrong. I have a data set where this happens and the effective sampling rate is a negative value. Taking the absolute value of the diff(timestamps) array will stop this from happening.

Not sure what the performance other people have been getting looks like, but I have experienced very slow loading when using the load_xdf() function in MATLAB.

My files are ~5GB, 64 channels EEG @ 2048 Hz and an audio recording with 2 channels @ 44.1 kHz. Loading times for each file are ~1hr.

This issue occurs on Windows, Ubuntu, and CentOS using MATLAB 2014b and 2018b.

Is this expected behavior? How can I speed up loading?

Transfered from Bugzilla https://sccn.ucsd.edu/bugzilla/show_bug.cgi?id=1908

Ramon Martinez-Cancino 2016-08-04 17:22:48 PDT
Hi Christian,
I hope this email finds you well.
We've received this report on Bugzilla about an issue with load_xdf.m. As this belong to one plugin you wrote I'm FW-ing it (although I'm not sure if you are still taking care of it)
I made some testings and the thing is that in line 424 of load_xdf.m, the output of the min() is zero since
find([abs(remaining_data-cur_end_time) > abs(remaining_data-next_begin_time),true],1)-1

is zero too when using this data.I tried taking the next point that is not zero from the min() by taking the two first minimum values and checking if the first one is zero or not, and seems to work (at least does not crash). Can you take a look into this, please.

Thanks in advance,

Ramon
[reply] [−]DescriptionAaron 2016-06-29 07:08:19 PDT
"EEGLAB error in function load_xdf() at line 424: subscript indices must either be real positive integers or logicals."

Trying to import a file and cannot get past this step. Worked previously with same devices - now it is rejected. Unsure of the reasons.

File can be downloaded at: https://we.tl/iMwQb94j8W

In a slight rush to analyse data and I ca't access anything at the moment.

Many thanks and appreciate any help you can offer!

The Matlab importer chokes on files when a stream has the ID 0, such as this
example file:

streamid = fread(f,1,'uint32')
id = length(streams)+1;
idmap(streamid) = id; %#ok<SPRIX>  <- here

Hi, I have tried to plot the channel data through Matlab, but however, the event markers are all at time = 0. Also, I have tried some solutions in the previous issues but still don't work.
The data was EEG data from Muse2 Headband and the events were from python (pylsl) as a marker stream.

This is the code:

import serial
from pylsl import StreamInfo, StreamOutlet, local_clock

serial_port = 'COM4'
baud_rate = 9600
ser = serial.Serial(serial_port, baud_rate)

info = StreamInfo('MyMarkerStream', 'Markers', 1, 0, 'string', 'myuid34234')
outlet = StreamOutlet(info)

while True:
    try:
        line = ser.readline().decode('utf-8').strip()
        timestamp = local_clock()
        print(type([line]))
        outlet.push_sample([line], timestamp)
        print(f"Sent data: {line} hz at time: {timestamp}")
    except KeyboardInterrupt:
        print("Exiting...")
        ser.close()
        break
    except Exception as e:
        print(f"An error occurred: {e}")

(Edited by Chad to add code-formatting tags around the code)

Prompted by a user issue here:
labstreaminglayer/App-LabRecorder#107

LabRecorder puts clock offsets in the footer, but it also puts clock offsets throughout the recording. It shouldn't be strictly necessary to process the footer.

load_xdf should gracefully handle a corrupt footer.

Related: xdf-modules/pyxdf#83

sccn/eeglab#535

Has anyone ever evaluated the accuracy of load_xdf's drift correction algorithm? It is a non-standard algorithm (i.e. not part of Matlab) for fitting a first order curve to correct for clock drift between sending and receiving devices

https://github.com/xdf-modules/xdf-Matlab/blob/master/load_xdf.m#L505-L516

The reason I ask is that it has come to my attention that 1st order curves may not be appropriate for typical drift. For example, Here is the difference between LSL Markers and a screen flip signal recorded via an EEG device.

The total marker/EEG drift changes from about 27 to 23 ms over the course of several hours. This is definitely enough to muck up ERP/ERSP analysis. Such second order curves are pretty typical but an nth order curve is probably what is best.

If I were to rewrite this, I would use polyfit and do away with the ADMM method using the Huber loss function as minimization criteria but maybe there is a compelling reason for using such a robust tool. I confess that the main reason I would choose polyfit is that I don't fully understand how to adapt the code to return an nth order set of mapping coefficients. scipy has analogs to polyfit but I don't know quite what to do in C++, but there must a be a good lib out there for that stuff.

This line is wrong: https://github.com/xdf-modules/xdf-Matlab/blob/master/load_xdf.m#L530

What can happen is that in the case of a clock reset, the timestamps can go from a very high to a very small range of values. In this case, this line of code will fail to spot the large negative spike in the diff(timestamps) array and the result will be completely wrong. I have a data set where this happens and the effective sampling rate is a negative value. Taking the absolute value of the diff(timestamps) array will stop this from happening.

Hi everyone,

I am asking this for a colleague in China. "When we use LSL to collect respiratory and electromyographic data which was collected by bio-radio wireless physiological recorder. Data in a few minutes can be loaded in MATLAB workspace, but when the amount of data getting larger (such as 10 minutes or so), it will report the following error. How should it be solved?" I've attached his screenshots of the error encountered.

I've already asked on Slack and Chadwick suggested that it's probably a locale problem but to also post this issue here. Unfortunately, my Chinese is poor so I wasn't really able to communicate properly with my colleague. I've attached the file that is creating the error. I've also attached a file that works for comparison. My colleague is using 2019 MATLAB with the latest version of LSL.

Kind regards,
Nathan

data.zip

I simultaneously recorded EEG and audio using the audio capture software and synchronized the streams in LSL. When loading the audio streams using pop_loadxdf (version: 1.18) I receive the following message: 'Warning: Rank deficient, rank = 1, tol = 4.574081e+03'. The error is caused in line 600 in load_xdf when the jitter is removed:
'mapping = temp(k).time_stamps(indices) / [ones(1,length(indices)); indices];'
As a result, trigger markers, which I use to epoch the EEG and audio streams, are not correct in the audio stream.
The error occurs only if an audiostream has a length of approximately more than 62.000.000 data points. I recorded with a sampling rate of 44100, thus I have roughly 23 min of data.

Here an example how to reproduce the error in MATLAB 2020b:

% this throws an error in matlab 2020b
% Warning: Rank deficient, rank = 1, tol = 4.366797e+03.
nrSamples =65000000;
time_stamps=linspace(0,26,nrSamples);
indices=1:length(time_stamps);
mapping = time_stamps(indices) / [ones(1,length(indices)); indices];

%this does not, only differenc is nrSamples
nrSamples =60000000;
time_stamps=linspace(0,26,nrSamples);
indices=1:length(time_stamps);
mapping = time_stamps(indices) / [ones(1,length(indices)); indices];

I appreciate any help!

Original text from @krigolson:

Hi, I have loaded some EEG data that has event markers in it - and the time stamps appear to be not synchronized so when I load the data into EEGLAB the marker latencies are all the same and 1. I am using the latest version of all the software and note sure what is going wrong here. HELP!

The first stream was EEG data from a Muse Headband and the second stream was from MATLAB as a marker stream.

moved from: sccn/xdf#59