vislab / eeg-clean-tools Goto Github PK

Hi,

I tried preprocess a file (80 MB) but got an out of memory error. I was using parallel pool.

I am on Mac M1 MAX, 64 GB RAM, Matlab 2020b, PrepPipeline version 0.56.0 (8.01.2021).

Running Prep...
Using option file in directory ~
pop_editoptions: Options were not modified
Checking for boundary events
Preliminary detrend to compute reference
pop_eegfiltnew() - performing 1651 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase, non-causal)
firfilt(): |====================| 100%, ETE 00:00
Line noise removal
Find reference
Robust referencing
pop_eegfiltnew() - performing 1651 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase, non-causal)
firfilt(): |====================| 100%, ETE 00:00
Warning: [MATLAB:lang:StackOverflow]
Error using path (line 71)
Out of memory. The likely cause is an infinite recursion within the program. 
> In prepPipeline (line 174)

Warning: The following error was caught while executing 'onCleanup' class destructor:
Error using restoreEEGOptions (line 43)
Cannot restore EEG options. Unknown error occurred.
 Backup options file
 /private/var/folders/xz/_cqvg2j14rvb3bs26x3kwpr80000gn/T/tpedfbb5fc_11c8_43e8_9e26_59c31c22a50a.m
 Current options file
 ~/eeg_options.m

Error in prepPipeline>cleanup (line 187)
   restoreEEGOptions(backupFile, currentFile);

Error in prepPipeline>@()cleanup(backupOptionsFile,currentOptionsFile,warningsState) (line 46)
    finishup = onCleanup(@() cleanup(backupOptionsFile, currentOptionsFile, ...

Error in onCleanup/delete (line 80)
            obj.task();

Error in prepPipeline (line 182)
end

Best,
Dawid

If reference or earlier step fails, post-processing should not occur. This must be fixed.

Also structure has both postprocess and postProcess fields

Hello! Using a version of the removeTrend function ported to Python, I noticed that frequencies below the detrendCutoff in removeTrend are not completely attenuated when using the local method: sappelhoff/pyprep#125

I was wondering if this is the desired output, since it isn't clear to me why the window size is computed by dividing 1.5 by the cutoff frequency:

Also, I tried to test the original MATLAB code, but I wasn't able to debug it myself. Any tips there would be appreciated.

Script:

%% simulate signal
duration = 120;
frequency = [0.2, 0.4, 0.8, 1.6, 3.2];
amplitude = [1, 2, 3, 2, 1];
srate = 200;

nSamples = round(duration * srate);
period = 1 / srate;
seconds = (1:nSamples).*period;
data = zeros(1, nSamples);
for i=1:length(frequency)
    data = data + amplitude(i) * sin(2 * pi * frequency(i) * seconds);
end
%% format input
EEG.data = data;
EEG.srate = srate;
detrendIn = struct('detrendChannels', 1, 'detrendType', 'linear', ...
                    'detrendCutoff', 0.5, 'detrendStepSize', 0.02, ...
                    'detrendCommand', []);
[EEG, detrendOut] = removeTrend(EEG, detrendIn);

Error:

testRemoveTrend

Unrecognized function or variable 'change_row_to_column'.

Error in localDetrend (line 35)
data = change_row_to_column(data);

Error in removeTrend (line 83)
        localDetrend(EEG.data(detrendOut.detrendChannels, :)', ...

Error in testRemoveTrend (line 30)
[EEG, detrendOut] = removeTrend(EEG, detrendIn);

Hi everyone, I have been working on a project that requires the preprocessing of EEG data. I would like to use one of the PyPrep noisy channel functions to mark bad channels, and I am testing different parameters within these functions.

How do you recommend I choose the best parameters for my EEG dataset? I would appreciate your insight on this question.

Hi there,

I think I've discovered a small bug in the re-referencing logic of the PREP pipeline, but I wanted to make sure it isn't intentional behaviour before modifying anything on my end.

In the above loop, iterations starts at zero and only get incremented at the end, after the "check if we should break the loop" logic. As a consequence, it means that the code actually does a maximum of 5 iterations by default, even though the default maximum is documented to be 4. Additionally, it means that PREP needs to do at least three iterations of this loop, even if the detected bad channels are unchanged between the first and second iterations. Looking at the original PREP publication, the pseudocode for this loop doesn't say anything about a minimum of iterations, so I'm wondering whether this is a bug or an intentional behaviour I don't understand.

Thanks in advance, and thank you for creating and maintaining this software!

Conflicts with the name in the signal processing toolbox.

Make it more obvious what the errors are when it fails.

PREP changes the option file to 'option_savetwofiles' disabled but it's not changed back when it is done.

Experimenter performs a 20Hz impedance test for a long time, which causes Prep to flag all channels as bad. Would like to be able to exclude this period from the counting.

It seems that EEGLAB uses this on the main menu. Something to do for next release

Thank you so much for your reply! I process my data with PREP again following your advices, step by step.
Veog channel's number is the 20. I still not clear with should I do the channel location firstly.

After the channel location process through eeglab edit. I ran Prep.



In China, the line frequency is 50hz

Excluded 20th channel under reference channnels and evaluation channels condition.

After the whole process, I went back to eeglab interface, which shows the data is still unrereference.

According to the introduction of PREP, I thought this process have completed the procedures for me:
1 high pass
2 line noise
3 rereference
So I just need to do a low pass process before ICA, without another rereference and other preprocesses?

I am still a amateur in processing EEG data, thx for your patience!

Hello,
I was looking at the code for the line noise removal and if I understand correctly you are doing the smooth average of two consecutive windows in this line where datafitwin should be the new window and datafitwin0 should be the old window. But in line 70 of the same function you already assign datafitwin to datafitwin0 so that the smooth average is computed with the first Noverlap samples and the last Noverlap samples both of the new window which would be wrong.
Here is the whole code snippet:

for n = 1:nw
        indx = winstart(n):(winstart(n) + Nwin - 1);
        datawin = data(indx);
         [datafitwin, f0Sig] = ...
            fitSignificantFrequencies(datawin, f0, lineNoise);
%         [datafitwin, f0Sig, FvalSig, aSig, fSig, sig] = ...
%             fitSignificantFrequencies(datawin, f0, lineNoise);
        datafitwin0 = datafitwin;
%         datafitWinSave(:, n) = datafitwin0;  % Debugging
%         f0SigSave{n} = f0Sig; % Debugging
%         FvalSigSave{n} = FvalSig;  % Debugging
%         aSigSave{n} = aSig;  % Debugging
%         fSigSave{n} = fSig;  % Debugging
%         sigSave{n} = sig;   % Debugging
        f0Mask = f0Mask | f0Sig;
        if n > 1
            datafitwin(1:Noverlap)= smooth.*datafitwin(1:Noverlap) + ...
                (1 - smooth).*datafitwin0((Nwin - Noverlap + 1):Nwin);
        end;
        datafit(indx, :) = datafitwin;
end

So I think the assignment to datafitwin0 should be after doing the smooth average.
I would appreciate if you could clarify this.

Hello,

I am getting an error when I run PrepPipeline0.55.4
Here is a snip of the aftermath:

The same errors were present when I had the cleanline plugin installed, except the source code path was directed at cleanline directly, rather than in the PrePipeline utilities.

I have set the path in Matlab to EEGlab and both the VisLab-EEG-Clean-Tools and PrepPipeline before using the function. According to the report, the PREP pipeline has failed at the step of line noise removal with the following error:

prepPipeline failed removeLineNoise: Error using testSignificantFrequencies
Undefined function 'finv' for input arguments of type 'double'.
Line noise method: clean

May I ask how to solve the problem? Thank you so much.

Warning: The following error was caught while executing 'onCleanup' class destructor:
Error using restoreEEGOptions (line 43)
Cannot restore EEG options. No matching files were found.
Backup options file
C:\Users\KAYROB~1\AppData\Local\Temp\eeg_options_backup.m
Current options file
C:\Users\Kay Robbins\eeg_options.m

Error in prepPipeline>cleanup (line 225)
restoreEEGOptions(backupFile, currentFile);

Error in prepPipeline>@()cleanup(backupOptionsFile,currentOptionsFile,warningsState) (line 46)
finishup = onCleanup(@() cleanup(backupOptionsFile, currentOptionsFile, ...

Error in onCleanup/delete (line 60)
h.task();

Error in prepPipeline (line 41)
computationTimes= struct( 'detrend', 0, 'lineNoise', 0, 'reference', 0);

Error in level2Study/createLevel2Study (line 385)
[EEG, computationTimes] = prepPipeline(EEG, params);

Error in runESSLARGPrepPipelineNoInterpolation (line 30)
obj2.createLevel2Study(ess2Dir, 'forceRedo', false, ...

In prepPipeline (line 41)
In level2Study/createLevel2Study (line 385)
In runESSLARGPrepPipelineNoInterpolation (line 30)

Prepline 0.55 crashes during publishPrepReport, whether I specify additional parameters or not (e.g., use all defaults).

One example of parameters that leads to a crash:
'summaryFilePath', strcat('C:\Users\data', filename, '_PREP.html') ,...
'sessionFilePath', strcat('C:\Users\data', filename, '_PREP.pdf') ,...
'consoleID', 1,...
'publishOn', true);

In any test runs, Matlab credits line 73 of publishPrepReport: "movefile(tempReportLocation, sessionReportLocation);" for the error.

The code will run to completion if I comment out that line or if I set relativeReportLocation (line 48) equal to sessionFilePath. However, only the .html file will be properly named. The .PDF will only be named "prepReport" and is overwritten if more than 1 subject is processed in a loop.

Hello,

I'm attempting the run the prep pipeline for a set of 72Hr long continuous EEG recordings I have collected.

After running Prep on my EEG file, I get the following error:

"prepPipeline failed performReference: Error using catdirs (line 28) Out of memory. The likely cause is an infinite recursion within the program."

Any idea why this might be the case? Has anyone else had this error?

I spotted 2 typos in the report publishing functions:

• in getErrors.m, line 16:
  ~isfield('noiseDetection', 'errors')
  should be:
  isfield(noiseDetection, 'errors')
• in publishPrepReport.m, line 33:
  if (nargin < 5)
  should be:
  if (nargin < 4)

Dear PREP group,

I am working with clinical EEG data and I am using the PREP pipeline as an early preprocessing step.
In some of my EEG files, a high number of channels get flagged as bad channels and get interpolated, e.g. 7 out of 21 in the example attached (see screenshot).
However, in the PREP summary report, the error status for these files is still labelled as "good" (see screenshot).

From the PREP reference paper (Bigdely-Shalmo et al., 2015) I understand that one of the criteria used to flag datasets with issues is if >25% of the channels get interpolated:
"The criteria we currently use to flag datasets with serious issues include: [...]
Robust referencing interpolated more than 25% of the reference channels."

I am wondering why this is not flagged up in the summary report (in the case of the file attached, 33% of the channels were interpolated)?

Thank you for any help you might be able to provide!

Best wishes,
Irene

Dear PREP pipeline group
I have scaned your paper and read the introduction to prep, I appreciate what you have done, but I still felt confuse when I use this plugin on EEGlab. I check all the parameter setting choice, and find none directing to montage.

All in all, I want to want should I do with my data, before I use the tool. should I complete channel location first? should I discard the Veog channel. I cannot find any glue about that on your website.

After the process, I cannot see any change through this EEGlab interface. no channel location, no reference.

Dear VisLab Team,

I'm a master student at the Berlin School of Mind and Brain and I'm currently using the PREP Pipeline to preprocess the EEG dataset of my thesis.
I'm using it as an EEGlab plugin and I'm extremely impressed by your work.
However, I'm constantly encountering the same MATLAB error:

Warning: [MATLAB:nonExistentField]
Reference to non-existent field 'referenceOut'.
Error in prepPipeline (line 184)
EEG = removeTrend(EEG, EEG.referenceOut);
Error in pop_prepPipeline (line 75)
EEG = prepPipeline(EEG, params);

At the end of the pipeline the pdf Report coherently shows:

Reference errors: [prepPipeline failed postProcess: Reference to non-
existent field 'referenceOut'.
]

Do you have any idea about the possible cause of this error?
I've tried to "dig" into the different subfunctions but I haven't managed to find a solution yet.

The thing that puzzle me the most is that, as far as I can judge, the pipeline seems to correctly perform robust averaging and it is even able to remove bad interpolated channels (if specified).
But because of the aforementioned error messages, I don't know whether I can trust the completion of the process. What do you think?

Many thanks for your time and consideration.
Best regards,

Alberto Mariola

I am attempting to run PREP mainly for the service of bad channel identification, removal, and interpolation. When I attempt to do this via GUI or command line on my 40 channel raw .set files, it does not seem to be working properly.

Within the matlab command window, I can see the plugin is doing things, but in the end, bad channels are not removed or interpolated (despite my checking that box in postprocessing or setting params.removeInterpolatedChannel = true). What am I missing?

Is there any example code I could see, or should the runVEPPrepPipeline in examples be sufficient?

Might there be some data prerequisite I am missing?

Anything you can do to help me would be greatly appreciated.

Many thanks!
Spencer Fix

Cancel button doesn't cancel....

I run PREP on a 32 channel EEG data where many sets have 1-3 noisy channels. Since I want to continue with a matrix of full rank, I set 'removeInterpolatedChannels' to true. However if I do so, PREP seems to remove also channels that it does not note as interpolated in the report (and accordingly in EEG.etc.noiseDetection.reference.interpolatedChannels.all). I've seen this in multiple subjects and the only pattern I could recognize is that it seems to remove 2x the amount of channels that it tracks as interpolated (e.g. leaving me with 26 channels when 3 channels had been interpolated according to the report). Which channels are removed on top of the interpolated ones, seems sort of arbitrary to me (but most likely it isn't). Do you have any idea what could cause this issue? Is it a bug or a user-mistake?
If I set 'removeInterpolatedChannels' to false, it seems to work fine. The rank of the new EEG.data is then 32-nic-1 (nic = number of interpolated channels) which makes sense to me since it's now referenced to average (therefore the -1).
As a workaround, I'll keep it this way and remove the interpolated channels manually. I'm just unsure whether there's something else wrong with my usage of PREP. (Which is a great tool btw!)
These are the paramters that I used:

EEG = pop_prepPipeline(EEG,struct('ignoreBoundaryEvents', true, ...
    'referenceChannels', [1:4, 6:26, 28:32], ...
    'evaluationChannels',[1:4, 6:26, 28:32], ...
    'rereferencedChannels', [1:32], ...
    'ransacOff', false, ...
    'ransacSampleSize', 50, ...
    'ransacChannelFraction', 0.25, ...
    'ransacCorrelationThreshold', 0.75, ...
    'ransacUnbrokenTime', 0.4, ...
    'ransacWindowSeconds', 5, ...
    'srate', 250, ...
    'robustDeviationThreshold', 5, ...
    'correlationWindowSeconds', 1, ...
    'highFrequencyNoiseThreshold', 5, ...
    'correlationThreshold', 0.3, ...
    'badTimeThreshold', 0.01, ...
    'maxReferenceIterations', 4, ...
    'referenceType', 'Robust', ...
    'reportingLevel', 'Verbose', ...
    'interpolationOrder', 'Post-reference', ....
    'meanEstimateType', 'Median', ...
    'samples', EEG.pnts, ....
    'lineNoiseChannels', [1:32], ...
    'lineFrequencies', [50  100], ...
    'Fs', 250, ...
    'p', 0.01, ....
    'fScanBandWidth', 2, ...
    'taperBandWidth', 2, ...
    'taperWindowSize', 4, ...
    'pad', 0, ...
    'taperWindowStep', 1, ...
    'fPassBand', [0  125], ...
    'tau', 100, ...
    'maximumIterations', 10, ...
    'cleanupReference', false, ...
    'keepFiltered', true, ...
    'removeInterpolatedChannels', true, ...
    'reportMode', 'normal', ...
    'publishOn', true, ...
    'sessionFilePath', strcat(path_reports, filename,'_Report.pdf'), ...
    'summaryFilePath', strcat(path_reports, filename,'_Summary.html'), ...
    'consoleFID', 1));

Hi,
I just tried out performReference.m using the 'pre-reference' mode.
Matlab throws an error at line 102:
Reference to non-existent field 'interpoledChannels'.

I guess this is just a typo and should be changed to interpolatedChannels.

Best,
Wanja

GUI portion seems to give error, probably to MATLAB's change in figure representation. We need to address.

num2str must have a numerical input in some versions of matlab (Line 38) causing error

Dear PREP pipeline team

I have a question regarding the interpolation that happens based on robust average referencing.

Many researchers hold the view that interpolation of channels should not happen prior to ICA. What is your opinion about not interpolating prior to running ICA analysis? Does it differ from the rather conventional view as you implement interpolation prior to ICA, and if so, why?

Hence, it seems like I should remove interpolation in the post-process menu of PREP, would you agree?

Thank you in advance!

Was reading through the PREP re-referencing code while working on PyPREP, and noticed something that looks to be a bug in updateBadChannels.m:

Specifically, dropout channels aren't included in the ref.all union at the end, meaning that they're never excluded/interpolated during re-referencing. Additionally, ref.badChannelsFromRansac is included in the union twice, which makes it look like bad-by-dropout channels were meant to be included but got copy/pasted incorrectly.

Thanks in advance for taking a look!

Hello, I was using the pipeline then did ICA, epoching and wanted to do further artifact rejection, which requires chanlocs.labels, but channel information seems removed after the pipeline. The original channel number was 132, it became 95 after the interpolation, but is there a way to know which channels got removed so I still have channel location information for artifact rejection and further analyses? Thank you!

When running the pipeline with the parameter removeInterpolatedChannels set to true, and then trying to publish the report, prepReport can still try to access the removed channels (e.g. line 87 to plot the spectrum) leading to an error.

Hi,
as far as I could see it, there is currently no way to add back the online reference during the pipeline. In order to base the average reference on N + onlineRef channels I simply added onlineRef to params.referenceChannels and entering N in params.evaluationChannels. onlineRef is just a vector of zeros. The pipeline stopped with a matrix dimension mismatch error. This was because PREP checks also referenceChannels to be usable (in findUnusableChannels(signal, referenceOut.referenceChannels)), which is a good idea, but will at the end produce the error when reindexing channels that are not bad and one or more reference channels that are found to be bad but do not occur in evaluation channel list.

signal.data(targetChannels, :) = signaltemp.data(reindex, :); line 53 of interpolateChannels

I think simply replacing referenceChannels by evaluationChannels in findUnusableChannels is not a good idea as the reference signal should be clean.

Is there another to do it?
Best, Norman