djoshea / neuropixel-utils

Utilities for managing Neuropixel datasets and Kilosort results

neuropixel-utils

Neuropixel Utils is a toolkit written in Matlab for manipulating datasets collected by SpikeGLX (e.g. imec.ap.bin files) and the results produced by Kilosort / Kilosort 2. Please note that some of this functionality is redundant with the tools found in the Cortex Lab's spikes repository, authored By Nick Steinmetz, Mush Okun, and others. Here, we prioritize an organized, easy to use, object-oriented approach to accessing, manipulating, and visualizing the data. This reduces the need to worry about metadata.

See full documentation at https://djoshea.github.io/neuropixel-utils.

Neuropixel Utils facilitates the following data processing steps:

• Load and visualize raw neuropixel data from imec.ap.bin and imec.lf.bin files in Matlab
• Write custom pre-processing functions to apply to raw data either by writing a copy of the raw file or modifying it in place, optionally removing specific problematic time windows in the file
• Concatenate multiple Imec data files together while matching the amplifier gains
• Run Kilosort/Kilosort2, and load the results back into Matlab after manual inspection in Phy
• Plot drift maps using code adapted from the spikes repository
• Extract waveforms for each cluster from the raw data, optionally cleaning the snippets by subtracting templates for other clusters spiking during the same time window
• Visualize cluster electrical spiking images in space and cluster locations on the probe
• Determine trial boundaries in the file, and efficiently segment Kilosort results into individual trials

Neuropixel Utils was authored by Daniel J O'Shea (@djoshea) to facilitate precision artifact removal and careful inspection of raw data traces before running Kilosort, as well as post-hoc verification that the artifacts were removed successfully.

Download and install

To get started, clone the repo:

git clone https://github.com/djoshea/neuropixel-utils.git

And add it to your path in Matlab:

>> addpath('/path/to/neuropixel-utils')

Requirements

Neuropixel Utils requires Matlab R2019b or later.

neuropixel-utils's Issues

System requirements?

Could you add something to the readme and the GitHub Pages saying which versions of Matlab are supported, and if there are any other system requirements for running neuropixel-utils?

typo in function name

ImecDataset.saveTranformedDataset(...) is missing the s in transformed and should be
ImecDataset.saveTransformedDataset(...)

Issue with concatenating LF

An error was returned when concatenating files LF but not AP data using Neuropixel.ImecDataset.writeConcatenatedFileMatchGains. There was no field of 'badChannels' in the LFmeta.

I changed the union of badChannels with the 'if p.Results.writeLF ...' statement to the following and it works for what I have been doing but there is probably a more logical place to define the meta data.

% compute union of badChannels
for iM = 2:numel(imecList)
if ~isfield(meta, 'badChannels')
meta.badChannels = [];
else
meta.badChannels = union(meta.badChannels, imecList{iM}.badChannels);
end
end

I define the badChannels after concatenating all my Imec datasets. Should I be populating that field prior to concatenating?

Missing link in imec_dataset GH page

In https://djoshea.github.io/neuropixel-utils/imec_dataset/, at the text "If you have known trial boundaries in your file (see for more information), you can ", looks like the "see for" is missing a link that should be in there.

neuropixel-utils/imec_dataset/index.html

Line 896 in cad37c8

<p>If you have known trial boundaries in your file (see for more information), you can also excise the regions of time far from trial boundaries using the <code>TrialSegmentationInfo</code> instance. I&rsquo;ve found this to be useful to exclude time windows where the subject was asleep from further analysis.</p>

Error loading an imec Dataset , .imec..bin gets added to filename

Hey,

I saw on your page that it is possible to concatenate multiple files together, which is a great tool which I would love to use for a part of my dataset. I would like to follow specific cells through different recordings and the most reliable way would be by sorting the files together.
Unfortunately I already have problem with loading the first file.
I made a copy of the first file in a safety folder, I also made sure that only the ap.bin file and the meta file is in this folder.
When I try to load the file I get the following error message
imec = Neuropixel.ImecDataset('Z:\Users\Janna\IMECDATASET\20200402\001\NP\20200402_JA0033_Light_g0_t0.imec0.ap.bin');
Error using Neuropixel.ImecDataset (line 109)
Could not find AP bin file Z:\Users\Janna\IMECDATASET\20200402\001\NP\20200402_JA0033_Light_g0_t0.imec0.ap.bin.imec..bin

if I change the name of the file by just erasing the "0" the following error message appears:

imec = Neuropixel.ImecDataset('Z:\Users\Janna\IMECDATASET\20200402\001\NP\20200402_JA0033_Light_g0_t0.imec.ap.bin');
Reference to non-existent field 'imHpFlt'.

Error in Neuropixel.ImecDataset/readInfo (line 128)
df.highPassFilterHz = meta.imHpFlt;

Error in Neuropixel.ImecDataset (line 107)
df.readInfo();

As another step I also tried just a common prefix, this is the error message I get for that
imec = Neuropixel.ImecDataset('Z:\Users\Janna\IMECDATASET\20200402\001\NP\20200402_JA0033_Light');
Error using Neuropixel.ImecDataset.findImecFileInDir (line 1262)
No ap matches for Z:\Users\Janna\IMECDATASET\20200402\001\NP\20200402_JA0033_Light* exist

Error in Neuropixel.ImecDataset (line 98)
file = Neuropixel.ImecDataset.findImecFileInDir(fileOrFileStem, 'ap');

I am not sure if this is the right channel, I don't really think that the "0" behind the imec file name is the problem, but I also don't understand why the code is adding another ".imec..bin" to the full filename.

Do I have to add something or take away to load the dataset ? Or does it have something to do with the additional number behind the "imec" and that is why the code doesn't recognize the file ?

Best,
Janna

meta.snsShankMap may not be available if only writing lf.bin

If I only process and write lf.bin with imec.saveTransformedDataset, an error will occur when creating the ImecDataset object for the output file (line 2700).
Tracking the stack down led me here, where meta.snsShankMap is not available from the lf.meta.

neuropixel-utils/+Neuropixel/ImecDataset.m

Line 241 in 505b1bf

imec.snsShankMap = meta.snsShankMap;

Just FYI~

Common setup can erase raw data

If the user wants to write a filtered/CAR'd version of a file into the same directory containing the raw data (as most users first testing this will likely attempt), line 1398 of ImecDataset.m will clear this folder, deleting the raw data.

I'm guessing the assumed configuration is to specify an output path which isn't the same as the input, though I think most users will discover this the hard way.

Outdated function reference?

neuropixel-utils/+Neuropixel/KilosortMetrics.m

Line 1133 in ae56e18

TrialDataUtilities.Plotting.setMarkerOpacity(h, alpha);

This seems like an outdated function reference, and the new one is probably "Neuropixel.Utils.setLineOpacity"?

(Thank you for this awesome toolbox! I have written my own, also oop, toolkit for SpikeGLX and Kilosort1/2, but once seeing this beautifully designed and professionally written package I immediately threw mine away haha!!)

Output metadata is incorrect

Output files misspell at least these tags {~imroTbl, ~snsChanMap, ~snsShankMap}. This causes subsequent pipeline tools to crash.

Error while loading binaries from SpikeGLX (Release v.20190327) - port numbers on imec PXI card

A small error appears while loading "imec.ap.bin". This could be due to the fact that the latest release of SpikeGLX change the file name to "imecX.ap.bin" where X is the probe port number on the imec PXI card.

error on loading and working with kilosort dataset

Hi,

I am wondering if neuropixel utils is compatible with kilosort 2.5 and kilosort 3? I am able to load my kilosort 2.5 sorted data into the KilosortDataset object. However, I don't have a RawDataset and many other fields in my structure. Here's a picture for reference. Any thoughts on this?

Thanks so much!

turbo colormap not included

I get the following error trying to plot cluster waveforms - looks like a dependency on this function:
https://www.mathworks.com/matlabcentral/fileexchange/74662-turbo.

Might consider changing to a different default colormap or including turbo in this repo.

Unrecognized function or variable 'turbomap'.

Error in Neuropixel.KilosortMetrics/plotClusterWaveformAtCentroid (line 2206)
colormap = turbomap(numel(clusterInds));

error while trying to exclude specific time window

Hello,
I have been trying to excise time windows from the raw data using the timeShifts function with the following commands:

idxStartList=100;
idxStopList=round( imec.nSamplesAP/100 ) ;
timeShifts = Neuropixel.TimeShiftSpec.buildToExciseGaps(idxStartList, idxStopList);

[imecOut] = imec.saveTranformedDataset(cleanedPath, 'transformAP', { @Timeshifts },'writeAP', true, 'writeLF', false);

and got the following error:

Error in
Neuropixel.ImecDataset/writeConcatenatedFileMatchGains/writeCatFile
(line 1605)
data = fn(imecList{iF}, data,
chIds, source_idx);

Error in Neuropixel.ImecDataset.writeConcatenatedFileMatchGains
(line 1457)
writeCatFile(outFile, chIndsByFile, 'ap',
multipliers, chunkSize, p.Results.transformAP,
timeShiftsAP, dryRun);

Error in Neuropixel.ImecDataset/saveTranformedDataset (line 991)
imecOut =
Neuropixel.ImecDataset.writeConcatenatedFileMatchGains({df},
outPath, p.Results);

I'd appreciate your help with understanding what I've been doing wrong.
Many thanks,
Ido Maor