Common setup can erase raw data about neuropixel-utils

[djoshea]

Interesting, hadn't thought of this. What's the method you're calling that you're seeing this with? In theory it's only clearing files in the same directory with the same file "stem" (meaning everything up to but not including .imec.ap.bin). If this matches your raw data, you're trying to overwrite your raw data in place, which can't be done using writeConcatenatedFileMatchGains or saveTranformedDataset. These are designed to be saved only into a new path, or at least a new file stem. The reason being that you can't read from the file as it's being overwritten. So perhaps the correct behavior is to throw an error if outPath would match the current file.

Incidentally, if you're trying to overwrite the raw data in place (because of limited disk space), you need to use imec.modifyAPInPlace instead.

What's the call you're making to help diagnose this?

[etrautmann]

There are two use cases where I've run into this:

1. I'm currently extracting the sync channel and saving that as a .mat file in the output directory. The writeConcatenatedFileMatchGains() method clears the output folder and deletes that file before writing the transformed .bin file.

2. writing a modified file in the same directory as the original data with a modified name (perhaps most relevant for new users testing the code). For example, before I've fully groked the usage model, I might want to test the code and transform somefile.bin by making a somefile_CAR.bin in the same folder, which currently doesn't work because the stem directory is cleared before writing the transformed file.