the-lammert-lab / tinnitus-reconstruction

Reconstruct high-dimensional spectral representations of tinnitus using reverse correlation

Tinnitus Reconstruction via reverse correlation

Reconstructing high-dimensional representations of tinnitus using reverse correlation and compressed sensing.

Installing

For most users, it is best to install the MATLAB toolbox from the latest Release. For development, clone the following projects:

• mtools
• tinnitus-reconstruction
• ReadYAML (only for legacy use)
• yaml

Then add the functions to your path. The commands should look similar to this:

addpath ~/code/yaml
addpath ~/code/srinivas.gs_mtools/src
savepath

Finally, in the tinnitus-reconstruction/code directory, run setup.m as a MATLAB script.

Quickstart

Installing

Install the latest Release.

Running the experiments

There are multiple experiments available in this repository. All experiments can be found in the tinnitus-reconstruction/code/experiment/Protocols directory. The published, novel experimental protocol is in RevCorr.m.

To run any experiment, copy the template configuration file found here and modify it as you see fit.

Then, using a decibel meter (many are available for smartphones), record the decibel output through the headphones of sound that plays after running the following function:

play_calibration_sound()

Raise your system volume until the measured dB value is above 95dB. Save this value in the MATLAB workspace. For example cal_dB = 97.8; Don't worry, no sounds will be played above 65 dB unless made to do so by the user.

Then, run any of the Protocol functions with this measured dB value in your MATLAB prompt.

ThresholdDetermination(cal_dB)
LoudnessMatch(cal_dB)
PitchMatch(cal_dB)
RevCorr(cal_dB)

The function RunAllExp(cal_dB) uses one config file to run the protocol functions in the above order, repeating PitchMatch 3 times by default.

Results

To inspect reverse correlation results from AX (template sound) experiments, run:

pilot_reconstruction
reconstruction_viz

Use patient_reconstructions to inspect results from non-AX experiments.

Data can be collected from the other protocols by running

collect_data_pitch_match()
collect_data_thresh_or_loud()

Configuration files

An experiment is specified by a configuration file. These are YAML file that generally live in tinnitus-reconstruction/code/experiment/configs/, but you can put them anywhere.

A template config file can be found here. This template file includes inline comments that describe a sample configuration file and what fields are available.

In brief, the following fields are required to run a reverse correlation experiment:

• stimuli_type
• n_trials_per_block
• n_blocks
• subjectID

To run a threshold determination, loudness matching, or pitch matching experiment, the following two fields are additionally required.

• min_tone_freq
• max_tone_freq

Note: stimuli_type, n_trials_per_block, and n_blocks have no effect on non-reverse correlation protocols, but are still required to exist in a config file. Similarly, min_tone_freq and max_tone_freq do not interact with reverse correlation protocols and are not required in a config file.

Other config fields include:

• data_dir: path to directory where output files from the experiment should be saved
• stimuli_save_type: either waveform, spectrum, or bins. Determines in what form data from the experiment should be saved.
• various stimuli parameters including:
  • min_freq: the minimum frequency (in Hz) of the stimuli
  • max_freq: the maximum frequency (in Hz) of the stimuli
  • n_bins: how many tonotopic bins the stimuli has
  • duration: the duration of the stimuli (in seconds)
• stimuli hyperparameters specific to each stimulus generation type (see the stimuli class definitions for details)

You can load a config file into memory:

config = parse_config('path/to/config_file.yaml');

and generate a stimulus generation object from the config:

stimgen = eval([char(config.stimuli_type), 'StimulusGeneration()']);

To set parameters in a stimulus generation object, use the from_config() method, which takes a path or a config struct:

stimgen = GaussianPriorStimulusGeneration();
stimgen = stimgen.from_config('path/to/config_file.yaml'); % stimgen.from_config(config); 

You can generate a serialized experiment ID via:

expID = get_experiment_ID(config);

Or a hash via:

this_hash = get_hash(config);

Stimulus generation methods

Stimulus generation classes are defined here. All stimulus generation classes inherit from AbstractStimulusGenerationMethod. Non-binned classes inherit directly and binned classes through the intermediate class AbstractBinnedStimulusGeneratioMethod.

AbstractStimulusGenerationMethod

This class defines common properties of all stimulus generation methods. They are described here in brief with default values, e.g., property_name = default_value. They include:

• min_freq = 100: the minimum frequency of stimuli generated by this method (Hz).
• max_freq = 22000: the maximum frequency of stimuli generated by this method (Hz).
• duration = 0.5: the duration of stimuli generation by this method (seconds).
• n_trials = 100: the number of trials in a single block.
• Fs = 44100: the sample frequency of the stimuli generated by this method (samples/sec).
• nfft = duration*Fs: Dependant property, the number of fast Fourier transform points.

Methods:

• [y, spect, binned_repr] = subject_selection_process(self, signal)
• [stimuli_matrix, Fs, spect_matrix, binned_repr_matrix] = generate_stimuli_matrix(self)
• freq = get_freq(self)
• self = from_config(self, options)
• stim = synthesize_audio(X, nfft)

AbstractBinnedStimulusGenerationMethod

This abstract class includes three additional properties:

• n_bins = 100: The number of bins to use to represent the frequency axis.
• unfilled_dB = -100: The value of a bin labeled "unfilled" (dB).
• filled_dB = 0: The value of a bin labeled "filled" (dB)

Methods:

• [y, spect, binned_repr] = subject_selection_process(self,representation)
• [binnum, Fs, nfft, frequency_vector] = get_freq_bins(self)
• spect = get_empty_spectrum(self)
• binned_repr = spect2binnedrepr(self, T)
• T = binnedrepr2spect(self, binned_repr)
• [wav, X, binned_rep] = binnedrepr2wav(self, binned_rep, mult, binrange, new_n_bins, options)
• W = bin_signal(self, W, Fs)

StimulusGeneration classes

Any StimulusGeneration class such as GaussianPriorStimulusGeneration includes the generate_stimulus method.

[stim, Fs, X, binned_repr] = generate_stimulus(self)

Running an experiment

The RevCorr, ThresholdDetermination, LoudnessMatch, and PitchMatch functions run an experiment. You can invoke any of them in two ways:

RevCorr(cal_dB)
RevCorr(cal_dB, 'config', 'path_to_config_file')

In the first case, a dialog box opens and asks you to select a .yaml config file. In the second case, you specify the path to a config file directly. The cal_dB parameter is described above.

Collecting data

Data are saved in config.data_dir which is usually tinnitus-reconstruction/code/experiment/Data. For RevCorr, each block has a separate stimuli and response file saved for it, labeled by the subject ID and a unique hash. For the other experiments, each experiment has separate stimuli and responses saved and similarly marked.

You can use the collect_data functions to gather the data into output matrices.

[responses, stimuli] = collect_data('config', 'path_to_config_file');
[responses, stimuli, octave_responses, octave_stimuli] = collect_data_pitch_match('config', 'path_to_config_file')
[dBs, tones] = collect_data_thresh_or_loud('loudness', 'config', 'path_to_config_file')
[dBs, tones] = collect_data_thresh_or_loud('threshold', 'config', 'path_to_config_file')

Tinnitus representation reconstruction

You can use compressed sensing (cs), compressed sensing without a basis (cs_no_basis), linear regression (gs) or ridge regression (gs('ridge',true)).

To generate a reconstruction with one of these methods, pass it as a name-value argument with 'method':

[x, responses_output, stimuli_matrix_output] = get_reconstruction('config', config, 'method', 'cs');
[x, responses_output, stimuli_matrix_output] = get_reconstruction('config', config, 'method', 'linear');
[x, responses_output, stimuli_matrix_output] = get_reconstruction('config_file', 'path/to/config/file.yaml', 'method', 'cs_ridge');
[x, responses_output, stimuli_matrix_output] = get_reconstruction('config_file', 'path/to/config/file.yaml', 'method', 'linear_ridge');

Citation

@article{Hoyland2023,
	author={Hoyland, Alec and Barnett, Nelson V. and Roop, Benjamin W. and Alexandrou, Danae and Caplan, Myah and Mills, Jacob and Parrell, Benjamin and Chari, Divya A. and Lammert, Adam C.},
	journal={IEEE Open Journal of Engineering in Medicine and Biology}, 
	title={Reverse Correlation Uncovers More Complete Tinnitus Spectra}, 
	year={2023},
	volume={4},
	number={},
	pages={116-118},
	doi={10.1109/OJEMB.2023.3275051}
}

tinnitus-reconstruction's Issues

Fix AL's data post-processing (binning)

@alammert 's data was saved as spectra and then is post-processed to bins, but is adding high-frequency bin results that aren't real.

Reconstruction for 2-AFC experiment

Compare resynth results to resynth ground truth

Protocol code freeze

• Confirm that protocol works correctly
• Write template config file
• Freeze protocol code
• Create new release

Write documentation for pilot reconstructions

Add "repeat" button to experiment

Add button or key to repeat the sounds without recording any data.

Update all configs to use `target_signal_name` and update hashes

For each config file that we used to collect data:

1. Compute the hash of the vanilla config file.
2. Rename/add the target_signal_name and target_signal_filepath properties.
3. Compute the new hash for the modified config file.
4. Use the update_hash() function to update the hashes of the data files to match.

Better integration of PowerDistribution stimulus generation

PowerDistribution stimulus generation doesn't work like the others, since it requires loading a .mat or .csv file to get the distribution field. We should incorporate logic to do this automatically into stimgen.from_config(config) so that you can specify the filepath in the config file and have it load it at the same time as assigning values to all the other fields.

Add documentation for `generate_stimuli_matrix`

https://github.com/alec-hoyland/NSF-proposal/blob/4bb2dd70b8114393dd08b6b96c97d281a383a472/code/stimulus_generation/@AbstractStimulusGenerationMethod/AbstractStimulusGenerationMethod.m#L46

2AFC protocol

Write a new protocol function that implements 2-AFC task, where two stimuli are provided to the subject in an ABAC paradigm and then the subject rates which one (B or C) is closer to the target signal (A).

Number of completed trials not updating

In the Protocol, the number of completed trials is not updating after each block.

Filenames are too long

Use DataHash to generate a hash from the config struct contents and use that to name files.

Add `read_stimuli` method

% TODO

https://github.com/alec-hoyland/NSF-proposal/blob/ab80b98a1702e82da21d4efdc8002c940a167b20/code/utils/read_stimuli.m#L1

Results and conclusion sections

automate generating the subjectID

# TODO: automate generating the subjectID

https://github.com/alec-hoyland/NSF-proposal/blob/894a3b509cacc7697bcb54c70172e09e3811113d/code/experiment/configs/config_template.yaml#L18

Documentation using mkdocs

Create documentation like in the xolotl project by using mkdocs. This requires formatting each docstring in Markdown and then running a processor in Python to preprocess the docstrings and then build the documentation. The result is interactive documentation that can be webhosted.

Write function to speed up deployment

Get target audio from config file in a better way

Currently, the target audio name (e.g., buzzing) is extracted from the config file by looking at the path where the target audio is stored. This is bad for two reasons:

1. There is no standard for how to name these paths.
2. We need to specify either a standard for how to name the paths, or have a finite list of keywords to search for.

Instead, we should define what the target signal directly in the config file.
This does not change the fact that we still need an interim solution for the data already collected.

This issue is complete when the standard is updated in template config files and when existing data/config file pairs are modified to fit the new standard.

Rebuild paper with new data

Hashing breaks data gathering

Update data gathering functions to use new file naming paradigm

Write methods section

Why are reconstructed waveforms short?

Collect data

• Collect data for buzzing from @nelson-barnett
• Collect data for roaring from @nelson-barnett
• Collect data for screeching from @nelson-barnett
• Collect data for buzzing from @BenWR1124
• Collect data for roaring from @BenWR1124
• Collect data for screeching from @BenWR1124
• Collect data for buzzing from @alec-hoyland
• Collect data for roaring from @alec-hoyland
• Collect data for screeching from @alec-hoyland
• Collect data for buzzing from @alammert
• Collect data for roaring from @alammert
• Collect data for screeching from @alammert

Disordered output table from reconstructions

Automated release building

• Testing
• Rebuilding

How to do this?

Documentation for config files

Write a template config file that contains default values and also detailed explanation for what's each field should be. Also provide documentation for parse_config.

Create standalone deployment

collect_data fails to collect with arbitrary config file

Hi Alec,

So this is the command I am running,

[responses,stimuli]=collect_data('config','C:\Users\saman\OneDrive\Documents\GitHub\tinnitus-project\code\experiment\configs\config_custom.yaml')

And I have also attached the config file I am using. I had to go back and add the data_dir field so it could look in the correct place. I am also attaching what I am seeing in my data folder.

Config File:

subjectID: pilot&&subject=SJP&&stimuli_type=custom&&n_bins_filled_mean=20&&n_bins_filled_var=3
data_dir: '/home/alec/code/tinnitus-project/code/experiment/Data'
n_trials_per_block: 100
n_blocks: 10
total_trials: 1000
min_freq: 100
max_freq: 22000
n_bins: 100
n_bins_filled_mean: 20

Add option to binify the target signals in protocol

We should be able to choose to bin-ify the target signals for an A-X experiment.

Automatic config subject id generation

Create subject ids automatically based on the content of the config file.

Pilot reconstruction script sanity checking

• Run a baseline in-silico experiment to confirm zero correlation (i.e. r^2 ~ 0) when the responses are totally random.
• Plot reconstructions from the in-silico method (on the same plot as real results).
• Add a new column counting yesses vs noes for real data in the pilot reconstruction script.
• Write documentation for the script

Protocol bug re: string and double comparisons

Use `yaml` package instead of `ReadYAML`

The yaml package allows for reading and writing of yaml files, whereas the ReadYAML package only implements reading.

Run hyperparameter sweep

Error after selecting config files

Tried to run program by typing "Protocol"
Manually selected config file, test and test 2, and got the following error messages:

May be problem with config files, but I wanted to make sure

`ABSG.bin_signal` makes state changes it shouldn't

AbstractBinnedStimulusGeneration.bin_signal() changes the duration without warning the user. This is unexpected behavior that introduces bugs when nfft is no longer even.

Rebuild figures

1. Compute the reconstructions.
2. Rebuild the figures.
3. Update the paper.

Data validity checks for `collect_data` for 2AFC regime

Write data validity checks for the 2AFC regime. Currently they are disabled because they don't account for there being twice as many stimulus files (the tests except a 1:1 correspondence).

Brimijoin 1st bin always zero

@alammert reported that the first bin of Brimijoin stimuli is always zero. For this issue, try to reproduce the bug and then correct it.

Resynthesize audio from reconstructions

Write a one-liner into the reconstructions script that resynthesizes and saves a waveform of the reconstruction.

Print the number of trials completed when closing the experiment

Frequency bin generation for when max_freq does not equal Fs

Thanks, but this looks wrong to me. We’re really digging into DSP now.

Some facts: The spectrum should always have length nfft/2. And, its elements should always represent linearly-spaced frequencies from 0 to Fs/2. The only thing that would chanve these facts would be a change in Fs (which we won’t allow) or a change in bin_dur. The latter would still only change the spectrum length via nfft, but not the range of frequencies.

Some theory: Thus, changing the duration of a time domain signal changes the resolution of its frequency-domain representation, but not the range of frequencies that can be represented in the signal. Changing the range of frequencies can only be done by changing Fs. If you shorten the duration of the signal, what changes is the spacing between the frequencies associated with each element of the spectrum, not the range of frequencies.

Practically: Now, what we want is that the elements of the spectrum that have some power (>-100dB) are only those between minfreq and maxfreq. All others are -100dB. Between minfreq and maxfreq, the elements of the spectrum (that are associated with linearly-spaced frequencies) are grouped into n_bins number of logarithmic bins. Each element in a bin is assigned the same power value, as you know.

Figure 1

The first figure should be a diagram explaining the experimental protocol.

Webhost experiment

Figure out how to webhost the experiment so that it's easier for people to perform it.

• Maybe have an interactive config creator?
• Streamlit if Python, otherwise maybe host on WPI servers with MATLAB in the browser?
• Docker?

Debug BWR's 2-AFC data

@BenWR1124 's data isn't showing up in the pilot_reconstructions script. Find out what's wrong and fix it.

Rewrite pilot reconstructions script

The recent changes to the config structure has broken this script. It's a useful script. We should fix it.

Prevent mouse-clicks from making a decision

Mouse clicks will make a decision in the experiments, which is not what we want.

Figure 2

The second figure should show the reconstructions for the three target signals vs. the ground truths.

Write function to update hashes for config file/data pairs

Sometimes it is necessary to post-facto modify a config file. This occurs when we realize we need a new field to specify some feature of the data without knowing before the data are generated. Since data (via their filenames) are associated with the hashes of config files, when the config file changes, the config file and data become out-of-sync.

The solution to this is to write a function that updates the filenames of associated data files by taking in the old and new hashes for a modified config file.

ARO abstract