iusca / iusm-connpipe Goto Github PK

I am particularly concerned about the "apply_regressors" section.
There are various combinations of regressors and we need to test them all:

Level 1: Head Motion Regression vs AROMA
Level 2: aCompCore vs Physical regressors
Level 3: Global Signal regression (yes/no)
Level 4: Scrubbing

Jenya's comment in the MRtrix Streamline Tractography section:

tournier is one of several algorith options; best choice can vary depending on data quality (i.e. number of shells and directions)

Many of the QC features that we've created were not available at the time that we processed the HASDA data.
We'd like to run some portions of the pipeline to generate some of the QC statistics, without having to reprocess the data.

Matt and Andrea to work on this.

We are still trying to decide what format to use (i.e. a downloadable PDF or just stick to the README file and a wiki, using Githug's resources).

This would have to be done after completing T1_prepare_A. Perhaps as a pre-processing option for T1_Prepare_B, or a "post_T1_prepare_A".
The overlap mask needs to be renamed, and filled, then re-extraction needs to be done.

This should all be clearly documented in the QC log to make sure we know that the overlap mask was used as the brain mask.

There are cases where FSL won't output a motionRegressor_fd.txt file.

From nipy/nipype#1427:
"it looks like fsl_motion_outliers is not guaranteed to create an output text file. If the NIfTI image contains no timepoints above the threshold, fsl_motion_outliers will terminate with no output."

So, it seems like we can't rely on the file being generated. Probably a better logic is the following: if the metrics and FD and DVARS plots exist, but motionRegressor_fd.txt does NOT exist, then that means it ran but did not save output.

Therefore, the subject is a low-motion subject (with respect to the threshold), and code should NOT crash.

Incorporate new changes to script and add new parcellation options

PAline is not getting added to the acqparams.txt file. Seems like the number of B0 indices is not being correctly computed.
Check Matlab code f_preproc_DWI, line 254, where the PAcount is computed. Looking at Mario's example output, this PAcount value should be 9 but I am getting 0.

Revise Bandpass script; latest version is in hasda.

Current test data has a longer TR and no fieldmaps so some sections of the code have not been tested as they don't apply to this data.

Rachel identified some issues with the Demean and Detrend script. Code may have a line that was originally in the old version of the pipeline but is no longer needed.

Some error/warning messages need to be updated to reflect changes we've made in the pipeline and config file.

Currently, the denoising configuration variables are not available to T1_B without running T1_A.

The final stage of fMRI_A is to generate time series for each ROI. Currently, we save the NxT matrix as a.npz file because it is generated with python but many Matlab users will probably want to do analysis in Matlab so we should also save it as a .mat file, or at least as a .csv file that can be imported into matlab.

Mario has made many changes to the Unwarping section of the pipeline to make it more flexible to work with GRE filedmap data.

This changes affect the pipeline for all types of data; new flags have been introduced too. So, the code needs to be updated but we'll wait for the FC section to be completed and then we'll go back and revisit any changes/updates.

I found a cleaner way to pass variables to python scripts.
Use example in fMRI_A_EPI_ROIs.sh script and update all calls to pythons scripts.

Ideally, we'd want to provide the user with as much detail as possible of why the code crashed in the event that it does crash. We can possibly suggest what flags the user should revise in the config file... we already do this for some sections but it would be good to identify what errors cause the code to cash at various steps of the pipeline; i.e. what files are needed to run AROMA to avoid a crash? If they aren't present, then return a detailed error to user saying which files are missing.

Currently we used the defaults for ANTS - if there is a way to set (increase, preferably) the number of iterations, we'd want the user to be able to do that with a flag in the config file.

There are various cases where PCA cannot be computed because the number of voxels with signal is smaller than the number of time-points... this issue is not present in the Matlab pipeline because Matlab will compute Principal Components regardless of whether the dimensions of the input matrix are adequate. Instead, python will transpose the matrix to ensure that the dimensions are well defined to perform PCA.

The inverted matrix produces an error when attempting to perform the regression as there is a dimension mismatch and the regression matrix cannot be constructed.

To solve this:

• we need to check the dimensions of the PCA input
• if #voxels < #time-points then we compute the mean signal across all voxels, per time-point.
• we use this 1xT vector to construct regression matrix.

The readout-time has to be divided by the acceleration factor.

Seems like the get_readout script is not picking up the right header parameter and therefore it is defaulting to divide by 1. The header key is "ParallelPreductionFactorInPlane" but readout script is looking for dicomHeader[0x0051100b]....

We need two variables for this, one for EPI and another for DWI.
SOlution: rename the EPI config variable as config_epi_dcmFolder, and keep the other name for DWI as it is harder to replace throughout the code.

There are several configurations that are currently hard wired in the last two sections of DWI_B.
Jenya has left some comments about how those configurations could be specified by the user.

Some group discussion may be needed before those settings are made available in the config file...

HPC now has an ICA-AROMA module, so in theory, we don't need the package as a supplemental material (as we currently have it). Things to look into:

• Are there any python requirements in order to use the ICA-AROMA module?
• Is it compatible with other python packages needed for other sections of the pipeline?

Group results should be saved in various directories:

• parcellation
  -- Seed
  --- Whole Brain
  ---- Smoothed
  ---- Unsmoothed
  --- Seed Only

For ROI: Print to log file, and print ROI numbers that have all NAN to QC file.

We will release a new version of the code that will include only the bash/python code.
The matlab code will remain as a previous version that can always be retrieved and updated, as long as updates (pushes) are always done to the corresponding matlab branch.

Currently, since Jenya is still developing in Matlab, he can continue to push any updates to his jenya-wip branch. Andrea will continue to work on her andrea-wip-bash branch, but this branch will be regularly merged onto the master branch, which other collaborators can pull from to get the most recent version of the code.

Proposed date to cut the new release is 12/18/19.

Andrea and Matt will work on this.

Group by:

• ConnPipe parameters
  -- Parcellation
  --- ROI
  ---- Whole Brain
  ----- Smoothed_fwhmX
  ----- Unsmoothed
  ---- Seed Ony

substitute message in following files:

fMRI_A_EPI_ReadHeaders
get_readout
t1_prepare_A
DWI_A
fMRI_A_EPI_SpinEchoUnwarp

This section computes the average time series for the regions of the parcellation(s) selected by the user. Output was originally saved in a .mat file... what are we saving it in now?

May need the .mat files for fMRI_B which is currently a Matlab script

The script is called by fMRI_A_EPI_RegOthers.sh but it is not generating the clean output files.

The PCA computation will fail if the number of voxels in the eroded masks is smaller than the number of time-points in the series.
If this happens, we need to increase the masks until numVoxels >= numTimePoints

The config file is somewhat messy at the moment. In some sections (specially the Nuisance Regression sections) it is difficult for the user to distinguish which flags are to be set/specified by the user, and which flags should not be modified by the user.

So two things are needed to make this more clear:

1. Add more comments and instructions in the config file and highlight flags that are to be set by the user while making it clear which flags are set by the program once the user has specified their settings. Flags will always be set with the default values/settings that "we" (Jenya and Mario?) recommend but user may have data that has different processing needs

2. Add a description of each flag that the user can set in the documentation. Perhaps this is where we explain why/what settings "we" recommend, and offer suggestions for other settings if data is different...

If user selects ANTS for brain extraction, we'll run BET anyway, and use the mask generated with BET to compare it with the mask generated with ANTS. We'll look at the volume of the overlap between the two masks, if the volume of the overlap is greater than some threshold, then we keep the ANTS mask, otherwise, we prompt the user to either try another template (MICCAI, NKI are available) or keep the mask generated with BET.

Use this information to determine pipeline flow and file names.
This should also allow us to define the scanner_params as a function of the scanner type, instead of hard-coding the tag key names (lines 227 - 233 in config.sh file)

• copy DWI series from Karmen's folder
• make slurm scripts so that Meredith can start submitting jobs