When fsl_sub handles a single task, it internally redirects the outputs with this code:

            /bin/sh <<EOF1 > ${LogDir}${JobName}.o$$ 2> ${LogDir}${JobName}.e$$
$@
EOF1

In the case where this is executing under CBRAIN, this will pollute the work directory with a lot of files named bash.oNNN and bash.eNNN instead of storing these in the CBRAIN capture files. The way to fix this is to remove the redirections altogether, if we're running in CBRAIN. (But: how to test that? Env variable?)

The Boutiques command-line starts with "cp -r inputdir outputdir" which simply copies the symbolic link as is, instead of making a real distinct copy:

[cbrain@lg-1r17-n05 admin_nbeck-FslProbtrackx2-T371048]$ ls -l
total 0
lrwxrwxrwx 1 cbrain cbrain 49 Dec 19 12:13 DTI_merged.bedpostX -> ../../../../DP_Cache/82/13/27/DTI_merged.bedpostX
lrwxrwxrwx 1 cbrain cbrain 49 Dec 19 14:32 probtrackx2_output -> ../../../../DP_Cache/82/13/27/DTI_merged.bedpostX
lrwxrwxrwx 1 cbrain cbrain 54 Dec 19 12:13 ROI_Executive_bin.nii.gz -> ../../../../DP_Cache/82/14/06/ROI_Executive_bin.nii.gz
lrwxrwxrwx 1 cbrain cbrain 53 Dec 19 12:13 ROI_Striatum_bin.nii.gz -> ../../../../DP_Cache/82/14/07/ROI_Striatum_bin.nii.gz

Currently, fsl_sub doesn't specify a tool config when it submits tasks to CBRAIN. The Bourreau picks the first tool_config it finds, which has several issues:

• the new task may be submitted to a different cluster;
• the new task may be submitted with a different version of FSL.
  So, at the moment, FSL melodic (group analyses) only work when there is only a single tool config for fsl_sub, with the same version as FSL melodic, and on the same cluster.

The solution to allow for more fsl_sub tool_configs is to:

• choose the proper tool_config_id in fsl_melodic (bourreau-side). The tool config should be on the same bourreau, with the same FSL version (fsl_sub currently use different versions than FSL, but this should be fixed in fsl_sub's Boutiques descriptor).
• pass this tool config id to fsl_sub, e.g. through an environment variable.
• make fsl_sub write the tool_config_id in its job submission to CBRAIN.

In particular in the "Registration summary" page, images corresponding to non-synchronized children analyses do not show.

The CivetStudy model stores all the CivetOutputs inside it, which leads to a lot of duplicated data.

Let's create a drop-in replacement, CivetVirtualStudy, that behaves the same way as far as users are concerned but the data is kept as just a list of IDs of the associated CivetOutputs.

Architecture elements:

• store the list of IDs in a CSV file at the top of the CivetVirtualStudy, to be reloaded using the methods provided by the CbrainFileList model. The CbrainFileList object is never to be saved into the DB (ensure it too to prevent programmers making mistakes)
• [REJECTED] maybe access the CSV files using the data provider streaming methods when the CivetVirtualStudy is not synchronized
• adjust the view code to provide view links to the CivetOutputs as needed
• override sync_to_cache() so that syncing a CivetVirtualStudy will sync the CivetOutputs, and create the symlinks to the caches
• [REJECTED] override sync_to_provider() so that the symlinks are NOT sent, and the final status is ProvNewer (because the cached version will be missing the symlinks I assume)

Once the model works well, we'll replace the places int he scientific code where CivetStudies are created so that CivetVirtualStudies are used instead.

Note: was discussed in Redmine issue 3216.

Melodic results should be grouped in a "MelodicStudy" when user selects option in launch form (defaults to yes). Launching melodic should launch a new task that will wait for the others to complete and then group the results. Replicate what is done in CivetCombiner.

In documentation, encourage the user to be careful about naming.

Was issue #24 in cbrain-plugins-fsl.

In CBRAIN-Melodic interface, there has to be an option to register to "expanded functional Image"

The input files will be presented in a .CSV file.
Each column of the file will correspond to

Column 1: RSfMRI_1.nii.gz (compulsory, must be UNIQUE)
Column 2: BET-MRI_1.nii.gz (compulsory, same file may be repeated across rows)
Columns3: BET-Highres-FMRI_1.nii.gz (optional)

Program fail if Column1 and 2 are empty.

We have a GIF file installed in the platform that should be moved to this plugin; it is used by the MINC and the CivetOutput userfiles. I suggest duplicating it in the static assets for each.

Once done, notify me so I can remove it from CBRAIN itself.

• No Docker container should be used for these tools as FSL is largely installed on the clusters.
• For bet, #6 should also be considered.
• Try to be as exhaustive as possible on the supported options.

Right now the Combiner will refuse to create a CivetStudy unless all CivetOutputs were built with the same 'prefix'.

Add a checkbox in the interface and allow the program to proceed even if the civet outputs have different prefixes. With a warning to the user.

Error message: no viewer code available at this moment for files of type 'NIfTI file (compressed)'

Need to start from previously pre-processed data.

Was issue #18 in cbrain-plugins-fsl

If CIVET is sent to 'recover' mode when its T1 input file has been cleared from the DP cache, it fails in a different way that CBRAIN can't detect right now, so the task is erroneously marked as Completed.

Two improvements to fix this:

• re-sync input files in cluster_commands() (which is invoked after error recovery) to make sure the T1 is there
• check for the string Stopped processing all pipelines. at the end of the CIVET output.

Was issue #22 in cbrain-plugins-fsl.

There is a selection box in the nii2minc task view code that says:

 Write voxel data in type [selection box]

There are three values in it, but the task code allows launching a task with the blank value still selected, which crashes on the bourreau side with

 CbrainError: Unexpected voxel int sign

Adjust after_form().

Add 4 input params (x,y,z,t) and a checkbox to enable/disable the dimension check. If checkbox is enabled, then any file that doesn't have the required dimension will be excluded from the analysis, with a warning.

In CBRAIN-MELODIC interface, user may chose to "Use custom template file", however there is no way for user to choose this template file.

It can be provided as an option to select a template file together with the CSV and FSF file before launching melodic.

Preferably, this could be added as the 4th column in the CSV file.

From one of our user:

Only a flag is missing (--model). Could you please check whether it's sth that you can fix easily?
Multi-Shell Model : Use this option if the data contain more than one non-zero bvalue. This will use the model described in Jbabdi et al. MRM 2012, where the diffusion coefficient is modelled using a Gamma distribution. This option will output the posterior mean of a diffusivity variance parameter mean_d_stdsamples. Command line argument: --model=2. It can be either 1 or 2 for singel-shell or multi-shell analysis.
the default is 1, but we should be able to run it for multishell data too.

The pattern matching code used to auto-fill the prefix and subject ID in the CIVET form
should not stop at the first failure to match a pattern. It should try to fill all the fields,
and those that cannot, it should mark them in the errors list so that the input box show
up in red.

As noticed by @prioux:

I had a quick look at the work directory of that task, and in the .qsub file
all the data files are always substitued with complete full path to the task's work
directory. Is that really necessary? The entire point of writing a task is to encapsulate
the processing locally in the work directory, to make things relocatable even accross
clusters. Full paths defeat this completely. Couldn't all operations be performed
with relative paths instead? e.g. instead of

# Extracts dimensions of file /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz
${FSLINFO} /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz | awk '$1=="dim1" || $1=="dim2" || $1=="dim3" || $1=="dim4" {print}'> /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz.fslinfo
# Extracts TR of file /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz
${FSLHD} /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz | awk '$1=="pixdim4" {print $2}' > /gs/project/eim-670-aa/cbrain/data/gridshare/31/01/02/nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz.trvalue

why not :

# Extracts dimensions of file nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz
${FSLINFO} nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz | awk '$1=="dim1" || $1=="dim2" || $1=="dim3" || $1=="dim4" {print}' > nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz.fslinfo
# Extracts TR of file nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz
${FSLHD} nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz | awk '$1=="pixdim4" {print $2}' > nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz.trvalue

Also, the bash commands could be split on multiple lines to make things prettier. And no need to repeat the file name in the comment

# Extracts dimensions
${FSLINFO} \
    nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz | \
    awk '$1=="dim1" || $1=="dim2" || $1=="dim3" || $1=="dim4" {print}' \
    > nmahani-FslMelodic-T310102/Spin002^^^^_resting_post1_SENSE_FSL_14_1.nii.gz.fslinfo

According with Claude:

The defaut is 9 for all versions of CIVET. The CBRAIN defaults should
match the command-line defaults

BET can run on 4D fMRI data by selecting option -F. If this option is selected, other ones, such as -B, -S, -R must become unwritable. Otherwise, the output will be seriously erroneous, and only one volume.

Was issue #27 in cbrain-plugins-fsl

We no longer SVN IDs, there is a single place in the code of

cbrain_task/niak_pipeline_fmri_preprocess/bourreau/niak_pipeline_fmri_preprocess.xml.erb

(line 32) where we invoke svn_id_pretty_file_rev_author_date() .

Perform a regex audit as in aces/cbrain#482

When rendering userfiles of that type, the view code contains full paths on the cluster sides for rendering the images, so it's all borken. See screenshot.

The image's SRC URL in the HTML is all weird:

577603/content?arguments=melodic-output-305651-3.gica//nfs3_ib/evans-ms.nfs/tank/nfs/evans/nobackup/share/prioux/gridshare/30/56/51/asimioni-FslMelodic-T305651/denoised_func_data_nonaggr_run1_s06202.nii_melodic-output-305651-3.ica/reg/example_func2standard1.png&content_loader=collection_file&content_viewer=off&viewer=image_file&viewer_userfile_class=ImageFile

Looking at the FSL scripts I discovered many of them are written in the TCL programming language, and proper escaping of arguments passed down the stack of utilities is really poor. This explains some of the failures Najma got because she had "^^^^" in her filenames, which are normal characters in filename in a non-interactive use, but mess up all the TCL scripts when they are substituted. It's poor programming on the FSL side, and there's nothing we can do.

I suggest that FSL tools in CBRAIN right from the start apply a simple, stricter filename validation step before they're even launched. In a task's before_form(), a qucik check of all the filenames selected by the user could be made to ensure they only contain alphanums. It's exactly the purpose of the callback before_form(), in fact.

Generates an exception:

ActionView::MissingTemplate: Missing template BrainPortal/cbrain_plugins/installed-plugins/userfiles/functional_nifti_file/views/_volume_viewer_loader with {:locale=>[:en], :formats=>[:html], :handlers=>[:erb, :builder]}. Searched in: * "BrainPortal/app/views" * "BrainPortal" * "/"

with trace:

BrainPortal/cbrain_plugins/installed-plugins/userfiles/minc_file/views/_volume_viewer.html.erb:540:in `_cbrain_plugins_installed_plugins_userfiles_minc_file_views__volume_viewer_html_erb__3427599092687694237_281994820'
(...)BrainPortal/app/views/userfiles/show.html.erb:246:in `_app_views_userfiles_show_html_erb__383784840519253085_295074060'

See also exception log 1178 in the CBRAIN service

MGZ is the compressed version of MGH file... So the model should be MghFile and not MgzFile.

For instance, "command not found" error is not handled because:

It doesn't write ERROR in stdout/stderr.
The output directory exists (it's empty) and can be transferred.

Was issue #21 in cbrain-plugins-fsl.

It seems difficult to ensure that the Melodic code is restartable. While it creates unique directories (.ica, .ica+, etc), there is absolutely no guarantee that restarting it in the same directory won't crash or produce side effects. The best solution to date seems to be a mere deletion of the content of the working directory. Name collisions of output files created in CBRAIN should also be tested. Has to be tested for all kinds of tasks (see tests in https://github.com/glatard/cbrain-tests-fsl-melodic)

The following are advanced options not desirable for all users, but would be helpful for expert users - (1) is particularly for members or affiliates of our lab interested in direct comparisons between Freesurfer and CIVET, (2)-(3) are more generally useful. Not urgent but would be nice.

There are 3 parts to this request that could work independently or together:

(1) To use an alternative template than the default fsaverage (namely, our CIVET template which I have already converted to FS format)
(2) To use smoothing levels other than the 5-25mm offered by -qcache
(3) To do either of the above (or default -qcache) after a finished run of recon-all without running the entire thing from scratch.

For example:

(1): run everything from scratch with $template = CIVET2_average

recon-all -s $subj -all -qcache -target CIVET2_average

(2): run everything from scratch with smoothing = 30mm (default fsaverage template)

recon-all -s $subj -all -qcache -measure thickness -fwhm 30

(1,2): run everything from scratch with smoothing = 30mm and $template = CIVET2_average

recon-all -s $subj -all -qcache -measure thickness -fwhm 30 -target CIVET2_average

Or simply remove the -all flag for any (3): (1,2,3): post-process a finished recon-all run with smoothing = 30mm and $template = CIVET21_average

recon-all -s $subj -qcache -measure thickness -fwhm 30 -target CIVET2_average

(3): Postprocess finished recon-all with default -qcache and default template fsaverage

recon-all -s $subj -qcache

I have placed the CIVET2_average template (good for 2.0 or 2.1) for you to play around with here:

http://www.bic.mni.mcgill.ca/users/llewis/CIVET2_average.tar.gz

Place it in the Freesurfer subjects directory / in same directory as fsaverage.

Thank you! Let me know if you have any questions.

Was issue #32 in cbrain-plugins-neuro.

When creating a large study, we get one long line for each subject CivetOutput added; I think it's a result of the addlog_to_userfiles_these_created_these() method ; maybe when there are lots of files we can reduce the amount of text.

In some case, CIVET raise the following error:

[2016-08-22 12:31:02 EDT] setup_and_submit_job() Exception raised while setting up: NoMethodError: undefined method match' for #<MincFile:0x000000112345z3> [2016-08-22 12:31:02 EDT] .../cbrain_plugins/installed-plugins/cbrain_task/civet/bourreau/civet.rb:134:insetup'
[2016-08-22 12:31:02 EDT] ..Bourreau/app/models/cluster_task.rb:676:in `block (2 levels) in setup_and_submit_job'

It's due to missing conversion between object and name.

Increase flexibility in naming FSL outputs, similarly to CIVET.

Was issue #28 in cbrain-plugins-fsl

The current 'patch' to fsl_sub to make it work with CBRAIN works like this, when an arrray of tasks need to be submitted:

• loop through each task:
  • create a .json file
  • wait up to 5 minutes for a .cbid file

Instead of that, loop through all the tasks, create all the .json files, then wait for all the .cbid files to appear.

It would be nice to be able to download a file from a collection directly, as a one-step process, rather than having to click on "Extract Files from Collection", wait to be taken back to the list of files, then have to scan the entire list of files for the one you just extracted, and then "Download."

The two view files for CivetCombiner crash abominably when the Combiner was created directly from a Civet's form.

That's because the view files still only expect the task's inputs to be IDs to files (CivetOutputs), but internally the task
supports providing no file ID but providing the IDs of Civet tasks instead; then it fetches the IDs from the tasks
(presumably, the task created these CivetOutputs).

When a user tries to VIEW the CivetCombiner or EDIT a CivetCombiner, it fails.

Non exhaustive list:

• Add parameter groups wherever relevant.
• Add mutual exclusion between input file and input dir in FSL anat.
• Add one of a set requirement between input files in ICA AROMA.

Following @Najmahan comment, it would also be useful to add the type of files that need to be selected to the description of these tools (e.g., for MINCBet "This tool processes a MINC file." or for FSL anat "This tool processes a NIFTI file or a .anat directory").

@ttaa9 I cannot assign this issue to you until you accept the invitation to the repo, but it is for you :)

Was issue #33 in cbrain-plugins-fsl

When files are assigned a more specific type of Minc1 or Minc2, the volume viewer special hook loader fails. That hook is at:

https://github.com/aces/cbrain-plugins-neuro/blob/master/userfiles/minc_file/views/_volume_viewer.html.erb#L529

We have an entire copy of the MincNavigator codebase in the plugin, as a viewer for files of type MincFile; we should simply remove the directory and provide a tiny script that pulls it from GitHub instead. That woudl simplify maintenance and code fixes on that side.