Breaking after SH fitting about tractoflow

[GuillaumeTh]

Hi @GuidoGuberman

Yes that's possible but that imply to do this type of modification for all processing step.
@jchoude @arnaudbore @mdesco do you want to have a flag to run or not the step for all the steps ?

[jchoude]

@GuidoGuberman when you say SH fitting, you mean the CSD (fODF), right? Then you wouldn't want to proceed with the tracking?

I personally don't think it's a good idea to add such a flag for all steps. The goal of a pipelining engine such as Nextflow is, in part, to automatize such control. What happens if someone flags DTI_metrics as not needing to run, but then doesn't understand why the registration step does not start?

IMHO, the simplest and most foolproof way to do this would be to copy this pipeline and remove the steps after FODF_metrics. The residual step might even be added there.

[arnaudbore]

I talked about this with @GuidoGuberman and he wanted to be able to compare responses function from different vendors. The main idea is not to skip a step but more to be able to stop after or before a specific step.

[jchoude]

I see.

I guess that after that, the goal would be to be able to resume from the stopping point?

[arnaudbore]

Exactly.

[jchoude]

Ok.

Again, this is my opinion from experience: this risks making things more complicated and error-prone.

What I would do in a case where I would need to do this would either be:

1. Split the pipeline in two at that point, and the resume operation would be to call the second part.

2. Run the full pipeline as is usual, and then analyze the RF and compare them through vendors. Then, if everything makes sense, the results are already ready. If something needs to be adjusted, the part of results that do not make sense can be discarded, and computing should resume at the point where results are missing. Computing is "cheap" compared to adding potential error sources.

Also, tangentially: in a case where the RF would differ wildly across vendors, what would be the expected strategy? This choice can also influence how this would be implemented.

For example, if some manual RF needs to be set, then a separate pipeline might be best. If it is added in the input directories at a later stage, -resume will just rerun everything.

[GuidoGuberman]

Hi @jchoude ,

Thanks for the tip.

Sorry I was pretty ambiguous in my question. I meant to ask two things. First, the idea is to compare FRFs (and other metrics) between sites, and if they are different, I would like to implement a harmonization approach (e.g.: Karamuyak et al 2019). I thought it would be a good idea to add the flag to avoid running the subsequent steps for no reason (i.e.: if there are site differences). Second, when I asked about the SH fitting in my original question, I was referring to what I thought was the output of an intermediary step in the CSD. If I'm not mistaken, the Tournier 2007 implementation of CSD first fits spherical harmonics to the signal, and then performs the deconvolution on the SH coefficients of the fitted signal. I was hoping that it might be possible to get the residuals from this SH fitting. This harmonization approach that I mentioned uses those residuals to measure site differences.

But to come back to the issue of the flag, honestly what you're suggesting works too. If adding that flag is going to cause more problems than solve them then I can just do what you suggest.

[jchoude]

Hi @GuidoGuberman

No problem :)

I now understand when you want to break. This would need to be coded differently, since the 2 parts (SH fit and deconv) are all done in the same script (in fact, in the same method in Dipy).

However, if you only want to get the pure SH fit and then work on the residual of those, an additionnal process could be added to the pipeline. We already have a script that does this pure fitting, so it wouldn't be long.

As for not running the steps if not needed, I would need to see exactly what the Paper does. I couldn't find it using only Karamuyak et al 2019, so if you have a DOI or something I would love to see it.

Depending on the way the harmonization is applied, the modifications needed might be small or large.

[GuidoGuberman]

Hi @jchoude ,
That's exciting to hear then! Here's the DOI for the paper I mentioned (10.1016/j.neuroimage.2018.08.073). The SH residuals are used to do a quality check of the data. The SH coefficients are used to find SH features that are rotationally invariant and then to learn a way to map these features from one site onto another site. In the paper, they use the differences in tensor metrics along major bundles to show site differences and to justify their approach. So ideally, to avoid having to do tractography for no reason, a break point could be inserted after the SH fitting (ideally the tensor fitting would have been done as well at this point)? I'm not sure whether performing the deconvolution would be necessary, since at best I could use the AFD to also show differences between sites, but if the FRFs are different between sites then wouldn't the AFD be expected to be different as well?

[arnaudbore]

I'm closing this issue since I don't feel this is a highly demanded feature.