samtools sort requests too much memory about methylseq

[sb43]

Yes, @wikiselev shared his rnaseq implementation with me.

[sb43]

Hi Phil,
Thanks for the quickfix, I have already tried the commit from your master branch and it worked fine.
We have now fully working methylseq pipeline implemented in openstack using WR in our group.
Thanks for your continuous support.
Cheers,
Shriram

[ewels]

Fantastic! Thanks for letting us know.. And great that you guys are able to use this in production! 😄

I hadn't realised that you were also at the Sanger until now. Are you already in contact with @wikiselev and @micans? They're also running nf-core pipelines on your OpenStack / Kubernetes infrastructure..

[ewels]

Hi @sb43,

Thanks for reporting this. I guess that it's not that samtools is being inefficient, but more that we're telling samtools to use exactly the amount of memory available, to the byte:

methylseq/main.nf

Lines 774 to 777 in 1d3f5cc

	samtools sort $bam \\
	-m ${task.memory.toBytes() / task.cpus} \\
	-@ ${task.cpus} \\
	-o ${bam.baseName}.sorted.bam

In most computational setups, the memory allocation is a little relaxed meaning that this isn't a problem, but we have seen similar problems in other pipelines. I guess that just reducing the requested amount a little should fix the problem. In fact, @apeltzer did exactly this in the rnaseq pipeline the other day: nf-core/rnaseq#158

Phil

[sb43]

Hi @ewels,

Shall I create custom main.nf as suggested nf-core/rnaseq#158 , currently I am using methylseq v1.3 --with-docker using default main.nf.
OR
Have you planned a bugfix release anytime soon?

Thanks,
Shriram

[ewels]

Ok, pull-request merged. You should now be able to run this now using nextflow run nf-core/methylseq -r dev (the -r dev will pull the development version of the code). It will then go out in the next stable release, which should be relatively soon I think.

I'll close this issue now, but let me know if you hit any problems and we can reopen. Would be great to hear how you get on too!

Cheers,

Phil

[micans]

Which of course is by and large nf-core/rnaseq, being forked from it. Some day the road will lead back ... Stijn

[suhrig]

It seems this issue has returned. I am running into the same situation where samtools sort requests exactly as much memory as was reserved for the task (or a little more), resulting in the task being killed. If I'm interpreting things correctly, the changes added in PR #82 got lost when the NextFlow configuration was split into modules. See here: https://github.com/nf-core/methylseq/blob/81f989c93e866a9b0bd0d9584e077b9b8f78affe/modules/nf-core/samtools/sort/main.nf#L24C33-L24C33