Memory usage of fill module about mindthegap

[clemaitre]

Yes, you can try to limit the number of cpu cores, with the option -nb-cores.

[Chris7]

Thanks, is the memory due to building the de-bruijn graph? Is there a better way to predict the memory usage of a sample?

[clemaitre]

Hi,
This is not the de bruijn graph itself nor its construction that uses such memory (we use the implementation given by the GATB library that is very light, and for this you can control the memory with the option -max-memory). The problem comes from an additional data structure that is used to mark the traversed nodes that is not well memory-optimized (a sort of hashtable for all the branching kmers of the graph, this is on the TODO list to change it to a lighter data structure...) and that must be duplicated for each thread.
Usually, even for large genomes or large number of reads, the number of branching kmers is not so high and this is not an issue. Note that high number of branching kmers can happen if the abundance-min parameter is not well set or with some atypical datasets (for instance, metagenomics).
I hope that reducing the number of threads made it possible for you to run MindTheGap.
Best,
Claire

[clemaitre]

Hi,
the implementation of the de bruijn graph has been modified in MindTheGap (uses now default options of implementation as in dbgh5). This should fix the memory issues.

[Chris7]

Great! Looking forward to trying the newest version out.

[Chris7]

This problem still persists with the newest install. Is there a reference I can look at to see how the memory allocation was fixed to understand why I am still getting 30-50GB of memory allocation per sample?

[clemaitre]

Hi,
We fixed a bug in the building of the de Bruijn graph. MindTheGap used previously non-default implementations for the data structures behind and for some datasets we observed that it could lead to a false graph with way too much branching nodes, which took later too much memory in the Fill module. MindTheGap now uses the default implementation of the dbg of gatb-core.
Did you run again MindTheGap starting from the reads (and not from the previously built h5 file) ?