Memory reservation in dbgh5 about gatb-core

[rchikhi]

Thanks for the bug report. Unfortunately I can't reproduce it. Can you reproduce it on a different file, or just this one? Regardless, can you please give me the input file? (e.g. https://transfer.sh)

I tried the following (on my machine and genocluster):

$ bin/dbgh5 -in ../test/db/microsnp.fa -kmer-size 12
[DSK: Collecting stats on microsnp       ]  100  %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu:  -1.0 %   mem: [  16,   16,   16] MB
[DSK: nb solid kmers found : 0           ]  100  %   elapsed:   0 min 2  sec   remaining:   0 min 0  sec   cpu: 111.1 %   mem: [ 223,  223,  223] MB B

EXCEPTION: This dataset has no solid kmers

and

seqtk seq ../test/db/reads3.fa.gz  | head -n 2000 > 1000kreads.fa
$ bin/dbgh5 -in 1000kreads.fa
[DSK: Collecting stats on 1000kreads     ]  100  %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu: 100.0 %   mem: [  28,   28,   28] MB
[DSK: Pass 1/1, Step 1: partitioning     ]  27.3 %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu: 175.0 %   mem: [  93,   93,   93] MB
[etc.. running fine]

[cguyomar]

I have exactly the same problem with the microsnp dataset

$ bin/dbgh5 -in ../test/db/microsnp.fa -kmer-size 12
[DSK: Collecting stats on microsnp       ]  100  %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu:  -1.0 %   mem: [  17,   17,   17] MB 
[DSK: Pass 1/1, Step 1: partitioning     ]  0    %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu:  -1.0 %   mem: [  46,   46,   75] MB 
EXCEPTION: Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 16 and capacity is 5242881152
Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 32 and capacity is 5242881152
Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 48 and capacity is 5242881152
Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 64 and capacity is 5242881152
Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 80 and capacity is 5242881152
Pool allocation failed for 0 bytes (kmers alloc), mainbuffer is null?. Current usage is 96 and capacity is 5242881152

$ bin/dbgh5 -in ../test/db/microsnp.fa -kmer-size 12 -max-memory 1000
[DSK: Collecting stats on microsnp       ]  100  %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu:   0.0 %   mem: [  17,   17,   17] MB 
[DSK: nb solid kmers found : 0           ]  100  %   elapsed:   0 min 0  sec   remaining:   0 min 0  sec   cpu: 105.3 %   mem: [  73,   73,   75] MB B 

EXCEPTION: This dataset has no solid kmers

[rchikhi]

I see. Maybe the problem is limited to machines with 8GB of RAM. I'll keep this issue open and get back to it at some point.

[rchikhi]

Have recently committed a possibly related memory allocation fix:
e99c5d7