Comments (14)
wwood
commented on August 18, 2024
Some more details:
The log file
/srv/sw/megahit/20141001/bin/sdbg_builder_cpu count -k 27 -m 2 --host_mem 912000000000 --gpu_mem 0 --max_read_len 270 --output_prefix ./megahit_out/tmp/k27 --num_cpu_threads 44 --num_output_threads 14 --input_file -
[Wed Oct 1 13:35:46 2014]: Extracting solid (k+1)-mers for k = 27
Host memory to be used: 912000000000
Number CPU threads: 44
Reading input...
Max Read length is 270
Estimate max number of reads can be loaded: 9499999999
Done reading input, 3565133146 reads in total
Time elapsed: 8929.7721s
18 words per read, 2 words per substring, 3 words per edge
lv.1 Available Memory: 637014551548, memory for reads: 256689586512:
max # lv.1 items = 159253637887
max # lv.2 items = 198121728
Filling read partition buckets...
Done!
Time elapsed: 377.9126s
Iteration 1, from bucket 0 to 24239
Scanning and filling offsets... done.
Number of large differentials: 0
[LV1] time elapsed: 1447.2667s
> Iteration [1,1], from bucket 0 to 14
Extracting substrings... done. Time elapsed: 473.5269s
Sorting substrings with CPU...
Sorting time elapsed: 13.2187s
> Iteration [1,2], from bucket 15 to 33
Extracting substrings... Counting time elapsed: 3.6452s
Counting time elapsed: 3.6914s
Counting time elapsed: 3.8071s
Counting time elapsed: 3.8158s
Counting time elapsed: 3.8687s
Counting time elapsed: 3.8902s
Counting time elapsed: 3.9505s
Counting time elapsed: 3.9996s
Counting time elapsed: 4.0991s
Counting time elapsed: 4.1983s
Counting time elapsed: 4.2060s
Counting time elapsed: 4.2584s
Counting time elapsed: 4.6759s
Counting time elapsed: 4.9194s
done. Time elapsed: 171.9542s
Sorting substrings with CPU...
Sorting time elapsed: 13.5855s
> Iteration [1,3], from bucket 34 to 49
Extracting substrings... Counting time elapsed: 3.4111s
Counting time elapsed: 3.4323s
Counting time elapsed: 3.6752s
Counting time elapsed: 3.8263s
... (snip)
> Iteration [7,229], from bucket 390621 to 390624
Extracting substrings... done. Time elapsed: 3.1408s
Sorting substrings with CPU...
Sorting time elapsed: 1.9405s
Linear part: 18.040201
Output time elapsed: 20.478769
Linear part: 3.180423
Output time elapsed: 3.638798
Done sorting!
Time elapsed: 54498.0511
All done!
Number of A C G T A- C- G- T- and $:
955581803 23350177529 34719302854 34785626485 23474233670 149752302 385114470 384055915 148855607
Total number of edges: 118352700635
Total number of ONEs: 116329340538
Total number of v$ edges: 955581803
Total number of $v edges: 955581803
/srv/sw/megahit/20141001/bin/assembler -s ./megahit_out/tmp/k27 -o ./megahit_out/tmp/k27 -t 44 --max_tip_len -1 --min_final_contig_len 200 --remove_low_local --low_local_ratio 0.2
[Thu Oct 2 21:14:49 2014]: Assembling contigs from SdBG for k = 27
Loading succinct de Bruijn graph: ./megahit_out/tmp/k27
Done. Time elapsed: 945.306707
Number of Edges: 118352700635
K value: 27
Number of CPU threads: 44
Removing tips with length less than 2
Accumulated tips removed: 25803436; time elapsed: 1067.8968
Removing tips with length less than 4
Accumulated tips removed: 78365783; time elapsed: 1144.5846
Removing tips with length less than 8
Accumulated tips removed: 175212015; time elapsed: 1485.4293
Removing tips with length less than 16
Accumulated tips removed: 337187680; time elapsed: 2393.5703
Removing tips with length less than 32
Accumulated tips removed: 580802466; time elapsed: 3881.5799
Removing tips with length less than 54
Accumulated tips removed: 764407925; time elapsed: 5092.2451
Tips removal done! Time elapsed(sec): 15065.307947
Number of bubbles: 38537938. Time elapsed: 9005.537910
Removing low local coverage...
total_depth: 248340346621
Simple path graph size: 1217741225, time for building: 26775.029208
Total length: 60537902427, N50: 68, Mean: 57, number of contigs: 1047116089
Maximum length: 18977
Time to output: 10161.837682
Number of unitigs removed: 70178826, time: 20309.421179
Total length: 50121920476, N50: 71, Mean: 58, number of contigs: 853138264
Maximum length: 33750
Done! Time elapsed(sec.): 65224.430098
/srv/sw/megahit/20141001/bin/iterate_edges_k61 -c ./megahit_out/tmp/k27.contigs.fa -m ./megahit_out/tmp/k27.multi -t 44 -k 27 -s 10 -o ./megahit_out/tmp/k37 -l 270 --addi_contig_file ./megahit_out/tmp/k27.addi.fa --addi_multi_file ./megahit_out/tmp/k27.addi.multi -r - -f fasta
[Fri Oct 3 22:20:03 2014]: Extracting iterative edges from k = 27 to 37
Reading contigs...
Read 19695850 contigs, total length: 1073741874
Adding to hash...
Reading contigs...
The tmp directory:
total 1.1T
-rw-rw-r-- 1 uqbwoodc abisko 7.3G Oct 3 22:18 k27.addi.fa
-rw-rw-r-- 1 uqbwoodc abisko 144M Oct 3 22:18 k27.addi.multi
-rw-rw-r-- 1 uqbwoodc abisko 6.9G Oct 2 03:16 k27.cand
-rw-rw-r-- 1 uqbwoodc abisko 102G Oct 3 22:18 k27.contigs.fa
-rw-rw-r-- 1 uqbwoodc abisko 1.2M Oct 2 03:16 k27.counting
-rw-rw-r-- 1 uqbwoodc abisko 7.2G Oct 2 21:14 k27.dn
-rw-rw-r-- 1 uqbwoodc abisko 45G Oct 2 03:20 k27.edges.0
-rw-rw-r-- 1 uqbwoodc abisko 45G Oct 2 03:20 k27.edges.1
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.10
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.11
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.12
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.13
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.2
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.3
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.4
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.5
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.6
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.7
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.8
-rw-rw-r-- 1 uqbwoodc abisko 46G Oct 2 03:20 k27.edges.9
-rw-rw-r-- 1 uqbwoodc abisko 67 Oct 2 21:14 k27.f
-rw-rw-r-- 1 uqbwoodc abisko 4.9G Oct 3 22:18 k27.final.contigs.fa
-rw-rw-r-- 1 uqbwoodc abisko 14G Oct 2 21:14 k27.isd
-rw-rw-r-- 1 uqbwoodc abisko 14G Oct 2 21:14 k27.last
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.0
-rw-rw-r-- 1 uqbwoodc abisko 477M Oct 2 06:00 k27.mercy.1
-rw-rw-r-- 1 uqbwoodc abisko 478M Oct 2 06:00 k27.mercy.10
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.11
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.12
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.13
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.14
-rw-rw-r-- 1 uqbwoodc abisko 478M Oct 2 06:00 k27.mercy.15
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.16
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.17
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.18
-rw-rw-r-- 1 uqbwoodc abisko 474M Oct 2 06:00 k27.mercy.19
-rw-rw-r-- 1 uqbwoodc abisko 473M Oct 2 06:00 k27.mercy.2
-rw-rw-r-- 1 uqbwoodc abisko 473M Oct 2 06:00 k27.mercy.20
-rw-rw-r-- 1 uqbwoodc abisko 472M Oct 2 06:00 k27.mercy.21
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.22
-rw-rw-r-- 1 uqbwoodc abisko 477M Oct 2 06:00 k27.mercy.23
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.24
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.25
-rw-rw-r-- 1 uqbwoodc abisko 471M Oct 2 06:00 k27.mercy.26
-rw-rw-r-- 1 uqbwoodc abisko 468M Oct 2 06:00 k27.mercy.27
-rw-rw-r-- 1 uqbwoodc abisko 471M Oct 2 06:00 k27.mercy.28
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.29
-rw-rw-r-- 1 uqbwoodc abisko 474M Oct 2 06:00 k27.mercy.3
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.30
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.31
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.32
-rw-rw-r-- 1 uqbwoodc abisko 473M Oct 2 06:00 k27.mercy.33
-rw-rw-r-- 1 uqbwoodc abisko 477M Oct 2 06:00 k27.mercy.34
-rw-rw-r-- 1 uqbwoodc abisko 478M Oct 2 06:00 k27.mercy.35
-rw-rw-r-- 1 uqbwoodc abisko 478M Oct 2 06:00 k27.mercy.36
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.37
-rw-rw-r-- 1 uqbwoodc abisko 477M Oct 2 06:00 k27.mercy.38
-rw-rw-r-- 1 uqbwoodc abisko 475M Oct 2 06:00 k27.mercy.39
-rw-rw-r-- 1 uqbwoodc abisko 473M Oct 2 06:00 k27.mercy.4
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.40
-rw-rw-r-- 1 uqbwoodc abisko 474M Oct 2 06:00 k27.mercy.41
-rw-rw-r-- 1 uqbwoodc abisko 477M Oct 2 06:00 k27.mercy.42
-rw-rw-r-- 1 uqbwoodc abisko 476M Oct 2 06:00 k27.mercy.5
-rw-rw-r-- 1 uqbwoodc abisko 480M Oct 2 06:00 k27.mercy.6
-rw-rw-r-- 1 uqbwoodc abisko 478M Oct 2 06:00 k27.mercy.7
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.8
-rw-rw-r-- 1 uqbwoodc abisko 479M Oct 2 06:00 k27.mercy.9
-rw-rw-r-- 1 uqbwoodc abisko 221G Oct 2 21:14 k27.mul
-rw-rw-r-- 1 uqbwoodc abisko 2.0G Oct 3 22:18 k27.multi
-rw-rw-r-- 1 uqbwoodc abisko 56G Oct 2 21:14 k27.w
-rw-rw-r-- 1 uqbwoodc abisko 0 Oct 3 22:20 k37.edges.0
-rw-rw-r-- 1 uqbwoodc abisko 0 Oct 3 22:20 k37.rr.pb
from megahit.
voutcn
commented on August 18, 2024
Hello Ben,
Please check the error message corresponds to errno 32 on your system, using the command:
$ perl -le '$!=32;print$!'
On my system the errno is translated to Broken pipe. If you got the same result, please run the following command to run the subprogram "iterate_edge" alone and see whether there are more detailed err messages.
./cmd | /srv/sw/megahit/20141001/bin/iterate_edges_k61 -c ./megahit_out/tmp/k27.contigs.fa -m ./megahit_out/tmp/k27.multi -t 44 -k 27 -s 10 -o ./megahit_out/tmp/k37 -l 270 --addi_contig_file ./megahit_out/tmp/k27.addi.fa --addi_multi_file ./megahit_out/tmp/k27.addi.multi -r - -f fasta
Regards,
Dinghua
from megahit.
wwood
commented on August 18, 2024
rerunning:
$ time ./cmd | /srv/sw/megahit/20141001/bin/iterate_edges_k61 -c ./megahit_out/tmp/k27.contigs.fa -m ./megahit_out/tmp/k27.multi -t 44 -k 27 -s 10 -o ./megahit_out/tmp/k37 -l 270 --addi_contig_file ./megahit_out/tmp/k27.addi.fa --addi_multi_file ./megahit_out/tmp/k27.addi.multi -r - -f fasta >issue3.stdout 2>issue3.stderr
real 513m46.647s
user 14902m8.680s
sys 182m57.540s
$ perl -le '$!=32;print$!'
Broken pipe
There was no obvious errors on issue3.stderr (empty file) and issue3.stdout was:
Reading contigs...
Read 19695850 contigs, total length: 1073741874
Adding to hash...
Reading contigs...
Read 19688367 contigs, total length: 1073741846
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19744086 contigs, total length: 1073741837
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19741417 contigs, total length: 1073741826
Done!
Adding to hash...
Reading contigs...
Read 19770807 contigs, total length: 1073741849
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19744205 contigs, total length: 1073741833
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19718340 contigs, total length: 1073741874
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19721519 contigs, total length: 1073741973
Done!
Adding to hash...
Reading contigs...
Read 19692259 contigs, total length: 1073741973
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19686045 contigs, total length: 1073741832
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19695874 contigs, total length: 1073741855
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19687531 contigs, total length: 1073741843
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19714688 contigs, total length: 1073741929
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19710079 contigs, total length: 1073741855
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19753386 contigs, total length: 1073741860
Done!
Adding to hash...
Reading contigs...
Read 19779371 contigs, total length: 1073741843
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19769145 contigs, total length: 1073741874
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19728265 contigs, total length: 1073741828
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19739240 contigs, total length: 1073741932
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19742895 contigs, total length: 1073741835
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19773316 contigs, total length: 1073741861
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19772910 contigs, total length: 1073741839
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19718684 contigs, total length: 1073741833
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19717931 contigs, total length: 1073741833
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19750811 contigs, total length: 1073742223
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19723045 contigs, total length: 1073741860
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19723922 contigs, total length: 1073741869
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19711178 contigs, total length: 1073741883
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19713687 contigs, total length: 1073741833
Done!
Adding to hash...
Reading contigs...
Read 19701001 contigs, total length: 1073741845
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19708652 contigs, total length: 1073741826
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19717499 contigs, total length: 1073742105
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19711153 contigs, total length: 1073741872
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19710722 contigs, total length: 1073741825
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19728568 contigs, total length: 1073741890
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19682551 contigs, total length: 1073742027
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19644792 contigs, total length: 1073741851
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19642654 contigs, total length: 1073741930
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19647771 contigs, total length: 1073741829
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19700835 contigs, total length: 1073741829
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19671564 contigs, total length: 1073741827
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19684774 contigs, total length: 1073741826
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19732001 contigs, total length: 1073741847
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19761009 contigs, total length: 1073741913
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19760036 contigs, total length: 1073741843
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19737590 contigs, total length: 1073741855
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19760775 contigs, total length: 1073741860
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19729195 contigs, total length: 1073741844
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19707786 contigs, total length: 1073741843
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19628511 contigs, total length: 1073741827
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19658760 contigs, total length: 1073741824
Done!
Adding to hash...
Reading contigs...
Done!
Writing internal edges...Read 19666316 contigs, total length: 1073741930
Done!
Adding to hash...
Reading contigs...
Read 6093174 contigs, total length: 331620697
Done!
Writing internal edges...Done!
Adding to hash...
Reading contigs...
Read 0 contigs, total length: 0
Done!
Writing internal edges...Done!
Number of crusial kmers: 1945390540
Reading contigs...
Read 18099602 contigs, total length: 1073741849
Writing internal edges...Reading contigs...
Read 18354316 contigs, total length: 1073742472
Done!
Writing internal edges...Reading contigs...
Read 18547204 contigs, total length: 1073741824
Done!
Writing internal edges...Reading contigs...
Read 18383491 contigs, total length: 1073741855
Done!
Writing internal edges...Reading contigs...
Read 1907517 contigs, total length: 109532178
Done!
Writing internal edges...Reading contigs...
Read 0 contigs, total length: 0
Done!
Number of crusial kmers: 1945390540
words per read: 18
words per read: 18
Total: 67108864, aligned: 36866992. Iterative edges: 665632971
Total: 134217728, aligned: 73272183. Iterative edges: 1494629926
Total: 201326592, aligned: 112829208. Iterative edges: 1904844003
Total: 268435456, aligned: 152175844. Iterative edges: 2182736245
Total: 335544320, aligned: 187738004. Iterative edges: 2683299742
Total: 402653184, aligned: 224963435. Iterative edges: 3166659064
Total: 469762048, aligned: 264706566. Iterative edges: 3397162495
Total: 536870912, aligned: 302159136. Iterative edges: 3604024020
Total: 603979776, aligned: 338109757. Iterative edges: 4049328414
Total: 671088640, aligned: 376602374. Iterative edges: 4323539453
Total: 738197504, aligned: 415925810. Iterative edges: 4480010578
Total: 805306368, aligned: 451743337. Iterative edges: 4775520856
Total: 872415232, aligned: 488828951. Iterative edges: 5090773232
Total: 939524096, aligned: 528374950. Iterative edges: 5233052861
Total: 1006632960, aligned: 565425648. Iterative edges: 5404979849
Total: 1073741824, aligned: 600839651. Iterative edges: 5713714906
Total: 1140850688, aligned: 639819853. Iterative edges: 5871674013
Total: 1207959552, aligned: 679060138. Iterative edges: 5997141386
Total: 1275068416, aligned: 713849149. Iterative edges: 6233478882
Total: 1342177280, aligned: 751740563. Iterative edges: 6441124657
Total: 1409286144, aligned: 791276005. Iterative edges: 6531835113
Total: 1476395008, aligned: 824911711. Iterative edges: 6631829869
Total: 1543503872, aligned: 853680187. Iterative edges: 6728589269
Total: 1610612736, aligned: 880422607. Iterative edges: 6808404807
Total: 1677721600, aligned: 909174943. Iterative edges: 6933272685
Total: 1744830464, aligned: 944987880. Iterative edges: 7032349429
Total: 1811939328, aligned: 981367805. Iterative edges: 7304815314
Total: 1879048192, aligned: 1019517585. Iterative edges: 7534876985
Total: 1946157056, aligned: 1059951520. Iterative edges: 7622559139
Total: 2013265920, aligned: 1098986066. Iterative edges: 7716933786
Total: 2080374784, aligned: 1134770062. Iterative edges: 7948172068
Total: 2147483648, aligned: 1172650816. Iterative edges: 8148489835
Total: 2214592512, aligned: 1212908685. Iterative edges: 8221815087
Total: 2281701376, aligned: 1253154775. Iterative edges: 8305475245
Total: 2348810240, aligned: 1287829723. Iterative edges: 8485170406
Total: 2415919104, aligned: 1325274992. Iterative edges: 8657959061
Total: 2483027968, aligned: 1365334647. Iterative edges: 8718390818
Total: 2550136832, aligned: 1403788102. Iterative edges: 8782115686
Total: 2617245696, aligned: 1439887520. Iterative edges: 8969286433
Total: 2684354560, aligned: 1478049233. Iterative edges: 9101674675
Total: 2751463424, aligned: 1517997948. Iterative edges: 9148945580
Total: 2818572288, aligned: 1555394504. Iterative edges: 9223860464
Total: 2885681152, aligned: 1591420389. Iterative edges: 9394626612
Total: 2952790016, aligned: 1629942418. Iterative edges: 9487028576
Total: 3019898880, aligned: 1669489866. Iterative edges: 9526290915
Total: 3087007744, aligned: 1706288422. Iterative edges: 9606225523
Total: 3154116608, aligned: 1742921037. Iterative edges: 9767821124
Total: 3221225472, aligned: 1782157868. Iterative edges: 9845768889
Total: 3288334336, aligned: 1822036599. Iterative edges: 9874443019
Total: 3355443200, aligned: 1856546866. Iterative edges: 9913745623
Total: 3422552064, aligned: 1885964999. Iterative edges: 9938404132
Total: 3489660928, aligned: 1913157288. Iterative edges: 9957746916
Total: 3556769792, aligned: 1941128665. Iterative edges: 9975779772
Total: 3565711229, aligned: 1943604677. Iterative edges: 9976802341
Writing iterative edges...
Done!
What next?
from megahit.
voutcn
commented on August 18, 2024
Hi Ben, thanks for your feedback. The subprogram seems okay.
Please check out the branch https://github.com/voutcn/megahit/tree/debug_1T_soil and run MEGAHIT with the same parameters. i.e.
$ time megahit --k-min 27 --k-max 87 --k-step 10 -t44 -m 912000000000 -l 270 --input-cmd './cmd' --cpu-only -o PATH_TO_YOUR_LAST_OUTPUT_DIR
PATH_TO_YOUR_LAST_OUTPUT_DIR is the path to your output directory of the last run. According to your log it should be /srv/sw/megahit/20141001/megahit_out.
This branch contains the several commits after the first release (0.1.1-beta) which fixed some bugs, and skips the first two steps of MEGAHIT as you have already build the graph for k=27.
from megahit.
wwood
commented on August 18, 2024
Thanks a lot for your help. Do you mean PATH_TO_YOUR_LAST_OUTPUT_DIR as ./megahit_out not /srv/sw/megahit/20141001/megahit_out? The latter is where the software is installed, and there is no megahit_out directory there.
from megahit.
wwood
commented on August 18, 2024
Perhaps I should clarify, $PWD is /srv/projects/abisko/assemblies/63_erio_ultradeeps_after20140929_megahit
from megahit.
voutcn
commented on August 18, 2024
To avoid any ambiguity, use the absolute path to the output directory of your last run :)
from megahit.
wwood
commented on August 18, 2024
OK, recompiling now then. While I'm here I noticed this which I'm guessing is a bug (but not one that had a negative impact on me):
~/bioinfo/megahit_debug_1T_soil$ make
[[ -d ./bin/ ]] || mkdir ./bin/
/bin/sh: 1: [[: not found
(snip)
from megahit.
voutcn
commented on August 18, 2024
Thank you Ben. I change the command to mkdir -p bin/. This should be more compatible.
from megahit.
voutcn
commented on August 18, 2024
Hi Ben,
How is the assembly going? If you still encounter problems, feel free to contact me.
If it completed successfully, would you mind sharing data volume (in base-pairs) and assembly stat (This script may be useful )? We don't have data & hardware of that scale to test MEGAHIT, thus your feedbacks are warmly appreciated.
Dinghua
from megahit.
wwood
commented on August 18, 2024
Hey, its still going as I had to Ctrl z it a few times to run other things.
I'll get back to you when it finishes - appreciate the effort you are
putting into this.
On 12 Oct 2014 18:18, "Dinghua Li" [email protected] wrote:
Hi Ben,
How is the assembly going? If you still encounter problems, feel free to
contact me.
If it completed successfully, would you mind sharing data volume (in
base-pairs) and assembly stat (This script
http://wiki.bioinformatics.ucdavis.edu/index.php/Count_fasta.pl may be
useful )? We don't have data & hardware of that scale to test MEGAHIT, thus
your feedbacks are warmly appreciated.Dinghua
—
Reply to this email directly or view it on GitHub
#3 (comment).
from megahit.
wwood
commented on August 18, 2024
Worked. finish just sends me an email.
$ time ~/bioinfo/megahit_debug_1T_soil/megahit --k-min 27 --k-max 87 --k-step 10 -t24 -m 912000000000 -l 270 --input-cmd './cmd' --cpu-only -o /srv/projects/abisko/assemblies/63_erio_ultradeeps_after20140929_megahit/megahit_out >issue3.1.stdout 2>issue3.1.stderr; finish
^Z
[1]+ Stopped ~/bioinfo/megahit_debug_1T_soil/megahit --k-min 27 --k-max 87 --k-step 10 -t24 -m 912000000000 -l 270 --input-cmd './cmd' --cpu-only -o /srv/projects/abisko/assemblies/63_erio_ultradeeps_after20140929_megahit/megahit_out > issue3.1.stdout 2> issue3.1.stderr
real 309m17.093s
user 0m0.000s
sys 0m0.000s
$ time fg; finish
~/bioinfo/megahit_debug_1T_soil/megahit --k-min 27 --k-max 87 --k-step 10 -t24 -m 912000000000 -l 270 --input-cmd './cmd' --cpu-only -o /srv/projects/abisko/assemblies/63_erio_ultradeeps_after20140929_megahit/megahit_out > issue3.1.stdout 2> issue3.1.stderr
real 5938m31.760s
user 90610m49.836s
sys 3537m21.892s
$ cat issue3.1.stderr
[Wed Oct 8 17:51:03 2014]: Extracting iterative edges from k = 27 to 37
[Sat Oct 11 01:40:43 2014]: Building graph for k = 37
[Sat Oct 11 13:00:04 2014]: Assembling contigs from SdBG for k = 37
[Sun Oct 12 03:20:25 2014]: Extracting iterative edges from k = 37 to 47
[Sun Oct 12 08:18:17 2014]: Building graph for k = 47
[Sun Oct 12 22:35:28 2014]: Assembling contigs from SdBG for k = 47
[Mon Oct 13 08:42:23 2014]: Extracting iterative edges from k = 47 to 57
[Mon Oct 13 12:00:28 2014]: Building graph for k = 57
[Mon Oct 13 18:10:52 2014]: Assembling contigs from SdBG for k = 57
[Mon Oct 13 23:11:11 2014]: Extracting iterative edges from k = 57 to 67
[Tue Oct 14 01:07:38 2014]: Building graph for k = 67
[Tue Oct 14 05:02:12 2014]: Assembling contigs from SdBG for k = 67
[Tue Oct 14 08:03:18 2014]: Extracting iterative edges from k = 67 to 77
[Tue Oct 14 09:14:26 2014]: Building graph for k = 77
[Tue Oct 14 12:20:29 2014]: Assembling contigs from SdBG for k = 77
[Tue Oct 14 14:32:45 2014]: Extracting iterative edges from k = 77 to 87
[Tue Oct 14 15:12:38 2014]: Building graph for k = 87
[Tue Oct 14 17:28:07 2014]: Assembling contigs from SdBG for k = 87
[Tue Oct 14 18:49:33 2014]: Merging to output final contigs.
[Tue Oct 14 18:53:51 2014]: ALL DONE.
Stats. Of contigs >5kb (largest 210kb)
Total length of sequence: 831703406 bp
Total number of sequences: 93782
N25 stats: 25% of total sequence length is contained in the 8615 sequences >= 14603 bp
N50 stats: 50% of total sequence length is contained in the 27811 sequences >= 8666 bp
N75 stats: 75% of total sequence length is contained in the 56398 sequences >= 6280 bp
Total GC count: 502803073 bp
GC %: 60.45 %
In total
Total length of sequence: 24377051875 bp
Total number of sequences: 59868535
N25 stats: 25% of total sequence length is contained in the 3635532 sequences >= 869 bp
N50 stats: 50% of total sequence length is contained in the 14309751 sequences >= 418 bp
N75 stats: 75% of total sequence length is contained in the 33036265 sequences >= 267 bp
Total GC count: 14759664907 bp
GC %: 60.55 %
I'm not really sure how to judge how good this assembly is, but I do know it isn't easy - complex community and plenty of related strains. Based on these assembly stats alone, I'm not unimpressed, particularly for a single-ended assembler. Thanks for your help - not sure whether the issue should be closed?
from megahit.
voutcn
commented on August 18, 2024
Thanks a lot. Would you also sharing the number of reads and total base-pairs of your input data? Yes it is difficult to evaluate the assembly quality of metagenomics data. Anyway at this stage we want to have some idea on how large volume MEGAHIT can handle.
Please feel free to close this issue. Thanks for your feedback again.
from megahit.
wwood
commented on August 18, 2024
3,565,711,229 reads, 389,996,298,082 bp (30-237 nt long each read). Most are paired but then I suppose megahit treats them as single anyway.
from megahit.
Related Issues (20)
- Problem with big assembly HOT 1
- Error with latest OS X Bioconda recipe HOT 1
- Assembly contiguity & sequencing depth
- Usage on Mac - megahit_core read2sdbg
- Ubuntu WSL and Open-Suse server megahit_core read2sdbg
- Stuck with exit code 1 for a while please help HOT 2
- Hi~I got this error: 'Exit code -9' HOT 3
- == Error == system call for: "['']" finished abnormally, OS return value: 2
- Will circular DNA be reported? HOT 3
- Exit Code -6
- Exit code 1
- Running into a permission error while running a test data
- How to solve memory shortages?
- Megahit running for almost a month now
- MEGAHIT output: scaffolds HOT 3
- How to tell what the "peak memory" or "maximum memory needed" would be for a job based on the log?
- Is there a default for the --preset option?
- Error: Exit code -7
- Merge simple edges in the assembly graph? HOT 1
- Illumina SE
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from megahit.