TERefiner_1 error about repdenovo

[simoncchu]

I think you need to set the bamtools path in the makefile (under TERefiner folder). Usually, it works well and don't need to re-compile. But I do need to release a user-friendly installation package.

[Milad021]

And I have this in my repertory !
30mer.temp_contigs.fa Asm_30_4880_29 Asm_40_64434_39 Asm_60_112338_59 contigs.fa.itself.sort.bam contigs.fa_no_dup.fa.merge.info 40mer.temp_contigs.fa Asm_30_610_29 Asm_40_8054_39 Asm_60_14042_59 contigs.fa.itself.sort.bam.bai dumped_30mers.txt 50mer.temp_contigs.fa Asm_30_78089_29 Asm_50_119523_49 Asm_60_1755_59 contigs.fa_no_dup.fa dumped_40mers.txt 60mer.temp_contigs.fa Asm_30_9761_29 Asm_50_14940_49 Asm_60_28084_59 contigs.fa_no_dup.fa.merged.fa dumped_50mers.txt Asm_30_1220_29 Asm_40_1006_39 Asm_50_1867_49 Asm_60_3510_59 contigs.fa_no_dup.fa.merged.fa.fai dumped_60mers.txt Asm_30_156178_29 Asm_40_128868_39 Asm_50_29880_49 Asm_60_438_59 contigs.fa_no_dup.fa.merged.fa.itself.sort.bam kmers_fq.fastq Asm_30_19522_29 Asm_40_16108_39 Asm_50_3735_49 Asm_60_56169_59 contigs.fa_no_dup.fa.merged.fa.itself.sort.bam.bai original_contigs_before_merging.fa Asm_30_2440_29 Asm_40_2013_39 Asm_50_466_49 Asm_60_7021_59 contigs.fa_no_dup.fa.merged.fa.no_dup.fa reads_coverage.txt Asm_30_305_29 Asm_40_32217_39 Asm_50_59761_49 Asm_60_877_59 contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai Asm_30_312356_29 Asm_40_4027_39 Asm_50_7470_49 contigs.fa contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam Asm_30_39044_29 Asm_40_503_39 Asm_50_933_49 contigs.fa.fai contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam.bai

[simoncchu]

Hi @Milad021 ,
Yes, it is important. Have you tried to re-compile TERefiner_1? Also, check the input of TERefiner, whether they are empty or not.

[Milad021]

I tried to recompile TERefiner_1, but I got an error. In fact it seems that it doesn't found the api/BamReader.h .
I tried multiple links from bamtools repository and build folder to REPdenovo/bamtools-master, and the file is never found

when i do REPdenovo/bamtools$ find ./ -name lib, i get nothing and it's asked when i do make for TERefiner.

Thanks for your time

[Milad021]

In fact there is ( maybe since recently ) two separated include folders for bamtools.
I did a little modification in your Makefile before to success compiling with lot of warnings.
Here is the thing :

BAMTOOLS=../bamtools/build/src/api/include
BAMTOOLSSHARED=../bamtools/build/src/include

CFLAGS =  -O3 -Wall -static -I$(BAMTOOLS) -I$(BAMTOOLSSHARED) -L$(BAMTOOLS_LD) -Wl,-rpath,$(BAMTOOLS_LD)

now it's compiled without errors, but i'm not sure everything is right.

[Milad021]

Unfortunately still the same problem. And the same files are generated in my directory
"""""""""""""
[bwa_index] Pack FASTA... 0.01 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 0.11 seconds elapse.
[bwa_index] Update BWT... 0.00 sec
[bwa_index] Pack forward-only FASTA... 0.00 sec
[bwa_index] Construct SA from BWT and Occ... 0.06 sec
[main] Version: 0.7.15-r1140
[main] CMD: bwa index ./Siek14_Repeatelm_12X/contigs.fa
[main] Real time: 0.707 sec; CPU: 0.184 sec
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 2624 sequences (671503 bp)...
[M::mem_process_seqs] Processed 2624 reads in 1.304 CPU sec, 1.306 real sec
[main] Version: 0.7.15-r1140
[main] CMD: bwa mem -a ./Siek14_Repeatelm_12X/contigs.fa ./Siek14_Repeatelm_12X/contigs.fa
[main] Real time: 1.437 sec; CPU: 1.312 sec
Running command: /usr/local/src/REPdenovo/TERefiner_1 -P -b ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam -r ./Siek14_Repeatelm_12X/contigs.fa -o ./Siek14_Repeatelm_12X/contigs.fa
_no_dup.fa -c 0.9 -g ...
rm: impossible de supprimer './Siek14_Repeatelm_12X/contigs.fa.itself.bam': Aucun fichier ou dossier de ce type
rm: impossible de supprimer './Siek14_Repeatelm_12X/contigs.fa.itself.bam': Aucun fichier ou dossier de ce type
rm: impossible de supprimer './Siek14_Repeatelm_12X/contigs.fa.itself.bam': Aucun fichier ou dossier de ce type
"""""""""""""""

My directory

""""""""""""""""
contigs.fa
contigs.fa.fai
contigs.fa.itself.sort.bam
contigs.fa.itself.sort.bam.bai
contigs.fa_no_dup.fa
contigs.fa_no_dup.fa.merged.fa
contigs.fa_no_dup.fa.merged.fa.fai
contigs.fa_no_dup.fa.merged.fa.itself.sort.bam
contigs.fa_no_dup.fa.merged.fa.itself.sort.bam.bai
contigs.fa_no_dup.fa.merged.fa.no_dup.fa
contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai
contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam
contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam.bai
contigs.fa_no_dup.fa.merge.info
dumped_30mers.txt
dumped_40mers.txt
dumped_50mers.txt
dumped_60mers.txt
kmers_fq.fastq
original_contigs_before_merging.fa
reads_coverage.txt
"""""""""""""""""""
My config.txt :
""""""""
MIN_REPEAT_FREQ 100
RANGE_ASM_FREQ_DEC 2
RANGE_ASM_FREQ_GAP 0.8
K_MIN 30
K_MAX 60
K_INC 10
K_DFT 30
READ_LENGTH 151
GENOME_LENGTH 650000000
MIN_CONTIG_LENGTH 100
ASM_NODE_LENGTH_OFFSET -1
IS_DUPLICATE_REPEATS 0.85
COV_DIFF_CUTOFF 0.5
MIN_SUPPORT_PAIRS 20
MIN_FULLY_MAP_RATIO 0.2
TR_SIMILARITY 0.85
TREADS 15
BWA_PATH GLOBAL
SAMTOOLS_PATH GLOBAL
JELLYFISH_PATH GLOBAL
#VELVET_PATH /usr/bin
VELVET_PATH /home/dygap/mbehzadi/src/velvet
REFINER_PATH /usr/local/src/REPdenovo/TERefiner_1
CONTIGS_MERGER_PATH /usr/local/src/REPdenovo/ContigsMerger
OUTPUT_FOLDER ./Siek14_Repeatelm_12X/
VERBOSE 1
""""""""
grep '>' original_contigs_before_merging.fa | wc -l
2624
Thank you

[simoncchu]

Can you post the reported errors by running this command?

/usr/local/src/REPdenovo/TERefiner_1 -P -b ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam -r ./Siek14_Repeatelm_12X/contigs.fa -o ./Siek14_Repeatelm_12X/contigs.fa
_no_dup.fa -c 0.9 -g

[Milad021]

Thank you for the speedy respons ! I will do it tomorrow, as soon as I have access to my PC.

[simoncchu]

In case you will meet other issues, you can similarly compile ContigsMerger on your server. Then copy the newly compiled TERefiner_1 and ContigsMerger to the same level as main.py. Then try again.

[Milad021]

Hi @ReedWarbler ,
By running this command, I have this error message :
""""""""""""
$ /usr/local/src/REPdenovo/TERefiner_1 /usr/local/src/REPdenovo/TERefiner_1 -P -b ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam -r ./Siek14_Repeatelm_12X/contigs.fa -o ./Siek14_Repeatelm_12X/contigs.fa_no_dup.fa -c 0.9 -g
Index file not found, now create it!!!
Index file cannot be created!!!
Bamtools ERROR: could not open input BAM file: ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam
Cannot parse bam file
""""""""""""

[simoncchu]

I think the error is caused before this step. The /Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam is empty?
Also, can u similarly compile ContigsMerger on your server. Then copy the newly compiled TERefiner_1 and ContigsMerger to the same level as main.py. Then try again?

[Milad021]

Hi @ReedWarbler I'm so sorry for my late response, because I will not be in the lab until Tuesday. As soon as I'm there I'll let you know. But if I can not manage TERefiner and ContigsMerger, can I send you my original_contigs_before_merging.fa ? And My question is ,will I need TERefiner and ContigsMerger for Scaffolding step?

[Milad021]

Hi,
the file doesn't look empty :

678K mai   31 17:30 contigs.fa.itself.sort.bam

the head is :

BAM�sl�@HD	VN:1.3	SO:coordinate
@SQ	SN:NODE_1_length_176_cov_1.988636_30_31235_312356	LN:204
@SQ	SN:NODE_2_length_106_cov_2.000000_30_31235_312356	LN:134
@SQ	SN:NODE_6_length_76_cov_1.973684_30_31235_312356	LN:104
@SQ	SN:NODE_7_length_72_cov_1.972222_30_31235_312356	LN:100

The ContigsMerger is compiled on the machine without error, I'm not sure if the error can from the change I made on the makefile for TERefiner. If not, I have no idea where does it came from.

[simoncchu]

Can u view this file?

samtools view contigs.fa.itself.sort.bam NODE_1_length_176_cov_1.988636_30_31235_312356

[Milad021]

NODE_1_length_176_cov_1.988636_30_31235_312356	256	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	204M	*	0	0	*	*	NM:i:0	MD:Z:204	AS:i:204
NODE_2_length_236_cov_2.258475_30_15617_312356	256	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	204M60H	*	0	0	*	*	NM:i:1	MD:Z:8A195	AS:i:199
NODE_3_length_161_cov_2.869565_30_7808_312356	256	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	42H147M	*	0	0	*	*	NM:i:1	MD:Z:43G103	AS:i:142
NODE_8_length_242_cov_1.995868_40_12886_128868	256	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	17H204M59H	*	0	0	*	*	NM:i:0MD:Z:204	AS:i:204
NODE_12_length_229_cov_1.995633_50_11952_119523	272	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	14H204M59H	*	0	0	*	*	NM:i:0MD:Z:204	AS:i:204
NODE_83_length_174_cov_2.045977_50_3040_119523	272	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	62H160M	*	0	0	*	*	NM:i:0	MD:Z:160	AS:i:160
NODE_83_length_174_cov_2.045977_50_3040_74700	272	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	62H160M	*	0	0	*	*	NM:i:0	MD:Z:160	AS:i:160
NODE_82_length_106_cov_2.028302_50_3040_37350	272	NODE_1_length_176_cov_1.988636_30_31235_312356	1	0	62H92M	*	0	0	*	*	NM:i:0	MD:Z:92	AS:i:92
NODE_3_length_142_cov_1.992958_60_11233_112338	272	NODE_1_length_176_cov_1.988636_30_31235_312356	4	0	200M	*	0	0	*	*	NM:i:0	MD:Z:200	AS:i:200
NODE_365_length_167_cov_4.095809_30_3904_195220	272	NODE_1_length_176_cov_1.988636_30_31235_312356	10	0	195M	*	0	0	*	*	NM:i:3	MD:Z:34G33G105G20	AS:i:180
NODE_535_length_160_cov_4.800000_30_3040_97610	272	NODE_1_length_176_cov_1.988636_30_31235_312356	13	0	17H171M	*	0	0	*	*	NM:i:4	MD:Z:31G31T1G69G35	AS:i:151
NODE_12_length_167_cov_2.029940_60_3040_112338	272	NODE_1_length_176_cov_1.988636_30_31235_312356	20	0	185M40H	*	0	0	*	*	NM:i:2	MD:Z:24G33G126	AS:i:175
NODE_12_length_167_cov_2.029940_60_3040_70210	272	NODE_1_length_176_cov_1.988636_30_31235_312356	20	0	185M40H	*	0	0	*	*	NM:i:2	MD:Z:24G33G126	AS:i:175
NODE_12_length_167_cov_2.029940_60_3040_35105	272	NODE_1_length_176_cov_1.988636_30_31235_312356	20	0	185M40H	*	0	0	*	*	NM:i:2	MD:Z:24G33G126	AS:i:175
NODE_148_length_88_cov_4.352273_40_3221_128868	272	NODE_1_length_176_cov_1.988636_30_31235_312356	79	0	126M	*	0	0	*	*	NM:i:1	MD:Z:82A43	AS:i:121
NODE_204_length_88_cov_4.613636_40_3040_80540	272	NODE_1_length_176_cov_1.988636_30_31235_312356	79	0	126M	*	0	0	*	*	NM:i:2	MD:Z:11T70A43	AS:i:116

[simoncchu]

I mean this file: ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam, because u said:

By running this command, I have this error message :
""""""""""""
$ /usr/local/src/REPdenovo/TERefiner_1 /usr/local/src/REPdenovo/TERefiner_1 -P -b ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam -r ./Siek14_Repeatelm_12X/contigs.fa -o ./Siek14_Repeatelm_12X/contigs.fa_no_dup.fa -c 0.9 -g
Index file not found, now create it!!!
Index file cannot be created!!!
Bamtools ERROR: could not open input BAM file: ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam
Cannot parse bam file
""""""""""""

[Milad021]

Sorry I did n't understand.
I give you the head of the file ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam
Next I show you the output of the command you asked me for :

samtools view contigs.fa.itself.sort.bam NODE_1_length_176_cov_1.988636_30_31235_312356

Thanks for responding fast each time !

I did again all the REPdenovo commands and now i get a strange error :

/usr/local/src/REPdenovo/TERefiner_1 -P -b ./Siek14_Repeatelm_12X/contigs.fa.itself.sort.bam -r ./Siek14_Repeatelm_12X/contigs.fa -o ./Siek14_Repeatelm_12X/contigs.fa -c 0.9 -g
*** Error in `/usr/local/src/REPdenovo/TERefiner_1': free(): invalid next size (fast): 0x0000000001b97fd0 ***
======= Backtrace: =========
[0x54b087]
[0x550912]
[0x55112e]
[0x4215c0]
[0x423e9d]
[0x4027b5]
[0x5336b3]
[0x53393e]
[0x40699a]
======= Memory map: ========
00400000-00652000 r-xp 00000000 fe:04 921790                             /usr/local/src/REPdenovo/TERefiner_1
00851000-0085d000 rw-p 00251000 fe:04 921790                             /usr/local/src/REPdenovo/TERefiner_1
0085d000-00864000 rw-p 00000000 00:00 0 
01b35000-027e1000 rw-p 00000000 00:00 0                                  [heap]
7fab14000000-7fab1402c000 rw-p 00000000 00:00 0 
7fab1402c000-7fab18000000 ---p 00000000 00:00 0 
7fab18df3000-7fab18ecc000 rw-p 00000000 00:00 0 
7fab18fe4000-7fab1900e000 rw-p 00000000 00:00 0 
7ffe8ab88000-7ffe8aba9000 rw-p 00000000 00:00 0                          [stack]
7ffe8abd9000-7ffe8abdb000 r--p 00000000 00:00 0                          [vvar]
7ffe8abdb000-7ffe8abdd000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Abandon

I get a lot of free RAM

[Milad021]

Hi @ReedWarbler do you have any idea ?