cuffcomapre can't find transcript.gtf files about perl-for-bioinformatics

[biocoder]

Yes. Please comment out these lines:

cuffcompare:
 - -i /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/assembly_list.txt

Change overwriteOutputDir: NO to overwriteOutputDir: YES.

Also, please make sure your $SHELL is /bin/bash. See here: https://github.com/biocoder/Perl-for-Bioinformatics/tree/master/NGS-Utils#caveats

If after following these steps, it is still failing, try executing the cuffcompare command with -V option to see if cuffcompare prints any extra debug information.

/usr/local/bin/cuffcompare -V -T -o /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare/lncRNApipe_cuffcmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genes.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_jJMDrZlC1UCY.2cells/transcripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_jJMDrZlC1UCY.6h/transcripts.gtf 

If none of the suggestions above work, please let me know.

Cheers.

[yeroslaviz]

Hi, thanks for the fast answer. Unfortunately this didn't work.
the two lines are commented out and the overwrite is active.
I have also checked for the bash/dash problem.

There are 2 choices for the alternative sh (providing /bin/sh).                                                                                                                                                                                              
  Selection    Path            Priority   Status
------------------------------------------------------------
  0            /bin/dash        200       auto mode
* 1            /bin/bash        100       manual mode
  2            /bin/dash        200       manual mode

But when I am running the script again, it goes all the way, but doesn't manage to run cuffcompare.

This is the output of the log file from the postAssembly step:

cat lncRNApipe/lncRNApipe-test/run/lncRNApipe.postAssembly.run_Gvj2dFgCL1ZI.log 

Tue Feb 13 10:05:47 2018        Validating options...

Tue Feb 13 10:05:47 2018        Starting ☲☴ lncRNApipe Pipeline...

Tue Feb 13 10:05:47 2018        ########################### Module 1: Running cuffcompare... #########################################

Making output directory for cuffcompare [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare ]


Command call:
-------------
/usr/local/bin/cuffcompare -T -o /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare/lncRNApipe_cuffcmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genes.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.2cells/transcripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf 

Error: cannot locate input file: /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf

INFO!!
--------
Using local FASTA reference [ genome.fa ] to fetch sequences to maintain consistency.
** Requires Bio::SeqIO module to be installed and available **


Tue Feb 13 10:05:47 2018        ########################### Module 2: Running categorize_ncRNAs.pl ###################################

Cannot find Cuffcompare tracking [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare/lncRNApipe_cuffcmp.tracking ] file...

ERROR

Tue Feb 13 10:05:47 2018        ☲☴ lncRNApipe Pipeline aborted(?)



Tue Feb 13 10:05:47 2018        Validating options...

Tue Feb 13 10:05:47 2018        Starting ☲☴ lncRNApipe Pipeline...

Tue Feb 13 10:05:47 2018        ########################### Module 1: Running cuffcompare... #########################################

Making output directory for cuffcompare [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffdiff_known_ncRNAs/cuffcompare ]


Command call:
-------------
/usr/local/bin/cuffcompare -T -o /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffdiff_known_ncRNAs/cuffcompare/lncRNApipe_cuffcmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/zf_noncode2016.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.2cells/transcripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf 

Error: cannot locate input file: /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf

INFO!!
--------
Using local FASTA reference [ genome.fa ] to fetch sequences to maintain consistency.
** Requires Bio::SeqIO module to be installed and available **


Tue Feb 13 10:05:47 2018        ########################### Module 2: Running categorize_ncRNAs.pl ###################################

Cannot find Cuffcompare tracking [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffdiff_known_ncRNAs/cuffcompare/lncRNApipe_cuffcmp.tracking ] file...

ERROR

Tue Feb 13 10:05:47 2018        ☲☴ lncRNApipe Pipeline aborted(?)

but the files are there:

ll /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/
total 4204
drwxrwxr-x 2 yeroslaviz yeroslaviz    4096 Feb 13 10:05 ./
drwxrwxr-x 4 yeroslaviz yeroslaviz    4096 Feb 13 10:05 ../
-rw-rw-r-- 1 yeroslaviz yeroslaviz   92361 Feb 13 10:06 genes.fpkm_tracking
-rw-rw-r-- 1 yeroslaviz yeroslaviz  149342 Feb 13 10:06 isoforms.fpkm_tracking
-rw-rw-r-- 1 yeroslaviz yeroslaviz  460485 Feb 13 10:06 run.log
-rw-rw-r-- 1 yeroslaviz yeroslaviz       0 Feb 13 10:05 skipped.gtf
-rw-rw-r-- 1 yeroslaviz yeroslaviz 3588043 Feb 13 10:06 transcripts.gtf

any ideas?

Thanks
Assa

P.S.

When running the cuffcompare command separately, it creates the following output:

/usr/local/bin/cuffcompare -T -o tmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genes.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.2cells/transcripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf 
Warning: Could not connect to update server to verify current version. Please check at the Cufflinks website (http://cufflinks.cbcb.umd.edu).

yeroslaviz@cotopaxi:MarcoN.RNASeq.GLD2$ ll -rt
total 9292
...
-rw-rw-r--  1 yeroslaviz yeroslaviz  348739 Feb 13 10:14 tmp.tracking
-rw-rw-r--  1 yeroslaviz yeroslaviz  134225 Feb 13 10:14 tmp.loci
-rw-rw-r--  1 yeroslaviz yeroslaviz 3335119 Feb 13 10:14 tmp.combined.gtf
-rw-rw-r--  1 yeroslaviz yeroslaviz    2780 Feb 13 10:14 tmp.stats

[yeroslaviz]

There is another strange behavior I have found. Your pipeline creates scripts for each of the steps.
SO i have lloked at the script created for the postAssembly step - lncRNApipe_postAssembly.run_Gvj2dFgCL1ZI.sh.
In the file I can see the command it treis to execute with the cuffcompare step (line 41).

perl /local/Assa/projects/MarcoN.RNASeq.GLD2/lncRNApipe/lncRNApipe --run /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run --cpu 16 --cuffcompare '-r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genes.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.2cells/transcripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_Gvj2dFgCL1ZI.6h/transcripts.gtf ' --cat-ncRNAs '-sample-names "2cells,6h" -len 200 -min-exons 1 -ov 80 -inc -ignore-genePred-err' --get '-ov 80 -sf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/zf_noncode2016.gtf -sff gtf' --fetch --cpc --inf --rna > /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/lncRNApipe.postAssembly.run_Gvj2dFgCL1ZI.log 2>&1

When I copy paste this command to the terminal it works and run ( i have stopped it at the CPC step, but it seems to run with no problems. )

Any ideas where the problem might be?

thanks
Assa

[biocoder]

It is strange. I never encountered the error specific to your case. The Error: cannot locate input file: error is an error from cuffcompare which is installed at /usr/local/bin. Just to rule out any issues with that binary:

Can you edit the file /local/Assa/projects/MarcoN.RNASeq.GLD2/lncRNApipe/.lncRNApipe.depconf and replace /usr/local/bin/cuffcompare with /local/Assa/projects/MarcoN.RNASeq.GLD2/lncRNApipe/.lncRNApipe.depbin/linux_cuffcompare and retry?

Do you see the same error with cuffcompare then?

[yeroslaviz]

yes it is the same error, just the path to the cuffcompare is different.

cat /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/lncRNApipe.postAssembly.run_2FHTl2i3VOJus.log

Tue Feb 13 16:50:08 2018        Validating options...

Tue Feb 13 16:50:08 2018        Starting ☲☴ lncRNApipe Pipeline...

Tue Feb 13 16:50:08 2018        ########################### Module 1: Running cuffcompare... #########################################

Making output directory for cuffcompare [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare ]


Command call:
-------------
/local/Assa/projects/MarcoN.RNASeq.GLD2/lncRNApipe/.lncRNApipe.depbin/linux_cuffcompare -T -o /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuf
fcompare/lncRNApipe_cuffcmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genes.gtf -s /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//
lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.2cells/transcripts.gtf /home/yeroslaviz/pro
jects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.6h/transcripts.gtf

Error: cannot locate input file: /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.6h/transcripts.gtf

INFO!!
--------
Using local FASTA reference [ genome.fa ] to fetch sequences to maintain consistency.
** Requires Bio::SeqIO module to be installed and available **


Tue Feb 13 16:50:08 2018        ########################### Module 2: Running categorize_ncRNAs.pl ###################################

Cannot find Cuffcompare tracking [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffcompare/lncRNApipe_cuffcmp.tracking ] file...

ERROR

Tue Feb 13 16:50:08 2018        ☲☴ lncRNApipe Pipeline aborted(?)



Tue Feb 13 16:50:08 2018        Validating options...

Tue Feb 13 16:50:08 2018        Starting ☲☴ lncRNApipe Pipeline...

Tue Feb 13 16:50:08 2018        ########################### Module 1: Running cuffcompare... #########################################

Making output directory for cuffcompare [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffdiff_known_ncRNAs/cuffcompare ]


Command call:
-------------
/local/Assa/projects/MarcoN.RNASeq.GLD2/lncRNApipe/.lncRNApipe.depbin/linux_cuffcompare -T -o /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuf
fdiff_known_ncRNAs/cuffcompare/lncRNApipe_cuffcmp -r /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/zf_noncode2016.gtf -s /home/yeroslaviz/projects/
MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/genome.fa /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.2cells/trans
cripts.gtf /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.6h/transcripts.gtf

Error: cannot locate input file: /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cufflinks/run_2FHTl2i3VOJus.6h/transcripts.gtf

INFO!!
--------
Using local FASTA reference [ genome.fa ] to fetch sequences to maintain consistency.
** Requires Bio::SeqIO module to be installed and available **


Tue Feb 13 16:50:08 2018        ########################### Module 2: Running categorize_ncRNAs.pl ###################################

Cannot find Cuffcompare tracking [ /home/yeroslaviz/projects/MarcoN.RNASeq.GLD2/lncRNApipe//lncRNApipe-test/run/cuffdiff_known_ncRNAs/cuffcompare/lncRNApipe_cuffcmp.tracking 
] file...

ERROR

Tue Feb 13 16:50:08 2018        ☲☴ lncRNApipe Pipeline aborted(?)

[yeroslaviz]

The problem was that cuffcompare started working before cufflinks was done and there fore has shown the error message. This was fixed and the workflow now runs all the way through.

apparently there is still some inkorrekt notification in the las status.log file after the postAssembly step. when no novel ncRNAs are found a "bailing" notification is written to the log file and therefore a notification in the last log file says "pipeline failed".