morinlab / kronos_pipelines Goto Github PK

Currently, we parallelize on chromosome. However, the largest chromosomes (e.g. chr1, chr2) still take 18-20 hours to run on genomes. We can decrease the wall time for this pipeline by splitting up the chromosomes in smaller chunks.

One simple splitting approach is to split at the centromeres, but we still might want to split even more than that.

Hello,

I run the Kronos version of TITAN using the default --num_jobs and --num_pipelines in launch.sh file. The other setting were okay. Submitting high number of jobs caused issues in login nodes and lead to killing the jobs and restarting the nodes by system. I rerun the pipeline using lower numbers for --num_jobs(100) and --num_pipelines(10) and it is working fine so far.

Arezoo

It appears the warning given by samtools is causing the second step in the Sequenza pipeline to fail. All I see in the log is:

Warning: The index file is older than the data file: /genesis/extscratch/morinlab/shared/rmorin/sequenza_DLBCL/data/bam_alignments/PAN44N.rmdup.bam.bai
Warning: The index file is older than the data file: /genesis/extscratch/morinlab/shared/rmorin/sequenza_DLBCL/data/bam_alignments/PAN44.rmdup.bam.bai

Due to a bug in Kronos v2.0.4, when the --no_prefix option is specified, this breaks parallelized tasks because the output file names aren't updated properly. So, parallelized tasks all write to the same output file, causing multiple issues.

I've opened an issue with Jafar (see link). The obvious workaround is to omit the --no_prefix option with the caveat of uglier file names. I will restore it once the bug is fixed.