GTF and BED files into Channels about chipseq

[chuan-wang]

No more bug after nextflow v0.32. Codes fixed in PR #53

[drejom]

Actually, it's very reproducible now - can't get it to run without this error at all. I managed to get one run to start yesterday and it made it part the way through, but no joy today. I've tried with --genome GRCh37 using iGenomes and a modified GRCh37 (with an additional construct) with --gtf --bed --fasta --bwa_index specified as above and get the same error in both cases.

Usually its the error from line 209, but I've also had it stop at line 214. I'll try tidying up the failed runs and see if I can reproduce it consistently...sorry its been an unbridled frenzy of job resubmissions!

[drejom]

Thanks @ewels, totally see what yr getting at...I'll take this approach in the future for sure. Was just puzzled because resubmitting the job witout modification failed. In any case, nextflow run nf-core/chipseq -profile test,singularity -c nextflow.dback -r dev runs.

Looks like @pditommaso is on to it - running with @chuan-wang's PR #73 all goes smoothly and I can resume the failed run from yesterday. Can probably merge the PR and close this issue?

[drpatelh]

Great @drejom Thanks for giving it a go! I should say that it is likely to change in portions over the next few weeks as I test and document it. Also, Im sure I will find edge cases where I will need to go back to the blackboard! However, if you want some quick and dirty results for now then Im happy to provide bugfixes until I create an official PR. I guess the best place to chat would be the chipseq channel on Slack.

I have fixed the channel naming for single-end, and added an option called --skipDiffAnalysis that allows you to skip the differential analysis step. Not sure whats going on with the latter...

[drejom]

Righto, I have it running now...catch you on slack. Thanks so much!

[chuan-wang]

Apologize. This issue has to be reopen after more tests. I restore the previous code because the bug is still there.

[drejom]

Still having this issue (NXF_VER 19.01.1). Strangely its temperamental - some runs it works without a problem, then rerunning the exact command (ie arrowing up on the prompt) it fails.

[ewels]

I'm a little lost as to exactly what the problem is here. Could you please paste the nextflow output from a failed run?

[drejom]

This ran part way through before failing due to a walltime limit. Trying to rerun, I get:

nextflow run nf-core/chipseq  -c ../../config/nextflow.dback      -with-singularity $HOME/singularity-images/nfcore-chipseq-dev.img     -profile singularity -resume   --singleEnd --reads '../../data/*.fastq'     --saveReference --genome GRCh37     --singleEnd --saturation --blacklist_filtering --allow_multi_align      --macsconfig "../../config/macsconfig.csv"     --fasta "../../data/GRCh37_pMoHIV1Plasmid.fa"     --gtf "../../data/GRCh37_pMoHIV1Plasmid.gtf"  --bed "../../data/GRCh37_pMoHIV1Plasmid.bed"   --bwa_index "../../data/BWAIndex/genome.fa"     --outdir ../../results/nf-chipseq      --email [email protected] -r dev
N E X T F L O W  ~  version 19.01.0
Launching `nf-core/chipseq` [silly_bhaskara] - revision: 492bc8cb60 [dev]
WARN: The `into` operator should be used to connect two or more target channels -- consider to replace it with `.set { bwa_index }`
ERROR ~ 1

 -- Check script 'main.nf' at line: 183 or see '.nextflow.log' file for more details
ERROR ~ No FastQ file found for sample in MACS config: [pcDNA-1, pcDNA-1-INP, pcDNA-2, pcDNA-2-INP, pcDNA-3, pcDNA-3-INP, F2-1, F2-1-INP, F2-2, F2-2-INP, F2-3, F2-3-INP, ZFP-1, ZFP-1-INP, ZFP-2, ZFP-2-INP, ZFP-3, ZFP-3-INP]

 -- Check '.nextflow.log' file for details

[ewels]

Ok, so there are a couple of error messages here.

Firstly, line 183 is to do with setting sample details based on an input file:

chipseq/main.nf

Lines 175 to 187 in 492bc8c

	/*
	* Create a channel for macs config file
	*/
	Channel
	.from(macsconfig.readLines())
	.map { line ->
	list = line.split(',')
	chip_sample_id = list[0]
	ctrl_sample_id = list[1]
	analysis_id = list[2]
	[ chip_sample_id, ctrl_sample_id, analysis_id ]
	}
	.into { vali_para; macs_para; saturation_para }

So there's something funny about how nextflow is reading and parsing this file (../../config/macsconfig.csv).

The second message is from line 209:

chipseq/main.nf

Lines 207 to 210 in 492bc8c

	def missing_samples = config_samples - config_samples.intersect(fastq_samples)
	if(!missing_samples.isEmpty()){
	exit 1, "No FastQ file found for sample in MACS config: ${missing_samples}"
	}

This kind of figures, as if the macsconfig parsing has failed then there will be no intersection between the FastQ inputs and the config file.

Could you please post .nextflow.log so that we can look to see if nextflow is giving us any more information about what is going wrong?

[drejom]

Thanks for your help once more @ewels, much appreciated!
Heres the nextflow.log

For what its worth, ../../config/macsconfig.csv looks like:

pcDNA_1,pcDNA_1_INP,pcDNA_1
pcDNA_2,pcDNA_2_INP,pcDNA_2
pcDNA_3,pcDNA_3_INP,pcDNA_3
F2_1,F2_1_INP,F2_1
F2_2,F2_2_INP,F2_2
F2_3,F2_3_INP,F2_3
ZFP_1,ZFP_1_INP,ZFP_1
ZFP_2,ZFP_2_INP,ZFP_2
ZFP_3,ZFP_3_INP,ZFP_3

[Note that I changed '-' to '_' in macsconfig and renamed the fastqs in the interim to see if it helped, but no diff]

[ewels]

This is weird.. And how reproducible is this error? 1 in how many runs? Do you always get exactly the same error message?

@pditommaso - can you see anything suspicious in this log file? The traceback doesn't mean much to me..

[ewels]

In order to fix this, I think we need to get away from messy environments with job resubmissions and get to a clearer picture - a small reproducible example if possible.

What happens if you run this in a clean directory?

nextflow run nf-core/chipseq -profile test,singularity

Does the error start if you then add in your config file to that command (-c ../../config/nextflow.dback)..? You see what I'm getting at - trying to add in one piece at a time until we get to the same mode of failure. Once we can reliably reproduce the problem, we can start to investigate it.

Phil

[pditommaso]

What it looks suspicious is the definition of chip_sample_id and other var in the global scope. I would strongly suggest to define them locally e.g.

 Channel 
     .from(macsconfig.readLines()) 
     .map { line -> 
         list = line.split(',') 
         def chip_sample_id = list[0] 
         def ctrl_sample_id = list[1] 
         def analysis_id = list[2] 
         [ chip_sample_id, ctrl_sample_id, analysis_id ] 
     }

note: the use of def

[ewels]

Thanks @pditommaso!

PR merged - @drejom if you could test it that would be great. If it resolves the problem we can close the issue... 😉

Phil

[drejom]

Hi again...I've just given this another go with a new experiment and find the same issue with this latest commit with PR#73 merged: "No FastQ file found for sample in MACS config."

Initially, the pipeline started and looked to be ok. I interrupted the job and resubmitted, but now have this error again. nextflow.1.log is the initial successful run, nextflow.log shows the error. If I start the run again in a clean working dir, it proceeds without error.

Running with the test profile I cannot reproduce this error, ie I can interrupt and resubmit
nextflow run nf-core/chipseq -profile test,singularity -c nextflow.apollo -r dev -resume
multiple times without any complaints.

Any other ideas @pditommaso or @chuan-wang?

[drpatelh]

@drejom I am working on a PR for an updated version of the pipeline. I just need to add in a little more functionality, docs and test it a bit more. I'm not sure how easy it's going to be to debug and fix your issue. If this is an independent run of the pipeline (i.e. you don't need to process these samples in the same way as older samples) then you can try and run the pipeline using:

nextflow run drpatelh/chipseq -c ../../config/nextflow.apollo -profile singularity -resume --design ../../config/design.csv --genome GRCh37 --outdir ../../results/nf-chipseq.GRCh37 --email [email protected]

Based on the parameters you are using the only notable difference is that the input data has to be provided in a design file.
https://github.com/drpatelh/chipseq/blob/master/docs/usage.md#full-design

This should overcome the issues you are facing because the read information and controls are provided in a single file that is validated using a python script beforehand.

I am still working on the docs so it's not all there yet but it would be a good test if you can get it to run through.

[drejom]

Awesome @drpatelh that's great news. I've given it a go and can get it running though not to completion. I've found a few issues popping up - I can move the discussion to your repo if you enable issues there?

Eg the process deseqConsensusPeakSet stops with the error:
Error: package or namespace load failed for ‘DESeq2’ in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]): there is no package called ‘xfun’

And to get the pipeline running with single end data, lines 709 and 712 of main.nf should read

ch_filter_bam_flagstat.into instead of ch_filter_flagstat.into
ch_filter_bam_stats_mqc.set instead of ch_filter_stats_mqc.set

[drpatelh]

Should be fixed in #76