NextFlow pipeline process hangs after silently dying in SGE about nextflow

[pditommaso]

Yes, the best is that you manage to set bash as the default SGE shell. Have a look to this
http://stackoverflow.com/questions/2020957/how-to-change-the-default-shell-for-the-sun-grid-engine

An alternative workaround consists in adding the following entry in your nextflow.config:

process.clusterOptions = '-S /bin/bash' 

[pditommaso]

Does it run correctly without configuring the SGE executor?

Also do you launch it in a shared folder? I mean, your working directory is accessible by the cluster nodes?

[pditommaso]

Unable to reproduce

[andrewcstewart]

The pipeline works fine without SGE executor, and it’s on a cluster shared folder.  I’m also able to run non-nextflow jobs on the cluster (it’s a single host cluster just for testing).  I believe it’s an SGE problem, so I’ll investigate on that side.  Just curious if this is a common problem for nextflow users.

—
Sent from Mailbox

On Mon, Nov 24, 2014 at 11:45 PM, Paolo Di Tommaso
[email protected] wrote:

Does it run correctly without configuring the SGE executor?
Also do you launch it in a shared folder? I mean, your working directory is accessible by the cluster nodes?

---

Reply to this email directly or view it on GitHub:
#21 (comment)

[pditommaso]

We are using Nextflow with SGE heavily, so it should work.

In the directory where you are launching it, there is a folder starting with work/ad/b159ca (it is the directory where the jobs is actually executed) could you tar it and attach it to this issue, so I can try to debug it?

Also you want to try to execute the pipeline using the following command:

nextflow -trace nextflow.executor.GridTaskHandler run <pipeline name>

It will dump the qsub command line used to submit the job execution.

[andrewcstewart]

Will do (on Monday btw) .

[andrewcstewart]

Ok, here's the output of the sub command dump:

nextflow -trace nextflow.executor.GridTaskHandler run tutorial.nf
N E X T F L O W  ~  version 0.11.2
[warm up] executor > sge


TRACE: submit splitLetters (1) > cli: [qsub, -wd, /home/astewart/pipelines/pipeline1/work/6f/f5be1c08597d0961db1b75d9d888a7, -N, nf-splitLetters_1, -o, /dev/null, -j, y, -terse, -V, -notify, -q, all.q, .command.run]
TRACE: submit splitLetters (1) > exit: 0
35


[6f/f5be1c] Submitted process > splitLetters (1)

I'll attach the tarball as well.
Edit: forgot you can't upload attachments to github issues. Here's the file contents as a gist: https://gist.github.com/andrewcstewart/c91fff97ca836c65d41c

[pditommaso]

When nextflow launches a job with the SGE, it looks for a file named ".exitcode" that notifies the job termination (and eventually reports the error exit status).

I don't see this file in the ones you attached. Also I've noticed that you are running the job in the SGE cluster by using the Docker engine.

It may be difficult to have Docker running with SGE. Did you try to execute that example without using the Docker feature?

[andrewcstewart]

Hm, interesting clue. So I could try manually creating .exitcode for a frozen job and see if that forces nextflow to continue past that point. My suspicion then would be that there's something wrong with my SGE configuration. I can try to run the qsub command manually.

Yes, I've tried running it without Docker. I should have provided the simpler example. I've updated the .command files in my my gist after running without Docker: https://gist.github.com/andrewcstewart/c91fff97ca836c65d41c

[pditommaso]

Yes, I would suggest that you try to replicate the problem by running the qsub command manually without using docker.

Also you may want to add the set -x debug directive just after the shebang header in the bash script so that you will able to track what is going wrong.

[andrewcstewart]

Yes, I already replicated the problem by running qsub without docker (I updated the gist with the results)

—
Sent from Mailbox

On Mon, Dec 1, 2014 at 1:55 PM, Paolo Di Tommaso [email protected]
wrote:

Yes, I would suggest that you try to replicate the problem by running the qsub command manually without using docker.

Also you may want to add the set -x debug directive just after the shebang header in the bash script so that you will able to track what is going wrong.

Reply to this email directly or view it on GitHub:
#21 (comment)

[pditommaso]

If so, change to the folder where the .command.run file is and the following qsub command

qsub -wd $PWD -N nf-splitLetters_1 -o $PWD/qsub.out -j y -terse -V -notify -q all.q .command.run

Could you please then copy the content of the qsub.out file created by it ?

[andrewcstewart]

Sure.

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Badly placed ()'s.

[pditommaso]

OK, I think we got it. It looks your SGE is using a wrong shell. Try to submit the job like, before, but using the following command:

qsub -S /bin/sh -wd $PWD -N nf-splitLetters_1 -o $PWD/qsub.out -j y -terse -V -notify -q all.q .command.run

[andrewcstewart]

Aha, that would make sense. Got a syntax error now.

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
Badly placed ()'s.
/var/spool/gridengine/default/napa/job_scripts/41: line 6: syntax error near unexpected token `<'
/var/spool/gridengine/default/napa/job_scripts/41: line 6: `    done < <(ps -e -o pid= -o ppid=)'

[pditommaso]

Still no joy, my mistake sorry. It should be bash not sh. Please try this:

qsub -S /bin/bash -wd $PWD -N nf-splitLetters_1 -o $PWD/qsub.out -j y -terse -V -notify -q all.q .command.run

[andrewcstewart]

Aha, success!

Ok, it finished with:

qsub -S /bin/bash -wd $PWD -N nf-splitLetters_1 -o $PWD/qsub.out -j y -terse -V -notify -q all.q .command.run

I now have two files, chunk_aa and chunk_ab.

So I'm guessing I need to set a default shell in SGE or modify the cextflow.config or something like that?

[andrewcstewart]

Excellent. Thanks so much for your time! I'm stoked to try out nextflow on SGE. (I might even try to work docker back in there. That's a really sweet combo).

[pditommaso]

Good! Feel free to join the Nextflow google group if you need further help or to simply share you experience with it.

https://groups.google.com/forum/#!forum/nextflow

Best, p