KtServerService error about cactus

[lassancejm]

Did you get pass this error?

I am seeing a lot of message similar to yours, but I am uncertain what is causing this error or how to get passed it.

Tips are welcome!

JM

INFO:toil.leader:Issued job 'KtServerService' A/q/jobvU4WyV with job batch system ID: 3 and cores: 0, disk: 2.8 G, and memory: 50.3 G
INFO:toil.leader:Job ended successfully: 'KtServerService' A/q/jobvU4WyV
WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' A/q/jobvU4WyV
WARNING:toil.leader:A/q/jobvU4WyV    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:A/q/jobvU4WyV    INFO:toil:Running Toil version 3.19.0-0feb1d4d1b4fc66062fc4dbc5d8f7fb046df39e6.
WARNING:toil.leader:A/q/jobvU4WyV    WARNING:toil.resource:'JTRES_487db7e35980364bd76c90413e988c94' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
WARNING:toil.leader:A/q/jobvU4WyV    WARNING:toil.resource:'JTRES_487db7e35980364bd76c90413e988c94' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
WARNING:toil.leader:A/q/jobvU4WyV    INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
WARNING:toil.leader:A/q/jobvU4WyV    (No info could be read for "-p": geteuid()=56685 but you should be root.)
WARNING:toil.leader:A/q/jobvU4WyV    INFO:cactus.shared.common:Running the command ['ktserver', '-port', '19362', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/tmp/toil-d26cb8ff-cabd-4073-9dff-c78622313dd7-24b9bf83-4de6-4cc2-806a-6f9b1f77386a/tmpsPKhar/e5afee6e-3501-486e-8a88-350bae38f4cf/tQlq7b6/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/tmp/toil-d26cb8ff-cabd-4073-9dff-c78622313dd7-24b9bf83-4de6-4cc2-806a-6f9b1f77386a/tmpsPKhar/e5afee6e-3501-486e-8a88-350bae38f4cf/tmpXUtETZ.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
WARNING:toil.leader:A/q/jobvU4WyV    CRITICAL:toil.lib.bioio:Error starting ktserver.
WARNING:toil.leader:A/q/jobvU4WyV    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '19362', '-host', '10.31.138.118', 'TERMINATE']
WARNING:toil.leader:A/q/jobvU4WyV    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '19362', '-host', '10.31.138.118', 'TERMINATE']
WARNING:toil.leader:A/q/jobvU4WyV    Process ServerProcess-1:
WARNING:toil.leader:A/q/jobvU4WyV    Traceback (most recent call last):
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
WARNING:toil.leader:A/q/jobvU4WyV        self.run()
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
WARNING:toil.leader:A/q/jobvU4WyV        self.tryRun(*self.args, **self.kwargs)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
WARNING:toil.leader:A/q/jobvU4WyV        raise RuntimeError("KTServer failed. Log: %s" % f.read())
WARNING:toil.leader:A/q/jobvU4WyV    RuntimeError: KTServer failed. Log: 2019-08-28T08:50:23.431847-05:00: [SYSTEM]: ================ [START]: pid=111118
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:23.432018-05:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:23.432332-05:00: [SYSTEM]: applying a snapshot file: db=0 ts=1566409684634000000 count=20479544 size=15187944389
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618643-05:00: [ERROR]: [DB]: :: 9: system error: too short region
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618705-05:00: [ERROR]: could not apply a snapshot: system error: too short region
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618807-05:00: [SYSTEM]: starting the server: expr=:19362
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618869-05:00: [SYSTEM]: server socket opened: expr=:19362 timeout=200000.0
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618880-05:00: [SYSTEM]: listening server socket started: fd=4
WARNING:toil.leader:A/q/jobvU4WyV    
WARNING:toil.leader:A/q/jobvU4WyV    CRITICAL:toil.lib.bioio:Error starting ktserver.
WARNING:toil.leader:A/q/jobvU4WyV    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job t/V/jobTdd5ip/g/tmpFoOJp5-_serialiseJob-stream used 401.26% (11.4 GB [12187840512B] used, 2.8 GB [3037418200B] requested) at the end of its run.
WARNING:toil.leader:A/q/jobvU4WyV    Traceback (most recent call last):
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/toil/worker.py", line 324, in workerScript
WARNING:toil.leader:A/q/jobvU4WyV        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
WARNING:toil.leader:A/q/jobvU4WyV        returnValues = self._run(jobGraph, fileStore)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
WARNING:toil.leader:A/q/jobvU4WyV        returnValues = self.run(fileStore)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/toil/job.py", line 1644, in run
WARNING:toil.leader:A/q/jobvU4WyV        startCredentials = service.start(self)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/cactus/pipeline/ktserverToil.py", line 33, in start
WARNING:toil.leader:A/q/jobvU4WyV        snapshotExportID=snapshotExportID)
WARNING:toil.leader:A/q/jobvU4WyV      File "/n/home01/lassance/.conda/envs/ENV_CACTUS/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 62, in runKtserver
WARNING:toil.leader:A/q/jobvU4WyV        raise RuntimeError("Unable to launch ktserver in time. Log: %s" % log)
WARNING:toil.leader:A/q/jobvU4WyV    RuntimeError: Unable to launch ktserver in time. Log: 2019-08-28T08:50:23.431847-05:00: [SYSTEM]: ================ [START]: pid=111118
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:23.432018-05:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:23.432332-05:00: [SYSTEM]: applying a snapshot file: db=0 ts=1566409684634000000 count=20479544 size=15187944389
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618643-05:00: [ERROR]: [DB]: :: 9: system error: too short region
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618705-05:00: [ERROR]: could not apply a snapshot: system error: too short region
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618807-05:00: [SYSTEM]: starting the server: expr=:19362
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618869-05:00: [SYSTEM]: server socket opened: expr=:19362 timeout=200000.0
WARNING:toil.leader:A/q/jobvU4WyV    2019-08-28T08:50:36.618880-05:00: [SYSTEM]: listening server socket started: fd=4
WARNING:toil.leader:A/q/jobvU4WyV    
WARNING:toil.leader:A/q/jobvU4WyV    ERROR:toil.worker:Exiting the worker because of a failed job on host holy2b17215.rc.fas.harvard.edu
WARNING:toil.leader:A/q/jobvU4WyV    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' A/q/jobvU4WyV with ID A/q/jobvU4WyV to 5

[RomainFeron]

Hi @lassancejm, if I remember correctly, my solution was to run Cactus with singularity. It ran better this way. Sorry I can't help you more !

[jasonsydes]

Hi @RomainFeron, thanks for posting about this. I'm receiving the same error as well. Unfortunately, we're running under Cactus under Singularity already... Do you happen to know what version of cactus you're using? (Looks like ours was installed on Oct 9, 2019...)

[RomainFeron]

Hi @jasonsydes,
I have not tried to run Cactus since I posted this issue; I was cloning cactus from the git repository, so the version I ran was ~1 year older than yours.

[jasonsydes]

Hi @RomainFeron, thanks for that info, that's helpful.