citiususc / bigseqkit Goto Github PK

View Code? Open in Web Editor NEW

46.0 46.0 3.0 235 KB

BigSeqKit: a parallel Big Data toolkit to process FASTA and FASTQ files at scale

License: GNU General Public License v3.0

Dockerfile 0.50% Shell 0.03% Go 84.25% Python 15.22%

bigseqkit's People

Contributors

Stargazers

Watchers

Forkers

animesh genostack wook2014

bigseqkit's Issues

Error when running ignis-submit in wsl2

This should use jre and lib in ignis folder as I understand. But why it still report this error?

Exception in thread "main" java.lang.NoSuchMethodError: 'void com.fasterxml.jackson.core.JsonGenerator.writeStartArray(java.lang.Object, int)' at com.fasterxml.jackson.databind.ser.impl.StringArraySerializer.serialize(StringArraySerializer.java:170) at com.fasterxml.jackson.databind.ser.impl.StringArraySerializer.serialize(StringArraySerializer.java:22) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:722) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:166) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4110) at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3437) at com.github.dockerjava.core.DefaultInvocationBuilder.encode(DefaultInvocationBuilder.java:292) at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:121) at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:37) at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13) at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:609) at org.ignis.scheduler.Docker.createDriverContainer(Docker.java:318) at org.ignis.submitter.Submit.call(Submit.java:186) at org.ignis.submitter.Submit.call(Submit.java:46) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine.execute(CommandLine.java:2078) at org.ignis.submitter.Submit.main(Submit.java:284)

seqkit stats in single machine with 20-32 threads

Hi, would bigseqkit improve the speed of seqkit stats in single machine with 20-32 threads? Thx

platform question

Since ignis-submit calls exec ${IGNIS_HOME}/backend/jre/bin/java and that java is for x86-64 linux, does that mean that is the only OS that can be used?

How to run in a HPC cluster?

Following the guide .

We assume that the cluster has installed Slurm and Singularity.

We do have them.

However, the first step failed:

ignis-deploy images singularity --host ignishpc/full ignis_full.sif
DockerException:  Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

And the second step requires docker, which is not allowed and available. Our HPC administrators do not allow installing docker.

docker run --rm -v $(pwd):/target ignishpc/slurm-submitter ignis-export /target

Failed to run locally.

Hi, I'm trying to install and run BigSeqKit locally.

Here are the commands. There's an error of Permission denied, but it remains after changing ownership of the ignis directory.

# no error
sudo docker pull ignishpc/full

cd ~/Downloads
# no error
docker run --rm -v $(pwd):/target ignishpc/submitter ignis-export /target

# set current directory as job directory
export IGNIS_DFS_ID=$(pwd)
# set docker as scheduler
export IGNIS_SCHEDULER_TYPE=docker
# set where docker is available
export IGNIS_SCHEDULER_URL=/var/run/docker.sock

./ignis/bin/ignis-submit ignishpc/full bigseqkit
Mar 10, 2024 09:03:37 <ERROR> org.ignis.submitter.Submit.call(Submit.java:271): java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:179) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:37) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.11.jar:?]
        at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:609) ~[docker-java-core-3.2.11.jar:?]
        at org.ignis.scheduler.Docker.createDriverContainer(Docker.java:318) ~[ignis-scheduler-docker-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:186) [ignis-submitter-1.0.jar:?]
        at org.ignis.submitter.Submit.call(Submit.java:46) [ignis-submitter-1.0.jar:?]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.access$1300(CommandLine.java:145) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at picocli.CommandLine.execute(CommandLine.java:2078) [picocli-4.6.1.jar:task ':jar' property 'archiveVersion']
        at org.ignis.submitter.Submit.main(Submit.java:284) [ignis-submitter-1.0.jar:?]
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:63) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.<init>(LinuxDomainSocket.java:41) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:137) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:409) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:164) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:174) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:135) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:128) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67) ~[httpclient5-5.0.3.jar:5.0.3]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:175) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        ... 19 more
Caused by: com.sun.jna.LastErrorException: [13] Permission denied
        at com.github.dockerjava.transport.LinuxDomainSocket.connect(Native Method) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.connect(LinuxDomainSocket.java:49) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.open(DomainSocket.java:69) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:59) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.LinuxDomainSocket.<init>(LinuxDomainSocket.java:41) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138) ~[docker-java-transport-3.2.11.jar:?]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:137) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:409) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:164) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:174) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:135) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:128) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) ~[httpclient5-5.0.3.jar:5.0.3]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67) ~[httpclient5-5.0.3.jar:5.0.3]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:175) ~[docker-java-transport-httpclient5-3.2.11.jar:?]
        ... 19 more

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.