ecogenomics / checkm Goto Github PK

View Code? Open in Web Editor NEW

324.0 324.0 73.0 6.78 MB

Assess the quality of microbial genomes recovered from isolates, single cells, and metagenomes

Home Page: https://ecogenomics.github.io/CheckM/

License: GNU General Public License v3.0

Python 100.00%

checkm's People

Contributors

Stargazers

Watchers

Forkers

ctskennerton hunter-cameron wwood ofanoyi fw1121 iueayhu yjarosz palc zhssakura liupfskygre kehuixie chinaargs jayrbolton prehensilecode hadrieng grupo-de-oceanografia-costeira vinisalazar ravinpoudel pythseq suzuki-shm juleskers loisgordils asafpr nickp60 ctb audy luernerliu hcdenbakker mza0150 msm772083034 rheannna tarah28 bernt-matthias mengqingren ashwinssudarshan zhaoxia413 avanisinghal1 wanliu2019 jsgounot driftbio seedpcseed dyxstat bikmi zjyzjjzmt alexpersa7 wangdi2014 computational-genomics-lab mmastert rommelramos azeem-svg chaolab wook2014 yananzh jbitencourt alienzj luisvacpr mianlee jjacobson95 maxibor misialq 25280841 puncocharm ivanv87 mvries spitfiredd louis-mg daxlal muweidong2021 crowmane420 abubakariabdulwasid sophieseuleson

checkm's Issues

Prodigal output parsing problem

Hi, the prodigal output format seems to have changed between versions 2.60 and 2.61/2.62. The new output can't be parsed.

Example of old output line:

# Model Data: version=Prodigal.v2.60;run_type=Single;model="Ab initio";gc_cont=35.88;transl_table=11;uses_sd=1

New output line:

# Model Data: gc_cont=35.88;transl_table=4;uses_sd=1

The error is:

  File "/apps/python/lib/python2.7/site-packages/checkm/prodigal.py", line 178, in __parseGFF
    self.translationTable = line.split(';')[4]
IndexError: list index out of range

Also, another small request: In the file "checkm", please change the line

versionFile = open(os.path.join(binDir, '..', 'checkm', 'VERSION'))

import checkm; versionFile = open(os.path.join(checkm.__path__[0], 'VERSION'))

or something similar, this would be more stable.

"checkm tetra" fails on lowercase sequence data

Depending on the assembler and primary sequence data, contigs can contain lowercase letters. This leads to 'nan' values in the output of the tetra command. The workaround is to map the sequence data to upper case before. However, this should be handled in Python very easily.

missing_duplicate_genes_50.tsv

Hey Donovan, Mike,

it was good meeting both of you at ISME a few weeks ago.
I've tried installing and running CheckM, but run into an issue when running 'lineage_set'. The treeParser.py script seems to be looking for a file called 'missing_duplicate_genes_50.tsv' (line 385) in the genome_tree/ folder of the data directory.

I can't find that file, neither locally nor on https://data.ace.uq.au/public/CheckM_databases/

hope you can fix that (and that I'm not overlooking something silly here)
Thanks!
Daan Speth

unknown command data

Following the installation instructions here: https://github.com/Ecogenomics/CheckM/wiki/Installation

After installing version 0.4 using pip, I ran 'checkm data update', and got the following error:

unknown command: data

The checkm help also does not list a 'data' command:

checkm
checkm [options] command filenames

Commands:

checkm write [checkm filename (default:checkm.txt) [filepath (default='.')]]
- writes a checkm manifest file to disc for the files in the given filepath.
Use -r to include all files under a given path in a single manifest.

checkm print [filepath (default='.')]
- As for 'write', but will print the manifest to the screen.

checkm multi [checkm filename (default:checkm.txt) [filepath (default='.')]]
- writes a checkm manifest file to disc for the files in the given filepath, recursively creating a manifest file within each subdirectory and using the '@' designation in the parent checkm files above it.

checkm check [checkm filename (default:checkm.txt)]
- checks the given checkm manifest against the files on disc.
Use -m to recursively scan through any multilevel checkm files it finds in this manifest as well.

checkm remove_multi [checkm filename (default:checkm.txt)]
- scans through the checkm file, recursively gathering a list of all included checkm manifests, returning the list of files.
Use the option '-f' or '--force' to cause the tool to try to delete these checkm files.

checkm tree_qa throws AttributeError

Hi,

I am using checkM version 1.0.3. I installed all yesterday, ran the "checkm test" and no issues came up.

Now, I got an error when analyzing my own genome bins, as explained below:

command 1: checkm tree -t 20 ./ ./checkm_tree -x .fa

completed with no issues

command 2: checkm tree_qa -f ./checkm_tree.txt ./checkm_tree

error

[CheckM - tree_qa] Assessing phylogenetic markers found in each bin.

Reading HMM info from file.
Parsing HMM hits to marker genes:
Finished parsing hits for 3600 of 3600 (100.00%) bins.

Unexpected error: <type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/srv/sw/checkm/1.0.3/bin/checkm", line 709, in
checkmParser.parseOptions(args)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/main.py", line 1197, in parseOptions
self.treeQA(options)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/main.py", line 177, in treeQA
treeParser.printSummary(options.out_format, options.tree_folder, RP, options.bTabTable, options.file, binStats)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/treeParser.py", line 45, in printSummary
self.reportBinTaxonomy(outDir, resultsParser, bTabTable, outFile, binStats, bLineageStatistics=False)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/treeParser.py", line 633, in reportBinTaxonomy
binIdToTaxonomy = self.getBinTaxonomy(outDir, binIds)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/treeParser.py", line 216, in getBinTaxonomy
domainNode = self.__findDomainNode(node)
File "/srv/sw/checkm/1.0.3/lib/python2.7/site-packages/checkm/treeParser.py", line 237, in __findDomainNode
curNode = curNode.parent_node

AttributeError: 'NoneType' object has no attribute 'parent_node'

Would it be possible to clarify? Maybe I am running the commands incorrectly or my installation is faulty ...

Thank you very much in advance for your help.

Dieter.

CheckM installation check error

Hi,
I don't have admin rights on our server and our IT guy installed CheckM for us. The command options show up just fine on type checkm, but when on trying the installation check test command I got the following error. It doesn't make a whole lot of sense to me, but my guess is that Python version 2.6 is one issue and have already requested an upgrade. But can you have a peek at the error message and let me know if something else is missing too (e.g. Error: Unrecognized format, trying to open hmm file /tmp/be078517-7f20-4792-9116-a243633e5715 for reading) so that I can ask our admin to install it ?

Command used : checkm test ~/checkm_test_results

[CheckM - Test] Processing E.coli K12-W3310 to verify operation of CheckM.

[Step 1]: Verifying tree command.

[CheckM - tree] Placing bins in reference genome tree.

Identifying marker genes in 1 bins with 1 threads:
Finished processing 0 of 1 (0.00%) bins.
Error: Unrecognized format, trying to open hmm file /tmp/be078517-7f20-4792-9116-a243633e5715 for reading.

Finished processing 1 of 1 (100.00%) bins.

Saving HMM info to file.

Unexpected error: <type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/usr/bin/checkm", line 717, in
checkmParser.parseOptions(args)
File "/usr/lib/python2.6/site-packages/checkm/main.py", line 1267, in parseOptions
self.test(options)
File "/usr/lib/python2.6/site-packages/checkm/main.py", line 1156, in test
verifyEcoli.run(self, options.output_dir)
File "/usr/lib/python2.6/site-packages/checkm/test/test_ecoli.py", line 64, in run
parser.tree(options)
File "/usr/lib/python2.6/site-packages/checkm/main.py", line 129, in tree
markerSetParser.writeBinModels(binIdToModels, hmmModelInfoFile)
File "/usr/lib/python2.6/site-packages/checkm/markerSets.py", line 471, in writeBinModels
with gzip.open(filename, 'wb') as output:
AttributeError: GzipFile instance has no attribute 'exit'

Thanks,
Neha

Error when output folder is the current directory

Is there any way to de-contamination?

Hi ~

Since checkm can find the contamination as multiple copies of marker genes, is there any way to locate and remove these copies in contigs/scaffolds files?

Best
Yifan

AttributeError when using checkm coverage

When I run "checkm coverage" I keep getting AttributeErrors. an outputfile is produced, but the coverage of every contig in every bin is given as zero.
I am using sorted an indexed Bam files (mappings done with bowtie, then converted to .bam-format, sorted and indexed using samtools).

This is the error I run into:
Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python2.7/dist-packages/checkm/coverage.py", line 208, in __workerThread elif read.is_secondary or read.is_supplementary: AttributeError: 'pysam.calignmentfile.AlignedSegment' object has no attribute 'is_supplementary'

I am using CheckM v1.0.4
the command I ran is:
checkm coverage -x fa -t 8 ./ checkm_coverage.out input/myinput_reads.BAM

Edit: my pysam version is 0.8.2.1 (in case that may be part of the problem)

checkm lineage_wf memory usage

Hi,
I've been recently started to use CheckM to evaluate the bins from my metagenomic data but I'm getting stuck in the first step of lineage_wf due to the amount of memory the program is using. I even tried the -r option but it exceeds 80 GB of requested memory instead of being limited to ~14GB as exposed in the Quick Start page of the CheckM wiki.

Any solution for this? Thank you

My bins folder contains 507 bins

My command:
checkm lineage_wf -r -t 8 -x fa ./bins ./outdir

Log:

[CheckM - tree] Placing bins in reference genome tree.

Identifying marker genes in 507 bins with 8 threads:
Finished processing 0 of 507 (0.00%) bins.^M [.....] Finished processing 507 of 507 (100.00%) bins.^M
Saving HMM info to file.

Calculating genome statistics for 507 bins with 8 threads:
Finished processing 0 of 507 (0.00%) bins.^M [.....] Finished processing 507 of 507 (100.00%) bins.^M

Extracting marker genes to align.
Parsing HMM hits to marker genes:
Finished parsing hits for 1 of 507 (0.20%) bins.^M [.....] Finished processing 507 of 507 (100.00%) bins.^M$
Extracting 43 HMMs with 8 threads:
Finished extracting 0 of 43 (0.00%) HMMs.^M [.....] Finished processing 507 of 507 (100.00%) bins.^M
Aligning 43 marker genes with 8 threads:
Finished aligning 0 of 43 (0.00%) marker genes.^M [.....] Finished processing 507 of 507 (100.00%) bins.^M

Reading marker alignment files.
Concatenating alignments.
Placing 507 bins into the genome tree with pplacer (be patient).
=>> PBS: job killed: vmem 135223554048 exceeded limit 85899345920

problem with pfam.hmm formatting?

I think I'm having issue with the pfam format of the "phyla.hmm" file that was created by

checkm data update
however, it could be something else. As a test, I tried the lineage_wf command with the ecoli genome you supplied in test_data. Please see errors below. Thanks.

what I did to install and test checkm lineage_wf

installing with pip

pip install numpy
pip install checkm-genome
./bin/checkm data update
I typed "DATA" to save data in

I added prodigal, hmmer, and pplacer to PATH

test installation with test data (ecoli)

./bin/checkm lineage_wf DATA/test_data/ Results

The output was

[CheckM - tree] Placing bins in reference genome tree.

Identifying marker genes in 1 bins with 1 threads:
Finished processing 0 of 1 (0.00%) bins.

Error: Unrecognized format, trying to open hmm file /global/projectb/scratch/jfroula/965c9772-6a8a-48d3-9b67-a5a541d84085 for reading.

Finished processing 1 of 1 (100.00%) bins.

Saving HMM info to file.

Calculating genome statistics for 1 bins with 1 threads:
Finished processing 0 of 1 (0.00%) bins.
Process Process-4:
Traceback (most recent call last):
File "/usr/common/usg/languages/python/2.7.4/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/common/usg/languages/python/2.7.4/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "/global/scratch2/sd/jfroula/CheckM/git/CheckM/checkm/binStatistics.py", line 113, in __processBin
codingDensity, translationTable, numORFs = self.calculateCodingDensity(binDir, genomeSize, scaffoldStats)
File "/global/scratch2/sd/jfroula/CheckM/git/CheckM/checkm/binStatistics.py", line 229, in calculateCodingDensity
codingBasePairs = self.__calculateCodingBases(aaGenes, seqStats)
File "/global/scratch2/sd/jfroula/CheckM/git/CheckM/checkm/binStatistics.py", line 240, in __calculateCodingBases
seqStats[scaffoldId]['# ORFs'] = seqStats[scaffoldId].get('# ORFs', 0) + 1
KeyError: 'AC_000091_1'

Extracting marker genes to align.
Parsing HMM hits to marker genes:
Finished parsing hits for 1 of 1 (100.00%) bins.
[Errno 2] No such file or directory: 'Results/bins/637000110/hmmer.tree.txt'

Extracting 43 HMMs with 1 threads:
Finished extracting 0 of 43 (0.00%) HMMs.