althonos / pyfastani

Cython bindings and Python interface to FastANI, a method for fast whole-genome similarity estimation.

🐍⏩🧬 PyFastANI

Cython bindings and Python interface to FastANI, a method for fast whole-genome similarity estimation. Now with multithreading!

🗺️ Overview

FastANI is a method published in 2018 by Chirag Jain et al. for high-throughput computation of whole-genome Average Nucleotide Identity (ANI). It uses MashMap to compute orthologous mappings without the need for expensive alignments.

pyfastani is a Python module, implemented using the Cython language, that provides bindings to FastANI. It directly interacts with the FastANI internals, which has the following advantages over CLI wrappers:

• simpler compilation: FastANI requires several additional libraries, which make compilation of the original binary non-trivial. In PyFastANI, libraries that were needed for threading or I/O are provided as stubs, and Boost::math headers are vendored so you can build the package without hassle. Or even better, just install from one of the provided wheels!
• single dependency: If your software or your analysis pipeline is distributed as a Python package, you can add pyfastani as a dependency to your project, and stop worrying about the FastANI binary being present on the end-user machine.
• sans I/O: Everything happens in memory, in Python objects you control, making it easier to pass your sequences to FastANI without needing to write them to a temporary file.
• multi-threading: Genome query resolves the fragment mapping step in parallel, leading to shorter querying times even with a single genome.

This library is still a work-in-progress, and in an experimental stage, but it should already pack enough features to be used in a standard pipeline.

🔧 Installing

PyFastANI can be installed directly from PyPI, which hosts some pre-built CPython wheels for x86-64 Unix platforms, as well as the code required to compile from source with Cython:

$ pip install pyfastani

In the event you have to compile the package from source, all the required libraries are vendored in the source distribution, so you'll only need a C/C++ compiler.

Otherwise, PyFastANI is also available as a Bioconda package:

$ conda install -c bioconda pyfastani

💡 Example

The following snippets show how to compute the ANI between two genomes, with the reference being a draft genome. For one-to-many or many-to-many searches, simply add additional references with m.add_draft before indexing. Note that any name can be given to the reference sequences, this will just affect the name attribute of the hits returned for a query.

🔬 Biopython

Biopython does not let us access to the sequence directly, so we need to convert it to bytes first with the bytes builtin function. For older versions of Biopython (earlier than 1.79), use record.seq.encode() instead of bytes(record.seq).

import pyfastani
import Bio.SeqIO

sketch = pyfastani.Sketch()

# add a single draft genome to the mapper, and index it
ref = list(Bio.SeqIO.parse("vendor/FastANI/data/Shigella_flexneri_2a_01.fna", "fasta"))
sketch.add_draft("S. flexneri", (bytes(record.seq) for record in ref))

# index the sketch and get a mapper
mapper = sketch.index()

# read the query and query the mapper
query = Bio.SeqIO.read("vendor/FastANI/data/Escherichia_coli_str_K12_MG1655.fna", "fasta")
hits = mapper.query_sequence(bytes(query.seq))

for hit in hits:
    print("E. coli K12 MG1655", hit.name, hit.identity, hit.matches, hit.fragments)

🧪 Scikit-bio

Scikit-bio lets us access to the sequence directly as a numpy array, but shows the values as byte strings by default. To make them readable as char (for compatibility with the C code), they must be cast with seq.values.view('B').

import pyfastani
import skbio.io

sketch = pyfastani.Sketch()

ref = list(skbio.io.read("vendor/FastANI/data/Shigella_flexneri_2a_01.fna", "fasta"))
sketch.add_draft("Shigella_flexneri_2a_01", (seq.values.view('B') for seq in ref))

mapper = sketch.index()

# read the query and query the mapper
query = next(skbio.io.read("vendor/FastANI/data/Escherichia_coli_str_K12_MG1655.fna", "fasta"))
hits = mapper.query_genome(query.values.view('B'))

for hit in hits:
    print("E. coli K12 MG1655", hit.name, hit.identity, hit.matches, hit.fragments)

⏱️ Benchmarks

In the original FastANI tool, multi-threading was only used to improve the performance of many-to-many searches: each thread would have a chunk of the reference genomes, and querying would be done in parallel for each reference. However, with a small set of reference genomes, there may not be enough for all the threads to work, so it cannot scale with a large number of threads. In addition, this causes the same query genome to be hashed several times, which is not optimal. In pyfastani, multi-threading is used to compute the hashes and mapping of query genome fragments. This allows parallelism to be useful even when a only few reference genomes are available.

The benchmarks below show the time for querying a single genome (with Mapper.query_draft) using a variable number of threads. Benchmarks were run on a i7-8550U CPU running @1.80GHz with 4 physical / 8 logical cores, using 50 bacterial genomes from the proGenomes database. For clarity, only 5 randomly-selected genomes are shown on the second graph. Each run was repeated 3 times.

🔖 Citation

PyFastANI is scientific software; it was presented among other optimized software at the European Student Council Symposium (ESCS) 2022 during ECCB 2022. Please cite both PyFastANI and FastANI if you are using it in an academic work, for instance as:

PyFastANI (Larralde, 2022), a Python library with optimized bindings to FastANI (Jain et al., 2018).

🔎 See Also

Computing ANI for metagenomic sequences? You may be interested in pyskani, a Python package for computing ANI using the skani method developed by Jim Shaw and Yun William Yu.

💭 Feedback

⚠️ Issue Tracker

Found a bug ? Have an enhancement request ? Head over to the GitHub issue tracker if you need to report or ask something. If you are filing in on a bug, please include as much information as you can about the issue, and try to recreate the same bug in a simple, easily reproducible situation.

🏗️ Contributing

Contributions are more than welcome! See CONTRIBUTING.md for more details.

⚖️ License

This library is provided under the MIT License.

The FastANI code was written by Chirag Jain and is distributed under the terms of the Apache License 2.0, unless otherwise specified in vendored sources. See vendor/FastANI/LICENSE for more information. The cpu_features code was written by Guillaume Chatelet and is distributed under the terms of the Apache License 2.0. See vendor/cpu_features/LICENSE for more information. The Boost::math headers were written by Boost Libraries contributors and is distributed under the terms of the Boost Software License. See vendor/boost-math/LICENSE for more information.

This project is in no way not affiliated, sponsored, or otherwise endorsed by the original FastANI authors. It was developed by Martin Larralde during his PhD project at the European Molecular Biology Laboratory in the Zeller team.