Issues when trying to use pyani about pyani

[sbridel]

Well, i clone your repo and then i enter

python3.4 ./pyani/bin/genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email myownmail@hello
... and It works..

I don't know what the issue but your script is amazing. Congrats

[sbridel]

python3.4 ./pyani/bin/average_nucleotide_identity.py pyani/tests/test_ani_data/ -o tests/test_ANIb_output/ -m ANIb -g
Traceback (most recent call last):
  File "./pyani/bin/average_nucleotide_identity.py", line 169, in <module>
    from pyani import (anib, anim, tetra, pyani_config, pyani_files,
ImportError: cannot import name 'pyani_tools'

won't work

average_nucleotide_identity.py -i pyani/tests/test_ani_data/ -o pyani/tests/test_ANIb_output/ -m ANIb -g
Traceback (most recent call last):
  File "/usr/local/bin/average_nucleotide_identity.py", line 806, in <module>
    results = methods[args.method][0](infiles, org_lengths)
  File "/usr/local/bin/average_nucleotide_identity.py", line 532, in unified_anib
    logger=logger)
  File "/usr/local/lib/python3.4/dist-packages/pyani/run_multiprocessing.py", line 45, in run_dependency_graph
    cumretval += multiprocessing_run(cmdset, workers, verbose)
  File "/usr/local/lib/python3.4/dist-packages/pyani/run_multiprocessing.py", line 86, in multiprocessing_run
    for cline in cmdlines]
  File "/usr/local/lib/python3.4/dist-packages/pyani/run_multiprocessing.py", line 86, in <listcomp>
    for cline in cmdlines]
AttributeError: 'module' object has no attribute 'run'

won't work too

[widdowquinn]

Hi Seb,

When you run the scripts, are you running them in the repository root?

If you have installed pyani with pip install pyani or python setup.py install, then the installer will place the executable scripts genbank_get_genomes_by_taxon.py and average_nucleotide_identity.py in your usual applications/binaries directory. The intention is that you can then run them from any location, and they will use the installed pyani modules.

If you try to run them in the repository root, Python gets confused about the import location (as pyani is a subdirectory of the repository as well as an installed module name).

This error may go away if you change directory to another location (e.g. ~/Desktop) and run your analysis as

$ genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 \
    -l Caulobacter_downloads.log --email [email protected]
$ average_nucleotide_identity.py -i $PATH_TO_REPO/pyani/tests/test_ani_data/ \
    -o $PATH_TO_REPO/pyani/tests/test_ANIb_output/ -m ANIb -g

L.

[widdowquinn]

And thank you for the compliment on pyani!

[sbridel]

Well i first install it with pip3

When i use the commande genbank_get_genomes_by_taxon it fails (first message)
When i use the command average_nucleotide_identity, it fails again (last message)

I tried to clone all your repository into a folder, then i tried

cd pyani
python3.4 ./pyani/bin/genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email myownmail@hello

and works

but

python3.4 ./pyani/bin/average_nucleotide_identity.py pyani/tests/test_ani_data/ -o tests/test_ANIb_output/ -m ANIb -g

failed (this pyani folder is from git clone)

I'm very happy to having successfully fetching the genome i wanted from NCBI.
Now i really want to perform a ANi analysis with your tool but I don't know what's going on

---

EDIT
I think it's only installation issues.
You may rely consider of virtualizing your ANI tools with Docker or Galaxy.
As far as i know it's the only ANI software available which can be used in command line..
very usefull when you need to make an ANI with 40 genomes (all vs all)
I want to perform this ANI so much that i tried Jspecies... I have to click over 1600 times on the mouse for selecting all

[widdowquinn]

Hi Seb,

There's clearly some optimisation I can do on the installation instructions ;) Here's how I'd do it myself, to get the current repo working (inside a virtualenv, but it ought also to work with the Python installed on your system) and do the Caulobacter analysis:

$ cd ~/Desktop/
$ virtualenv venv-pyani -p python3.6
$ source venv-pyani/bin/activate
$ git clone https://github.com/widdowquinn/pyani.git
$ cd pyani
$ pip install -r requirements.txt
$ python setup.py install
$ which genbank_get_genomes_by_taxon.py
/Users/lpritc/Desktop/venv-pyani/bin/genbank_get_genomes_by_taxon.py
$ cd ..
$ genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email myownmail@hello
$ average_nucleotide_identity.py -i Caulobacter_downloads -o Caulobacter_ANI --labels Caulobacter_downloads/labels.txt --classes Caulobacter_downloads/classes.txt -g --gmethod seaborn --gformat pdf,png -v -l Caulobacter_ANI.log

The output logs and images are attached for reference:

Caulobacter_downloads.txt
Caulobacter_ANI.txt

The issue you're having indicates I need to do a release that updates the PyPI version, and I'll do this shortly.

A Docker installation is potentially somewhere in the future. Packaging for Galaxy is more complex and may not happen for some time.

The impossibility of using JSpecies for large datasets is exactly why I wrote pyani. I've used it to classify over 800 isolates simultaneously. I've no idea how long JSpecies would take for that, even if I had the patience to click all the boxes ;)

If you install pyani as above and run the scripts from a directory other than the repo root, does this solve the problem for you?

L.

[sbridel]

Thanks.
I have some issue setting virtualenv since python3.6 was installed by anaconda and now i dont know how to make it works

Running virtualenv with interpreter /home/sebastien/anaconda3/bin/python3.6
Using base prefix '/home/sebastien/anaconda3'
New python executable in venv-pyani/bin/python3.6
Not overwriting existing python script venv-pyani/bin/python (you must use venv-pyani/bin/python3.6)
venv-pyani/bin/python3.6: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
ERROR: The executable venv-pyani/bin/python3.6 is not functioning
ERROR: It thinks sys.prefix is '/home/sebastien/Bureau' (should be '/home/sebastien/Bureau/venv-pyani')
ERROR: virtualenv is not compatible with this system or executable

[widdowquinn]

The virtualenv in my example was left in just so that you can see what I did. If system or Anaconda python works for you, then please use it, by all means.

I expect the cleanest way to make your venv-pyani specifically Python 3.6 is to delete it, and recreate it. That's what I usually do (package installation is typically quick, for me).

[sbridel]

Ok !!! Sorry i dont undertsand what you said.

I install python 3.6 and then

 git clone https://github.com/widdowquinn/pyani.git
$ cd pyani
$ pip install -r requirements.txt
$ python setup.py install

which genbank_get_genomes_by_taxon.py
$HOME/anaconda3/bin/genbank_get_genomes_by_taxon.py

This script has been installed in anaconda3 bin so I used

$HOME/anaconda3/bin/genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email email@to_me

WORKS

and then

$HOME/anaconda3/bin/average_nucleotide_identity.py -i Caulobacter_downloads -o Caulobacter_ANI --labels Caulobacter_downloads/labels.txt --classes Caulobacter_downloads/classes.txt -g --gmethod seaborn --gformat pdf,png -v -l Caulobacter_ANI.lo

works but failed at the end (plotting)

INFO: Formats requested: pdf,png
INFO: Graphics format: pdf
INFO: Graphics method: seaborn
/home/sebastien/anaconda3/lib/python3.6/site-packages/matplotlib/cbook.py:136: MatplotlibDeprecationWarning: The axisbg attribute was deprecated in version 2.0. Use facecolor instead.
  warnings.warn(message, mplDeprecation, stacklevel=1)
Traceback (most recent call last):
  File "/home/sebastien/anaconda3/bin/average_nucleotide_identity.py", line 4, in <module>
    __import__('pkg_resources').run_script('pyani==0.2.1.dev0', 'average_nucleotide_identity.py')
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 744, in run_script
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1506, in run_script
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/pyani-0.2.1.dev0-py3.6.egg/EGG-INFO/scripts/average_nucleotide_identity.py", line 791, in <module>
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/pyani-0.2.1.dev0-py3.6.egg/EGG-INFO/scripts/average_nucleotide_identity.py", line 631, in draw
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/pyani-0.2.1.dev0-py3.6.egg/pyani/pyani_graphics.py", line 163, in heatmap_seaborn
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/pyani-0.2.1.dev0-py3.6.egg/pyani/pyani_graphics.py", line 117, in get_seaborn_clustermap
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 1252, in clustermap
    **kwargs)
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 1087, in plot
    row_linkage=row_linkage, col_linkage=col_linkage)
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 991, in plot_dendrograms
    ax=self.ax_row_dendrogram, rotate=True, linkage=row_linkage)
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 717, in dendrogram
    label=label, rotate=rotate)
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 535, in __init__
    self.dendrogram = self.calculate_dendrogram()
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/seaborn/matrix.py", line 615, in calculate_dendrogram
    color_threshold=-np.inf)
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/scipy/cluster/hierarchy.py", line 2227, in dendrogram
    is_valid_linkage(Z, throw=True, name='Z')
  File "/home/sebastien/anaconda3/lib/python3.6/site-packages/scipy/cluster/hierarchy.py", line 1421, in is_valid_linkage
    % name_str)
ValueError: Linkage 'Z' uses the same cluster more than once.

[widdowquinn]

Great - it sounds like you sorted out the executable/library issue, but you now have hit a problem in scipy.

Can you tell me which version of scipy you are using? The version I have locally is scipy-0.19.0 and you can find which version you have available by running the command pip freeze, as below:

$ pip freeze
appdirs==1.4.3
biopython==1.68
cycler==0.10.0
matplotlib==2.0.0
numpy==1.12.1
packaging==16.8
pandas==0.19.2
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2016.10
scipy==0.19.0
seaborn==0.7.1
six==1.10.0

[widdowquinn]

It is possible that you are using scipy v0.18.1 which is known to have this bug: scipy/scipy#6785.

If that is the case, the solution would be to install a later version of scipy.

[sbridel]

Thanks god i'm about to use your ANI.
scipy==0.18.1
Indeed, i need to install 0.19 !

---

EDIT
I install using conda (since anaconda3 seems to have successfully install your PYANI) :

conda install -c anaconda scipy=0.19.0 

👍

I tested your ANI without plots, it works finally 👍
I'm testing ANI + plotting right now 🥇 👍

[sbridel]

Well it finally works !!!! Awesome graphics !!!
i need to understand all graphics and changing palettes colors will be nice (i love changing colors) but well your works is amazing !!! I hope to find some interesting results now

[widdowquinn]

Thank you for persisting! I'm so glad it works for you, now!

Good luck with your research, and please remember you can cite this paper when your own manuscript comes out ;) :

Pritchard et al. (2016)

@Article{C5AY02550H,
author ="Pritchard, Leighton and Glover, Rachel H. and Humphris, Sonia and Elphinstone, John G. and Toth, Ian K.",
title  ="Genomics and taxonomy in diagnostics for food security: soft-rotting enterobacterial plant pathogens",
journal  ="Anal. Methods",
year  ="2016",
volume  ="8",
issue  ="1",
pages  ="12-24",
publisher  ="The Royal Society of Chemistry",
doi  ="10.1039/C5AY02550H",
url  ="http://dx.doi.org/10.1039/C5AY02550H",
abstract  ="Soft rot Enterobacteriaceae (SRE) are bacterial plant pathogens that cause blackleg{,} wilt and soft rot diseases on a broad range of important crop and ornamental plants worldwide. These organisms (spanning the genera Erwinia{,} Pectobacterium{,} Dickeya{,} and Pantoea) cause significant economic and yield losses in the field{,} and in storage. They are transmissible through surface water{,} by trade and other movement of plant material and soil{,} and in some cases are subject to international legislative and quarantine restrictions. Effective detection and diagnosis in support of food security legislation and epidemiology is dependent on the ability to classify pathogenic isolates precisely. Diagnostics and classification are made more difficult by the influence of horizontal gene transfer on phenotype{,} and historically complex and sometimes inaccurate nomenclatural and taxonomic assignments that persist in strain collections and online sequence databases. Here{,} we briefly discuss the relationship between taxonomy{,} genotype and phenotype in the SRE{,} and their implications for diagnostic testing and legislation. We present novel whole-genome classifications of the SRE{,} illustrating inconsistencies between the established taxonomies and evidence from completely sequenced isolates. We conclude with a perspective on the future impact of widespread whole-genome sequencing and classification methods on detection and identification of bacterial plant pathogens in support of legislative and policy efforts in food security."}

[sbridel]

I will not forgive to cite your paper. You have done a very powerful tools and perform high bioinformatics programming. I'm doing a PhD thesis in bioinformatics so i hope my own manuscript will come out soon!