instamatic-dev / edtools

Collection of tools for automated processing and clustering of electron diffraction data

edtools

Collection of tools for automated processing and clustering of batch 3-dimensional electron diffraction (3D ED) datasets.

The source for this project is available here.

Installation

Install using pip install edtools. Installation should take less than 20 seconds on a normal desktop.

Find the latest releases for the versions that have been tested on.

OS Requirement

Windows 10 or newer.

Software Requirements

• Python 3.6+ including numpy, scipy, matplotlib, and pandas libraries
• sginfo or cctbx.python must be available on the system path for edtools.make_shelx
• Access to WSL
• XDS package must be installed properly under WSL

Package dependencies

Check pyproject.toml for the full dependency list and versions.

Documentation

See the documentation at https://edtools.readthedocs.io.

Pipeline tools

At any step, run edtools.xxx -h for help with possible arguments.

autoindex.py

Looks for files matching XDS.INP in all subdirectories and runs them using XDS.

In:  XDS.INP
Out: XDS data processing on all files

Usage:

edtools.autoindex

extract_xds_info.py

Looks files matching CORRECT.LP in all subdirectories and extracts unit cell/integration info. Summarizes the unit cells in the excel file cells.xlsx and cells.yaml. XDS_ASCII.HKL files matching the completeness / CC(1/2) criteria are listed in filelist.txt. Optionally, gathers the corresponding XDS_ASCII.HKL files in the local directory. The cells.yaml file can be used as input for further processing.

In:  CORRECT.LP
Out: cells.yaml
     cells.xlsx
     filelist.txt

Usage:

edtools.extract_xds_info

find_cell.py

This program a cells.yaml file and shows histogram plots with the unit cell parameters. This program mimicks CELLPARM and calculates the weighted mean lattice parameters, where the weight is typically the number of observed reflections (defaults to 1.0). For each lattice parameter, the mean is calculated in a given range (default range = median+-2). The range can be changed by dragging the cursor on the histogram plots.

Alternatively, the unit cells can be clustered by giving the --cluster command, in which a dendrogram is shown. The cluster cutoff can be selected by clicking in the dendrogram. The clusters will be written to cells_cluster_#.yaml.

In:  cells.yaml
Out: mean cell parameters
     cells_*.yaml (clustering only)

Usage:

edtools.find_cell cells.yaml --cluster

make_xscale.py

Prepares an input file XSCALE.INP for XSCALE and corresponding XDSCONV.INP for XDSCONV. Takes a cells.yaml file or a series of XDS_ASCII.HKL files as input, and uses those to generate the XSCALE.INP file.

In:  cells.yaml / XDS_ASCII.HKL
Out: XSCALE.INP

Usage:

edtools.make_xscale cells.yaml -c 10.0 20.0 30.0 90.0 90.0 90.0 -s Cmmm

cluster.py

Parses the XSCALE.LP file for the correlation coefficients between reflection files to perform hierarchical cluster analysis (Giordano et al., Acta Cryst. (2012). D68, 649–658). The cutoff threshold can be selected by clicking in the dendrogram window. The program will write new XSCALE.LP files to subdirectories cluster_#, and run XSCALE on them, and (if available), pointless.

In:  XSCALE.LP
Out: cluster_n/
	filelist.txt
	*_XDS_ASCII.HKL
	XSCALE processing
	Pointless processing
	shelx.hkl
	shelx.ins (optional)

Usage:

edtools.cluster

Helper tools

make_shelx.py

Creates a shelx input file. Requires sginfo to be available on the system path to generate the SYMM/LATT cards.

In:  cell, space group, composition
Out: shelx.ins

Usage:

edtools.make_shelx -c 10.0 20.0 30.0 90.0 90.0 90.0 -s Cmmm -m Si180 O360

run_pointless.py

Looks for XDS_ASCII.HKL files specified in the cells.yaml, or on the command line and runs Pointless on them.

In:  cells.yaml / XDS_ASCII.HKL
Out: Pointless processing

update_xds.py

Looks files matching CORRECT.LP in all subdirectories, and updates the cell parameters / space group as specified.

In:  XDS.INP
Out: XDS.INP

Usage:

edtools.update_xds -c 10.0 20.0 30.0 90.0 90.0 90.0 -s Cmmm

find_rotation_axis.py

Finds the rotation axis and prints out the inputs for several programs (XDS, PETS, DIALS, Instamatic, and RED). Implements the algorithm from Gorelik et al. (Introduction to ADT/ADT3D. In Uniting Electron Crystallography and Powder Diffraction (2012), 337-347). The program reads XDS.INP to get information about the wavelength, pixelsize, oscillation angle, and beam center, and SPOT.XDS (generated by COLSPOT) for the peak positions. If the XDS.INP file is not specified, the program will try to look for it in the current directory.

In:  XDS.INP, SPOT.XDS
Out: Rotation axis

Usage:

edtools.find_rotation_axis [XDS.INP]

Demo of using edtools to process batch 3D electron diffraction datasets

See the demo at https://edtools.readthedocs.io/en/latest/examples/edtools_demo.html.

edtools's Issues

python setup.py install fails

Work-around: python setup.py develop.
Solution: edtools should be added to setup.py as a package.

Installation on Ubuntu 18.04 missing .yaml

Hi!
I installed edtools on Ubuntu 18.04 by sudo python3 setup.py install, but it appears not to put the .yaml files in the right place:

$ edtools.make_shelx
Traceback (most recent call last):
  File "/usr/local/bin/edtools.make_shelx", line 11, in <module>
    load_entry_point('edtools==0.1.0', 'console_scripts', 'edtools.make_shelx')()
  File "/home/fcx32934/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/fcx32934/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2853, in load_entry_point
    return ep.load()
  File "/home/fcx32934/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2444, in load
    return self.resolve()
  File "/home/fcx32934/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2450, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/dist-packages/edtools-0.1.0-py3.6.egg/edtools/make_shelx.py", line 7, in <module>
    TABLE = yaml.load(open(fin, "r"), Loader=yaml.Loader)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.6/dist-packages/edtools-0.1.0-py3.6.egg/edtools/atomlib.yaml'

Am I doing something wrong?

openpyxl missing as dependency

Hiya,

Unfortunately, I'm still a novice with git and entirely unfamiliar with poetry, so I don't dare to add a pull-request (yet?)...

Issue

When installing edtools via pip, the package openpyxl is not being installed. This leads to an error when executing edtools.extract_xds_info as the writing of the .xlsx file fails (see below error message).
I don't know if pandas is supposed to require openpyxl itself, so maybe it's actually only an issue in that current version, v1.4.2? I added a full list of environment packages as .txt below.

System: Win11, miniconda3 (fresh conda environment), tested in py3.8, py3.9 and py3.10

Workaround

pip install openpyxl

Suggested fix

add openpyxl to requirements/dependencies

Best,
Daniel

Traceback (most recent call last):
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\danie\Miniconda3\envs\edtools\Scripts\edtools.extract_xds_info.exe\__main__.py", line 7, in <module>
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\site-packages\edtools\extract_xds_info.py", line 437, in main
    cells_to_excel(xdsall)
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\site-packages\edtools\extract_xds_info.py", line 211, in cells_to_excel
    df.to_excel(fn)
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\site-packages\pandas\core\generic.py", line 2345, in to_excel
    formatter.write(
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\site-packages\pandas\io\formats\excel.py", line 888, in write
    writer = ExcelWriter(  # type: ignore[abstract]
  File "C:\Users\danie\Miniconda3\envs\edtools\lib\site-packages\pandas\io\excel\_openpyxl.py", line 49, in __init__
    from openpyxl.workbook import Workbook
ModuleNotFoundError: No module named 'openpyxl'

package_list.txt

make_shelx: implicit single elements ignored in composition

I was running edtools.make_shelx like this:

edtools.make_shelx -c 7.487 11.041 13.775 90 92.939 90 -s "P2(1)/n" -m C15 H12 N2 O

I just noticed that this does not produce SFAC values for the oxygen atom. It works okay if O1 is used instead though.

This did not stop solving the structure 😁, but as it is a silent omission it might be catching people out. Could we please recognise a plain O as if it were O1? Or at the very least report that not all of the composition line was processed?