diamondlightsource / fast_dp Goto Github PK

eg /dls/i03/data/2018/nt18231-9/processed/tmp/2018-11-15/07-35-33-8f6f713d/INS2_29_2_1_/fast_dp/fast_dp-report.html is 12.6 MB.
.json is another 12 MB

I suspect there are a few digits that can be shaved off

The file https://github.com/DiamondLightSource/fast_dp/blob/master/bin/launch_fast_dp should be removed from this repository as it is entirely DLS-specific, and we may want to change the launch script independent of the installed fast_dp version. I have copied a modified version to mx-scripts/auto-xia2/launch_fast_dp.

see html logs for xia2; fast_ep for inspiration... would be very useful to replace current very minimal text version...

See https://docs.python.org/2/library/resource.html

Hi,
looking how Aimless is run, I guess that if the user wants to get scaled unmerged data, the right file is xds_sorted.mtz.
Is that correct? Is this file available for download? (I can't remember how the web interface works).

Leaving it to it's own devices seems to change the absolute scale of the data, which can be an issue if it decides that the correct scale factor is 0.02 - means that all your intensities are < 1 which apparently SHELXC treats as meaningful...

If given e.g. --spacegroup=P212121 as input, then fast_dp should output the reflection file in P212121, rather than P222 as it currently does.

Since this is a dependency anyway it would seem sane to make it the only way to read image headers. That way we would get for no cost proper support for full imgCIF files etc. See also SCI-7774 in DLS JIRA.

I'm trying to get fast_dp working on a new CentOS7 machine.

I install cctbx from source
installed fast_dp as per instructions

Fast_DP version 1.3
Starting image: /20190501/burn0/lyso5_1_E1_00001.cbf
Running on: als-enable
Number of jobs: 1
Number of cores: 0
Processing images: 1 -> 100
Rotation range: 107.00 -> 207.00
Template: lyso5_1_E1_#####.cbf
Wavelength: 1.25511
Working in: /home/username/process
All autoindexing results:
Lattice      a      b      c  alpha   beta  gamma
Autoindexing failed
Fast DP error: 'CLIBD'

From fast_dp.error file:

$ cat fast_dp.error 
Traceback (most recent call last):
  File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/fast_dp.py", line 532, in main
    finst.process()
  File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/fast_dp.py", line 263, in process
    self._xds_inp, input_cell=self._input_cell_p1
  File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/autoindex.py", line 133, in autoindex
    spacegroup_to_lattice(r),
  File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/cell_spacegroup.py", line 78, in spacegroup_to_lattice
    os.path.join(os.environ["CLIBD"], "symop.lib"), "r"
  File "/home/username/cctbx/build/../base/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'CLIBD'

Have

 SPOT_RANGE=       1      33
 SPOT_RANGE=     238     270
 SPOT_RANGE=     536     540

33 images, 33 images, 5 images => suspect last should be 5 degrees not 5 images

Beam position x,y settings get flipped between index and integrate/correct stages.

i.e. the first P1 scaling step; adding data > 360 degrees adds no new information...

For Eiger data sets this is a reasonable risk; should explicitly catch and handle

!!! ERROR !!! generic library is specified but file does not exist

more usefully though, if someone has set -l on the command line and fast_dp cannot find the plugin; throw the error then.

Please could fast_dp generate a scaled_unmerged.mtz? Just needs to add the following input to aimless:
output unmerged

At moment it says:

Starting image: ../thaum-11_1_0001.cbf
Fast DP error: 'detector_class'

If you look at line 306-309 of fast_dp.py:

            self._refined_beam = (
                beam_pixels[1] * float(self._xds_inp["QY"]),
                beam_pixels[0] * float(self._xds_inp["QX"]),
            )

It appears to me that the refined beam is being ordered with Y first and X second. I think this is propagating into the fast_dp.xml file where refinedXBeam and refinedYBeam are swapped with respect to the starting values. SynchWeb (master branch) has been written to accept these swapped beam centres, but the changes that were made to Synchweb in order to display things properly for fast_dp means that, at least for me, the other pipeline results (autoPROC, xia2-3d, xia2-dials) indicate a badly misaligned beam_xy in my image headers... which I know they are not.

Latest revision 20170601 has different approach to cluster usage and appears to ignore the forkintegrate command - this can make fast_dp no longer fast. In INTEGRATE.LP see

 INPUT PARAMETER VALUES
 ----------------------
 MAXIMUM_NUMBER_OF_PROCESSORS=  20  (0: automatic choice of OpenMP threads)
 number of OpenMP threads used  20
 MAXIMUM_NUMBER_OF_JOBS=  20  (0:automatic choice of forked main tasks)
 number of forked INTEGRATE tasks   1

Working with XDS authors to resolve what is happening here...

Currently only fast_dp.log and fast_dp.mtz are included as AutoProcProgramAttachment.

Suggestion to add:
fast_dp_unmerged.mtz
aimless.log
autoindex.log
pointless.log
xtriage.log

Any others?

https://jira.diamond.ac.uk/browse/MXSW-953

https://github.com/dectris/neggia

need to:

1. read master file => header information
2. generate XDS.INP's for INDEX INTEGRATE CORRECT phases
3. use plugin in place of HDF5ToXDS type process

This code really should not be in the repo any more... somewhere in dlstbx maybe, but not here.

i.e. in preference to running with e.g. UGE as a queue management system, instead run the parallel bits on node-001 node-002 or similar named on command-line

Will require passing info through to distributed forkxds script

fast_dp has become incompatible with recent dials releases due to deprecation and removal of some features. Please fix.

Does not matter so much for H5ToXDS but matters a lot when using neggia

Some tests would be useful

PyPI does not like our Github flavoured markdown:
https://pypi.org/project/fast-dp/

$ fast_dp image_0001.cbf --spacegroup="C121"
Fast_DP version 1.3
Spacegroup C121 not recognised: ignoring
...

This is because it checks the name against a hardcoded list in $CLIBD/symop.lib:

It would be better instead if it used sgtbx to interpret space group symbols is more permissive in what it supports as input.

Running tests without XDS present give:

        for step in ["XYCORR", "INIT", "COLSPOT", "IDXREF"]:
>           lastrecord = open("%s.LP" % step).readlines()[-1]
E           FileNotFoundError: [Errno 2] No such file or directory: 'XYCORR.LP'

fast_dp/autoindex.py:106: FileNotFoundError

This is because run_job doesn't check and provides no way to tell if the process actually ran, so autoindex just carries on as if it's worked: