diamondlightsource / fast_dp Goto Github PK
View Code? Open in Web Editor NEWFast DP: Fast Data Processsing with XDS
Home Page: https://pypi.python.org/pypi/fast_dp
License: Apache License 2.0
Fast DP: Fast Data Processsing with XDS
Home Page: https://pypi.python.org/pypi/fast_dp
License: Apache License 2.0
Currently only fast_dp.log and fast_dp.mtz are included as AutoProcProgramAttachment.
Suggestion to add:
fast_dp_unmerged.mtz
aimless.log
autoindex.log
pointless.log
xtriage.log
Any others?
If given e.g. --spacegroup=P212121
as input, then fast_dp should output the reflection file in P212121, rather than P222 as it currently does.
For Eiger data sets this is a reasonable risk; should explicitly catch and handle
!!! ERROR !!! generic library is specified but file does not exist
more usefully though, if someone has set -l on the command line and fast_dp cannot find the plugin; throw the error then.
i.e. the first P1 scaling step; adding data > 360 degrees adds no new information...
https://github.com/dectris/neggia
need to:
see html logs for xia2; fast_ep for inspiration... would be very useful to replace current very minimal text version...
Please could fast_dp generate a scaled_unmerged.mtz? Just needs to add the following input to aimless:
output unmerged
At moment it says:
Starting image: ../thaum-11_1_0001.cbf
Fast DP error: 'detector_class'
This code really should not be in the repo any more... somewhere in dlstbx maybe, but not here.
Does not matter so much for H5ToXDS but matters a lot when using neggia
If you look at line 306-309 of fast_dp.py
:
self._refined_beam = (
beam_pixels[1] * float(self._xds_inp["QY"]),
beam_pixels[0] * float(self._xds_inp["QX"]),
)
It appears to me that the refined beam is being ordered with Y first and X second. I think this is propagating into the fast_dp.xml
file where refinedXBeam
and refinedYBeam
are swapped with respect to the starting values. SynchWeb (master branch) has been written to accept these swapped beam centres, but the changes that were made to Synchweb in order to display things properly for fast_dp
means that, at least for me, the other pipeline results (autoPROC, xia2-3d, xia2-dials) indicate a badly misaligned beam_xy in my image headers... which I know they are not.
The file https://github.com/DiamondLightSource/fast_dp/blob/master/bin/launch_fast_dp should be removed from this repository as it is entirely DLS-specific, and we may want to change the launch script independent of the installed fast_dp version. I have copied a modified version to mx-scripts/auto-xia2/launch_fast_dp.
Since this is a dependency anyway it would seem sane to make it the only way to read image headers. That way we would get for no cost proper support for full imgCIF files etc. See also SCI-7774 in DLS JIRA.
I'm trying to get fast_dp working on a new CentOS7 machine.
I install cctbx from source
installed fast_dp as per instructions
Fast_DP version 1.3
Starting image: /20190501/burn0/lyso5_1_E1_00001.cbf
Running on: als-enable
Number of jobs: 1
Number of cores: 0
Processing images: 1 -> 100
Rotation range: 107.00 -> 207.00
Template: lyso5_1_E1_#####.cbf
Wavelength: 1.25511
Working in: /home/username/process
All autoindexing results:
Lattice a b c alpha beta gamma
Autoindexing failed
Fast DP error: 'CLIBD'
From fast_dp.error file:
$ cat fast_dp.error
Traceback (most recent call last):
File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/fast_dp.py", line 532, in main
finst.process()
File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/fast_dp.py", line 263, in process
self._xds_inp, input_cell=self._input_cell_p1
File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/autoindex.py", line 133, in autoindex
spacegroup_to_lattice(r),
File "/home/username/cctbx/base/lib/python2.7/site-packages/fast_dp/cell_spacegroup.py", line 78, in spacegroup_to_lattice
os.path.join(os.environ["CLIBD"], "symop.lib"), "r"
File "/home/username/cctbx/build/../base/lib/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'CLIBD'
Some tests would be useful
Hi,
looking how Aimless is run, I guess that if the user wants to get scaled unmerged data, the right file is xds_sorted.mtz
.
Is that correct? Is this file available for download? (I can't remember how the web interface works).
Leaving it to it's own devices seems to change the absolute scale of the data, which can be an issue if it decides that the correct scale factor is 0.02 - means that all your intensities are < 1 which apparently SHELXC treats as meaningful...
$ fast_dp image_0001.cbf --spacegroup="C121"
Fast_DP version 1.3
Spacegroup C121 not recognised: ignoring
...
This is because it checks the name against a hardcoded list in $CLIBD/symop.lib:
fast_dp/fast_dp/cell_spacegroup.py
Lines 90 to 109 in 3216065
It would be better instead if it used sgtbx to interpret space group symbols is more permissive in what it supports as input.
Have
SPOT_RANGE= 1 33
SPOT_RANGE= 238 270
SPOT_RANGE= 536 540
33 images, 33 images, 5 images => suspect last should be 5 degrees not 5 images
eg /dls/i03/data/2018/nt18231-9/processed/tmp/2018-11-15/07-35-33-8f6f713d/INS2_29_2_1_/fast_dp/fast_dp-report.html
is 12.6 MB.
.json
is another 12 MB
I suspect there are a few digits that can be shaved off
Latest revision 20170601 has different approach to cluster usage and appears to ignore the forkintegrate command - this can make fast_dp no longer fast. In INTEGRATE.LP see
INPUT PARAMETER VALUES
----------------------
MAXIMUM_NUMBER_OF_PROCESSORS= 20 (0: automatic choice of OpenMP threads)
number of OpenMP threads used 20
MAXIMUM_NUMBER_OF_JOBS= 20 (0:automatic choice of forked main tasks)
number of forked INTEGRATE tasks 1
Working with XDS authors to resolve what is happening here...
PyPI does not like our Github flavoured markdown:
https://pypi.org/project/fast-dp/
Beam position x,y settings get flipped between index and integrate/correct stages.
i.e. in preference to running with e.g. UGE as a queue management system, instead run the parallel bits on node-001 node-002 or similar named on command-line
Will require passing info through to distributed forkxds script
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.