Comments (19)
The following is my attempt at rationalising the current state of masking in format classes/imagesets:
Dynamic masking
I.e. a mask that varies image-to-image, with a dependence on the goniometer/scan.
Overload the get_goniometer_shadow_masker()
method in the format class, e.g.:
def get_goniometer_shadow_masker(self, goniometer=None):
if goniometer is None:
goniometer = self.get_goniometer()
return GoniometerMaskerFactory.smargon(goniometer)
(see
Lines 500 to 504 in df55d9a
and set the _dynamic_shadowing
attribute of the format class to True
.
This method is called by Format.get_masker()
if the format class has the _dynamic_shadowing
attribute set to True
and goniometer
is an instance of MultiAxisGoniometer
:
Lines 269 to 281 in df55d9a
Format.get_masker()
is called from Format.get_imageset()
, where it is passed as an argument to ImageSetData
during the construction of the ImageSweep
object:
Lines 427 to 447 in df55d9a
ImageSweep.get_mask()
calls ImageSweep.get_dynamic_mask()
, which then calls the `get_mask() method of the dynamic masker object:
Lines 1244 to 1267 in 2086723
ImageSweep.get_dynamic_mask()
also applies ImageSet.get_trusted_range_mask()
:
Line 1266 in 2086723
Lines 856 to 872 in 2086723
Goniometer shadow masker
Goniometer shadow maskers should be an instance of dxtbx.masking.GoniometerShadowMasker
(or subclass):
https://github.com/cctbx/dxtbx/blob/master/masking/goniometer_shadow_masking.h#L33-L225
There is a GoniometerMaskerFactory
class that provides methods for constructing varying different types of GoniometerShadowMasker
that can be re-used in different format classes:
Lines 31 to 213 in 2086723
Static mask
I.e. a mask that is common to all images in the imageset.
The method ImageSet.get_static_mask()
combines the mask generated by Imageset.get_untrusted_rectangle_mask()
with an optional external mask that is set to the external_lookup_
attribute of the ImageSet
:
Lines 838 to 854 in 2086723
Lines 816 to 836 in 2086723
Lines 802 to 814 in 2086723
imageset.external_lookup.mask.data
is set in e.g. dials.import
or dials.generate_mask
:
The functionality that is currently missing is that there is no way for a format class to override/update the static mask. What is proposed in #65 (comment) and #70 is to add a get_static_mask()
method to Format
which returns either a mask or None and then set that in the ImageSet
when it is instantiated. This can be done as follows:
imageset.external_lookup.mask.data = ImageBool(mask_flex_array_or_tuple_of_flex_arrays)
from dxtbx.
override/update the static mask
Where does "merging of a user-provided mask" happen?
from dxtbx.
@biochem-fan this i
Where does "merging of a user-provided mask" happen?
This happens in the call to ImageSet.get_static_mask()
which combines it with the untrusted rectangle mask:
Lines 838 to 846 in 2086723
The external_lookup.mask.data
attribute can be set in e.g. dials.import
or dials.generate_mask
. What is being suggested is that we create a mechanism that allows the format class to set/modify imageset.external_lookup.mask.data
when constructing the imageset.
from dxtbx.
the format class to set/modify imageset.external_lookup.mask.data
Does it mean Format.get_static_mask()
should take an existing user-provided mask, merge it with the format specific mask and return a new mask? That is:
imageset.external_lookup.mask.data = format.get_static_mask(imageset.external_lookup.mask.data)
Then ImageSet.get_static_mask(mask)
further merges it with the untrusted rectangle mask?
Shouldn't the first merging step with the user-provided mask (external_lookup.mask
) happen in ImageSet
?
from dxtbx.
the format class to set/modify imageset.external_lookup.mask.data
Does it mean
Format.get_static_mask()
should take an existing user-provided mask, merge it with the format specific mask and return a new mask? That is:imageset.external_lookup.mask.data = format.get_static_mask(imageset.external_lookup.mask.data)
Then
ImageSet.get_static_mask(mask)
further merges it with the untrusted rectangle mask?
Yes, that is what I understand of @jmp1985's suggestion.
Shouldn't the first merging step with the user-provided mask (
external_lookup.mask
) happen inImageSet
?
This issue with this approach is that it would require further modification to Imageset
and likely other places in dxtbx/imageset.h
to pass and store a format class-specified static mask, whereas the above suggestion would only need changes to Format.py.
from dxtbx.
@biochem-fan do you have a small example file (a single image would suffice) that we can use to exercise the format class-defined mask in dxtbx/format/FormatHDF5SaclaMPCCD.py
? If this was small enough we might be able to add it to https://github.com/dials/data-files, or possibly better on zenodo as long as the files aren't compressed (see e.g. https://github.com/dials/data/blob/8b7f34c9467997fc49dedf2c563b946a2b770d33/dials_data/definitions/vmxi_thaumatin.yml).
from dxtbx.
from dxtbx.
@rjgildea Isn't it simply modifying ImageSet.get_static_mask(mask)
? My worry is that by delegating the merging step to implementation in each Format class, a buggy class might ignore the user-specified mask. It also causes code duplication among many classes.
@phyy-nx That image was created by an early version of the pipeline and does not contain the mask. Please try https://drive.google.com/open?id=0ByqmCYlnqv6UU1RXb053MG1ucmc (11 MB).
from dxtbx.
@rjgildea Isn't it simply modifying
ImageSet.get_static_mask(mask)
? My worry is that by delegating the merging step to implementation in each Format class, a buggy class might ignore the user-specified mask. It also causes code duplication among many classes.
No, the merging step would be in the Format baseclass, probably somewhere around this point in Format.get_imageset()
which would call Format.get_static_mask()
:
https://github.com/dials/dxtbx/blob/master/format/Format.py#L427-L447
In the case of FormatHDF5SaclaMPCCD
it would look pretty much the same as the existing (currently unused) FormatHDF5SaclaMPCCDget_mask()
function:
@phyy-nx That image was created by an early version of the pipeline and does not contain the mask. Please try https://drive.google.com/open?id=0ByqmCYlnqv6UU1RXb053MG1ucmc (11 MB).
Thanks!
from dxtbx.
@rjgildea OK, this makes sense.
from dxtbx.
Unfortunately it's not quite as simple as I'd hoped. Even if the imageset is returned here with the imageset.external_lookup.mask.data
set by the format class:
dxtbx/model/experiment_list.py
Lines 227 to 230 in 2086723
the mask is then overwritten again here:
dxtbx/model/experiment_list.py
Lines 249 to 280 in 2086723
from dxtbx.
I've made a start towards this here 0e36727. This works if you run dials.image_viewer
directly on the image file:
$ dials.image_viewer MPCCD-Phase3-21528-5images.h5 show_mask=True
However it doesn't work when loading from an experiments json file, due to the imageset.external_lookup.mask
being overwritten as mentioned above:
$ dials.import MPCCD-Phase3-21528-5images.h5
$ dials.image_viewer imported.expt show_mask=True
from dxtbx.
@biochem-fan @phyy-nx do you also have example images for testing the masking in FormatHDF5SaclaRayonix.py and FormatPYunspecified.py?
from dxtbx.
@rjgildea Currently the Rayonix detector at SACLA does not need a detector specific mask. The get_mask()
function is not used now (self.mask
is always None
). But they might become relevant in future (i.e. when the detector gets damaged over time).
from dxtbx.
from dxtbx.
@rjgildea Currently the Rayonix detector at SACLA does not need a detector specific mask. The
get_mask()
function is not used now (self.mask
is alwaysNone
). But they might become relevant in future (i.e. when the detector gets damaged over time).
It would probably still be a good idea to have an example dataset available so we can ensure test coverage of this format class.
from dxtbx.
@keitaroyam or @phyy-nx, do you have a test image from Rayonix we can make public?
from dxtbx.
@phyy-nx are the FormatPYunspecifiedInMemory/FormatPYunspecifiedStillInMemory classes still needed/expected to work? As far as I can tell they don't work as it stands:
5cdffe4#diff-e59100cf97d8b4af3427ae450ca1476eR34-R46
$ pytest --regression tests/format/test_FormatPYunspecified.py::test_FormatPYunspecifiedStillInMemory --runxfail
...
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_FormatPYunspecifiedStillInMemory ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
dials_regression = '/Users/rjgildea/software/cctbx/modules/dials_regression'
@pytest.mark.xfail
def test_FormatPYunspecifiedStillInMemory(dials_regression):
filename = os.path.join(
dials_regression,
"image_examples/LCLS_CXI/shot-s00-2011-12-02T21_07Z29.723_00569.pickle",
)
assert not FormatPYunspecifiedStillInMemory.understand(filename)
with open(filename, "rb") as f:
d = pickle.load(f)
assert FormatPYunspecifiedStillInMemory.understand(d)
> mem_imageset = FormatPYunspecifiedStillInMemory.get_imageset(d) # noqa F841
tests/format/test_FormatPYunspecified.py:46:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
format/Format.py:321: in get_imageset
format_instance = Class(filenames[0], **format_kwargs)
format/FormatPYunspecifiedStill.py:70: in __init__
FormatPYunspecifiedInMemory.__init__(self, data, **kwargs)
format/FormatPYunspecified.py:227: in __init__
FormatPYunspecified.__init__(self, data, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, image_file = '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE', kwargs = {}
def __init__(self, image_file, **kwargs):
"""Initialise the image structure from the given file."""
if not self.understand(image_file):
> raise IncorrectFormatError(self, image_file)
E IncorrectFormatError: (<dxtbx.format.FormatPYunspecifiedStill.FormatPYunspecifiedStillInMemory object at 0x117146890>, '/Users/rjgildea/software/cctbx/modules/dxtbx/DISTANCE')
format/FormatPYunspecified.py:46: IncorrectFormatError
from dxtbx.
from dxtbx.
Related Issues (20)
- Use module_offset in multi-panel nexus
- EIGER X 16M at SPring-8 BL41XU HOT 9
- [3.17] seems incompatible with Python 3.12 HOT 5
- [3.17] error: ‘uint32_t’ does not name a type HOT 2
- [3.17] : DeprecationWarning: SelectableGroups dict interface is deprecated.
- [3.17] PytestUnknownMarkWarning: Unknown pytest.mark.regression
- Support for Rigaku Arc detectors HOT 5
- Generalise support for dynamic shadowing in dxtbx HOT 1
- NXmx bugs
- `FormatEigerNXmxFilewriter.get_raw_data` fails when number of linked data files is large HOT 3
- dxtbx.format.FormatNXmxDLS16M.FormatNXmxDLS16M no longer supports legacy data without VDS
- `FormatSMVADSCmlfsom` is too promiscuous HOT 1
- dlsnxs2cbf: produces invalid files from i04-1 Eiger 9M HOT 10
- More general support for handling of hit indices
- Define what we mean by array range HOT 11
- Treat everything internally as count-from-zero
- Figure out what is needed to remove image range
- Rename get_array_range -> get_z_range HOT 1
- Create branches in dxtbx and dials to explore impacts of removing image range etc. HOT 1
- Setting `geometry.goniometer.axis=` for a multi-axis goniometer should be disallowed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dxtbx.