spacetelescope / roman_datamodels

Datamodel support for the roman calibration pipeline

Roman Datamodels Support

Installation

The easiest way to install the latest roman-datamodels release into a fresh virtualenv or conda environment is

pip install roman-datamodels

Detailed Installation

The roman-datamodels package can be installed into a virtualenv or conda environment via pip. We recommend that for each installation you start by creating a fresh environment that only has Python installed and then install the roman_datamodels package and its dependencies into that bare environment. If using conda environments, first make sure you have a recent version of Anaconda or Miniconda installed. If desired, you can create multiple environments to allow for switching between different versions of the roman-datamodels package (e.g. a released version versus the current development version).

In all cases, the installation is generally a 3-step process:

• Create a conda environment
• Activate that environment
• Install the desired version of the roman-datamodels package into that environment

Details are given below on how to do this for different types of installations, including tagged releases, DMS builds used in operations, and development versions. Remember that all conda operations must be done from within a bash shell.

Installing latest releases

You can install the latest released version via pip. From a bash shell:

conda create -n <env_name> python
conda activate <env_name>
pip install roman-datamodels

Note
Alternatively, you can also use virtualenv to create an environment; however, this installation method is not supported by STScI if you encounter issues.

You can also install a specific version (from roman-datamodels 0.1.0 onward):

conda create -n <env_name> python
conda activate <env_name>
pip install roman-datamodels==0.5.0

Installing the development version from Github

You can install the latest development version (not as well tested) from the Github main branch:

conda create -n <env_name> python
conda activate <env_name>
pip install git+https://github.com/spacetelescope/roman_datamodels

Installing for Developers

If you want to be able to work on and test the source code with the roman-datamodels package, the high-level procedure to do this is to first create a conda environment using the same procedures outlined above, but then install your personal copy of the code overtop of the original code in that environment. Again, this should be done in a separate conda environment from any existing environments that you may have already installed with released versions of the roman-datamodels package.

As usual, the first two steps are to create and activate an environment:

conda create -n <env_name> python
conda activate <env_name>

To install your own copy of the code into that environment, you first need to fork and clone the roman_datamodels repo:

cd <where you want to put the repo>
git clone https://github.com/spacetelescope/roman_datamodels
cd roman_datamodels

Note
Installing via setup.py (python setup.py install, python setup.py develop, etc.) is deprecated and does not work.

Install from your local checked-out copy as an "editable" install:

pip install -e .

If you want to run the unit or regression tests and/or build the docs, you can make sure those dependencies are installed too:

pip install -e ".[test]"
pip install -e ".[docs]"
pip install -e ".[test,docs]"

Need other useful packages in your development environment?

pip install ipython pytest-xdist

roman_datamodels's Issues

The area array should be removed from the test utils

The area array is not be stored with level 2 data, that array should be removed from functions like mk_level2_image, along with anywhere else it is used.

https://github.com/spacetelescope/roman_datamodels/blob/main/src/roman_datamodels/testing/utils.py#L302

Data model constructors should return object instance used in constructor if right type

Currently if one supplies an existing instance of a datamodel (say RampModel) to one of the datamodel constructors (say RampModel), and the instance is the same type as the constructor, the constructor should return that instance. Currently it raises an error.

This may require some trickery to effectively replace self with the instance (I will have to brush up on this)

Unable to open association file with `datamodels.open`

Running:

import roman_datamodels.datamodels as dm
dm.open('r90001-a3001_image_001_asn.json')

Results in:

ile ~/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/asdf/asdf.py:1064, in AsdfFile._open_generic_file(cls, self, generic_file, uri, validate_checksums, extensions, _get_yaml_content, _force_raw_types, strict_extension_check, ignore_missing_extensions, **kwargs)
   1063 msg = "Input object does not appear to be an ASDF file or a FITS with ASDF extension"
-> 1064 raise ValueError(msg)

ValueError: Input object does not appear to be an ASDF file or a FITS with ASDF extension

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 dm.open('r90001-a3001_image_001_asn.json')

File ~/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/roman_datamodels/datamodels/_utils.py:93, in rdm_open(init, memmap, **kwargs)
     90 if "asn_n_members" in kwargs:
     91     del kwargs["asn_n_members"]
---> 93 asdf_file = init if isinstance(init, asdf.AsdfFile) else _open_path_like(init, memmap=memmap, **kwargs)
     94 if (model_type := type(asdf_file.tree["roman"])) in MODEL_REGISTRY:
     95     return MODEL_REGISTRY[model_type](asdf_file, **kwargs)

File ~/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/roman_datamodels/datamodels/_utils.py:54, in _open_path_like(init, memmap, **kwargs)
     52     asdf_file = asdf.open(init, **kwargs)
     53 except ValueError as err:
---> 54     raise TypeError("Open requires a filepath, file-like object, or Roman datamodel") from err
     56 # This is only needed until we move min asdf version to 3.0
     57 if AsdfInFits is not None and isinstance(asdf_file, AsdfInFits):

TypeError: Open requires a filepath, file-like object, or Roman datamodel

The error originates from roman_datamodels attempting to open the json file as as asdf file in _open_path_like here:

roman_datamodels/src/roman_datamodels/datamodels/_utils.py

Line 93 in 16c16d3

asdf_file = init if isinstance(init, asdf.AsdfFile) else _open_path_like(init, memmap=memmap, **kwargs)

There appears to be checks against the file suffix:

roman_datamodels/src/roman_datamodels/datamodels/_utils.py

Lines 97 to 104 in 16c16d3

	if isinstance(init, str):
	exts = Path(init).suffixes
	if not exts:
	raise ValueError(f"Input file path does not have an extension: {init}")
	# Assume json files are asn and return them
	if exts[0] == "json":
	return init

that occur after the call to _open_path_like (which only returns an AsdfFile), moving the suffix checks prior to _open_path_like was insufficient to fix the issue due to at least the following:

• the suffix comparison is against json not .json
• if the suffix comparison is fixed (replaced with .json) the functions returns a string
  Having rdm_open return a string is incompatible with stpipe which expects that the return value can be used as a context and the following error is seen:

Traceback (most recent call last):
  File "/Users/bgraham/.pyenv/versions/romancal_doctests/bin/strun", line 26, in <module>
    step = Step.from_cmdline(sys.argv[1:])
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/stpipe/step.py", line 186, in from_cmdline
    return cmdline.step_from_cmdline(args)
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/stpipe/cmdline.py", line 375, in step_from_cmdline
    step, step_class, positional, debug_on_exception = just_the_step_from_cmdline(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/stpipe/cmdline.py", line 308, in just_the_step_from_cmdline
    parameter_cfg = step_class.get_config_from_reference(
  File "/Users/bgraham/.pyenv/versions/3.10.6/envs/romancal_doctests/lib/python3.10/site-packages/stpipe/pipeline.py", line 182, in get_config_from_reference
    with cls._datamodels_open(dataset, asn_n_members=1) as model:
AttributeError: __enter__

Expose public API for making test data

As projects begin to add support for roman data files it would be helpful to have a documented public API for making 'test' data (similar to what is currently done in romancal using the maker_utils).

See spacetelescope/astrocut#105 (comment) for a specific example where a level 2 file (with complete WCS) would be useful.

And https://github.com/spacetelescope/jdaviz/blob/e8f125ebe375f938ed3a7b2af97e1a1c3dd5b531/jdaviz/configs/imviz/tests/utils.py#L241 for another example.

Should we drop the `random_utils` module?

The main use for random_utils was the factories module, which was removed by #197 in favor of just using the maker_utils.

Note that outside of the now removed factories (and the tests for the random_utils) there is very little in the way of any actual code using the random_utils. Indeed, there are no direct uses of the random_utils outside of roman_datamodels itself (i.e. they are not used anywhere in romancal at this time). There are two exceptions, both in the maker_utils (imported here):

1. roman_datamodels/src/roman_datamodels/maker_utils/_common_meta.py

   Line 210 in a271364

   aper["name"] = kwargs.get("name", f"WFI_{generate_positive_int(17) + 1:02d}_FULL")

2. roman_datamodels/src/roman_datamodels/maker_utils/_common_meta.py

   Line 490 in a271364

   meta["gw_acq_exec_stat"] = kwargs.get("gw_acq_exec_stat", generate_string("Status ", 15))

While outside the scope of this issue, #215 proposes removing these do make the maker_utils fully deterministic; meaning that, if #215 is pursued there will be no uses of random_utils at all meaning we should just drop random_utils.

If we decide against #215, we only actually need two functions from all the random_utils so I would instead propose moving those functions inside maker_utils as part of the private API and then remove random_utils.

Metadata display

In version 0.16.1 (the current released version on PyPI), typing metadata only returns an object instead of displaying the information. For example:

import roman_datamodels as rdm
test = rdm.open('r0000501001001001001_01101_0001_WFI01_cal.asdf')
test.meta.instrument

returns:
<roman_datamodels.stnode.WfiMode at 0x11deeb6d0>

rather than a nested representation of the content. Enclosing it like dict(test.meta.instrument) does print it out, but this is not like earlier behavior and will be not as useful for users. Can we have some way to display the nested data within metadata sections just by typing, e.g., test.meta.instrument?

Inconsistent fields between factories and `rad` schemas

This test:

roman_datamodels/tests/test_factories.py

Lines 30 to 46 in 626e0da

	@pytest.mark.parametrize("node_class", [c for c in stnode.NODE_CLASSES if isinstance(c, stnode.TaggedObjectNode)])
	def test_no_extra_fields(node_class, manifest):
	instance = create_node(node_class)
	instance_keys = set(instance.keys())
	schema_uri = next(t["schema_uri"] for t in manifest["tags"] if t["tag_uri"] == instance.tag)
	schema = asdf.schema.load_schema(schema_uri, resolve_references=True)
	schema_keys = set()
	subschemas = [schema]
	if "allOf" in schema:
	subschemas.extend(schema["allOf"])
	for subschema in subschemas:
	schema_keys.update(subschema.get("properties", {}).keys())
	diff = instance_keys - schema_keys
	assert len(diff) == 0, "Factory instance has extra keys: " + ", ".join(diff)

is currently being skipped by our unit tests.

I believe that this is a bug in the parameterization, where isinstance should be issubclass, i.e. correcting line 30 to:

@pytest.mark.parametrize("node_class", [c for c in stnode.NODE_CLASSES if issubclass(c, stnode.TaggedObjectNode)])

This then runs test_no_extra_fields on all the datamodel objects constructed from the rad schemas. Almost all of the factories pass this test. However, there are 3 models that do not pass:

• Pixelarea: extra keys: meta
• Target: extra keys: source_type_apt
• RefFile: extra keys: distortion

Are these errors in the factories for these classes, or are they missing fields in the rad schemas which should be added?

Strange `ResourceWarning`

Background: in PR #142, the use of pytest-openfiles was switched out for catching ResourceWarnings directly with pytest. This should work because Python will issue a ResourceWarning if a file handle is left open when the file goes out of scope (that is leaving a file open). For the most part this will work as a drop in replacement; however, under certain circumstances pytest is unable to raise the ResourceWarning as an error, resulting in a PytestUnraisableExceptionWarning. Typically, these are raised when testing error messages using the pytest.raises context manger. The interesting thing is that these PytestUnraisableExceptionWarnings are being issued for things not caught by pytest-openfiles. For example, in #143 some new warnings show up in the CI:

tests/test_filetype.py::test_filetype
  /home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  
  Traceback (most recent call last):
    File "/home/runner/work/roman_datamodels/roman_datamodels/tests/test_filetype.py", line 19, in test_filetype
      file_8 = filetype.check(open(DATA_DIRECTORY / "fake.asdf"))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/runner/work/roman_datamodels/roman_datamodels/tests/data/fake.asdf' mode='r' encoding='UTF-8'>
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

tests/test_models.py::test_make_saturation
  /home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/_pytest/unraisableexception.py:[78](https://github.com/spacetelescope/roman_datamodels/actions/runs/4659291564/jobs/8245992937?pr=143#step:10:79): PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  
  Traceback (most recent call last):
    File "/home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/jsonschema/validators.py", line 909, in resolve_from_url
      document = self.store[url]
                 ~~~~~~~~~~^^^^^
    File "/home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/jsonschema/_utils.py", line 28, in __getitem__
      return self.store[self.normalize(uri)]
             ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  KeyError: 'asdf://stsci.edu/datamodels/roman/schemas/reference_files/ref_common-1.0.0'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
             ^^^^^^^^^^^^^^^^^^^^^^
  ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/pytest-of-runner/pytest-0/popen-gw0/test_add_model_attribute0/testreadnoise2.asdf'>
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

tests/test_open.py::test_path_input
  /home/runner/work/roman_datamodels/roman_datamodels/.tox/test-xdist/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/contextlib.py", line 2[89](https://github.com/spacetelescope/roman_datamodels/actions/runs/4659291564/jobs/8245992937?pr=143#step:10:90), in helper
      return _GeneratorContextManager(func, args, kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/pytest-of-runner/pytest-0/popen-gw0/test_asdf_in_fits_error0/AsdfInFits.fits'>
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

All of these warnings are fixed by #149.

However, these are not the issue here. If one runs the tests twice in a row, a new PytestUnraisableExceptionWarning appears, such as:

tests/test_models.py::test_make_saturation
  /Users/wjamieson/.pyenv/versions/3.11.2/envs/roman_datamodels/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
  
  Traceback (most recent call last):
    File "/Users/wjamieson/.pyenv/versions/3.11.2/envs/roman_datamodels/lib/python3.11/site-packages/jsonschema/validators.py", line 909, in resolve_from_url
      document = self.store[url]
                 ~~~~~~~~~~^^^^^
    File "/Users/wjamieson/.pyenv/versions/3.11.2/envs/roman_datamodels/lib/python3.11/site-packages/jsonschema/_utils.py", line 28, in __getitem__
      return self.store[self.normalize(uri)]
             ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  KeyError: 'asdf://stsci.edu/datamodels/roman/schemas/reference_files/saturation-1.0.0'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/wjamieson/.pyenv/versions/3.11.2/envs/roman_datamodels/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
             ^^^^^^^^^^^^^^^^^^^^^^
  ResourceWarning: unclosed file <_io.BufferedReader name='/private/var/folders/wr/y4mz2y553jn8wwblt_9wtn240002qx/T/pytest-of-wjamieson/pytest-191/test_add_model_attribute0/testreadnoise2.asdf'>
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Some observations about this

1. The test that emits the PytestUnraisableExceptionWarning is not consistent, except that it is in a test following:

   roman_datamodels/tests/test_models.py

   Lines 555 to 571 in 626e0da

   	def test_add_model_attribute(tmp_path):
   	# First make test reference file
   	file_path = tmp_path / "testreadnoise.asdf"
   	utils.mk_readnoise(filepath=file_path)
   	readnoise = datamodels.open(file_path)
   	readnoise["new_attribute"] = 77
   	assert readnoise.new_attribute == 77
   	with pytest.raises(ValueError):
   	readnoise["_underscore"] = "bad"
   	file_path2 = tmp_path / "testreadnoise2.asdf"
   	readnoise.save(file_path2)
   	readnoise2 = datamodels.open(file_path2)
   	assert readnoise2.new_attribute == 77
   	readnoise2.new_attribute = 88
   	assert readnoise2.new_attribute == 88
   	with pytest.raises(ValidationError):
   	readnoise["data"] = "bad_data_value"

2. The precise error (in the above example the KeyError) that is emitted, while the PytestUnraisableExceptionWarning occurs is also inconsistent.
3. What will make the warning consistently disappear is removing:

   roman_datamodels/tests/test_models.py

   Lines 570 to 571 in 626e0da

   	with pytest.raises(ValidationError):
   	readnoise["data"] = "bad_data_value"

   from test_add_model_attribute.
4. This PytestUnraisableExceptionWarning only gets emitted when running the entire test sweet twice in a row. That is I cannot reproduce the issue in isolation as the tests are currently (even if I isolate to just tests/test_models.py). However, if I remove the pytest.raises context from line 570 (meaning test_add_model_attribute will fail), then I can reproduce the warning by just running tests/test_models.py. This warning will again occur in some later test.

The error appears to be that the testreadnoise2.asdf file is not being closed. If one plays around with the test and closing the underlying files when an error is raised, one can switch the offending file to testreadnoise.asdf (the first file in the test).

This seems like a very deep bug, and it is a bit disturbing that I cannot even isolate it to a single run of the unit tests (let alone a single test module or test). I know it is real because I can reproduce it by following the steps above.

My testing was done in a new environment created by installing roman_datamodels using PR #149's branch on Python 3.11.2.

Fix empty DataModel initialization

Issue RCAL-612 was created on JIRA by William Jamieson:

It has been reported that initializing an empty DataModel of any type results in the model not being initialized around a DNode instance. This results in the inability to directly add any data to the model without reaching into its private state.

The init for DataModel should be updated to handle this case.

Updates to identify association files for processing

Update roman_datamodels to handle romancal pipeline processing of associations.

random_utils need to let user set seed

I don't see any way to set a seed for https://github.com/spacetelescope/roman_datamodels/blob/main/src/roman_datamodels/random_utils.py

Also while you are at it, maybe use the new API: https://numpy.org/doc/stable/reference/random/generator.html

Datamodels should allow creation of new attributes

While the node machinery allows this, the top level datamodel class does not. It should.

DOC: datamodels stuff should be here not in pipeline docs

https://roman-pipeline.readthedocs.io/en/latest/roman/datamodels/models.html

For a user that only cares about the output file, not running the pipeline, they won't look there.

Object nodes should allow new attributes to be set with dot notation if present in schema

Currently, only attributes that have already been assigned can be set using .foo = 'bar'. Otherwise, the attribute needs to be set using ['foo'] = 'bar'. For example, this doesn't work:

node = stnode.WfiImage()
node.cal_logs = []

but this does:

node = stnode.WfiImage()
node["cal_logs"] = []
node.cal_logs = ["Hello"]

It would be convenient if attributes described by the schema could always be set with the dot operator.

Should we remove randomness from the `maker_utils`?

It is my understanding that one of the main differences between the maker_utils and the factories was that the maker_utils were supposed to deterministically generate the data to fill a datamodel. However, there are three exceptions to this:

1. roman_datamodels/src/roman_datamodels/maker_utils/_common_meta.py

   Line 210 in a271364

   aper["name"] = kwargs.get("name", f"WFI_{generate_positive_int(17) + 1:02d}_FULL")

2. roman_datamodels/src/roman_datamodels/maker_utils/_common_meta.py

   Line 490 in a271364

   meta["gw_acq_exec_stat"] = kwargs.get("gw_acq_exec_stat", generate_string("Status ", 15))

3. roman_datamodels/src/roman_datamodels/maker_utils/_datamodels.py

   Line 370 in a271364

   exptypes = choices(["SCIENCE", "CALIBRATION", "ENGINEERING"], k=shape[product_idx])

All three of these are fairly straight forward to resolve. Thus should we simply replace this randomness with something deterministic?

Deprecate and/or remove `testing.factories`

During the last Roman developers tag-up (June 2, 2023), PR #193 was discussed.

As part of the discussion it was pointed out that #193's changes (mostly in adding through automatic testing) enable the maker_utils to construct objects equivalent to those constructed by the testing.factories. Meaning that both sets of construction functions largely duplicate each others function (generating a valid stnode object). Thus it was proposed that we entirely remove the testing.factories because the maker_utils have proven to be more useful. A less drastic option would be to first deprecate testing.factories and then remove it.

Currently, romancal does not use any of the testing.factories methods in its actual code, it only mentions them in its documentation, which would be relatively easy to update to mention maker_utils instead. The testing.factories are used quite extensively for testing roman_datamodels. However, since #193 adds tests to ensure that the maker_utils stay in sync with RAD in exactly the same way as the current tests ensuring testing.factories is, it is a rather simple process to transition all the tests relying on testing.factories to using maker_utils.

The primary difference is maker_utils mostly fills the stnode objects with fixed so called "dummy" data (e.g. data arrays of all zeros, and NOSTR for strings), while testing.factories attempts to fill all items with randomly generated data.

Note that there are a few places where maker_utils is non-deterministic (it makes random choices):

• roman_datamodels/src/roman_datamodels/maker_utils.py

  Line 252 in d616487

  aper_number = generate_positive_int(17) + 1

• roman_datamodels/src/roman_datamodels/maker_utils.py

  Line 1243 in d616487

  guidewindow["meta"]["gw_acq_exec_stat"] = generate_string("Status ", 15)

• roman_datamodels/src/roman_datamodels/maker_utils.py

  Line 1183 in d616487

  exptypes = choices(["SCIENCE", "CALIBRATION", "ENGINEERING"], k=shape[product_idx])

For consistency with the rest of maker_utils it might be advantageous to update these to use deterministic (fixed) choices.

This issue is intended to be a place for the Roman developers to discuss removing testing.factories or other alternatives.

Resolve flake8 issues

This is just a reminder to address complaints from flake8 and enable the TOXENV=style job in the CI workflow.

ASDF validation fails on opening L2 data product

I'm trying to load a calibrated WFI image (available on Box) from the SOC Build 22Q4_B7 Test Suite, provided by @tddesjardins. I try:

import roman_datamodels

roman_datamodels.open("WFI01_cal.asdf")

which fails with:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[7], line 6
----> 6 roman_datamodels.open("WFI01_cal.asdf")

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/roman_datamodels/datamodels.py:425, in open(init, memmap, target, **kwargs)
    423 try:
    424     kwargs["copy_arrays"] = not memmap
--> 425     asdffile = asdf.open(init, **kwargs)
    426 except ValueError:
    427     raise TypeError("Open requires a filepath, file-like object, or Roman datamodel")

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:1839, in open_asdf(fd, uri, mode, validate_checksums, extensions, ignore_version_mismatch, ignore_unrecognized_tag, _force_raw_types, copy_arrays, lazy_load, custom_schema, strict_extension_check, ignore_missing_extensions, _compat, **kwargs)
   1828     readonly = mode == "r" and not copy_arrays
   1830 instance = AsdfFile(
   1831     ignore_version_mismatch=ignore_version_mismatch,
   1832     ignore_unrecognized_tag=ignore_unrecognized_tag,
   (...)
   1836     _readonly=readonly,
   1837 )
-> 1839 return AsdfFile._open_impl(
   1840     instance,
   1841     fd,
   1842     uri=uri,
   1843     mode=mode,
   1844     validate_checksums=validate_checksums,
   1845     extensions=extensions,
   1846     _force_raw_types=_force_raw_types,
   1847     strict_extension_check=strict_extension_check,
   1848     ignore_missing_extensions=ignore_missing_extensions,
   1849     **kwargs,
   1850 )

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:927, in AsdfFile._open_impl(cls, self, fd, uri, mode, validate_checksums, extensions, _get_yaml_content, _force_raw_types, strict_extension_check, ignore_missing_extensions, **kwargs)
    925 if close_on_fail:
    926     generic_file.close()
--> 927 raise e

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:912, in AsdfFile._open_impl(cls, self, fd, uri, mode, validate_checksums, extensions, _get_yaml_content, _force_raw_types, strict_extension_check, ignore_missing_extensions, **kwargs)
    910 generic_file = generic_io.get_file(fd, mode=mode, uri=uri)
    911 try:
--> 912     return cls._open_generic_file(
    913         self,
    914         generic_file,
    915         uri,
    916         validate_checksums,
    917         extensions,
    918         _get_yaml_content,
    919         _force_raw_types,
    920         strict_extension_check,
    921         ignore_missing_extensions,
    922         **kwargs,
    923     )
    924 except Exception as e:
    925     if close_on_fail:

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:975, in AsdfFile._open_generic_file(cls, self, generic_file, uri, validate_checksums, extensions, _get_yaml_content, _force_raw_types, strict_extension_check, ignore_missing_extensions, **kwargs)
    969         raise ValueError(
    970             "Input object does not appear to be an ASDF file. Cannot check "
    971             "if it is a FITS with ASDF extension because 'astropy' is not "
    972             "installed"
    973         ) from None
    974 elif file_type == util.FileType.ASDF:
--> 975     return cls._open_asdf(
    976         self,
    977         generic_file,
    978         validate_checksums=validate_checksums,
    979         extensions=extensions,
    980         _get_yaml_content=_get_yaml_content,
    981         _force_raw_types=_force_raw_types,
    982         strict_extension_check=strict_extension_check,
    983         ignore_missing_extensions=ignore_missing_extensions,
    984         **kwargs,
    985     )
    986 else:
    987     raise ValueError("Input object does not appear to be an ASDF file or a FITS with ASDF extension")

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:878, in AsdfFile._open_asdf(cls, self, fd, validate_checksums, extensions, _get_yaml_content, _force_raw_types, strict_extension_check, ignore_missing_extensions, **kwargs)
    876 if get_config().validate_on_read:
    877     try:
--> 878         self._validate(tree, reading=True)
    879     except ValidationError:
    880         self.close()

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/asdf.py:618, in AsdfFile._validate(self, tree, custom, reading)
    616 else:
    617     tagged_tree = yamlutil.custom_tree_to_tagged_tree(tree, self)
--> 618 schema.validate(tagged_tree, self, reading=reading)
    619 # Perform secondary validation pass if requested
    620 if custom and self._custom_schema:

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/asdf/schema.py:682, in validate(instance, ctx, schema, validators, reading, *args, **kwargs)
    679     ctx = AsdfFile()
    681 validator = get_validator(schema, ctx, validators, ctx.resolver, *args, **kwargs)
--> 682 validator.validate(instance)
    684 additional_validators = [_validate_large_literals]
    685 if ctx.version >= versioning.RESTRICTED_KEYS_MIN_VERSION:

File ~/miniconda3/envs/jdaviz-only/lib/python3.10/site-packages/jsonschema/validators.py:314, in create.<locals>.Validator.validate(self, *args, **kwargs)
    312 def validate(self, *args, **kwargs):
    313     for error in self.iter_errors(*args, **kwargs):
--> 314         raise error

ValidationError: mismatched tags, wanted 'tag:stsci.edu:asdf/unit/quantity-1.1.0', got 'tag:stsci.edu:asdf/core/ndarray-1.0.0'

Failed validating 'tag' in schema['properties']['data']:
    {'properties': {'unit': {'enum': ['electron / s'],
                             'tag': 'asdf://stsci.edu/datamodels/roman/tags/unit-1.0.0'},
                    'value': {'datatype': 'float32',
                              'ndim': 2,
                              'tag': 'tag:stsci.edu:asdf/core/ndarray-1.0.0'}},
     'tag': 'tag:stsci.edu:asdf/unit/quantity-1.1.0',
     'title': 'Science data, excluding border reference pixels.'}

On instance['data']:
    {'byteorder': 'little',
     'datatype': 'float32',
     'shape': [4088, 4088],
     'source': 8}

Using py3.10 with:

asdf-astropy==0.3.0
asdf==2.14.3
asdf-astropy==0.3.0
asdf-coordinates-schemas==0.1.0
asdf-standard==1.0.3
asdf-transform-schemas==0.3.0
asdf-unit-schemas==0.1.0
asdf-wcs-schemas==0.1.1
jsonschema==4.17.3

Any tips?

Warnings are raised by the test suite

When I run the tests in roman_datamodels locally, I am currently getting 44 warnings. Ideally, we should address and correct these warnings as they appear in the suite, either by ignoring them (if we feel they are irrelevant) or by resolving the cause of the waning itself.

The warning report I am currently getting is:

tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
tests/test_models.py:234
  /Users/wjamieson/projects/tmp/astropy-vounit/roman_datamodels/tests/test_models.py:234: PytestUnknownMarkWarning: Unknown pytest.mark.set_up_list_of_l2_files_data - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.set_up_list_of_l2_files_data(2, "asdf")

tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
tests/test_models.py:265
  /Users/wjamieson/projects/tmp/astropy-vounit/roman_datamodels/tests/test_models.py:265: PytestUnknownMarkWarning: Unknown pytest.mark.set_up_list_of_l2_files_data - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.set_up_list_of_l2_files_data(2, "asdf")

tests/test_factories.py: 7 warnings
tests/test_stnode.py: 5 warnings
  /Users/wjamieson/.pyenv/versions/3.11.2/envs/astropy-vounit/lib/python3.11/site-packages/erfa/core.py:154: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
    warnings.warn('ERFA function "{}" yielded {}'.format(func_name, wmsg),

tests/test_factories.py: 7 warnings
tests/test_stnode.py: 5 warnings
  /Users/wjamieson/.pyenv/versions/3.11.2/envs/astropy-vounit/lib/python3.11/site-packages/erfa/core.py:154: ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)"
    warnings.warn('ERFA function "{}" yielded {}'.format(func_name, wmsg),

tests/test_factories.py::test_factory_method_implemented[IpcRef]
tests/test_factories.py::test_instance_valid[IpcRef]
tests/test_stnode.py::test_serialization[IpcRef]
  /Users/wjamieson/.pyenv/versions/3.11.2/envs/astropy-vounit/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86: RuntimeWarning: overflow encountered in reduce
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

tests/test_stuserdict.py::test_init
  /Users/wjamieson/projects/tmp/astropy-vounit/roman_datamodels/tests/test_stuserdict.py:17: DeprecationWarning: Passing 'dict' as keyword argument is deprecated
    user_dict_4 = STUserDict(dict=input_1)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Add WCS to metadata in create_wfi_image

Over in jdaviz we're working on supporting Roman L2 data products (spacetelescope/jdaviz#1822). I've been testing our Roman ASDF support on some old SOC build files, but having smaller/artificial ImageModels would be helpful. Today I came across rdm.testing.factors.create_wfi_image, which nearly does what we need!

On main, the synthetic WFI image created by this method only has a photometry key in the metadata, but no wcs:

roman_datamodels/src/roman_datamodels/testing/factories.py

Line 1259 in 932afed

raw["meta"]["photometry"] = create_photometry()

Would it be possible to add a simple WCS, too? We do something like this in jdaviz for testing.

Thanks!

Catch and remove JWST kwarg asn_n_members

In stpipe the pipeline.py module the line 164 has
with cls._datamodels_open(dataset, asn_n_members=1) as model:
and so asn_n_members is getting passed to the roman datamodels_open as a kwarg
This is a quick fix to remove that argument for roman_datamodels if present.

Enable validation of all types assigned to Node attributes.

The previous PR [#253] that allowed Node attributes to be assigned other Node instances did not permit instant validation. This should be corrected.

remove python 3.8 from ci workflows

Remove python version 3.8 from the CI workflows to match romancal

Updates for gw functions in make models

update to test_models for gw updates in rad

Run coverage on test code in addition to the package code

Running coverage on the code that makes up the test suite can help to catch bugs in the tests including:

• tests that share the same name (coverage will show that only one of these tests was run and pytest will not complain when this happens)
• never used conditional statements (here is one innocuous example from this package)
• empty parameterization of tests

DOC: Add a section on file I/O

In particular, I can't find the docs on how to write a roman_datamodels.datamodels.ImageModel out into .asdf file. I thought it would have a .write method but it does not. A search on "write" on the docs search bar took forever, not sure why.

Re-add inline data representations

Issue RCAL-611 was created on JIRA by William Jamieson:

Previous refactoring of roman_datamodels removed the ability for DNode objects to print out useful representations of themselves. This should be resolved by adding a *_repr_* for DNode.

Validation errors triggered by MsosStackModel

I installed romancal version 11, and added a function to convert the stack images to the MsosStackModel and then save as an asdf file. Here is the piece of codes:

wcs = getWcs(**kwargs)
meta['wcs']=wcs
msosStack =  stnode.MsosStack()

msosStack["meta"] = meta
msosStack['data'] = fluxData  #u.Quantity(fluxData, ru.electron / ru.DN, dtype=np.float64)
msosStack['uncertainty'] = unctData
msosStack['mask'] = mask
msosStack['coverage'] = coverageMap

af = asdf.AsdfFile()
msosStackModel = rdm.MsosStackModel(msosStack)
af.tree = {'roman': msosStackModel}

line 294: msosStackModel = rdm.MsosStackModel(msosStack)

52b8672f1d4e:~/fits2Asdf# python3 fitsToasdf.py
Traceback (most recent call last):
File "/home/fits2Asdf/fitsToasdf.py", line 428, in
fits2MsosStack(fluxfile, unctfile, outfile, **kwargs)
File "/home/fits2Asdf/fitsToasdf.py", line 294, in fits2MsosStack
msosStackModel = rdm.MsosStackModel(msosStack)
File "/usr/lib/python3.10/site-packages/roman_datamodels/datamodels/_core.py", line 88, in init
asdffile.tree = {"roman": init}
File "/usr/lib/python3.10/site-packages/asdf/asdf.py", line 638, in tree
self._validate(asdf_object, custom=bool(tree))
File "/usr/lib/python3.10/site-packages/asdf/asdf.py", line 665, in _validate
schema.validate(tagged_tree, self, reading=reading)
File "/usr/lib/python3.10/site-packages/asdf/schema.py", line 677, in validate
validator.validate(instance)
File "/usr/lib/python3.10/site-packages/jsonschema/validators.py", line 314, in validate
raise error
jsonschema.exceptions.ValidationError: 'calibration_software_version' is a required property

Datamodels constructors should raise an exception if the top level node of an asdf file has the wrong type.

The current code does not check to see, when using a data model constructor, that the asdf file provided has a top node that has a type consistent with the data model class. The implementation to provide this should allow for specific cases where they are mismatched for conversion functions to be used, but this is expected to be infrequent.

Currently there exists a stub method to do this, but it currently does nothing.

Support Opening Datamodels as Another Type

Add support for opening a datamodel as another type, and having everything "under the hood" matching the new type (eg. opening a WFI ScienceRaw Level 1 file/model as RampModel).

Calls to missing (removed?) classes in testing module

There are some references to roman_datamodels.stnode.Wcsinfo:

roman_datamodels/src/roman_datamodels/testing/factories.py

Lines 1177 to 1203 in 481c3e3

	def create_wcsinfo(**kwargs):
	"""
	Create a dummy Wcsinfo instance with valid values for attributes
	required by the schema.
	Parameters
	----------
	**kwargs
	Additional or overridden attributes.
	Returns
	-------
	roman_datamodels.stnode.Wcsinfo
	"""
	raw = {
	"dec_ref": random_utils.generate_float(-90.0, 90.0),
	"ra_ref": random_utils.generate_positive_float(360.0),
	"roll_ref": random_utils.generate_float(),
	"s_region": random_utils.generate_string("Spatial extent "),
	"v2_ref": random_utils.generate_float(),
	"v3_ref": random_utils.generate_float(),
	"v3yangle": random_utils.generate_float(),
	"vparity": random_utils.generate_int(),
	}
	raw.update(kwargs)
	return stnode.Wcsinfo(raw)

but there is no Wcsinfo in stnode on main.

Search broken in docs

Search results in constant loading. I did not look closely but I suspect it is the same issue that was fixed in:
spacetelescope/jwst#7524

Resource set-up question

What is the purpose of

roman_datamodels/setup.cfg

Lines 38 to 39 in 4ecff71

	[options.package_data]
	roman_datamodels.resources = manifests/*.yaml, schemas/*.yaml, schemas/**/*.yaml

Validate Should Detect Incorrect NDArray DType

Validate should detect when the dtype of an ndarray object does not match the dtype specified in the schema.

Should we add `gwcs` as a dependency.

An issue arose where someone could not open L2 pipeline results using just roman_datamodels the issue turned out to be that gwcs was missing from the environment. Thus should gwcs be a dependency of roman_datamodels

Fix Dimension Order in Factories

In the factories, ngroups appears after ncolumns & nrows, when it should be before them (e.g. ngroups x ncolumns x nrows).