eradiate / eradiate

Eradiate: a next-generation radiative transfer model for Earth observation applications

Eradiate Radiative Transfer Model

Eradiate is a modern radiative transfer simulation software package for Earth observation applications. Its main focus is accuracy, and for that purpose, it uses the Monte Carlo ray tracing method to solve the radiative transfer equation.

Detailed list of features

• Spectral computation
  Solar reflective spectral region

  Eradiate ships spectral data within from 280 nm to 2400 nm. This range can be extended with additional data (just ask for it!).
  Line-by-line simulation

  These are true monochromatic simulations (as opposed to narrow band simulations). Eradiate provides monochromatic absorption datasets spanning the wavelength range [250, 3125] nm. It also supports user-defined absorption data provided it complies with the dataset format specifications.
  Band simulation

  These simulations computes results in spectral bands. The correlated k-distribution (CKD) method with configurable quadrature rule is used. This method achieves a trade-off between performance and accuracy for the simulation of absorption by gases. Eradiate ships with absorption datasets suitable for use within the CKD method in spectral bands of variable width (including 1 nm and 10 nm wavelength bands and 100 cm^-1 wavenumber bands), from 250 nm up to 3125 nm. It also supports user-defined absorption data provided it complies with the dataset format specifications.
• Atmosphere
  One-dimensional atmospheric profiles

  Both standard profiles, e.g. the AFGL (1986) profiles, and customized profiles are supported.
  Plane-parallel and spherical-shell geometries

  This allows for more accurate results at high illumination and viewing angles.
• Surface
  Lambertian and RPV reflection models

  Model parameters can be varied against the spectral dimensions.
  Detailed surface geometry

  Add a discrete canopy model (either disk-based abstract models, or more realistic mesh-based models).
  Combine with atmospheric profiles

  Your discrete canopy can be integrated within a scene featuring a 1D atmosphere model in a fully coupled simulation.
• Illumination
  Directional illumination model

  An ideal illumination model with a Delta angular distribution.
  Many irradiance datasets

  Pick your favourite—or bring your own.
• Measure
  Top-of-atmosphere radiance and BRF computation

  An ideal model suitable for satellite data simulation.
  Perspective camera sensor

  Greatly facilitates scene setup: inspecting the scene is very easy.
  Many instrument spectral response functions

  Our SRF data is very close to the original data, and we provide advice to further clean up the data, trading off accuracy for performance.
• Monte Carlo ray tracing
  Mitsuba renderer as radiometric kernel

  We leverage the advanced Python API of this cutting-edge C++ rendering library.
  State-of-the-art volumetric path tracing algorithm

  Mitsuba ships a null-collision-based volumetric path tracer which performs well in the cases Eradiate is used for.
• Traceability
  Documented data and formats

  We explain where our data comes from and how users can build their own data in a format compatible with Eradiate's input.
  Transparent algorithms

  Our algorithms are researched and documented, and their implementation is open-source.
  Thorough testing

  Eradiate is shipped with a large unit testing suite and benchmarked periodically against community-established reference simulation software.
• Interface
  Comprehensive Python interface

  Abstractions are derived from computer graphics and Earth observation and are designed to feel natural to EO scientists.
  Designed for interactive usage

  Jupyter notebooks are now an essential tool in the digital scientific workflow.
  Integration with Python scientific ecosystem

  The implementation is done using the Scientific Python stack.
  Standard data formats (mostly NetCDF)

  Eradiate uses predominantly xarray data structures for I/O.

Installation and usage

For build and usage instructions, please refer to the documentation.

Support

Got a question? Please visit our discussion forum.

Authors and acknowledgements

Eradiate is developed by a core team consisting of Vincent Leroy, Sebastian Schunke, Nicolas Misk and Yves Govaerts.

Eradiate uses the Mitsuba 3 renderer, developed by the Realistic Graphics Lab, taking advantage of its Python interface and proven architecture, and extends it with components implementing numerical methods and models used in radiative transfer for Earth observation. The Eradiate team acknowledges Mitsuba creators and contributors for their work.

The development of Eradiate is funded by the Copernicus programme through a project managed by the European Space Agency (contract no 40000127201/19/I‑BG). The design phase was funded by the MetEOC-3 project (EMPIR grant 16ENV03).

Citing Eradiate

The most general citation is as follows:

@software{Eradiate,
    author = {Leroy, Vincent and Nollet, Yvan and Schunke, Sebastian and Misk, Nicolas and Govaerts, Yves},
    license = {LGPL-3.0},
    title = {Eradiate radiative transfer model},
    url = {https://github.com/eradiate/eradiate},
    doi = {10.5281/zenodo.7224314},
    year = {2024}
}

If you want to reference a specific version, you can update the previous citation with doi, year and version fields populated with metadata retrieved from our Zenodo records. Example:

@software{Eradiate,
    author = {Leroy, Vincent and Nollet, Yvan and Schunke, Sebastian and Misk, Nicolas and Govaerts, Yves},
    license = {LGPL-3.0},
    title = {Eradiate radiative transfer model},
    url = {https://github.com/eradiate/eradiate},
    doi = {10.5281/zenodo.10411036},
    year = {2023},
    version = {0.25.0},
}

License

Eradiate is free software licensed under the GNU Lesser General Public License (v3).

Project status

Eradiate is actively developed. It is beta software.

eradiate's Issues

[BUG] Could NOT find Threads (missing: Threads_FOUND)

Bug report

Summary

cmake --preset default -> Could NOT find Threads (missing: Threads_FOUND)

Description

Unfortunately, I'm trying to build Eradiate with WSL on WINDOWS, but I'm getting an error message. Below you will find the error message.

Steps to reproduce

1. Install UBUNTU using WSL on Windows 10
2. Install all requirements following the installation guide

System configuration

• Operating system: Operating system: Windows 10 Professional & Ubuntu
• Compiler: Clang 11.0.0
• CMAKE: cmake version 3.22.1
• Python version: 3.8.13

Error message

(eradiate) gk@pixbox-PC:~/eradiate$ cmake --preset default
Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  MI_DEFAULT_VARIANTS="scalar_mono;scalar_mono_double;scalar_rgb;scalar_spectral;llvm_rgb;llvm_ad_rgb"
  MI_ENABLE_EMBREE="OFF"

-- The CXX compiler identification is Clang 11.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++-11 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Mitsuba v3.0.1
-- Mitsuba: targeting the native CPU architecture (specify MI_NATIVE_FLAGS to change this).
-- Mitsuba: using libc++.
-- Found Python: /home/gk/miniconda3/envs/eradiate/bin/python3.8 (found version "3.8.13") found components: Interpreter Development Development.Module Development.Embed
-- pybind11 v2.8.0 dev2
CMake Warning (dev) at /home/gk/miniconda3/envs/eradiate/share/cmake-3.22/Modules/CMakeDependentOption.cmake:84 (message):
  Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
  Syntax.  Run "cmake --help-policy CMP0127" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
  ext/mitsuba/ext/pybind11/CMakeLists.txt:101 (cmake_dependent_option)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Mitsuba: building the following variants:
--  * scalar_mono
--  * scalar_mono_double
--  * scalar_rgb
--  * scalar_spectral
--  * llvm_rgb
--  * llvm_ad_rgb
-- Dr.Jit v0.2.1
-- Dr.Jit: building the CUDA & LLVM JIT backend.
-- Dr.Jit-Core: LLVM will be loaded dynamically at runtime.
-- Dr.Jit-Core: CUDA will be loaded dynamically at runtime.
-- Dr.Jit-Core: OptiX support enabled.
-- Dr.Jit: building the autodiff backend.
-- Dr.Jit: building the Python plugin.
-- Configure ILMBASE Version: 2.5.5 Lib API: 25.0.4
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Could NOT find Threads (missing: Threads_FOUND)
CMake Error at ext/mitsuba/ext/openexr/IlmBase/config/IlmBaseSetup.cmake:138 (message):
  Unable to find a threading library which is required for IlmThread
Call Stack (most recent call first):
  ext/mitsuba/ext/openexr/IlmBase/CMakeLists.txt:35 (include)


-- Configuring incomplete, errors occurred!
See also "/home/gk/eradiate/build/CMakeFiles/CMakeOutput.log".
See also "/home/gk/eradiate/build/CMakeFiles/CMakeError.log".
(eradiate) gk@pixbox-PC:~/eradiate$

Support for Windows users

Feature request

Summary

Support for this OS should be trivial to implement but we lack a proper test infrastructure.
Usage of Github actions or other CI technologies must be studied in order to provide this feature with proper support for users.

Reason and benefit

• There is a need from our user base for this as expressed in #371
• Generally speaking, the Eradiate team wants to reach a larger user base, and we should not rely on WSL to reach MS windows users.
• It would facilitate Eradiate installation for some people

Details

• The Mitsuba kernel and its wheel must be compiled and uploaded to PyPI for Windows
• We need to define a way to test Eradiate under Windows
• We potentially need dedicated hardware in the dev team

Implement a process level locking mechanism to safely run eradiate in a multi-threaded context

Feature request

Summary

The eradiate data module does not implement any locking mechanism. This seems to be problematic when having several instances of Eradiate accessing the filesystem through xarray concurrently from different threads. I observed this behavior while using dask for distributed computations. Dask workers may be multi threaded, yet eradiate shall not run in a multi threaded context. Support of this setup has not been foreseen and I doubt we actually want to implement it. Mutli-threaded should be left to Mitsuba in our context.

However, Dask does not allow to set the max number of concurrent tasks on a worker. Limiting the number of threads in dask puts a hard limit on the CPU resources a worker has access to. As a result, a dask worker run one task at a time, or has access to multiple threads for Mitsuba computations. This is a serious limitation.

On top of this, concurrent access errors are difficult to debug, especially in a distributed setup.

Reason and benefit

This request describes the implementation of a process level locking mechanism in Eradiate, to solve the mentioned issues by:

• Protecting some clear critical sections in the eradiate data module
• Protecting the multi-threaded parts of eradiate as a critical section, since it is not suitable to have several mitsuba instances running from different python level threads.
• Provide multi-threaded execution handling facilities to easily
  • handle any potential critical section
  • assess the number of python threads in the current process and adapt the behavior accordingly
  • reproduce the mentioned issues in regression tests
  • activate/configure/deactivate the locking mechanisms for each critical section
  • help avoiding deadlocks

Details

A separated module should be implemented to handle the logic. I would call it locks, and place it at src/eradiate/locks.py. This is a suggestion that may be moved and renamed to anything more suitable. The default behavior should be defined using the eradiate.config facility

User interface

Concerning the interface, @leroyvn suggested a contextual implementation. I would call the context manager protect. This context manager would allow to specifically activate some categories of critical sections:

# Use default locking behavior (protect everything?)
experiment = eradiate.scenes.experiments.AtmosphereExperiment(...)
eradiate.run(experiment)

# Protect the eradiate data and mitsuba run critical section categories
with eradiate.protect(categories=["data", "run"]):
    experiment = eradiate.scenes.experiments.AtmosphereExperiment(...)
    eradiate.run(experiment)

# Protect the eradiate data critical section category only
with eradiate.protect(categories=["data"]):
    experiment = eradiate.scenes.experiments.AtmosphereExperiment(...)
    eradiate.run(experiment)

Developers interface

As already mentioned, a critical section would be listed in a category. For instance, the eradiate.data module would implement a critical section per dataset, each of which would be listed in the "data" category.

I suggest to use a combination of a context manager and a python decorators to provide eradiate developers with convenient ways to declare critical sections.

with eradiate.locks.declare(category="data", name="sentinel_2a-msi-8a.nc"):
    # critical section start
    ...
    xr.open_dataset(...)
    ...
    # critical section end
# not critical

or:

class DummyExample:
    ...
    @eradiate.locks.declare(category="data") # the function/method name could be used for this use case
    def method_with_critical_section(self):
        # critical section start
        ...
        xr.open_dataset(...)
        ...
        # critical section end

The decorator would simply use the context manager internally. Here is a rough implementation sketch:

class CriticalSection:
    ...
    def __enter__(self, category, name):
         ... # filter activated categories using the configuration
         self.locks[category][name].acquire()

    def __exit__(self):
        ...
        self.locks[category][name].unlock()

    def __call__(self, func, category):
        def wrapped(*args, **kwargs):
            with self(category, func.__name__):
                return func(*args, **kwargs)
        return wrapped

eradiate.locks.declare = CriticalSection()

Configuration

A locks attribute could be added to eradiate.config as a mapping, providing the boolean activated entry and a set of default protected critical section names entry called sections. The eradiate.protect context manager would then simply change the config.

eradiate.config.locks
> { "activated": True, "sections": { "data" } }

Default behavior should be decided. I advocate for at least protecting the eradiate data critical sections by default.

Additional features

The module could detect mutlti-threaded instances and issue warnings when critical sections are not protected

Some inspection features should be provided in order to list the critical sections with their statuses

This locking mechanism needs to be tested to work with dask workers multi-threading

Testing

This is an issue. I do not see how I can reproduce the concurrent access issues I have with a 100% chance. The best test I can think of for now would be likely to reproduce the errors I faced when opening several CKD datasets. The implemented critical sections would ensure that these never fail, but if the locks module was improperly used/implemented, we would end up with tests that could detect the issue

Import error while fetching the Eradiate data using the CLI

The command eradiate data fetch fails on my configuration on ref f51aa97 (main, from 25th aug 2022).

Here is the traceback from the CLI:

(msp) msp@168a3187bd2e:~$ eradiate data fetch
Reading file list from '/home/msp/sources/eradiate/resources/downloads.yml'
Fetching 'ckd/absorption/1nm/afgl_1986-us_standard-1nm-v3.nc'
Traceback (most recent call last):
  File "/home/msp/.conda/envs/msp/bin/eradiate", line 8, in <module>
    sys.exit(main())
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/msp/.conda/envs/msp/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/msp/sources/eradiate/src/eradiate/cli.py", line 178, in fetch
    path = eradiate.data._store.data_store.fetch(filename)
  File "/home/msp/sources/eradiate/src/eradiate/util/lazy_loader.py", line 93, in __getattr__
    raise AttributeError(f"module '{package_name}' has no attribute '{name}'")
AttributeError: module 'eradiate.data' has no attribute '_store'

The following line seems to be the culprit:

eradiate/src/eradiate/cli.py

Line 178 in f51aa97

path = eradiate.data._store.data_store.fetch(filename)

This might be linked to the recent implementation of the lazy loader. I don't have a lot of knowledge on how the lazy loader works in details, but after a bit of investigation, explicitely importing the _store submodule should fix the problem. Here is a proposed fix:

from eradiate.data import _store

[...]

def fetch(files, from_file):

    [...]
    
    for filename in files:
        try:
            console.print(f"[blue]Fetching '{filename}'[/]")
            path = _store.data_store.fetch(filename)
        except DataError:

[...]

Could the lazy loader load the submodules in an inline fashion, given some additional work? I have no idea how to do so.

This is low priority on my side, since I already downloaded everything. People may also be able to download the datasets manually, but it's far from being convenient.

Convert spectral radiance into radiance

Summary

Provide a standard method to convert from spectral radiance into radiance.

Reason and benefit

Current simulation results refer to a single wavelength in the form of spectral radiance (W·sr−1·m−2·nm−1). For many applications, however, it is important to integrate these results over a defined bandwidth in order to obtain a total radiance (W·sr−1·m−2) that is visible to a detector within this band.

Details

I have an application where I need to model the radiance outside the field of view for an instrument. In combination with a point source transmittance measurement on the ground, this allows me to estimate the out-of-field stray light for various in-orbit scenes. To do this, I have to convert the wavelength-related spectral radiance values into a total radiance for our bandpass filter. I think it would be helpful for different users to implement a tutorial or a standardized method on how to implement this conversion / integral correctly.

[BUG] BSDF constructor inconsistency

Bug report

Summary

There is an inconsistency in the constructor for the LambertianBSDF. When the reflectance is given as a dictionary, no unit needs to be specified and conversion to reflectance units is handled automatically. However, when the reflectance is given as a spectrum object, the unit must be added by the user. Otherwise the BSDF constructor will raise an error, citing incompatible units, DIMENSIONLESS instead of REFLECTANCE.

My suspicion is, that the converter that handles the dictionary automatically attaches the correct units.

Description

Please describe your issue in detail

Steps to reproduce

The first of the two assignments passes, the second raises an error.

import eradiate.scenes as ertsc

ground_bsdf_dict = ertsc.bsdfs.LambertianBSDF(
    reflectance = {
        "type": "interpolated",
        "wavelengths": [440, 550, 620],
        "values": [0.067269, 0.096786, 0.125058]
    }
)

ground_bsdf_obj = ertsc.bsdfs.LambertianBSDF(
    reflectance=ertsc.spectra.InterpolatedSpectrum(
        wavelengths=[440, 550, 620], 
        values=[0.067269, 0.096786, 0.125058]
    )
)

System configuration

Please provide information regarding your system configuration

• Operating system: ...
• Compiler: ...
• Python version: ... (get it from python --version)
• Eradiate version: ... (get it from eradiate show)

[BUG] Experiment KernelContext (context_init) is not initialized correctly & zero radiance

Bug report

Summary

Experiment KernelContext (context_init) is not initialized correctly, the default wavelength is 550 nm, regardless of the wavelength_range of the MolecularAtmosphere.

Description

My aim is to launch an experiment, using an Atmosphere, in the wavelength interval [1626 nm, 1690nm] with a resolution of 0.1 nm.
The maximum resolution of the CKD mode is 1nm, so I tried to use the mono mode by adapting the https://eradiate.readthedocs.io/en/stable/rst/user_guide/spectral_discretization.html#id2 section of the user guide.
The KernelContext created automatically (KernelContex.new()) doesn't take into account the wavelength interval used, which results in an exception during experiment initialization (albedo estimation).
I managed to get around this problem by creating my own initialization function.
By doing this, I'm able to launch the experiment, unfortunately all the points of the radiance vector obtained equal zeros.

Steps to reproduce

The code that build the experiment:

from typing import ClassVar

import eradiate
import eradiate.scenes as ertsc
import numpy as np
import numpy.typing as npt
from dataclasses import dataclass, field
from eradiate import unit_registry as ureg
from scipy.optimize import Bounds
from eradiate.scenes.core import traverse
from eradiate.kernel._render import mi_traverse
import mitsuba as mi

import gesat.sun as sun
from gesat.atmosphere import (
    AtmosphericProfile,
    RadiativeTransferModel,
    TOASpectralRadiance,
)
from gesat.geometry import Coordinates, observation_angles


@dataclass
class EradiateAtmosphericProfile(AtmosphericProfile):
    ch4_concentration: float  # ppb
    atmosphere: ertsc.atmosphere.MolecularAtmosphere = field(init=False)
    CH4_IDX: ClassVar[int] = 0

    def __post_init__(self):
        self.atmosphere = ertsc.atmosphere.MolecularAtmosphere.ussa_1976(
            concentrations={
                "CH4": self.ch4_concentration
                * 1e-9  # mixing ratio in ppb
                * ureg.dimensionless,
            },
            wavelength_range=(1626 * ureg.nm, 1690 * ureg.nm),
        )

    def error_covariance_matrix(self) -> npt.NDArray[np.float32]:
        return np.eye(2, dtype=np.float32)

    def state_vector_bounds(self) -> Bounds:
        return Bounds(lb=np.array([0.0]), ub=np.array([10000.0]))

    def state_vector(self) -> npt.NDArray[np.float32]:
        return np.array([self.ch4_concentration])

    def update(self, state_vector: npt.NDArray[np.float32]) -> None:
        ch4 = state_vector[EradiateAtmosphericProfile.CH4_IDX]
        self.atmosphere = ertsc.atmosphere.MolecularAtmosphere.ussa_1976(
            concentrations={
                "CH4": ch4 * 1e-9 * ureg.dimensionless,  # mixing ratio in ppb
            },
            wavelength_range=(1626 * ureg.nm, 1690 * ureg.nm),
        )


@dataclass
class Eradiate(RadiativeTransferModel):
    sensor_position: Coordinates
    surface_bsdf: ertsc.bsdfs.LambertianBSDF = field(init=False)
    illumination: ertsc.illumination.DirectionalIllumination | None = field(init=False)
    measure: ertsc.measure.MultiDistantMeasure = field(init=False)

    def __post_init__(self):
        self.surface_bsdf = ertsc.bsdfs.LambertianBSDF()

    def initialize(
        self, simulation_time: np.datetime64, location: Coordinates
    ) -> AtmosphericProfile:
        angles = sun.sun_position(location, simulation_time)
        self.illumination = ertsc.illumination.DirectionalIllumination(
            zenith=angles.zenith * ureg.deg, azimuth=angles.azimuth * ureg.deg
        )
        angles = observation_angles(location, self.sensor_position)
        eradiate.set_mode("mono")
        self.measure = ertsc.measure.MultiDistantMeasure.hplane(
            id="toa_radiance",
            zeniths=angles.zenith * ureg.deg,
            azimuth=angles.azimuth * ureg.deg,
            srf={
                "type": "multi_delta",
                "wavelengths": np.arange(start=1626.0, stop=1690.0, step=0.1) * ureg.nm,
            },
        )
        return EradiateAtmosphericProfile(ch4_concentration=1700.0)

    def run(self, atmospheric_profile: AtmosphericProfile) -> TOASpectralRadiance:
        assert isinstance(atmospheric_profile, EradiateAtmosphericProfile)
        assert self.illumination is not None
        exp = eradiate.experiments.AtmosphereExperiment(
            surface=self.surface_bsdf,  # type: ignore
            atmosphere=atmospheric_profile.atmosphere,
            illumination=self.illumination,
            measures=self.measure,
        )
        ## remove this line to see the KernelContext initialisation issue
        _init_exp(exp)
        results = eradiate.run(exp)
        return TOASpectralRadiance(data=np.squeeze(results["radiance"].to_numpy()))


def _init_exp(exp: eradiate.experiments.AtmosphereExperiment) -> None:
    kdict_template, umap_template = traverse(exp.scene)

    try:
        ctx = eradiate.contexts.KernelContext(
            si=eradiate.spectral.index.MonoSpectralIndex(w=1650 * ureg.nm),
            kwargs=exp._context_kwargs,
        )
        exp.mi_scene = mi_traverse(
            mi.load_dict(kdict_template.render(ctx=ctx)),  # type: ignore
            umap_template=umap_template,
        )
    except RuntimeError as e:
        raise RuntimeError(f"(while loading kernel scene dictionary){e}") from e

    # Remove unused elements from Mitsuba scene parameter table
    assert exp.mi_scene is not None
    exp.mi_scene.drop_parameters()

The code I use to run the experiment:

from gesat.geometry import Coordinates
import numpy as np

from gesat.rtm import Eradiate


def test_eradiate() -> None:
    sensor_position = Coordinates.geodetic(np.float64(2.3522219), np.float64(48.856614))
    observer_position = Coordinates.geodetic(
        np.float64(-1.553621), np.float64(47.218371)
    )

    model = Eradiate(sensor_position)
    profile = model.initialize(np.datetime64("2020-12-04 12:00:30"), observer_position)
    toa_radiance = model.run(profile)
    assert toa_radiance.data.shape == (640, )

System configuration

• Operating system: Ubuntu 20.06 WSL
• Compiler: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
• Python version: Python 3.11.0
• Eradiate version:

── System ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
OS: Ubuntu 20.04.3 LTS
Python: 3.11.0 (main, Mar  1 2023, 18:26:19) [GCC 11.2.0]

── Versions ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• eradiate 0.23.2
• drjit 0.4.1
• mitsuba 3.2.1
• eradiate-mitsuba 0.0.2

── Available Mitsuba variants ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• scalar_mono
• scalar_mono_double
• scalar_rgb
• scalar_spectral
• llvm_rgb

── Configuration ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• ERADIATE_AZIMUTH_CONVENTION: AzimuthConvention.EAST_RIGHT
• ERADIATE_DATA_STORE_URL: http://eradiate.eu/data/store/
• ERADIATE_DOWNLOAD_DIR: /home/seb/projects/gesat_demo/$ERADIATE_SOURCE_DIR/resources/downloads
• ERADIATE_OFFLINE: False
• ERADIATE_PROGRESS: 2 (2)
• ERADIATE_SMALL_FILES_REGISTRY_REVISION: master
• ERADIATE_SMALL_FILES_REGISTRY_URL: https://raw.githubusercontent.com/eradiate/eradiate-data
• ERADIATE_SOURCE_DIR: None

[BUG] Run Eradiate with Docker Desktop

Bug report

Summary

run Eradiate with Docker Desktop & Kernel died

Description

Unfortunately, I am not able to run the examples given in the tutorials. After the command "import eradiate" the kernel died, and no computation has been started - see image attached

Steps to reproduce

1. Run Docker desktop
2. Paste this command into a terminal: docker run -p "8888:8888" --rm -it rayference/eradiate-jupyterlab
3. Open container with server and insert the token
4. Run parts of canopy2_tutorial

System configuration

• Operating system: Windows 10 Professional & Docker Desktop 4.10.1 (82475)

image:

[BUG] Crash with PyPI package v0.24.2.post1 and xarray 2023.8

Bug report

Description

When launching a simulation with the PyPI package, in CKD mode, the code raises a ValueError. The stack trace is as follows:

Traceback (most recent call last):
  File "/home/leroyv/Documents/src/personal/sandboxes/eradiate-debug/debug_20230914.py", line 39, in <module>
    results = eradiate.run(exp)
              ^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/experiments/_core.py", line 645, in run
    exp.process(spp=spp, seed_state=seed_state)
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/experiments/_core.py", line 449, in process
    self.init()
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/experiments/_core.py", line 431, in init
    mi.load_dict(kdict_template.render(ctx=self.context_init)),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/kernel/_kernel_dict.py", line 151, in render
    value = v(ctx) if isinstance(v, InitParameter) else v
            ^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/kernel/_kernel_dict.py", line 40, in __call__
    return self.evaluator(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/scenes/atmosphere/_core.py", line 599, in <lambda>
    self.eval_albedo(ctx.si).m_as(ureg.dimensionless),
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/scenes/atmosphere/_molecular_atmosphere.py", line 254, in eval_albedo
    return self.radprops_profile.eval_albedo(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/functools.py", line 946, in _method
    return method.__get__(obj, cls)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/radprops/_core.py", line 332, in _
    return self.eval_albedo_ckd(w=si.w, g=si.g, zgrid=zgrid).squeeze()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/radprops/_afgl1986.py", line 152, in eval_albedo_ckd
    sigma_t = self.eval_sigma_t_ckd(w=w, g=g, zgrid=zgrid)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/radprops/_afgl1986.py", line 248, in eval_sigma_t_ckd
    sigma_a = self.eval_sigma_a_ckd(w=w, g=g, zgrid=zgrid)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/eradiate/radprops/_afgl1986.py", line 178, in eval_sigma_a_ckd
    ds = ds.reindex({"bd": idx})
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/xarray/core/dataset.py", line 3665, in reindex
    return alignment.reindex(
           ^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/xarray/core/alignment.py", line 890, in reindex
    aligner = Aligner(
              ^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/xarray/core/alignment.py", line 162, in __init__
    self.indexes, self.index_vars = self._normalize_indexes(indexes)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/xarray/core/alignment.py", line 209, in _normalize_indexes
    for idx, index_vars in Indexes(xr_indexes, xr_variables).group_by_index():
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/leroyv/miniforge3/envs/eradiate-debug/lib/python3.11/site-packages/xarray/core/indexes.py", line 1435, in __init__
    raise ValueError(
ValueError: unmatched keys found in indexes and variables: {'bin', 'index'}

Steps to reproduce

1. In a fresh environment, install Eradiate from PyPI: pip install eradiate==0.24.2.post1. Make sure that xarray 2023.8 is used.
2. Run the reproducer below.

import eradiate
import eradiate.scenes as ertsc
from eradiate import unit_registry as ureg
from eradiate.scenes.atmosphere import MolecularAtmosphere
from eradiate.scenes.spectra import SolarIrradianceSpectrum
from eradiate.spectral import WavelengthSet

eradiate.set_mode("ckd_double")

surface_bsdf = ertsc.bsdfs.LambertianBSDF(reflectance=0.5)
atmosphere = MolecularAtmosphere.afgl_1986(model="us_standard")
irradiance = SolarIrradianceSpectrum(dataset="coddington_2021-0.025_nm")
illumination = ertsc.illumination.DirectionalIllumination(
    zenith=15.0, azimuth=10.0, irradiance=irradiance
)
spectral_range = WavelengthSet.arange(
    400.0 * ureg.nm,
    410.0 * ureg.nm,
    10 * ureg.nm,
)
measure = ertsc.measure.MultiDistantMeasure.hplane(
    id="toa_brf",
    zeniths=10.0,
    azimuth=190.0,
    srf=ertsc.spectra.MultiDeltaSpectrum(wavelengths=spectral_range.wavelengths),
    spp=1000,
)
exp = eradiate.experiments.AtmosphereExperiment(
    default_spectral_set=spectral_range,
    surface=surface_bsdf,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=measure,
)
results = eradiate.run(exp)

System configuration

eradiate show output:

── System ─────────────────────────────────────────────────────────────────────────────────

CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
OS: Ubuntu 22.04.3 LTS
Python: 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

── Versions ───────────────────────────────────────────────────────────────────────────────

• eradiate 0.24.2.post1
• drjit 0.4.1
• mitsuba 3.2.1
• eradiate-mitsuba 0.0.2

── Available Mitsuba variants ─────────────────────────────────────────────────────────────

• scalar_mono
• scalar_mono_double
• scalar_rgb
• scalar_spectral
• llvm_rgb

── Configuration ──────────────────────────────────────────────────────────────────────────

• ERADIATE_AZIMUTH_CONVENTION: AzimuthConvention.EAST_RIGHT
• ERADIATE_DATA_STORE_URL: http://eradiate.eu/data/store/
• ERADIATE_DOWNLOAD_DIR: 
/home/leroyv/Documents/src/personal/sandboxes/eradiate-debug/.eradiate_downloads
• ERADIATE_OFFLINE: False
• ERADIATE_PROGRESS: 2 (2)
• ERADIATE_SMALL_FILES_REGISTRY_REVISION: master
• ERADIATE_SMALL_FILES_REGISTRY_URL: 
https://raw.githubusercontent.com/eradiate/eradiate-data
• ERADIATE_SOURCE_DIR: None

[BUG] jit_llvm_init(): no suitable vector ISA found, shutting down LLVM backend

Bug report

Summary

cmake --build build -> failed

Description

Unfortunately, I'm trying to build Eradiate on LINUX , but I'm getting an error messages.

• FAILED: ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o
• FAILED: ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o
• jit_llvm_init(): no suitable vector ISA found, shutting down LLVM backend..

Steps to reproduce

1. Install Eradiate on Linux following the installation guide
2. cmake --preset default

(eradiate) grit@s2ng-eradiate:~/eradiate $ cmake --preset default
Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  MI_DEFAULT_VARIANTS="scalar_mono;scalar_mono_double;scalar_rgb;scalar_spectral;llvm_rgb;llvm_ad_rgb"
  MI_ENABLE_EMBREE="OFF"

-- The CXX compiler identification is Clang 11.0.0
-- The C compiler identification is Clang 11.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++-11 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang-11 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Mitsuba v3.0.1 
-- Mitsuba: targeting the native CPU architecture (specify MI_NATIVE_FLAGS to change this).
-- Mitsuba: using libc++.
-- Found Python: /home/grit/miniconda3/envs/eradiate/bin/python3.8 (found version "3.8.13") found components: Interpreter Development Development.Module Development.Embed 
-- pybind11 v2.8.0 dev2
CMake Warning (dev) at /home/grit/miniconda3/envs/eradiate/share/cmake-3.22/Modules/CMakeDependentOption.cmake:84 (message):
  Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
  Syntax.  Run "cmake --help-policy CMP0127" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
  ext/mitsuba/ext/pybind11/CMakeLists.txt:101 (cmake_dependent_option)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Mitsuba: building the following variants:
--  * scalar_mono
--  * scalar_mono_double
--  * scalar_rgb
--  * scalar_spectral
--  * llvm_rgb
--  * llvm_ad_rgb
-- Dr.Jit v0.2.1 
-- Dr.Jit: building the CUDA & LLVM JIT backend.
-- Dr.Jit-Core: LLVM will be loaded dynamically at runtime.
-- Dr.Jit-Core: CUDA will be loaded dynamically at runtime.
-- Dr.Jit-Core: OptiX support enabled.
-- Dr.Jit: building the autodiff backend.
-- Dr.Jit: building the Python plugin.
-- Configure ILMBASE Version: 2.5.5 Lib API: 25.0.4
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Configure OpenEXR Version: 2.5.5 Lib API: 25.0.4
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /usr/bin/clang-11
-- Found OpenMP_C: -fopenmp=libomp (found version "5.0") 
-- Found OpenMP_CXX: -fopenmp=libomp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")  
-- Tests are disabled. Set FASTFLOAT_TEST to ON to run tests.
-- Mitsuba: using builtin implementation for CPU ray tracing.
-- Found Sphinx: /home/grit/miniconda3/envs/eradiate/bin/sphinx-build  
CMake Warning at ext/mitsuba/CMakeLists.txt:450 (message):
  

  Created a default 'mitsuba.conf' configuration file.  You will
  probably want to edit this file to specify the desired configurations
  before starting to compile.


-- Eradiate Plugins: targeting the native CPU architecture (specify MITSUBA_NATIVE_FLAGS to change this).
-- Eradiate Plugins: using libc++.
-- Building Eradiate plugins
-- Configuring done
-- Generating done
-- Build files have been written to: /home/grit/eradiate/build
(eradiate) grit@s2ng-eradiate:~/eradiate $ cd build
(eradiate) grit@s2ng-eradiate:~/eradiate/build $ ls
build.ninja  CMakeCache.txt  CMakeFiles  cmake_install.cmake  ext  src
(eradiate) grit@s2ng-eradiate:~/eradiate/build $ cd 
(eradiate) grit@s2ng-eradiate:~ $ cd eradiate/

3. cmake --build build

(eradiate) grit@s2ng-eradiate:~/eradiate $ cmake --build build
[41/873] Building CXX object ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o
FAILED: ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o 
/usr/bin/clang++-11 -DDRJIT_ENABLE_AUTODIFF=1 -DDRJIT_ENABLE_CUDA=1 -DDRJIT_ENABLE_JIT=1 -DDRJIT_UNROLL=" " -Ddrjit_python_EXPORTS -I/home/grit/eradiate/ext/mitsuba/ext/drjit/include -I/home/grit/eradiate/ext/mitsuba/ext/drjit/ext/drjit-core/include -I/home/grit/eradiate/ext/mitsuba/ext/drjit/ext/drjit-core/ext/nanothread/include -isystem /home/grit/miniconda3/envs/eradiate/include/python3.8 -isystem /home/grit/eradiate/ext/mitsuba/ext/pybind11/include -stdlib=libc++ -D_LIBCPP_VERSION -fcolor-diagnostics -O3 -DNDEBUG -fPIC -fvisibility=hidden   -fno-math-errno -ffp-contract=fast -fno-trapping-math -march=native -Wall -Wextra -Wno-unused-local-typedefs -flto=thin -Os -std=gnu++17 -MD -MT ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o -MF ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o.d -o ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda.cpp.o -c /home/grit/eradiate/ext/mitsuba/ext/drjit/src/python/bind_cuda.cpp
Killed
[43/873] Building CXX object ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda_ad.cpp.o
ninja: build stopped: subcommand failed.
(eradiate) grit@s2ng-eradiate:~/eradiate $ echo CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES
(eradiate) grit@s2ng-eradiate:~/eradiate $ echo $CUDA_VISIBLE_DEVICES
0
(eradiate) grit@s2ng-eradiate:~/eradiate $ cmake --build build
[1/831] Building CXX object ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o
FAILED: ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o 
/usr/bin/clang++-11 -DDRJIT_ENABLE_AUTODIFF=1 -DDRJIT_ENABLE_CUDA=1 -DDRJIT_ENABLE_JIT=1 -DDRJIT_UNROLL=" " -Ddrjit_python_EXPORTS -I/home/grit/eradiate/ext/mitsuba/ext/drjit/include -I/home/grit/eradiate/ext/mitsuba/ext/drjit/ext/drjit-core/include -I/home/grit/eradiate/ext/mitsuba/ext/drjit/ext/drjit-core/ext/nanothread/include -isystem /home/grit/miniconda3/envs/eradiate/include/python3.8 -isystem /home/grit/eradiate/ext/mitsuba/ext/pybind11/include -stdlib=libc++ -D_LIBCPP_VERSION -fcolor-diagnostics -O3 -DNDEBUG -fPIC -fvisibility=hidden   -fno-math-errno -ffp-contract=fast -fno-trapping-math -march=native -Wall -Wextra -Wno-unused-local-typedefs -flto=thin -Os -std=gnu++17 -MD -MT ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o -MF ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o.d -o ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm.cpp.o -c /home/grit/eradiate/ext/mitsuba/ext/drjit/src/python/bind_llvm.cpp
Killed
[3/831] Building CXX object ext/mitsuba/ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_llvm_ad.cpp.o
ninja: build stopped: subcommand failed.
(eradiate) grit@s2ng-eradiate:~/eradiate $ cmake --build build
[266/829] Generating srgb.coeff
Optimizing spectra ................................................................................................................................................................................................ done.
[825/829] cd /home/grit/eradiate/build/ext/mitsuba/src/python && /h...itsuba/python/mitsuba/ /home/grit/eradiate/build/ext/mitsuba/python
jit_llvm_init(): no suitable vector ISA found, shutting down LLVM backend..
[829/829] Linking CXX shared library ext/mitsuba/plugins/sphericalcoordsvolume.so
(eradiate) grit@s2ng-eradiate:~/eradiate $ cmake --build build
[2/2] cd /home/grit/eradiate/build/ext/mitsuba/src/python && /home/...itsuba/python/mitsuba/ /home/grit/eradiate/build/ext/mitsuba/python
jit_llvm_init(): no suitable vector ISA found, shutting down LLVM backend..

4. eradiate show

jit_llvm_init(): no suitable vector ISA found, shutting down LLVM backend..
─────────────────────────────────────────────────────────────── Versions ────────────────────────────────────────────────────────────────

• eradiate 0.22.4.post67+gd52619b
• mitsuba 3.0.1

────────────────────────────────────────────────────── Available Mitsuba variants ───────────────────────────────────────────────────────

• scalar_mono
• scalar_mono_double
• scalar_rgb
• scalar_spectral
• llvm_rgb
• llvm_ad_rgb

───────────────────────────────────────────────────────────── Configuration ─────────────────────────────────────────────────────────────

• ERADIATE_SOURCE_DIR: /home/grit/eradiate
• ERADIATE_DATA_PATH: None
• ERADIATE_DATA_STORE_URL: http://eradiate.eu/data/store/
• ERADIATE_DOWNLOAD_DIR: /home/grit/eradiate/resources/downloads
• ERADIATE_OFFLINE: False
• ERADIATE_PROGRESS: <ProgressLevel.KERNEL: 2>
• ERADIATE_AZIMUTH_CONVENTION: AzimuthConvention.EAST_RIGHT

5. export CUDA_VISIBLE_DEVICES="" doesn`t solve the issue

System configuration

• Operating system: Ubuntu 20.04.5 LTS
• Compiler: Clang 11.0.0
• Cmake: cmake-3.22.1-h1fce559_0
• Python version: 3.8.13Ubuntu 20.04.5 LTS
• Eradiate version: eradiate 0.22.4.post67+gd52619b
• Mitsuba version: mitsuba 3.0.1

Spectral response function to have one leading zero and one trailing zero

Feature request

Summary

Provide a way to have spectral response data start with a zero and end with a zero.
This is a feature request from @Ygovs and @lucio-f (please correct if the details are wrong)

Reason and benefit

Pre-requisite of another software that takes advantage of the SRF datasets shipped with Eradiate, whose SRF integration procedure is more accurate when the spectral response data have a leading zero and a trailing zero.

Details

Update the src/eradiate/srf_tool.py module to add this feature. The SRF datasets with leading and trailing zero can then be locally generated for use within the said software. Eradiate does not require leading/trailing zeros in the spectral response data.

[BUG] Quickstart does not work

Bug report

Summary

To test eradiate I simply tried running the sequence proposed in the Quickstart.
An error occurred when importing the eradiate.scenes module

Description

The code is very simple:

import eradiate

import numpy as np

import eradiate.scenes as ertsc
from eradiate import unit_registry as ureg
eradiate.set_mode("mono")

if __name__ == "__main__":

    surface_bsdf = ertsc.bsdfs.LambertianBSDF()
    atmosphere = ertsc.atmosphere.MolecularAtmosphere.ussa_1976(has_absorption=False)
    illumination = ertsc.illumination.DirectionalIllumination(
        zenith=15.0, azimuth=0.0
    )
    measure = ertsc.measure.MultiDistantMeasure.hplane(
        id="toa_brf",
        zeniths=np.arange(-75, 76, 5),
        azimuth=0,
        srf={"type": "multi_delta", "wavelengths": 550.0 * ureg.nm},
        spp=10000,
    )

    exp = eradiate.experiments.AtmosphereExperiment(
        surface=surface_bsdf,
        atmosphere=atmosphere,
        illumination=illumination,
        measures=measure,
    )

    results = eradiate.run(exp)
    print(results)

When running it I got:

Traceback (most recent call last):
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/enum.py", line 276, in __set_name__
    enum_member = enum_class._value2member_map_[value]
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'noinit'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/seb/projects/gesat_demo/src/gesat/rtm.py", line 7, in <module>
    eradiate.set_mode("mono")
    ^^^^^^^^^^^^^^^^^
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/site-packages/lazy_loader/__init__.py", line 77, in __getattr__
    submod = importlib.import_module(submod_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/site-packages/eradiate/_mode.py", line 9, in <module>
    from .attrs import documented, parse_docs
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/site-packages/eradiate/attrs.py", line 62, in <module>
    class DocFlags(enum.Flag):
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/enum.py", line 538, in __new__
    raise exc
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/enum.py", line 296, in __set_name__
    and _is_single_bit(value)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/miniconda3/envs/gesat_demo/lib/python3.11/enum.py", line 98, in _is_single_bit
    num &= num - 1
           ~~~~^~~
TypeError: unsupported operand type(s) for -: 'str' and 'int'

Steps to reproduce

When simply running 'eradiate show', the same error occurs

System configuration

• Operating system: Ubuntu 22.04 WSL
• Compiler: .gcc
• Python version: ... Python 3.11.0
• Eradiate version: ... 0.23.2

RAMI benchmark solver runtime error: "Unreferenced attribute orientation in distant"

Bug report

Summary

I tried to implement the RAMI scene, and ran into the following error:

RuntimeError: ​[xml_v.cpp:269] Unreferenced attribute orientation in distant

Description

I tried to implement some code on the new RAMI scene/solver, and ran into the following error whilst running app.run(). I guess it is an error is the distance measure, but not sure?

---

RuntimeError                              Traceback (most recent call last)
<ipython-input-30-b72f3165c8a5> in <module>
----> 1 app.run()

~/Documents/test/eradiate/eradiate/solvers/rami/app.py in run(self)
    436         Essentially chains :meth:`process` and :meth:`postprocess`.
    437         """
--> 438         self.process()
    439         self.postprocess()
    440 

~/Documents/test/eradiate/eradiate/solvers/rami/app.py in process(self)
    289         """
    290         self._raw_results = None  # Unset raw results for error detection
--> 291         self._raw_results = self._runner.run()
    292 
    293     def postprocess(self):

~/Documents/test/eradiate/eradiate/solvers/onedim/runner.py in run(self)
     73 
     74         # Run computation
---> 75         kernel_scene = self.kernel_dict.load()
     76         for sensor in kernel_scene.sensors():
     77             kernel_scene.integrator().render(kernel_scene, sensor)

~/Documents/test/eradiate/eradiate/scenes/core.py in load(self)
    104         self.check()
    105         from eradiate.kernel.core.xml import load_dict
--> 106         return load_dict(self)
    107 
    108 

RuntimeError: ​[xml_v.cpp:269] Unreferenced attribute orientation in distant

Steps to reproduce

Please enumerate the steps to follow in order to reproduce the issue

import eradiate
from eradiate.solvers.rami.app import RamiSolverApp
from eradiate.scenes.biosphere import HomogeneousDiscreteCanopy
from eradiate.scenes.surface import LambertianSurface
from eradiate.scenes.illumination import DirectionalIllumination
from eradiate.scenes.measure import DistantMeasure
from eradiate.solvers.rami.app import RamiScene

eradiate.set_mode("mono_double", wavelength=550.0)

canopy = HomogeneousDiscreteCanopy.from_parameters()
surface = LambertianSurface(reflectance=0.15)
illumination = DirectionalIllumination(zenith=30., azimuth=45.)
measure = DistantMeasure(id="toa_brf",film_resolution=(32, 32),spp=1000)

scene = RamiScene(
    canopy=canopy,
    surface=surface,
    illumination=illumination,
    measures=measure)

app = RamiSolverApp(scene=scene)

app.run()

Configuration file

Please include the configuration file used in your computation here, if applicable

System configuration

Please provide information regarding your system configuration

• Operating system: Mac OS Big Sur 11.1
• Compiler: clang version 4.0.0
• Python version: 3.7.6
• Eradiate version: ...

[BUG] HAPKE BSDF produces invalid values on DEM surfaces

Bug report

Summary

Using the Hapke BSDF model on a DEM surface produces invalid samples.

Description

When I use the Hapke BSDF on a flat surface, I get meaningful results. If I attach the BSDF with the same paramters to a DEM surface (50x50km Algeria5 in this case) I get a very large number of invalid values. The number of invalid values varies with the parametrs, but I did not find a configuration, which produces no invalid samples.

Using the RPV BSDF on the same DEM produces zero invalid samples.

Steps to reproduce

These are the parameters I used.

HapkeBSDF(
    w=0.6,
    h=0.043,
    B_0=0.936,
    theta=14.1,
    b=0.189,
    c=0.452
)

Building CXX object bilambertian.cpp.o fails: "call to 'abs' is ambiguous"

Bug report

Summary

When building Eradaite the following the process stops with the following error message:

[2/3] Building CXX object src/plugins/src/bsdfs/CMakeFiles/bilambertian.dir/bilambertian.cpp.o
FAILED: src/plugins/src/bsdfs/CMakeFiles/bilambertian.dir/bilambertian.cpp.o
../src/plugins/src/bsdfs/bilambertian.cpp:154:47: error: call to 'abs' is ambiguous

Description

Complete error log is attached,
Eradiate_Build_error.log

together with the preset output.
Eradiate_Build_error.log

Steps to reproduce

Error os produced during the build cmake --build build commando

System configuration

Please provide information regarding your system configuration

• Operating system: MacOS 10.15.7
• Compiler: ...
• Python version: 2.7.16
• Eradiate version: Most recent Git clone

SOLID composite solar spectral irradiance data set

Enhancement

Summary

Add the SOLID composite solar spectral irradiance data set.

Reason and benefit

This particular solar spectral irradiance data set is recommended by the Committee on Earth Observation Satellites (CEOS).

The users will benefit from a larger choice of solar spectral irradiance data sets, including recent and trusted data sets.

Details

If the data is available for everyone and for free, the data should simply be formatted to the eradiate data set format and added to eradiate-data. Then the eradiate.data.solar_irradiance_spectra module must be updated to include this new data set in the list of supported data sets.

[BUG] Installation fails under Windows

Bug report

Summary

Installation fails under Windows

"Eradiate is delivered through PyPI and can be installed using the pip. This is the recommended way to install Eradiate."

• Installation fails under Windows

Description

 pip install eradiate==0.24.3 

  Obtaining dependency information for eradiate==0.24.3 from https://files.pythonhosted.org/packages/55/ab/b0f674af2766b638e5d24ebddd26e4b8173c0805df6891b1e37759af8e1a/eradiate-0.24.3-py3-none-any.whl.metadata
  Using cached eradiate-0.24.3-py3-none-any.whl.metadata (8.4 kB)
INFO: pip is looking at multiple versions of eradiate to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement eradiate-mitsuba==0.0.2 (from eradiate) (from versions: 0.0.1rc5, 0.0.1rc6, 0.0.1rc7, 0.0.1rc8, 0.0.1rc9)
ERROR: No matching distribution found for eradiate-mitsuba==0.0.2

Steps to reproduce

pip install eradiate==0.24.3 under Windows

System configuration

• Operating system: Windows 11
• Python version: v3.11
• Eradiate version: v.0.24.3

Invalid Eradiate results on a white Lambertian surface

Is this the bug we deserve?

Summary

It may be related to this mitsuba issue, AKA the batman bug
My Eradiate results seem a bit off on a simple white lambertian surface for some specific viewing geometries

This can have a significant impact on Eradiate bias wrt other models

Description

The scene is a very simple scattering only (or Rayleigh) atmosphere on top of a white lambertian surface.
The results around vza=0 are quite far from the expected value.

I'm still not sure the problem does not come from my Eradiate config, since I'm refactoring a big part of my code.
However, I can not find any problem even after reviewing the config several times. May I ask a double check?
I may have missed something while plotting the results?

Correct me if I'm wrong, but the batman bug generally produces results with a BRF exactly equal to the surface reflectance. It's not the case here.

I tried different SPP values, up to 10000. It does not look like a noise issue.

Configuration file

The exact configuration I'm using is available here

Steps to reproduce

Assuming you downloaded the configuration file in local, here is a simple python code to reproduce my computations

import json
import eradiate
eradiate.set_mode("ckd_double")

with open("white_scattering_config.json") as f:
    config = json.load(f)
    experiment = eradiate.experiments.Rami4ATMExperiment(**config)
    experiment.run()
    ds = experiment.results["pipeline_measure"]
    ds.squeeze().brf.plot()
    ds.squeeze().mean(dim="w").brf.plot()

System configuration

• Operating system: Linux fenrir 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxGNU/Linux
• Compiler: clang-9
• Python version: Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) [GCC 9.4.0] on linux
• Eradiate version: from commit f83994c

Simple experiment produces zero radiance with Apple M2 CPU [BUG]

Bug report

Summary

I am creating this issue to report a bug that occurs on Apple M2 CPU (initially reported by @Srheft in #332).

Description

It seems that although Eradiate can be successfully installed on a system with an Apple M2 CPU, a simple experiment such as the experiment that is created in the First steps with Eradiate tutorial, produces incorrect results ; the radiance values are zero.

Steps to reproduce

Run the First steps with Eradiate tutorial.

System configuration

── System ──────
CPU: b'Apple M2 Pro'
OS: 13.4
Python: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:05:36) [Clang 14.0.6 ]
── Versions ─────────
• eradiate 0.23.1.post1.dev26+g9a3bdcee
• drjit 0.4.1
• mitsuba 3.2.1
── Available Mitsuba variants ──
• scalar_mono
• scalar_mono_double
• scalar_rgb
• scalar_spectral
• llvm_mono_double
• llvm_ad_rgb
── Configuration ─────────
• ERADIATE_AZIMUTH_CONVENTION: AzimuthConvention.EAST_RIGHT
• ERADIATE_DATA_STORE_URL: http://eradiate.eu/data/store/
• ERADIATE_DOWNLOAD_DIR: /Users/seftekh1/eradiate/resources/downloads
• ERADIATE_OFFLINE: False
• ERADIATE_PROGRESS: ProgressLevel.KERNEL (2)
• ERADIATE_SOURCE_DIR: /Users/seftekh1/eradiate

Investigate high /tmp usage on Linux system

It seems that Eradiate does not cleanup its temporary files. It is a problem for long lasting computation as files accumulate quite quickly in /tmp. I had more than ~100GB in a night of computations.

eradiate/src/eradiate/scenes/atmosphere/_core.py

Lines 367 to 371 in f83994c

	def update(self) -> None:
	"""
	Update internal state.
	"""
	self.cache_dir.mkdir(parents=True, exist_ok=True)

The update function overwrites the current cache directory location without removing the previous one on disk.
I can't see anywhere else in the codebase a line removing the cachedir.

Eradiate uses the tempfile module to handle the creation of temporary dicts.

eradiate/src/eradiate/scenes/atmosphere/_core.py

Line 4 in f83994c

import tempfile

eradiate/src/eradiate/scenes/atmosphere/_core.py

Lines 340 to 349 in f83994c

	cache_dir: pathlib.Path = documented(
	attr.ib(
	factory=lambda: pathlib.Path(tempfile.mkdtemp()),
	converter=pathlib.Path,
	validator=attr.validators.instance_of(pathlib.Path),
	),
	doc="Path to a cache directory where volume data files will be created.",
	type="path-like",
	default="Temporary directory",
	)

This module does not provide any mechanism for cleaning up the temporary files.

Did I miss a cleanup mechanism somewhere?
Would it be sufficient and reliable to add a call to shutil.rmtree in this update function?

Cannot configure CMake for compilation

Bug report

Summary

I cannot configure CMake for compilation with the requested command cmake --preset default

Description

Following the installation procedure for the mac, I reached the compilation step for the radiometric kernel. Everything went well up to this point. I am in the main eradiate directory. Trying to configure CMake for the compilation with the requested command cmake --preset default returns an error

CMake Error: Could not read presets from xxx/xxx/xxx/eradiate: File not found

Steps to reproduce

Please enumerate the steps to follow in order to reproduce the issue. If possible, please provide a minimal reproducible example.

1. cd eradiate
2. cmake --preset default

The only preset file cmake will understand is in eradiate/ext/mitsuba/
when typing cmake --preset eradiate
it returns

  CMAKE_BUILD_TYPE="Release"
  MI_DEFAULT_VARIANTS="scalar_mono;scalar_mono_double;scalar_rgb;scalar_spectral;llvm_rgb"
  MI_ENABLE_EMBREE="OFF"

-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- The C compiler identification is AppleClang 12.0.0.12000032
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at CMakeLists.txt:60 (message):
  In-tree builds are not permitted.  To recover, delete 'CMakeCache.txt', the
  'CMakeFiles' directory and inform CMake about the source (-S) and build
  (-B) paths.  For example to compile to a directory labeled 'build' using
  the Ninja generator, enter

    $ rm -Rf CMakeCache.txt CMakeFiles
    $ cmake -S . -B build -G Ninja
    $ cmake --build build


-- Configuring incomplete, errors occurred!

System configuration

Please provide information regarding your system configuration

• Operating system: Mac OS 10.15.7
• Compiler: Apple clang version 12.0.0 (clang-1200.0.32.21)
• Python version: Python 3.8.17
• Eradiate version: v0.23.1
• cmake version 3.26.4

Thank you very much in advance for your help.

[BUG] Conda-lock env solver fails to resolve drjit dependency for OSX using pip.

Bug report

Summary

          As a result of the poetry solver issue, users who may want to use the conda environments `optional` or `dependencies` (from the locks or the env yaml files) will need to perform the installation of the `eradiate-mitsuba` pkg on their own using pip. This is a very minor issue, since these conda environments are only provided for rare corner cases. The `dependencies` layer is used for the wheel main deps list, its purpose isn't to be installed this way with conda. The `optional` layer is only a convenience layer, and most likely won't be used. The new option `-p` for `pip-deps` allow the script to generate conda envs with the pip deps when the solver will be fixed. Complete `dependencies` and `optional` environments can be setup using the pip locks anyway.

Originally posted by @wint3ria in #338 (comment)

Description

We can not recreate the lock files for OSX using conda-lock for now. This is due to a failure in conda-lock to retrieve the DrJIT package for the platform. Wheels are indeed generated for DrJIT on OSX.

An issue has been posted here conda/conda-lock#436 by @leroyvn

Steps to reproduce

Recreating the lock files for OSX wil fail with the following traceback:

conda-lock --kind explicit --no-mamba --file requirements/conda/environment-dependencies.yml --filename-template "requirements/conda/environment-dependencies-{platform}.lock" -p osx-64

Locking dependencies for ['osx-64']...
INFO:conda_lock.conda_solver:osx-64 using specs ['python 3.8.*', 'pip *', 'aenum', 'attrs >=22.2', 'click', 'dessinemoi >=23.1.0', 'environ-config', 'lazy_loader >=0.1', 'matplotlib >=3.3', 'netcdf4', 'numpy', 'pint', 'pinttrs >=23.2.0', 'pooch', 'portion', 'rich', 'ruamel.yaml', 'scipy', 'shellingham !=1.5.1', 'tqdm', 'typer >=0.9.0', 'xarray >=0.19,!=0.20.*']
Traceback (most recent call last):
  File "/home/msp/.conda/envs/eradiate/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/conda_lock.py", line 1266, in lock
    lock_func(
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/conda_lock.py", line 996, in run_lock
    make_lock_files(
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/conda_lock.py", line 359, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/conda_lock.py", line 746, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/conda_lock.py", line 682, in _solve_for_arch
    pip_deps = solve_pypi(
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/pypi_solver.py", line 294, in solve_pypi
    link = chooser.choose_for(op.package)
  File "/home/msp/.conda/envs/eradiate/lib/python3.8/site-packages/conda_lock/_vendor/poetry/installation/chooser.py", line 72, in choose_for
    raise RuntimeError(
RuntimeError: Unable to find installation candidates for drjit (0.4.1)

This only affects the dependencies and optional environment locks on OSX.

Workaround

Use pip locks on OSX instead, or manually install the eradiate-mitsuba package using pip on OSX

Proposed solutions

• Fix the conda-lock solver
• Create a Mitsuba & DrJIT packages for conda so that we can get rid of the pip deps for this solver

System configuration

• Operating system: Linux (Ubuntu)
• Compiler: N/A
• Python version: 3.8.16
• Eradiate version: N/A, WIP on the DMS. See #330 #338

Create an Eradiate package [2/3]

This feature request is the second step in the creation of a PyPI Eradiate package. This issue is created to discuss the requirements and possible implementations of a patched dependencies management system (DMS) suitable for the wheel distribution. A few design ideas have been developed, and are left here for the record. The design number 3 is the one implemented in this PR. This PR is the second in a series of 3 PRs related to the packaging. The first one #328 focused on the implementation of a wheel for Eradiate and its upload to PyPI, and the last one will focus on the release process, testing of the different envs, and documentation.

Summary

Since #328 , an Eradiate package is available through Pypi. It can be installed alongside the Eradiate-specific Mitsuba rendering kernel compiled for that purpose using the following pip command:

pip install "eradiate[production]"

The [production] suffix bothers us very much, because using the default, simpler, and more intuitive command pip install eradiate would not install Mitsuba, and thus provide an incomplete installation to end users.

Reason and benefit

We could stick to this state of affairs and gently suggest anyone who do not install the package correctly to RTFM, but I am a depressive maniac, not a mean developer. As a result, we need to find a solution to this cosmetic problem.

On top of this inconvenience, the current DMS forces the wheel to define the dev, tests, docs, and optional dependency layers as optional dependency sets. These sets are defined for development or CI/CD purposes only, for use cases where having access to an Eradiate source repository is mandatory. As such, including them in the Eradiate wheel definition, only used for production by end users working outside of the sources, is an anti-pattern and might only produce confusion.

Furthermore, no convenient way to switch deps layer has been provided to developers so far, to the best of my knowledge. This could be a nice to have feature.

Details

The following subsections are descriptions of the different setups we thought about and their issues.

Design 0

We stick to the current setup as of #328 . Eradiate installation is documented for prod with highlights on the importance of using the eradiate[production] deps set. It is simple. It is ready, we can move on to other topics. It is ugly. Some users will fail to see the doc. It's a bit weird to have a broken install when simply doing pip install eradiate.

Design 1

We add eradiate_mitsuba to the main layer. Hence it is installed anyway in any setup. If an Eradiate source repository is detected by the eradiate code, then it has the precedence over the Eradiate package and the Eradiate Mitsuba kernel package.

The problems with this setup are the following:

• The mitsuba command precedence can not be defined by the wheel and will depend on the user environment. As a result, we can not guarantee which binary would be called when a developer performs a CLI mitsuba operation: the packaged one or the one built in source?
• It won't be possible to have "light-weight" layers not requiring Mitsuba for some specific CI/CD steps for instance.
• It breaks the deps layering concept in the first place. There is no good reason to have layers if we then have special cases for some packages
• The wheel uses the optional dependencies as defined in the pyproject.toml. We would thus end up with inconsistent layer definitions between the unused dev dep sets defined in the wheel, and the actual dev deps layers used when working on the Eradiate source tree.
• I am not sure if it's easily maintainable or not

Design 2

We deploy two different packages to Pypi:

• eradiate-dev
• eradiate

eradiate-dev is the renamed package implementation as of #328 . eradiate is just a shallow package, requiring eradiate-dev[production].

It's rather simple to implement, and (only) fixes the cosmetic issue.

Problems:

• It will create some confusion.
• Uninstalling eradiate through pip uninstall eradiate would not uninstall eradiate-dev, and thus the Eradiate actual implementation would still be available after uninstalling
• It does not solve the problems of dev deps being uploaded to Pypi as optional deps of the prod package.

Design 3

I tend to believe now that a lot of the problems we see RN are due to the fact we define the layers in the pyproject.toml file as optional dependencies. The pyproject.toml file is here to define the deps of our production wheel. I don't see real good reasons to have the dev layers defined there in the end.

Since we already have a set of script that generate requirements*.txt files in the requirements directory, we could rewrite our pyproject.toml to include those latter instead. This is fairly easy to do using the dynamic field of the pyproject.toml, and I have succeeded in doing so while testing a bit. The requirements/*.py scripts read the pyproject.toml optional dependency sets in the first place, so we would indeed need to reformulate them to source their informations on the layers from somewhere else.

The idea here is to redefine the layers of the DMS in another file than pyproject.toml. The scripts in the requirements folder would then read the layers definitions and create requirement-*.txt files. Finally, the pyproject.toml only imports the requirement lists necessary for its default and optional dependency sets. Extending the requirements/layered.yml would be a good candidate for defining the content of the source layers.

Additional improvements

As of #328 , conda deps are assumed to be transparently translatable from Pypi package names. This is not really the case, and we could instead generate conda environments that add a pip deps if the exact same conda package can not be found. Installing pip deps using conda env definitions is supported by Conda. This is the case in particular for the eradiate_mitsuba package that does not have a conda counterpart. As a result, #328 provides a simple fix for this specific package. The idea here would be to generalize this process. Will not fix: we prefer to handle these cases manually to avoid shipping incorrect deps without any warning.

Dependency layers production and packaging requires a bit more testing in the CI. Out of the scope of the current PR. Should be done in subsequent PR (3/3) updating the CI, env testing, and release process.

Nice to have: provide a simple way to switch layer in dev mode

Test failure: eradiate/radprops/tests/test_rad_profile.py::test_us76_approx_rad_profile

Bug report

Summary

The test_rad_profile.py fails with an HDF error which look like a file format issue.

Description

I had this issue on both the testing docker container and my development environment
Please find attached the output log of pytest

test-logs.txt

Steps to reproduce

You simply need to run pytest eradiate in the source folder. The additional datasets for Eradiate need to be loaded.

Configuration file

N/A

System configuration

• Operating system: MacOS, MacOS+Docker, Ubuntu, Ubuntu+Docker
• Compiler: clang++
• Python version: 3.7.10 Anaconda
• Eradiate version: 0.0.1