atmtools / arts Goto Github PK

Most importantly, rename doit_i_field. What did we decide?

Other similar stuff?

Move general functions from doit-files, to e.g. m_cloudbox.

Describe the bug
The reason OEM does not work is sometimes very unclear. In the attached file, a simple test case runs through some code trying to retrieve water vapor in a simplified setup from some data. The code fails. I am sure it fails because the inversion of the measurement error covariance matrix is not very good for this particular case, This is reported simply as "Error during OEM computation.", which is somewhat vague. Should we perhaps introduce a "oem_checkedCalc" function, and let this deal with error messages?

Anyways, there is a larger problem here as well. If I run OEM again after seeing the std::cout-error, I get a segmentation fault.

To Reproduce
Unzip this file:
tmp.zip

Change the XMLDIR variable at the top of the python file to a string pointing at your own arts-xml-dir. Run to see the segmentation fault. If you run while changing "lm" to "li", again in the settings the oem_model variable, then the error persists but the segmentation fault is not there, so it looks like a "lm" problem.

If applicable, error output:

                                MAP Computation                                
Formulation: Standard
Method:      Levenberg-Marquardt

 Step     Total Cost         x-Cost         y-Cost    Conv. Crit.   Gamma Factor
--------------------------------------------------------------------------------
    0   -6.30048e+17              0   -6.30048e+17                    100.000000
--------------------------------------------------------------------------------

Error during OEM computation.

                                      ----                                      


                                MAP Computation                                
Formulation: Standard
Method:      Levenberg-Marquardt

 Step     Total Cost         x-Cost         y-Cost    Conv. Crit.   Gamma Factor
--------------------------------------------------------------------------------
    0            nan            nan   -6.30048e+17                    100.000000
Segmentation fault (core dumped)

Expected behavior
Either tell me the error in clear text in a throw/raise call, or ensure it is dealt with quietly

System info (please complete the following information):

• OS: Ubuntu 20.04

Richard: I have created the first version of a test targeting Jacobians by iyActiveSingleScat. Presently it just checks that things run. The test runs with iyActiveSingleScat but crashes with iyActiveSingleScat2.

The test is called TestIyActive_wfuns.arts

@m-brath I have now introduced spectral_radiance field and worked on methods to set this WSV. I leave it to you to revise the part from this WSV to heating rates. As mentioned, the connection to the cloudbox shall be removed. There are a lot of TBD in the description of WSVs and WSMs for those calculations. In addition, try to start the descriptions with a one-lines, as done for most other WSVs and WSMs.

Just a crazy idea, triggered by the line catalogue issue I juste created:

How about creating a local arts-xml-data folder during build and install, initially empty.

Then create new set of reading routines ReadArtsData(...).

They would look in the local arts-xml-data for the given file. If its there, they behave just like ReadXML. But if the file does not exist, they download it from the online master arts-xml-data location. (We could even think about a feature that checks if the online version is newer and re-downloads.)

I have no idea how easy it is to access the internet from within C++. It certainly opens a big (perhaps Pandoran) box. But if that is possible, the rest should be pretty simple, it seems to me.

Variables and methods now named as _transmission should be renamed to _transmitance?

Let us keep _trans for simplicity

Hi @riclarsson,

I'm trying to read an ARTS catalog with the new pyarts classes interface and would like to output the species and some parameters of each line. Accessing f0 and i0 is no problem and I'm sure I'm just completely blind here, but how do I access the species name?

import pyarts

ws = pyarts.workspace.Workspace()

ws.ReadHITRAN(filename="HITRAN2016.par", fmin=1e9, fmax=1.1e9)
ws.WriteXML("ascii", ws.abs_lines, "hitran.xml")

aalines = pyarts.classes.ArrayOfAbsorptionLines()
aalines.readxml("hitran.xml")

all_lines = [
    {"f0": line.f0, "i0": line.i0} for alines in aalines for line in alines.lines
]

all_lines.sort(key=lambda x: x["f0"])
for line in all_lines:
    print(line)

The species name is in the xml file as an attribute, but I couldn't find a way to access it through the AbsorptionLines python class.

• Implement consistent treatment of surface between RT4 and Disort.
• Possibly utilize BRDF support offered by C Disort to implement non-Lambertian surfaces.

Can be started after PR #80 is merged.

As far as I understand there is a circular dependency in the way ARTS is built. The
current process is:

1. Build make_auto_workspace to create auto_workspace.h
2. Build make_auto_md_h and make_auto_md_ccto createauto_md.hBUT:make_auto_md_ccdepends onauto_workspace.hthroughworkspace_ng.h```

The problem is that auto_workspace.h contains the calls to the constructors of all
groups. Now if a group wants to call an agenda it has to import auto_md.h so it
depends on make_auto_md_cc, which closes the loop.

This becomes problematic when you want to use a polymorphic class as a workspace
variable because linking make_auto_md_cc then requires linking the vtable
which in turn requires definitions of all virtual functions. If now one of those functions calls an agenda,
i.e. depends on auto_md.cc, you end up with a dependency loop that cannot be satisfied.

Breaking this dependency cycle would also help to avoid rebuilding workspace_ng.cc and
thus auto_md.* every time a header for a group is changed and thus help to avoid
unnecessary recompilations. This should help to speed up compilation quite a bit.

Proposal

I think the solution is to modify the auto_workspace.h to separate the definition of
the WorkspaceMemoryHandler interface from the implemetation of the (de-)allocation
functions.

The WorkspaceMemoryHandler could then become a global object which is initialized
within main.cc.

@stefanbuehler As far as I understand, Fig 5.2 (and then also Eq 5.1) does not match the implementation. The weights are stored in another order. For example, the first and last weights are at opposite corners, while the figure places them along one side.

It would also be good to mark in Fig 5.2 what is the row and column dimension.

If problematic to update the figure, maybe just clarify that the description is schematic.

That said I wished that all parts of ARTS were as well documented as the implementation of tensors and interpolation.

/P

This is a bug I have been trying to fix this afternoon but gave up before meandring home. I need it fixed in the newlinerecord branch because a lot of functions demand being able to set the quantum numbers correctly. It seems python String interpretation is not the same as arts own interpretation of String. Because this is just odd otherwise. I really cannot understand the bug. I don't even always get any "gdb" output saying it touched any arts file. Even one time, I got the final output to just say "H2O2", which is even more scary...

I am unsure this should be here or under typhon. It seems a lot can be improved in the interface to deal with things not being converted to other types as easy as they should be (e.g., int is not understood under python to also be float when required). Perhaps this is another instance of limited conversion routines?

Describe the bug
There is an error with the input towards QuantumIdentifierSet running under python interface.

To Reproduce
Steps to reproduce the behavior:

1. Run in a .arts file:

Arts2 {
QuantumIdentifierCreate(test)
QuantumIdentifierSet(out=test, string_initializer="O2-66 TR UP v1 1 LO v1 1")
Print(test)
}

2. Run in *.py file

from typhon.arts.workspace import Workspace
arts = Workspace()
arts.QuantumIdentifierCreate("test")
arts.QuantumIdentifierSet(out=arts.test, string_initializer="O2-66 TR UP v1 1 LO v1 1")
arts.Print(arts.test)

If applicable, error output:

In arts, the output is:

Executing ARTS.
Command line:
arts test.arts 
Version: arts-2.3.1277 (git: 2b0d7522)
Executing Arts
{
- QuantumIdentifierCreate
- QuantumIdentifierSet
- Print
O2-66 TR UP v1 1 LO v1 1
}
This run took 0.05s (0.13s CPU time)

In python the output is:

Loading ARTS API from: /home/richard/Work/arts/build/src/libarts_api.so
Segmentation fault (core dumped)

Expected behavior

At least print the correct output!

System info (please complete the following information):
Tried on a ubuntu 18.04 system, which complains about free() under the python code, and under a ubuntu 19.10 system, which gives the complain above.

Describe the bug
The way to define covmat_sx in ARTS that I am aware of is two-fold. You either define a covmat_block as a Sparse larger than (0, 0) and call your choice of retrievalAdd* or you add a block matrix using covmat_sxAddBlock before retrievalDefClose. The latter works but the former does not for retrievalAddFreqShift.

To Reproduce
Run this code changing OK to False and the error is seen.

If applicable, controlfile code:

from typhon.arts.workspace import Workspace
import scipy as sp
import numpy as np


OK = True

arts = Workspace(0)
arts.retrievalDefInit()
arts.f_grid = np.linspace(1e9, 2e9, 3)

if not OK:
    arts.covmat_block = sp.sparse.csc.csc_matrix(1e10*np.diag(np.ones((1))))

arts.retrievalAddFreqShift(df = 50e3)

if OK:
    arts.covmat_sxAddBlock( block = sp.sparse.csc.csc_matrix(1e10*np.diag(np.ones((1)))) )

arts.retrievalDefClose()

If applicable, error output:

Exception: Call to ARTS WSM retrievalDefClose failed with error: *covmat_sx* does not contain a diagonal block for each retrieval quantity in the Jacobian.
 Fails test (!covmat_sx.has_diagonal_blocks(ji_t)) for ji_t   0   0

Expected behavior
Both methods should work as when OK=True.

This Jacobian is now zero for zero wind? Formerly correct? If not, find another way to do the calculations. If yes, find a practical workaround.

We have now
scat_dataCalc
scat_data_checkedCalc
Very hard to understand the difference. They should be merged.

What do we do with all errors and warning caused by bad normalisation of the phase function?

Do we have spherical harmonics soon enough so we can forget this?
Or shall we have automatic rescaling instead of errors and warnings?

I am unable to edit and submit this but there is a misspelled name in the "branches" link

Expected name: "Stable branches"

Actual name: "Stale branches"

Make some checks and when done make this iyActive

Once DOIT has been updated (#207), a lot of the old RT code becomes obsolete. I will have a look at removing it. Please ping me @m-brath when you are done with #207 .

@riclarsson I made a test with the latest version of second order integration in iyEmissionStandard. I got basically identical results with first and second order integration. I assume this means that the max criterion has the result that the algorithm falls back to first order more or less for each RT step. Maybe this is what you meant in some comment you made. Anyhow, as it is now second order ends up pretty much useless as it does not improve accuracy, but still use more calculations.

But was it really necessary to introduce the max criterion? My memory is that the large deviation for y I reported about did not involve I, but the later Stokes elements. Maybe my memory is wrong, but maybe there was just a problem for higher stokes_dim and polarised situations?

With the OEM module in place, I think it would be a good idea to have the ability to process raw-data in ARTS to generate lists of y and covmat_se/covmat_seps from some raw-time series. (The latter does not exist today, since covmat_se is defined to contain it.)

Is this an idea accepted by others? Or do you want to keep rawdata processing out of ARTS still? I think it fits to have it given the addition of OEM.

My design of this would be very simply in the beginning but could of course evolve if it helps other observation schemes. An agenda would be created, called rawdata_agenda. This agenda outputs a ybatch, an ArrayOfCovarianceMatrix covmat_sepsbatch, and an ArrayOfIndex ybatch_count. It takes only a Matrix rawdata as input. The methods inside this agenda could then require more information.

In the original implementation I intend to use this for I have a simple repeating cold-atm-hot-atm measurement cycle. The method for generating ybatch from this would be called ybatchFromRawdataCAHA and require an additional hot and cold temperature Vector input. (Just to keep the start of this simple.) Additionally, a TimeAverageBatchCalc method would be created that takes a time Vector and a time_target Numeric to reduce the size of ybatch and generate ybatch_count and covmat_sepsbatch at the correct sizes.

Problem one is that Workspace outputs text despite being asked not to output text.

Code example:

from typhon.arts.workspace import Workspace, arts_agenda

x = Workspace(0)

Expected output to screen: None

Output to screen: verbosityInit

/////////////////////////////////////////////////////////////////////////////////////////////////

Problem two is that arts_agenda also outputs text to screen

Code:

from typhon.arts.workspace import Workspace, arts_agenda

def f(arts):
    arts.Print("HELLO")

x = arts_agenda(f)

Expected output to screen: None

Output to screen: verbosityInit

This bug persist even if Workspace(0) has been called prior to arts_agenda(f)

/////////////////////////////////////////////////////////////////////////////////////////////////

Problem three is something I have not found out where it is. It has to do with Copy also being output to the screen.

Hi,

When looking in m_cloudbox for a method, I noticed some methods that got me to think. There are still some unclarities around scat_species.

First of all, it used in the new PSD system. Is scat_species a good name here? Should it be renamed to something like psd_id_string?

Further, the description of scat_species needs to be updated.

There are also some old methods that use scat_species:
ScatElementsSelect
ScatSpeciesExtendTemperature
ScatSpeciesMerge
Does that make sense now?

When looking at this, also these methods should be revised:
particle_fieldCleanup
ScatSpeciesInit
ScatElementsPndAndScatAdd
ScatSpeciesPndAndScatAdd
ScatElementsToabs_speciesAdd
ScatSpeciesScatAndMetaRead

Do we need all these now? Good naming of the WSMs?

Richard found an interesting C++ library:

https://github.com/usnistgov/SCATMECH

The library could be useful in several ways. It would give us Mie and Rayleigh code. It also contains T-matrix (seems to be well documented) that potentially could free us from the fortran T-matrix code. In a longer perspective, it seems very useful for setting up models for the surface.

Not urgent now, but Oliver please take a look. Are there any showstoppers?

Give clearer instructions on how to install missing Python packages. Especially avoid confusion about the package providing the lark module is called lark-parser on PyPi.

Since scat_species_XXX_field have been removed, cloudbox_limits_old is probably not needed anymore in cloudboxSetAutomatically.

Describe the bug
After creating an ARTS variable with the name "np", .value cannot be called anymore for any variables that map to numpy types.

To Reproduce

import pyarts

ws = pyarts.workspace.Workspace()
ws.VectorSetConstant(ws.f_grid, 10, 1.)

print("OUT1: ", ws.f_grid.value)
ws.VectorCreate("np")
print("OUT2: ", ws.f_grid.value)

If applicable, error output:

OUT1:  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
Traceback (most recent call last):
  File "test_value.py", line 8, in <module>
    print("OUT2: ", ws.f_grid.value)
  File "/home/olemke/anaconda3/lib/python3.8/site-packages/pyarts/workspace/variables.py", line 357, in value
    return np.asarray(self)
AttributeError: 'WorkspaceVariable' object has no attribute 'asarray'

Expected behavior
Defining an ARTS Variable with the name "np" should not break access to the .value method or the *Create methods need to catch illegal names.

System info (please complete the following information):

• OS: Linux/macOS
• Version 2.5.0

The following test currently fail because they use the obsolete abs_lineshapeDefine:

 80 - arts.ctlfile.nocheck.instruments.hirs.TestHIRS_reference (Failed)
 82 - arts.ctlfile.nocheck.instruments.seviri.TestSEVIRI (Failed)
 83 - arts.ctlfile.nocheck.instruments.mviri.TestMVIRI (Failed)

To Reproduce
Run ctest

Expected behavior
Tests shouldn't fail

Hi,

The main point of this post is to summarize a conversation I had with @erikssonpatrick about the use of full models via continua.cc in ARTS. We both encountered problems where the continua.cc solution was sub-optimal because we had to change values. There is a need to change how these continua are done.

We have many models in continua.cc using line absorption and true continua absorption interchangeably to give full absorption model code. This comes with several problems and limitations. Not only is the continua.cc code difficult to follow but some questions about its reliability is also difficult to answer, for example:

1. Does the true continua really match the absorption lines when either lines or continua is updated (we have done this but I don't know if we have checked)?
2. How do we estimate the model errors when everything is hard-coded (we cannot today)?
3. How do we append other absorption lines while making sure we do not accidentally add a line that is in the hard-coded model (we need quantum number encoding)?

The advantage is that at least some of the full models are well-tested and known to work. They are the 'industry standard', but we do not have clear ways to improve them and to update them to newer models as new papers come out. The continua.cc code is just too messy and difficult to work with.

I implemented two solutions recently. One was to hard-code an absorption line model external to continua.cc via a method so that all the model parameters' derivatives can be computed and thus their errors understood. The other is to adopt AbsorptionLines to use the full model's parameters (which already can propagate all the errors). @erikssonpatrick informed me he has recently done the latter as well for a different project. The differences between our approaches is that I enforced the same temperature behavior as the full model whereas he manipulated the standard line parameters to match the model. My solution is, of course, more in line with the intent of the full models, but it comes with the cost of complication. Essentially, I have introduced a new way to compute the line strength and thus a new way to deal with LTE radiative transfer into our line shape calculations. We need to allow not just one but many definitions of the line strength for this to work, since the temperature dependency is not clear otherwise.

I am sure one of these three solutions is the way forward for ARTS. We need to separate the full models - in a method that is easy to document - from the amalgamation of different physics that makes using the true continua models necessary. Since we want to be able to describe the physics via our own line catalogs, we need to be able to describe these continua in a way that can be matched to the line catalogs. Any thoughts on how to go forward here?

My personal view is that hard-coded non-continua.cc models are the best way to do this for now. It is clear what you are doing when you include a model, and it is clear what the individual model is doing. We then might want to fit these to the ARTS absorption line parameters, or compute our own parameters from the original data, but at least we have the original model to compare towards, and we can then easily allow others to use said model.

For now creates the issue just to have something to cite in milestone fr ARTS3

Is it possible to get IBA to use another atmosphere for downwelling radiation?

A possible solution could be to let all atmospheric fields by input to the surface agendas.

For some reason, nlte_field is already this. If the solution above not used, investigate if this can be removed.

This issue keeps track of fixing the issues in commit 08de9d4. If the problems cannot be resolved, the code needs to be removed again at some point.

I have said this before but was advised to open an issue here.

In short

Our current LineRecord contains a lot of copies between lines that are not line specific but absorption band or species specific. Additionally, because of the variety in the data a LineRecord can contain, it is not possible to store and read an ArrayOfLineRecord reliably and efficiently as binary data. We can address this by making an ArrayOfLineRecord class to store the metadata and size information of all the lines in the array.

In more details: size

We currently have global quantum numbers and local quantum numbers in the same variable. There are 32 quantum numbers stored per level in a LineRecord. This means 1024 bits per line is stored in RAM. Less than 10 of these numbers are not band specific, so at least 700 bits can be saved per line if we make the separation and can throw away the global numbers.

We currently store line shape information, that is the way to compute the parameters required for every line, with independent species information. If we can guarantee that the line shape calculations are the same for every line in an ArrayOfLineRecord, several optimizations are possible in the code. Also, the broadening metadata information is stored as an Index describing the type, two bools describing if self and air broadening is present, and an ArrayOfSpeciesTag. A single SpeciesTag contains 64 bits of information, and the average size is 2 SpeciesTag per line (for air and self broadening). This means that every line could store about 144 bits less of information if the broadening data was made global.

Additionally, the species and isotopologue is stored per LineRecord but could be made global.
The line shape normalization, the line shape mirroring, the line shape cutoff frequency, the line shape line mixing pressure limit, the line shape population type, the reference temperature of the lines, and the LineRecord version are also stored per LineRecord but could arguably be made global. This is an additional 72 bits of information less per line that could be stored globally once.

In total, per LineRecord, we could store something like 900 bits less. This is a significant part of the size of a LineRecord, which today is about 1904 bits large in average. So the storage gains would be good.

In more details: storage

Another advantage of this change would be that the entire ArrayOfLineRecord would have a predictable size. This means that each LineRecord can be read and stored to binary files. This could give a large increase in speed of reading and writing larger line databases for multiple uses.

The tag of such an XML file would look something like:

<ArrayOfLineRecord version="0" nlines="40" species="O2-66" broadeningspecies="SELF N2 H2O CO2 H2 He BATH" lineshape="VP" mirroringtype="LP" zeemaneffect="true" normalizationtype="VVH" cutofflimit="810e9" linemixinglimit="1e0" populationdistribution="LTE">

This might look like a beast, but it really does not change much from how the data looks like today even if we store the ArrayOfLineRecord with just a single line in ascii form. All the sizes should be made clear from the tag, so that each line is known to be the same number of bits large. This allows fast and efficient binary IO.

Additional benefits: code readability

Lastly, another advantage is that ArrayOfLineRecord specific optimizations in the line-by-line code could be made.

One such optimization is that the partition function today has to checked for every absorption line so that the isotoplogue and reference temperature has not been changed from the last one. If this is instead just computed once we have better predictability of the code, since the partition function is then going to be constant.

The same can be said about the line shape volume mixing vector. If this is known to be constant the first time it is computed, the code checking that it is the same can be removed.

Also, the Doppler broadening would be known from the beginning of the cross-section calculations.

These changes would likely not affect the speed of the code execution by much, but making as many things as possible constant expressions helps greatly with readability. And the fewer non-constants we have, the easier it will be to ensure that parallel code executes efficiently.

Only keep working files. Merge each of them into a single controlfile, if feasible.

If somebody needs the full planetary toolbox, it can be found in 2.2

I think we should distribute spectral line catalogue data together with ARTS.

As baseline, perhaps two catalogues, the one by Agnes Perrin, to be used for planetary calculations up to 3THz. And one based on the latest HITRAN edition. (These two are already available, with the caveat that the HITRAN one is based on the 2012 version, so not exactly new.)

The format right now is ARTSCAT-5, but for the release it could be ARTSCAT-6, the version that is split by isotopologue and has all parameters that are common for the entire file at the beginning of the file.

There should be documentation of the format, perhaps best directly with the catalogue files (with a short note in the userguide that it can be found there)

I wonder whether subversion is the best system for such data. The current size of the split HITRAN catalogue is 200mb (each file is zipped). Subversion style version control does not seem to be terribly useful for this kind of data.

The goal is to have a simple short step-by-step list for new users to get up and running LBL calculations.

We could also consider Simon's suggestion to have a WSM that pulls the catalogue from an online location, storing a local cash. This could be done on the species (in ARTSCAT-6 isotopologue) level, then the individual files are quite small.

But if we start such a system, wouldn't it be tempting to do the same for other arts xml data?

Describe the bug
The python agendas sets assertion instead of showing runtime error. This makes setting up a runfile difficult

To Reproduce

import pyarts
@pyarts.workspace.arts_agenda
def abs_xsec_agenda(ws):
    pass
ws = pyarts.workspace.Workspace()
ws.abs_xsec_agenda = abs_xsec_agenda

Terminal output

terminate called after throwing an instance of 'std::runtime_error'
  what():  The agenda abs_xsec_agenda must generate the output WSV abs_xsec_per_species,
but it does not. It only generates:

Aborted (core dumped)

Expected behavior
Show the error instead of calling terminate, crashing the process (terminal output is not shown in spyder, so external software must be used to find the cause of the error)

System info (please complete the following information):

• OS: Ubuntu 20.04

When I run ctest -R from the command line, also the python "version" is launched. Should it work like that? A bit annoying that it ends with a fail when all is OK as far as I am concerned.

Here is an example:

cirrus|build: ctest -R TestAntenna2D
Test project /home/patrick/GIT/ARTS/arts/build
Start 130: arts.ctlfile.fast.artscomponents.antenna.TestAntenna2D
1/2 Test #130: arts.ctlfile.fast.artscomponents.antenna.TestAntenna2D .......... Passed 3.20 sec
Start 131: python.arts.ctlfile.fast.artscomponents.antenna.TestAntenna2D
2/2 Test #131: python.arts.ctlfile.fast.artscomponents.antenna.TestAntenna2D ...***Failed 0.02 sec
/usr/bin/python3: can't open file 'controlfiles/artscomponents/antenna/TestAntenna2D.py': [Errno 2] No such file or directory

50% tests passed, 1 tests failed out of 2

Total Test time (real) = 3.22 sec

The following tests FAILED:
131 - python.arts.ctlfile.fast.artscomponents.antenna.TestAntenna2D (Failed)
Errors while running CTest

Add the possibility to add description to workspace groups for the online documentation. Should follow the same implementation as WSVs.

As far as I can see, we no longer need the perturbation option for Jacobians.

I will remove the methods and functions directly related to this. If time allows, I will also check out if this allows removing arguments from some of the agendas.

Richard: I leave to you to decide if the internal bookkeeping shall be cleaned up. Now all Jacobians will be considered as analytical, affecting e.g. the FOR_ANALYTICAL_JACOBIANS_DO macro.

If no isssues arise on thunder or mistral, we switch to C++17 as default.

A workshop to celebrate the first 20 years. Also ARTS-3?

The selection of absorption models in xsec_continuum_tag in continua.cc needs improving. It's convoluted and has performance issues due to lots of string comparisons.

There is a replacement of iyHybrid using the new interface. If this is merged as the default, get_diydx in jacobian.h/jacobian.cc can be removed.

I made the interface but @erikssonpatrick told me he or someone else would setup a test. I no longer remember the state of this but would like to remove get_diydx since I had a minor look at jacobian.h/cc today.

Anyways, what is the status? I think a decision needs to be made before ARTS 3.

We have now two systems for describing PSDs. Only the new system supports retrievals and the old shall be removed. Things to be done:

• Clean up the code
• Remove or revise instructions in developer guide
• Revise or make new checks for scattering solvers

Right now abs_linesReadSpeciesSplitCatalog just returns empty lines if no catalogue files are found (for example if the path is mistyped).

I think it should trow a runtime error if no file exists for a species tag that is a line-by-line species. (This type of silent failure leads to bugs that are hard to find.)

Optionally, in case anyone needs the behaviour of silent failure for a script, there could be an input argument "robust" with default 0, that triggers the silent behaviour if set to 1. This would be consistent with DOBatchCalc.

Describe the bug
Cannot run through TestOEM in accepted build configuration.

To Reproduce
Install ubuntu 20.04 or upgrade to ubuntu 20.04. Install dependencies. "make check"

Expected behavior
Works within reasonable time.

System info (please complete the following information):

• OS: Ubuntu 20.04 base
• Version: Latest

Additional context
Switching to libopenblas from liblapack3 and libblas3 made the code run. (Thanks Oliver!)

Suggested solution
Check for liblapack3 and libblas3 and give appropriate warning. (I am not good enough at cmake to make this check)

Have a quick look at what parts of AUG to remove.
First step is to initiate the review.

Remove or "activate" cfiles in /controlfiles that are not used in any test.
See also #211