The available slurm resource control we provide is restrictive and doesn't provide the capability we need. We should just pass slurm_args as a tuple and splat them into the sbatch script.

This will mainly be useful for single-column experiments, where the precompilation time is a significant proportion of the overall model runtime.

Since this package will be used for other component models than Atmos, we should rename it to reflect this. This will be straightforward because the package is not registered.

The simplest name would be CalibrateModel.jl. Ollie also suggested ClimaCalibrate

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Set up the basic framework for the repo, github actions, the future Julia package, and project tomls.

We should upgrade sooner rather than later, before more 1.8.5 infrastructure is added.
We need to wait for Atmos to update first.

Could I be a kill-joy here are suggest the name CalibrateClimaAtmos because I expect folks may use this repo as somewhat of a template in future for calibration pipelines too; so starting a naming convention that scales would be a good idea.

Also i hate it ;)

Once ClimaAtmos#2341 is finalized and merged, we can add full GPU support.
This will just require setting proper sbatch arguments for the driver, pipeline.sh. We could do this quickly by adding an additional GPU driver, but I would rather have a unified interface.
I am planning on doing this by introducing a proper command line interface, which will also allow a simpler interface for running experiments.

We will need to :

• determine the correct GPU settings. This will be quick.
• Add an argument parser and change the way the pipeline takes in configurations. This will be the bulk of the work, but won't be too complicated because the pipeline is still quite simple.

This addresses the following issue of the O3.1.1 Milestone:

CliMA/ClimaCoupler.jl#638

Components

• implement the SF perfect example, using Businger parameters
• use (and modify where needed) the CalibrateAtmos interface
• show convergence of parameters
• debug insensitivity (better observation data needed)
• ensure the test still runs using the bash scripts
• adapt Atmos code for the new interface, ensure it runs as before

Convergence (a_m and a_b params)

notes / TODOs

• SF - Why are some Businger params still hard coded? - fixed in CliMA/SurfaceFluxes.jl#165
• pipeline.jl runs without bash scripts and was here for debugging. As part of a separate PR, we can generalize it, so a similar script can be used by other lightweight models. For the heavyweight models, we may be able to wrap the bash / SLURM script execution into Julia's pipeline functions, so we can call the scripts from the model repos.
• we should use the saved output to retrieve ustar in the plotting script

There is no real mechanism to handle failures in the pipeline (particularly in forward model runs).
FailureHandler from EnsembleKalmanProcesses will be used in order to fill the model output with NaNs and properly perform the update step with missing values.
See SampleSuccGauss for more info.

Derecho uses PBS instead of slurm, so we will need to create a new job controller with the same funcitonality as the slurm controller.

Given the new atmosphere component release - can we relax this version requirement?

https://github.com/CliMA/CalibrateAtmos.jl/blob/e16d189eca5668629968a4cc5b12fdb5209793f6/experiments/sphere_held_suarez_rhoe_equilmoist/Project.toml#L16

This might be a bit confusing, separate into 2 functions?

https://github.com/CliMA/CalibrateAtmos.jl/blob/cbb51f82bf92770eaf2363f79d2310b779d84f0c/src/emulate_sample.jl#L86C12-L86C33

We don't export anything, we should export useful methods/types:

An initial list:

• CaltechHPC
• calibrate
• ExperimentConfig
• get_prior
• kwargs

Purpose

The goal is to have calibration scripts for EDMF in ClimaAtmos.jl. This is an equivalent of https://github.com/CliMA/CalibrateEDMF.jl for the standalone EDMF repository.

Cost/Benefits/Risks

Cost/Risk: The initial cost and risk are minimal. We will be mostly porting over and adapting the scripts from CalibrateEDMF.jl

Benefits: We need a calibration pipeline for EDMF in the ClimaAtmos model. The EDMF code is being refactored. It would be extremely helpful to be able to calibrate the refactored code to see if we can recover the current model results.

People and Personnel

(@ the personnel required/requested to implement the software solution)

• Lead: TBD
• Collaborators: @trontrytel , @odunbar, @costachris, @nefrathenrici
• Reviewers: @haakon-e, @szy21

Components

(Describe the main components of the software solution)

Inputs

(Describe the inputs to the software solution)

Results and deliverables

• Calibrate a ClimaAtmos.jl SCM EDMF Bomex case (one of the cases we run in the CI) with the data from the standalone TurbulenceConvection.jl model.
• Add more cases to calibration pipeline.
• Create a semi-automatic way to trigger calibrations. (This would help with the code refactoring of EDMF)

Task Breakdown And Tentative Due Date

(A preliminary list of PRs and a preliminary timeline of PRs, milestones, and key results)

• Retire the namelist interface of EDMF in ClimaAtmos and move to toml and config file structure. See the discussion here: CliMA/ClimaAtmos.jl#1259, PR CliMA/ClimaParams.jl#103 (@nefrathenrici , @trontrytel)
• #2
• Port over the CalibrateEDMF scripts using the toml files (@costachris , @haakon-e)
• Modify CalibrateEDMF framework to read hdf5 output from CA (instead of TC.jl netCDF stats files).
• Review the calibration workflow. Suggest tests and default settings, (@odunbar)

Proposed Delivery Date

(Your best estimate of the date this software solution will be completed)

Likely need to set JULIA_CPU_TARGET: 'broadwell;skylake'

ERROR: Unable to find compatible target in system image.
Stacktrace:
 [1] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
   @ Base ./loading.jl:1052
 [2] _tryrequire_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String)
   @ Base ./loading.jl:1442
 [3] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1816
 [4] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [5] macro expansion
   @ ./loading.jl:1648 [inlined]
 [6] macro expansion
   @ ./lock.jl:267 [inlined]
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611

Specific Issue

PRs such as this one in ClimaAtmos.jl need to call EKP directly for use due to newer features/uses as they do not exist in ClimaCalibrate.jl
CliMA/ClimaAtmos.jl#3223 (comment)

Specific solution

Add interfaces to EKP Observation/ObservationSeries objects and minibatch calls into CAL.

More general issue

At present one can't interface at all with EKP, the idea being that everything is called within ClimaCalibrate objects.

• this means one shouldn't need to directly load EKP into their package. But all of the options/settable objects are EKP objects so this happens anyway.
  E.g. the CAL.initialize()

import ClimaCalibrate as CAL
import EnsembleKalmanProcesses as EKP
CAL.initialize(
    ensemble_size,
    observations,
    prior,
    output_dir,
    process = EKP.Inversion(); # <- need EKP
    scheduler = EKP.DefaultScheduler(eki_timestep), # <- ...
    localization = EKP.Localizers.SECNice() # <- ...
    # abc = EKP.xyz() <- ...
)

More general solution

Can we allow users to configure their own EKP and then pass this into the ClimaCalibrate wrapper?
i.e. should our guiding principle be that ClimaCalibrate should not depend on the EKP configuration in order to work

• this has the benefit that there is lower maintenance on EKP features / objects.
  This may be acheivable by simply adding extra methods to take in the EKP objects, so you do not "have" to use the wrapper.

For example, in your e.g. ClimaAtmos script

import EnsembleKalmanProcesses as EKP

initial_ensemble = EKP.construct_initial_ensemble(rng_ekp, prior, ensemble_size)
ekp_obj = EnsembleKalmanProcess( 
    intial_ensemble,
    observation,
    process = EKP.Inversion();
    scheduler = EKP.DefaultScheduler(eki_timestep), 
    localization = EKP.Localizers.SECNice(),
)

Then we can pass this into ClimaCalibrate so that it doesn't need to know anything about the EKP configuration

import ClimaCalibrate as CAL
CAL.initialize(ekp_obj, output_dir, prior) # 

where we have added a new initialize which

function initialize(ekp_obj, output_dir, prior) 
    save_eki_state(eki, output_dir, 0, prior)
    return eki
end

CES causes issues with precompilation due to its use of Conda and PyCall. I plan to remove these dependencies in the source package, but a quick fix is to only conditionally load CES.

The Climate Modeling Alliance

Software Design Issue 📜

Purpose

We will implement a pipeline to calibrate on the moist held-suarez experiment in ClimaAtmos. We will calibrate on the horizontal temperature gradient.
Once this is complete, we will generalize to a pipeline that can handle other cases.

Cost/Benefits/Risks

Primary risk is developer time, but this will save us a lot of time for manual calibration in the long run.

People and Personnel

Components

• EKP interface: Configure EKP, construct the prior distribution, update ensemble
• HPC interface: this should be GPU and CPU compatible, ideally running on the cluster and the a100 server.
• Atmos interface: Configure and run the atmosphere model.
• Data processing utilities: Needed for constructing the observation map
• Visualization utilities: Used for comparing model output and determining useful information for the observation map

Inputs

• EKP configuration: ensemble and iteration sizes
• Prior distribution TOML
• Observation map
• Steady-state restart file

Results and Deliverables

• Fully working and reproducible perfect-model calibration for moist held-suarez
• Simple experiment configuration interface with tests
• Inverse problem for the HS case

Task Breakdown And Schedule

• #5
• Create interface
• Moist HS observation map
• Demonstrate moist HS perfect model calibration

SDI Revision Log

CC

@tapios @simonbyrne @cmbengue

Issue

https://github.com/CliMA/ClimaCalibrate.jl/blob/61ae9262a6567983737b1196a894513d2def7074/src/ekp_interface.jl#L320C5-L320C37

Currently the EKP.update_ensemble!( has a return value too which is triggered by early termination (see here) due to adaptive timesteppers

Solution

Allow a return value
terminate = EKP.update_ensemble!(
and then use !isnothing(terminate) to break the update loop