climateimpactlab / dodola

Containerized application for running individual tasks in a larger, orchestrated CMIP6 bias-adjustment and downscaling workflow.

dodola

Containerized application for running individual tasks in a larger, orchestrated CMIP6 bias-adjustment and downscaling workflow.

This is under heavy development.

Features

Commands can be run through the command line with dodola <command>.

Commands:
    adjust-maximum-precipitation  Adjust maximum precipitation in a dataset
    apply-dtr-floor               Apply a floor to diurnal temperature...
    apply-non-polar-dtr-ceiling   Apply a ceiling to diurnal temperature...
    apply-qdm                     Adjust simulation year with quantile...
    apply-qplad                   Adjust (downscale) simulation year with...
    cleancmip6                    Clean up and standardize GCM
    correct-wetday-frequency      Correct wet day frequency in a dataset
    get-attrs                     Get attrs from data
    prime-qdm-output-zarrstore    Prime a Zarr Store for regionally-written...
    prime-qplad-output-zarrstore  Prime a Zarr Store for regionally-written...
    rechunk                       Rechunk Zarr store in memory.
    regrid                        Spatially regrid a Zarr Store in memory
    removeleapdays                Remove leap days and update calendar
    train-qdm                     Train quantile delta mapping (QDM)
    train-qplad                   Train Quantile-Preserving, Localized...
    validate-dataset              Validate a CMIP6, bias corrected or...

See dodola --help or dodola <command> --help for more information.

Example

From the command line, run one of the downscaling workflow's validation steps with:

dodola validate-dataset "gs://your/climate/data.zarr" \
  --variable "tasmax" \
  --data-type "downscaled" \
  -t "historical"

The service used by this command can be called directly from a Python session or script

import dodola.services

dodola.services.validate(
    "gs://your/climate/data.zarr", 
    "tasmax",
    data_type="downscaled",
    time_period="historical",
)

Installation

dodola is generally run from within a container. dodola container images are currently hosted at ghcr.io/climateimpactlab/dodola.

Alternatively, you can install a bleeding-edge version of the application and access the command-line interface or Python API with pip:

pip install git+https://github.com/ClimateImpactLab/dodola

Because there are many compiled dependencies we recommend installing dodola and its dependencies within a conda virtual environment. Dependencies used in the container to create its conda environment are in ./environment.yaml.

Support

Additional technical documentation is available online at https://climateimpactlab.github.io/dodola/.

Source code is available online at https://github.com/ClimateImpactLab/dodola. This software is Open Source and available under the Apache License, Version 2.0.

dodola's Issues

add analog method to downscaling service

currently the only downscaling method supported is the delta shift spatial disaggregation method. Need to add an analog-inspired method

bug in QDM bias correction service

Currently there is a bug in the .train step in QDM, the line

model.train(gcm_training_ds[train_variable], obs_training_ds[train_variable])

has ref (obs) and hist (GCM historical) swapped, it should be

model.train(obs_training_ds[train_variable], gcm_training_ds[train_variable])

Drop rechunker for simple xarray rechunk?

The rechunker package has had some inconsistencies and stability issues handling different input Zarr stores. Some require more love and care than others.

I've been having much more success/stability with very basic xarray v0.17.0 rechunking:

import os
import xarray as xr
from adlfs import AzureBlobFileSystem

fs = AzureBlobFileSystem()
ds = xr.open_zarr(fs.get_mapper(os.environ.get("IN_ZARR")))

ds=ds.chunk({
        "time": int(os.environ.get("TIME_CHUNK")),
        "lat": int(os.environ.get("LAT_CHUNK")),
        "lon": int(os.environ.get("LON_CHUNK")),
    }
)
del ds[os.environ.get("VARIABLE")].encoding["chunks"]

ds.to_zarr(fs.get_mapper(os.environ.get("OUT_ZARR")), mode="w")

This would be more memory-conservative if we use chunks in xr.open_dataset(..., chunks={...}, engine="zarr") rather than the Dataset.chunk() method -- which will read in everything at once. <- Not sure this is faster when we have large memory available, like we do.

Should swap that in and dump the rechunk dependency and see how it goes. If we have certain cases that work better with rechunker (larger data), then consider adding it as an option.

fix dimension naming in regridding service

Currently the regridded output from the regridding service outputs a zarr store that has 2-d latitude and longitude named (y, x) for a 1-degree regular grid that has been regridded bilinearly. This naming is something that xesmf does during the regridding. This is a problem for other services (e.g. rechunker) that requires lat/lon dimensions not to be named y, x, so it needs to be cleaned up before writing out.

CI (GH actions) workflow not triggered on pull request.

Our currentCI workflow only triggers on push events. We need it to also trigger on pull request to main.

The addition should look something like this.

on:
  push:
    branches: "*"
  pull_request:
    branches: "main"

Should be an easy fix. We only triggered on push events because CI was just getting set up -- so we don't need to worry about breaking legacy behavior.

Stumbled across this in PR #2.

bug in rechunking service

When I run the following: dodola rechunk clean-dev/ACCESS-ESM1-5.historical.1x1.zarr -v tasmax -c time=60225 -c lat=20 -c lon=20 --maxmemory 113004000 --out clean-dev/ACCESS-ESM1-5.historical.1x1.rechunked.zarr, I'm getting this error:

Traceback (most recent call last):
  File "/anaconda/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola==0.1.0a0', 'console_scripts', 'dodola')())
  File "/anaconda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/anaconda/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/anaconda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/anaconda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/anaconda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/anaconda/lib/python3.8/site-packages/dodola-0.1.0a0-py3.8.egg/dodola/cli.py", line 105, in rechunk
    coord_chunks = {c.split("=")[0]: int(c.split("=")[1]) for c in chunk}
  File "/anaconda/lib/python3.8/site-packages/dodola-0.1.0a0-py3.8.egg/dodola/cli.py", line 105, in <dictcomp>
    coord_chunks = {c.split("=")[0]: int(c.split("=")[1]) for c in chunk}
IndexError: list index out of range

Add components for quantile delta mapping (QDM) bias correction

We have fragments of bias correction logic in dodola. Lots of the new development has been prototyped into an argo workflow template.

Where we're able, we should turn these components into commands for dodola. For example:

• find-qdm-window - Given a zarr store with a simulation, find the min and max years window that we can adjust with a QDM model.
• train-qdm - Given a reference and historical simulation zarr store, train a QDM model and output the model to a zarr store. Should accept kind="+" (for temperature) or kind="*" (for precip).
• apply-qdm - Apply a trained QDM model to a given simulation, for a given year.

The tricky bit is that QDM bias correction can't happen entirely within one or two dodola steps. We rely on some cloud-specific infrastructure to to reliably fan-out and fan-in when we apply the QDM model adjustment for a year and then collect those yearly results into a single output.

add CMIP6 cleanup service

We need to create a service to clean up raw CMIP6 files.

The service should:

• ingest CMIP6 raw models (as they are chunked from the Pangeo archive, in time) as zarr stores
• remove additional unneeded dimensions, e.g. height, member_id, time_bnds
• if leap days are present in the climate model, remove leap days and update the calendar to noleap
  compute with an optional pre-existing regridding weights file
• save the cleaned up model as a zarr store

ZeroDivisionError error from maxmem in `dodola rechunk`?

Getting an traceback from dodola rechunk with ZeroDivisionError from the rechunker package (traceback below).

It appears that max_mem arg to rechunker.rechunk() might require an int. It does not take a str, as advertised. Need to confirm this. If this is the case, we should correct our CLI help and docstrs to reflect that it requires an int. Be sure to cast input to int in dodola/cli.py. Most importantly, remember to notify the rechunker folk about this.

Traceback (most recent call last):
  File "/opt/conda/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/dodola/dodola/cli.py", line 110, in rechunk
    services.rechunk(
  File "/opt/dodola/dodola/services.py", line 20, in service_logger
    func(*args, **kwargs)
  File "/opt/dodola/dodola/services.py", line 117, in rechunk
    plan = rechunker_rechunk(
  File "/opt/conda/lib/python3.8/site-packages/rechunker/api.py", line 296, in rechunk
    copy_spec, intermediate, target = _setup_rechunk(
  File "/opt/conda/lib/python3.8/site-packages/rechunker/api.py", line 377, in _setup_rechunk
    copy_spec = _setup_array_rechunk(
  File "/opt/conda/lib/python3.8/site-packages/rechunker/api.py", line 482, in _setup_array_rechunk
    read_chunks, int_chunks, write_chunks = rechunking_plan(
  File "/opt/conda/lib/python3.8/site-packages/rechunker/algorithm.py", line 129, in rechunking_plan
    write_chunks = consolidate_chunks(shape, target_chunks, itemsize, max_mem)
  File "/opt/conda/lib/python3.8/site-packages/rechunker/algorithm.py", line 70, in consolidate_chunks
    headroom = max_mem // chunk_mem
ZeroDivisionError: integer division or modulo by zero

Get CI/CD to properly tag container images on release

CI/CD only pushes dev tags on commits to main. We need it to also properly tag containers with a version and push them on Github Releases or tagged commits or something.

Should take care of this before our first proper release.

Remember, Docker container tags don't have the v! So, v0.1.0 should be tagged on the container as 0.1.0.

regrid converts regridded output to float64

We use xesmf to regrid and I think it has been converting input data (which may be float32) to float64 (a la JiaweiZhuang/xESMF#66).

Most direct solution to this is to gather input dtypes and and cast output back after regridding. Or maybe allow this as an option...?

Either way, this is doubling the size of many output files unless you validate for the change in encoding dtypes.

add wet day frequency correction

Because of dry-day GCM biases and the mixed discrete-continuous nature of precip data, we need to add a wet day frequency correction to our downscaling workflow. This will involve two steps:

1. pre-processing: replace zeros in obs and model data with nonzero uniform random values below a trace threshold (probably we will set this to be 0.05 mm/day)
2. post-processing: all values in the bias corrected data below that threshold will be set to zero

@brews what's your opinion on this being one vs two services? I think it could just be one with a "pre-process" or "post-process" option argument, that seem reasonable to you?

update QDM implementation to include quantile information for use in analog-inspired downscaling

To get quantile information, we need to add the following to our QDM implementation:

with set_options(sdba_extra_output=True):
    QDM.train(ref, hist)
    biascorrected = QDM.adjust(sim)

Rechunk has unused "variable" argument

dodola rechunk has a -v/--variable required option that never gets passed to the actual rechunk service.

Unless rechunker.rechunk() requires a DataArray (and we're not giving it that), I think we can drop this "variable" option from the CLI.

Migrate service logging into a decorator

In dodola/services.py we explicitly log when the service starts and when the service stops. We can add some sugar to this by writing a simple decorator for each of the service functions. It was remember to send the logging messages on start and stop for us.

Zarr store read breaks with adlfs==0.7.0

Looks like the recent adlfs release v0.7.0 has broken zarr store streaming from Azure storage.

With adlfs v0.7.0 the recent dodola build errors on when opening zarr stores it could previously read (see output below).

A dodola-like local env appears to read and write the same zarr store with adlfs v0.6.3. The v0.7.0 bump then gives similar errors in this local environment to what we see below.

Should dig into this more and let the adlfs folk know.

The error:

Traceback (most recent call last):
  File "/opt/conda/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/dodola/dodola/cli.py", line 145, in regrid
    services.regrid(
  File "/opt/dodola/dodola/services.py", line 20, in service_logger
    func(*args, **kwargs)
  File "/opt/dodola/dodola/services.py", line 147, in regrid
    ds = storage.read(x)
  File "/opt/dodola/dodola/repository.py", line 51, in read
    x = open_zarr(self.get_mapper(url_or_path))
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/zarr.py", line 666, in open_zarr
    ds = open_dataset(
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/api.py", line 554, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/xarray/backends/zarr.py", line 324, in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/zarr/hierarchy.py", line 1188, in open_group
    return Group(store, read_only=read_only, cache_attrs=cache_attrs,
  File "/opt/conda/lib/python3.8/site-packages/zarr/hierarchy.py", line 120, in __init__
    meta = decode_group_metadata(meta_bytes)
  File "/opt/conda/lib/python3.8/site-packages/zarr/meta.py", line 104, in decode_group_metadata
    raise MetadataError('unsupported zarr format: %s' % zarr_format)
zarr.errors.MetadataError: unsupported zarr format: None

add precip unit conversion to CMIP6 cleaning

CMIP6 precip is in kg/m2/s, needs to be mm/day (e.g. multiply by 24 * 60 * 60). We should add this in to the standardize_gcm function, could be a flag similar to leapday removal.

AttributeError in /opt/dodola/dodola/core.py::standardize_gcm() from dodola cleancmip6

Caught this running

dodola cleancmip6 clean-dev/ACCESS-ESM1-5.historical.zarr \
    scratch/clean-cmip6-dev-wv4k5/historical-standardized.zarr

in an Argo workflow.

Here is the log:

clean-cmip6-dev-wv4k5-27916977: INFO:dodola.repository:Read clean-dev/ACCESS-ESM1-5.historical.zarr
clean-cmip6-dev-wv4k5-27916977: Traceback (most recent call last):
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/bin/dodola", line 33, in <module>
clean-cmip6-dev-wv4k5-27916977:     sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
clean-cmip6-dev-wv4k5-27916977:     return self.main(*args, **kwargs)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 782, in main
clean-cmip6-dev-wv4k5-27916977:     rv = self.invoke(ctx)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
clean-cmip6-dev-wv4k5-27916977:     return _process_result(sub_ctx.command.invoke(sub_ctx))
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
clean-cmip6-dev-wv4k5-27916977:     return ctx.invoke(self.callback, **ctx.params)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
clean-cmip6-dev-wv4k5-27916977:     return callback(*args, **kwargs)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/dodola/dodola/cli.py", line 64, in cleancmip6
clean-cmip6-dev-wv4k5-27916977:     services.clean_cmip6(x, out, drop_leapdays, storage=_authenticate_storage())
clean-cmip6-dev-wv4k5-27916977:   File "/opt/dodola/dodola/services.py", line 26, in service_logger
clean-cmip6-dev-wv4k5-27916977:     func(*args, **kwargs)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/dodola/dodola/services.py", line 183, in clean_cmip6
clean-cmip6-dev-wv4k5-27916977:     cleaned_ds = standardize_gcm(ds, leapday_removal)
clean-cmip6-dev-wv4k5-27916977:   File "/opt/dodola/dodola/core.py", line 146, in standardize_gcm
clean-cmip6-dev-wv4k5-27916977:     ds_cleaned = ds.isel.drop(dims_to_drop)
clean-cmip6-dev-wv4k5-27916977: AttributeError: 'function' object has no attribute 'drop'

Add basic numerical validation?

There is an interest in adding some kind of numerical validation, the kind that's manually called on zarr stores.

Initial ideas are for sanity checks... e.g. "Does the file have temperatures for the surface of the sun?", "Does the file have negative precipitation values?", etc.

Might be worth prototyping these checks in argo manually. Think about standardizing vocabulary here too -- there are many different kinds of "validation"...

Add dask-kubernetes, distributed, and better better subprocess handling in container

We could use dask-kubernetes and distributed as controlled versions in the container environment. These are more often used in prototype workflows and polishing.

Similarly, we need to improve the way (sub)processes are handled in the container as jobs are becoming more complex. Signals and general killing could be handled with more grace (i.e. just used tini). Want to keep things from hanging or becoming zombies.

Add CONTRIBUTING doc & outline release process. Remind brews to version setup.py?

Need to consider being more disciplined about updating the version information in setup.py if users are installing this with pip on the side. That said, this is still primarily an application to be run from a Docker container. Need to stay focused on that.

update buildweights with domain file option

The buildweights service also needs to be updated to accommodate a domain file in addition to a specified target resolution.

train-qdm gives ValueError: Expected index 'quantiles' to be PandasIndex

From ClimateImpactLab/downscaleCMIP6#122 (comment). A bias-correction workflow running QDM training with dodola:0.4.0 fails with ValueError: Expected index 'quantiles' to be PandasIndex.

The full log/error is

/opt/conda/lib/python3.9/site-packages/xarray/core/indexing.py:1379: PerformanceWarning: Slicing is producing a large chunk. To accept the large
chunk and silence this warning, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):
    ...     array[indexer]

To avoid creating the large chunks, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):
    ...     array[indexer]
  return self.array[key]
/opt/conda/lib/python3.9/site-packages/dask/array/core.py:4338: PerformanceWarning: Increasing number of chunks by factor of 18
  result = blockwise(
/opt/conda/lib/python3.9/site-packages/xarray/core/indexing.py:1379: PerformanceWarning: Slicing is producing a large chunk. To accept the large
chunk and silence this warning, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):
    ...     array[indexer]

To avoid creating the large chunks, set the option
    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):
    ...     array[indexer]
  return self.array[key]
/opt/conda/lib/python3.9/site-packages/dask/array/core.py:4338: PerformanceWarning: Increasing number of chunks by factor of 18
  result = blockwise(
Traceback (most recent call last):
  File "/opt/conda/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/opt/dodola/dodola/cli.py", line 85, in train_qdm
    services.train_qdm(
  File "/opt/dodola/dodola/services.py", line 28, in service_logger
    func(*args, **kwargs)
  File "/opt/dodola/dodola/services.py", line 63, in train_qdm
    storage.write(out, qdm.ds)
  File "/opt/dodola/dodola/repository.py", line 42, in write
    x.to_zarr(url_or_path, mode="w", compute=True)
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/dataset.py", line 1922, in to_zarr
    return to_zarr(
  File "/opt/conda/lib/python3.9/site-packages/xarray/backends/api.py", line 1448, in to_zarr
    writes = writer.sync(compute=compute)
  File "/opt/conda/lib/python3.9/site-packages/xarray/backends/common.py", line 167, in sync
    delayed_store = da.store(
  File "/opt/conda/lib/python3.9/site-packages/dask/array/core.py", line 1028, in store
    result.compute(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dask/base.py", line 285, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dask/base.py", line 567, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/dask/threaded.py", line 79, in get
    results = get_async(
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 514, in get_async
    raise_exception(exc, tb)
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 325, in reraise
    raise exc
  File "/opt/conda/lib/python3.9/site-packages/dask/local.py", line 223, in execute_task
    result = _execute_task(task, data)
  File "/opt/conda/lib/python3.9/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/parallel.py", line 305, in _wrapper
    raise ValueError(
ValueError: Expected index 'quantiles' to be PandasIndex(array=Float64Index([               0.005,                0.015,                0.025,
              0.034999999999999996,                0.045,                0.055,
                             0.065,  0.07500000000000001,                0.085,
                             0.095,  0.10500000000000001,                0.115,
                             0.125,                0.135,  0.14500000000000002,
                             0.155,                0.165,  0.17500000000000002,
                             0.185,                0.195,  0.20500000000000002,
                             0.215,                0.225,  0.23500000000000001,
                             0.245,                0.255,                0.265,
                             0.275,  0.28500000000000003,                0.295,
                             0.305,                0.315,                0.325,
                             0.335,  0.34500000000000003,  0.35500000000000004,
                             0.365,                0.375,                0.385,
                             0.395,                0.405,  0.41500000000000004,
                             0.425,                0.435,                0.445,
                             0.455,                0.465,  0.47500000000000003,
                             0.485,                0.495,                0.505,
                             0.515,                0.525,                0.535,
                             0.545,                0.555,   0.5650000000000001,
                0.5750000000000001,                0.585,                0.595,
                             0.605,                0.615,                0.625,
                             0.635,                0.645,                0.655,
                             0.665,                0.675,                0.685,
                0.6950000000000001,   0.7050000000000001,                0.715,
                             0.725,                0.735,                0.745,
                             0.755,                0.765,                0.775,
                             0.785,                0.795,                0.805,
                0.8150000000000001,   0.8250000000000001,   0.8350000000000001,
                             0.845,                0.855,                0.865,
                             0.875,                0.885,                0.895,
                             0.905,                0.915,                0.925,
                             0.935,   0.9450000000000001,   0.9550000000000001,
                             0.965,                0.975,                0.985,
                             0.995],
             dtype='float64'), dtype=dtype('float64')). Received PandasIndex(array=Float64Index([0.004999999888241291, 0.014999999664723873,  0.02500000037252903,
               0.03500000014901161,  0.04500000178813934, 0.054999999701976776,
               0.06499999761581421,  0.07500000298023224,  0.08500000089406967,
                0.0949999988079071,  0.10499999672174454,  0.11500000208616257,
                             0.125,  0.13500000536441803,  0.14499999582767487,
                0.1550000011920929,  0.16500000655651093,  0.17499999701976776,
                0.1850000023841858,  0.19499999284744263,  0.20499999821186066,
                0.2150000035762787,  0.22499999403953552,  0.23499999940395355,
               0.24500000476837158,   0.2549999952316284,  0.26499998569488525,
                0.2750000059604645,   0.2849999964237213,  0.29499998688697815,
                0.3050000071525574,   0.3149999976158142,  0.32499998807907104,
               0.33500000834465027,   0.3449999988079071,  0.35499998927116394,
               0.36500000953674316,                0.375,  0.38499999046325684,
               0.39500001072883606,   0.4050000011920929,  0.41499999165534973,
               0.42500001192092896,   0.4350000023841858,   0.4449999928474426,
               0.45500001311302185,   0.4650000035762787,   0.4749999940395355,
               0.48500001430511475,   0.4950000047683716,   0.5049999952316284,
                0.5149999856948853,   0.5249999761581421,   0.5350000262260437,
                0.5450000166893005,   0.5550000071525574,   0.5649999976158142,
                 0.574999988079071,   0.5849999785423279,   0.5950000286102295,
                0.6050000190734863,   0.6150000095367432,                0.625,
                0.6349999904632568,   0.6449999809265137,   0.6549999713897705,
                0.6650000214576721,    0.675000011920929,   0.6850000023841858,
                0.6949999928474426,   0.7049999833106995,   0.7149999737739563,
                0.7250000238418579,   0.7350000143051147,   0.7450000047683716,
                0.7549999952316284,   0.7649999856948853,   0.7749999761581421,
                0.7850000262260437,   0.7950000166893005,   0.8050000071525574,
                0.8149999976158142,    0.824999988079071,   0.8349999785423279,
                0.8450000286102295,   0.8550000190734863,   0.8650000095367432,
                             0.875,   0.8849999904632568,   0.8949999809265137,
                0.9049999713897705,   0.9150000214576721,    0.925000011920929,
                0.9350000023841858,   0.9449999928474426,   0.9549999833106995,
                0.9649999737739563,   0.9750000238418579,   0.9850000143051147,
                0.9950000047683716],
             dtype='float64', name='quantiles'), dtype=dtype('float64')) instead.

This if from workflow dc6-dev-7pfjb.

Remove our need for "custom" environment variables for storage auth, use adlfs defaults

Looks like adlfs grabs storage account, keys, name, etc from sensible environment variables (AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY).

We should consider just having users pass configs through these defaults. This seems like more sensible behavior if we want to support other fsspec-compatible storage options.

This has the potential to break backwards compatibility. Beware. Try to get this change in before our first release.

Add rechunker service

We need a service for rechunking climate data.

Roughly base it on ClimateImpactLab/downscaleCMIP6#75

The alternative is to run rechunking as a "script"-like step in the Argo workflow, kinda like how raw test data is downloaded (e.g. https://github.com/ClimateImpactLab/downscaleCMIP6/blob/master/workflows/downloadraw-workflow.yaml).

This service is intended for a necessary rechunking step in the body of the downscaling workflow, not rechunking for initial data cleaning.

Add basic plots (mostly for validation)

There are some general, common plots we could quickly put behind a dodola validate ... command (for example).

This might be worth a new module or breaking dodola.core into submodules.

We'd want CLI command to output PDFs or jpg/pngs. Internal function should maybe work on matplotlib axes? Zarr stores would be input to the CLI commands.

Clean up Azure storage authentication configuration in CLI

Right now we explicitly as for Azure storage authentication as an input to every CLI sub-command. Users can pass these in as CLI options, or more commonly, via environment variables. The problem is that dodola/cli.py functions are pretty cluttered with setup and documentation for these authentication inputs. There is quite a bit of repetition because this bit of authentication is needed for every CLI subcommand we have.

We can clean this up and reduce the chance for error by write a little internal function that grabs this authentication information from environment variables (ditch support for the CLI argument options) and just use this little credential-grabbing function everywhere. This would simplify the code and consolidate the logic. I don't know of any end-users that really need to use CLI arguments to pass credentials so no problem there. Everyone should be passing this through environment variables anyways.

Add basic logging framework

We should set up logging early on so we don't wind up with a mess of print() from different people debugging. Hope this will make it easier to grow into.

`azure.core.pipeline.policies.http_logging_policy` is waaay to chatty with messages to INFO

We should restrict or ignore logging from azure.core.pipeline because it spams INFO in our logs.

I think this change will be in the logging config setup in dodola/cli.py. I'm okay with a very simple solution right now, a comprehensive logging system for the app can wait.

add leap day removal service

Some GCMs have leap days, and others do not. We need to add a service that removes leap days.

Note and pin package changes in environment.yaml

We had a couple changes to environment.yaml starting in PR #95.

In environment.yaml:

• On the pandas line, we need to note that we don't directly depend on pandas and this version pin is a workaround to a regression (#96).

• On the pip xclim line, we need to pin the xclim version. Also, try switch the package install to a conda dependency rather than pip.

We should do this after the main features are merged into main, just before the release.

Container CI/CD builds fail on requirements install

CI/CD has failed (https://github.com/ClimateImpactLab/dodola/runs/1660654604). conda is choking on our custom pip install from requirements.txt. This underlines that we should prob just use conda environments.

cleancmip6 reads in all data when doing precipitation unit conversion?

From this run it looks like the dodola cleancmip6 might be trying to read in an entire Zarr store to memory when it does precipitation unit checks/conversion.

Support generic "fsspec"-like input for data files/stores

It would be nice if we generally support fsspec-style {protocol}{path} (e.g. az://bogus/data.zarr) for input and output data configs. This would make it easier moving data between multiple cloud providers. Users can assume they'd pass in credentials through default environment variables (see #55).

This is more consistent behavior with tools our users are already using. I think didn't set it up this way initially because I was having trouble with adlfs when we first started prototyping the pipeline tools.

Want to get this behavior in before our first release.

Drawback to this is that we become more sensitive to breaking changes in the fsspec-verse -- which is still pretty young.

ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn'

CI/CD is currenly failing on several PRs with errors like

2021-04-06T19:53:04.1755264Z ============================= test session starts ==============================
2021-04-06T19:53:04.1756609Z platform linux -- Python 3.8.5, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /opt/conda/bin/python
2021-04-06T19:53:04.1757449Z cachedir: .pytest_cache
2021-04-06T19:53:04.1757812Z rootdir: /
2021-04-06T19:53:06.4964488Z collecting ... collected 0 items / 3 errors
2021-04-06T19:53:06.4964919Z 
2021-04-06T19:53:06.4965329Z ==================================== ERRORS ====================================
2021-04-06T19:53:06.4965945Z _____________ ERROR collecting opt/dodola/dodola/tests/test_cli.py _____________
2021-04-06T19:53:06.4967683Z ImportError while importing test module '/opt/dodola/dodola/tests/test_cli.py'.
2021-04-06T19:53:06.4968544Z Hint: make sure your test modules/packages have valid Python names.
2021-04-06T19:53:06.4969120Z Traceback:
2021-04-06T19:53:06.4969700Z opt/conda/lib/python3.8/importlib/__init__.py:127: in import_module
2021-04-06T19:53:06.4970448Z     return _bootstrap._gcd_import(name[level:], package, level)
2021-04-06T19:53:06.4971151Z opt/dodola/dodola/tests/test_cli.py:3: in <module>
2021-04-06T19:53:06.4971677Z     import dodola.cli
2021-04-06T19:53:06.4972179Z opt/dodola/dodola/cli.py:8: in <module>
2021-04-06T19:53:06.4972820Z     from dodola.repository import adl_repository
2021-04-06T19:53:06.4973519Z opt/dodola/dodola/repository.py:5: in <module>
2021-04-06T19:53:06.4974191Z     from adlfs import AzureBlobFileSystem
2021-04-06T19:53:06.4975190Z opt/conda/lib/python3.8/site-packages/adlfs/__init__.py:1: in <module>
2021-04-06T19:53:06.4975986Z     from .spec import AzureDatalakeFileSystem
2021-04-06T19:53:06.4976980Z opt/conda/lib/python3.8/site-packages/adlfs/spec.py:21: in <module>
2021-04-06T19:53:06.4977646Z     from fsspec.asyn import (
2021-04-06T19:53:06.4978749Z E   ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py)
2021-04-06T19:53:06.4979807Z _________ ERROR collecting opt/dodola/dodola/tests/test_repository.py __________
2021-04-06T19:53:06.4980943Z ImportError while importing test module '/opt/dodola/dodola/tests/test_repository.py'.
2021-04-06T19:53:06.4981829Z Hint: make sure your test modules/packages have valid Python names.
2021-04-06T19:53:06.4982420Z Traceback:
2021-04-06T19:53:06.4982981Z opt/conda/lib/python3.8/importlib/__init__.py:127: in import_module
2021-04-06T19:53:06.4983727Z     return _bootstrap._gcd_import(name[level:], package, level)
2021-04-06T19:53:06.4984479Z opt/dodola/dodola/tests/test_repository.py:2: in <module>
2021-04-06T19:53:06.4985219Z     from dodola.repository import memory_repository
2021-04-06T19:53:06.4985940Z opt/dodola/dodola/repository.py:5: in <module>
2021-04-06T19:53:06.4986614Z     from adlfs import AzureBlobFileSystem
2021-04-06T19:53:06.4987559Z opt/conda/lib/python3.8/site-packages/adlfs/__init__.py:1: in <module>
2021-04-06T19:53:06.4988356Z     from .spec import AzureDatalakeFileSystem
2021-04-06T19:53:06.4989352Z opt/conda/lib/python3.8/site-packages/adlfs/spec.py:21: in <module>
2021-04-06T19:53:06.4990009Z     from fsspec.asyn import (
2021-04-06T19:53:06.4991117Z E   ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py)
2021-04-06T19:53:06.4992143Z __________ ERROR collecting opt/dodola/dodola/tests/test_services.py ___________
2021-04-06T19:53:06.4993239Z ImportError while importing test module '/opt/dodola/dodola/tests/test_services.py'.
2021-04-06T19:53:06.4994113Z Hint: make sure your test modules/packages have valid Python names.
2021-04-06T19:53:06.4994699Z Traceback:
2021-04-06T19:53:06.4995259Z opt/conda/lib/python3.8/importlib/__init__.py:127: in import_module
2021-04-06T19:53:06.4996262Z     return _bootstrap._gcd_import(name[level:], package, level)
2021-04-06T19:53:06.4996994Z opt/dodola/dodola/tests/test_services.py:10: in <module>
2021-04-06T19:53:06.4997731Z     from dodola.repository import memory_repository
2021-04-06T19:53:06.4998441Z opt/dodola/dodola/repository.py:5: in <module>
2021-04-06T19:53:06.4999126Z     from adlfs import AzureBlobFileSystem
2021-04-06T19:53:06.5000086Z opt/conda/lib/python3.8/site-packages/adlfs/__init__.py:1: in <module>
2021-04-06T19:53:06.5000998Z     from .spec import AzureDatalakeFileSystem
2021-04-06T19:53:06.5002010Z opt/conda/lib/python3.8/site-packages/adlfs/spec.py:21: in <module>
2021-04-06T19:53:06.5002670Z     from fsspec.asyn import (
2021-04-06T19:53:06.5003785Z E   ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.8/site-packages/fsspec/asyn.py)
2021-04-06T19:53:06.5004671Z =========================== short test summary info ============================
2021-04-06T19:53:06.5005242Z ERROR opt/dodola/dodola/tests/test_cli.py
2021-04-06T19:53:06.5005867Z ERROR opt/dodola/dodola/tests/test_repository.py
2021-04-06T19:53:06.5006542Z ERROR opt/dodola/dodola/tests/test_services.py
2021-04-06T19:53:06.5007643Z !!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!

full raw output is here: https://pipelines.actions.githubusercontent.com/pI0rznBOREHLzDzYIIx7CqNOfWgyB9RQgLKZdXq33tb6VZrNeK/_apis/pipelines/1/runs/134/signedlogcontent/4?urlExpires=2021-04-06T20%3A09%3A34.3217143Z&urlSigningMethod=HMACV1&urlSignature=2ZXkCGQS7hkAlqFFdNbkWjVxd3YJbIIcLS3o3zBKyZY%3D

Looking back at successful and the recent unsuccessful jobs, it looks like fsspec update to v0.9.0 might be the root cause.

Overzealous exception blocking from dodola.service.log_service()

dodola.service.log_service() currently decorates all services and logs their entrance and exit. It also logs Exceptions but it is a bit too heavy handed. Exceptions raised when the stack enters different services are correctly logged when run from (for e.g.) a container in a cluster but log_service() also blocks exceptions raised from automated system tests. In the case of automated system tests, pytest or CI/CD usually just gives a stacktrace describing a very confused open_zarr() looking for a Zarr store or Group that doesn't exist.

Here is example output where I put a raise ValueError("uh-oh!") at the opening of dodola.services.regrid() to get:

FAILED                           [ 50%]
dodola/tests/test_services.py:154 (test_regrid[bilinear])
regrid_method = 'bilinear'

    @pytest.mark.parametrize("regrid_method", ["bilinear", "conservative"])
    def test_regrid(regrid_method):
        """Test that services.regrid regrids output"""
        goal_shape = (360, 90)
    
        # Make fake input data.
        ds_in = grid_global(30, 20)
        ds_in["fakevariable"] = wave_smooth(ds_in["lon"], ds_in["lat"])
    
        fakestorage = memory_repository(
            {
                "a_file_path": ds_in,
            }
        )
    
        regrid(
            "input_path",
            out="output_ds",
            method=regrid_method,
            storage=fakestorage,
        )
>       actual_shape = fakestorage.read("output_ds")["fakevariable"].shape

test_services.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../repository.py:51: in read
    x = open_zarr(self.get_mapper(url_or_path))
../../../../miniconda3/envs/dodola_dev/lib/python3.8/site-packages/xarray/backends/zarr.py:675: in open_zarr
    ds = open_dataset(
../../../../miniconda3/envs/dodola_dev/lib/python3.8/site-packages/xarray/backends/api.py:572: in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
../../../../miniconda3/envs/dodola_dev/lib/python3.8/site-packages/xarray/backends/zarr.py:296: in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

store = <fsspec.mapping.FSMap object at 0x7fe48a162f40>, mode = 'r'
cache_attrs = True, synchronizer = None, path = '', chunk_store = None
storage_options = None

    def open_group(store=None, mode='a', cache_attrs=True, synchronizer=None, path=None,
                   chunk_store=None, storage_options=None):
        """Open a group using file-mode-like semantics.
    
        Parameters
        ----------
        store : MutableMapping or string, optional
            Store or path to directory in file system or name of zip file.
        mode : {'r', 'r+', 'a', 'w', 'w-'}, optional
            Persistence mode: 'r' means read only (must exist); 'r+' means
            read/write (must exist); 'a' means read/write (create if doesn't
            exist); 'w' means create (overwrite if exists); 'w-' means create
            (fail if exists).
        cache_attrs : bool, optional
            If True (default), user attributes will be cached for attribute read
            operations. If False, user attributes are reloaded from the store prior
            to all attribute read operations.
        synchronizer : object, optional
            Array synchronizer.
        path : string, optional
            Group path within store.
        chunk_store : MutableMapping or string, optional
            Store or path to directory in file system or name of zip file.
        storage_options : dict
            If using an fsspec URL to create the store, these will be passed to
            the backend implementation. Ignored otherwise.
    
        Returns
        -------
        g : zarr.hierarchy.Group
    
        Examples
        --------
        >>> import zarr
        >>> root = zarr.open_group('data/example.zarr', mode='w')
        >>> foo = root.create_group('foo')
        >>> bar = root.create_group('bar')
        >>> root
        <zarr.hierarchy.Group '/'>
        >>> root2 = zarr.open_group('data/example.zarr', mode='a')
        >>> root2
        <zarr.hierarchy.Group '/'>
        >>> root == root2
        True
    
        """
    
        # handle polymorphic store arg
        clobber = mode != "r"
        store = _normalize_store_arg(
            store, clobber=clobber, storage_options=storage_options, mode=mode
        )
        if chunk_store is not None:
            chunk_store = _normalize_store_arg(chunk_store, clobber=clobber,
                                               storage_options=storage_options)
        path = normalize_storage_path(path)
    
        # ensure store is initialized
    
        if mode in ['r', 'r+']:
            if contains_array(store, path=path):
                raise ContainsArrayError(path)
            elif not contains_group(store, path=path):
>               raise GroupNotFoundError(path)
E               zarr.errors.GroupNotFoundError: group not found at path ''

../../../../miniconda3/envs/dodola_dev/lib/python3.8/site-packages/zarr/hierarchy.py:1166: GroupNotFoundError

Add support for Azure Data Lake Gen2 storage

We need to add support for storing Zarr files in ADL via dodola/repository.py. While we're at it we can clean out the old demo RepositoryAbc subclasses and just focus on Zarr/ADL and the subclass used for testing.

We'd prob want to use adlfs for this (https://github.com/dask/adlfs).

We can start by passing in auth keys via the CLI/environment variables (click has a nice interface for this). It would be nice to also support the usual clientSecret, clientId, tenant credentials needed to authenticate service principals but I think it's okay if we need to iron this out later.

Don't worry too much about testing any new Repository subclasses for ADL support. We need to iron out details about Zarr configs on write and I figure storage tests will be white-box anyways...

add downscaling service

We also need to add a service that includes an implementation of Quantile Preserving Spatial Disaggregation (QPSD), our downscaling method.

update quantiles in QDM implementation

As of now, endpoints are added to nquantiles by equally_spaced_nodes since eps is set in the EmpiricalQuantileMapping base class train method inherited by QuantileDeltaMapping in xclim. eps=None results in equally_spaced_nodes not adding endpoints, which is what we want. We can update as follows:

quantiles = equally_spaced_nodes(100, eps=None)
QDM = QuantileDeltaMapping(..., nquantiles=quantiles)

Need weight-file creation service

We need to create weights files for later downscaling stages of the workflow.

A good starting point is create a service-level integration test and work up from there.

Add Docker container build specs

This app will run in a Docker container for production. We need to begin specifying the Docker build and test.

Really quickly, this would need:

• Dockerfile Keep it as light as we can.
• .dockerignore Keep out cruft, including .git.
• Be able to run package tests on installed application within the container (optional)

This doesn't need to test, build or release as a part of CI/CD just yet. This issue is just to focus on the container spec.

regridding bug with cli

The regridding service inherited the same bug from buildweights as in #32, e.g.

  File "/anaconda/envs/dodola/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola==0.1.0a0', 'console_scripts', 'dodola')())
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
TypeError: regrid() got an unexpected keyword argument 'targetresolution'

same fix as in #32 should take care of it.

error when running buildweights

When I run the following on Azure, dodola buildweights clean-dev/ACCESS-ESM1-5.historical.zarr -m bilinear --outpath clean-dev/ACCESS-ESM1-5.historical.1x1.weights.nc, I'm getting this error:

Traceback (most recent call last):
  File "/anaconda/envs/dodola/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola==0.1.0a0', 'console_scripts', 'dodola')())
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/anaconda/envs/dodola/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
TypeError: buildweights() got an unexpected keyword argument 'targetresolution'

I think it's because targetresolution in the cli interface should actually be target_resolution, but I'll take a closer look tomorrow and test it.

NotImplementedError from dodola downscale

Got funny error from a command. I think the command would have looked like:

downscale "az://scratch/dc6-dev-75bpr/fine-climatology-regrid.zarr" \
    --trainvariable "tasmax" \
    --outvariable "tasmax" \
    --yclimocoarse "az://scratch/foobar/coarse-climatology-regrid.zarr" \
    --yclimofine "az://scratch/foobar/fine-climatology-regrid.zarr" \
    --method "BCSD" \
    --domain_file "az://support/domain.0p25x0p25.zarr" \
    --adjustmentfactors "az://scratch/foobar/downscale-adjustmentfactors.zarr" \
    --out "az://scratch/foobar/biascorrected-downscaled.zarr"

The error from logging:

dc6-dev-n9fqz: Traceback (most recent call last):
dc6-dev-n9fqz:   File "/opt/conda/bin/dodola", line 33, in <module>
dc6-dev-n9fqz:     sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1134, in __call__
dc6-dev-n9fqz:     return self.main(*args, **kwargs)
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1059, in main
dc6-dev-n9fqz:     rv = self.invoke(ctx)
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1665, in invoke
dc6-dev-n9fqz:     return _process_result(sub_ctx.command.invoke(sub_ctx))
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1401, in invoke
dc6-dev-n9fqz:     return ctx.invoke(self.callback, **ctx.params)
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 767, in invoke
dc6-dev-n9fqz:     return __callback(*args, **kwargs)
dc6-dev-n9fqz:   File "/opt/dodola/dodola/cli.py", line 150, in downscale
dc6-dev-n9fqz:     services.downscale(
dc6-dev-n9fqz:   File "/opt/dodola/dodola/services.py", line 27, in service_logger
dc6-dev-n9fqz:     func(*args, **kwargs)
dc6-dev-n9fqz:   File "/opt/dodola/dodola/services.py", line 190, in downscale
dc6-dev-n9fqz:     adjustment_factors, downscaled_ds = apply_downscaling(
dc6-dev-n9fqz:   File "/opt/dodola/dodola/core.py", line 193, in apply_downscaling
dc6-dev-n9fqz:     model = SpatialDisaggregator(var=train_variable)
dc6-dev-n9fqz:   File "/opt/conda/lib/python3.9/site-packages/skdownscale/spatial_models/sd.py", line 24, in __init__
dc6-dev-n9fqz:     raise NotImplementedError(
dc6-dev-n9fqz: NotImplementedError: functionality for spatial disaggregation of tasmax has not yet been added

I think this is because trainvariable needs to be either temperature or precipitation and this is a restriction from skdownscale....?

If we need any of the variable parameter(s) vocabulary is limited then this should be in documentation, and I don't think it is?

Build and deploy Docker containers as part of CI/CD

We should build and deploy a Docker container to a registry as a part of CI/CD.

Optionally, consider testing with the built container image.

"height" is not cleaned from CMIP6 Datasets with dodola.core.standardize_gcm

Running some raw CMIP6 runs through dodola cleancmip6 with v0.2.0 or main does not remove the "height" coordinate variable from the Dataset.

Here is a print from one of the cleaned Datasets:

opening az://clean/ACCESS-ESM1-5/training/r1i1p1f1/tasmax.zarr
<xarray.Dataset>
Dimensions:    (bnds: 2, lat: 145, lon: 192, time: 7300)
Coordinates:
    height     float64 ...
  * lat        (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0
    lat_bnds   (lat, bnds) float64 dask.array<chunksize=(145, 2), meta=np.ndarray>
  * lon        (lon) float64 0.0 1.875 3.75 5.625 ... 352.5 354.4 356.2 358.1
    lon_bnds   (lon, bnds) float64 dask.array<chunksize=(192, 2), meta=np.ndarray>
  * time       (time) object 1995-01-01 12:00:00 ... 2014-12-31 12:00:00
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(7300, 1), meta=np.ndarray>
Dimensions without coordinates: bnds
Data variables:
    tasmax     (time, lat, lon) float32 dask.array<chunksize=(365, 145, 192), meta=np.ndarray>
Attributes: (12/50)
    Conventions:             CF-1.7 CMIP-6.2
    activity_id:             CMIP
    branch_method:           standard
    branch_time_in_child:    0.0
    branch_time_in_parent:   21915.0
    cmor_version:            3.4.0
    ...                      ...
    table_info:              Creation Date:(30 April 2019) MD5:e14f55f257ccea...
    title:                   ACCESS-ESM1-5 output prepared for CMIP6
    tracking_id:             hdl:21.14100/67543e90-bb83-44b1-8a30-e1fbb5ada0d...
    variable_id:             tasmax
    variant_label:           r1i1p1f1
    version:                 v20191115

It's important that we remove this "height" coordinate because it causes the Quantile Delta Mapping training step to fail in a very hard and obtuse way.

I think the problem is that we aren't correctly detecting that "height" is present with

dodola/dodola/core.py

Line 279 in 1e372ab

if "height" in ds.dims:

Instead, we might check that "heights" is an element in ds.variables or ds.coords ?

Edit: This got past unit testing because that test was also checking for membership against ds.dims. Should have checked that the test failed properly before it actually passed. Should update how the test checks membership also in any fix.

Add regridding service

We created a service to create the weights file for regridding (#9), but we need a service for doing the actual zarr-to-zarr regridding.

The service should:

• be able to regrid to a user-input grid spec (similar to dodola buildweights)
• ideally take zarr store input and stream to zarr store output
• compute with an optional pre-existing regridding weights file

add QDM to bias correction code and multiple methods to cli

Need to add QDM method (Cannon et al., 2015) to bias correction code options in conjunction with a method argument that defines that at the CLI level.

dodola biascorrect depends on a muddy storm of position-dependent arguments on CLI

Running dodola biascorrect form the commandline requires a mess of position-dependent arguments. It would be cleaner for end-users if we could switch these from arguments to options so that the parameters are pass in explicitly with flags like -m or --method.

unit tests failing because of timeslice issue

We're currently getting a bunch of test errors coming from Line 98 in core.py in the adjust quantile mapping function coming from how we're slicing time:

=================================== FAILURES ===================================
__________ test_adjust_quantiledeltamapping_year_kind[additive kind] ___________

variable_kind = '+', expected = -1.0

    @pytest.mark.parametrize(
        "variable_kind, expected",
        [
            pytest.param("+", -1.0, id="additive kind"),
            pytest.param("*", 0.5, id="multiplicative kind"),
        ],
    )
    def test_adjust_quantiledeltamapping_year_kind(variable_kind, expected):
        """Test that QDM 'kind' is handled"""
        # Setup input data.
        target_variable = "fakevariable"
        n_simdays = 100 * 365  # 100 years of daily simulation.
    
        model_bias = 2.0
        ts_sim = np.ones(n_simdays, dtype=np.float64)
        sim = _timeseriesfactory(
            ts_sim * model_bias, start_dt="2015-01-01", variable_name=target_variable
        )
    
        target_year = 2026
    
        # Yes, I'm intentionally training the QDM to a different bias. This is to
        # spurn a difference between "kind" adjustments...
        qdm = _train_simple_qdm(
            target_variable="fakevariable", kind=variable_kind, additive_bias=model_bias + 1
        )
>       adjusted_ds = adjust_quantiledeltamapping_year(
            simulation=sim,
            qdm=qdm,
            year=target_year,
            variable=target_variable,
        )

opt/dodola/dodola/tests/test_core.py:109: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opt/dodola/dodola/core.py:98: in adjust_quantiledeltamapping_year
    simulation = simulation[variable].sel(
opt/conda/lib/python3.9/site-packages/xarray/core/dataarray.py:1271: in sel
    ds = self._to_temp_dataset().sel(
opt/conda/lib/python3.9/site-packages/xarray/core/dataset.py:2365: in sel
    pos_indexers, new_indexes = remap_label_indexers(
opt/conda/lib/python3.9/site-packages/xarray/core/coordinates.py:421: in remap_label_indexers
    pos_indexers, new_indexes = indexing.remap_label_indexers(
opt/conda/lib/python3.9/site-packages/xarray/core/indexing.py:274: in remap_label_indexers
    idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
opt/conda/lib/python3.9/site-packages/xarray/core/indexing.py:121: in convert_label_indexer
    indexer = index.slice_indexer(
opt/conda/lib/python3.9/site-packages/pandas/core/indexes/base.py:5686: in slice_indexer
    start_slice, end_slice = self.slice_locs(start, end, step=step)
opt/conda/lib/python3.9/site-packages/pandas/core/indexes/base.py:5888: in slice_locs
    start_slice = self.get_slice_bound(start, "left")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = CFTimeIndex([2015-01-01 00:00:00, 2015-01-02 00:00:00, 2015-01-03 00:00:00,
             2015-01-04 00:00:00, 2015-01-...-30 00:00:00,
             2114-12-31 00:00:00],
            dtype='object', length=36500, calendar='noleap', freq='D')
label = '2015-12-17', side = 'left', kind = None

    def get_slice_bound(self, label, side: str_t, kind=None) -> int:
        """
        Calculate slice bound that corresponds to given label.
    
        Returns leftmost (one-past-the-rightmost if ``side=='right'``) position
        of given label.
    
        Parameters
        ----------
        label : object
        side : {'left', 'right'}
        kind : {'loc', 'getitem'} or None
    
        Returns
        -------
        int
            Index of label.
        """
        assert kind in ["loc", "getitem", None]
    
        if side not in ("left", "right"):
            raise ValueError(
                "Invalid value for side kwarg, must be either "
                f"'left' or 'right': {side}"
            )
    
        original_label = label
    
        # For datetime indices label may be a string that has to be converted
        # to datetime boundary according to its resolution.
>       label = self._maybe_cast_slice_bound(label, side)
E       TypeError: _maybe_cast_slice_bound() missing 1 required positional argument: 'kind'

opt/conda/lib/python3.9/site-packages/pandas/core/indexes/base.py:5798: TypeError

Edit: Can see this in PR #95 CI job https://github.com/ClimateImpactLab/dodola/runs/2975167693.