jonking93 / dash

A package for paleoclimate data assimilation workflow.

DASH

A Matlab toolbox for paleoclimate data assimilation workflow.

• Citation
• Summary
• Install DASH,
• Getting started
• Feedback and contributions
• Repository branches

Citation

Please cite:

King, J. M., Tierney, J., Osman, M., Judd, E. J., & Anchukaitis, K. J. (2023). DASH: A MATLAB Toolbox for Paleoclimate Data Assimilation. Geoscientific Model Development, (in review).

when using the toolbox.

Summary

DASH is a Matlab toolbox designed to facilitate paleoclimate data assimilation. It includes modules to help:

• Catalogue and organize climate data,
• Design state vector ensembles,
• Implement proxy system models, and
• Run Kalman filter, particle filter, and optimal sensor algorithms.

Install DASH

There are 3 ways to install DASH:

1. Using Github,
2. Via MATLAB's Add-On explorer, or
3. Using MATLAB File Exchange

Github

1. Navigate to the most recent stable release: Latest Release
2. Under the release assets, download the file: DASH-<version>.mltbx
3. Open the downloaded file. This should automatically install the DASH toolbox in your MATLAB environment. (If you previously installed a DASH toolbox using a .mltbx file, this will update your toolbox to the latest version)

MATLAB Add-On Explorer

1. Click on the Home tab in the MATLAB editor,
2. Click on the Add-Ons or Get Add-Ons button,
3. Search for DASH and click on the entry for the toolbox (Its tagline is "A Matlab toolbox for paleoclimate data assimilation")
4. Click on the Add button in the top-right corner and follow the instructions.

MATLAB File Exchange

1. Navigate to the submission for the toolbox: DASH on FileExchange
2. Click the Download button in the top right corner and select the Toolbox option. This should download a file with the name DASH-<version>.mltbx.
3. Open the downloaded file (its name should follow the pattern DASH-<version>.mltbx). This should automatically install the DASH toolbox in you MATLAB environment.

Getting Started

To get started with the DASH toolbox, enter:

>> dash.doc

in the MATLAB console. This will open the DASH documentation, which includes resources for starting with DASH. In particular, we recommend checking out the DASH tutorial. The tutorial begins with an overview of the DASH toolbox, and then provides step-by-step instructions and examples for using the components of DASH. We recommend budgeting several hours to complete the tutorial.

If you have not yet installed the toolbox, you can find the DASH documentation online here: DASH Documentation

And the tutorial is available here: DASH Tutorial

Feedback / Contributions

Find a bug, or have an idea for a cool new feature? We welcome feedback! For bug reports, suggestions, or anything else - send us an email at [email protected]

Interested in contributing? Either send us an email, or submit a pull request to the DASH repository to get started.

Branches

The following is an overview of the branches of the DASH repository.

main

This branch holds the most up-to-date source code for the DASH toolbox. This branch is intended for developers rather than users. It may contain active development and may not be stable. If you are looking to use the DASH toolbox, we recommend downloading and opening the DASH-<version>.mltbx file from the most recent stable release.

gh-pages

This branch holds the source code used to implement the online DASH documentation website.

dash's Issues

PSM: R error incorporates PSM error. Add dynamic R generation

DA: Cov localization should allow a toggle for whether to calculate a mean or convert to NaN

V2: Consider .grid and .ens extensions

Need to add vector R back to getPSMOutput

Ens: Overlap should be a property of coupled variables. NOT a property of the ensemble as a whole.

PSM: vslite bias correction is just a placeholder

Metadata duplicate checking takes too long in buildMetadata

V3: Write DA to file

DA: Enable all-at-once computation

GRID: Enable tripolar metadata

V3: Implement code for fixed R

V2: Enable writing ensemble to file

Ensemble dimensions should have subsetted metadata

ALL: Mark trailing singleton dimensions and do not return when regridding

V3: Joint updates for time-dependent PSMs

PSMs that are time dependent work fine for serial, but not joint.

PSMs that incorporate proxy memory are completely unsupported.

PSM: Investigate nargout in parfor and also handles

V2: Consider using metadata arrays that are not cells by default.

Maybe return Yi and Yf for appended method?

ensMeta in the design should be enabled for vectors for sequences

V3: how best to slice serial Ye output?

Grid: Consider ending gridfiles with a .grid extension

Design: Warn if a grid file is not on the path

Design: Check that indices, dimID, varNames, etc are always set to a column or row

DA: regridAnalysis should return metadata for plotting

Kalman chunking

You can recycle any kalman gain that uses the same "obs" array. Potentially large speed gains for successful PSMs with a joint update.

DA: Removed time-dependence from Dash

It made the joint ENSRF much cleaner. But might want to eventually add it back in.
The alternative is building a second PSM array.

DESIGN: Something needs to check that the ensemble metadata matches the size of the sequence index

V3: Variable Covariance localization

V3: Dash that can disable variance or mean (for joint)

Design: Overlap should be a scalar for each variable

Design: Should coupling / syncing match all ens dimensions are remove previous syncs?

Energy statistic / Npdft

estat is super slow. Either speed it up, or have Npdft just accept iterations and then calculate once at the end.

Design: Coupling by default

1. Should be able to couple multiple variables with a single function call.

2. Should have a couple 'all' flag

3. Perhaps automatically couple variables in a finalizeDesign function.
   (But this could be questionable, how to set state seq and mean indices?)
   Maybe have a copyIndices function for vars and template?

Gridfile Metadata should be in cells

Design: getCoupledStateIndex and getCoupledEnsIndex are practically duplicates

DA: Dash needs error checking

Ens: Save Ens to file. Consider .ens extension.

VS Lite PSM needs efficient error checking

DA: AAAAAAAAaaaaaaa, put error checking in Dash!!!!!!!!!!!

V3: Tall ensembles

Implement dash to handle ensembles that do not fit in working memory.

V3: Return Ye and update as outputs

DA: Need to monitor for exploded gain

Place the "var" field in getKnownIDs and remove any hardcoding

Check that gridfile metadata is always a column

Build quickstart

PSM: VS-Lite standardization.

Any VS-Lite standardization will change as multiple proxies are updated. This is because the mean of each run can change as the analysis updates.

Perhaps there should be a baseline standardization given at the beginning of each PSM?
This would be along the lines of: Do a bulk Ye calculation before starting dash, then save the mean and std of this bulk calculation to use in all dash updates.

V2: Check ensemble for NaNs or create addEnsemble support fxn

build "designDisp" or some function to let the user look at a state vector design.

Design: Check that the intersect in getMatchingMeta should not be stable

Ens: General messiness

1. Make a more intuitive directory format
2. More intelligent use of varDex, sequences, etc. Most of these functions should be called a single time and pass the outputs.