linkedearth / pyleotutorials

Jupyter-based, science-driven tutorials for using the LinkedEarth data-software Python ecosystem

PyleoTutorials: A Gentle Introduction to the Pyleoclim Package.

A collection of science-driven tutorials for using the Pyleoclim Python package. Contributed by: Deborah Khider, Julien Emile-Geay, Alexander James, Feng Zhu, Jordan Landers, Pin-Tzu Lee

Contents

The notebooks folder contains Jupyter Notebooks that illustrate the main classes of the Pyleoclim user interface, using data hosted either in the Cloud or in the data directory.

The notebooks are organized in three levels:

• Level 0: basic tutorials to understand the main classes and their uses, requiring minimal knowledge of Python
• Level 1: more advanced tutorials involving customization and more knowledge of Python
• Level 2: tutorials involving more advanced data analysis techniques, requiring some domain knowledge to apply and interpret properly.

Read-only

You may start by browsing this repository to consult its notebooks, which are rendered by the Github website. If you like what you see, we encourage you to experiment with them, as per the following section.

How to run these notebooks

Using myBinder

myBinder allows you to run the notebooks in a no-install cloud container. This is the best solution if you just want to get you feet wet with the code. Simply click on the "launch Binder" badge at the top of the page. The notebooks will be automatically pulled into your workspace. Beware: it takes a few minutes for the BinderHub to be spun up the first time around. After repeated uses (not just by you) it becomes much faster, but the first time you launch might need to coincide with a coffee break, a run around the nearest park, or catching up on the literature.

Local installation

If it's love at first sight and you want to take Pyleoclim home with you, you may use the environment.yml file with conda to install the required packages within an anaconda or miniconda installation. That is, open a terminal, navigate to a folder containing environment.yml , then run the command :

conda env create -f environment.yml

This will install all the required packages and their dependencies. It may take a few minutes the first time, but after that you will have the power of Pyleoclim at your local disposal.

If you are new to conda, just know that it is an environment manager for Python. What is an environment, you may ask? Think of it as a piece on your computer where all the Python libraries that you need for a specific application live in perfect harmony. To access this paradise, you need to activate the environment. If you used the environment.yml file:

conda activate pyleo

Using the LinkedEarth JupyterHub

Coming Soon (September 2022)

If you have access to a decent internet connection and do not want to install Python and Pyleoclim on your machine, you can also run these (and your own) notebooks on the LinkedEarth JupyterHub. To do so, please contact LinkedEarth to obtain an account, which may take 1-2 business days to review. Use of the LinkedEarth Hub implies that you subscribe to the LinkedEarth code of conduct which pretty much amounts to being a decent human being. This is a good solution if you intend on using Pyleoclim in your own work, particularly for paleoclimate data-model comparisons.

If you choose this route, know that the PyleoTutorials notebooks will not be directly pulled to the hub (unlike the myBinder solution). You will first need to download, clone or fork this repository onto your local machine. You can do so by clicking on the green 'code' button at the top (right above the list of files in the repository):

To learn more about cloning/forking, see this post. Downloading is just a one time action.

Once the notebooks and data are on your local machine, you can upload them to your private instance on the hub using this tutorial.

Please note that the notebooks use specific paths that you will either need to correct in the cells or reproduce on the hub (i.e., the notebooks and data folders).

Whichever path you choose, happy Pyleocliming, and please consider joining our user community on Discourse. It is the best place to provide feedback on any of this, and make requests for new tutorials.

Acknowledgements

This work was supported by NSF Grant ICER 2126510.

pyleotutorials's Issues

Series - How to create from various sources

We should have several notebooks illustrating how to get a Series (or LiPD Series) object from:

1. a csv through Pandas (show how to skip lines and enter headers; the LR04 dataset needs line skipping at the beginning)
2. a LiPD file (maybe use either ODP846 or Crystal Cave).

And show a simple plot

Correlation notebook run time

The L2_correlations notebook is pretty heavy (minimum 37 minute run time, which really slows down CI), could be worth discussing moving to paleobooks and replacing with a more lightweight example.

Basic operations with MulitpleSeries/EnsembleSeries

1. common_time
2. stackplot (including showing how to change the labels for an existing Series/MS object)

• colormap changes
• Labels (none, some, changes)
• shift factor.

For ensembleSeries

1. Envelope/traces

anti_alias filter in spectral analysis notebook

Pyleoclim now has a superb anti-alias feature for PSD objects. The spectral analysis notebook should nod to it.

Include new wavelet/coherence capabilities

Recent updates to Pyleoclim allow to more flexibly specify the time and scale axes for Scalogram and Coherence objects. The Wavelet tutorial should illustrate how to use them.

Filtering and detrending tutorial

Re-use from PaleoHack Notebook #3

Summary plots

Work through an example to "pretty up" a summary plot for publication, with the various options. Can reuse the example in the Pyleoclim manuscript using Nino3.4 data.

It would be interesting to first show the default plot and look at ways to change some attributes.

Enable CI for the notebooks

We should be able to re-use the notebooks to run the "tests". Only thing to be careful about is that some of the notebooks require user input (i.e., choose your timeseries from a lipd file). So this particular line for the tests notebook would have to be rewritten to use the number parameter.

I propose to create a test folder that contain the test notebooks that are updated for no user inputs. Or we modify to not allow for user inputs. This problem should disappear once we integrate pylipd.

Interpolation vs bin vs gkernel and effect on spectral analysis

Explore the difference between the different imputation method and effect on spectral analysis

Include tutorials on how to get age ensembles quantiles for value and time and export to pandas, numpy

Following an open request for Pyleoclim, the quantiles function in EnsembleSeries can now support either calculating the quantiles along the value axis or the time axis.

In addition, the time/values can be exported to Pandas DataFrame or numpy arrays for further manipulation.

Suggest adding the new tutorials in L1_working_with_age_ensembles.

Update basic tutorials

Somewhere in an old Pyleoclim version (or on @khider's hard drive), you will find a tutorial called pyleoclim_ui_tutorial.ipynb and one called plot_styles.ipynb. These tutorials need to be moved here and made to work with 0.8.1.

See LinkedEarth/Pyleoclim_util#216

Age ensemble notebook

Create notebook with examples of moving from age ensembles + series to ensemble series:

• From lipd file
• From csv (maybe superfluous depending on NOAA format)
• From pangaea
• From noaa

QuanSight work showcase

Going with Pyleoclim release 0.11.0, we need to showcase the multiple improvements brought about by the work of the QuanSight team.

• all tutorials should start using load_dataset() whenever possible
• the pandas work per se should go into a dedicated L0 notebook called "Pandas in Pyleoclim". This will repurpose the first half of the paleopandas playground.
• the work on MultipleSeries overloaded methods (+, -, &) should go into “L0_basic_MSES_manipulation.ipynb”. This will repurpose the second half of the paleopandas playground

Update tutorials with new Series.surrogates

Todo:

• new notebook on how to use surrogates
• update correlation nb
• update WTC in wavelet nb (phaseran and ar1sim)
• update spectral notebook

Spectral Analysis

Show how the various methods work and the effect of parameters (e.g, c for WWZ, n50 for Lomb-Scargle, BW for MTM, windows for Welch), the frequency vector options + effect of detrending/binning.

main --> jupyterbook automation

updates are not automated; see if we can set up a GitHub actions to automatically update :

• notebooks
• environment
  from the main branch to jupyterbook

Pyleoclim correlation methods

explain the different ways significance is assessed, especially over a grid.

tutorial on how to edit subplots

Hi all - would love to see a tutorial on how to edit subplots of a pyleoclim figure. Specifically, I want to override a PCA plot so I can plot my own loadings.

Right now, the way I'm handling this is by calling

fig, gs = pca.modeplot()
ax = subplot(gs[3,1])

and then plotting everything to ax. gs is a GridSpec and fig is a Figure.

ReadMe file

I started a readme file, but it needs more precise guidance on:

1. installation using some environment.yml file (the one from Pyleoclim_util? Please confirm)
2. how to sign up for the Hub

I also need feedback on the various levels of tutorials.

We should probably acknowledge the PaleoCube grant there too

Highlighting intervals tutorial

Build a tutorial showing how to highlight intervals:

• overlay
• above
• below

with labels:

• added to the legend (e.g. if there are categories of labels)
• per interval with pointers

Bonus:

• labelling specific points on a figure
• making an event timeline?

Fodder: geologic timescale (Cenozoic), MIS

Pretty faces for notebook authors

Our current model for author contributions comes from EarthCube, and let's just say it's not optimal:

Alexander James, Department of Earth Sciences, University of Southern California

Author = {"name": "Alexander James", "affiliation": "Department of Earth Sciences, University of Southern California", "email": "[email protected]", "orcid": "0000-0001-8561-3188"} 

Julien Emile-Geay, Department of Earth Sciences, University of Southern California 

Author = {"name": "Julien Emile-Geay", "affiliation": "Department of Earth Sciences, University of Southern California", "email": "[email protected]", "orcid": "0000-0001-5920-4751"}

On the other hand, it's possible to have our pretty avatars on there, as in Jordan's paleobook repo.

I suggest we do this for all the notebooks in this repo. @jordanplanders would it take you long?

L0-Getting Started Notebook - Change link for intro to Python

Right now, the tutorials link to the old version of our training material. Once LeapFROGS is up and running, need to switch to that.

Edit: Actually needed in most notebooks

Creating Multiple Panels

Create a tutorial showing how Matplotlib subplots can be used in combination with returned fig/ax from Pyleoclimn figures. An example is from the correlation figure in the Pyleoclim manuscript.

showcase stripes() for Series and MultipleSeries in tutorials

• L0_basic_ts_manipulation.ipynb
• L0_basic_MSES_manipulation.ipynb

Update PCA notebook

the "data wrangling" section needs to showcase the newly developed MGS time/lat plot, and the soon-to-be developed Resolution class (and plot) for MultipleSeries.

Detrending

A helpful tutorial would be on detrending:

• rehashing the Series.detrend() docstring, which is pretty complete at this point
• illustrating how to use SSA for detrending.
• effect of detrending on spectral/wavelet analysis

Working with LiPD files

For LiPD directory:

• mapping

For LiPD files:

• dashboard.

Also show how to manipulate and the use of various LiPDs to LiPDSeries transforms.

Beef up the NOAA example

Show an example on how to use their API and grab relevant datasets to Series or GeoSeries

Update SSA tutorial

• Describe the SSA res object
• For the missing values section, use the knee method

Publication-ready figures

L0_a_quickstart.ipynb mentions "L1_publication_ready_figures.ipynb", but there is no such notebook. Do we want to create one, and if so, what figures would be most newsworthy?

SSA Tutorial

Use SSA tutorial from Julien

Basic timeseries manipulation with Pyleoclim

• Slicing
• standadizing
• gaussianize
• stats

Optional (the functions needs some TLC)

• Detecting gaps

pyLipd integration into tutorials

L1_working_with_LiPD.ipynb should be updated to reflect pyLipd updates, when the latter are ready for prime time.

Note: Euro2k example can go either there or in the MSES tutorial

Coherence/Wavelet

Repurpose the example from the documentation to create a tutorial on how to interpret coherence and wavelet plots.