thomasmbury / ewstools

Python package for early warning signals (EWS) of bifurcations in time series data.

ewstools

A Python package for early warning signals (EWS) of bifurcations in time series data.

Overview

Many systems in nature and society have the capacity to undergo critical transitions--sudden and profound changes in dynamics that are hard to reverse. Examples include the outbreak of disease, the collapse of an ecosystem, or the onset of a cardiac arrhythmia. From a mathematical perspective, these transitions may be understood as the crossing of a bifurcation (tipping point) in an appropriate dynamical system model. In 2009, Scheffer and colleagues proposed early warning signals (EWS) for bifurcations based on statistics of noisy fluctuations in time series data (Scheffer et al. 2009). This spurred massive interest in the subject, resulting in a multitude of different EWS for anticipating bifurcations (Clements & Ozgul 2018). More recently, EWS from deep learning classifiers have outperformed conventional EWS on several model and empirical datasets, whilst also providing information on the type of bifurcation (Bury et al. 2021).

ewstools is an accessible toolbox for computing, analysing and visualising EWS in time series data. It complements an existing EWS package in R (Dakos et al. 2012). Given the recent surge in popularity of the Python programming langauge (PYPL, 2022), a Python-based implementation of EWS should be useful.

The package provides:

• An intuitive, object-oriented framework for computing EWS for a given time series
• Time series detrending methods using
  • A Gaussian kernel
  • LOWESS (Locally Weighted Scatterplot Smoothing)
• Computation of CSD-based early warning signals including:
  • Variance and associated metrics (standard deviation, coefficient of variation)
  • Autocorrelation (at specified lag times)
  • Higher-order statistical moments (skewness, kurtosis)
  • Power spectrum and associated metrics
  • Various entropy measures
• Computation of Kendall tau values to quantify trends
• Application of deep learning classifiers for bifurcation prediction as in Bury et al. 2021.
• Visualisation tools to display output
• Built-in theoretical models to test EWS

ewstools makes use of pandas for dataframe handling, numpy for fast numerical computing, plotly for visuliastion, lmfit for least-squares minimisation, arch for bootstrapping methods, EntropyHub for entropy computations, statsmodels and scipy for detrending methods, and TensorFlow for deep learning.

Install

Requires Python 3.7 or later. You can install ewstools with pip using the commands

pip install --upgrade pip
pip install ewstools

Jupyter notebook is required for the tutorials, and can be installed with the command

pip install jupyter notebook

Package dependencies are

'pandas>=0.23.0',
'numpy>=1.14.0',
'plotly>=2.3.0',
'lmfit>=0.9.0', 
'arch>=4.4',
'statsmodels>=0.9.0',
'scipy>=1.0.1',

and should be installed automatically. To use the deep learning functionality, you will need to install TensorFlow with version later than 2.0 and earlier than 2.12.

To install the latest development version, use the command

pip install git+https://github.com/thomasmbury/ewstools.git#egg=ewstools

NB: the development version comes with the risk of undergoing continual changes, and has not undergone the level of scrutiny of official releases.

Tutorials

1. Introduction to ewstools
2. Spectral EWS
3. Deep learning classifiers for bifurcation prediction

Quick demo

First we need to import ewstools and collect data to analyse. Here we will run a simulation of the Ricker model, one of the models stored in ewstools.models.

import ewstools
from ewstools.models import simulate_ricker
series = simulate_ricker(tmax=500, F=[0,2.7])
series.plot();

We then make a TimeSeries object, which takes in our data and a transition time (if desired). EWS are not computed beyond the transition time.

ts = ewstools.TimeSeries(data=series, transition=440)

We can then detrend, compute EWS and calculate Kendall tau statistics by applying methods to the TimeSeries object:

ts.detrend(method='Lowess', span=0.2)
ts.compute_var(rolling_window=0.5)
ts.compute_auto(lag=1, rolling_window=0.5)
ts.compute_auto(lag=2, rolling_window=0.5)
ts.compute_ktau()

Finally, we can view output as an interactive Plotly figure (when run in a Jupyter notebook) using

ts.make_plotly()

More detailed demonstrations can be found in the tutorials, and all methods are listed in the documentation.

Documentation

Available on ReadTheDocs.

Issues

If you have any suggestions or find any bugs, please post them on the issue tracker. I also welcome any contributions - please get in touch if you are interested, or submit a pull request if you are familiar with that process.

Acknowledgements

This work is currently supported by an FRQNT (Fonds de recherche du Québec - Nature et Technologies) postdoctoral research scholarship awarded to Dr. Thomas Bury. In the past, it has also been supported by NSERC (Natural Sciences and Engineering Research Council) Discovery Grants awarded to Dr. Chris Bauch and Dr. Madhur Anand.

Citation info

If you like the respoitory, please give it a star :D

If your research makes use of it, please cite

Bury, Thomas M. "ewstools: A Python package for early warning signals of bifurcations in time series data." Journal of Open Source Software 8.82 (2023): 5038.

ewstools's Issues

Rolling window size as a proportion of pre-bifurcation data

Currently rolling window size is provided as a proportion of the length of the time series given. This is fine if only pre-bifurcation is fed into compute_ews. However, if pre- and post- bifurcation data are fed into compute_ews, along with upto defined as the time of bifurcation, then it makes more sense to define the rolling window parameter as the proportion of the length of the pre-bifurcation data.

Spectral EWS tutorial link

Hi,
The link for the spectral EWS tutorial in the pypi page, and the jupyter notebook link in https://pypi.org/project/ewstools/1.0.1/#demos don't work.

Thanks for the hard work ;)
Alexis

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

typo in the tutorial for spectral ews

At the end of the tutorial In[22] reads

ts_rm_x.make_plotly() Image(fig.to_image())

and it should be

fig = ts_rm_x.make_plotly() Image(fig.to_image())

Issue with power spectrum computation using ewstools

I am using the ewstools package to compute the power spectrum for a time series starting in 1854 and ending in 1905 with a rolling window size of 29. However, the output only provides two unique time points for the power spectrum, which is unexpected.

Below is the relevant code snippet:

import ewstools

#Assuming amoc_pcs_series is my time series data
ts = ewstools.TimeSeries(data=amoc_pcs_series, transition=2000)
rw_size = 29

ts.compute_spectrum(rolling_window=rw_size, ham_length=len(amoc_pcs))
ts.pspec

The output looks like this:

Unique time points: [1882. 1902.]

I expected around 22 power spectrum plots, but I only get two. I would appreciate any help or guidance on how to resolve this issue or if there's a misunderstanding on my part regarding the package's functionality.

Installed ewstools using pip but shows AttributeError

Hi,

I am exploring this library for my research on detecting tipping points. I made a time-series data based on typhoon speeds and would like to run it through this module. I am following the tutorial notebook where the time series class was initiated but this is where I encountered the AttributeError. Screenshot shown below:

I wonder if there's a suggested fix on this? Thanks and hope to hear from you soon.

Regards,
Gabriel

Error while uploading existing model in deeplearning tutorial

ValueError: File format not supported: filepath=..\ewstools\saved_classifiers\bury_pnas_21\len500\best_model_1_1_len500.pkl. Keras 3 only supports V3 .keras files and legacy H5 format files (.h5 extension). Note that the legacy SavedModel format is not supported by load_model() in Keras 3. In order to reload a TensorFlow SavedModel as an inference-only layer in Keras 3, use keras.layers.TFSMLayer(C:\Users\Murali\Desktop\Mtech\Sem4\stock\ewstools\saved_classifiers\bury_pnas_21\len500\best_model_1_1_len500.pkl, call_endpoint='serving_default') (note that your call_endpoint might have a different name).

Any suggestion on how to handle this error?

bug in the deep learning tutorial

There is a bug in the Deep learning classifiers for bifurcation prediction (tutorial_deep_learning.ipynb).
In particular, when plotting the ensemble average. The statement
ts_may.make_plotly(ens_avg=True)

makes the program crash.
The bug is in the function 'make_plotly', specifically with the statement:
df_plot = ts.dl_preds.groupby(['time']).mean().reset_index()

The bug is caused by the presence of column 'classifier' within the 'ts.dl_preds' frame.

A quick fix can be made by replacing the above statement with the following two:
df = ts.dl_preds.drop(['classifier'], axis=1) # JJV
df_plot = df.groupby(['time']).mean().reset_index() # JJV
I verified that the redefined function 'make_plotly' with the fix works flawlessly.
Thank you for developing the (ewstools) package. It is excellent and it has been very useful in my research.
I have some suggestions on how to further extend the package, which I would like to discuss with the ewstools development team.

Julio Valdes
National Research Council Canada
[email protected]