martinvonk / spei

A simple Python package to calculate and visualize some popular drought indices such as the SPI, SPEI and SGI.

SPEI

SPEI is a simple Python package to calculate drought indices for time series such as the SPI (Standardized Precipitation Index), SPEI (Standardized Precipitation Evaporation Index), and SGI (Standardized Groundwater Index). This package uses popular Python packages such as Pandas and Scipy to make it easy and versatile for the user to calculate the drought indices. Pandas Series are great for dealing with time series; providing interpolation, rolling average, and other manipulation options. SciPy enables us to use all different kinds of distributions to fit the data.

For the calculation of potential evaporation, take a look at pyet. This is another great package that uses pandas Series to calculate different kinds of potential evaporation time series.

Please feel free to contribute or ask questions!

If you happen to use this package, please cite: Vonk, M. A. (2024). SPEI: A simple Python package to calculate and visualize drought indices (vX.X.X). Zenodo. https://doi.org/10.5281/zenodo.10816741.

Available Drought Indices

Drought Index	Abbreviation	Literature
Standardized Precipitation Index	SPI	1
Standardized Precipitation Evaporation Index	SPEI	2
Standardized Groundwater Index	SGI	3,4
Standardized Streamflow Index	SSFI	5

The package is not limited to only these four drought indices. If any of the distributions in the Scipy library is valid, the drought index can be calculated.

Installation

To get the latest stable version install using:

pip install spei

To get the development version download or clone the GitHub repository to your local device. Install using:

pip install -e <download_directory>

Literature

1. B. Lloyd-Hughes and M.A. Saunders (2002) - A Drought Climatology for Europe. DOI: 10.1002/joc.846
2. S.M. Vicente-Serrano, S. Beguería and J.I. López-Moreno (2010) - A Multi-scalar drought index sensitive to global warming: The Standardized Precipitation Evapotranspiration Index. DOI: 10.1175/2009JCLI2909.1
3. J.P. Bloomfield and B.P. Marchant, B. P. (2013) - Analysis of groundwater drought building on the standardised precipitation index approach. DOI: 10.5194/hess-17-4769-2013
4. A. Babre, A. Kalvāns, Z. Avotniece, I. Retiķe, J. Bikše, K.P.M. Jemeljanova, A. Zelenkevičs and A. Dēliņa (2022) - The use of predefined drought indices for the assessment of groundwater drought episodes in the Baltic States over the period 1989–2018. DOI: 10.1016/j.ejrh.2022.101049
5. E. Tijdeman, K. Stahl and L.M. Tallaksen (2020) - Drought characteristics derived based on the Standardized Streamflow Index: A large sample comparison for parametric and nonparametric methods. DOI: 10.1029/2019WR026315

Note that the method for calculating the drought indices does not come from these articles and SciPy is used for deriving the distribution. However the literature is helpful as a reference to understand the context and application of drought indices.

Alternatives

There are other great packages available to calculate these indices. However, they are either written in R such as SPEI or don't have the Standardized Groundwater Index such as climate_indices. Additionaly, these packages provide ways to analyse spatial data and calculate potential evaporation. This makes these packages complex, because it is easier to only deal with time series. However, support for spatial data is something on the to-do list so help is appreciated.

spei's Issues

Invalid frequency passed to pd.Grouper

Invalid frequency string passed to pd.Grouper.

The group_yearly_df function passes either "YE" or "Y" to pd.Grouper, depending on the pandas version. Using pandas version '2.1.4', the frequency passed to pd.Grouper is invalid. The correct string is "Y", but "YE" is passed.

The following error was caused:

File ~/.local/lib/python3.10/site-packages/spei/utils.py:85, in group_yearly_df(series)
83 grs = {}
84 freq = "YE" if pd_version >= "2.1.0" else "Y"
---> 85 for year_timestamp, gry in series.groupby(Grouper(freq=freq)):
86 index = validate_index(gry.index)
87 gry.index = to_datetime(
88 "2000-" + index.strftime(strfstr), format="%Y-" + strfstr
89 )

File /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/lib/python3.10/site-packages/pandas/core/resample.py:2046, in TimeGrouper.init(self, freq, closed, label, how, axis, fill_method, limit, kind, convention, origin, offset, group_keys, **kwargs)
2043 if convention not in {None, "start", "end", "e", "s"}:
2044 raise ValueError(f"Unsupported value {convention} for convention")
-> 2046 freq = to_offset(freq)
2048 end_types = {"M", "A", "Q", "BM", "BA", "BQ", "W"}
2049 rule = freq.rule_code

File offsets.pyx:4460, in pandas._libs.tslibs.offsets.to_offset()

File offsets.pyx:4557, in pandas._libs.tslibs.offsets.to_offset()

ValueError: Invalid frequency: YE

Question : Exception has occurred: ValueError -- Formating issue of the input ?

Hi, I with spei_series = si.spei(input, timescale=30, dist=scs.fisk, fit_freq="MS")
I get
Exception has occurred: ValueError
No objects to concatenate. What is wrong with my input ?

Here is a view of my input series
1961-01-01 897.0
1961-02-01 -69.0
1961-03-01 -706.0
1961-04-01 86.0
1961-05-01 -380.0
...
2020-08-01 -960.0
2020-09-01 -363.0
2020-10-01 1067.0
2020-11-01 -215.0
2020-12-01 1849.0

How to calculate in monthly scale？

I have a monthly data, but I don't konw hw to use this code. Thanks

Switch to Object-Oriented

The original setup was functional. But it is better to transform to SI class and save all data and fitted distributions and logic there.

ValueError: cannot set using a list-like indexer with a different length than the value

Hello,

I am having the following error when running si.spei() with a series without any missing value and the index as datetimeindex: ValueError: cannot set using a list-like indexer with a different length than the value

Here is the traceback:
`File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/spei/si.py:170, in spei(series, dist)
143 def spei(series: Series, dist: ContinuousDist = fisk) -> Series:
144 """Method to compute the Standardized Precipitation Evaporation Index
145 [spei_2010]_.
146
(...)
167 Journal of Climate, 23, 1696-1718, 2010.
168 """
--> 170 return compute_si_ppf(series=series, dist=dist)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/spei/si.py:56, in compute_si_ppf(series, dist, index, sgi, prob_zero)
54 cdf = compute_cdf_nsf(data=data)
55 ppf = norm.ppf(cdf)
---> 56 si.loc[data.index] = ppf
57 return si

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/indexing.py:849, in _LocationIndexer.setitem(self, key, value)
846 self._has_valid_setitem_indexer(key)
848 iloc = self if self.name == "iloc" else self.obj.iloc
--> 849 iloc._setitem_with_indexer(indexer, value, self.name)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/indexing.py:1837, in _iLocIndexer._setitem_with_indexer(self, indexer, value, name)
1835 self._setitem_with_indexer_split_path(indexer, value, name)
1836 else:
-> 1837 self._setitem_single_block(indexer, value, name)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/indexing.py:2077, in _iLocIndexer._setitem_single_block(self, indexer, value, name)
2074 self.obj._check_is_chained_assignment_possible()
2076 # actually do the set
-> 2077 self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
2078 self.obj._maybe_update_cacher(clear=True, inplace=True)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/internals/managers.py:394, in BaseBlockManager.setitem(self, indexer, value)
389 if using_copy_on_write() and not self._has_no_reference(0):
390 # if being referenced -> perform Copy-on-Write and clear the reference
391 # this method is only called if there is a single block -> hardcoded 0
392 self = self.copy()
--> 394 return self.apply("setitem", indexer=indexer, value=value)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
350 applied = b.apply(f, **kwargs)
351 else:
--> 352 applied = getattr(b, f)(**kwargs)
353 result_blocks = extend_blocks(applied, result_blocks)
355 out = type(self).from_blocks(result_blocks, self.axes)

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/internals/blocks.py:1041, in Block.setitem(self, indexer, value, using_cow)
1038 values = values.T
1040 # length checking
-> 1041 check_setitem_lengths(indexer, value, values)
1043 value = extract_array(value, extract_numpy=True)
1044 try:

File ~/miniconda3/envs/FluxNet/lib/python3.8/site-packages/pandas/core/indexers/utils.py:168, in check_setitem_lengths(indexer, value, values)
162 indexer = np.array(indexer)
163 if not (
164 isinstance(indexer, np.ndarray)
165 and indexer.dtype == np.bool_
166 and indexer.sum() == len(value)
167 ):
--> 168 raise ValueError(
169 "cannot set using a list-like indexer "
170 "with a different length than the value"
171 )
172 if not len(indexer):
173 no_op = True

ValueError: cannot set using a list-like indexer with a different length than the value`

I am running Python 3.8, Numpy 1.24.3, Pandas 2.0.3 and Scipy 1.10.1

ValueError: zero-size array to reduction operation minimum which has no identity

Hello!

I've been trying several datasets and I keep getting the same error:

Traceback (most recent call last):

Cell In[31], line 3
si.ssfi(pd.Series(df_mean_month['vazao']), dist=scs.fisk)

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/spei/si.py:251 in ssfi
ssfi.fit_distribution()

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/spei/si.py:394 in fit_distribution
fd = Dist(

File :7 in init

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/spei/dist.py:66 in post_init
pars, loc, scale = self.fit_dist(data=data_fit, dist=self.dist)

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/spei/dist.py:93 in fit_dist
fit_tuple = dist.fit(data, scale=std(data))

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2620 in fit
start = self._fitstart(data)

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2366 in _fitstart
loc, scale = self._fit_loc_scale_support(data, *args)

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2703 in _fit_loc_scale_support
data_a = np.min(data)

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/numpy/core/fromnumeric.py:2953 in min
return _wrapreduction(a, np.minimum, 'min', axis, None, out,

File ~/miniconda3/envs/spyder-env/lib/python3.12/site-packages/numpy/core/fromnumeric.py:88 in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

ValueError: zero-size array to reduction operation minimum which has no identity

Question: Time scale/span for SPEI1, SPEI3, SPEI6 etc.

What is the time scale of the calculated SPEI?

Problems to run the code with my own data.

Hello, how are you?

I'm having trouble running the code with my own data. Could you help me? When I run the cell related to the calculation of SPEI, I get the following error:

ValueError: zero-size array to reduction operation minimum which has no identity

To check if the error is a problem with my database, I exported your precipitation data and imported it again for testing. When I do this, the same error occurs as well. Do you have any clues as to why this error is happening?

Details of my code below:

#%% SPEI CODE

import spei as si # si for standardized index
import pandas as pd
import scipy.stats as scs
import matplotlib.pyplot as plt

Load Time Series

fileurl = "https://raw.githubusercontent.com/pastas/pastas/master/doc/examples/data"
prec = (
pd.read_csv(f"{fileurl}/rain_nb1.csv", index_col=0, parse_dates=True)
.squeeze()
.multiply(1e3)
) # to mm
evap = (
pd.read_csv(f"{fileurl}/evap_nb1.csv", index_col=0, parse_dates=True)
.squeeze()
.multiply(1e3)
) # to mm
head = pd.read_csv(f"{fileurl}/head_nb1.csv", index_col=0, parse_dates=True).squeeze()

fig, ax = plt.subplots(3, 1, figsize=(12, 8), sharex=True)
prec.plot(ax=ax[0], legend=True, grid=True)
evap.plot(ax=ax[1], color="C1", legend=True, grid=True)
head.plot(ax=ax[2], color="k", legend=True, grid=True);

Calculate SPI

f = 90 # days
series = prec.rolling(f, min_periods=f).sum().dropna()
series

spi3_gamma = si.spi(series, dist=scs.gamma, fit_freq="M")
spi3_gamma

tmin, tmax = pd.to_datetime(["1994", "1998"])
plt.figure(figsize=(8, 4))
spi3_gamma.plot(label="gamma")
plt.xlim(tmin, tmax)
plt.legend()
plt.grid()
plt.title("SPI");

f, ax = plt.subplots(1, 1, figsize=(12, 8))
si.plot.si(spi3_gamma, ax=ax)
ax.set_xlim(pd.to_datetime(["1994", "1998"]))
ax.grid()
ax.set_ylabel("SPI3", fontsize=14)
plt.show()

#%% SPEI CODE WITH MY DATA

prec.to_csv('prec.csv')

rainfall_teste = pd.read_csv("C:/Users/luizf/OneDrive/Doutorado POLI-USP/10- Análise de Dados Pluviométricos_R01/SPI/prec.csv", sep=',')

rainfall_teste = rainfall_teste.set_index('date')['rain']

rainfall_teste.to_csv('rainfall_teste')

Calculate SPI

f = 90 # days
series_rainfall_teste = rainfall_teste.rolling(f, min_periods=f).sum().dropna()
series_rainfall_teste

series_rainfall_teste.info()

spi3_gamma_rainfall_teste = si.spi(series_rainfall_teste, dist=scs.gamma, fit_freq="M")
spi3_gamma_rainfall_teste

ValueError: zero-size array to reduction operation minimum which has no identity