sdtaylor / pyphenology

Plant phenology models in python with a scikit-learn inspired API

pyPhenology

Plant phenology models in python with a scikit-learn inspired API

Full documentation

http://pyphenology.readthedocs.io/en/master/

Installation

Requires: scipy, pandas, joblib, and numpy

Install via pip

pip install pyPhenology

Or install the latest version from Github

pip install git+git://github.com/sdtaylor/pyPhenology

Usage

A Thermal Time growing degree day model:

from pyPhenology import models, utils
observations, predictors = utils.load_test_data(name='vaccinium')
model = models.ThermalTime()
model.fit(observations, predictors)
model.get_params()
{'t1': 85.704951490688927, 'T': 7.0814430573372666, 'F': 185.36866570243012}

Any of the parameters in a model can be set to a fixed value. For example the thermal time model with the threshold T set to 0 degrees C

model = models.ThermalTime(parameters={'T':0})
model.fit(observations, predictors)
model.get_params()
{'t1': 26.369813953905265, 'F': 333.76534368004388, 'T': 0}

Citation

If you use this software in your research please cite it as:

Taylor, S. D. (2018). pyPhenology: A python framework for plant phenology modelling. Journal of Open Source Software, 3(28), 827. https://doi.org/10.21105/joss.00827

Bibtex:

@article{Taylor2018,
author = {Taylor, Shawn David},
doi = {10.21105/joss.00827},
journal = {Journal of Open Source Software},
mendeley-groups = {Software/Data},
month = {aug},
number = {28},
pages = {827},
title = {{pyPhenology: A python framework for plant phenology modelling}},
url = {http://joss.theoj.org/papers/10.21105/joss.00827},
volume = {3},
year = {2018}
}

Acknowledgments

Development of this software was funded by the Gordon and Betty Moore Foundation's Data-Driven Discovery Initiative through Grant GBMF4563 to Ethan P. White.

pyphenology's Issues

up scipy req to >= 1.2.0

to incorporate parallelism in differential evolution

add warning/error for dropping data

pyPhenology/pyPhenology/models/utils.py

Line 174 in 2ab71e9

if drop_missing:

-when NA stuff is dropped here (due to missing weather data, probably stuff over water), then issue an error cause it then won't match predictions.
-or, probably better, drop sites w/out weather data in the validation

documentation

• badges
• all primary functions/classes documented
• short vignette/site
• one more short example
  rmse of all models on held out data
• seed optimizer parameter
• graph showing timings
• julien date in the data structure needs completion
• basin hopping optimizer

save all model iterations in WeightedEnsemble

Currently only the final models are saved. but to refit the weights with the newest data will require all the old iterations.

issue warning when latitude/longitude is in species observations

the naive model expects these in the predictor data, but throws an error if they're in both predictor and observation data. Both are kinda reasonable.

setup sensible optimizer defaults

Sensible defaults have been setup for for the differential evolution and basin hopping for various scenarios.

• testing A quick method meant for testing code
• practical Should produce realistic results on desktop systems in a short period. (name of this open to suggestions)
• intensive Designed to find the absolute optimal solution. Can potentially take hours to days when there is a large number of observations.

python installation

• installer
• conda-forge
• pypi

propogate model error in bootstrap method

should be able to have a 'none' aggregation method in the boostrap prediction function

pyPhenology/pyPhenology/models/ensemble_models.py

Line 41 in 30445f7

def predict(self,to_predict=None, temperature=None, aggregation='mean', **kwargs):

independent score class

need a scorer that is outside of the model class.

improve testing

-have a single fit/predict test for the individual models to test their only real method (_apply_model)

-pick one model to do most of the testing with, each within a test__ function for pytest

-use @pytest.mark.parametrize to iterate over known value testing

• 1 test_ script for each of those ^

some simple tests to impliement

various checks to do here here data loading functions

https://codecov.io/gh/sdtaylor/pyPhenology/src/300864e2a38356c2c36d4110c2478db35a90b0e2/pyPhenology/utils.py

fix ensemble nestedness for fit methods

Like #113 but for fit()

other optimizer methods

• differential evolution
• simulated annealing
• basin hopping
  BH in scipy doesn't constrain to (low,high), so will have to do that manually
• full search /brute force

put equal weight option in ensemble

This will simplify having an ensemble of bootstrapped models.

sequential model issue

triangle response should work the same with the following two setups but it doesn't

    temperature[right_side] -= t_opt
    temperature[right_side] /= (t_max - t_opt)

    temperature[right_side] -= t_max
    temperature[right_side] /= (t_opt - t_max)

spring warming and parallel models

spring warming model is essentially a uniforc model w/ some fixed params

parallel model is mix of the alternating and parallel model (ie. triangle response for chilling and an exponential function for F)

see implementation in Melaas et al. 2016
but note these are implemented slightly differently in basler 2016 (and probably other papers)

configurable optimization

differential evolution, simulated annealing, full search, etc, with configurable parameters for all

change saved model format to json

Need to be able to store the model type in the file, and whether it's a bootstrap model or no.

would give the ability to load a model with a generic function like

model = utils.load_saved_model('params_file.json')

To that end it would also be good to have a descriptor in the model class available, such as

model.model_metadata
{'model_type':'ThermalTime',
'parameters_fit':True,
etc

or

model.model_type
{'model_type':'Bootstrap',
'core_model':'ThermalTime',
'num_bootstraps':50,
'parameters_fit':True

parallelization

for the ensemble models, make internal parallelization an option by passing the number of cores to use. but also make more complex parallel stuff (MPI, DASK) and option by having a method which just passes each job as an iter object.

put score method in bootstrap model

travis testing

create test cases and have testing on travis

general data cleaning methods

potentially methods to clean some raw data and put it in a format used by the package

• takes dates of phenophases (a la NPN) and convert to DOY with some filter

need more debug statements in the ensemble methods

setting debug=True in the bootstrap and weighted ensemble models doesn't give any useful info.

don't require doy column in model.predict()

model.predict() enforces the to_predict dataframe having a doy column, even though I'm pretty sure it's not actually used.

make version method

like pyPhenology.__version__

easily done in the retriever https://github.com/weecology/retriever/blob/master/retriever/_version.py

but potentially more automated like in xarray https://github.com/pydata/xarray/blob/master/xarray/_version.py

update travis badge in docs

now using gh actions

update docs

lots of updates needed for new data structure, and new ways to save/load files

• from #52
• from #57, #58

fix bootstrap prediction

bootstrapmodel.predict() without any arguments should return the prediction of the input data. but after some thought I realized that isn't actually working since the input data is being shuffled underneath. so I need to fix this.

boostrapmodel.predict(newstuff = blah blah), ie. with new data should be working.

models to add

• uniforc
• unichill
• alternating
• macro scale budburst
• m1 - requires daylength calculation and associated site_info
• linear
• sequential
• parallel
• DORMPHOT

add schwartz spring index models

Like what the NPN map use. These would be "fixed" in that they couldn't be fit on new data, but can only be used for predictions.

split out models to their own files

ensemble model

need an ensemble model to combine different core models. probably with a few different methods. simple means, bayesian model averaing, AIC weights.

make a base ensemble class

There are a lot of duplicated things in ensemble_models.py. especially with the newly added Ensemble()

use TypeError

I think a lot of my assert statements can be replaced with raise TypeError to be more descriptive

add in model fitting time

would be useful for debugging or optimizing
this is mostly dependent on the timing for _apply_model()

be more clear about data structure

I've pulled the data structures straight from my original dataset study models, so I need to clarify (and probably provide code for) how things get converted to that structure.

deal with warnings

wow geez, a lot of new warnings from numpy and scipy to maintain

need parameter check in bootstrap model

if parameters is not a saved file to load then core_model and num_bootstraps must be set

add docs contact/help section

Either my email, github issues page, and/or maybe a google docs form

add a few more example datasets

Like ones with more sites, or a lot of data.

make missing temperature check more robust

If there are unexpected columns in the obs data.frame with NA's, they will be dropped due to "lack" of temp data, cause the super long temp data.frame will end up with NA's

incorporate weather data retrieval

accept a list of lat/long/ years, check to see if a nearby station is available, and download and format weather data to pyPhenology format

utils.get_weather_data('site_info.csv', data_file='', temp_folder='', accept_all=False, distance_buffer)

distance_buffer:
allowed distance from a site to a weather station

accept_all:
be prompted before downloading large files (can exceed 100mb). or proceeding if

data_file:
final weather data file to be written. will be in the format used in this package.

make linear model a bit better

instead of having start date and end date for spring, do start date and length. that way it's be easier to have them as parameters to estimate.

put in check for bad fits

If fitted parameters produce a bunch of non-predictions(doy 999), then put in a warning saying so.

maybe:
"Model did not converge well, doy values of 999 in prediction output. Perhaps try with optimizer_params='intensive', or with fewer parameters to estimate".