timechop

Generate temporal validation time windows for matrix creation

Note: Timechop is now a bundled component of Triage, and future development will take place there. To utilize Timechop within your custom pipeline, you may still import it from there.

In predictive analytics, temporal validation can be complicated. There are a variety of questions to balance: How frequently to retrain models? Should the time between rows for the same entity in the train and test matrices be different? Keeping track of how to create matrix time windows that successfully answer all of these questions is difficult.

That's why we created timechop. Timechop takes in high-level time configuration (e.g. lists of train label spans, test data frequencies) and returns all matrix time definitions.

Timechop currently works with the following:

feature_start_time - data aggregated into features begins at this point
feature_end_time - data aggregated into features is from before this point
label_start_time - data aggregated into labels begins at this point
label_end_time - data aggregated is from before this point
model_update_frequency - amount of time between train/test splits
training_as_of_date_frequencies - how much time between rows for a single entity in a training matrix
max_training_histories - the maximum amount of history for each entity to train on (early matrices may contain less than this time if it goes past label/feature start times)
training_label_timespans - how much time is covered by training labels (e.g., outcomes in the next 1 year? 3 days? 2 months?)
test_as_of_date_frequencies - how much time between rows for a single entity in a test matrix
test_durations - how far into the future should a model be used to make predictions (in the typical case of wanting a single prediction set immediately after model training, this should be set to 0 days)
test_label_timespans - how much time is covered by test predictions (e.g., outcomes in the next 1 year? 3 days? 2 months?)

Here's an example of a typical set-up with a single prediction immediately after training and models built at an annual frequency:

from timechop.timechop import Timechop

chopper = Timechop(
    feature_start_time=datetime.datetime(1990, 1, 1, 0, 0), 
    feature_end_time=datetime.datetime(2017, 1, 1, 0, 0),
    label_start_time=datetime.datetime(2014, 1, 1, 0, 0),
    label_end_time=datetime.datetime(2017, 1, 1, 0, 0),
    model_update_frequency='1 year',
    training_as_of_date_frequencies=['6 months'],
    max_training_histories=['2 years'],
    training_label_timespans=['6 months'],
    test_as_of_date_frequencies=['1 days'],
    test_durations=['0 days'],
    test_label_timespans=['6 months']
)
result = chopper.chop_time()
print(result)

[
    {
        'feature_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'feature_start_time': datetime.datetime(1990, 1, 1, 0, 0),
        'label_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'label_start_time': datetime.datetime(2014, 1, 1, 0, 0),
        'test_matrices': [{
            'as_of_times': [
                datetime.datetime(2014, 7, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2014, 7, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2014, 7, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2015, 1, 1, 0, 0),
            'test_as_of_date_frequency': '1 days',
            'test_label_timespan': '6 months',
            'test_duration': '0 days'
        }],
        'train_matrix': {
            'as_of_times': [
                datetime.datetime(2014, 1, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2014, 1, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2014, 1, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2014, 7, 1, 0, 0),
            'max_training_history': '2 years',
            'training_as_of_date_frequency': '6 months',
            'training_label_timespan': '6 months'
        }
    },
    {
        'feature_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'feature_start_time': datetime.datetime(1990, 1, 1, 0, 0),
        'label_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'label_start_time': datetime.datetime(2014, 1, 1, 0, 0),
        'test_matrices': [{
            'as_of_times': [
                datetime.datetime(2015, 7, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2015, 7, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2015, 7, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2016, 1, 1, 0, 0),
            'test_as_of_date_frequency': '1 days',
            'test_label_timespan': '6 months',
            'test_duration': '0 days'
        }],
        'train_matrix': {
            'as_of_times': [
                datetime.datetime(2014, 1, 1, 0, 0),
                datetime.datetime(2014, 7, 1, 0, 0),
                datetime.datetime(2015, 1, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2015, 1, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2014, 1, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2015, 7, 1, 0, 0),
            'max_training_history': '2 years',
            'training_as_of_date_frequency': '6 months',
            'training_label_timespan': '6 months'
        }
    },
    {
        'feature_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'feature_start_time': datetime.datetime(1990, 1, 1, 0, 0),
        'label_end_time': datetime.datetime(2017, 1, 1, 0, 0),
        'label_start_time': datetime.datetime(2014, 1, 1, 0, 0),
        'test_matrices': [{
            'as_of_times': [
                datetime.datetime(2016, 7, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2016, 7, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2016, 7, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2017, 1, 1, 0, 0),
            'test_as_of_date_frequency': '1 days',
            'test_label_timespan': '6 months',
            'test_duration': '0 days'
        }],
        'train_matrix': {
            'as_of_times': [
                datetime.datetime(2014, 1, 1, 0, 0),
                datetime.datetime(2014, 7, 1, 0, 0),
                datetime.datetime(2015, 1, 1, 0, 0),
                datetime.datetime(2015, 7, 1, 0, 0),
                datetime.datetime(2016, 1, 1, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2016, 1, 1, 0, 0),
            'first_as_of_time': datetime.datetime(2014, 1, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2016, 7, 1, 0, 0),
            'max_training_history': '2 years',
            'training_as_of_date_frequency': '6 months',
            'training_label_timespan': '6 months'
        }
    }
]

And a second example with multiple testing dates and showing how the train matrices behave at the edge cases, showing the effects of some of the other paramters:

from timechop.timechop import Timechop

chopper = Timechop(
	feature_start_time=datetime.datetime(1990, 1, 1, 0, 0), 
    feature_end_time=datetime.datetime(2010, 1, 16, 0, 0),
    label_start_time=datetime.datetime(2010, 1, 1, 0, 0),
    label_end_time=datetime.datetime(2010, 1, 16, 0, 0),
    model_update_frequency='5 days',
    training_as_of_date_frequencies=['1 days'],
    max_training_histories=['5 days'],
    training_label_timespans=['1 day'],
    test_as_of_date_frequencies=['3 days'],
    test_durations=['5 days'],
    test_label_timespans=['3 days']
)
result = chopper.chop_time()
print(result)

[
    {
        'feature_end_time': datetime.datetime(2010, 1, 16, 0, 0),
        'feature_start_time': datetime.datetime(1990, 1, 1, 0, 0),
        'label_end_time': datetime.datetime(2010, 1, 16, 0, 0),
        'label_start_time': datetime.datetime(2010, 1, 1, 0, 0),
        'test_matrices': [{
            'as_of_times': [
                datetime.datetime(2010, 1, 3, 0, 0),
                datetime.datetime(2010, 1, 6, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2010, 1, 6, 0, 0),
            'first_as_of_time': datetime.datetime(2010, 1, 3, 0, 0),
            'matrix_info_end_time': datetime.datetime(2010, 1, 9, 0, 0),
            'test_as_of_date_frequency': '3 days',
            'test_label_timespan': '3 days',
            'test_duration': '5 days'
        }],
        'train_matrix': {
            'as_of_times': [
                datetime.datetime(2010, 1, 1, 0, 0),
                datetime.datetime(2010, 1, 2, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2010, 1, 2, 0, 0),
            'first_as_of_time': datetime.datetime(2010, 1, 1, 0, 0),
            'matrix_info_end_time': datetime.datetime(2010, 1, 3, 0, 0),
            'max_training_history': '5 days',
            'training_as_of_date_frequency': '1 days',
            'training_label_timespan': '1 day'
        }
    },
    {
        'feature_end_time': datetime.datetime(2010, 1, 16, 0, 0),
        'feature_start_time': datetime.datetime(1990, 1, 1, 0, 0),
        'label_end_time': datetime.datetime(2010, 1, 16, 0, 0),
        'label_start_time': datetime.datetime(2010, 1, 1, 0, 0),
        'test_matrices': [{
            'as_of_times': [
                datetime.datetime(2010, 1, 8, 0, 0),
                datetime.datetime(2010, 1, 11, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2010, 1, 11, 0, 0),
            'first_as_of_time': datetime.datetime(2010, 1, 8, 0, 0),
            'matrix_info_end_time': datetime.datetime(2010, 1, 14, 0, 0),
            'test_as_of_date_frequency': '3 days',
            'test_label_timespan': '3 days',
            'test_duration': '5 days'
        }],
        'train_matrix': {
            'as_of_times': [
                datetime.datetime(2010, 1, 2, 0, 0),
                datetime.datetime(2010, 1, 3, 0, 0),
                datetime.datetime(2010, 1, 4, 0, 0),
                datetime.datetime(2010, 1, 5, 0, 0),
                datetime.datetime(2010, 1, 6, 0, 0),
                datetime.datetime(2010, 1, 7, 0, 0)
            ],
            'last_as_of_time': datetime.datetime(2010, 1, 7, 0, 0),
            'first_as_of_time': datetime.datetime(2010, 1, 2, 0, 0),
            'matrix_info_end_time': datetime.datetime(2010, 1, 8, 0, 0),
            'max_training_history': '5 days',
            'training_as_of_date_frequency': '1 days',
            'training_label_timespan': '1 day'
        }
    }
]

The output of Timechop works as input to the architect.Planner.

jasonyanwenl / timechop Goto Github PK

timechop's Introduction

timechop

timechop's People

Contributors

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent