Coder Social home page Coder Social logo

dsc-removing-trends-lab-seattle-ds-080519's Introduction

Removing Trends - Lab

Introduction

In this lab, you'll practice your detrending skills!

Objectives

In this lab you will:

  • Use a log transformation to minimize non-stationarity
  • Use rolling means to reduce non-stationarity
  • Use differencing to reduce non-stationarity
  • Use rolling statistics as a check for stationarity
  • Create visualizations of transformed time series as a visual aid to determine if stationarity has been achieved
  • Use the Dickey-Fuller test and conclude whether or not a dataset is exhibiting stationarity

Detrending the Air passenger data

In this lab you will work with the air passenger dataset available in 'passengers.csv'. First, run the following cell to import the necessary libraries.

# Import necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pylab as plt
%matplotlib inline
  • Import the 'passengers.csv' dataset
  • Change the data type of the 'Month' column to a proper date format
  • Set the 'Month' column as the index of the DataFrame
  • Print the first five rows of the dataset
# Import 'passengers.csv' dataset
data = None

# Change the data type of the 'Month' column
data['Month'] = None

# Set the 'Month' column as the index
ts = None

# Print the first five rows

Plot this time series.

# Plot the time series

Create a stationarity check

Your next task is to use the code from previous labs to create a function stationarity_check() that takes in a time series and performs stationarity checks including rolling statistics and the Dickey-Fuller test.

We want the output of the function to:

  • Plot the original time series along with the rolling mean and rolling standard deviation (use a window of 8) in one plot
  • Output the results of the Dickey-Fuller test
# Create a function to check for the stationarity of a given time series using rolling stats and DF test
# Collect and package the code from previous labs

Use your newly created function on the ts timeseries.

# Code here

Perform a log and square root transform

Plot a log transform of the original time series (ts).

# Plot a log transform

Plot a square root transform of the original time series (ts).

# Plot a square root transform

Going forward, let's keep working with the log transformed data before subtracting rolling mean, differencing, etc.

Subtracting the rolling mean

Create a rolling mean using your log transformed time series, with a time window of 7. Plot the log-transformed time series and the rolling mean together.

# your code here
roll_mean = None
fig = plt.figure(figsize=(11,7)) 

Now, subtract this rolling mean from the log transformed time series, and look at the 10 first elements of the result.

# Subtract the moving average from the log transformed data
data_minus_roll_mean = None

# Print the first 10 rows

Drop the missing values from this time series.

# Drop the missing values

Plot this time series now.

# Plot the result

Finally, use your function check_stationarity() to see if this series is stationary!

# Your code here

Based on the visuals and on the Dickey-Fuller test, what do you conclude?

# Your conclusion here

Subtracting the weighted rolling mean

Repeat all the above steps to calculate the exponential weighted rolling mean with a halflife of 4. Start from the log-transformed data again. Compare the Dickey-Fuller test results. What do you conclude?

# Calculate Weighted Moving Average of log transformed data
exp_roll_mean = None

# Plot the original data with exp weighted average
  • Subtract this exponential weighted rolling mean from the log transformed data
  • Print the resulting time series
# Subtract the exponential weighted rolling mean from the original data 
data_minus_exp_roll_mean = None

# Plot the time series

Check for stationarity of data_minus_exp_roll_mean using your function.

# Do a stationarity check

Based on the visuals and on the Dickey-Fuller test, what do you conclude?

# Your conclusion here

Differencing

Using exponentially weighted moving averages, we seem to have removed the upward trend, but not the seasonality issue. Now use differencing to remove seasonality. Make sure you use the right amount of periods. Start from the log-transformed, exponentially weighted rolling mean-subtracted series.

After you differenced the series, drop the missing values, plot the resulting time series, and then run the stationarity check() again.

# Difference your data
data_diff = None

# Drop the missing values


# Check out the first few rows
data_diff.head(15)

Plot the resulting differenced time series.

# Plot your differenced time series
# Perform the stationarity check

Your conclusion

# Your conclusion here

Summary

In this lab, you learned how to make time series stationary through using log transforms, rolling means, and differencing.

dsc-removing-trends-lab-seattle-ds-080519's People

Contributors

loredirick avatar alexgriff avatar sumedh10 avatar taylorhawks avatar

Watchers

James Cloos avatar Kevin McAlear avatar  avatar Mohawk Greene avatar Victoria Thevenot avatar Belinda Black avatar Bernard Mordan avatar raza jafri avatar  avatar Joe Cardarelli avatar The Learn Team avatar Sophie DeBenedetto avatar  avatar  avatar Matt avatar Antoin avatar  avatar  avatar Amanda D'Avria avatar  avatar Nicole Kroese  avatar Kaeland Chatman avatar Lisa Jiang avatar Vicki Aubin avatar Maxwell Benton avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.