You've seen two basic time series models now, the random walk and white noise models. In this lesson, you'll learn about two other very important time series models that are widely used to understand and predict future values in stochastic processes: the Autoregressive (AR) and Moving Average (MA) models.
You will be able to:
- Explain what autoregressive means in an autoregressive model
- Explain what a moving average model means
- Describe how AR and MA can be combined to form an ARMA model
An autoregressive (AR) model is when a value from a time series is regressed on previous values from the same time series.
In words, the mathematical idea is the following:
Or, mathematically:
Some notes based on this formula:
- If the slope is 0, the time series is a white noise model with mean
$\mu$ - If the slope is not 0, the time series is autocorrelated
- Bigger slope means bigger autocorrelation
- When there is a negative slope, the time series follows an oscillatory process
We simulated some time series below. Have a look at them, and make sure this follows your intuition looking at the formula.
Note that simply having a value for phi (
Let's look at the autocorrelation plots as well.
The oscillatory process of the time series with
Next, let's look at the partial autocorrelation plots.
For each of these PACFs, we notice a high value for 1 lag, then autocorrelations of 0, except for the second one. This is no big surprise, as the slope parameter is fairly small, so the relationship between a value and the next one is fairly limited.
The Moving Average model can be described as the weighted sum of today's and yesterday's noise.
In words, the mathematical idea is the following:
Or, mathematically:
Some notes based on this formula:
- If the slope is 0, the time series is a white noise model with mean
$\mu$ - If the slope is not 0, the time series is autocorrelated and depends on the previous white noise process
- Bigger slope means bigger autocorrelation
- When there is a negative slope, the time series follow an oscillatory process
For the Moving Average Model we also simulated some time series with varying parameters below.
When there is a positive
Let's look at the ACF plots.
Remember that MA processes have autocorrelations, but because of the structure of the MA formula (regressing it on the noise term of the previous observation) there is only a dependence for one period, and the autocorrelation is zero for lags 2 and higher.
If
Next, let's look at the partial autocorrelation plots.
For PACFs, a typical structure is that there is a strong correlation with the 1-period lag (strength depending on
Let's look at the formulas of AR and MA again:
- AR:
$Y_t = \mu + \phi * Y_{t-1}+\epsilon_t$ - MA:
$Y_t = \mu +\epsilon_t + \theta * \epsilon_{t-1}$
Note that these models are constructed in a way that processes only depend directly on the previous observation in the process. These are known as "1st order models", and denoted by AR(1) and MA(1) processes respectively. Let's look at AR(2) and MA(2).
- AR(2):
$Y_t = \mu + \phi_1 * Y_{t-1}+\phi_2 * Y_{t-2}+\epsilon_t$ - MA(2):
$Y_t = \mu +\epsilon_t + \theta_1 * \epsilon_{t-1}+ \theta_2 * \epsilon_{t-2}$
Needless to say, this can be extended to higher-orders as well! Generally, the order of an AR model is denoted
A quick overview of how higher order models affect the ACF and PACF:
Considering a time series that was generated by an autoregression (AR) process with an order of
With a time series generated by a moving average (MA) process with an order
Now that we've seen AR and MA models, it is important to note that there is no reason why AR and MA models would not coexist. That's where ARMA models come in, which basically means that in this model, a regression on past values takes place (AR part) and also that the error term is modeled as a linear combination of error terms of the recent past (MA part). Generally, one denotes ARMA as ARMA(p, q).
An ARMA(2,1) model is given by:
A short table to summarize ACF and PACF for AR(p), MA(q), and ARMA(p, q):
AR(p) | MA(q) | ARMA(p, q) | |
---|---|---|---|
ACF | Tails off | Cuts off after lag q | Tails off |
PACF | Cuts off after lag p | Tails off | Tails off |
Seeing the table above, you might get an idea of why ACF and PACF are so useful when modeling! What you generally will try to do for any time series analysis is:
- Detrend your time series using differencing. ARMA models represent stationary processes, so we have to make sure there are no trends in our time series
- Look at ACF and PACF of the time series
- Decide on the AR, MA, and order of these models
- Fit the model to get the correct parameters and use for prediction
To learn more about AR, MA, and ARMA, have a look at lessons 1 and 2 here.
Great! Now that you have learned the basics of AR, MA, and ARMA models, let's look at some time series and how to model them in the next lesson!