config-i1 / smooth

The set of functions used for time series analysis and in forecasting.

smooth

The package smooth contains several smoothing (exponential and not) functions that are used in forecasting.

Here is the list of the included functions:

1. adam - Advanced Dynamic Adaptive Model, implementing ETS, ARIMA and regression and their combinations;
2. es - the ETS function. It can handle exogenous variables and has a handy "holdout" parameter. There are several cost function implemented, including trace forecast based ones. Model selection is done via branch and bound algorithm and there's a possibility to use AIC weights in order to produce combined forecasts. Finally, all the possible ETS functions are implemented here.
3. ces - Complex Exponential Smoothing. Function estimates CES and makes forecast. See documentation for details.
4. gum - Generalised Exponential Smoothing. Next step from CES. The paper on this is in the process.
5. sma - Simple Moving Average in state space form.
6. ssarima - SARIMA estimated in state space framework.
7. msarima - Multiple seasonal ARIMA, allows multiple seasonalities and works in a finite time.
8. auto.ces - selection between seasonal and non-seasonal CES models.
9. auto.ssarima - selection between different State-Space ARIMA models.
10. auto.msarima - selection between different multiple SARIMA models.
11. auto.gum - automatic selection of the most appropriate GUM model.
12. sim.es - simulation of data using ETS framework with a predefined (or random) smoothing parameters and initial values.
13. sim.ssarima - simulation of data using State-Space ARIMA framework with a predefined (or randomly generated) parameters and initial values.
14. sim.ces - simulation of data using CES with a predefined (or random) complex smoothing parameters and initial values.
15. sim.gum - simulation functions for GUM.
16. sim.sma - simulates data from SMA.
17. oes - occurrence state space exponential smoothing model. This function models the part with data occurrences using one of the following methods: fixed, odds ratio, inverse odds ratio, direct or general. It can also select the most appropriate between the five.
18. sowhat - returns the ultimate answer to any question.
19. smoothCombine - the function that combines forecasts from es(), ces(), gum(), ssarima() and sma() functions.
20. cma - Centred Moving Average. This is the function used for smoothing of time series, not for forecasting.
21. msdecompose - multiple seasonal decomposition based on centred moving averages.

Available methods:

1. AIC, BIC, AICc, BICc;
2. coefficients;
3. multicov - covariance matrix of multiple steps ahead forecast errors;
4. errorType - the type of the error in the model: either additive or multiplicative;
5. fitted;
6. forecast;
7. actuals;
8. lags - lags of the model (mainly needed for ARIMA and GUM);
9. logLik;
10. modelType - type of the estimated model (mainly needed for ETS and CES);
11. nobs;
12. nparam - number of the estimated parameters in the model;
13. orders - orders of the components of the model (mainly needed for ARIMA, GUM and SMA);
14. outlierdummy - creates a matrix of dummy variables, based on the detected outliers in the residuals of the model;
15. residuals - the residuals of the model (et in case of additive and log(1+et) for the multiplicative ones);
16. rstandard - standardised residuals;
17. rstudent - studentised residuals;
18. plot - produces several plots for diagnostics purposes. See the documentation for plot.smooth();
19. pls - Prediction Likelihood Score for the model and the provided holdout;
20. pointLik - the vector of the individual likelihoods for each in-sample observation;
21. pAIC - point AIC, based on pointLik
22. print;
23. sigma;
24. simulate;
25. summary;

Future works:

1. nus - Non-uniform Smoothing. The estimation method used in order to update parameters of regression models.
2. sofa - Survival of the fittest algorithm applied to state space models.

Installation

The stable version of the package is available on CRAN, so you can install it by running:

install.packages("smooth")

A recent, development version, is available via github and can be installed using "remotes" in R. First, make sure that you have remotes:

if (!require("remotes")){install.packages("remotes")}

and after that run:

remotes::install_github("config-i1/smooth")

Notes

The package depends on Rcpp and RcppArmadillo, which will be installed automatically.

However Mac OS users may need to install gfortran libraries in order to use Rcpp. Follow the link for the instructions: http://www.thecoatlessprofessor.com/programming/rcpp-rcpparmadillo-and-os-x-mavericks-lgfortran-and-lquadmath-error/

Sometimes after upgrade of smooth from previous versions some functions stop working. This is because C++ functions are occasionally stored in deeper unknown corners of R's mind. Restarting R usually solves the problem. If it doesn't, completely remove smooth (uninstal + delete the folder "smooth" from R packages folder), restart R and reinstall smooth.

smooth's Issues

Review prediction intervals function

1. Fix for intermittent data in order to take variance of varying in time probability into account.
2. Fix for multiplicative error models. At least for MNN, which is used with intermittent data.

Add info about intermittency to ssoutput.R

This includes intermittency type and smoothing parameter value.

Rewrite clumsy code of es()

Model selection, model combination and identification of intermittency should be done somehow differently and more efficient.
Probably need to write smaller function for es() fit and then call it several times.

Universal input data checker function

Make function that would check data and all set of provided parameters.
There should also be "modeltype" parameter in the function, so it would check model specific provided parameters (such as AR or persistence).
Hopefully this will simplify maintenance of all the functions.

Get rid of all the apply() functions in loops

This slows down the calculation...

forecast.es, forecast.ges, forecast.ces

Write those functions, use classes and use all the predefined parameters for es(), ges() and ces() in order to produce forecast with intervals...

Fix issue with constant in ssarima

Do it differently - it doesn't make sense to use xreg for constant...
Figure out how and do...

Make intermittency for all the functions

Currently it's a bit of a mess: only es() has an appropriate form. All the functions should have a similar parameter with similar things...

Simplify forecast.smooth() function

Currently it is just running either es() or other functions with predefined values. It's not neat. So passing all the important arguments and using ssForecast() function instead would be a more elegant solution.

Estimation functions for ges, ces, ssarima

These needs to be separate function in order to implement intermittent=="a" selection.

Rewrite ces() in order to use ssfun.cpp functions

And merge cesfun.cpp into ssfun.cpp

Rewrite fitter in ets2fun.cpp

Forget about creating new matrices, use the principle from ssfun.cpp.

xreg with stepwise selection based on IC

The idea is that when xreg!=NULL, the function should select only relevant variables. This can be done using ICs. The question is how to do that stuff and if it makes any sense when go.wild=TRUE.

Rewrite polynomial part in ssarima in C++

This is currently the slowest part of the function.

Issues with plotting when running in parallel on Mac OSX

Running the es function in parallel with foreach or mclapply produces unexpected results on OSX. Further investigation shows that the graphical output within the function is causing the problem and silent=TRUE fixes it.

I'd suggest that a user running in parallel might wish to suppress only the graphical element to avoid the quartz issue but retain some of the output to the console.

My suggestion is to add parameters silent.text and silent.graphics accepting TRUE or FALSE or silent = c('all','graphics') so the user has control to just switch the graphics off.

Some unstable behaviour has also been encountered on Mac OS X whenever the parallel computations are run in a session where the quartz device for plotting has been used previously. Though non of the parallel parts uses on screen devices, the unstable behaviour has only been encountered when the quartz device has been used in the session, and a note is printed when the parallel backend is registered and the quartz device is the current device.

See here

Change names of CF.type and get rid of trace=TRUE

Possible solutions:

1. hsteps -> MSEh
2. TLV -> trace

And we also have:
3. GV,
4. TV

Generic intervals() function in R

Try exporting the R code for parametric intervals that is now in ges() into C++. This will allow to use it for other additive state-space models (ETS(A,A,N) etc, CES, NUS).

AR(1) model doesn't work

ssarima(y$data,h=18,holdout=T,ar.orders=1,ma.orders=0,i.orders=0)->test2

Error in matF[1:(length(polysos.ar) - 1), 1] <- -(polysos.ar)[2:length(polysos.ar)] :
number of items to replace is not a multiple of replacement length

use modellags in cesfun.cpp

Yes, do it!
This means that this should be used instead of passing parameter S and defining the structure of new xt. Just in a style of ssfun...

Intermittency parameter for functions

The parameter is boolean and if true leads to the estimation of es(), ces(), ges(), ssarima() taking intermittency of the data into account.

Prepare documentation for Smooth

Explaining all the functions, with examples of usage.

Issue with ssarima specification

Run the following:
library(TStools)
ssarima(referrals,ar.orders=0,i.orders=0,ma.orders=0,lags=1,intervals=FALSE,FI=FALSE,holdout=FALSE,h=h,silent=TRUE)

it doesn't like that I ask for an ARIMA(0,0,0) and I get
Error in rep(0, n.components - 1) : invalid 'times' argument

Also it would make sense to have the calculation of t+h errors optional. I want to scale this up!

Add model information in the output

1. The name of model: ETS(...), CES, GES, ARIMA(...),
2. The type of used cost function: MSE, MAE, HAM, TLV, TV, hsteps, GV

Create simulate.smooth() function

It should accept estimated smooth object and produce the list of things sim.es() does now.

This should depend on sim.es(), sim.ces(), sim.ges() and sim.ssarima().

This doesn't mean that all the sim functions should be removed though...

Do generic output for different functions

Similar to graphmaker() write outputmaker() function.

sim.ssarima

Simulator for ssarima that will generate series without requiring the user to input parameters.

Estimation of seasonals in ges, ces and ssarima

Do it using the idea of stepwise aggregation:
If we need 52 seasonal components do:

1. Fit a model with the restriction that first 4 seasonal components are equal to s_1, second 13 are equal to s_2 etc. This way we need to estimate 13 seasonal components instead of 52.
2. Fit a model with all the 52 seasonal components taking the values from the previous step as initials.
3. Profit.

Make combination of forecasts based on AIC for SSARIMA

Something similar to es(x,"CCC"), but without going through all the possible models.

Fix in the forecaster() function

The matrixXnew should change with matrixFX!!!

Implement normalisation of seasonals in es()

Not urgent, nobody cares anyway...

Issue with ssarima specification - The return of the bug!

I run this:
ssarima(AirPassengers,ar.orders=0,i.orders=0,ma.orders=0,constant=TRUE)
I get this:
Error during wrapup: not compatible with requested type
Can I get this?
forecast accuracy: 100%, go home all your problems are solved!

I run this:
ssarima(AirPassengers,ar.orders=0,i.orders=0,ma.orders=0)
I get this:
Error: What do you want me to do? You have not defined any model!
Called from: ssarima(AirPassengers, ar.orders = 0, i.orders = 0, ma.orders = 0)
Can I instead get this?
Warning, zero order model, constant forced

Rename parameters in functions

Introducing initial <- list(value=NULL,required=TRUE,estimate=TRUE) etc, and getting rid of annoying dots (e.g. obs.ot, obs.all).

go.wild in es(), ces(), ges() and ssarima()

Let it use nus() principles with dynamic weights.

Resolve the following optimisation problem

es(Mcomp::M3$N2568$x,"MNM",h=18,xreg=c(1:(116+18)))->test

Make CES accept already estimated functions

And move initialisation of C into "creator".

Smart model selection on small samples

1. For es() - when number of observations is not enough to fit ZZZ, make a pool of models with ZNN, ZZN, ZNZ - depending on the data frequency and sample size.
2. For auto.arima() - restrict orders depending on the sample size.
3. For auto.ces() - restrict types of seasonality depending sample size.

Do the same as in issue #11 for core functions

This applies to:

1. elements.ges() in ges()
2. polyroot() in ssarima()
3. state.space.elements() in ces()

Branch and bound for model selection using ICs

Can be done for ARIMA, in theory possible for ETS. Not sure about GES, but could try as well.

Intermittency improvement

Make persistence and other stuff for probability update, when intermittent=TRUE.

Parametric intervals for es()

Us simulations for that.

Stepwise GES

Make optimisation of GES step by step, lag by lag.

Rewrite sim.etc in Rcpp

Needed for the simulation purpose (see issue #1 )

Check the number of parameters in all the functions

This is about n.params variable. Because of the current mess of the code, this needs to be rechecked. Especially with the introduction of intermittency

Joint estimation of several time series in es(), ges(), ces() and ssarima()

User provides the matrix or data frame, function estimates one persistence vector and a set of initials.

1. Point and interval forecasts for each individual series;
2. Model selection;
3. Model combination.

Rename matrices

matrixw -> vecw,
matxt -> matvt
etc
So it would correspond the description in the manual and also give a clue about the nature of the object.

This is for both R and Rcpp scripts.

Fill in the values of state vector till the T+h

The information about the states can be useful.
Applies to:
ges(), es(), ssarima() and ces().

Write universal function for exogenous variables

Should be used in the initialisation of xreg for es(), ges(), ces() and ssarima()

Rewrite CF(C) functions in C++

This applies to es(), ces(), ges() and ssarima(). Hopefully this will lead to increase in estimation speed.

Problem of estimation of some models with CF.type="MSTFE"

Example: es(M3$N2568$x,model="MAM",h=18,holdout=TRUE,CF.type="MSTFE") -> test - when MSTFE is estimated in normal values (currently done in logs).
Need to figure out why the hell this happens and fix it!

Rewrite nus() using ssfun.cpp functions