guannan / modelselection

Tutorial on model assessment, model selection and inference after model selection

Model assesment, selection and inference after selection

Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.

Talks

• Use of reference models in variable selection at Laplace's demon seminar series
  • Video
• Model assessment, comparison and selection at Master class in Bayesian statistics, CIRM, Marseille
  • Slides
  • Video
• Model assessment and model selection aka Basics of cross-validation tutorial at StanCon 2018 Helsinki
  • Slides
  • Video
  • Code for figures
• Regularized horseshoe talk at StanCon 2018 Asilomar
  • Slides
  • Video
• Model selection tutorial at StanCon 2018 Asilomar
  • Slides
  • Video

Outline of the StanCon 2018 Asilomar tutorial and links to notebooks

• Basics of predictive performance estimation
• When cross-validation is not needed
  • Simple model we trust - betablockers
• When cross-validation is useful
  • We don't trust the model - roaches
  • Complex model with posterior dependencies - collinear
• On accuracy of cross-validation
• Cross-validation and hierarchical models
• When cross-validation is not enough
  • large number of models - diabetes
• loo 2.0
• Projection predictive model selection
  • collinear
  • diabetes
  • projpred

Additional case studies

• loo 2.0
  • Bayesian Stacking and Pseudo-BMA weights using the loo package
  • Leave-one-out cross-validation for non-factorizable models
  • Approximate leave-future-out cross-validation for time series models
• Projection predictive model selection (projpred) examples
  • collinearity - mesquite
  • random data vs original data - candy
  • winequality-red
  • bodyfat
  • See also projpred quick start vignette
• LOO-R^2

See also

• When LOO and other cross-validation approaches are valid
• Parsimonious principle vs integration over all uncertainties
• Comments on Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection
• I am the supercargo
• Nice horseshoe example in Bayes Sparse Regression case study by Michael Betancourt

References

• Afrabandpey, H., Peltola, T., Piironen, J., Vehtari, A., and Kaski, S. (2019). Making Bayesian predictive models interpretable: A decision theoretic approach. arXiv preprint arXiv:1910.09358
• Bürkner, P.-C., Gabry, J., Vehtari, A. (2018). Leave-one-out cross-validation for non-factorizable normal models. arXiv:1810.10559
• Bürkner, P.-C., Gabry, J., Vehtari, A. (2020). Approximate leave-future-out cross-validation for time series models. Journal of Statistical Computation and Simulation, doi:10.1080/00949655.2020.1783262. Online. Preprint arXiv:1902.06281
• Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997–1016. Preprint
• Gelman, A., Goodrich, B., Gabry, J., and Vehtari, A. (2018). R-squared for Bayesian regression models. The American Statistician, doi:10.1080/00031305.2018.1549100. Online.
• Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2019). Bayesian leave-one-out cross-validation for large data. Thirty-sixth International Conference on Machine Learning, PMLR 97:4244--4253. Online.
• • Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2020). Leave-one-out cross-validation for Bayesian model comparison in large data. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 108:341-351. Online. preprint arXiv:2001.00980.
• Paananen, T., Piironen, J., Bürkner, P.-C., and Vehtari, A. (2020). Implicitly adaptive importance sampling. arXiv:1906.08850
• Pavone, F., Piironen, J., Bürkner, P.-C., and Vehtari, A- (2020). Using reference models in variable selection. arXiv preprint arXiv:2004.13118
• Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian predictive methods for model selection, Statistics and Computing 27(3), 711–735. Online
• Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:905-913. Online
• Piironen, J., and Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors. In Electronic Journal of Statistics, 11(2):5018-5051. Online
• Piironen, J., and Vehtari, A. (2018). Iterative supervised principal components. Proceedings of the 21th International Conference on Artificial Intelligence and Statistics, accepted for publication. arXiv preprint arXiv:1710.06229
• Piironen, J., Paasiniemi, M., and Vehtari, A. (2020). Projective Inference in High-dimensional Problems: Prediction and Feature Selection. Electronic Journal of Statistics, 14(1):2155-2197. Online. Preprint arXiv:1810.02406
• Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5):1413–1432. arXiv preprint.
• Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2019). Pareto smoothed importance sampling. arXiv preprint.
• Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models. JMLR, 17(103):1–38. Online
• Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive methods for model assessment, selection and comparison, Statistics Surveys 6, 142–228. Online
• Williams, D. R., Piironen, J., Vehtari, A., and Rast, P. (2018). Bayesian estimation of Gaussian graphical models with projection predictive selection. arXiv:1801.05725
• Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091, Online