A. Henzi, M. Puke, T. Dimitriadis, and J. Ziegel (2023). A safe Hosmer-Lemeshow test. Preprint, arXiv:2203.00426.
-
The simulation code can be found in the simulation folder. To execute the simulation, please run the
sim_HLstyle_oos.R
script. The results from each replication will be saved in an rds file. In order to carry out the simulation, the script relies on the functions contained in thenle.R
,eHL.R
, andoracle_eHL.R
files located in the main folder. Those files do the following:-
nle.R
: Thenle.R
function facilitates the generation of data for the simulation study of the Hosmer-Lemeshow type calibration test. Using this function, one can calculate the parameters$\beta$ , which are described in section 3 of the document. -
eHL.R
: TheeHL.R
function is responsible for calculating the e-value by utilizing the sample split and the isotonic/monotonic recalibrated probabilities. It is based on Algorithm 2, which is described in Section 2.5 of the document. -
oracle_eHL.R
: Calculates the optimal (hypothetical) e-value.
-
-
To obtain the results presented in Table 1 of the document, please refer to the
table.R
script. Similarly, to generate Figure 1, please run theplots_simulation_main.R
script. -
The
sim_instability_appendix.R
script generates the results presented in Figure 4 of the document. To access the results from each run, please refer to the correspondingrds
file.
-
In application, we analyze (re-)calibration of probability predictions for the binary event of credit card defaults in Taiwan in the year 2005. The results can be replicated using the file
credit.R
. -
The data analyzed here was obtained from the UCI Machine Learning Repository. To reproduce the study’s findings, it is necessary to access and download the dataset from the aforementioned webpage.
The software versions that were used to run these analyses are
- R 4.2.1
readxl
(1.4.1)dplyr
(1.0.10)tidyr
(1.2.1)ggplot2
(3.4.0)ResourceSelection
(0.3-5)kableExtra
(1.3.4)foreach
(1.5.2)tidyverse
(1.3.2)doParallel
(1.0.17)
The time-intensive simulations were performed on 100 cores of the
cluster of the data
laboratory of the
University of Hohenheim. See chapter Simulations
for details. We are
grateful for the provided computing power.
Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
Yeh, I. C., & Lien, C. H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473-2480.