stephenbates19 / nestedcv

Nested cross-validation for accurate confidence intervals for prediction error.

nestedcv

Nested cross-validation for accurate confidence intervals for prediction error.

This method was introduced in the paper Cross-validation: what does it estimate and how well does it do it? by Stephen Bates, Trevor Hastie, and Robert Tibshirani, available at https://arxiv.org/abs/2104.00673.

To install, run

	devtools::install_github("stephenbates19/nestedcv")

Two vignettes showing basic usage are included.

The numerical experiments from the paper can be found at https://github.com/stephenbates19/nestedcv_experiments.

nestedcv's Issues

Error running nestcv.train on regression

Hi,

Thank you for your lovely package. I ran into an error when trying to run a regression on the iris data set using the Earth algorithm, which I aim to use for my data.

I hope you can help me.

`
library (nestedcv)

x <- iris[, 1:3]
colnames(x) <- c("marker1", "marker2", "marker3")
x <- as.data.frame(apply(x, 2, scale))
y = iris[, 4]

hyper_grid <- expand.grid(
degree = 1:2,
nprune = seq(2, 20, length.out = 10)
)

cvrf <- nestcv.train(x = x,
y = y,
method = "earth",
n_outer_folds = 5,
linpreds=FALSE,
metric = "RMSE",
trControl = trainControl(method = "cv", number = 3),
tuneGrid =hyper_grid,
cv.cores = 1)
summary(cvrf)

`

Error in data.frame(obs = output$testy, pred = output$predy) :
arguments imply differing number of rows: 150, 0

Regards,
Bernard

I can not find git hub repo specifically addressing problem of prediction for unseen data

GREAT WORK THANKS

Only may you share some links to code to predict data sets with unseen data
of course all validation data has unseen data
but I can not find git hub repo specifically addressing problem of prediction for unseen data