Comments (3)
Outside of resampling, all preprocessing quantities are computed using the training set. So you could get your mode or means on the training set and impute all samples using these. You test set or unknown samples probably have a different distribution (based on what was sampled), but the model is expecting something consistent with the training set.
As a pathological counter-example, suppose you are predicting one new sample. You couldn't impute a missing predictor value if you used the current sample set.
This specific issue isn't the point though; it's one of consistency. Non-training set samples should be preprocessed in a manner that is consistent with what was used to build the model.
For resampling, using the terminology here, the preprocessing is recomputed for each analysis set and these same values (e.g. the mean or mode in your imputation example) are used on the corresponding assessment set.
For 10-fold CV, this results in 10 different imputation values but this is the point; we want to measure the variation in the preprocessing steps and let that variation show its impact on the the assessment set performance measures.
apply the same values from the training set to the test set, which may lead to over-fitting.
It wouldn't lead to overfitting since the test set is being imputed with values contained in other samples. Quite the opposite; we measure overfitting using this strategy.
from fes.
Great answer. Thank you so, so much. I am looking forward to reading the rest of your book, and your other content too. Maybe you can add this content into the book, as it's a great explanation.
I was researching this question, and I ran across this resource (the first answer in the link below.) It was discussing how when you split up your training and test sets, it's possible that one runs into issues where there is a mismatch between all the possible categorical values in the training set and the test set, due to the random split. Therefore, this needs to be avoided when encoding. If this isn't in your book, you may want to mention this.
https://www.kaggle.com/c/avito-demand-prediction/discussion/56550
Good luck!
from fes.
Thanks.
We address that in chapter 5.
from fes.
Related Issues (20)
- Typo in 1.4 Feature Selection
- Figure 6.5: knots are not plotted HOT 1
- Typo in Intro to Chapter 10
- different output for chapter 11.2 initial ROC analysis HOT 1
- Figure 2.2b plot output in graphic does not match code
- Section 3.4.1 variance reduction not Sqrt(R) HOT 1
- Averaging fold criteria not usually advisable HOT 4
- Leave-one-out cross validation is very valuable HOT 2
- Average {RMSE vs. MSE} over V-folds
- Error in ames_pairwise.R
- Typo in 2.2
- Code missing for results mentioned in Chapter 11.2.1?
- Feature Engineering and Selection: A Practical Approach for Predictive Models (Chapman & Hall/CRC Data Science Series) (English Edition) 1ª Edição, eBook Kindle HOT 1
- Trouble with plotting roc curve(yardstick) in some models HOT 1
- Chapter 6, 9, 10, 12 typos. Version 2019-06-21 HOT 1
- Chapter two - First reference and Table 2.1
- 02_Predicting_Risk_of_Ischemic_Stroke- Coefficients for RFE model HOT 1
- Codes for Section 6.2 are missing HOT 1
- Typo in 2.3
- Typo in preface HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fes.