datasciencedojo / intrototextanalyticswithr Goto Github PK
View Code? Open in Web Editor NEWPublic repo for the Data Science Dojo YouTube tutorial series "Introduction to Text Analytics with R".
Public repo for the Data Science Dojo YouTube tutorial series "Introduction to Text Analytics with R".
In IntroToTextAnalytics_Part4
train.tokens.df <- cbind(Label = train$Label, data.frame(train.tokens.dfm))
R says:
Warning message:
'as.data.frame.dfm' is deprecated.
Use 'convert(x, to = "data.frame")' instead.
See help("Deprecated")
What would be the fix for this?
I actually found it:
train.tokens.df <- cbind(Label = train$Label, as.data.frame(train.tokens.dfm))
Thanks
Hi. Great tutorial. Just a quick note on Session 11: when creating cosine similarities with spam message feature on training data you should exclude the observation itself from the spam messages list:
# cosine similarities with spam messages and vice versa!
spam.indexes <- which(train$Label == "spam")
train.svd$SpamSimilarity <- rep(0.0, nrow(train.svd))
for(i in 1:nrow(train.svd)) {
spam.indexesCV <- setdiff(spam.indexes,i)
train.svd$SpamSimilarity[i] <- mean(train.similarities[i, spam.indexesCV])
}
This solves the data leakage problem leading to over-fitting. The RF results on test data with updated feature are much better:
# Drill-in on results
confusionMatrix(preds, test.svd$Label)
Confusion Matrix and Statistics
Reference
Prediction ham spam
ham 1445 32
spam 2 192
Accuracy : 0.98
95% CI : (0.972, 0.986)
No Information Rate : 0.866
P-Value [Acc > NIR] : < 2e-16
Kappa : 0.907
Mcnemar's Test P-Value : 0.000000658
Sensitivity : 0.999
Specificity : 0.857
Pos Pred Value : 0.978
Neg Pred Value : 0.990
Prevalence : 0.866
Detection Rate : 0.865
Detection Prevalence : 0.884
Balanced Accuracy : 0.928
'Positive' Class : ham
Hi,
Thank you for the awesome tutorial. I am trying to use similar process to my project. Kindly can you suggest which form is good to store the train.tokens.idf data. As after I deploy this, the training idf data will be used while testing. I tried storing it in a text file but I am not sure weather the data is stored in the same format.
Please help
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.