Comments (5)
Hi!
The output
accuracy: 0.951048951048951
log loss: 0.1111237013184977
from this example is given by the last two lines of the corresponding script:
print("accuracy:", accuracy_score(y_test, label_predictions))
print("log loss:", log_loss(y_test, probability_predictions))
In the example, the reported performance is a test performance, not a training performance, as the test set was held out during training.
Does that answer the question?
from gama.
Hi!
The output
accuracy: 0.951048951048951 log loss: 0.1111237013184977
from this example is given by the last two lines of the corresponding script:
print("accuracy:", accuracy_score(y_test, label_predictions)) print("log loss:", log_loss(y_test, probability_predictions))In the example, the reported performance is a test performance, not a training performance, as the test set was held out during training.
Does that answer the question?
ah, yes, I am sorry for that, I got confused. I meant the training performance, not the test performance.
Thank you.
from gama.
Sorry for the slow replies. To get the training score, just all use score
with the training data: automl.score(X_train, y_train)
. If you want to access the 5-fold cross validation score that was found during the search process, it's a little bit more tricky (making it more easily accessible is in the works), assuming you have the (default) BestFit postprocessing:
max(map(lambda ind: ind.fitness.values, automl._final_pop))
should output a (score, length)
tuple, e.g. (-0.0908208740423977, -2)
, where the first value is the score for the given metric. The score is negated if the metric was a loss, because internally the scorers follow scikit-learns conversion to always ensure "bigger is better".
I hope this answers the question!
from gama.
Ah, I see, great answer. Yes, the cv score is what I was after, as this is the training performance estimation. What is the length
in the outputted tuple and how is it negative? Also, I came across an error when using automl.score()
, should I post this here or start a new issue?
Thank you for your help
from gama.
Please open a new issue with a minimal working example included 👍
Length represents the length of the pipeline in number of steps (not including imputation and categorical encoding). Internally all scores are represented s.t. they can be maximized, which is why metrics that should be minimized (loss, length) are negated.
from gama.
Related Issues (20)
- Update developer docs
- Bug with LinearSVC training HOT 2
- warm starting error because of empty array in the search space HOT 4
- Better warm starting with automatically converting pipeline or pipeline string to gama individual string HOT 6
- package is incompatible. HOT 2
- Make it easier to access model evaluation results
- KeyError: 'ignore' when running example HOT 1
- Add link to license in readme HOT 1
- Hyperparameter check HOT 2
- Gama re-evaluates already evaluated pipelines HOT 4
- Evaluating individuals after deadline HOT 2
- [Researcher] Developing a new AutoML system around GAMA as outer wrapper? HOT 6
- Using `numpy` arrays as data source may lead to errors if inferred encoding is used
- `Changelog` always post message, even if it already sent it before
- Unable to warm-start AsyncEA HOT 7
- [Implementation] Developing a Custom Auto-ML System Variant with GAMA: Questions & Concerns HOT 3
- Issue with Pickling Local Object in Custom Gama Classifier Implementation HOT 12
- AttributeError in Ensemble Class when Accessing 'validation_score'
- Addition of a new search algorithm for GAMA HOT 2
- [Feature Proposal] Bayesian Optimisation Implementation HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gama.