Comments (5)
Hi @dzlab,
This code works fine with unbalanced datasets (the example we use in the book is highly unbalanced!) The split is random, so you should get approximately the same class proportions in the train, eval and test splits. If you need to enforce stratified sampling, I don't believe this is supported in TFX. But you could prepare separate files for each split and pass them into your TFX pipeline as detailed here: https://www.tensorflow.org/tfx/guide/examplegen#custom_inputoutput_split
Hope that's helpful!
from building-machine-learning-pipelines.
Hi @dzlab, you can extend the executor of the ExampleGen component to overwrite the split workflow according to your needs. The benefit is that the component will then be executed via Apache Beam (a big advantage of TFX over for example Kubeflow Pipelines SDK). Apache Beam can process the data itself or outsource it to Spark or Flink. And all steps are getting tracked in the metadata store.
from building-machine-learning-pipelines.
@dzlab You can find an example of how to overwrite the executor in the example: https://github.com/Building-ML-Pipelines/building-machine-learning-pipelines/blob/master/chapters/adv_tfx/Custom_TFX_Components.ipynb (check out the 2nd component)
from building-machine-learning-pipelines.
Yeah stratified sampling is what I was looking for, i think it will be tricky to do it outside TFX. If I have a large dataset, I would need yet another infra (e.g. spark) for just prepare data for TFX. What would you recommend?
from building-machine-learning-pipelines.
Cool that answers my question.
from building-machine-learning-pipelines.
Related Issues (20)
- Makefile `install-deps` fails HOT 1
- How to read JSON files with FileBasedExampleGen HOT 1
- Data Preprocessing - Handling Multilabel Data HOT 1
- Why Infra Validator Component is not even covered? HOT 4
- The way to extend BaseExampleGenExecutor has changed in version 0.23 HOT 1
- GOOGLE_APPLICATION_CREDENTIALS HOT 1
- Unable to install requirements HOT 14
- Can't run Kubeflow sample code HOT 4
- context.run(statistics_gen) interactive pipeline throwing error HOT 1
- Do we need to create schema for train dataset or entire dataset to compare with eval and test dataset? HOT 1
- Modification for `pre-experiment-pipeline/experiment_6Mar.ipynb` HOT 1
- is it possible to get TFserving tutorial for the existing pipeline and for the same data?
- Dockerfile for creating image used to Train HOT 2
- tfx.utils.dsl_utils isn't supported anymore and raises errors. HOT 1
- Interactive Pipeline crashes on different stages HOT 2
- Chapter 2 example error HOT 1
- There issues installing the requirements.txt in ubutu 22.04 HOT 1
- Issue installing requirements HOT 1
- problem while serving with docker container
- Examples pass to evaluator component
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from building-machine-learning-pipelines.