Comments (1)
TL;DR
This papers identifies several key obstacles in machine learning research and gives some inspirations for machine learning research that matters, aiming to address the gap between research and real-world problems.
Key Obstacles
- Machine learning for machine learning sake.
- Hyper-focus on benchmark datasets.
- Reproducibility for evaluation results.
- Meaningful interpretation of evaluation results (e.g., error analysis, why the particular datasets were chosen, etc.)
- De-emphasized the need to learn how to formulate problems and define features, leaving young researchers unprepared to tackle new problems.
- Hyper-focus on abstract metrics
- Abstract metrics like accuracy, RMSE, F-measure ignores problem-specific details.
- Performance obtained by training a model M on dataset X may not reflect M’s performance on other datasets drawn from the same problem.
Necessary Components of Any Research with a Real Impact
- Determine what data should be collected.
- Select or extract relevant features.
- Choose an appropriate learning method.
- Select an meaningful evaluation method.
- Interpret the results.
- Involve domain experts.
- Publicize the results to the relevant scientific community.
- Persuade users to adopt the technique.
Making Machine Learning Matter
- In addition to traditional measures of performance, we can measure dollars saved, lives preserved, time conserved, effort reduced, quality of living increased, and so on. Focusing our metrics on impact will help motivate upstream restructuring of research efforts. They will guide how we select data sets, structure experiments, and define objective functions. At a minimum, publications can report how a given improvement in accuracy translates to impact for the originating problem domain.
- Involve domain experts: They could provide an independent assessment of the performance, utility, and impact of the work in relevant domain.
- Consider potential impact when selecting which research problems to tackle, not merely how interesting or challenging they are from the ML perspective.
Examples of Impact Challenges of Machine Learning that Matters:
- A law passed or legal decision made that relies on the result of an ML analysis.
- $100M saved through improved decision making provided by an ML system.
- A conflict between nations averted through highquality translation provided by an ML system.
- A 50% reduction in cybersecurity break-ins through ML defenses.
- A human life saved through a diagnosis or intervention recommended by an ML system.
- Improvement of 10% in one country’s Human Development Index (HDI) (Anand & Sen, 1994) attributable to an ML system.
Obstacles to ML Impact
- Jargon: Communication problem between peoples in and out of ML domains.
- Risk of deploying ML system to real world applications.
- Complexity: The field has not yet matured to a point where researchers from other areas can simply apply ML to the problem of their choice.
Read More
- Deep Reinforcement Learning That Matters by Peter Henderson et al. AAAI 2018
- Machine Learning Research that Matters for Music Creation: A Case Study by Bob L. Sturm et al. Journal of New Music Research 2018.
from papernotes.
Related Issues (20)
- Neural Architecture Search
- A Recipe for Training Neural Networks HOT 1
- SinGAN: Learning a Generative Model from a Single Natural Image HOT 1
- Few-Shot Unsupervised Image-to-Image Translation HOT 1
- A Style-Based Generator Architecture for Generative Adversarial Networks
- Unsupervised Data Augmentation for Consistency Training HOT 1
- How to Read a Paper HOT 1
- Selfie: Self-supervised Pretraining for Image Embedding HOT 1
- NeurIPS 2019 Notes
- Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates HOT 1
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning HOT 1
- Bayesian Deep Learning
- Knowledge Distillation
- CVPR 2020 Tutorial Talk: Automated Hyperparameter and Architecture Tuning
- Extensive CVPR 2020 Highlighted Tutorials and Papers!
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application
- Why Normalizing Flows Fail to Detect Out-of-Distribution Data
- Knowledge Distillation Meets Self-Supervision & Prime-Aware Adaptive Distillation HOT 4
- Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels HOT 3
- Hyperspherical Prototype Networks HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from papernotes.