The course includes 6 modules that cover the skills of a machine learning engineer.
Skills of a machine learning engineer:
- The ML Engineer should be proficient in all aspects of model architecture, data pipeline interaction, and metrics interpretation.
- A Machine Learning Engineer designs, builds, and productionizes ML models to solve business challenges.
- The ML Engineer needs familiarity with foundational concepts of application development, infrastructure management, data engineering, and data governance.
- Through an understanding of training, retraining, deploying, scheduling, monitoring, and improving models, the ML Engineer designs and creates scalable solutions for optimal performance.
Part 1 - Data Governance:
- Make an initial setup using
DVC
tool and add a dataset ✅ - Define a DVC pipeline that will:
- preprocess data ✅
- train a model ✅
- evaluate the model ✅
- generate a feature importance plot with a model agnostic method ✅
- The pipeline should be reproducible using
dvc repro
✅ - Run experiments and save metrics using
dvc metrics
✅
Part 2 - CI/CD and testing:
- Create unit tests for python code from Part 1 ✅
- Create a
github action
which at least performs:- code quality check ✅
- auto-formatting with black ✅
- linting with pylint - fail if less than a threshold example ✅
- run unit tests ✅
Part 3 - Experiment tracking:
- Conduct several experiments:
- use different features ✅
- hyperparameter search ✅
- different models ✅
- Results of each experiment should be tracked in MLFlow ✅
- Best model artifacts should be logged as well ✅
Part 4 - Model deployment in online mode (RestAPI):
- Prepare model for deployment ✅
- Use FastAPI for the deployment ✅
- Wrap everything into Docker ✅
- Also prepare the service for cluster deployment ❌
Part 5 - Python packaging:
- Prepare python code for packaging ✅
- Create and run tests ✅
- Publish your package at pypi ✅
Part 6 - ETL pipeline link :
- Create an ETL pipeline as an Airflow dag ✅
- Besides Pandas use Pyspark for data manipulation ✅
**Part 7 - Batch mode model deployment: **
- Create a pipeline for model training link:
- use the output from Part 6 as an input ✅
- define a logic for retraining ✅
- define a logic for model versioning ✅
- Create a pipeline for model serving link:
- use an output from Part 6 as an input ✅
- use a model from the model training pipeline as an input ✅
conda activate your_env
poetry config virtualenvs.path "path/to/your/conda/envs"
poetry config virtualenvs.create false
poetry install