gracikk-ds / ml-ops

The course includes 6 modules that cover the skills of a machine learning engineer.

ML Engineering course by EPAM

The course includes 6 modules that cover the skills of a machine learning engineer.

Skills of a machine learning engineer:

1. The ML Engineer should be proficient in all aspects of model architecture, data pipeline interaction, and metrics interpretation.
2. A Machine Learning Engineer designs, builds, and productionizes ML models to solve business challenges.
3. The ML Engineer needs familiarity with foundational concepts of application development, infrastructure management, data engineering, and data governance.
4. Through an understanding of training, retraining, deploying, scheduling, monitoring, and improving models, the ML Engineer designs and creates scalable solutions for optimal performance.

Course modules:

---

Part 1 - Data Governance:

• Make an initial setup using DVC tool and add a dataset ✅
• Define a DVC pipeline that will:
  • preprocess data ✅
  • train a model ✅
  • evaluate the model ✅
  • generate a feature importance plot with a model agnostic method ✅
• The pipeline should be reproducible using dvc repro ✅
• Run experiments and save metrics using dvc metrics ✅

---

Part 2 - CI/CD and testing:

• Create unit tests for python code from Part 1 ✅
• Create a github action which at least performs:
  • code quality check ✅
  • auto-formatting with black ✅
  • linting with pylint - fail if less than a threshold example ✅
  • run unit tests ✅

---

Part 3 - Experiment tracking:

• Conduct several experiments:
  • use different features ✅
  • hyperparameter search ✅
  • different models ✅
• Results of each experiment should be tracked in MLFlow ✅
• Best model artifacts should be logged as well ✅

---

Part 4 - Model deployment in online mode (RestAPI):

• Prepare model for deployment ✅
• Use FastAPI for the deployment ✅
• Wrap everything into Docker ✅
• Also prepare the service for cluster deployment ❌

---

Part 5 - Python packaging:

• Prepare python code for packaging ✅
• Create and run tests ✅
• Publish your package at pypi ✅

---

Part 6 - ETL pipeline link :

• Create an ETL pipeline as an Airflow dag ✅
• Besides Pandas use Pyspark for data manipulation ✅

---

**Part 7 - Batch mode model deployment: **

• Create a pipeline for model training link:
  • use the output from Part 6 as an input ✅
  • define a logic for retraining ✅
  • define a logic for model versioning ✅
• Create a pipeline for model serving link:
  • use an output from Part 6 as an input ✅
  • use a model from the model training pipeline as an input ✅

---

conda activate your_env
poetry config virtualenvs.path "path/to/your/conda/envs"
poetry config virtualenvs.create false
poetry install