A modern approach to data science and machine learning using Python & Docker.
- Use a modern Python development stack geared towards automation and best practices.
- Harness Docker for a reproducible, portable development environment and ease transition to production.
- Docker
- Bonus: GNU make to make full use of the
Makefile
Note: This has only been tested on macOS. Linux support is assumed. Windows support is untested.
make docker-run
Automatically pulls the latest image from Docker Hub the first time it is run. Subsequent runs will use local copy and will be faster. Copy the link to the Jupyter Lab server and paste it into a browser of your choice to access the Jupyter Lab.
By default, the current working directory $PWD
will be used as the local directory that will be mapped to /root/work
directory on the Docker container.
make docker-run host_volume=/full/path/to/local/folder
Use the host_volume
option to specify the local folder to be used by the Docker container. The specified folder will be available under /root/work
in the Docker container.
make docker-build
This step requires creating an account and a repository on Docker Hub (free for public images). Update the docker_hub_repo` variable in Makefile
to point to the correct repo on Docker Hub.
make docker-push
- Uses
pyenv
for managing Python version - Uses Python Development Master (
pdm
) for managing dependencies and packaging - Uses Cookiecutter for project scaffolding
- Keeps the common packages and libraries related to Python development and DS/ML projects in a global space to avoid reinstalling for every project
- Keeps a local copy of the cookiecutter project template in the final image
- Aims for a small final image (work in progress).
Python Development
- cookiecutter
- nox
- pre-commit
- flake8
- sphinx
- sphinx-click
- furo
- black
- pytest
- coverage
- typer
- mypy
Basic Python data science packages
- ipython
- jupyterlab
- numpy
- scipy
- matplotlib
- pandas
- seaborn
- statsmodels