ericyaang / dst

yet another custom data science template via cookiecutter

data science template

in this repo u can look at default template for ds/ml/dl/.. projects or similar

how to use

• before creating a new project from this template, u need to install the next dependencies

  • cookiecutter

    • linux

      pip install cookiecutter

    • macos

      brew install cookiecutter

  • github cli

    • linux

      look at the linux installation instructions

    • macos

      • install

        brew install gh

      • upgrade

        brew upgrade gh

• after go to the directory where u want to create your project and run

  cookiecutter gh:vtrokhymenko/dst

• follow the instruction

using the next project structure

├── .github/                           <- some actions
│   ├── workflows/  
│   │   ├── ci.yml  
│   │   └── dependency-review.yml  
│   └── dependabot.yml  
│  
├── config/                            <- often it's yaml-files with some parameters
│  
├── data/  
│   ├── external/                      <- data from third party sources
│   ├── interim/                       <- intermediate data that has been transformed
│   ├── processed/                     <- the final, canonical data sets for modeling
│   ├── raw/                           <- the original, immutable data dump
│   ├── features/                      <- another
│   └── README.md  
│  
├── docs/                              <- a default sphinx project (see sphinx-doc.org for details)
│  
├── experiments/                       <- for any experiments
│   └── README.md  
│  
├── models/                            <- trained & serialized models, model predictions, or model summaries
│   └── README.md  
│  
├── notebooks/                         <- notebooks for research naming convention is a number (for ordering), the creator's initials,
│                                         and a short `-` delimited description, eg `1.0-jqp-initial-data-exploration`
│  
├── references/                        <- data dictionaries, manuals, and all other explanatory materials
│   └── README.md  
│  
├── tests/                             <- test for project
│   └── __init__.py
│  
├── {{ cookiecutter.project_name }}/   <- source code
│   ├── __init__.py                    <- propose generate with `mkinit`
│   ├── data/                          <- scripts to download or generate data
│   ├── models/                        <- scripts to train models and then use trained models to make predictions
│   └── visualization/                 <- scripts to create exploratory and results oriented visualizations
│  
├── .gitignore                         <- default for python
├── .pre-commit-config.yaml            <- custom pcc with `reorder_python_imports`, `black`, `flake8`, `pyright`, `mypy`, `pre-commit-hooks`..  
├── LICENSE                            <- will be created if u choose
├── README.md
└── requirements.txt                   <- propose generate with `pipreqs`

other similar templates

• cookiecutter-data-science
  • cdst
  • python-package-template
• ocean
• kedro

propose to use next tools

• gh – github on the terminal
• dvc – open-source version control system for ds projects
• cml – continuous machine learning | ci/cd for ml/dl
• renovate - yet another dependency management
• hydra – to configuring complex applications
• pipreqs – autogenerate pip requirements
• pre-commit – framework for managing & maintaining multi-language pre-commit hooks
• code style/review/formatter/typer
  • awesome-python-typing
  • codefactor
  • snyk
  • deepsource
  • prettier
  • pycodestyle
  • pyre-check
  • pyright
  • restyled (autopep8, black, isort, prettier-markdown, reorder-python-imports, yapf)
  • super-linter (pylint, flake8, awesome-flake8-extensions, black)
  • yapf
  • pyupgrade
  • vulture
• tests
  • codecov
  • coveragepy
  • pytest (guide)
  • pytest-cov
  • mutmut
• profiler/debugger
  • birdseye
  • heartrate
  • palanteer
  • py-spy
  • pyheat
  • snoop
  • viztracer
  • memray
• spellcheckers
  • yaspeller
  • pyspelling
  • vale