coefficientsystems / modern-data-cookiecutter

Cookiecutter template for Coefficient projects.

Modern Data Cookiecutter

Modern Cookiecutter template for Python-based data science/engineering/analytics projects. Created by Coefficient.

Inspired by:

• cookiecutter-data-science
• cookiecutter-hypermodern-python
• govcookiecutter

Usage

# Install cookiecutter (if you haven't already done so)
pipx install cookiecutter

# Install the latest release directly from this repo
cookiecutter https://github.com/CoefficientSystems/modern-data-cookiecutter

# Follow instructions in docs/getting_started.md or docs/quickstart.md

Features

📦 Managing Python & Package Dependencies

• Work across multiple Python versions using pyenv
• Virtualenv management using virtualenv and pyenv-virtualenv
• Packaging and dependency management with Poetry
• Automated dependency upgrades with poetry-plugin-up and Dependabot

👷 Code Quality, Testing & CI

• Linting pre-configured using pre-commit, Flake8 and pylint
• Code auto-formatting with Black, autoflake, add-trailing-comma
• Import sorting with isort
• Automated Python syntax upgrades with pyupgrade
• Testing with pytest
• Continuous integration with GitHub Actions
• Twelve Factor app principles

🤖 Data Science

• Jupyter Lab/Notebook configured with jupyter-black autoformatter
• Directory structure designed specifically for data science & analytics projects (adapted from cookiecutter-data-science)

🔒 Security Features

• Security audit with pip-audit and detect-secrets
• Keep secrets & config out of code by loading as environment variables using direnv

📚 Documentation, IDE Settings & Release Notes

• Get started easily using Getting Started or the Quickstart with further documentation on how to use the packages in this cookiecutter available in docs/
• VS Code Settings configures features like "Format on save", isort configuration, bracket pair colorization, Pylance and mypy alongside recommended extensions
• Automated release notes using towncrier

Contributing

Please see CONTRIBUTING.md for guidance on how to request features, report issues, setup a development environment, test this cookiecutter and contribute to development.

modern-data-cookiecutter's Issues

Release 0.2

• #19
• #25
• #26
• #27
• #51
• #37
• #56
• #108
• #87
• #55
• Improve README for new users (#96)
• Add towncrier entries
• Release to main
• Tag release

Deprecated Linters VS Code Modal.

https://github.com/microsoft/vscode-python/wiki/Migration-to-Python-Tools-Extensions

Add linters, pyproject.toml and .vscode/settings.json

Update linters to use Ruff for the performance speedup where appropriate.

flake8-broken-line

https://github.com/wemake-services/flake8-broken-line/ + any other extensions created by https://github.com/wemake-services?

Fix Poetry Plugins Deprecation Warning.

Warning: poetry-plugin-export will not be installed by default in a future version of Poetry.
In order to avoid a breaking change and make your automation forward-compatible, please install poetry-plugin-export explicitly. See https://python-poetry.org/docs/plugins/#using-plugins for details on how to install a plugin.
To disable this warning run 'poetry config warnings.export false'.

install poetry-up plugin and poetry-plugin-export via pipx inject?

Replace flake8 with ruff

Relevant links:

• https://github.com/charliermarsh/ruff
• wemake-services/flake8-eradicate#269 (see last comments)
• https://www.reddit.com/r/Python/comments/11syxd0/the_ruff_python_linter_is_insanely_good/
• https://community.plone.org/t/drop-black-isort-flake8-and-use-ruff/16413
• https://news.ycombinator.com/item?id=34062974
• https://www.reddit.com/r/Python/comments/x30lcs/github_charliermarshruff_an_extremely_fast_python/
• https://www.reddit.com/r/Python/comments/10lsovt/ruff_a_new_fast_and_correct_python_checkerlinter/
• https://www.reddit.com/r/Python/comments/yzhykq/python_linter_comparison_2022_pylint_vs_pyflakes/

Lots of other good linters etc are referenced in these threads. There may also be other Rust-based CLI tools that are worth investigating as part of this repo.

Re-add Python versions back into precommit

These have been lost, e.g. hard-coded to 3.9 here: https://github.com/CoefficientSystems/coefficient-cookiecutter/blob/develop/%7B%7Bcookiecutter.repo_name%7D%7D/.pre-commit-config.template.yaml#L5

Setup Poetry project

Release 0.1

Issues

• #1
• #5
• tutorial1 ("Getting to know")
• tutorial2 ("Create from scratch")
• "Project templates made easy"
• #6
• #8
• #10
• #12
• Add ccds inspired data directory layout
• #14

Better gitignores

# Data
data/**/*.*
!**/data/**/README.md
!**/data/**/*.gitkeep
!**/data/**/file.csv

Add jupyter-black

https://github.com/n8henrie/jupyter-black

Release 0.3

• #172
• #72
• pytest-coverage
• Github CI
• Gitlab CI
• Towncrier updates
• Contributers guide

---

Maybe?

• Inspiration from https://github.com/pydanny/cookiecutter-django/ / https://cookiecutter-django.readthedocs.io/en/latest/project-generation-options.html
• https://pyup.io/
• https://github.com/wemake-services/wemake-django-template
• https://github.com/globality-corp/flake8-logging-format
• https://github.com/hynek/structlog
• https://github.com/willmcgugan/rich

pyproject.toml updates

python = "~3.9.7"
# Everything below here is alphabetically sorted

Add mypy

tutorial0 ("Lean the basics")

https://cookiecutter.readthedocs.io/en/1.7.2/first_steps.html

Make a shortened version of Getting Started

eg. a cheatsheet of commands right at the top

Consider replacing or removing flake8-eradicate

This clashes with flake8 >= 6 with no repo updates since Dec 2022. Open issue is here:
wemake-services/flake8-eradicate#269

Options:

1. Wait until this issue is resolved and flake8-eradicate supports flake8 6+
2. Switch to replacements e.g. ruff as suggested in above thread

Run poetryup to upgrade packages

Add towncrier

Automate manual test of generated project

Do this using pytest in the outer project

Investigate differences between subprocess.popen and subprocess.call

Tips:

• https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html
• https://docs.pytest.org/en/latest/how-to/fixtures.html#teardown-cleanup-aka-fixture-finalization

pre-commit autoupdate

Run pre-commit autoupdate in outer repo, manually copy the version changes into the inner precommit

Add option for a good jupyter-friendly setup

Adding in the env's kernel should be automated in project setup

Add pre-commit

Remove deprecated python.sortImports.args from settings.json

This can be removed from settings.json (double check using latest isort):

"python.sortImports.args": ["--recursive"],

because

[ERROR 2022-5-29 22:19:11.680]: /Users/john/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/isort/main.py:1261: UserWarning: W0501: The following deprecated CLI flags were used and ignored: --recursive!
  warn(
/Users/john/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/isort/main.py:1265: UserWarning: W0500: Please see the 5.0.0 Upgrade guide: https://pycqa.github.io/isort/docs/upgrade_guides/5.0.0.html

Add instructions for installing on Linux

For Vertex Workbench (Debian):

# Install deps
# from https://bgasparotto.com/install-pyenv-ubuntu-debian
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev

# Auto-installer
curl https://pyenv.run | bash

.bashrc not working on workbench? Added to .bash_profile:

conda deactivate

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

# Load pyenv-virtualenv automatically by adding
# the following to ~/.bashrc:
eval "$(pyenv virtualenv-init -)"

# Add this for Poetry
export PATH=$HOME/.local/bin:$PATH

Installed pyenv-virtualenvwrapper plugin using git clone: https://github.com/pyenv/pyenv-virtualenvwrapper

pre-commit updates

# Update all versions in this file by running:
#   $ pre-commit autoupdate

Update all packages

Remove "coefficient" from repo name etc in setup

Fix poetryup

poetryup is failing on install with the following message:

CalledProcessError

  Command '['git', '--git-dir', '/var/folders/98/2gsv1cnj48n3k_s4zfh6mxqc0000gn/T/pypoetry-git-poetryup37axluxm/.git', '--work-tree', '/var/folders/98/2gsv1cnj48n3k_s4zfh6mxqc0000gn/T/pypoetry-git-poetryup37axluxm', 'checkout', 'fdfaf4d446249f8f69db613bac285e818a7df92f']' returned non-zero exit status 128.

  at ~/.poetry/lib/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

It looks like this is coming from this part of the outer pyproject.toml

poetryup = { git = "https://github.com/john-sandall/poetryup.git", rev = "fdfaf4d446249f8f69db613bac285e818a7df92f" } # pragma: allowlist secret

Poetry TOML Confiiguration Files Location Deprecation

Change the getting_started and quickstart docs to fix these issues in pre-commit.

poetry-export...................................................................Failed
- hook id: poetry-export
- files were modified by this hook

Configuration file exists at /Users/jamie/Library/Preferences/pypoetry, reusing this directory.

Consider moving TOML configuration files to /Users/jamie/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.

What breaks if I just move these files? Are there environment variables or other installs that break somewhere if I do this?

Replicate dependabot changes in inner repo

List of dependabot PRs which made changes to outer repo but not inner repo:

• #29
• #32
• #41
• #44
• #46
• #49
• #50

Base package doesn't pass pre-commit

pylint..........................................................................Failed
- hook id: pylint
- exit code: 16

************* Module tests.test_gershwin
tests/test_gershwin.py:1:0: C0114: Missing module docstring (missing-module-docstring)
tests/test_gershwin.py:6:0: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 6.67/10 (previous run: 6.67/10, +0.00)

Todo

• Add module docstring to tests

Add bracket pair colourization

Uninstall Bracket Pair Colorizer (1 & 2), VS Code has this functionality built-in, enable via settings.json

{
    "editor.bracketPairColorization.enabled": true,
    "editor.guides.bracketPairs":"active"
}

https://github.com/CoenraadS/Bracket-Pair-Colorizer-2#readme

isort in Jupyter

It would be great to have isort work in Jupyter Notebooks

Add .gitignore & .envs

Create useful coefficient-cookiecutter with Poetry & tests

Tutorial

Please can you create a beginner friendly tutorial for this? I know there's docs/quickstart.md but this feels written for people who know what they're doing?