priceloop / conventions

∞ Priceloop Engineering Conventions for Scala, Python, Git Workflow etc

👩‍💻 Conventions

Our Priceloop Engineering conventions for Scala, Python, Git Workflow etc.

📖 Contents

• Scala Styling Guide
• Python Styling Guide
• Naming Conventions
• Git Commit Structure
• Git Branch Flow
• Secrets
• Language
• License

Scala Styling Guide

We use Scalafmt to format our code. Use this config in your projects.

Python Styling Guide

Use the following pre-commit hooks:

• isort
• pylint following Google styling guide
• black formater with the following flags:
  • -l 110 -S -t
  • VSCode settings.json

"python.formatting.blackArgs": ["-l", "110", "-S", "-t"]

• flake8

• docstrings: Google conventions

• Quotes: double

• mypy

• Use pre-commit with the following hooks:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.2.3
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: debug-statements
      - id: requirements-txt-fixer
      - id: flake8

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.720
    hooks:
      - id: mypy
        args: [--allow-redefinition, --ignore-missing-imports]

  - repo: local
    hooks:
      - id: isort
        name: "Sort imports"
        language: system
        types: [file, python]
        entry: isort
      - id: pylint
        name: "PyLint"
        language: system
        types: [file, python]
        files: .
        exclude: test_*.py
        entry: python3 -m pylint
      - id: black
        name: "Black"
        language: system
        pass_filenames: false
        entry: black .
        args: [--safe, --quiet, "--line-length=110", "--skip-string-normalization"]

Naming Conventions

Git Branch Naming

Your branch name should follow the format type-scope.

Types:

• feat
• fix
• chore

Storage buckets, i.e. s3, gcs

ai.priceloop.{{bucket_purpose}}.{{environment}}

e.g. ai.priceloop.data.prod

{{environment}} = {prod, stage}

{{bucket_purpose}} can potentially be broken down to

{{layer}}.{{bucket_purpose}}, e.g.

ai.priceloop.ml.metadata
ai.priceloop.frontend.assets
ai.priceloop.dataplatform.metadata

Namespace

• Long: priceloop-production / priceloop-staging

• Short: priceloop-prod / priceloop-stage

Git Commit Structure

• In general, follow the Conventional Commit guidelines
• Be descriptive
• Use all lower-case
• Limit punctuations (no dots, no commas)
• Short (under 70 characters is best)

👉 Follow the git commit structure for commits that are getting merged. In branches, when you do something temporary (e.g. testing ideas etc), you don't need to follow it

Git Branch Flow

We follow the following PRs flow with user facing feature(s):

develop → { git checkout -b } → feature branch → {PR} develop → {PR} master

CI/CD

We use Github Actions as the CI solution. Every repository with the codebase for user facing features must contain the following CI configuration/workflow definition files:

.github
└── workflows
    ├── develop.yaml
    ├── master.yaml
    └── pr_branches_check.yaml

Where

• develop.yaml - defines the CI pipeline to deploy to the stage env
• master.yaml - defines the CI pipeline to deploy to the prod env
• pr_branches_check.yaml - defines the check to validate if PR is compliant with the branch flow

pr_branches_check.yaml

name: PR Branches Check

on:
  pull_request:
    types:
      - opened
      - reopened

jobs:
  develop_to_master:
    if: github.base_ref == 'master'
    runs-on: ubuntu-latest
    steps:
      - name: Check if PR to "master" is from "develop"
        if: ${{ github.head_ref != 'develop' }}
        run: |
          echo "Wrong source branch, only develop->master PR's allowed"
          exit 1
  any_to_develop:
    if: github.base_ref == 'develop'
    runs-on: ubuntu-latest
    steps:
      - name: Check if PR is to "develop"
        if: ${{ github.head_ref == 'refs/heads/master' }}
        run: |
          echo "Wrong source branch, only "!master"->develop PR's allowed"
          exit 1

GitHub Branches Protection Rules

Every Github repository containing user facing features must have the protection rules for the master and develop branches.

Secrets

⚠️ Never put any username, password on Git, always use local environments or secrets manager to access services

• Metadata considered as secrets:
  • AWS keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  • Access tokens (any oauth2 tokens which have long life time)
  • Access credentials, i.e. combination of password and user name
  • Webhook URLs
• Metadata which can be stored to git:
  • Bucket name, path to object
  • Database name
  • Resource ID, URI ← it may be a subject of change/dispute

Note: in case of doubt whether the info is a secret/internal or can be public, treat it at a higher security level, i.e. as a secret. It costs nothing to downgrade security level, but it's quite expensive to upgrade it 😃

How to share secrets

To share between peers:

• Please use gpg to encrypt opaque text secret.
• Slack DM is acceptable, but remember to delete the secret message after it was received!

We use AWS Secret Manager and github secrets for secrets to be used programmatically.

Language

• Stick to American English e.g. optimize instead of optimise or behavior instead of behaviour. In doubt check Google 😃

License

• For private repos, no License necessary
• For public repos, we should use MIT License