caravancoop / configstore Goto Github PK

Add backend that uses https://github.com/jpadilla/django-dotenv (MIT) to parse a file

Packaging note: define an extra to make the dependency optional

A project that uses docker-compose installed with pip will have this dependency installed: https://pypi.org/project/python-dotenv/
It installs a dotenv package that conflicts with the dotenv module installed by django-dotenv
Handle this with another dependency extra and some compat code in the dotenv backend

Amazon has a nice settings & secrets storage, but it is not integrated automatically into ECS. Docker secrets for example are tied to the swarm/service concepts, which don’t exist in the ECS environment.

We need a new backend built on boto3 that fetches parameters. Credentials are not needed for the class, since boto will be able to auto-configure itself from the Instance or Task IAM role. We will need a prefix however, as parameters don’t have namespacing; this would also serve as a switch to enable the backend (similar to how DotEnvBackend needs a path in its constructor):

if 'MYAPP_AWS_SSM_PREFIX' in os.environ:
    store.add_backend(configstore.AwsSsmBackend(os.environ['MYAPP_AWS_SSM_PREFIX']))

Not in scope:

• types (list, int, etc; this could be added later for all backends)
• get many / pre-fetch all parameters for prefix

Legitimate exceptions could be hidden by the try/except in the main function. I propose that the backends should return a special value to indicate key not found; I think None would be fine.

Dependabot was set up to create pull requests against the branch master, but couldn't find it.

If the branch has been permanently deleted you can update Dependabot's target branch from your dashboard.

View the update logs.

• switch to pytest
• add tox config with all python versions and pyflakes
• add circleci config

Would it be useful to add syntax shortcuts to make client code shorter?

store = configstore.Store()
store += configstore.EnvVarBackend()

if something:
    store += configstore.DotEnvBackend()

DEBUG = store['DEBUG']
DATABASES = {'default': dj_database_url.parse(store['DATABASE_URL'])}

We have unit tests for each class but not for the whole system.

envvar: run in sub-processes?

dotenv: test missing file, unreadable file, empty file, invalid file

docker secret: run with fake filesystem? run in container?

aws ssm: use moto to connect to a fake SSM API

Proposal:

• Normal PRs go into a develop branch (git-flow style)
• To make a release, we use bumpversion to update files and make a PR to master
• The CI system already runs setup.py check, safety check, check-manifest for all branches
• For release branches, we could push sdist and wheel to TestPyPI (or a Caravan devpi server)
• When the PR is merged, CircleCI makes a tag (unsigned?), pushes it to the repo and uses twine to push wheel and sdist to PyPI

The pattern we want to encourage is to call get_settings when an app is initialized and store the results in the appropriate structure (django settings module, Pyramid app config, etc.). It would be nice to have a method to clean up the internal data of the backends, and if necessary release internal resources.

Proposed API: add Store.close that calls backend.close

Bonus: add Store.__enter__ and Store.__exit__ to call close automatically

The DictBackend class used in test_store should be promoted to an official backend.

Use cases include:

• Define all default values in one spot instead of using get_setting(x, default)
• For app tests, cleaner to define test settings in a dict instead of e.g. os.environ
• Load settings from another data source without having to write a backend class

Things that get reimplemented in projects:

• convert string value to boolean (on, yes, 1, true / off, no, 0, false; case-insensitive)
• lists (separators? space / newline / comma)

Not sure about more; for integers for example we could catch and ignore ValueError but whould it be a good thing? Review projects to see need.

• Sphinx
• How to use
  • django
  • pyramid
• Backend reference
• How to write a backend
• RTD setup

Switch to flit + pyproject.toml

Should give a smaller footprint to release and still be pip-installable from github.

Add type declarations (I think Python 3 native syntax, and drop 2.7 support)

Run mypy from tox

Run mypy in CI