kiwicom / konfetti

Python configuration management system with an intuitive API

🎊 konfetti 🎊

Description

konfetti is a Python configuration management library that simplifies the process of setting up your application to run on your company’s infrastructure.

This library will help you to retrieve secrets from Vault, manages the access to settings required by our monitoring services, such as Datadog and Sentry, and set up tests for evaluating your app's behavior.

Konfetti manages your app's configuration settings through lazy evaluation: It only calls and configures what your app needs and when it needs it.

Key benefits:

Configurable lazy evaluation - You can choose the moment when Konfetti will evaluate your the configuration of your app.

Faster & simpler test runs - No need for evaluating the configuration of the whole project if it's not used.

Faster and flexible testing - Isolating small parts of your application no longer requires you to perform a complete setup for each test.

Integration with popular Web Application Frameworks - Konfetti can seamlessly work with Django, Flask, and Celery.

The interface design and features are heavily inspired by decouple, Django, envparse and dynaconf.

Installation

To install konfetti via pip run the following command:

pip install konfetti

Quickstart

Before Konfetti can perform its tasks, you'll need to create a settings module and then tell Konfetti the location of this module.

1. Creating the Settings Module

Please find the application settings, for your production, local, or other environments, using the following path: app_name/settings/production.py

Next, please review the below code block and copy the relevant parts in your settings file.

> ⚠️ Variables need to be named with all uppercase letters, other variables will be ignored

> ⚠️ If your app requires Vault access, then you'll need to specify `VAULT_ADDR` and `VAULT_TOKEN` in the settings module

# app_name/settings/production.py
from konfetti import env, vault

VAULT_ADDR = env("VAULT_ADDR")
VAULT_TOKEN = env("VAULT_TOKEN")

DEBUG = env("DEBUG", default=False)
DATABASE_URI = vault("path/to/db")

Apart from the import statement from konfetti import env, vault, you can remove the settings for the features that you don't use.

If, for instance, you don’t use a database, then you can remove the DATABASE_URI variable. Depending on your settings, it might also be called DB_URI, or similar.

Furthermore, you can remove VAULT_ADDR and VAULT_TOKEN if your app doesn’t require secrets.

2. Configuring the Access Point

# app_name/settings/__init__.py
from konfetti import Konfig, AsyncVaultBackend

config = Konfig(vault_backend=AsyncVaultBackend("/secret/team"))

In your app's environment variables, please add the KONFETTI_SETTINGS variable with the path to your settings module. In the case of the code block above, it would be:

export KONFETTI_SETTINGS=app_name.settings.production

Alternatively the access point could be initiated from an object, importable string, mapping or a JSON file:

Object

class TestSettings:
    VALUE = "secret"
config = Konfig.from_object(TestSettings, ...)

Importable string

config = Konfig.from_object("path.to.settings", ...)

# If the config is in the same module
SECRET = vault("/path/to")["secret"]
config = Konfig.from_object(__name__, ...)

Mapping

config = Konfig.from_mapping({"SECRET": 42}, ...)

JSON

config = Konfig.from_json("/path/to.json")

Usage

The settings module/class with configuration options shouldn't be accessed directly, because the aforementioned features are implemented in the access point level.

from app_name.settings import config

async def something():
    await config.DATABASE_URI  # asynchronously taken from Vault
    debug = config.DEBUG  # Usual sync access

Documentation

For full documentation, please see https://konfetti.readthedocs.io/en/latest/

Or you can look at the docs/ directory in the repository.

Python support

Konfetti supports Python 2.7, 3.5, 3.6, 3.7 and 3.8

License

The code in this project is licensed under MIT license. By contributing to konfetti, you agree that your contributions will be licensed under its MIT license.

konfetti's Issues

Extra casts

For example,

VALUE = env("VALUE", cast=list)

will parse "1,2,3" into ["1", "2", "3"],

also some kind of subcast could be helpful

similar could be done for dict

Support for callable defaults

Usage example:

def read_vault_rancher_token():
    """Read token for thief vault stored in a secret rancher file."""
    open("/run/secrets/vault_token") as secret:
        return secret.read().strip()

VAULT_TOKEN = env("VAULT_TOKEN", default=read_vault_rancher_token)

Additionally, we can add a convention that this callable will receive an instance of KiwiConfig as a first argument

Support for username / password authorization for Vault

Yaml files config support

Maintain a connection pool for Vault

could be helpful and speed up things if connections will be reused, but there should be an API to clean up the resources.

Implement hypothesis tests

E.g. we could implement strategies for different config options and generate config modules on the flight - this will help to validate corner cases, especially for nested structures

Copy vault variable in `[]` access

It will allow better reusage of variables:

dd_vault = vault("datadog")

DATADOG = {
    "ip": VAULT_DATADOG["ip"],
    "port": 8125,
    "app_key": VAULT_DATADOG["app_key"],
    "api_key": VAULT_DATADOG["api_key"],
}

Currently, the same variable is referenced in all cases which leads to accumulating keys, the actual vault variable looks like this:
vault("datadog")["ip"]["app_key"]["api_key"]

Utilize asyncio.gather in `Konfig.require`

If multiple required keys are taken from Vault, then it will be beneficial to access them concurrently, now it is a "for" loop.

Hooks to `Konfig`?

Could be a place to init something after the config is loaded:

config = KiwiConfig(...)

@config.after_load
def init_tracing():
    ddtrace.tracer.configure(hostname=config.DD_HOST, port=config.DD_PORT)`

Ini files config support

SECRET = config.ini("VAR_NAME", default=42, cast=int)

Add mypy to CI

Tracing support?

E.g. optionally trace all requests to Vault

Add changelog

Add a parameter for prefix in environment variables

Goal - avoid incidental usage of some production credentials loaded in the env
An idea to discuss

Add per-option feature to disable caching or setup custom TTL

E.g. if somebody doesn't want to cache a particular variable:

NOT_CACHED = vault("path", ttl=None)

Now we cache the whole secret on the Vault level, not per-item here

Combining multiple configs

Use case.
The main config is in a file, some extras are stored separately in JSON / YAML:

# settings.py

DEBUG = env("DEBUG", default=False, cast=bool)

config = Konfig.from_object(__name__)
config.extend_with_json("path/to.json")
# config.add ?
# config.add_json ?

Django integration

django settings module:

SOMETHING = vault("/path/to")

In the code:

from django.conf import settings

...

The Django's singleton should be used

Database-based config

Schema-based validation for config

KiwiConfig could take marshmallow schema optionally to validate the config just after the module loading

environs supports this - https://github.com/sloria/environs/blob/master/environs.py

Alternative - add validator parameter to config options

Add pytest test skipping based on requried options

@pytest.mark.require_config("USER")
def test_something():

Implement overriding for Vault

Something like

with config.vault.override({"path/to": {"something": "else"}}):

NOTE. For os.environ overriding there could be some issues - the same lower case and upper case string - are there any collisions?

Retries for Vault

some network error could happen on Vault access - there should be a possibility to setup retrying logic

Document architecture decisions

http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions

https://github.com/npryce/adr-tools

Consider changing override rules for Vault

Vault supports paths in different cases - /path/to/UPPER and /path/to/upper are different paths. Having override always working with uppercase version makes impossible to work with both paths. Even though this is a very unlike case, we can have a different approach - use an actual path's name as an environment variable name:

Vault: /path/to/secret/

Env: /path/to/secret/ (maybe treat ones without trailing/leading slashes as equivalents)

Possible benefits:

• Clear mapping between Vault path and env var name; they are the same - /path/to/secret/ will make less confusion than PATH__TO__SECRET
• Possibility to override paths in different cases. Which actually doesn't matter much, but still removes some artificial restrictions

However, possibly it would be better to support both styles at least for some time - to have thief-style non-key level variables overrides working without changing.

Evaluate dictionaries on access

Example:

OPTION = {
    "one": env("ONE"),
    "second": env("SECOND"),
}

dicts will not be recursively evaluated on access. But the usage is:

ddtrace.configure(**config.DATADOG_CONFIG)

Flask integration

Flask.config should be an instance of Konfig

Move documentation to readthedocs.io