Coder Social home page Coder Social logo

grimen / python-config2 Goto Github PK

View Code? Open in Web Editor NEW
20.0 5.0 9.0 119 KB

Python environment configuration simplified - highly inspired by `node-config`.

Home Page: https://pypi.org/project/config2

License: MIT License

Makefile 4.69% Python 95.31%
python configuration config environment environments settings application configurations env node-config

python-config2's Introduction

config2 PyPI version Build Status Coverage Status

Python environment configuration simplified.

Introduction

Config2 (for Python) - which is highly inspired by node-config - organizes hierarchical configurations for your app deployments.

It lets you define a set of default parameters, and extend them for different deployment environments (development, qa, staging, production, etc.).

Configurations are stored in configuration files within your application, and can be overridden and extended by environment variables, command line parameters, or external sources.

This gives your application a consistent configuration interface shared among a growing list of npm modules also using node-config.

NOTE: This project is more or less in pair with node-config implementation, with exception for some fluff that could be considered too much magic such as deployment specific multi-instance deployments which I so far haven't found any good motivation for, and some other questionable advanced features mentioned in the wiki pages.

Project Guidelines

...based on node-config project guidelines:

  • Simple - Get started fast
  • Powerful - For multi-node enterprise deployment - excluded because with power comes responsability
  • Flexible - Supporting multiple config file formats
  • Lightweight - Small file and memory footprint
  • Predictable - Well tested foundation for module and app developers

Install

Install using pip:

$ pip install config2

Use

1. Assuming we have a python application project...

some_project
└── app.py

app.py - some app making serious $$$

# business logic
print('$$$')

2. Let's add some environment specific config files...

some_project
└── config
    ├── default.yml
    ├── development.yml
    ├── foo.yml
    └── production.yml
└── app.py

default.yml - with some bogus nested settings shared for all environments (defaults)

a1: DEFAULT 1
a2:
    b1: [1, 2, 3]
    b2:
        -   foo
        -   bar
    b3:
        c1: 1
        c2: "DEFAULT 2"

development.yml - with some bogus nested settings overriden for development environment (overriden)

a2:
    b2:
        -   DEV 1
    b3:
        c2: "DEV 2"
some_key_only_for_dev: true

foo.yml - with some bogus nested settings overriden for foo environment (overriden)

a2:
    b2:
        -   FOO 1
    b3:
        c2: "FOO 2"
some_key_only_for_foo: true

production.yml - with some bogus nested settings overriden for production environment (overriden)

a2:
    b2:
        -   PROD 1
    b3:
        c2: "PROD 2"
some_key_only_for_prod: true

3. Let's now run the app using various environments...

$ python app.py

from config2.config import config

config.get_env() # => None
config.get() # => {'a1': 'DEFAULT 1', 'a2': {'b1': [1, 2, 3], 'b2': ['foo', 'bar'], 'b3': {'c1': 1, 'c2': 'DEFAULT 2'}}}

config.a1 # => 'DEFAULT 1'
config.a2 # => {'b1': [1, 2, 3], 'b2': ['foo', 'bar'], 'b3': {'c1': 1, 'c2': 'DEFAULT 2'}}
config.a2.b3.c2 # => 'DEFAULT 2'

print('$$$')

$ ENV=development python app.py

from config2.config import config

config.get_env() # => 'development'
config.get() # => {'a1': 'DEFAULT 1', 'a2': {'b1': [1, 2, 3], 'b2': ['DEV 1'], 'b3': {'c1': 1, 'c2': 'DEV 2'}}, 'some_key_only_for_dev': True}

config.a1 # => 'DEFAULT 1'
config.a2 # => {'b1': [1, 2, 3], 'b2': ['DEV 1'], 'b3': {'c1': 1, 'c2': 'DEV 2'}}
config.a2.b3.c2 # => 'DEV was here 2'

config.some_key_only_for_dev # => True

config.some_key_only_for_foo # => AttributeError
config.some_key_only_for_prod # => AttributeError

print('$$$')

$ ENV=foo python app.py

from config2.config import config

config.get_env() # => 'foo'
config.get() # => {'a1': 'DEFAULT 1', 'a2': {'b1': [1, 2, 3], 'b2': ['FOO 1'], 'b3': {'c1': 1, 'c2': 'FOO 2'}}, 'some_key_only_for_foo': True}

config.a1 # => 'DEFAULT 1'
config.a2 # => {'b1': [1, 2, 3], 'b2': ['FOO 1'], 'b3': {'c1': 1, 'c2': 'FOO 2'}}
config.a2.b3.c2 # => 'FOO was here 2'

config.key_only_for_foo # => True

config.some_key_only_for_dev # => AttributeError
config.some_key_only_for_prod # => AttributeError

print('$$$')

$ ENV=production python app.py

from config2.config import config

config.get_env() # => 'production'
config.get() # => {'a1': 'DEFAULT 1', 'a2': {'b1': [1, 2, 3], 'b2': ['PROD 1'], 'b3': {'c1': 1, 'c2': 'PROD 2'}}, 'some_key_only_for_foo': True}

config.a1 # => 'DEFAULT 1'
config.a2 # => {'b1': [1, 2, 3], 'b2': ['PROD 1'], 'b3': {'c1': 1, 'c2': 'PROD 2'}}
config.a2.b3.c2 # => 'PROD was here 2'

config.some_key_only_for_prod # => True

config.some_key_only_for_dev # => AttributeError
config.some_key_only_for_foo # => AttributeError

print('$$$')

etc.

4. Optionally, let's now introduce custom config environment variables...

some_project
└── config
    ├── custom-environment-variables.yml
    ├── default.yml
    ├── development.yml
    ├── foo.yml
    └── production.yml
└── app.py

custom-environment-variables.yml - with mappings of config keys to environment variables

a1: A1
a2:
    b3:
        c2: C2

5. Let's now run the app using custom environment variables to override config...

$ A1=x C2=y python app.py

from config2.config import config

config.get_env() # => None
config.get() # => {'a1': 'x', 'a2': {'b1': [1, 2, 3], 'b2': ['foo', 'bar'], 'b3': {'c1': 1, 'c2': 'y'}}}

config.a1 # => 'x'
config.a2 # => {'b1': [1, 2, 3], 'b2': ['foo', 'bar'], 'b3': {'c1': 1, 'c2': 'y'}}
config.a2.b3.c2 # => 'y'

print('$$$')

Test

Clone down source code:

$ make install

Run colorful tests, with only native environment (dependency sandboxing up to you):

$ make test

Run less colorful tests, with multi-environment (using tox):

$ make test-tox

About

This project was mainly initiated - in lack of existing alternatives - to be used at our work at Markable.ai to have common code conventions between various programming environments where Python (research, CV, AI) and Node.js (I/O, APIs, UIs, scripts) currently are most used.

License

Released under the MIT license.

python-config2's People

Contributors

grimen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

python-config2's Issues

YAML load warning

Versions

  • Config2 - 0.3.1
  • Python - 3.6.3

Issue
Following YAML loading warning is been thrown:

/usr/local/lib/python3.6/site-packages/config2/serializers/yaml_.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Dev dependencies not separated

First of all, thanks for building this library! I was banging my head against a wall trying to find a simple-yet-powerful application config approach like node-config in the Python ecosystem.

Unfortunately, pip-installing config2 installs its development dependencies as well. As of 2020, I would suggest to simply replace the setuptools-setupextras-based approach with Poetry to reduce manual boilerplate code. I would be happy to submit a PR if you welcome this.

I also have a few more problems and solutions in mind. Let me know if you welcome contributions, and I will file them in separate issues. Cheers!

Broken module requirements (mybad)

Hi Jonas,

I have an issue during the install of the new version of config2 (0.3.2).

It seems that there is an issue with the requirements of the mybad module.
Please find attached the log file (clean install from a docker image).

$ pip install config2
[...]
Collecting mybad>=0.1.4 (from config2)
  Downloading https://files.pythonhosted.org/packages/fb/62/5366bda200b009452f4446ade7758f0a18be21a3131ca5e541d4b07e6d4e/mybad-0.2.0.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-s_jj15d_/mybad/setup.py", line 8, in <module>
        import setupextras
    ModuleNotFoundError: No module named 'setupextras'

Thank you for your help (and for the package ;))!

Jean-Baptiste

error-config2.log

How to run this command on windows system ENV=development python app.py

I' unable to execute the ENV=development python app.py on windows 10 .
I tried executing the following command

set ENV=development & python app.py

Output:
DEFAULT 1
{'b1': [1, 2, 3], 'b2': ['foo', 'bar'], 'b3': {'c1': 1, 'c2': 'DEFAULT 2'}}
DEFAULT 2
config.some_key_only_for_dev 'some_key_only_for_dev'
config.some_key_only_for_foo 'some_key_only_for_foo'
config.some_key_only_for_foo 'some_key_only_for_prod'
$$$
Excepted result
DEFAULT 1
{'b1': [1, 2, 3], 'b2': ['DEV 1'], 'b3': {'c1': 1, 'c2': 'DEV 2'}}
DEV 2
config.some_key_only_for_dev True
config.some_key_only_for_foo 'some_key_only_for_foo'
config.some_key_only_for_foo 'some_key_only_for_prod'

Feature: `config.truthy` helper

Useful for less verbose environment flag parsing.

Example:

config.truthy(None) # => False
config.truthy(False) # => False
config.truthy(0) # => False
config.truthy("0") # => False
config.truthy("false") # => False
config.truthy("False") # => False
config.truthy("FALSE") # => False
config.truthy("no") # => False # e.g. YAML
config.truthy("No") # => False # e.g. YAML
config.truthy("NO") # => False # e.g. YAML
config.truthy("") # => False
config.truthy("nil") # => False
config.truthy("null") # => False

config.truthy(True) # => True
config.truthy(1) # => True
config.truthy("1") # => True
config.truthy("true") # => True
config.truthy("True") # => True
config.truthy("TRUE") # => True
config.truthy("yes") # => True # e.g. YAML
config.truthy("Yes") # => True # e.g. YAML
config.truthy("YES") # => True # e.g. YAML
config.truthy({}) # => True
config.truthy([]) # => True

config.get_env() AttributeError

As I follow the README, I come across the issue that config.get_env() raises an AttributeError
By looking through config.py, I can't seem to find the function too.
What am I missing here?

pip install config2 still installs release 0.2.0

Hi,

I still get release 0.2.0 if I run "pip install config2", which contains the bogus custom-environment-variables.yml handling. I need to install from Github directly to get version 0.2.1.

Also, get_env() still does not work.

However, very nice project! Perfectly fits my needs.

Best - André.

how to use multiple config file in one enviroment

Sorry, cant find contact to write PM
As i understand the one file for one environment
But is it possible to load multiple file in default enviroment?

if I use:
from config2.config import config
config.values

it return values from all enviroment but this is not able with using dict keys as method

is it possible?

thanks

defaults override custom-envoronment-variables

Hi,

It appears that the default.yml is overriding the custom-environment-variables.yml defined configuration.

Python Version: 2.7.8

Project Structure

  • ./requirements.txt
  • ./.python-version
  • ./config/custom-environment-variables.yml
  • ./config/default.yml
  • ./main.py
  • ./env

main.py

from config2.config import config

import os

print(os.environ['APP']) 
print('port', config.port) 

config/default.yml

port: 3000

config/custom-environment-variables.yml

port: APP

When run with python main.py config returns the default (3000) as expected
When run with APP=1111 python main.py config still returns 3000 where I expected 1111 however the APP environment variable is returning 1111.

Is virtualenv causing a problem here?

requirements.txt

attributedict==0.1.8
blessings==1.7
Click==7.0
colour-runner==0.1.1
config2==0.1.4
deepdiff==3.3.0
deepmerge==0.0.4
easypackage==0.1.8
filelock==3.0.9
itsdangerous==0.24
Jinja2==2.10
jsonpickle==1.0
MarkupSafe==1.0
pluggy==0.7.1
py==1.6.0
Pygments==2.2.0
PyYAML==3.13
six==1.11.0
toml==0.10.0
tox==3.5.2
virtualenv==16.0.0
Werkzeug==0.14.1

Thanks for your work btw

EDIT:
Just to add to this, defining configuration in another file works fine.
i.e config/foo.yml
and executing ENV=foo python main.py will return the correct configuration.

Enhancement: Error messages not very human friendly - should be very specific why something failed (user error vs system error)

Example:

[2019-01-16 20:54:39 -0500] [11635] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/Users/grimen/.pyenv/versions/3.6.6/envs/3.6.6-default/lib/python3.6/site-packages/attributedict/collections.py", line 223, in __getattr__
    return self.__getitem__(key)
  File "/Users/grimen/.pyenv/versions/3.6.6/envs/3.6.6-default/lib/python3.6/site-packages/attributedict/collections.py", line 165, in __getitem__
    result = self.__dict__.__getitem__(key)
KeyError: 'nn'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.