bswck / configzen

Manage configuration with pydantic.

Proven practical knowledge of Python, code quality assurance and delivering production-ready, documented & well-tested software.
With numerous FOSS contributions and passion for learning, I am #opentowork—reach me out:

• via LinkedIn (bswck),
• via Discord (bswck),
• via e-mail ([email protected]).

🔗 My resume.

configzen's Issues

Configuration transclusion and query language

Describe the use case of a new functionality

Structuring complex configuration without repeating yourself.
In detail, the functionality should bring new query language for loading and managing the configuration.
This would definitely help to maintain a neat and tidy, yet complex configuration system.

Example Use

Having

class MyI18n(ConfigModel):
    locale: str
    timezone: str

we could configure it with a single file

# path/to/i18n.yaml
locale: pl_PL
timezone: Europe/Warsaw

and then load it with

MyI18n.load("path/to/path/to/i18n.yaml")

However, if we want to maintain multiple configuration files that should actually do something similar to 'transcluding' this configuration internally, we need to either have identical copies of the common configuration in them (and then if one changes, the other may be inconsistent with it), or have something that performs the transclusion from path/to/i18n.yaml in those configuration files. That transclusion would happen without changing the documents, since the configzen library is built on abstraction over file formats – thus, in a YAML configuration we could then even import a JSON config.

Assuming that both for production and development mode of an example considered application we use the following code to load configuration:

class MyConfig(ConfigModel):
    i18n: MyI18n
    # other independent config variables...

config = MyConfig.load("path/to/development.yaml" if __debug__ else "path/to/production.yaml")

then instead of maintaing separately i18n section in file

# path/to/development.yaml
i18n:  # as in path/to/i18n.yaml
  locale: pl_PL
  timezone: Europe/Warsaw
# other independent config variables...

and separately maintaing i18n section in file

# path/to/production.yaml
i18n:  # as in path/to/i18n.yaml
  locale: pl_PL
  timezone: Europe/Warsaw
# other independent config variables...

and wasting time on worrying whether they are the same as they should,
we could have

# path/to/development.yaml
i18n: 
  .import: path/to/i18n.yaml
# other independent config variables...

and

# path/to/production.yaml
i18n: 
  .import: path/to/i18n.yaml
# other independent config variables...

Please note that I could rewrite these examples into JSON, TOML or anything else supported by anyconfig and it would still work identically.

These import() statements could be even part of a rich query language, that could merge imported configurations, such as

options: 
 .merge:
   .import: path/to/base.yaml
   .import: path/to/overrides.yaml

to merge dictionaries out of imported base.yaml and overrides.yaml.
Namely, that would result in

AppropriateConfigModel(options={**anyconfig.load("base.yaml"), **anyconfig.load("overrides.yaml")})

Moreover, accessing particular scopes of the imported configurations would be considered useful.
We could make a default-settings.yaml with default configs dedicated for other models.

# path/to/default-settings.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  port: 5432

and then, in the application development configuration:

# path/to/development.yaml
i18n: 
  .import(i18n): path/to/default-settings.yaml
database:
  name: postgres_dev
  password: 
    .import(database.password): path/to/default-settings.yaml
  port: 
    .import(database.port): path/to/default-settings.yaml

and in the application production configuration:

# path/to/production.yaml
i18n: 
  .import(i18n): path/to/default-settings.yaml
database:
  .import: path/to/default-settings.yaml

# path/to/production.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  .import(database): path/to/default-settings.yaml
  name: postgres_dev

And this way, we would create inheritance of configurations:

# path/to/production.yaml
.import: path/to/default-settings.yaml  # base configuration
database:
  # inform that we inherit from path/to/default-settings.yaml database section, 
  # because now we just overwrite that imported section and YAML will 
  # otherwise forget about it (it does not merge)
  .import(database): .
  # overwrite desired option with our custom value
  name: postgres_dev

which would be equivalent to:

# path/to/production.yaml
# start import(path/to/default-settings.yaml)
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  port: 5432
# end .import: path/to/default-settings.yaml
database:
  # start .import(database): path/to/default-settings.yaml
  name: postgres
  password: myapp
  # end .import(database): path/to/default-settings.yaml
  name: postgres_dev

that eventually evaluates to

# path/to/production.yaml
i18n:
  locale: pl_PL
  timezone: Europe/Warsaw
database:
  name: postgres
  password: myapp
  name: postgres_dev

Additional context

No response

Add unit tests

Describe the use case of a new functionality

Software without unit tests is like a child without shoes.

Prone to pain.

Example Use

(Does not apply)

Additional context

No response

`wrap_module` has limitations with relative imports

Describe the bug

importlib.import_module requires the package argument to be non-null in the case of a relative import (e.g. importlib.import_module("..abc") is not allowed). This causes problems when trying to import through a directory in the wrap_module method. Take the following directory tree as an example:

.
├── module
│   └── conf.py
└── my_config.py

This will cause a TypeError when trying to call wrap_module to import my_config in configzen, because the top level module directory is not a python package, causing the passed value to import_module to be None.

To reproduce

# the error is relative to the filesystem, so heres a quick script that would generate a repro
import os

os.mkdir("./repro")
os.mkdir("./repro/module")

with open("./repro/module/conf.py", "w") as f:
    f.write(
        """from configzen import ConfigModel

class Test(ConfigModel):
    host: str
    port: int


Test.wrap_module(".my_config")"""
    )

with open("./repro/my_config.py", "w") as f:
    f.write(
        """host = '0.0.0.0'
port = 5000"""
    )

print("now try python3 repro/module/conf.py")

Error

Traceback (most recent call last):
  File "/home/zero/Desktop/Projects/Python/repro/module/conf.py", line 8, in <module>
    Test.wrap_module(".my_config")
  File "/home/zero/.local/lib/python3.10/site-packages/configzen/model.py", line 2430, in wrap_module
    importlib.import_module(module_name, package=package)
  File "/usr/lib/python3.10/importlib/__init__.py", line 121, in import_module
    raise TypeError(msg.format(name))
TypeError: the 'package' argument is required to perform a relative import for '.my_config'

Versions

• OS: Arch Linux
• Python: 3.10.10

Additional context

No response

Support comment preservation

Describe the use case of a new functionality

Let's go. I'll put a plan here, once I have one.
The initial idea is to keep the initial configuration structure in some kind of graph and then use it when exporting the configuration.

Example Use

Model.export(preserve_comments=True)
(after migration to pydantic v2)
Model.export_model(preserve_comments=True)

Additional context

No response

CLI for config operations

Use case

[config.yml]

foo:  # a comment
  bar: 5
  biz:
    baz: 10

> configzen foo.bar  # automatically detect it's about config.yml`
5

> configzen foo.bar=6  # set foo.bar to 6, preserve comments`
[config.yml] Set foo.bar to 6

Example usage

No response

Additional context

No response

Support 3.8+

Describe the use case of a new functionality

End of support: 2024-10 (ref).

Example Use

Does not apply.

Additional context

Credit to @ZeroIntensity for the idea.

Support for autosave

Describe the use case of a new functionality

I guess save config on every assignment/update(). Not sure how that could look like yet, especially in the asynchronous environment where that would be blocking.

Example Use

(Does not apply)

Additional context

No response

Implement own async-reading utilities

Describe the use case of a new functionality

Something like https://docs.hikari-py.dev/en/latest/_modules/hikari/files/#AsyncReader.

Example Use

poetry remove aiofiles :)

Additional context

No response

Add support for CLI model fields

Describe the use case of a new functionality

Easily change settings just from process arguments. There's probably something like that in Hydra, unsure.

Example Use

from configzen import ConfigModel, ConfigField, cli_param

class MyModel(ConfigModel):
    debug: bool = ConfigField(alias="debug_mode", cli_param=cli_param("-d", "--debug"))
    verbose: bool = ConfigField.cli_param("-v", "--verbose")

    def _cli_callback(self):
        ...

Integrate with argparse:

1. via arguments

   parser = argparse.ArgumentParser()
   ...
   arguments = MyModel.cli.get_parser_arguments()
   for nf_args, options in arguments.items():
       parser.add_argument(*nf_args, **options)

2. as a subparser

   parser = argparse.ArgumentParser()
   subparsers = parser.add_subparsers()
   ...
   subparsers.add_parser(MyModel.cli.as_parser("name"))

Integrate with click:

if __name__ == "__main__":
    MyModel.cli()

Integrate with typer:

app = typer.Typer()
app.add_typer(MyModel.cli.typer())

Additional context

No response

Migrate to pydantic v2

Describe the use case of a new functionality

I wish I really didn't have to.

Example Use

Does not apply

Additional context

https://docs.pydantic.dev/latest/migration/

LinkedRoute

Use case

A config route that supports type-safety and runtime validation.

If we have

class ConfItem(BaseConfiguration):
    pair: tuple[int, int]


class Conf(BaseConfiguration):
    item: ConfItem

then instead of writing

Conf.at("item.pair[0]")

write

Conf.at(Conf.item.pair[0])

where

Conf.item.pair[0]

is in fact

LinkedRoute(Conf, Route([GetAttr("item"), GetAttr("pair"), GetItem(0)]))

at runtime.

Example usage

No response

Additional context

No response

Contrib: changing the configuration with paginated UIs

Describe the use case of a new functionality

One of the reasons for the creation of configzen was its scope management usefulness.
For example when modifying the settings through the application that uses configzen, one might want to make a nice UI for modifying particular configuration. When configuration gets really big, it might be really hard to fit it one form.

New functionality: abstract framework for configzen UIs and pagination.

Example Use

(Does not apply)

Additional context

No response

Support for Python configuration

Describe the use case of a new functionality

Allow Python modules to serve as configuration resources.

Example Use

^extend: programmatic_config.py
...

Additional context

No response

Properly internally type-annotate binary blobs

Describe the use case of a new functionality

Does not apply.

Example Use

Does not apply.

Additional context

No response

Add coverage

Describe the use case of a new functionality

No serious package doesn't use coverage.

Example Use

Does not apply

Additional context

No response