martvanrijthoven / dicfg

Dicfg is a configuration system that supports dependency injection via object interpolation in config files.

Dicfg is a configuration system that supports dependency injection via object interpolation in config files.

Main Features

• Loading of predefined config files (YAML and JSON).
• Overwrite config with user_config files/dictionaries, command line interface, and/or presets.
• Customize merge/replace behavior for dictionaries and lists.
• Interpolation support for sub-config files, config variables, environment variables, and python objects.
• Build object instances directly in the config.
• Dependency injection via object interpolation: configure all object dependencies directly in the config.
• Use object attribute interpolation for referencing object attributes directly in the config file.

Installation

pip install dicfg

conda install -c conda-forge dicfg

Docs

https://martvanrijthoven.github.io/dicfg/

dicfg's Issues

ConfigReader.read() fails on reading cli_config

The read() method of ConfigReader attempts to read the cli arguments and merge them with the user_config. There are two issues with this:

1. I find this unexpected behaviour and would argue that it makes sense to make this functionality optional, e.g. by adding an optional flag parameter to the read method, specifying whether to add cli arguments or not. A potential situation in which I can see this causing issues is if I run my main script with certain cli arguments that are meant for other parts of the script and that can potentially conflict with config parameters in my user_config. For example, if I were to specify an X number of cpus as a cli arg, but I want Y cpus for the batchiterator (configured in the iterator_config file).

2. The more important issue is that the _read_cli method causes an error in my case:

    File "/usr/local/lib/python3.9/site-packages/wholeslidedata/iterators/batchiterator.py", line 150, in create_batch_iterator
      config = config_reader.read(user_config=user_config, presets=presets)
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 102, in read
      cli_config = self._read_cli(sys.argv[1:])
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 138, in _read_cli
      dicts.append(_create_dict_from_keys(keys, value))
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 160, in _create_dict_from_keys
      value = ast.literal_eval(value)
    File "/usr/local/lib/python3.9/ast.py", line 62, in literal_eval
      node_or_string = parse(node_or_string, mode='eval')
    File "/usr/local/lib/python3.9/ast.py", line 50, in parse
      return compile(source, filename, mode, flags,
    File "<unknown>", line 1
      <a_redacted_path>
      ^
  SyntaxError: invalid syntax

ast.literal_eval raises a SyntaxError because the value it tries to read is a string containing some path, e.g. /data/pathology/users/robin/some_path, which it cannot parse. It would need to have an extra set of quotation marks around it to be parsed properly by literal_eval,.

Add features to docs

• name in config (v0.1.0)
• Reference mapping symbol ":"
• Presets declarations in config
• Don't build option
• explain yaml functionality anchors and aliases and how to use
• Explain the decision to avoid inner auto-config loading and use an intermediate variable at the context level instead.