martvanrijthoven / dicfg Goto Github PK

The read() method of ConfigReader attempts to read the cli arguments and merge them with the user_config. There are two issues with this:

1. I find this unexpected behaviour and would argue that it makes sense to make this functionality optional, e.g. by adding an optional flag parameter to the read method, specifying whether to add cli arguments or not. A potential situation in which I can see this causing issues is if I run my main script with certain cli arguments that are meant for other parts of the script and that can potentially conflict with config parameters in my user_config. For example, if I were to specify an X number of cpus as a cli arg, but I want Y cpus for the batchiterator (configured in the iterator_config file).

2. The more important issue is that the _read_cli method causes an error in my case:

    File "/usr/local/lib/python3.9/site-packages/wholeslidedata/iterators/batchiterator.py", line 150, in create_batch_iterator
      config = config_reader.read(user_config=user_config, presets=presets)
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 102, in read
      cli_config = self._read_cli(sys.argv[1:])
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 138, in _read_cli
      dicts.append(_create_dict_from_keys(keys, value))
    File "/usr/local/lib/python3.9/site-packages/dicfg/reader.py", line 160, in _create_dict_from_keys
      value = ast.literal_eval(value)
    File "/usr/local/lib/python3.9/ast.py", line 62, in literal_eval
      node_or_string = parse(node_or_string, mode='eval')
    File "/usr/local/lib/python3.9/ast.py", line 50, in parse
      return compile(source, filename, mode, flags,
    File "<unknown>", line 1
      <a_redacted_path>
      ^
  SyntaxError: invalid syntax

ast.literal_eval raises a SyntaxError because the value it tries to read is a string containing some path, e.g. /data/pathology/users/robin/some_path, which it cannot parse. It would need to have an extra set of quotation marks around it to be parsed properly by literal_eval,.