Seems, like conform.new can't accept all config types on generation.

proxy = [{:default_route, {{127, 0, 0, 1}, 1813, "secret"}},
         {:options, [{:type, :realm}, {:strip, true}, {:separator, '@'}]},
         {:routes, [{'test', {{127, 0, 0, 1}, 1815, "secret"}}]}]

config :eradius, :servers,
  proxy: {'127.0.0.1', [1812, 1813]}

config :eradius,
  session_nodes: [node],
  proxy: [{ {:eradius_proxy, 'proxy', proxy }, [{'127.0.0.1', "secret"}] }],
  logfile: './radius.log',
  client_ip: {0, 0, 0, 0}

** (Protocol.UndefinedError) protocol Enumerable not implemented for "secret"
    (elixir) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir) lib/enum.ex:112: Enumerable.reduce/3
    (elixir) lib/enum.ex:1398: Enum.reduce/3
    (elixir) lib/enum.ex:1106: Enum.map_reduce/3
    (elixir) lib/macro.ex:226: Macro.do_postwalk/3
    (elixir) lib/macro.ex:234: Macro.do_postwalk/3
    (elixir) lib/enum.ex:1102: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir) lib/macro.ex:226: Macro.do_postwalk/3


07:57:38.606 [error] Error in process <0.46.0> with exit value: {#{'__exception__'=>true,'__struct__'=>'Elixir.Protocol.UndefinedError',description=>nil,protocol=>'Elixir.Enumerable',value=><<6 bytes>>},[{'Elixir.Enumerable','impl_for!',1,[{file,"lib/enum.ex"},{line,1}]},{'Elixir.Enumerable',reduce,3,[{file...

Hello,
I know that if the config.exs imports a config from a submodule, conform imports the settings from the imported config and adds them to the schema when using mix conform.new. Never the less one would have to recreate the schema every time a dependency changes its settings or translations.
As a feature request I would like to ask for a function to import configurations of dependencies to add them into the configuration of the main applications.

As an example of how it should be used:
deps/dependency/config/dependency.schema.exs

[
  mappings: [
    "dependency.setting": [
      doc: "Setting of dependency",
      datatype: :binary,
      default: "foo"
  ],
  translations: []
]

config/myapp.schema.exs

[
  import_settings: [
    :dependency
  ],
  mappings: [
    "myapp.setting": [
      doc: "Setting of myapp",
      datatype: :binary,
      default: "bar"
  ],
  translations: []
]

using mix conform.configure results in

# Setting of dependency
dependency.setting = "foo"

# Setting of myapp
myapp.setting = "bar"

The schemas must also be imported somehow of course to use the translations.

Sincerely,
Tobias

When running mix conform.effective you have to have a myapp.conf file already built. This wasn't immediately obvious to me, so I ended up running it expecting to see some output generated by my schema. This resulted in an undefined function error.

I think it would be pretty easy and helpful to add a check for whether or not a .conf file exists before executing conform.effective, and print a warning instead of users seeing a stacktrace.

For configuring guardian's ttl, the configuration expects a tuple in the form of {30, :days}. No matter what I do, conform seems to always turn this into [30, :days], which causes guardian's internal pattern matching to fail. Is there anyway to configure this?

Here's a link to guardian:
https://github.com/ueberauth/guardian#installation

I'm just going to say ahead of time this is my first Elixir/Erlang deploy (with or without exrm/conform) so be kind. I'm also assuming this is my own error and not a bug. I'm just not sure of a better way to get help/feedback. I appreciate your time.

I am using exrm 1.0.0-rc7 and conform 1.0.0-rc8.

When I build on the same machine I want to deploy on, the conf file that has the correct values for our production env are not being applied correctly. Specifically, I have a list of kafka brokers defined as a [list: :ip] data type in the schema. The conf file has this list of broker ips defined, but app start fails.

From the crash dump:

$ head erl_crash.dump
=erl_crash_dump:0.3
Wed Jan 13 22:41:57 2016
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient'
System version: Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]

When I enter the console I get a KafkaNetwork exception as it's trying to connect to a local Kafka instance (the default value in the schema).

$ gretel console
Using /app/gretel/current/releases/0.0.1/gretel.sh
Exec: /app/gretel/current/erts-7.2/bin/erlexec -boot /app/gretel/current/releases/0.0.1/gretel -boot_var ERTS_LIB_DIR /app/gretel/current/erts-7.2/../lib -env ERL_LIBS /app/gretel/current/lib -config /app/gretel/current/running-config/sys.config -args_file /app/gretel/current/running-config/vm.args -user Elixir.IEx.CLI -extra --no-halt +iex -- console
Root: /app/gretel/current
/app/gretel/current
Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]


22:50:26.996 [info]  Child Logger.ErrorHandler of Supervisor Logger.Supervisor started
Pid: #PID<0.68.0>
Start Call: Logger.Watcher.watcher(:error_logger, Logger.ErrorHandler, {true, true, 500}, :link)
Restart: :permanent
Shutdown: 5000
Type: :worker

22:50:27.012 [info]  Application logger started at :"[email protected]"

22:50:27.014 [info]  Application poison started at :"[email protected]"

22:50:27.014 [error] Could not connect to broker "127.0.0.1" on port 9092
{"Kernel pid terminated",application_controller,"{application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient',send_sync_request,3,[{file,\"lib/kafka_ex/network_client.ex\"},{line,30}]},{'Elixir.Enum',do_find_value,3,[{file,\"lib/enum.ex\"},{line,2490}]},{'Elixir.KafkaEx.Server',metadata,5,[{file,\"lib/kafka_ex/server.ex\"},{line,249}]},{'Elixir.KafkaEx.Server',init,1,[{file,\"lib/kafka_ex/server.ex\"},{line,26}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]},{'Elixir.KafkaEx',start,[normal,[]]}}}"}

Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) ({application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient'

# schema file
# ...
    "kafka_ex.brokers": [
      doc: "Provide documentation for kafka_ex.brokers here.",
      to: "kafka_ex.brokers",
      datatype: [ list: :ip ],
      default: [{"127.0.0.1", 9092}]
    ],
# ...

# gretel.conf
kafka_ex.brokers = some.ip.range:9092, some.ip.range:9092, some.ip.range:9092

I have placed my gretel.conf in the release version directory /app/gretel/current/releases/0.0.1/gretel.conf. With previous versions of the hex package, the gretel.conf was copied to /app/gretel/current/running-config/gretel.conf, but this current version of conform doesn't do that.

At this point I'm unsure where to go. As indicated I've tried rolling back to the last stable releases of both exrm and conform, but that ran me into a separate set of issues. Specifically, running escript with the built conform file was failing with a message saying that conform_escript function was undefined or something.

I'm sorry if that's too much info, but trying to do my homework here and not submit a bogus issue. It appears that I'm missing a simple step, but I'll be damned if I can find it. Any help would be greatly appreciated. Thanks!

I used mix conform.new to generate a .schema from my config.exs in which I had this:

config :logger, :console,
  level: :debug,
  format: "$time $metadata[$level] $levelpad$message\n"

the generated .schema (and .config) didn't properly include the \n leading to no newlines in my logger.

I guess the recommended workaround is to write my own translation for logger.console.format that special cases \n characters?

It would be very nice to have validation types.

Example:

  datatype: Module

Where Module implements behaviour, implements function validate, which returns {:ok, value} | {:error, error}

It will allow to build own validation libraries on top of conform.

Hi there.

I have the following chunk of awesomeness in my config.exs

config :rocket, Rocket.Endpoint,
  url: [host: "localhost"],
  root: Path.dirname(__DIR__),
  secret_key_base: "Hoopdie doopdie doo",
  debug_errors: false,
  http: [
    dispatch: [
      {:_, [
          {"/ws",               Pixie.Adapter.Cowboy.HttpHandler, {Pixie.Adapter.Plug, []}},
          {"/messagerocket.js", :cowboy_static,                   {:file, Path.join(Path.dirname(__DIR__),"priv/static/messagerocket.js")}},
          {:_,                  Plug.Adapters.Cowboy.Handler,     {Rocket.Endpoint, []}}
        ]
      }
    ]
  ]

And when I run mix conform.new I get the following amazing backtrace:

** (FunctionClauseError) no function clause matching in Macro.do_traverse_args/4
    (elixir) lib/macro.ex:290: Macro.do_traverse_args({Pixie.Adapter.Plug, []}, [], #Function<16.54478476/2 in Macro.postwalk/3>, #Function<11.54688206/2 in Conform.Utils.Code.stringify/1>)
    (elixir) lib/macro.ex:269: Macro.do_traverse/4
    (elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir) lib/macro.ex:282: Macro.do_traverse/4
    (elixir) lib/macro.ex:277: Macro.do_traverse/4
    (elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir) lib/macro.ex:282: Macro.do_traverse/4

I am so excited to raise this bug for you to fix. You must be very happy as the maintainer of a popular open source project to serve my every whim.

All kidding aside, this is with conform 2.1.1 on Elixir 1.3.2 with Erlang 19. If there's any way I can help you diagnose this then feel free to ask.

config.exs contains

config :engine,
  host_id: "some uuid",

the generated engine.schema.exs:

    "engine.host_id": [
      commented: false,
      datatype: :binary,
      default: "nice uuid",
      doc: "Provide documentation for engine.host_id here.",
      hidden: false,
      to: "engine.host_id"
    ],

when I generate engine.conf, it contains what I understand to be spurious double quotes:

# Provide documentation for engine.host_id here.
engine.host_id = "nice uuid"

And then this seems to be rather wrong:

$ mix conform.effective | grep host_id
 engine: [host_id: "\"nice uuid\"", middlewares: [],

Not a showstopper, I am cleaning up the spurious double-quotes. But buggy.

I'm trying to run the test suite, from master, and I get this as a failing test:

 mix test
...............

  1) test can generate config as Elixir terms from .conf and schema with imports (ConfTranslateTest)
     test/conf_translate_test.exs:57
     ** (MatchError) no match of right hand side value: {:ok, []}
     stacktrace:
       test/conf_translate_test.exs:83: anonymous fn/5 in ConfTranslateTest.test can generate config as Elixir terms from .conf and schema with imports/1
       (ex_unit) lib/ex_unit/capture_io.ex:146: ExUnit.CaptureIO.do_capture_io/2
       (ex_unit) lib/ex_unit/capture_io.ex:119: ExUnit.CaptureIO.do_capture_io/3
       test/conf_translate_test.exs:66

............

Finished in 8.3 seconds (0.1s on load, 8.2s on tests)
28 tests, 1 failure

Randomized with seed 187604

I've looked at the code for that test and the sys.config file remains empty-- not sure what's exactly going on...

hey guys,
I have an umbrella app with 5 apps within umbrella. I use distillery to build the release, and edeliver for deployment.
Distillery is configured to build single release for whole umbrella application which as well results in generating a single sys.config file.

Configuration is rather complex and I was hoping to use conform, but I'm struggling a bit with making it work. For example, when I run mix conform.new it goes into each app and generates a schema for each app under umbrella. Next task mix conform.configure generates .conf files for each app in respective config directory as well.

Now that's what I'm struggling to understand, please correct me if I'm wrong:

• conform is intended for use for built release and translate .conf into sys.config files according to rules in schema file
• schema and config files were generated for individual applications
• distillery is configured to produce single umbrella release which produces a single sys.config (with configs of 5 apps lumped together)

How do I generate single schema and single .conf file with this setup? and how does conform fits into this?

Thank you,
Alex

It would be nice to be able to define a module of functions that can be reused for translations. Either as an importable script, or defined inline. It seems conform.effective handles this already, so potentially the only thing that needs to happen is to fix exrm's conform plugin, and work on some kind of support for importing a script containing the functions to use.

Currently exrm leverages conform to merge all dependency schemas and generate a single schema

    # Get top-level schema...
    schema = app |> String.to_atom |> Conform.Schema.read
    # Get schemas from all dependencies
    dep_schemas = Conform.Schema.coalesce
    # Merge together
    merged = Conform.Schema.merge(dep_schemas, schema)

this would be useful as a conform mix task which one could leverage to output the schema to a file without needing to build a release.

something like mix conform.schema [outfilepath]

Sorry, it was my fault, it was 2 o'clock night.

My project has the config file like following:

config.exs:

use Mix.Config

config :sts,
  key: :value,
  limit: 42

import_config "#{Mix.env}.exs"

config/dev.exs:

use Mix.Config

config :sts,
  env: :dev

config/test.exs:

use Mix.Config

config :sts,
  env: :test

I generate schema.exs as: mix conform.new,
and I find the sts.env is test not dev, shouldn't it dev as default?

Furthermore, when I input as: MIX_ENV=dev mix conform.new, it doesn't change the sts.env

Right now if you have app A which depends on app B, and both have conform schemas, you have to duplicate B's schema info in A if you want to use the .conf to configure both A and B. We can improve the usability and maintainability of configs by merging schemas from dependencies into the schema of the top-level app when generating it.

Potential issues:

• Cyclical dependencies, how do we handle them?
• How deep do we go? One level? N-levels? All the way?
• Are there undesired side-effects by supporting this functionality

There is, as far as I can tell, no way to encode the following config option:

config :my_app, prop_with_quotes: ~s(foo "bar" baz)

I can not reopen #98 so I open a new issue.

on mix conform.effective utf8 encoded content in .conf file is supported,
but in a release it is still transcoded to latin-1.

conform does not support keys which contain .

desired sys.config

[
{my_app, [{'Elixir.Bar', value}]}
]

Conform splits the to keypath on . which makes the above sys.config impossible.

keys can only be atoms

desired sys.config

[{other_app, [{<<"some.string">>, value}]}]

Conform expects each key component to be converted to an atom

Example:

[
  mappings: [
   "db.user": [
      doc: """
      username:password
      """,
      to: "app",
      datatype: :binary,
      default: "root:"
    ]
  ],

  translations: [
    "app": fn
      _, user_pass, acc ->
        case String.split(user_pass, ":") do
          [user, pass] ->
            repo_conf = acc[App.Repo] || []
            put_in((acc || []), [App.Repo], [{:username, user}, {:password, pass} | repo_conf])
          _ ->
            IO.puts("Username and password should be separated with `:`, got: #{user_pass}")
            exit(1)
        end
    end]
]

Conform.Schema.read!("config") |> Conform.Schema.stringify |> IO.puts

transforms to:

[
  mappings: [
    "db.user": [
      doc: """
      username:password
      """,
      to: "app",
      datatype: :binary,
      default: "root:"
    ]
  ],
  translations: [
    app: fn _, user_pass, acc ->
      case String.split(user_pass, ":") do
        userpass ->
          repo_conf = acc[App.Repo] || []
          put_in(acc || [], [App.Repo], [{:username, user}, {:password, pass} | repo_conf])
        _ ->
          IO.puts("Username and password should be separated with `:`, got: #{user_pass}")
          exit(1)
      end
    end
  ]
]

Changes the line to: [user, pass] -> to userpass

CC: @umatomba

Compiling conform in Elixir 1.3 shows three warnings that should be addressed:

warning: the variable "key" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:

    case int do
      1 -> atom = :one
      2 -> atom = :two
    end

should be written as

    atom =
      case int do
        1 -> :one
        2 -> :two
      end

Unsafe variable found at:
  lib/conform/translate.ex:189

warning: function Mix.Dep.children/0 is undefined or private
  lib/conform/schema.ex:395

warning: function Mix.Dep.children/0 is undefined or private
  lib/mix/tasks/conform.archive.ex:24

When using an umbrella application, the config files for any of the apps (but especially the primary application) do not get packaged as part of the exrm release.

I was able to replicate this with https://github.com/bitwalker/exrm-umbrella-test.

$ MIX_ENV=prod mix conform.new
$ MIX_ENV=prod mix conform.configure
$ cd apps/master_app
$ MIX_ENV=prod mix release
$ find . -type f -name "*.conf"

I expected there to be a master_app.conf in rel/master_app/releases/0.0.1/

Hi all (I hope people are listening!),

I've been working on some significant improvements to how conform works today. One of the biggest problems I've been working on is a more intuitive way to do mappings/translations for complex types. For instance, let's say you want to get the following config output:

[ lager: [ handlers: [
  lager_console_backend: [ level: :info ],
  lager_file_backend: [ level: :error, path: "/var/log/error.log" ],
  lager_file_backend: [ level: :info, path: "/var/log/info.log" ] ]

So you write a schema that looks like this:

[ mappings: [
   "lager.handlers.*": [
     to: "lager.handlers",
     datatype: [list: :complex],
     default: [],
     ... ],
   "lager.handlers.console.level": [
     to: "lager.handlers.lager_console_backend.level",
     datatype: [enum: [:info, :error]],
     default: :info,
     ...],
   "lager.handlers.file.*": [
      to: "lager.handlers.lager_file_backend",
      datatype: [list: :complex],
      default: [],
      ...],
   "lager.handlers.file.error": [
      to: "lager.handlers.lager_file_backend.error",
      datatype: :binary,
      default: "/var/log/error.log",
      ...],
    "lager.handlers.file.info": [
      to: "lager.handlers.lager_file_backend.info",
      datatype: :binary,
      default: "/var/log/info.log",
      ...] ],
  translations: [
    "lager.handlers.lager_file_backend.error": fn _mapping, {_, path} ->
      [level: :error, path: path]
    end,
    "lager.handlers.lager_file_backend.info": fn _mapping, {_, path} ->
      [level: :info, path: path]
    end
  ] ]

For one, the above does not work. In fact I've found that there isn't actually a way to do this properly that works to cover a variety of edge cases. It is also unintuitive, contains a lot of what is effectively boilerplate/garbage mappings, and already contains bits which are difficult to understand in context (for instance, what args are passed to the translations, what are their values?).

I'm working on, and would like feedback on my solution to these problems. I'm planning on conforming more closely to the way cuttlefish handles this, so our schema would look more like the following:

[ mappings: [
   "lager.handlers.$backend.level": [
     to: "lager.handlers.lager_$backend_backend.level",
     datatype: [enum: [:info, :error]],
     default: :info, ... ],
   "lager.handlers.file.error": [
      to: "lager.handlers.lager_file_backend.error",
      datatype: :binary,
      default: "/var/log/error.log",
      ...],
    "lager.handlers.file.info": [
      to: "lager.handlers.lager_file_backend.info",
      datatype: :binary,
      default: "/var/log/info.log",
      ...] ],
  translations: [
    "lager.handlers": fn conf ->
        file_handlers = Conform.Conf.get(conf, "lager.handlers.lager_file_backend.$level")
        |> Enum.map(fn {[_, _, backend, level], path} -> {backend, [level: level, path: path]} end)
        console_handlers = Conform.Conf.get(conf, "lager.handlers.lager_console_backend")
        |> Enum.map(fn {[_, _, backend], conf} -> {backend, conf} 
        console_handlers ++ file_handlers
    end
  ] ]

In a nutshell:

• Wildcards in mappings are now bound to variable names, and those variables can be used in the to key path.
• After all mappings are performed, the translations are run, translation keys define the path at which the translated value will be placed. The result of applying the translation will overwrite the value at that path if one exists.
• A new module Conform.Conf will be exposed to schemas, which provides functions for querying the configuration state for data. In the above example, get is a function which allows you to fetch values based on a key path, which can contain variables. The result of get will always be a keyword list, where the key is the tokenized path (as atoms), and the value is the configuration at that path.
• The conf argument will also contain configuration read from config.exs that isn't exposed via the .conf.

I think the above significantly reduces the complexity of conform's implementation, while also providing more powerful tools for manipulating the configuration which is output.

As part of this sweep of changes, I'm also planning on introducing validators, which can be applied to mappings (prior to translations). And the following new mapping settings:

• commented - Boolean, if true will write the default value to the conf, but comment it out.
• required - Boolean, if true will throw an error if the value is not present in the conf
• hidden - Boolean, will not write the setting to the conf when generated, in this way you can define advanced settings which can be configured, but which won't be shown to users of the .conf by default

I'd like to get feedback on all of this, as they are significantly large breaking changes. I wish we could avoid such a large set of changes, but after a lot of work in private, I'm beginning to see how the current architecture is not very flexible moving forward, and is currently error prone for some cases. I really think this is worth the pain, but I want to make sure others feel that way as well.

For many of you, these changes may not actually affect you, but for those using translations, or wildcard mappings, it will.

warning: redefining @doc attribute previously set at line 10
  lib/conform/schema.ex:17: Conform.Schema (module)

warning: redefining @doc attribute previously set at line 9
  lib/conform/translate.ex:16: Conform.Translate (module)

warning: function Enum.member/2 is undefined or private
  lib/conform/schema.ex:255

warning: function TranslateErorr.exception/1 is undefined (module TranslateErorr is not available)
  lib/conform/translate.ex:398

I must add I'm new to the elixir world, so these may be my own fault..

The Readme says

I currently am planning on supporting user-defined types, but that work has not yet been completed.

but then it also shows an example of creating a custom type.

I tried using one, but it doesn't seem to be invoked at all, parsing the corresponding value as a binary instead.

Issue: UTF8 characters in config files are not supported / handled properly.

Code: The code at https://github.com/bitwalker/conform/blob/master/src/conform_parse.erl#L155 converts utf8 encoded content explicitly to latin1.

Effect: So if we have e.g. a value like "ú..." (U+00FA) in the config file the output will be <<250,...>> which is no valid utf8 content. (the correct output would be <<195, 186,...>>)

Question 1: Why? Is there a problem with leaving content encoded as it is?

Question 2: Can the code please be updated / fixed to support utf8 encoded config files?

I have a config.exs value that is a list. When I go to make a release, I get a failure in conform.effective. I'm not sure if I'm intended to be able to use lists in a config, or how that would end up looking in my ops conf file when generating an exrm release.

If this is intended behaviour, maybe a more useful warning? If not, I'm happy to help put together a PR to fix this.

Invalid input configuration (even before schema checking) will cause errors displayed but exit status is 0 (success).

% ./conform --conf /etc/passwd --schema /dev/null --config /dev/null

Invalid conf file at line 1, column 1:
    root:x:0:0:root:/root:/bin/bash

echo $?
0

#67

As indicated in the pull request, failures occur when there is only a single sublist defined for a nested list config.

In my config.exs, have this config:

use Mix.Config

config :conform_test, test: [
    {:hd_prime,<<
0xC7,0x1C,0xAE,0xB9,0xC6,0xB1,0xC9,0x04,0x8E,0x6C,0x52,0x2F,0x70,0xF1,0x3F,0x73,
0x98,0x0D,0x40,0x23,0x8E,0x3E,0x21,0xC1,0x49,0x34,0xD0,0x37,0x56,0x3D,0x93,0x0F,
0xFA,0x33,0x6F,0x6E,0x0A,0xC9,0x25,0x13,0x95,0x43,0xAE,0xD4,0x4C,0xCE,0x7C,0x37,
0xDC,0x97,0x46,0x51,0x29,0x69,0x32,0x84,0x54,0xF1,0x8F,0xAF,0x8C,0x59,0x5F,0x64,
0x24,0x77,0xFE,0x96,0xBB,0x2A,0x94,0x1D,0x5B,0xCD,0x1D,0x4A,0xC8,0xCC,0x49,0x88,
0x0D,0x81,0x15,0xF6,0x35,0xB1,0x05,0xEE,0x2E,0x4E,0x15,0xD0,0x4B,0x24,0x54,0xBF,
0x48,0x19,0x8A,0x0A,0xA7,0xC1,0x40,0x58,0x22,0x94,0x93,0xD2,0x25,0x30,0xF4,0xDB,
0x07,0x08,0xFA,0x9B,0x37,0x8E,0x3C,0x4F,0x3A,0x90,0x60,0xBE,0xE6,0x7C,0xF9,0xA4,
0x20,0xFD,0x51,0xF6,0x94,0x58,0x70,0x5A,0xC6,0x8C,0xD4,0xFE,0x6B,0x6B,0x13,0xAB,
0x0D,0xBA,0x74,0xD8,0xA8,0x4B,0x2A,0x14,0xB3,0x14,0x4E,0x0E,0xF1,0x28,0x47,0x54
    >>}
]

When i execute the mix conform.new the following errors raised:

➜  conform_test mix conform.new

You already have a schema at config/conform_test.schema.exs.
Do you want to overwrite this schema with a new one?
 [Yn] 

** (FunctionClauseError) no function clause matching in Conform.Utils.Code.to_heredoc/3
    lib/conform/utils/code.ex:213: Conform.Utils.Code.to_heredoc(<<199, 28, 174, 185, 198, 177, 201, 4, 142, 108, 82, 47, 112, 241, 63, 115, 152, 13, 64, 35, 142, 62, 33, 193, 73, 52, 208, 55, 86, 61, 147, 15, 250, 51, 111, 110, 10, 201, 37, 19, 149, 67, 174, 212, 76, 206, 124, 55, 220, 151, ...>>, :open, "\"\"\"\n      ")
    lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
    lib/conform/utils/code.ex:63: Conform.Utils.Code.format_list/2
    lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
    lib/conform/utils/code.ex:62: Conform.Utils.Code.format_list/2
    lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
    lib/conform/utils/code.ex:63: Conform.Utils.Code.format_list/2
    lib/conform/schema.ex:289: Conform.Schema.stringify/2

You can try this test project at https://github.com/developerworks/conform_test

See my test project again, https://github.com/alco/conformist/commit/11df8b2f204578353694ae055d3f1d52fea26e28.

Every time I call mix release, it keeps recompiling conform. The reason could be that conform builds an escript during the release process. I haven't had time to confirm that though.

$ mix release
Building release with MIX_ENV=prod.
==> conform
Compiled src/conf_parse.erl
lib/conform/schema.ex:1: warning: redefining module Conform.Schema
Compiled lib/conform/schema/transform.ex
Compiled lib/conform/sysconfig.ex
Compiled lib/conform/schema/validator.ex
Compiled lib/conform/schema/mapping.ex
Compiled lib/conform.ex
Compiled lib/conform/type.ex
lib/mix/tasks/conform.archive.ex:1: warning: redefining module Mix.Tasks.Conform.Archive
Compiled lib/conform/types/enum.ex
Compiled lib/conform/validators/range_validator.ex
lib/mix/tasks/conform.configure.ex:1: warning: redefining module Mix.Tasks.Conform.Configure
Compiled lib/conform/parse.ex
lib/mix/tasks/conform.effective.ex:1: warning: redefining module Mix.Tasks.Conform.Effective
Compiled lib/mix/tasks/conform.archive.ex
lib/mix/tasks/conform.new.ex:1: warning: redefining module Mix.Tasks.Conform.New
lib/mix/tasks/conform.release.ex:1: warning: redefining module Mix.Tasks.Conform.Release
Compiled lib/mix/tasks/conform.release.ex
Compiled lib/conform/utils/utils.ex
Compiled lib/conform/conf.ex
Compiled lib/mix/tasks/conform.configure.ex
Compiled lib/mix/tasks/conform.new.ex
Compiled lib/conform/utils/code.ex
Compiled lib/mix/tasks/conform.effective.ex
Compiled lib/conform/schema.ex
Compiled lib/conform/translate.ex
Generated conform app
Generated escript conform with MIX_ENV=prod
==> The release for conformist-0.0.1 is ready!
==> You can boot a console running your release with `$ rel/conformist/bin/conformist console`

$ mix release
==> conform
Compiled src/conf_parse.erl
Compiled lib/conform/schema/transform.ex
Compiled lib/conform/sysconfig.ex
Compiled lib/conform/schema/mapping.ex
Compiled lib/conform/schema/validator.ex
Compiled lib/conform.ex
Compiled lib/conform/type.ex
Compiled lib/conform/conf.ex
Compiled lib/conform/validators/range_validator.ex
Compiled lib/conform/types/enum.ex
Compiled lib/conform/parse.ex
Compiled lib/mix/tasks/conform.archive.ex
Compiled lib/mix/tasks/conform.release.ex
Compiled lib/conform/utils/utils.ex
Compiled lib/mix/tasks/conform.configure.ex
Compiled lib/mix/tasks/conform.new.ex
Compiled lib/conform/utils/code.ex
Compiled lib/mix/tasks/conform.effective.ex
Compiled lib/conform/schema.ex
Compiled lib/conform/translate.ex
Generated conform app
Building release with MIX_ENV=prod.
==> conform
Generated escript conform with MIX_ENV=prod
==> The release for conformist-0.0.1 is ready!
==> You can boot a console running your release with `$ rel/conformist/bin/conformist console`

If you clone this project https://github.com/alco/exrmtest and run mix conform.release in it, it'll work the first time. But every subsequent attempt produces the following output:

$ mix conform.release
==> conform
Unchecked dependencies for environment dev:
* neotoma (Hex package)
  the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
* ex_doc (Hex package)
  the dependency is not available, run "mix deps.get"
* earmark (Hex package)
  the dependency is not available, run "mix deps.get"
==> exrmtest
** (Mix) Can't continue due to errors on dependencies

When running it with MIX_ENV=prod, it only complains about neotoma:

$ MIX_ENV=prod mix conform.release
==> conform
Unchecked dependencies for environment prod:
* neotoma (Hex package)
  the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
==> exrmtest
** (Mix) Can't continue due to errors on dependencies

The errors appear to come from the invocation of escript.build but I haven't been able to investigate it further.

Elixir v1.2.3.

# mix release
Building release with MIX_ENV=prod.

You have dependencies (direct/transitive) which are not in :applications!
The following apps should to be added to :applications in mix.exs:

        conform_exrm -> conform -> neotoma => neotoma is missing from conform

Continue anyway? Your release may not work as expected if these dependencies are required! [Yn]:

Versions used:
conform_exrm 0.3.0
exrm 1.0.1
conform 1.0.0-rc8

The API has changed for transforms a while ago, this is rather confusing.

The source code has embedded code samples which are closer to reality. The new API format is rather impractical, here is a code snippet for a transform that takes a value of "format abcde" and turns it into {:format, "abcde"}, useful for myapp.some_crypto_key = base64 YTM0NZomIzI2OTsmIzM0NTueYQ==

  transforms: [
    "myapp.some_crypto_key": fn confpid ->
        # discard the wrappers, get at the value
        [{_,val}] = Conform.Conf.get(confpid, "myapp.some_crypto_key")
        #IO.puts "encry #{inspect val}"
        [k, v] = String.split(to_string(val))
        k = String.to_atom(k)
        {k, v}
      end
    ],

https://github.com/basho/cuttlefish supports generating the vm.args from the app.conf, is something similar planned for conform? Just being able to set the node name and cookie would already solve all my problems.

Hi,

Currently some tasks, like conform.effective validates the environment used to execute it. Is this really necessary? For instance I use more than just three valid environments, and the task requires it to pass this test:

unless env in [:test, :dev, :prod] do
   error "The value provided for --env is not a valid environment"
   exit(:normal)
end

I've already modified the code locally to do what I want, and I've noticed no issues, if this isn't necessary I could create a pull request.

Tested with conform 2.0. If the application's sys.config contains a tuple:

[{cqerl,[{maps,true},
         {cassandra_nodes,[{<<"127.0.0.1">>,<<"9042">>}]},...

and the schema contains:

    "cqerl.cassandra_nodes": [
      datatype: [ list: :ip ],
      default: [ {"127.0.0.1", "9099"} ], # note canary in port value
      to: "cqerl.cassandra_nodes"
     ],

conform will generate an erroneous output sys.config with a list of lists (instead of a list of tuples) holding the data from the input sys.config list of tuples:

{cqerl,[{cassandra_nodes,[[<<"127.0.0.1">>,<<"9042">>]]},

I'm hesitant to consider this a bug outright, but I'll describe an example scenario.

If I create a release with 5 sublists defined for a particular key in prod.exs and need to deploy that release to a different environment where only 3 sublists are necessary (via a different .conf file), the merging process gives me the 3 sublists that I want, but also the last 2 from the initial prod.exs. I can see how a strict merge and a replacement of the list both make sense in various scenarios.

I seem to be fighting an uphill battle with sublists, so maybe I'm approaching the problem incorrectly, but figured I'd open it to discussion.

I have updated my test project to also use the latest exrm, https://github.com/alco/conformist/commit/1a7ee58744b520a23b7defde1c5d2d3993c158fa.

When I run mix release for the first time (or after removing the _build directory), it builds the release. When I try to run it for the second time, I get the following error:

$ mix release
Building release with MIX_ENV=prod.
==> conform
Unchecked dependencies for environment prod:
* exrm (Hex package)
  the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
==> Failed to execute before_release hook for Elixir.ReleaseManager.Plugin.Conform!
==> conformist
** (Mix) Can't continue due to errors on dependencies

On the readme, the schema is listed as having mappings, and translations (as in cuttlefish), but in the generated schema, there's mappings, transforms, and validators. This leads me to think that the readme is quite a bit out of date - is this correct, or am I reading it wrong? Is there a canonical place for up to date documentation?

Hello @bitwalker, I have some issues:

I need to merge schemes from dependency of my application and some of its dependencies in release. I see that I can use extends attribute for this purpose, but in the same way I see that conform uses Mix.Dep.children/0 to collect dependency schemes.

This leads to following problem: What if I want to merge schema from not direct dependency, but for example schema from dependency of a dependency? In this way we will get:

** (CaseClauseError) no case clause matching: []
    lib/mix/tasks/conform.archive.ex:34: anonymous fn/4 in Mix.Tasks.Conform.Archive.run/1
    (elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/mix/tasks/conform.archive.ex:23: Mix.Tasks.Conform.Archive.run/1
    lib/conform_exrm.ex:27: ReleaseManager.Plugin.Conform.before_release/1
    lib/mix/tasks/release.ex:248: anonymous fn/2 in Mix.Tasks.Release.execute_before_hooks/1
    (elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/mix/tasks/release.ex:77: Mix.Tasks.Release.do_run/1
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2

Can I do it with the current conform? If not, what do you think If we will replace Mix.Dep.children from the conform.archive.ex with project_path/deps/extended_dep/config/schema?

I will send PR soon for following discuss.

And also I have second question. I see that some transformers are arirty 1 and some transformers are arity 3 in conform's README.md. For all of my transformers I'm getting:

** (Conform.Schema.SchemaError) Invalid transform for my_mapping, it must be a function of arity 1.

error. How can I use transformers with arity 3?

Thank you.

While

config :logger, :backends, [
  :console,
  {ExSyslog, :exsyslog_error},
  {ExSyslog, :exsyslog_debug}
]

works fine with 1.0.0-pre's mix conform.new, and appears to generate a correct .schema.exs file:

    "logger.backends": [
      commented: false,
      datatype: [
        list: :atom
      ],
      default: [
        :console,
        "ExSyslog": :exsyslog_error,
        "ExSyslog": :exsyslog_debug
      ],
      doc: "Provide documentation for logger.backends here.",
      hidden: false,
      to: "logger.backends"
    ],

, there is a failure when generating a new conf using mix conform.configure:

╰─$ mix conform.configure
** (ArgumentError) argument error
    :erlang.atom_to_binary({:ExSyslog, :exsyslog_error}, :utf8)
    lib/conform/translate.ex:275: Conform.Translate.write_datatype/3
    (elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
    (elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir) lib/enum.ex:1043: Enum.map/2
    lib/conform/translate.ex:290: Conform.Translate.write_datatype/3
    lib/conform/translate.ex:55: anonymous fn/2 in Conform.Translate.to_conf/1
    (elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3

Not sure if this is a dupe of #20, but figured you might want to know since this is probably a very common configuration option for Elixir in production. Thanks for conform and exrm!!! ❤️

On top of #89 I am debugging in issue around https://github.com/bitwalker/conform/blob/master/lib/mix/tasks/conform.archive.ex#L33 . I can only describe it as "the accumulator drops items". My example schema "extends" two other schemas. Those are found, but the accumulator somehow drops them.

Pulling my hair out on this one.

Having problem with version

{:exrm, "> 1.0.0-rc5"},
{:conform, "> 1.0.0-rc4"}

MIX_ENV=prod mix release
Building release with MIX_ENV=prod.
==> conform
Unchecked dependencies for environment prod:

• exrm (Hex package)
  the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
  ==> Failed to execute before_release hook for Elixir.ReleaseManager.Plugin.Conform!
  ==> qrgateway
  ** (Mix) Can't continue due to errors on dependencies

It works ok without conform

For some configuration, the current state of conform doesn't offer enough flexibility. If someone wants a list or map of values, for example:

[[name: "buzz", type: :person, age: 25],
 [name: "fido", type: :dog, age: 5]]

Currently there is no way to define a mapping from a value in the .conf to that format. The proposal for supporting this scenario looks something like the following:

# my_app.conf
my_app.complex_list.buzz.type = person
my_app.complex_list.buzz.age = 25
my_app.complex_list.fido.type = dog
my_app.complex_list.fido.age = 5

# my_app.schema.exs
[mappings: [
    "my_app.complex_list.*": [
    to: "my_app.complex_list",
    datatype: [complex: {"my_app.complex_list.*.type", "my_app.complex_list.*.age"}],
    default: []
  ]
  "my_app.complex_list.*.type" [
    to: "my_app.complex_list",
    datatype: :atom,
    default: :undefined
  ],
  "my_app.complex_list.*.age" [
    to: "my_app.complex_list",
    datatype: :integer,
    default: 0
  ]]
translations: [
  "my_app.complex_list.*": fn
    _, {name, {type, age}}, nil -> [[name: name, type: type, age: age]]
    _, {name, {type, age}}, acc -> [[name: name, type: type, age: age] | acc]
  ]]

Stated more plainly, the tasks to implement this are as follows:

• Automatically initialize accumulators to []
• Add wildcard support to keys.
• Add a complex data type, which takes a tuple of keys to mappings which will populate that position in the tuple passed to the translation (or simply returned if no translation is provided)
• When a complex datatype is encountered, get the mappings for that datatype, and extract the values from the .conf parse result, and create a tuple with those mapped values.
• Skip further processing of the mappings used by the complex type

I'm open to suggestions, improvements, etc. Please provide examples of all three files (.conf, .schema.exs, and sys.config) so that it's clear what the inputs and outputs are.

Why do you use custom stringify here instead of Macro.to_string?

Link to original issue: bitwalker/distillery#92

We need to run conform's pre_start hook pre_upgrade as well now that the hooks for it are in distillery master.

jfyi, this is part of Phoenix app default config:

config :myapp, Myapp.Endpoint,
  live_reload: [
    patterns: [
      ~r{priv/static/.*(js|css|png|jpeg|jpg|gif)$},
      ~r{web/views/.*(ex)$},
      ~r{web/templates/.*(eex)$}
    ]
  ]

and this is actually translated into the following:

    "myapp.Elixir.Myapp.Endpoint.live_reload.patterns": [
      doc: "Provide documentation for myapp.Elixir.Myapp.Endpoint.live_reload.patterns here.",
      to: "myapp.Elixir.Myapp.Endpoint.live_reload.patterns",
      datatype: [
        list: :binary
      ],
      default: [
        %{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 157, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 112, 0, 47, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "priv/static/.*(js|css|png|jpeg|jpg|gif)$"},
        %{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 106, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 119, 0, 120, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "web/views/.*(ex)$"},
        %{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 116, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 119, 0, 120, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "web/templates/.*(eex)$"}
      ]
    ],

For example here is the schema generated for a conf entry that takes a module name:

    "backend": [
      to: "my_app.Elixir.MyApp.backend",
      datatype: :atom,
      default: MyApp.SomeBackend
    ],

In the conf file, you have to prefix the value with Elixir.:

backend = Elixir.MyApp.SomeBackend