yogthos / config

Library for managing environment variables in Clojure using EDN configuration files

yogthos/config

A library for managing configuration using environment variables and EDN configuration files.

The configuration is resolved in the following order, the variables found in later configurations will replace those declared earlier:

1. config.edn on the classpath
2. .lein-env file in the project directory
3. .boot-env file in the project directory
4. EDN file specified using the config environment variable
5. Environment variables
6. Java System properties

The library parses configuration keys into Clojure keywords with names lowercased, then _ and . characters converted to dashes, e.g:

• foo_bar -> foo-bar
• Foo_bar -> foo-bar
• Foo.BAR -> foo-bar

namespaced keys can be declared using __:

• db__spec -> db/spec

The values are parsed using the following strategy:

1. [0-9]+ -> number
2. ^(true|false)$ -> boolean
3. \w+ -> string
4. try parse as EDN, and return the original value as the default

following environment variables:

* BOOL=true
* text="true"
* number=15
* quoted-number="12"
* db__spec="jdbc:sqlite:myapp_dev.db"
* edn_string="{:foo :bar :baz [1 2 \"foo\"]}"
* unparsed.text="some text here"

are translated as:

* :bool          true,
* :text          "true",
* :number        15,
* :quoted-number "12",
  :db/spec       "jdbc:sqlite:myapp_dev.db"
* :edn-string    {:foo :bar, :baz [1 2 "foo"]},
* :unparsed-text "some text here"

Installation

Include the following dependency in your project.clj file:

Usage

external configuration

In most cases you'll likely want to specify the configuration file using the config environment variable. We will add the dependency to our project.clj file and specify the configuration file location using :jvm-opts:

(defproject edn-config-test "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.0"]
                 [yogthos/config <VERSION>]]
  ;; configuration will be read from the dev-config.edn file               
  :jvm-opts ["-Dconfig=dev-config.edn"]               
  :main edn-config-test.core)

embedded configuration

In some cases you may wish to package configuration in the jar along with the application. In this case the config.edn file must be present on the classpath.

Let's take a look at setting up separate configurations for development and production by adding profiles to project.clj. We'll create dev and prod profiles each pointing to a different resource path containing the desired configuration.

First, we'll need to create a config folder in the root of the project. Under the config we will create dev and prod folders. Each of this will contain a file called config.edn.

We'll create a development configuration in config/dev/config.edn:

{:db "jdbc:sqlite:dev.db"}

and a production configuration in config/prod/config.edn:

{:db "jdbc:sqlite:prod.db"}

Next, we will add the dependency and the profiles to our project.clj:

(defproject edn-config-test "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.0"]
                 [yogthos/config <VERSION>]]
  :profiles {:prod {:resource-paths ["config/prod"]}
             :dev  {:resource-paths ["config/dev"]}}
  :main edn-config-test.core)

Accessing the configuration

We can now access the config variables the config.edn found under the resource path specified in the profile. There are two ways of doing this. We can load a version of config defined as config.core/env:

(ns edn-config-test.core
  (:require [config.core :refer [env]])
  (:gen-class))

(defn -main []
  (println (:db env)))

The config in env can be reloaded at runtime by calling the reload-env function.

Alternatively, we can call the config.core/load-env explicitly to manage the state of the config in the app. For example, if we use the mount library, we could write the following:

(ns edn-config-test.core
  (:require [mount.core :refer [defstate]]
            [config.core :refer [load-env]])
  (:gen-class))

(defstate env
  :start (load-env))

  (defn -main []
    (mount.core/start)
    (println (:db env)))    

Packaging for release

The application can be packaged using a specific profile by using the Leiningen with-profile option. For example, if we wanted to package with the prod profile then we'd run the following:

lein with-profile prod uberjar

The resulting jar will contain the config found in config/prod/config.edn in case an embedded configuration was used:

java -jar target/edn-config-test.jar
=> jdbc:sqlite:prod.db

Alternatively, we can create a file called custom-config.edn that looks as follows:

{:db "jdbc:sqlite:prod-custom.db"}

Then we can start the app and pass it the config environment variable pointing to the location of the file:

java -Dconfig="custom-config.edn" -jar target/edn-config-test.jar
=> jdbc:sqlite:prod-custom.db

Attributions

The yogthos/config project is based on the environ library.

License

Distributed under the Eclipse Public License, the same as Clojure.

config's Issues

Why can't I read the edu file.

  :profiles {:uberjar {:aot [vend-license-service.core]}
             :dev {:resource-paths ["config/dev"]}}

This is my project.clj file.
├─config
│ └─dev
My edn file is placed in dev. config.edn

This is my packaging order.

lein with-profile dev uberjar

This is the command I run.
java -Dconfig="config.edn" -jar xxx-standalone.jar

The program runs, but the edn file cannot be read.

Reloading question

When working with config at the repl is there a recommended way to reload the env map?

Always cast value to string?

This is basically a clone of weavejester/environ#36.
The base problem being that environment variables contain string values whereas edn-data (in config files and in lein profiles) supports other data types such as integers.
This causes inconsistent/confusing behavior when reading a value from an environment variable or edn structure.

Update dependencies to mitigate CVE-2021-44228 in log4j

Update to at least

[org.clojure/tools.logging "1.2.1"]

from
https://github.com/clojure/tools.logging

to mitigate CVE-2021-44228 in log4j which is a transitive dependency it seems.

EDIT:

This was written by Alex Miller, the maintainer of tools.logging:

tools.logging doesn't actually have a dependency on log4j (you bring your own), but it does use log4j as a test dependency and this bumps all those deps to the new versions

Different order resolution between the doc and the lib behavior

The documentation and the lib code, hence its behavior, are different. The doc states that:

The configuration is resolved in the following order, the variables found in later configurations will replace those declared earlier:

1. config.edn on the classpath
2. EDN file specified using the config environment variable
3. .lein-env file in the project directory
4. Environment variables
5. Java system properties

The code reads first the .lein-env, then the values are overridden by the config.edn file and not the inverse as stated in the documentation. Personally, I would prefer the behavior defined in the documentation to allow a dev environment to override some stuff in config file and by env vars in prod environment.

(defn load-env
  "Generate a map of environment variables."
  [& configs]
  (let [env-props (merge-maps (read-system-env) (read-system-props))]
    (apply
      merge-maps
      (read-env-file ".lein-env")
      (read-env-file (io/resource ".boot-env"))
      (read-config-file "config.edn")
      (read-env-file (:config env-props))
      env-props
      configs)))

Would you accept a PR to fix the code?

Syntax errors in CONFIG are silently ignored

Thanks for this wonderful library, the usage is very convenient and I'm a happy user for already quite a while :)

I just stumbled over a behavior that was unexpected and I would love to fix it:

When a config file defined by the environment variable CONFIG or the Java System property -Dconfig contains a syntax error, yogthos/config does silently ignore it. I would expect that an exception is thrown or at least a warning is printed.

Having errors being silent ignored causes unnecessary trouble to users. An additional check can be implemented by developers using yogthos/config for each project, but I think it is something that should be provided directly by yogthos/config.

If you approve that change of behavior, I would be happy to provide a pull request?

Specify an S3 URL for config file

Hi, would you be interested in a PR that would allow specification of an S3 URL from which to load config.edn?

Off the top of my head the main issue would be adding an amazonica dependency to this lib, which would be a good thing to avoid. If the functionality sounds like something you'd be interested in I'll have a think to see if there's a neat way to make S3 support opt-in only.

loading external configuration

I used to loading configuration from external file in production environment, like /etc/myapp/config. The benefit is that the config file does not need to be put in VCS, and can be managed by tools like ansible and puppet. But with edn-config, the config file must be bundled in the jar. Is that best practice in clojure/java world?

update clojure to 1.9.0

give file path in :ressource-paths

I end up with the following structure :

config
├── dev
│   └── config.clj
├── prod
│   └── config.clj
...

There is only one file per folder. Why not specify a file path instead of a directory path ? What is the rationale behind it ?

Thanks

lein 2.6.1 breaks parsing .lein-env with aot compile

when uberwar with lein 2.6.1 i received:
WARNING: failed to parse .lein-env No dispatch macro for: =
I think is related to this
technomancy/leiningen#2096

reagent template config exception (NumberFormatException)

running lein do clean, run on a fresh reagent-template template (+shadow-cljs), I get:

Syntax error (NumberFormatException) compiling at (config/core.clj:100:1).
For input string: "112473406068224456955"

Possibly related to parseLong limitations with bigInt?

Management of private configurations

Hi, would you consider the option to add config-private.edn to the list of config resources? The idea is allow to store private configs in a separate file (not to be checked-in in CVS).

Thank you.

Namespaces of keywords are stripped out of configs not bundled into jar

We have a config file that looks something like this:

{:pulsar/url "abc"
 :postgres/url "xyz"}

...which works fine with a config.edn bundled into the jar. However when we do java -Dconfig=foo.edn -jar app.jar the namespaces are stripped out with a warning:

Warning: config key :pulsar/url has been corrected to :url
Warning: config key :postgres/url has been corrected to :url

This is caused by the sanitize-keys function which is called in read-env-file but not read-config-file:

config/src/config/core.clj

Lines 36 to 39 in 051e229

	(defn sanitize-key [k]
	(let [s (keywordize (name k))]
	(if-not (= k s) (println "Warning: config key" k "has been corrected to" s))
	s))

I'm happy to submit a pull request to make sanitize-keys support namespaced keywords but I wanted to check first what the intention of the sanitization was to make sure I took an approach consistent with the original aims of the function. Thanks!