henrikjoreteg / getconfig

Config file reader for node.js projects that detects environment based on NODE_ENV.

getconfig - config fetcher for node.js

Managing configs for different environments is kind of a pain.

In short I wanted it to:

• Be simple to understand and use
• Use NODE_ENV environment variable to grab appropriate config
• Let me just go const config = require('getconfig') from anywhere in the app and have it Just Work™
• Allow using different formats (via require hooks)

How to use

1. npm install getconfig
2. Create a config/default.json (or config/default.js, anything you can require() will work) file in the same folder as the main entry point (usually project root)
3. Just require getconfig like so from anywhere in your project:

const Config = require('getconfig');

4. That's it!

Where to put your config and what to call it

Getconfig looks for a config directory in the same folder as the main entry point of your app. Your configuration files should be contained within that directory.

The configuration files attempted by require, in order, are:

• config/default
• config/all
• config/{{NODE_ENV}}
• config/local

Note that we don't list extensions, that's because the files are loaded via node's require mechanism, so anything node can require will work.

In the event that NODE_ENV is not set, getconfig will attempt to load dev, devel, develop, and development in its place.

Environment variables

In a lot of situations it's simpler to pass configuration via environment variables, rather than hardcoding it into a config file.

Fortunately, getconfig can fill those in for you. Just set the value of a key to reference the environment variable and it will be expanded inline. For example:

{
    "envVariable": "$ENV_VAR"
}

Note that this will only work for environment variables whose names are within the character set of A-Z, 0-9, and _ (underscore). This is to prevent collisions with things like complex strings that may start with a $.

Environment variables can be made optional by specifying a second $ in the value's name, such as:

{
    "envVariable": "$$ENV_VAR"
}

When an optional environment variable is unspecified, it is removed from that layer of configuration. This allows you to specify a default in config/default and an optional value in config/dev, and if the optional value in config/dev is unset the value from config/default will be used.

Required environment variables may be used in string interpolation like so:

{
    "envVariable": "some ${ENV_VAR}"
}

However when used in interpolation like the above, no type conversions will be applied.

Additionally, since all environment variables are strings, getconfig can also perform type coercion for you by specifying a ::type suffix. The following types are supported out of the box:

{
    "stringArray": "$STRING_ARRAY_VALUE::array",
    "boolean": "$BOOL_VALUE::boolean",
    "booleanArray": "$BOOLEAN_ARRAY_VALUE::array:boolean",
    "date": "$DATE_VALUE::date",
    "dateArray": "$DATE_ARRAY_VALUE::array:date",
    "number": "$NUMBER_VALUE::number",
    "numberArray": "$NUMBER_ARRAY_VALUE::array:number",
    "object": "$OBJECT_VALUE::object",
    "objectArray": "$OBJECT_ARRAY_VALUE::array:object",
    "regex": "$REGEX_VALUE::regex",
    "regexArray": "$REGEX_ARRAY_VALUE::array:regex"
}

The array type is special in that it accepts an optional argument specified by an additional :type suffix, that allows creating an array of a typed value.

In addition to the built in types, it is possible to add your own types like so:

const Errors = require('getconfig/errors');
const Types = require('getconfig/types');

Types.custom = function (val, arg1, arg2) {
    // do some conversion here
    return val;
    // throw new Errors.ConversionError(); // if something failed
};

const Config = require('getconfig');

You can then use ::custom as a suffix in your config and environment variables will be processed through your custom function. If your custom function accepts arguments (in the example above arg1 and arg2) they can be passed like so $$ENV_VAR::custom:arg:arg. Note that every argument will be passed as a string and it is up to your custom function to handle them appropriately.

Self references

Your configuration can also reference variables within itself as part of string interpolation, so a config file like the following is valid:

{
    "port": "$PORT::number",
    "host": "$HOSTNAME",
    "url": "http://${self.host}:${self.port}"
}

This allows you to define a variable once and reuse it in multiple places. This has the same limitations as string interpolation with environment variables however, it will not apply type conversions, and the value referenced must exist. You can refer to deeper keys by using a dot as a path separator like so:

{
    "some": {
        "deep": {
            "value": "here"
        }
    },
    "top": "${self.some.deep.value}"
}

Note that if the contents of a self referencing key is only the reference (i.e. ${self.value} vs http://${self.value}) the reference will be copied not interpolated. This allows you to reference objects in multiple places like so:

{
    "postgres": {
        "user": "pg",
        "database": "test"
    },
    "clientOne": {
        "database": "${self.postgres}"
    },
    "clientTwo": {
        "database": "${self.postgres}"
    }
}

In the above example, the database key under both clientOne and clientTwo will be a reference to the top level postgres object, simplifying configuration reuse.

Explicitly setting the config location

In certain circumstances, when your app isn't run directly (e.g. test runners) getconfig may not be able to lookup your config file properly. In this case, you can set a GETCONFIG_ROOT environment variable to the directory where your config files are located.

.env files

In addition to the above behaviors, getconfig will attempt to load a .env file located in either the config directory or the project root (i.e. one level above the config directory). These .env files are parsed as close to the dotenv package as possible for compatibility's sake.

Cloud Functions

When used in a Google Cloud Function or an AWS Lambda function, getconfig will use the CODE_LOCATION or LAMBDA_TASK_ROOT environment variable, respectively, to automatically locate the appropriate root. This means that you can include your config directory in your function deployment and things should work as you would expect them to.

Note: Google Cloud Functions automatically set NODE_ENV to "production" when deployed, however Lambda leaves NODE_ENV unset by default.

Environment

getconfig will always fill in the getconfig.env value in your resulting config object with the current environment name so you can programatically determine the environment if you'd like. If no NODE_ENV is set it will also set getconfig.isDev to true.

CLI

getconfig also includes a small helper tool for debugging and inserting values from your config into shell scripts and the like, it can be used like:

getconfig [path] config.value

The path parameter is optional and allows you to define the root of your project, the default is the current working directory. The second parameter is the path within the config to print.

Changelog

• 4.5.0
  • Add support for .env files.
  • Fix bug with multiple variables in interpolation.
• 4.4.0
  • Add the all layer, which loads before the NODE_ENV layer but after default and can help eliminate the need for multiple files with the same contents.
• 4.3.0
  • Include a CLI tool.
• 4.2.0
  • Allow self references to work correctly outside of string interpolation.
• 4.1.0
  • Support string interpolation of environment variables and self references.
• 4.0.0
  • Total rewrite, now supports optional environment variables as well as type coercion and custom type processors.
• 3.1.0
  • Supports Google Cloud Functions and AWS Lambda functions out of the box.
• 3.0.0
  • Does not merge arrays from config layers, instead overwrites them entirely with the topmost config's array.
• 2.0.0
  • Total refactor, now stores config files in a directory and merges them on top of each other for simplicity.
• 1.0.0
  • Bumping major to get out of 0.x.x range per semver conventions.
  • dev enviroments now look for related config files. So if you've set your $NODE_ENV to development and it will still find a file called dev_config.json.
• 0.3.0 - Switching from JSON.parse to ALCE to allow single quotes and comments. Better readme.

License

MIT

if you dig it follow @HenrikJoreteg and/or @quitlahok on twitter.

getconfig's Issues

npm WARN package.json [email protected] No repository field.

Node 0.10 NPM is more strict, it looks like.

Self referencing to an object doesn't work

I don't know if it's meant to but it feels like it should based on the docs. It instead sets the value to the string "[object Object]"

If it were to copy the reference rather than try string interpolation it could support all existing types.

Variables in not working

Using 4.4.1 and having my variables in all.json and default.json just to be sure.. The final result is not parsed correctly for some reason.

"proxyUrl": "http://${self.hapi.host}:${self.hapi.port}"

Results in:
http://localhost:${self.hapi.port}

Is there maybe a bug in the second parameter parser? Also in the docs/readme i think there's an typo.
"url": "http://${self.host}:${self.port}
The string is not closed with a quotation mark "

allow lowercase environment variable names

right now getconfig is somewhat arbitrarily limiting environment variable usage to uppercase strings, this isn't necessary so let's not keep that limitation.

inheriting from a global config

It's nice to have a different configs for dev, prod, etc. But most likely those environment-specific config will share some common stuff.

So it would be nice to still load "config.«env».json" but even better to use it to extend a common "config.json"

And while we are at merging configs, in my company we have :

config.json  <-- global, root config, stored in github
config.development.json  <-- config adjustments for env "development", stored in github
config.development.local.json <-- final adjustments for current developper machine, NOT stored in github

I find it a nice scheme. What do you think ?

be more careful about the "test" environment

it is extremely undesirable to run npm test and inadvertently use the wrong configuration because of forgetting to set NODE_ENV=test

we can use some npm provided environment variables to more forcefully prevent this. particularly npm_lifecycle_event, which will be set to the name of the script that's currently running. we'll likely want the condition that forces the environment to test to be process.env.npm_lifecycle_event.startsWith('test') since scripts like test:client and test:server are common.

Self referencing object values happen too soon

if I have a config that has something like this:

default.json

{
  "foo": {
    "infoo": "one"
  },
  "bar": "${self.foo.infoo}"
}

local.json

{
    "foo": {
      "infoo": "two"
    }
}

The config that's returned is

{ getconfig: { isDev: true }, foo: { infoo: 'two' }, bar: 'one' }

This only happens on objects. if I were to reference self.foo in local.json it would have the value I expected.

Issue with browserify

Whenever I try to browserify code that uses getconfig, it errors out with this, [TypeError: Cannot read property 'range' of null].

String interpolate environment variables

I use this module a lot and I ran into cases several times where I would like to interpolate any environment variables within a string. To make it make it more clearer, a small example;

{
  "url": "http://${HOSTNAME}.com"
}

This would be a great addition to an already great library 👍

Add support for filling in environment variables

For stuff like deploying on Azure or Heroku, all actual sensitive data typically comes in via environment variables.

Would be sweet to be able to be able to do stuff like:

{
  "knex": {
    "client": "postgres",
    "connection": {
       "host": "$DB_CONNECTION_HOST",
       "user": "$DB_CONNECTION_USER",
       "password": "$DB_CONNECTION_PASSWORD"
    }
  }
}

what say ye @nlf ?

1.0.0

Please

Error in React "folder not found"

In my Create React app i want to use getconfig since it works so easy.
Sadly when implementing this i got an error Error: Unable to find a config directory
I've already tried to set the GETCONFIG_ROOT in a .env file. It's so strange...

This is the file i use.

import {FETCH_RECIPES} from "../constants/recipe";
import axios from "axios";
const config = require('getconfig');

export function fetchRecipes (id) {
    return dispatch => {
        axios.get(`${config.API_URL}/recipes/${id}`).then((response) => {
            return dispatch({
                type: FETCH_RECIPES,
                response: response.data
            });
        }).catch((error) => {
            return dispatch({
                type: FETCH_RECIPES,
                response: {
                    error: error.response.data
                }
            });
        });
    };
}

And the error as an image for now.

Wrong config dir using Phusion Passenger

Serving an express app with Passenger as application server
I got the Error: Unable to find a config directory

Passenger uses an helper script to boot the app (helper-scripts/node-loader.js)
so the findConfig function detect the config path starting from require.main.filename
which is a system directory and indeed it cannot store config files.

is it possible to change the lookup part to really try to use process.cwd() if looking on the entire tree of require.main.filename fails?

implement some means of retrieving secret values

there is some desire to allow getconfig to retrieve values from somewhere other than a file on disk or environment variables, however this requires a bit of planning to come up with a reasonable solution since many things in node are by their very nature asynchronous and getconfig is intended to work synchronously.

some possible solutions:

1. make getconfig asynchronous, this would be a new semver-major but would allow us to perform async actions without any additional work. instead of const config = require('getconfig') we could return a promise and have const config = await require('getconfig')

pros:

• minimal usage change

cons:

• node doesn't yet have top level await for commonjs (it does for esm modules) so there's some inconvenience factor
• it would definitely be a breaking change

2. use a built in worker thread and Atomics to run async actions and wait for their results

pros:

• can perform async actions (the desired effect) while appearing to still be synchronous

cons:

• requires built in implementation or working out a plugin system of some kind to allow multiple sources (or see 4 below)
• configuring external sources would require some additional syntax

3. allow a value to be populated by an arbitrary external command

pros:

• would allow users to get a value in a very flexible way while maintaining synchronous loading thanks to spawnSync/execSync

cons:

• configuring a value to be read through an external command would require a reasonable way of expressing the desired command
• the entire command would have to be provided for every value coming from the given source

4. (middle ground between 2 and 3) allow users to provide a javascript module within their project that retrieves a value however they choose, run their module in a worker thread and use atomics to wait for its result

pros:

• very flexible
• maintains overall synchronous nature
• non-breaking

cons:

• users would likely have to provide their own modules for data retrieval
• those modules will have to perform their own configuration somehow (who configures the config fetcher?)

config files naming scheme

Your module looks awesome !

I just have troubles with the naming scheme. In a file explorer or in an IDE, files are sorted by name. It's thus common to name related files with a pattern like that «common» -> «specific», the "common" part making sure that files appears close in a file listing.

In that case, in my current company, it gives :

config.development.json
config.production.json
config.test.json

Isn't it nicer ?

What differentiates from node "config" module?

I have been using the npm module "config" for the purpose you express in the README. Is there some problem with that module that you were trying to solve here? Just trying to understand when i may want to use this. :)

An isProd counterpart

It would be nice to have a isProd counterpart to check if your not in the production environment. Becomes handy for covering both NODE_ENV=development and NODE_ENV=test.

optionally support other configuration file types

JSON's lack of comments has always been a frustration, and the alternative of using javascript isn't really a great one.

there's no good reason we can't allow for different file types through the use of optional peer dependencies. if the dependency is available at the appropriate range, we support that file type. if not, we don't. using optional peer dependencies means that even npm 7 won't install extra dependencies by default, allowing JSON users to not install anything they don't use and any non-JSON users to install only the dependencies required for the formats they do use.

Allow JSON5 config files

It would be great if getConfig could accept JSON5 config files.

default environment

Hello,

using NODE_ENV is an express.js convention https://groups.google.com/forum/#!topic/nodejs/GGL-SGfWKkI

express itself will default to 'development' if not found (http://expressjs.com/api)

So I humbly suggest to change the default env value to 'development', not 'dev'

Azure Web App - Unable to find a config directory

I've deployed my app to azure and now am getting the following error:

Error: Unable to find a config directory
at Object.internals.findConfig (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:44:15)
at Object.internals.findConfig (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:47:22)
at Object.internals.findConfig (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:47:22)
at Object.internals.init (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:100:113)
at Object. (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:138:28)
at Module._compile (module.js:398:26)
at Object.Module._extensions..js (module.js:405:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)

After looking at the source code, I am thinking the problem may lie in here:

root = root || (require.main ? Path.dirname(require.main.filename) : process.cwd());

If I set the GETCONFIG_ROOT env variable to 'wwwroot' I get a different error:

Error: No config files found
at Object.internals.init (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:131:15)
at Object. (D:\home\site\wwwroot\node_modules\getconfig\getconfig.js:138:28)
at Module._compile (module.js:398:26)
at Object.Module._extensions..js (module.js:405:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (D:\home\site\wwwroot\server.js:6:14)
at Module._compile (module.js:398:26)

Any ideas?

Local is overwriting NODE_ENV

I have these env as json files in my config/ dir:

• default
• dev
• local
• production
• staging

Although I have NODE_ENV set to 'staging', the config 'local' seems to be included to & overwrites the values.

v: 3.1.0