miklish / aotearoa

AOtearoa is a command line tool that allows Application Operations teams to generate complex configurations for an application while having little to no knowledge of the application's configuration system beyond the few values they are entrusted to provide (e.g.: production credentials, certificates, private keys, etc...).

AOtearoa 2

New Release!

Aotearoa 2 is now available.

Highlighted feature summary

* metadata.yml is now optional

You are no longer required to define metadata for all your tokens. AO2 will generate default metadata for you (that can be used to provide command-line prompts for values, or for use by tools such as web form generator plugins).

* Inline properties

Properties such as default values, or instructions to encrypt values, can now be specified directly in your templates (rather than needing to be specified in a metadata.yml).

* Improved Maven compatibility

Using the -b (or --buildsafe) option, AO2 can now be executed as part of an application's Maven build process without causing Maven to prematurely exit.

NOTE:

AO2 is not fully backwards-compatible with AO1. However, in the updated documentation below, we point out exactly where AO2 differs from AO1 and—where possible—provide simple fixes that can be made to your existing scripts so you can leverage AO2, while continuing to use your existing templates, metadata, and values files.

Overview

AOtearoa is a command line tool that allows Application Operations teams to generate complex configurations for an application while having little to no knowledge of the application's configuration system beyond the few values they are entrusted to provide (e.g.: production credentials, certificates, private keys, etc...).

Template and Value files

Updated for AO2

AOtearoa starts with template files. A template is a normal text configuration file that contains 'tokens' in which values will be inserted. Some examples are shown below:

security.yml

# JWT signature public certificates. kid and certificate path mappings.
jwt:
  clockSkewInSeconds: 60
  certificate:
    '100': ${oauth_certificate}  
...
# Enable JWT verification flag.
enableVerifyJwt: ${jwt_security|true}
...

datasource.yml

PostgresDataSource:
  DriverClassName: org.postgresql.ds.PGSimpleDataSource
  jdbcUrl: jdbc:postgresql://${database_urls|out:comma-separated}/users?ssl=true&sslfactory=com.biz.SSLSocketFactory
...

secret.yml

...
tableauPassword: ${tableau_password|out:encrypt}
...

Let's take a look at the tokens in these examples.

${oauth_certificate}

This is a straight-forward token. You can specify the replacement value for this in a values file, and (optionally) provide additional instructions to AO2 on how to handle this token in a metadata file (we will describe what values and metadata files are below).

${jwt_security|true}

In this case, we are providing both the token name, and providing inline instructions to AO2 on how to handle this token.

The token name is jwt_security. The text following the | character is a default value. This means if AO2 does not find a value for the token jwt_security in the values file, it will inject the value true.

${database_urls|out:comma-separated}

As with the previous example, we are providing both the token name, and providing inline instructions to AO2 on how to handle the token.

In this case, we are specifying how we want AO2 to transform the value before injecting it into the template. The out: part specifies that this is an output transformation instruction, and the comma-separated portion specifies that we want to convert a list of values into a string of comma separated values before we inject it into the template.

We will go into full detail on all transformation options below.

${tableau_password|out:encrypt}

In this example, the token name is tableau_password, and we have specified that we want to encrypt our value before injecting it into our template.

Some example Values files

Below we can see two example values files. One for use in development, and one for use in production.

dev-values.yml

jwt_security:
  - false
database_urls:  
  - dev-postgres-01.biz.com:9000
  - dev-postgres-02.biz.com:9000
oauth_certificate:
  - oauth-dev.pem
tableau_password:
  - devpassword

prod-values.yml

jwt_security:
  - true
database_urls:  
  - tss-postgres-01.biz.com:7373
  - tss-postgres-02.biz.com:7373
  - tss-postgres-03.biz.com:7390  
oauth_certificate:
  - oauth.pem
tableau_password:
  - password  

As you can see in the examples above, you can use different value files for different environments.

Tokens

Updated for AO2

UPDATE NOTE:
In AO1, the default regex pattern was {{my-token}}. If you have templates using this older default pattern, then you will need to add the --regex "\{\{(.*?)\}\}" option to those scripts.

Tokens are named placeholders that are put in template files for the purpose of being replaced with other values.

In AO2, the simplest tokens have the following form

    ${token_name}

where token_name is the name of the token.

Although, by default, AOtearoa 2 will look for tokens of the form ${token_name}, AO2 can find tokens of any form by allowing you to specify the regex pattern it searches for. For example, to search for tokens of the form {{my-token}}, you can use the regex option on the command line as follows: --regex "\{\{(.*?)\}\}".

Inline Token Properties

New for AO2

Default Values

If AO2 cannot find a value to replace a token with, it will check to see if a default value has been defined. E.g.:

    ${token_name|defaultValue}

or

    ${token_name|"defaultValue"}

Note: You can also specify default values in a metadata file.

Output Transformations

You can also specify a transformation to be performed on a value prior to that value being injected into a template.

There are four transformations available:

• copy
• comma-separated
• encrypt
• decrypt

(Please see Transforms in the Metadata section below, for more details on what each of these transformations does).

You can specify a transformation as follows:

    ${token_name|out:<transformation-name>}

E.g.: To encrypt a value before injecting it, would be:

    ${token_name|out:encrypt}

If no transform is specified, then the copy transform will be used (which simply injects a value verbatim).

Again, transforms can also be specified in a metadata file.

Using multiple inline properties

Multiple properties can be specified in a token. One simply needs to separate them with a semicolon. E.g.:

    ${token_name|out:encrypt;defaultVal}

    ${token_name|"my default"; out:decrypt}

Metadata files

Updated for AO2

UPDATE NOTE 1:
In AO2, a metadata file is no longer required, though it can still be used to provide metadata for tokens. In AO2, if a value exists that has no corresponding metadata, then default metadata information will be created for that value. Inline metadata found in template files can also be used to customize the default-generated metadata.

However, to fully customize the metadata for a token, you may still need to define it in a metadata file (since, not all of the properties available in a metadata file can be specified inline).

UPDATE NOTE 2:
If you choose to use a metadata file (e.g.: in order to specify properties that you cannot specify inline), then note that you no longer need to provide a files property. In fact, AO2 ignores any files values it finds in legacy (AO1) metadata files.

Instead, you must specify the extensions of template files you wish AO2 to automatically check for tokens using the -x option on the command line. E.g.: -x yml -x xml.

In addition to templates and values, you can provide a metadata file that describes the values to be injected.

variables:
  jwt_security:
    min: 1
    max: 1
    output:
    - copy
    type: string
    prompt-text: Enable 'true', or disable 'false' JWT verification security
    defaults:
    - true
    
  database_urls:
    min: 1
    max: inf
    output:
    - comma-separated
    type: string
    prompt-text: Provide a list of one or more urls for the database

  oauth_certificate:
    min: 1
    max: 1
    output:
    - copy
    type: string
    prompt-text: Enter the name of the OAuth (Auth Server) certificate
    defaults:
    - cert.pem

  tableau_password:
    min: 1
    max: 1
    output:
    - encrypt
    type: string
    prompt-text: Tableau password (this will be encrypted)

Transforms

The metadata file allows the input to be transformed. In the current version, 4 types of transforms are supported:

• copy
• comma-separated
• encrypt
• decrypt

The copy transform simply pastes the value in as-is. (This is the default transform for inline properties)

The encrypt transform will encrypt the password using AES encryption. The encryption secret can be set on the command line using the -y or -key option. If an encryption secret is not specified on the command line, then the default Light4j secret will be used. E.g.: The tableau password (set as password above) will appear as CRYPT:iR7QkKyNDJ4wTRn9BEXROgm2iAkajmIv2ZR5VOsxMqs= in secret.yml.

The decrypt transform will decrypt the password using AES encryption. The decryption secret can be set on the command line using the -y or -key option. If a decryption secret is not specified on the command line, the default Light4j secret will be used to decrypt values.

Note: The encryption and decryption key are the same, since AES is a symmetric-key algorithm.

The comma-separated transform concatenates elements in the list and separates them with commas. E.g.: the list

  - value1
  - value2
  - value3

will be converted to value1,value2,value3 prior to being inserted.

Default Token Values

Also note that you can set default values for tokens in the metadata yaml. These will show up in prompts and web forms, and will be used in cases where values are not entered in a values file.

Default Metadata

If there is no metadata entry for a particular value or token, the AO2 will create one. E.g.: The default metadata for a token called MY_TOKEN would be:

  MY_TOKEN:
    min: 1
    max: 1
    output:
    - copy
    type: string
    prompt-text: Please enter the value for MY_TOKEN

Order of Usage for Metadata

AO2 will

1. Use metadata for a token from a metadata file, if it exists.
2. Otherwise, it will use metadata defined inline for a token, if it exists.
3. Otherwise, it will generate default metadata.

Values

There are numerous ways in which values can be specified to the template resolver.

Values file

Below see our example production value file.

jwt_security:
  - true
database_urls:  
  - tss-postgres-01.biz.com:7373
  - tss-postgres-02.biz.com:7373
  - tss-postgres-03.biz.com:7390
oauth_certificate:
  - oauth.pem
tableau_password:
  - password

All values must be entered as elements of a yaml list (even if there is only a single entry).

If a variable is specified in the metadata yaml but has no value in the values file, then this will cause an error unless either

1. a default value is specified in the metadata yaml, or
2. you specify --prompts on the command line, to instruct AO2 to request a value from the user on the command line

(Note that, specifying a default value inline will not prevent an error in this case, since the metadata defined in a metada file takes precedence over metadata defined inline).

Default Values from Metadata

The value injected into a token can come from the default value specified in a metadata file, or as an inline token property.

Command line prompts

If command line prompts are chosen as a method to input values (using the --prompts option), then on the command line, the user would see the following:

Type \\q to quit. Enter a blank line to use default (if default exists)
[Enable 'true', or disable 'false' JWT verification security : (true)]: ⏎
 >> value selected is 'true'

Next entry requires between 1 and inf values. Enter \\n to complete.
[Provide a list of one or more urls for the database]: 
[Value 1]: tss-postgres-01.biz.com:7373
[Value 2]: tss-postgres-02.biz.com:7373
[Value 3]: tss-postgres-03.biz.com:7390
[Value 4]: \\n

[Enter the name of the OAuth (Auth Server) certificate]: oauth.pem
[Tableau password (this will be encrypted)]: password
Resolving templates...done.

As you can see, the prompts that are shown are specified in the metadata file (or by auto-generated metadata), and the defaults that are shown are also specified in the metadata file.

Environment variables

You may also use environment variables as a way to specify variable values. E.g.: prior to running a test, you may wish to temporarily turn off security in your configurations, even though your values file has jwt_security set to 'true':

    $ export jwt_security=false

To enable values to be read from environment variables, you use the --env option on the command line.

File values with environment variable overrides

You can also use a value file and environment variables, where environment variables can be used to override value file settings to enable you to do testing (e.g.: override security to a 'false' setting, or turn the logging level to 'debug' temporarily) without modifying your production values file.

This will prevent you from having to set environment variables for all your settings, and only specify overrides as environment variables.

Local web form

Not yet implemented.

Log Level Output

You can select the verbosity of output that AO2 generates. There are 5 levels that exist: TRACE, DEBUG, WARN, INFO, and QUIET. DEBUG messages are low level operational messages, whereas INFO messages are higher level status messages.

On the most verbose extreme, a log level of TRACE will output messages of all levels. On the other end, a log level of QUIET will supress all output except for the ERROR messages that are displayed if AOtearoa 2 is forced to exit.

The table below shows what types of message are outputted for each log level.

Log Level	Trace	Debug	Warn	Info
Trace (t)	Y	Y	Y	Y
Debug (d)	N	Y	Y	Y
Warn (w)	N	N	Y	Y
Info (i)	N	N	N	Y
Quiet (q)	N	N	N	N

The command line switch for log levels is -l (or equivalently --loglevel), and the switch values are listed below:

Log Level	Value
Trace	t
Debug	d
Warn	w
Info	i
Quiet	q (or no value)

The default log level is DEBUG (this is used when no switch is used). QUIET can be specified by either -l q, --loglevel q, or by simply -l (with no value) or --loglevel (with no value).

Command line arguments

To run AOtearoa, you would run

    $ java -jar ao.jar [arguments...]

with the appropriate arguments:

Command	Description
-h, --help	Display help/usage information
-m, --metadata	Variable metadata file (optional)
-o, --outputdir	Output directory (required)
-v, --values	Value file (optional)
-p, --prompts	Use command line prompts to enter values (optional)
-e, --env	Get values from environment variables (optional)
-s, --server	Config Server URL (optional)
-r, --regex	regex pattern (optional - default is '\{\{(.*?)\}\}' )
-t, --templates	Template file folder (required)
-x, --extension	Template file extensions (required)
-k, --kmetadata	Keystore metadata template (optional)
-y, --key	Symmetric encryption/decryption secret (optional)
-b, --buildsafe	Allows AO2 to run as part of a Maven build process (optional)
-l, --loglevel	Sets the log level (optional)

Some examples are below:

Use value file with log level set to QUIET

$ java -jar ao.jar       \
    -m metadata.yml      \ # metadata file
    -t templates/        \ # template files directory
    -x xml -x yml        \ # injects into xml and yml files
    -o config-out/       \ # output folder
    -v values.yml        \ # values file
    -l

Use value file and set the symmetric encryption/decryption secret (no metadata.yml)

$ java -jar ao.jar       \
    -t    templates/     \ # template files directory
    -x    xml            \ # injects into xml files
    -x    yml            \ # injects into yml files
    -o    config-out/    \ # output folder
    -v    values.yml     \ # values file
    --key the_secret       # the AES encryption/decryption secret

Use prompts

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -x xml -x yml        \ # injects into xml and yml files
    -m metadata.yml      \ # metadata file
    -o config-out/       \ # output folder
    -p                     # values from prompts

Use environment variables with log level set to WARN

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -x sh -x yml         \ # injects into bash and yml files
    -m metadata.yml      \ # metadata file
    -o config-out/       \ # output folder
    -e                   \ # values from environment variables
    -loglevel w            # log level set to WARN

Use environment variables with log level set to TRACE

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -x sh -x yml         \ # injects into bash and yml files
    -m metadata.yml      \ # metadata file
    -o config-out/       \ # output folder
    -e                   \ # values from environment variables
    -loglevel t            # log level set to TRACE

Value file with environment variable overrides (no metadata.yml)

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -x xml -x yml        \ # injects into xml and yml files
    -o config-out/       \ # output folder    
    -v values.yml        \ # value file
    -e                     # with environment variable overrides

Maven safe mode enabled, and symmetric encryption/decryption secret (no metadata.yml)

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -x xml -x yml        \ # injects into xml and yml files
    -v values.yml        \ # value file
    -o config-out/       \ # output folder        
    --key the_secret     \ # the AES encryption/decryption secret
    -b                     # Maven safe-mode enabled

Keystores

AOtearoa can also simplify the creation of JKS keystores.

Example Keystore YAML:

certificates:
  tableau-webserver-cert:                                   # certificate-reference name
    format: pem                                             # certificate format
    prompt-text: Tableau webserver public certificate       # prompt text
    filename: tableaucert.pem                               # filename of certificate

  postgres-database-cert:                                   # certificate-reference name
    format: pem                                             # certificate format
    prompt-text: Postgres self-signed certificate           # prompt text
    filename: postgrescert.pem                              # filename of certificate

keystores:
  server.keystore:                                          # this will be the output keystore filename
    base-keystore-filename: idkeystore.jks                  # base keystore filename 
    keystore-password: ${KEYSTORE_PASS|out:encrypt}         # keystore password
    certificates:                                           # list of certificates to load into keystore

  client.truststore:                                        # this will be the output keystore filename
    keystore-password: ${TRUSTSTORE_PASS|out:encrypt}       # keystore password
    certificates:                                           # list of certificates to load into keystore
      tableau-webserver-cert: tableau-uat                   # certificate-reference : alias
      postgres-database-cert: postgres-uat                  # certificate-reference : alias

The YAML required to build keystores contains two main sections: a certificates section and a keystores section.

Certificates section

The main certificates section describes the certificates to be loaded into the keystores that are defined in the main keystores section. For each certificate, you will need to provide a cert reference name in order to reference the certificate in the keystores section. Beyond that, you also need to provide the cert's format (only pem is currently supported), prompt text, and the cert's file name. The certificate must reside in the template folder.

Keystores section

File name

For each keystore, the first item you specify is the keystore file name. This will be the file name of the keystore in output folder.

base-keystore-filename (optional)

base-keystore-filename is an optional value that can be set if you wish to load certs into an existing keystore rather than create a new keystore from scratch. The existing keystore must reside in the template folder.

keystore-password

This is the password to the keystore. If you are creating a new keystore, then this will be the password assigned to the keystore that is created. However, if you have specified that you want to load certs into an existing keystore (by using the base-keystore-filename option), then this password must be the password of the existing keystore.

certificates section

This subsection contains the (zero or more) certs you want loaded into the keystore. If you do not enter any certs in this section, AOtearoa will create an empty keystore for you.

For each cert that you do include, each line must contain the reference name of a certificate that has been defined in the main certificates section of the yaml, and the alias you want assigned to the certificate within the keystore. The certificate reference name and its alias are separated by a ':' character.

tokens are allowed

In the example shown above, you'll notice the existence of tokens throughout. Since the keystore yaml file is processed after values are read (either from user prompts, from a value file, web form, config server etc...), these values can be included in the keystore yaml. These values will be injected into the keystore yaml prior to processing of the keystore yaml file. Tokens can appear anywhere in the keystore yaml and are totally optional.

Example command line

Build configuration and keystores from templates and value file (no metadata)

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -k keystore.yml      \ # keystore template file
    -v values.yml        \ # values file
    -o config-out/         # output folder

How Keystore files are processed

During processing, AO2 will inject values into the Keystore file (assuming it contains tokens), it will then copy the Keystore file into the template folder.This means that the template folder cannot contain a file with the same (case insensitive) name as the keystore file. AO2 will generate an error if it sees a file name conflict.

Maven compatibility

New for AO2

Using the -b (or --buildsafe) option, AO2 can be executed as part of an application's Maven build process without causing Maven to prematurely exit.

Combining Files

While in the examples provided, we separate the metadata and keystore files, it is actually possible to combine these files into one. For example, if you were to combine metadata and keystore configuration into a single file called _combined.yml, then you could use the following command line.

$ java -jar ao.jar       \
    -t templates/        \ # template files directory
    -v values.yml        \ # values file
    -o config-out/       \ # output folder
    -m combined.yml      \ # metadata file
    -k combined.yml        # keystore template file

Note: Value files cannot be combined into a single file with metadata and keystore configuration.

aotearoa's Issues

Keystore metadata file is always assumed to be in template folder

Users are supposed to be able to specify the location of the keystore metadata file, however currently, aotearoa assumes that it is in the template folder regardless of what the user specifies.