An Ansible plugin to merge all variables in context with a certain suffix (lists or dicts only) and create a new variable that contains the result of this merge. This is an Ansible action plugin, which is basically an Ansible module that runs on the machine running Ansible rather than on the host that Ansible is provisioning.
This plugin is tested with the latest release of each minor version of Ansible >=
2.1
. Earlier releases of some minor versions may not be compatible. This
plugin is not compatible with combinations of older versions of Ansible and
newer versions of Python. The following combinations are tested:
Python | Ansible |
---|---|
2.7 | >= 2.1 |
>= 3.5, < 3.8 | >= 2.5 |
>= 3.8 | >= 2.8 |
-
Pick a name that you want to use to call this plugin in Ansible playbooks. This documentation assumes you're using the name
merge_vars
. -
pip install ansible_merge_vars
-
Create an
action_plugins
directory in the directory in which you run Ansible.By default, Ansible will look for action plugins in an
action_plugins
folder adjacent to the running playbook. For more information on this, or to change the location where ansible looks for action plugins, see the Ansible docs. -
Create a file called
merge_vars.py
(or whatever name you picked) in theaction_plugins
directory, with one line:from ansible_merge_vars import ActionModule
-
Create the
library
directory if it's not created yet:mkdir -p library
-
Create an empty
merge_vars
(or whatever name you picked) file in yourlibrary
directory:touch library/merge_vars
Ansible action plugins are usually paired with modules (which run on the
hosts being provisioned), and Ansible will automatically run an action plugin
when you call a module of the same name in a task. Prior to Ansible 2.4,
if you want to call an action plugin by its name (merge_vars
) in your tasks,
you need an empty file called merge_vars
in the place where ansible checks
for custom modules; by default, this is in a library
directory adjacent to the
running playbook.
The variables that you want to merge must be suffixed with __to_merge
.
They can be defined anywhere in the inventory, or by any other means; as long
as they're in the context for the running play, they'll be merged.
Let's say we've got a group someenvironment
in group_vars
with a file
users.yml
, with these contents:
users__someenvironment_users__to_merge:
user1: bob
user2: henry
and a group somedatacenter
in groups_vars
with a file users.yml
, with these
contents:
users__somedatacenter_users__to_merge:
user3: sally
user4: jane
and we're running a play against hosts that are in both of those groups. Then this task:
name: Merge user vars
merge_vars:
suffix_to_merge: users__to_merge
merged_var_name: merged_users
expected_type: 'dict'
will set a merged_users
var (fact) available to all subsequent tasks that looks like this (if it were to be declared in raw yaml):
merged_users:
user1: bob
user2: henry
user3: sally
user4: jane
Note that the variables get merged in alphabetical order of their names, with values from later dicts replacing values from earlier dicts. So this setup:
users__someenvironment_users__to_merge:
user1: bob
user2: jekyll
users__somedatacenter_users__to_merge:
user2: hyde
user3: sally
name: Merge user vars
merge_vars:
suffix_to_merge: users__to_merge
merged_var_name: merged_users
expected_type: 'dict'
would set a merged_users
var that looks like this (if it were to be declared in raw yaml):
merged_users:
user1: bob
user2: jekyll
user3: sally
With great power comes great responsibility...
Let's say we've got a someenvironment
group with an open_ports.yml
file
that looks like this:
open_ports__someenvironment_open_ports__to_merge:
- 1
- 2
- 3
and a somedatacenter
group with an open_ports.yml
file that looks like this:
open_ports__somedatacenter_open_ports__to_merge:
- 3
- 4
- 5
Then this task:
name: Merge open ports
merge_vars:
suffix_to_merge: open_ports__to_merge
merged_var_name: merged_ports
expected_type: 'list'
will set a merged_ports
fact that looks like this (because the variables are merged in alphabetical order):
merged_ports:
- 3
- 4
- 5
- 1
- 2
Notice that 3
only appears once in the merged result. By default, this
merge_vars
plugin will de-dupe the resulting merged value. If you don't want
to de-dupe the merged value, you have to declare the dedup
argument:
name: Merge open ports
merge_vars:
suffix_to_merge: open_ports__to_merge
merged_var_name: merged_ports
dedup: false
expected_type: 'list'
which will set this fact:
merged_ports:
- 3
- 4
- 5
- 1
- 2
- 3
A note about dedup
:
- It has no effect when the merged vars are dictionaries.
When dealing with complex data structures, you may want to do a deep (recursive) merge.
Suppose you have variables that define lists of users to add and select who should have admin privileges:
users__someenvironment_users__to_merge:
users:
- bob
- henry
admins:
- bob
and
users__somedatacenter_users__to_merge:
users:
- sally
- jane
admins:
- sally
You can request a recursive merge with:
name: Merge user vars
merge_vars:
suffix_to_merge: users__to_merge
merged_var_name: merged_users
expected_type: 'dict'
recursive_dict_merge: True
and get:
merged_users:
users:
- sally
- jane
- bob
- henry
admins:
- sally
- bob
When merging dictionaries and the same key exists in both, the recursive merge checks the type of the value:
- if the entry value is a list, it merges the values as lists (merge_list)
- if the entry value is a dict, it merges the values (recursively) as dicts (merge_dict)
- any other values: just replace (use last)
parameter | required | default | choices | comments |
---|---|---|---|---|
suffix_to_merge | yes | Suffix of variables to merge. Must end with __to_merge . |
||
merged_var_name | yes | Name of the target variable. | ||
expected_type | yes | dict, list | Expected type of the merged variable (one of dict or list) | |
dedup | no | yes | yes / no | Whether to remove duplicates from lists (arrays) after merging. |
recursive_dict_merge | no | no | yes / no | Whether to do deep (recursive) merging of dictionaries, or just merge only at top level and replace values |
Running ansible-playbook with -v
will cause this plugin to output the order in
which the keys are being merged:
PLAY [Example of merging lists] ************************************************
TASK [Merge port vars] *********************************************************
Merging vars in this order: [ u'group1_ports__to_merge', u'group2_ports__to_merge', u'group3_ports__to_merge']
ok: [localhost] => {"ansible_facts": {"merged_ports": [22, 1111, 443, 2222, 80]}, "changed": false}
TASK [debug] *******************************************************************
ok: [localhost] => {
"merged_ports": [
22,
1111,
443,
2222,
80
]
}
PLAY RECAP *********************************************************************
localhost : ok=6 changed=0 unreachable=0 failed=0
There are some example playbooks in the examples
directory that show how the
various features work in the context of an actual Ansible playbook. These
example playbooks are run as part of the test suite for this plugin; if you
would like to run them yourself, please see the Contributing
section for instructions on how to run the test suite.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
There is only one prerequisite to working on this project locally:
- You have the Python versions in the .python-version installed and on your path (probably with pyenv
A development workflow may look like this:
-
Clone this repository
-
Run
make dev-deps
- This will create a virtualenv
venv
in the root of this project and install all of the dependencies needed to build a release and run tests.
- This will create a virtualenv
-
Run
make test-all
-
Updating the
tox.ini
file and running all the tests against all of the combinations of Ansible releases and Python versions takes a lot of time. To run aginst just one combination, you can list all of the combinations available and tell tox to only run the tests for one combination:
$ venv/bin/tox -l
py27-ansible-2.1
py27-ansible-2.2
py27-ansible-2.3
py27-ansible-2.4
py27-ansible-2.5
py27-ansible-2.6
...
py35-ansible-2.5
py35-ansible-2.6
py36-ansible-2.7
py36-ansible-2.8
...
$ venv/bin/tox -e py36-ansible-2.5
...
```
If you have any ideas about things to add or improve, or find any bugs to fix, we're all ears! Just a few guidelines:
-
Please write or update tests (either example-based tests, property-based tests, or both) for any code that you add, change, or remove.
-
Please add an example playbook or update an existing example playbook in the
examples
folder. These example playbooks serve as the integration tests for this plugin. -
Please make sure that
make test-all
exits zero. This runs a code linter, all of the tests, and all of the examples against all supported versions of Python and Ansible. -
If the linting seems too annoying, it probably is! Feel free to do what you need to do in the
.pylintrc
at the root of this repository to maintain sanity. Add it to your PR, and we'll most likely take it.
Happy merging!