ansible-collections / community.hashi_vault

Ansible collection for managing and working with HashiCorp Vault.

community.hashi_vault Collection

Collection Documentation

Browsing the latest collection documentation will show docs for the latest version released in the Ansible package not the latest version of the collection released on Galaxy.

Browsing the devel collection documentation shows docs for the latest version released on Galaxy.

We also separately publish latest commit collection documentation which shows docs for the latest commit in the main branch.

If you use the Ansible package and don't update collections independently, use latest, if you install or update this collection directly from Galaxy, use devel. If you are looking to contribute, use latest commit.

Tested with Ansible

Please refer to the ansible-core support matrix to see which versions of ansible-core are still supported or end-of-life.

Generally, we release a new major version of this collection a little before the release of a new ansible-core version, which is around every 6 months. In that release, we will update the CI matrix to drop the core versions that are about to go EoL, and add in new core versions if they have not been added already.

We also regularly test against the devel branch (latest development commit).

See the CI configuration for the most accurate testing information.

Tested with Vault

We currently test against the latest patch version within the latest two minor versions of the latest major version of Vault. Put another way, we test against version Z.{Z|Y}.Z. For example as of this writing, Vault is on major version 1, with the latest two minors being 8 and 7. So we'll test Vault 1.8.Z and 1.7.Z where Z is the latest patch within those versions.

We do not test against any versions of Vault with major version 0 or against pre-release/release candidate (RC) versions.

If/when a new major version of Vault is released, we'll revisit which and how many versions to test against.

The decision of which version(s) of Vault to test against is still somewhat in flux, as we try to balance wide testing with CI execution time and resources.

See the CI configuration for the most accurate testing information.

Python Requirements

Python 2.6, 2.7, and 3.5 are not supported in version 2.0.0 or later of the collection.

Currently we support and test against Python versions:

• 3.6
• 3.7
• 3.8
• 3.9
• 3.10
• 3.11
• 3.12

Note that for controller-side plugins, only the Python versions supported by the Ansible controller are supported (for example, you cannot use Python 3.7 with Ansible core 2.12).

External requirements

The hvac Python library is required for this collection. For full requirements and details, see the collection's User Guide.

Included content

See the list of included content in the docsite.

Using this collection

See Ansible Using collections for more details.

Contributing to this collection

See the contributor guide in the devel collection documentation.

Releasing this collection (for maintainers)

Follow the instructions for releasing small collections in the Ansible community wiki.

Once the new collection is published and the Zuul job is finished, add a release in GitHub by manually running the GitHub Release workflow. You'll need to enter the version number, which should exactly match the tag used to release the collection.

Release notes

See the rendered changelog or the raw generated changelog.

FAQ

Q: Why not have a single collection of HashiCorp products instead of one just for Vault?

A: This was considered when the hashi_vault plugin was first moved from community.general to this collection. There are several reasons behind this:

• The other known HashiCorp content at that time (covering Consul, Nomad, Terraform, etc.) does not share implementation or testing with Vault content.
• The maintainers are also different. This being a community supported collection means separate maintainers are more likely to focus on goals that make sense for their particular plugins and user base.
• The HashiCorp products serve different goals, and even when used together, they have their own APIs and interfaces that don't really have anything in common from the point of view of the Ansible codebase as a consumer.
• It would complicate testing. One of the primary goals of moving to a new collection was the ability to increase the scope of Vault-focused testing without having to balance the impact to unrelated components.
• It makes for a smaller package for consumers, that can hopefully release more quickly.

Q: Why is the collection named community.hashi_vault instead of community.vault or community.hashicorp_vault or hashicorp.vault or any number of other names?

A: This too was considered during formation. In the end, hashi_vault is a compromise of various concerns.

• hashicorp.vault looks great, but implies the collection is supported by HashiCorp (which it is not). That doesn't follow the convention of denoting community supported namespaces with community.
• community.vault looks great at first, but "Vault" is a very general and overloaded term, and in Ansible the first "Vault" one thinks of is Ansible Vault. So in the naming, and even in the future of this collection and its content, we have to be mindful of avoiding and removing ambiguities between these products (and other Vaults out there).
• community.hashicorp_vault is descriptive and unambiguous but is unfortunately quite long.
• community.hashicorp would be good for a collection that aims to contain community-supported content related to all HashiCorp products, but this collection is only focused on Vault (see above question).
• community.hashicorp.vault (or any other 3-component name): not supported (also long).
• community.hashi_vault isn't perfect, but has an established convention in the existing plugin name and isn't as long as hashicorp_vault.

Roadmap

More information

• Ansible Collection overview
• Ansible User guide
• Ansible Developer guide
• Ansible Collections Checklist
• Ansible Community code of conduct
• The Bullhorn (the Ansible Contributor newsletter)
• Changes impacting Contributors

Licensing

GNU General Public License v3.0 or later.

See LICENSE to see the full text.

Parts of the collection are licensed under the BSD-2-Clause license.

community.hashi_vault's Issues

hashi_vault does not use VAULT_ADDR environment variable anymore

From @transferkraM on Feb 23, 2021 15:23

SUMMARY

Before 3.0, hashi_vault functions properly with VAULT_ADDR and VAULT_TOKEN environment variables. Starting with 3.0 ANSIBLE_HASHI_VAULT_ADDR is required.

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault lookup plugin.

ANSIBLE VERSION

NOTE: Various irrelevant path information is abbreviated as <>.

ansible 2.10.5
  config file = <>/ansible.cfg
  configured module search path = ['<>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = <>/.local/lib/python/versions/ansible/lib/python3.9/site-packages/ansible
  executable location = <>/.local/lib/python/versions/ansible/bin/ansible
  python version = 3.9.1 (default, Feb  6 2021, 06:49:13) [GCC 10.2.0]

CONFIGURATION

ANSIBLE_COW_SELECTION(<>/ansible.cfg) = random
ANSIBLE_NOCOWS(env: ANSIBLE_NOCOWS) = True
ANSIBLE_PIPELINING(<>/ansible.cfg) = True
ANSIBLE_SSH_ARGS(<>/ansible.cfg) = -F ssh_config -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/%p:%r@%h
DEFAULT_GATHERING(<>/ansible.cfg) = smart
DEFAULT_LOAD_CALLBACK_PLUGINS(<>/ansible.cfg) = True
DEFAULT_ROLES_PATH(<>/ansible.cfg) = ['<>/roles']
DEFAULT_VAULT_PASSWORD_FILE(<>/ansible.cfg) = <>/vault
DISPLAY_SKIPPED_HOSTS(<>/ansible.cfg) = False
HOST_KEY_CHECKING(<>/ansible.cfg) = False
INTERPRETER_PYTHON(<>/ansible.cfg) = auto_legacy_silent
RETRY_FILES_ENABLED(<>/ansible.cfg) = False

OS / ENVIRONMENT

pyenv virtualenvs|ag ansible

ansible (created from /usr)

pyenv activate ansible
python --version

Python 3.9.1

STEPS TO REPRODUCE

export VAULT_ADDR=http://127.0.0.42:12345
sudo tcpdump -i lo port 12345
ansible ...

export ANSIBLE_HASHI_VAULT_ADDR=$VAULT_ADDR   # e.g. http://127.0.0.42:12345 (see above)
nc -l -p 12345 -s 127.0.0.42:12345
sudo tcpdump -i lo port 12345
ansible ...

some_var: '{{ lookup("community.hashi_vault.hashi_vault", "secret=foo/data/bar")

EXPECTED RESULTS

• VAULT_ADDR is respected (again) and wins over all other possible configuration values!

ACTUAL RESULTS

• VAULT_ADDR is not evaluated/used

Copied from original issue: ansible/ansible#73698

[env] Support reading a ca certificate path from VAULT_CACERT

SUMMARY

Make ca_cert default to env:VAULT_CACERT

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault lookup plugin

ADDITIONAL INFORMATION

Presently, when using hashi_vault with a non-public CA, the user must specify the CA certificate as part of the lookup parameters.
It would be nice if hashi_vault matched the vault client's behavior, and used a CA from VAULT_CACERT if available.

Currently, I am using the following workaround

#inventory.yaml
all:
  vars:
    some_secret_thing: "{{ lookup('hashi_vault', 'secret=secret/data/somesecret:somevalue', ca_cert=lookup('env','VAULT_CACERT'))}}"

aws_auth - `aws_security_token` parameter is not used

SUMMARY

The aws_security_token option, when supplied, is not passed into the aws_iam_login call.

If using a boto profile or other session credentials, the session token from those will be used, but direct parameter binding values are never accessed.

Related: #118

ISSUE TYPE

• Bug Report

COMPONENT NAME

_auth_aws_iam_login

ANSIBLE VERSION

N/A

COLLECTION VERSION

1.3.2

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

Use temporary AWS creds like those from role assumption which require use of the session token, then pass the token to the auth method.

EXPECTED RESULTS

Success!

ACTUAL RESULTS

An error describing an invalid session token.

Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.hashi_vault'. Error was a <class 'hvac.exceptions.InvalidRequest'>, original message: error making upstream request: received error code 403 from STS: <ErrorResponse xmlns=\"https://sts.amazonaws.com/doc/2011-06-15/\">\n  <Error>\n    <Type>Sender</Type>\n    <Code>InvalidClientTokenId</Code>\n    <Message>The security token included in the request is invalid.</Message>\n  </Error>\n  <RequestId>23453748-f61b-4b59-8e72-13ce2c90fcf4</RequestId>\n</ErrorResponse>\n, on post https://vault/v1/auth/aws/login"}

Parameter 'mount_point' does not work with JWT auth

SUMMARY

The parameter mount_point does not work with JWT auth. It seems like the parameter must be renamed/aliased to path before invoking hvac.jwt_login(). Ref. hvac.jwt_login() documentation. I think this can be fixed quite easily with the same approach as used for aliasing the Vault role parameter:

community.hashi_vault/plugins/lookup/hashi_vault.py

Line 472 in a4374fd

params['role'] = params.pop('role_id')

But adding tests for this seems like a considerable job....

ISSUE TYPE

• Bug Report

COMPONENT NAME

lookup/hashi_vault.py

ANSIBLE VERSION

ansible 2.10.4
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.9.1 (default, Dec 12 2020, 13:15:12) [GCC 8.3.0]

CONFIGURATION

ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
DEFAULT_LOAD_CALLBACK_PLUGINS(env: ANSIBLE_LOAD_CALLBACK_PLUGINS) = True
DEFAULT_STDOUT_CALLBACK(env: ANSIBLE_STDOUT_CALLBACK) = yaml
INTERPRETER_PYTHON(env: ANSIBLE_PYTHON_INTERPRETER) = /usr/local/bin/python

OS / ENVIRONMENT

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

STEPS TO REPRODUCE

- name: Authenticate with a JWT using custom mountpoint/path
  ansible.builtin.debug:
      msg: "{{ lookup('community.general.hashi_vault', 'secret/hello:value', auth_method='jwt', role_id='myroleid', jwt='myjwt', url='https://myvault:8200', mount_point='mypath')}}"

EXPECTED RESULTS

hashi_vault should authenticate successfully to Vault using the custom mountpoint/path and debug the secret.

ACTUAL RESULTS

Error was a <class ''TypeError''>, original message: jwt_login() got an unexpected keyword argument ''mount_point'''

AWX and ANSIBLE_ prefixed environment variables - support a 2nd set of env vars? ansible vars?

SUMMARY

Nothing against your new environment variable naming scheme (#10 (comment)), but would you consider supporting a second set of environment variables which does not use the ANSIBLE_ prefix but maybe something like ANS_ or similar?

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashivault lookup plugin

ADDITIONAL INFORMATION

The problem we have with the ANSIBLE_ prefix is that AWX does not allow user credentials to induce environment variables starting with this prefix.

When creating a new credential type, you are responsible for avoiding collisions in the extra_vars, env, and file namespaces. Also, avoid environment variable or extra variable names that start with ANSIBLE_ because they are reserved.

SOURCE

[internals] hashi_vault - plugin should always retrieve options with get_option()

SUMMARY

As part of the work done in ansible-collections/community.general#23 toward separating vault-related functionality from plugin-specific functionality, a new class was introduced to contain the vault-specific stuff.

The options from the plugin class were passed wholesale into the new class, to be used directly as a dictionary.

Although the plugin class is using self.set_options(), which seems sufficient to do things like process env: and ini: blocks, it may not always provide all of the functionality that that self.get_option() does. For example, one of the proposed fixes for ansible/ansible#73051 involves moving the deprecation handling specifically into get_option().

Fixing the issue wholesale would be a bit involved, as the most direct fix either is to pass the entire LookupPlugin object into the HashiVault class, and have HashiVault call get_option() all the time, or to increase the scope of the validate_ methods to ensure they enumerate all optional options.

But ultimately it's a design flaw; the purpose of the HashiVault class was supposed to be toward moving functionality into a module_util, and having to pass the plugin object into there seems like an anti-pattern (the plugin should process relevant options and pass them into methods, but as written the plugin is only partially aware of which options are relevant for a given auth method), so I don't want to dedicate significant effort toward that type of temporary fix.

Rather, I'll approach this with a more manual two-step approach:

1. Immediate workaround: deprecation is the only possible place where this has an impact in the short term. As we add deprecations, we will add throwaway calls to self.get_option() in the appropriate validate_ method for parameters that aren't already accessed (optional ones mainly), and to the lookup class's run method itself for params that aren't specific to an auth method (like url) for any option which has a deprecation, to ensure the message gets processed. Everything else stays the same.

2. As efforts toward making a module_util continue, the interface between plugin and util must and will ensure that this pattern doesn't propagate into the future, by carefully considering where responsibility lies. The interface problem will then be resolved naturally.

If other issues arise from the lack of universal get_option() calls before the module_util or other work can refactor to fix the core issue, then those will be dealt with at that time.

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault.py

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Feature Request: Add support for azure auth method

We deploy to both AWS and Azure, and are adding HashiCorp Vault to provide a cloud-agnostic secrets management solution. I was just testing out this collection on AWS and the aws_iam_login auth method works great. Vault supports a similar azure auth method for Azure that uses the instances Managed Service Identity (MSI).

It would be great to have the azure auth method supported by this collection,

hashi_vault - [env] Update VAULT_TOKEN handling

SUMMARY

See #10

VAULT_TOKEN is a Vault CLI env var.

The proposal then is to add an Ansible-specific env var and lower the precedence of the existing one:

• Add ANSIBLE_HASHI_VAULT_TOKEN in env: in spec
• Process VAULT_TOKEN as a fallback if the option is not set directly or via spec-defined env
• 🎗this parameter already has another fallback action, which is to try to infer its value via the token_file and token_path parameters. This will remain, and it will still have lower precedence than the VAULT_TOKEN env var. This preserves the existing relative relationship.

ℹThis is not a breaking change

• existing ways of specifying the value keep their relative precedence, and the new env var didn't exist previously

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

Support "caching" token after initial login

SUMMARY

To avoid multiple logins, it'd be great if the lookup module supported a token cache after initial login

ISSUE TYPE

• Feature Idea

COMPONENT NAME

lookup

ADDITIONAL INFORMATION

Currently, it seems for multiple lookups, each lookup does log in + reads from vault.

Would you be open to supporting a feature that adds a token cache?

Here's an example of a now deprecated module that had this feature: https://github.com/jhaals/ansible-vault/blob/17fdb6fde3cb441bc7737770591b5ab5c79a3fe3/vault.py#L100-L108

userpass auth method needs tests

SUMMARY

Related: #66

It seems like the userpass auth method may never have been working, at least not with the version that's in the newer auth class (I'm unsure on the deprecated method in hvac).

And there are no tests for userpass so we don't have any way to catch regressions or problems.

The userpass method is easily testable since it has no external dependencies. Putting this issue here to track the need.

This will probably get done as part of the refactor project, when the auth method moves centrally.

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ldap auth method needs tests

SUMMARY

units can be added for some basic level of tests, though there's not much to this auth method from our end.

For integration we could look into setting up an LDAP server, but I may instead try use an MMock solution to test our end of it.

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

tests - test `with_` functionality in lookup plugins

SUMMARY

In ansible-collections/community.general#23, use of the lookup plugin via with_hashi_vault was fixed.

There should be tests to ensure this continues working, for the current and any future lookup plugins.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

lookup/hashi_vault.py: IAM auth does not support the `mount_point` parameter

SUMMARY

AWS IAM authentication does not take the mount_point parameter, but other auth methods do.

ISSUE TYPE

• Bug Report

A previous report of a similar issue (#767) was classified as a Bug.

COMPONENT NAME

lookup/hashi_vault.py

ANSIBLE VERSION

ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Jul 18 2020, 19:35:03) [GCC 9.2.0]

CONFIGURATION

No changes

OS / ENVIRONMENT

cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.6
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

STEPS TO REPRODUCE

1. Enable the AWS auth method in HashiCorp Vault with a custom mountpoint, e.g., obfuscated-iam.
2. Use the hash_vault lookup plugin with the options auth_method=aws_iam_login mount_point=obfuscated-iam

EXPECTED RESULTS

hashi_vault should authenticate successfully to Vault using the custom mountpoint.

The issue resides, or at least is fixable, in the validate_auth_aws_iam_login function, here. A naive fix is to add mount_point to the params list as below:

    def validate_auth_aws_iam_login(self, auth_method):
        params = {
            'access_key': self.get_option('aws_access_key'),
            'secret_key': self.get_option('aws_secret_key'),
            'mount_point': self.get_option('mount_point')   # <--------- a naive fix
        }
        ...

ACTUAL RESULTS

hashi_vault attempts to authenticate using the default aws mountpoint instead of the requested obfuscated-iam, i.e., /v1/auth/aws/login instead of /v1/auth/obfuscated-iam/login.

fatal: [example.com]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'hvac.exceptions.InvalidRequest'>, original message: missing client token, on post https://example.net/v1/auth/aws/login"}

Dropping Python 2 & 3.5 Support

UPDATE: Python 3.5 support will also be dropped

As clarified in the hvac thread, support for Python 3.5 will also be dropped along with 2.x, and for the same reasons below, this collection won't be able to easily support 3.5.

Python 3.6 will be the minimum version for community.hashi_vault starting in version 2.0.0, keeping parity with hvac.

---

Python 2 Support Will be Dropped in v2

As of the next major version, v2 (v2.0.0+) release of community.hashi_vault, Python 2 will no longer be directly supported or tested against. At that time the minimum Python version will likely be Python 3.5, or 3.6 (tbd).

Why?

We all know Py2 is EoL and projects are dropping support for it left and right, including Ansible (on the controller, where currently all content in this collection runs). This makes it harder and harder to keep supporting it, to work around CI and testing issues too. But the major driver for this decision is around the hvac library.

All of the content in this collection currently relies on hvac and is expected to indefinitely.

hvac still supports Python2, for now, however:

• Dropping of Py2 support is planned: hvac/hvac#582
• hvac's own CI hasn't been testing against Python 2 since Feb 1: hvac/hvac#582 (comment)
• And that's led to two recent releases that accidentally broke Python 2 support, 0.10.12 and 0.10.13 (see hvac/hvac#706 and hvac/hvac#711)
• And those issues have in turn renewed discussion of explicitly dropping support in the library rather imminently: hvac/hvac#706 (comment)

Even if hvac does not explicitly drop support, it's clear the lack of testing against it is a problem, and given the other difficulty of continuing to support it, I feel announcing the end of support in this collection's next major version is prudent.

---

This post will be updated with more information as it becomes available.

hashi_vault - add options for retries on failure

SUMMARY

Vault being an HTTP service, there are numerous reasons why a single request could fail to complete, and it would be great to have a way to retry those requests.

Unlike tasks, there's no easy way to "retry" a lookup plugin (or indeed most controller-side plugin types) on the whole, so this isn't easily solved outside the plugin itself.

Disorganized list of thoughts/considerations:

• offer some sort of backoff, maybe several backoff strategies
• include jitter
• be smart about which failures to retry on (not access denied, not found, etc.), can it be configurable without being too complicated?
• backoff module would provide dead-simple implementation, but an additional dependency
• can requests/urllib3 retries be used?
  • is that an add'l dependency, or does it come with requests already?
  • Could that be used properly with hvac adapters? If so, may be best supported solution, though more work to implement.

This will probably not be implemented until things move a little more central, so that it can be tested more easily, independently of specific plugins.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

hashi_vault mTLS support

SUMMARY

Currently, the lookup plugin does not support authorization for a client certificate.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

community.general/plugins/lookup/hashi_vault.py

ADDITIONAL INFORMATION

- debug:
    msg: "{{ lookup('community.general.hashi_vault', 'secret/hola:value', url='https://myvault:8200', token=vault_token,
      ca_cert='/cert/path/ca.crt', tls_key='/cert/path/client.key', tls_cert='/cert/path/client.crt') }}"

It may make sense to add an alias tls_ca for ca_cert to use parameters with names appropriate to the sense.

Add ansible_hashi_vault_token_file and ansible_hashi_vault_token_path variables

SUMMARY

Add variables equivalent to ANSIBLE_HASHI_VAULT_TOKEN_FILE and ANSIBLE_HASHI_VAULT_TOKEN_PATH, as it allow one to change this based in inventory settings

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

With #15
we get environment for setting vault token path and file, this is just expand their support for ansible variable

hashi_vault - lookup plugin should allow environment variables for Username / Password auth

SUMMARY

Moved from following issue in previous repository: Original Issue

Other authentication method parameters like token for instance can be passed via certain environment variables. There should be specific environment variables to pass "username", "password" and "mount_point" parameters to allow for cleaner and more flexible calls when using the lookup plugin with instance or userpass authentication.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault

ADDITIONAL INFORMATION

This would allow for global initializing of the parameters and solve problems of reuising them on each usage of the lookup plugin.

This would be an example of the current state:

---
- hosts: webservers
  gather_facts: yes
  environment:
    VAULT_AUTH_METHOD: "ldap"
  vars:
      first_var: "{{lookup('hashi_vault', 'secret=kv/example/example:first_var mount_point=ldap username=user_xyz password=******************')}}"
      second_var: "{{lookup('hashi_vault', 'secret=kv/example/example:second_var mount_point=ldap username=user_xyz password=******************')}}"
      another_var: "{{lookup('hashi_vault', 'secret=kv/example/example:another_var mount_point=ldap username=user_xyz password=******************')}}"
      yet_another_var: "{{lookup('hashi_vault', 'secret=kv/example/example:yet_another_var mount_point=ldap username=user_xyz password=******************)}}"

This would be an example of how the usage after implementation:

---
- hosts: webservers
  gather_facts: yes
  environment:
    VAULT_USERNAME: "user_xyz"
    VAULT_PASSWORD: "*******************"
    VAULT_AUTH_METHOD: "ldap"
    VAULT_MOUNT_POINT: "ldap"
  vars:
      first_var: "{{lookup('hashi_vault', 'secret=kv/example/example:first_var ')}}"
      second_var: "{{lookup('hashi_vault', 'secret=kv/example/example:second_var')}}"
      another_var: "{{lookup('hashi_vault', 'secret=kv/example/example:another_var')}}"
      yet_another_var: "{{lookup('hashi_vault', 'secret=kv/example/example:yet_another_var)}}"

hashi_vault - token used for auth may inadvertently be inferred

SUMMARY

See: ansible-collections/community.general#1213 (review)

What we found during the test run in the linked PR is that HVAC was using the root token of our Vault dev server used in CI. This masked an issue with the JWT auth method in HVAC not behaving like other auth methods ( hvac/hvac#644 ) because we ended up using the root token instead.

The only test that caught this was the one that expected an access denied error with the given auth, because of course the root token had permission. If that test had not been there, it would have gone unnoticed and JWT auth would have been completely broken, with passing tests.

In ansible-collections/community.general#23 this bug was fixed for the general case of a failed auth method using an otherwise implied token, but it seems the HVAC library itself is doing some inference that we haven't accounted for. This may be specific to Vault being in dev mode.

---

Proposed investigation steps:

• Investigate the HVAC codebase to determine how it's looking for implicit tokens
• Look at what conditions this happens under
• Plan further resolution

Possible remediation options:

• replicate all implicit token loading as first-class options, the way we have for VAULT_TOKEN and the token file created by vault login
• perform an immediate .logout() after we first create the HVAC client, ensuring we don't have anything implicit that we haven't loaded via the plugin
• come up with a better test for this scenario, hopefully one that isn't specific to any one auth method, to quickly catch regressions

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault.py

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

remove templating in conditionals in tests

SUMMARY

There's templating in conditionals in a bunch of tests. They work fine but throw warnings since conditionals are already "in" a template.

This is just a reminder to go through those and replace the templating.

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

hashi_vault - lookup failed with connection error

SUMMARY

Hi! we got a randomly error with hashi_vault lookup plugin

Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'community.general.hashi_vault'. Error was a <class 'requests.exceptions.ConnectionError'>, original message: HTTPSConnectionPool(host='vault.example.com', port=80): Max retries exceeded with url: /v1/auth/token/lookup-self (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6fa81917b8>: Failed to establish a new connection: [Errno -2] Name or service not known',))"

It doesn't seems that it hc vault server error. we couldn't find anything unusual on server side.
We need to know why this error is happened. Sometimes everything fine, sometimes not.

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.general.hashi_vault

ANSIBLE VERSION

ansible 2.10.5
  config file = /mnt/d/git/myoffice/ansible_playbooks/ansible.cfg
  configured module search path = ['/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dusachev/.local/lib/python3.6/site-packages/ansible
  executable location = /home/dusachev/.local/bin/ansible
  python version = 3.6.9 (default, Oct  8 2020, 12:12:24) [GCC 8.4.0]

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

here is example playbook that we use

---

- name: "Vault test playbook"
  hosts: testhost
  tasks:
    - name: "facts"
      set_fact:
        vault_secrets:
          secret1: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret1') }}"
          secret2: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret2') }}"
          secret3: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret3') }}"
          secret4: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret4') }}"
          secret5: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret5') }}"
          secret6: "{{ lookup('community.general.hashi_vault', 'kv/data/stands/teststand/secret6') }}"

EXPECTED RESULTS

ACTUAL RESULTS

retries - `urllib3` fallback import from vendored version in `requests` is not correct

SUMMARY

In implementing retries, we need urllib3.util.Retry. We first try to import a standalone version of urllib3, which is usually available.

We have a fallback in case that fails to try to import via the requests library, which has a vendored version of urllib3 as well:

from requests.packages.urllib3.util import Retry

However this is not the correct form; instead we must import urllib3 as a module:

from requests.packages import urllib3

And then use it from there.

See: https://docs.python-requests.org/en/master/community/updates/#id25

No bug reports or issues in the wild have yet come up as a result of this, to my knowledge.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

tests - consider flake8-pytest-style

SUMMARY

As suggested in: https://github.com/ansible-collections/community.hashi_vault/pull/128/files#r696697212

The collection has a fair amount of unit tests (and growing) and it would be good to catch issues and try to enforce some style consistency there.

thoughts:

• local usage: right now all tests are run with ansible-test, so will think about how to encourage folks to run it locally before CI
• in CI: would fit nicely into the unit test jobs probably
• can rules/settings be defined in a way that will work for local/CI runs and also work well with IDE/editor integration?

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

hashi_vault - Can't login using username/password (userpass auth method)

SUMMARY

I can use the lookup plugin to login with a user token, but I can't login using a username and password.

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault lookup plugin, possibly with the integration with the hvac library.

ANSIBLE VERSION

ansible 2.10.2
  config file = None
  configured module search path = ['/home/jota/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jota/Documents/Tecnativa/infra/ci-templates/.venv/lib/python3.9/site-packages/ansible
  executable location = /home/jota/Documents/Tecnativa/infra/ci-templates/.venv/bin/ansible
  python version = 3.9.2 (default, Feb 20 2021, 00:00:00) [GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]

CONFIGURATION

(empty)

OS / ENVIRONMENT

Fedora 33, running a playbook against localhost (local connection)

STEPS TO REPRODUCE

I created a simple playbook just to test this case:

- name: Test
  hosts: localhost
  connection: local
  vars:
    vault_username: MY_USER_NAME
    vault_password: MY_USER_PASSWORD
    vault_api: URL_TO_MY_SERVER
    vault_token: MY_USER_TOKEN
  tasks:
    - ansible.builtin.debug:
        msg:
          "{{ lookup('community.hashi_vault.hashi_vault',
          'secret=secret/data/SOME-SECRET/SUB-SECRET', token=vault_token, url=vault_api)
          }}"
    - ansible.builtin.debug:
        msg:
          "{{ lookup('community.hashi_vault.hashi_vault',
          'secret=secret/data/SOME-SECRET/SUB-SECRET', auth_method='userpass',
          username=vault_username, password=vault_password, url=vault_api) }}"

EXPECTED RESULTS

Both cases should work (retrieve secrets with a token and with a username/password pair)

ACTUAL RESULTS

The first command (with a token) works (proving my user has access to said secret and that its token is valid)
The second command does not work, giving the following error:

TASK [ansible.builtin.debug] *********************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.hashi_vault'. Error was a <class 'hvac.exceptions.InvalidRequest'>, original message: missing client token, on get URL_TO_MY_SERVER/v1/secret/data/SOME-SECRET/SUB-SECRET"}

cc @yajo

docs build - process notes and TODOs

SUMMARY

This issue is meant to keep track of things related to the docs build process, which builds and publishes collection documentation on PRs and pushes, in temporary sites outside of the published collection docs on docs.ansible.com.

Current State

On a PR (open, updated, resynced)

• The docs from the target are built (rendered)
• The docs (docsite changes only) from the PR are built
• Hashes of both are recorded to determine if the PR makes any differences
• A git diff of the builds is taken to show in markdown
• If there are changes, then:
  • The rendered docs are attached to the build as an artifact (expires in 7 days)
  • The rendered docs are published via https://surge.sh to a temporary site that's specific to the PR. This lets anyone browse the fully rendered docs by clicking a link.
  • A bot comment on the PR provides links to the rendered sites of both the target branch docs and the PR's docs for comparison, and shows the git diff output in an expandable section (unless it's too big, which crashes GHA), and it shows a list of modified files, each of which links to the published docsite.
• If there are no changes:
  • if there's an existing surge site (because there were previously changes in the PR), it is unpublished
  • if there's an existing PR comment (because of previous docs changes), it is removed.

In addition to the above, another build is triggered which does exactly the same thing, but it will build the entire set of docs using the entire PR contents. That means it will include changes to plugin and module documentation.

The key for this one is that it gets built in a separate GitHub "environment" with rules that require it to be approved on every commit. This happens because the build is still privileged and we cannot trust the PR contents, so every commit must be reviewed by a person before approving this build. But once it runs, we get the complete picture of the built docs.

On PR close or merge

• Any existing surge site is unpublished
• If there's an existing PR comment, it is updated
  • if the PR was closed, the comment is updated to say the site is unpublished
  • if the PR was merged, the comment is updated thanking for the contribution, and linking to the main surge docsite build (as the changes have now been incorporated there)

On Push (to main only)

• the docs are built, and the "main" surge site is updated.

Important things

The pull_request_target event

REQUIRED READING: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

When a GitHub Workflow is run for the pull_request event, from a fork of the repository (the norm in this and most open source projects), the rights are limited, compared to PRs opened from a branch on the same repo (which requires write access to the repo). In particular, on a PR from a fork:

• the token used within workflow does not have write access to the repository, so it cannot (for example) commit, create/remove/update comments, open issues, etc.
• the workflow cannot read secrets

These are important security considerations, because anyone can open a PR, and use it to execute untrusted code, leading to unauthorized commits on the repo, secret exposure, etc.

For the purposes of this doc build process, this is important because:

• we need to store a secret for the token required to publish to surge
• we need to be able to created/edit/delete comments in PRs

Enter pull_request_target

Workflows run under this event do have access to secrets, and do have write access to the repository. The difference is that both the workflow definition itself, and its defined "ref" refer to the target of the PR. That means that the workflow file is always loaded from the target branch (for example main) so that it cannot be manipulated in the PR, and it also means that using the checkout action with its defaults is going to clone the target (ex main`) and not the PR content.

The point of all this is: you can't trust the PR content when running in a context that has access to secrets and elevated privileges.

It is possible to checkout the PR contents, but you must be careful not to execute anything in it, and there's lots of indirect ways that using the content of a PR could lead to exploitation, even without direct "execution".

Bringing it back to docs build

We use pull_request_target so we can get the surge secret and so we can post PR comments.

But in order to render the docs in the PR, we need some content from the PR.

In our case, we bring in only docs/docsite/, which contains YAML and RST. While a PR could stick other files there, we won't be executing anything in that subdirectory.

Problems and difficulties with pull_request_target

Problem 1: Changing the workflow

Because the workflow is always loaded from the target, it's very difficult to make changes to the workflow via the PR process and test them from within the same PR. The version of the workflow will always be loaded from main.

I've dealt with this in two ways:

Commit/merge it now, use other PRs, update incrementally

• For this, I'd put up a PR with a change to keep track of the purpose and what's being changed, and then merge it right away.
• A new PR would be put up that tests the functionality (it will be using the version of the workflow now in main)
• If anything needs to be changed in the workflow, I make new commits, put the original workflow PR # in the commit message (for example fix type in docs workflow (#123)), and push the commit to main.
• By mentioning the PR, GitHub will automatically show it linked in the original PR, even though it was not part of the PR; that at least keeps some semblance of cohesiveness.

This isn't great; pushing commits directly to main, some of which are the type that squashing was made for, and squashing those after the fact means a force push (also not good).

Put workflow changes on a direct branch, open other PRs against the branch

• In this method, the PR making changes to the workflow must be in a branch in the actual repository, not in a fork
• A new PR is still needed to test the changes, but this time, the PR should be made against the above branch, not against main (that is, you'll be asking to merge the PR into the branch that the workflow PR is based on). This PR can be in a fork.
• Updates to the workflow are pushed to the workflow PR branch as usual (just remember to push upstream if your origin is your fork)
• One difficulty here is that if you re-run a CI run on the test PR, it will not use the updated state of the workflow branch, it will use the one it ran against before (the exact commit). So if you update your workflow branch, your test PRs need some other change to trigger their CI to run against the new workflow. I've been using empty commits for that a la git commit -m empty --allow-empty and that's enough to re-trigger.
• A test PR that doesn't contain actual changes you want will be closed anyway. A test PR with changes you might want to keep is going to be squashed, or a manual rebase is super easy (removing the empty commits altogether).

This method is a little more complex, but it avoids the direct pushes.

Both methods make it near impossible for someone without write access to the repo to meaningfully contribute to the docs build process.

Problem 2: which docs?

The docs build process includes plugin and module documentation, which is very nice. However because of the security implications of pll_request_target, we're only actually using the docs/docsite/ folder from the PR. So changes made to collection content other than the docsite, will not be taken into account in the docs build.

The docstrings for executable content live in .py files, and even though some of them are read via AST and not executed, doc fragments are executed, and as a result, I'm not sure that we can ever include them in PRs submitted on forks.

One idea I had was to possibly introduce a workflow_dispacth event-based workflow (manual invocation) that can be run by a maintainer. This workflow would run against a PR, and render the entire docsite. The idea would be that a maintainer who has reviewed the PR can decide to run the docs build against the entire PR contents, and the resulting docs build site would contain all changes. A way of sorting gating that process. It would have to be re-done on every new commit to a PR as well.

Another idea I had was to figure out if the PR was submitted on a branch in the same repo, which can implicitly be trusted (the submitter had write access already). This doesn't help much on the external contributor front, but in many/most collections, the maintainers are very active contributors (it would certainly help me!).

Both of the above will make for some tricky logic and conditionals in the workflows, making them harder to read and grok.

So this has been sorta-solved by the separate build that requires approval (scroll up).

Problem 3: workflows running in forks

Right now, my workflow compares where it's running from to ensure it's running in ansible-collections/community.hashi_vault, and skips many/most steps if it's not. This was really an indirect way of saying "a fork doesn't have the surge secret so don't bother running all this stuff, it'll fail anyway".

It would be better to check for the existence of a secret instead, so that forks could optionally add such a thing.

That leads to the possibility of using secrets to define things like the surge site name stub, etc.

It has some weird implications for PR comments though.

ISSUE TYPE

• Documentation Report

COMPONENT NAME

docs build

ANSIBLE VERSION

N/A

hashi_vault - Retrieve list of secrets from path

SUMMARY

In the documentation there is this example

- name: Return all secrets from a path
  ansible.builtin.debug:
    msg: "{{ lookup('community.general.hashi_vault', 'secret=secret/hello token=c975b780-d1be-8016-866b-01d0f9b688a5 url=http://myvault:8200') }}"

This does not seem to work for me.
In the code i cant find any reference to hvac's list function

client.secrets.kv.v2.list_secrets(mount_point="kv-v2", path="folder")

Am i doing something wrong, or is this a misnomer? As i understand it, the last part of the path is referred to as the "secret", which then contains keys/values.
In a path multiple secrets can exists, so "Return all secrets from a path" should list all secret-names in that path, or the complete secrets as dicts.
But currently it seems to only work when the path ends with a secret name, then listing all keys/values of that one secret...

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault

ANSIBLE VERSION

❯ ansible --version
ansible 2.10.5
  config file = /home/buehring/GIT/srv-sftp-proxy/modules/customization/ansible/ansible.cfg
  configured module search path = ['/home/buehring/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/buehring/.local/lib/python3.6/site-packages/ansible
  executable location = /home/buehring/.local/bin/ansible
  python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]

CONFIGURATION

Im using this part, which works fine as soon as i reference the full path to a secret, but not if i remove the last part to get a list of secrets

  vars:
    vault_token: "{{ lookup('env', 'VAULT_TOKEN')}}"
    vault_addr: "{{ lookup('env', 'VAULT_ADDR')}}"
    vault_data: "{{ lookup('hashi_vault', 'secret=infrastructure/data/srv-sftp-proxy/allgemein token={{ vault_token }} url={{ vault_addr }}')}}"

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

aws_iam_login method needs tests

SUMMARY

There's some difficulty to this one, as we don't have any AWS IAM stuff to set up a backend against.

Unit tests can be added for a lot of it though, especially the validation/option processing.

For some level of integration, I am considering introducing MMock, and trying to mock the Vault API end of things so that we never actually have to contact AWS.

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Galaxy link from "About" section in github is misspelled

SUMMARY

Current link:
http://galaxy.ansible.com/community/hash_vault
Correct Link:
https://galaxy.ansible.com/community/hashi_vault

ISSUE TYPE

• Documentation Report

COMPONENT NAME

n/a

ANSIBLE VERSION

n/a

New environment variable scheme causes issues in some usage

SUMMARY

Hi - I was pointed here by ansible/awx#10275.

In AWX, we let users create custom credentials to pass secrets/info to playbooks for use by modules. However, we do not allow any custom credentials that pass variables that start with ANSIBLE_ - allowing a user delegated access to a playbook to pass arbitrary ANSIBLE_* parameters is a security risk, as they can change ansible behavior in unexpected ways (think ANSIBLE_SSH_EXECUTABLE, ANSIBLE_COLLLECTION_PATHS, and other things).

This means that at least in AWX, we can't allow users to create a custom credential that uses the new variables in use by v2 of this collection.

Is it possible to keep the legacy variables?

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.hashi_vault

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

under AWX, or Ansible Automation Platform

cc @infra-monkey

hashi_vault - [env] Update VAULT_NAMESPACE to match guidelines & add INI

SUMMARY

See #10

VAULT_NAMESPACE is an environment variable used by the Vault CLI. I'm proposing the following change:

• Add ANSIBLE_HASHI_VAULT_NAMESPACE to env: in spec
• Add INI value as well
• Process VAULT_NAMESPACE as a fallback to populate the option when it hasn't been set via any other means.

ℹ This change is not breaking because lowering the precedence of the existing env var doesn't raise any existing value's precedence (there's no other env var or INI value currently).

ISSUE TYPE

• Feature Request

COMPONENT NAME

hashi_vault.py

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Feature Request : Support Vault Agent Credentials

SUMMARY

Support the use of a locally running Vault Agent process to interact with Vault. This relieves the user almost entirely have need to manage or interact with tokens. Example case:

• Vault Agent is configured to perform AWS IAM authentication.
• The Vault Agent has a TCP listener on 127.0.0.1:8100 (standard HashiCorp deployment examples)
• Ansible using hashi_vault then just instructs hvac to connect to the local listener and authentication is transparent.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

community.hashi_vault, authentication methods.

ADDITIONAL INFORMATION

Related: ansible/ansible#60728
It appears that this was suggested before, but deemed unsupported by the upstream Python hvac module, but the notes in the docs there suggest that the module is missing support for UNIX Sockets connections to the Vault Agent. This works fine when connecting to a local TCP Vault Agent listener.

hashi_vault - [env] update role_id and secret_id env vars to match guidelines

SUMMARY

See #10

VAULT_ROLE_ID and VAULT_SECRET_ID use the VAULT_ prefix.

Proposed change:

• ANSIBLE_HASHI_VAULT_ROLE_ID and ANSIBLE_HASHI_VAULT_SECRET_ID will be added in v0.2.0
• VAULT_ROLE_ID and VAULT_SECRET_ID will be deprecated in v0.2.0 and removed in v2.0.0
• The VAULT_ prefixed vars will not be preserved, as they are not currently in use by the Vault CLI for this (or any) purpose; future use would conflict and this should be avoided.

⚠This is a breaking change
📝Impact Notes:

• Affects users who use these vars (for the approle, aws_iam_login, and jwt auth methods) which are available in the following collections and versions:
  • community.general version 0.2.0 and above
  • community.hashi_vault version 0.1.0

Does not affect anyone using INI or direct parameter setting.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

Allow to continue execution of plugin when secret not found or permission denied

SUMMARY

Allow the plugin to continue without failure when Vault returns a permission denied error or a value of None (secret does not exist)

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault lookup plugin

ADDITIONAL INFORMATION

Vault can return a permission denied error, or None data when the secret does not exist or the user does not have permission to access it.
However, there might be situations where you need to follow the execution of the plugin and playbook, and it would be nice to get an empty answer instead of an exception thrown in those situations.

- name: obtain secrets
      loop:
        - secret1
        - secret2
        - secret3
      set_fact:
        secrets_dict:
          "{{ secrets_dict | d({}) | combine( { item : lookup('community.hashi_vault.hashi_vault', 'secret/data/secrets/' + item, auth_method='userpass', password=vault_password, username=vault_username, url=vault_api, allow_failure=True) } )}}"

The allow_failure option would prevent those 2 exceptions from being thrown, but instead make the plugin return an empty value, which could be treated easily in Ansible.

cc @yajo

hashi_vault - [env] INI value for vault URL should take precedence over VAULT_ADDR env var

SUMMARY

This is similar to ansible-collections/community.general#373

We take the value of VAULT_ADDR in the plugin to emulate the vault CLI and use the env var, which is commonly set. In ansible-collections/community.general#23 an INI entry was added, but if VAULT_ADDR is set the INI value won't be used.

Since VAULT_ADDR is a more general value, in the context of Ansible, we should prefer the INI value if both are set.

The fix would be similar to that in ansible-collections/community.general#902 , manually loading the value. Also adding an additional (spec defined) env var, ANSIBLE_HASHI_VAULT_ADDR which could override the INI.

General env var guidance for this plugin: #10

Plan:

• ANSIBLE_HASHI_VAULT_ADDR was introduced in v0.2.0
• VAULT_ADDR will had its precedence lowered in v1.0.0

⚠This is technically a breaking change
📝Impact Notes:

• It is unlikely to break most existing configurations, but it is possible. All of the following conditions must be met for a configuration to break:
  • The url parameter is not being passed explicitly to the plugin in the lookup() call
  • url is set in both the INI file and via the VAULT_ADDR env var
  • the values of the url in the above places are different
• With those conditions, the value of VAULT_ADDR is used before this change, and the other different value set in INI is used after this change.
• For users who are aware of this change before they consume it, it can be worked around in advance via the following remediations:
  • If the mismatched env and INI values are unintentional, setting them to be the same (or choosing one place to set them) can be done with no ill effects before and after the change.
  • If the mismatch is intentional (in one environment the INI value is used, while in another invocation the env var is being used to intentionally override INI), then this can be worked around:
    • In community.hashi_vault v0.1.0: ideally upgrade to v0.2.0 then follow the next step, otherwise: set both VAULT_ADDR and the not-yet-used ANSIBLE_HASHI_VAULT_ADDR to the same value in those invocations. After updating to a post-change collection (either v0.2.0 or v1.0.0), remove the VAULT_ADDR setting.
    • In community.hashi_vault v0.2.0: set ANSIBLE_HASHI_VAULT_ADDR instead of VAULT_ADDR. No other changes are needed for v1.0.0.

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault.py

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

INI value is specific to ansible, and should be used over the more system wide VAULT_ADDR.

ACTUAL RESULTS

INI value will never be used if VAULT_ADDR is set.

Unable to retrieve a single key from kv v2

SUMMARY

Unable to retrieve a single key from Key/Value v2 secret engine.

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault 1.0.0

STEPS TO REPRODUCE

---
- hosts: all
  gather_facts: no
  tasks:
  - debug:
      msg: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/path1/path2 url=http://vault.com auth_method=approle role_id=1234 secret_id=5678') }}"
  - debug:
      msg: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/path1/path2:key1 url=http://vault.com auth_method=approle role_id=1234 secret_id=5678') }}"
    failed_when: false

EXPECTED RESULTS

Single value or key/value pair is returned.

ACTUAL RESULTS

TASK [debug] *******************************************************************
ok: [server1] => {
    "msg": {
        "key1": "secret1",
        "key2": "secret2",
        "key3": "secret3"
    }
}
TASK [debug] *******************************************************************
fatal: [server1]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'community.hashi_vault.hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret secret/path1/path2  does not contain the field 'key1'. for hashi_vault lookup"}

hashi_vault - [env] update VAULT_AUTH_METHOD to match guidelines

SUMMARY

See #10

VAULT_AUTH_METHOD uses the VAULT_ prefix.

Proposed change:

• ANSIBLE_HASHI_VAULT_AUTH_METHOD will be added in v0.2.0
• VAULT_AUTH_METHOD will be deprecated in v0.2.0 and removed in v2.0.0
• VAULT_AUTH_METHOD will not be preserved, as it is not currently in use by the Vault CLI for this (or any) purpose; future use would conflict and this should be avoided.

⚠This is a breaking change
📝Impact Notes:

• Affects users who use VAULT_AUTH_METHOD which is available in the following collections and versions:
  • community.general version 0.2.0 and above
  • community.hashi_vault version 0.1.0

Does not affect anyone using INI or direct parameter setting.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

Remove default Vault address (http://127.0.0.1:8200) for the url option and make it required

The default value for the url option is http://127.0.0.1:8200, which is meant to somewhat correspond to the "defaults" used by Vault CLI and the hvac library.

For several reasons, the default is unlikely to be correct for production uses (it doesn't use TLS, runs on localhost, etc.).

Instead, my observation in real world use cases is that people tend to forget to set a Vault address and then are confused about why it isn't working because they see the access denied, timed out, etc. error message but don't really notice that they aren't connecting to the correct Vault server.

As a result, I think the default does more harm than good.

I've also received feedback along the same lines (here's one example), and I'm tending to agree with the critics.

We also provide many ways of specifying the address:

• direct argument
• INI config
• multiple environment variables
• (upcoming #49) ansible vars

So anyone who is using the default for its intended value, has several other ways of not specifying it, including several that don't require modification of playbooks/tasks/templates.

Nonetheless it would be a breaking change and would happen in the next major version

---

Request for Feedback

If you have any opinion on this change either way, I invite you to provide it here!

SUMMARY

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

Allow token helpers to be used for the `hashi_vault` command

SUMMARY

Allow token helpers to be used for the hashi_vault command.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

ADDITIONAL INFORMATION

Hey all. Thank you very much for creating this plugin!

I personally use a token helper for vault (https://www.vaultproject.io/docs/commands/token-helper), which means I do not have a VAULT_TOKEN env variable set by vault login (I currently set it manually as a workaround).

I dug around and noticed that the hvac dependency itself does not support helpers directly, but can be tweaked to fetch it (hvac/hvac#639).

Is that something we could do in this plugin? Would find this more clean for my use case. I also can work on the PR myself if I get a green light :)

hashi_vault should support secret v2 path (without adding /data/)

SUMMARY

The hashi_vault lookup currently needs to use an API path to kv_v2 secrets. It would be nice to drop this requirement, like terraform do.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault

ADDITIONAL INFORMATION

The current call:

- name: get secret (currently)
  debug:
    msg: "{{ lookup('hashi_vault', 'engine/path/data/secret/path`) }}"

The wanted call:

- name: get secret (wanted)
  debug:
    msg: "{{ lookup('hashi_vault', 'engine/path/secret/path`) }}"

In the terraform provider, they get the engine path from a call to /v1/sys/internal/ui/mounts/{{ FULL_SECRET_PATH }}.

tests - remove separate "generic" secrets engine, combine with kv v1

SUMMARY

The tests currently set up and test the "generic" secrets engine, along with KV versions 1 & 2.

Since Vault 0.8.3 generic is just an alias for kv (version 1), so we really don't need to test "against" both.

Removing the creation and setup is easy, but I need to see which tests are run against the gen path and ensure they have coverage in kv1 without blindly just changing them all, as some might be duplicated in both.

ISSUE TYPE

COMPONENT NAME

hashi_vault - environment variable guidelines

SUMMARY

The hashi_vault lookup takes several parameters via environment variables, and recently, some via INI (ansible.cfg) as well.

Most of the original vars were intended to re-use vars that are commonly used in the vault CLI., like VAULT_ADDR and VAULT_TOKEN.

Along the way some newer ones have been added that aren't related to the vault CLI but are strictly for the plugin, but also adopted the VAULT_ prefix, and some others used ANSIBLE_VAULT_ as their prefix.

An additional consideration is that in Ansible, environment variables have higher precedence (are preferred) over INI values. We can think of this in terms of specificity; the INI value is defined "ansible-wide" (for your specific implementation), while the env var is defined per invocation so it should be considered more specific, especially given that the env vars defined for plugin parameters are specific to that Ansible plugin (usually denoted with the ANSIBLE_ prefix) and so have no use outside of that context.

The precedence issue becomes less clear when we take values that have meaning outside of ansible, like the VAULT_ vars used in the vault CLI.

I believe we should consider then that an env var with meaning outside of Ansible has a lower precedence (is less preferred) than an INI value defined in ansible.cfg, as in the context of Ansible, an INI value is a more specific value.

Additional background on the ANSIBLE_ prefix:

• https://meetbot.fedoraproject.org/ansible-meeting/2019-02-26/ansible_core_irc_public_meeting.2019-02-26-19.02.html
• ansible/ansible#52967 (comment)

PROPOSAL

This proposal aims to provide guidelines on how we name environment variables, and how we handle their precedence.

Ansible-only vars

This applies to a parameter to the plugin that will not inherit any existing vault CLI env var.

1. The variable should use the prefix ANSIBLE_HASHI_VAULT_. This name is on the long side, however it keeps to the pattern of starting with ANSIBLE_. Also considered (and even used before this proposal) was ANSIBLE_VAULT_ however this may cause confusion with Ansible Vault (unrelated to this plugin).
2. This variable should follow existing Ansible precedence rules, and be specified in the env: key in the plugin's argspec, so that the standard plugin loader can take care of it.

Even if there is no specific existing VAULT_ env var, we don't know whether one will be introduced in the future, and so don't want to use these names.

Variables used outside of Ansible

This applies to env vars that have a meaning outside of Ansible, but whose value we want to use. Examples include the vault CLI env vars, but also the $HOME variable as used for a default for the token_path parameter.

1. As far as naming, we of course have no choice but to use the name as it exists.
2. We should NOT specify this value in the argspec. Instead, we must write code that retrieves this value in the case that the parameter was not already set via any other means. This enforces lowest precedence.
3. Consider adding an additional ANSIBLE_HASHI_VAULT_ var (following guidelines above to add it to the argspec) so that the value can be overridden for specific Ansible invocations, overriding both the non-Ansible var and INI value.

RELATED ISSUES & PRs

See this project: https://github.com/ansible-collections/community.hashi_vault/projects/1

Not all will be listed here, but want to point some specific examples.

• ansible-collections/community.general#373 / ansible-collections/community.general#902 -- here the issue of specifying $HOME in the argspec was revealed to make the INI value practically unusable, since $HOME is basically always defined already. The solution was to load its value in code only when the value wasn't populated by any other method. In this PR I also added Ansible-specific env vars for token_path and token_file, but used a VAULT_ prefix which I now realize was an anti-pattern (I expect to introduce a PR to correct that).
• #8 -- here I'm realizing that given VAULT_ADDR is probably the most common way of setting the address for a vault server, it becomes more difficult to use an INI value to enforce a specific server's usage in Ansible, without overriding the env var on every invocation. I propose a new Ansible-specific var and to load VAULT_ADDR in code as the least preferred source of the value.

COMPONENT NAME

hashi_vault.py

hashi_vault lookup plugin require auth/token/lookup-self policy on the Vault token to read secrets

SUMMARY

Hello folks 👋

I found a behavior that seems weird to me: I'm using the hashi_vault lookup plugin with the token auth_method, I'm just setting a Vault token with a minimal set of vault policies (principle of least privilege) to my CI/CD to run the playbook that fetches secrets and the CI/CD system cannot fetches the secrets.

By using the token (with the given policies) with the Vault CLI I don't get any trouble.

My policies contains only sufficient rules to fetch the secrets that I want (read and list on specific vault path).

By investigating the code I came to that:

1. When using the token auth_method, the function auth_token is called.

2. This function call is_authenticated

3. That latest function only return True if the token have the capability to lookup-itself

So I came to the conclusion that if I want my playbook to run I have to add the lookup-itself capability in my token policies. I made the test and it works.

However Vault gives the capability to request its API by just setting the VAULT_TOKEN env variable, so why do we need that is_authenticated check for the token part ?

Thanks for your reply 🙏

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault

ANSIBLE VERSION

ansible 2.10.3
  config file = /Users/***/***/ansible.cfg
  configured module search path = ['/Users/***/***/library']
  ansible python module location = /Users/***/venv/ansible/lib/python3.8/site-packages/ansible
  executable location = /Users/***/venv/ansible/bin/ansible
  python version = 3.8.6 (default, Oct  8 2020, 14:07:53) [Clang 11.0.0 (clang-1100.0.33.17)]

CONFIGURATION

DEFAULT_MODULE_PATH(env: ANSIBLE_LIBRARY) = ['/Users/***/library']
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = ['/Users/***/.ansible/roles', '/Users/***/***/roles']
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /Users/***/.ansible/vault_password
DEPRECATION_WARNINGS(env: ANSIBLE_DEPRECATION_WARNINGS) = False
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

OS / ENVIRONMENT

ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G95

STEPS TO REPRODUCE

 name: "Test"
  hosts: 127.0.0.1
  tasks:
  - debug:
      msg: "{{ lookup('community.general.hashi_vault', 'secret=kv/data/foo:secret')}}"

EXPECTED RESULTS

PLAY [Test] **************************************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [debug] *************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
    "msg": "bar"

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

ACTUAL RESULTS

LAY [Test] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [debug] ***********************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'community.general.hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Invalid Hashicorp Vault Token Specified for hashi_vault lookup."}

PLAY RECAP *************************************************************************************************************************************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Fix Unit tests in light of core/devel dropping support for Python < 3.8 and module_utils tests importing controller-side code

SUMMARY

See ansible-collections/overview#45 (comment) and ensuing discussion.

Need to figure out:

• Supported Python versions for this collection
• How that works with support for earlier core versions
• How to handle unit tests for module_utils intended to work against controller and remote side code

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

hashi_vault - [env] Fix token_file and token_path env vars

SUMMARY

See #10

In ansible-collections/community.general#902 I replaced/added env vars prefixed by VAULT_ to solve the underlying issue, however thinking about that after the fact has led to #10 .

I'm proposing the following:

• The VAULT_TOKEN_FILE and VAULT_TOKEN_PATH env vars will be deprecated in v0.2.0 and removed in v2.0.0.
• The ANSIBLE_HASHI_VAULT_TOKEN_FILE and ANSIBLE_HASHI_VAULT_TOKEN_PATH vars will be added in v0.2.0
• The VAULT_ variants will not be preserved, as they are not currently in use by the Vault CLI for this (or any) purpose; future use would conflict and this should be avoided.

⚠This is a breaking change
📝Impact Notes:

• Affects users who use VAULT_ prefixed environment variables which are available in the following collections and versions:
  • community.general version 0.2.0 and above
  • community.hashi_vault version 0.1.0
• Does not affect anyone using INI or direct parameter setting

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

Hashi_vault lookup issues reading data from Kv2 when on ansible version 2.9

SUMMARY

fatal: [xxx.xxx.com]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'ansible.errors.AnsibleError'>, original message: The secret secret/data/dbre/elasticsearch/ESBOO/kibanapwd does not contain the field 'content'. for hashi_vault lookup"}

im getting this error when ansible version is 2.9.. the same works when the ansible version sis 2.10.

any way to get hashi_vault lookup work with ansible 2.9 version

here is my lookup command
kibana_password: "{{ lookup('hashi_vault', 'secret=secret/data/dbre/elasticsearch/{{ clustername }}/kibanapwd:content auth_method=ldap username={{ vault_user }} password={{ vault_pass }} url={{ vault_url }} validate_certs=True cacert=/cacert/capath/ca.pem') }}"

ISSUE TYPE

• Bug Report

COMPONENT NAME

hashi_vault
ansible 2.9
valut kv2

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

hashi_vault: tests fail (unstable) on macOS 10.15

SUMMARY

The tests for the hashi_vault lookup plugin fail on macOS (added to CI in ansible-collections/community.general#971). They are currently disabled.

ISSUE TYPE

• Bug Report

COMPONENT NAME

plugins/lookup/hashi_vault.py

ANSIBLE VERSION

2.10

Certify collection for use with Red Hat

SUMMARY

Our organisation does not have access to galaxy.ansible.com so we are unable to use this collection easily but we do have access to the certified content in cloud.redhat.com. I'd like to propose that this collection is certified by following the process here

https://access.redhat.com/articles/4916901 "How do I get a Collection certified?"

ISSUE TYPE

• Recommendation

COMPONENT NAME

N/A

ADDITIONAL INFORMATION

Would provide a level of trust to enterprise organisation and allow secure organisation to use a supported collection.

hvac 0.10.6 - AppRole auth method deprecation

SUMMARY

As of hvac 0.10.6:

AppRole is the latest auth method to be moved into the newer auth_methods class within hvac, meaning the old method is now deprecated and will show a warning to all users.

As we've done with other auth methods, we should be checking for existence of the new method type, using it if available, and if not fallback to the old method and show our own warning about upgrading hvac.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault.py

ADDITIONAL INFORMATION

Warning shown during execution:

/root/ansible_collections/community/hashi_vault/plugins/lookup/hashi_vault.py:460: DeprecationWarning: Call to deprecated function 'auth_approle'. This method will be removed in version '0.12.0' Please use the 'login' method on the 'hvac.api.auth_methods.approle' class moving forward.
  self.client.auth_approle(**params)

token auth does not handle a case of `vault_file` being a directory

SUMMARY

Will need to do a little more testing/evaluation, but it appears that there are two issues:

1. When token auth validation happens, and it looks for the token from disk (via token_path and token_file), if the token_file exists and is a directory, that condition is not properly handled and it raises an exception.
2. The above seems to occur even when auth_method != token; which shouldn't be the case.

Will confirm both cases and post a PR to fix.

ISSUE TYPE

• Bug Report

COMPONENT NAME

module_utils/auth_methods

ANSIBLE VERSION

N/A

COLLECTION VERSION

1.3.2

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

hashi_vault connection - add option to control timeout

SUMMARY

The timeout is currently fixed to hvac's default of 30s with no way to customize it.

When everything is working correctly it's unlikely it'll ever be hit. When a URL is bad or the server is down, 30s can feel like a lifetime. As I implement retry support (#71), I realize that this point compounds a lot if the request is hanging waiting for a timeout on every retry as well.

When the request legitimately needs to take longer (some auth methods or secret engines could presumably take time), it would be impossible for it to ever finish.

Note on the above: I have seen a case of AWS auth for example being hit so often that it triggers throttling errors on the AWS APIs, and Vault's internal retries (which are not configurable) have it keep it going for longer than 30s, and in this case the timeout for this plugin proved problematic, as it was our main way of triggering widespread AWS auth calls and we couldn't get the "real" response from the Vault server.

---

There should be a configurable timeout option to better control this.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

hashi_vault

ADDITIONAL INFORMATION