ansible / awx-resource-operator

AWX Resource Operator

An operator that can be used along side Ansible AWX to launch automation jobs and create job templates by creating resources directly in kubernetes. This operator is built with Operator SDK and Ansible.

Table of Contents

• AWX Resource Operator
  • Overview
    • Prerequisites
  • Installing the Operator
  • Usage
    • AnsibleJob
    • JobTemplate
  • Building and Testing
  • Author
  • Code of Conduct
  • Get Involved

Overview

There are currently two Custom Resources provided by the awx-resource-operator:

• AnsibleJob - launches a job in the AWX instance specified in the the specified k8s secret (awx host url, token)
• JobTemplate - creates a job template in the AWX instance specified.

Prerequisites

• Install the kubernetes-based cluster of your choice:
  • Openshift
  • K8s
  • CodeReady containers
  • minikube
• Deploy AWX using the awx-operator
• Create an OAuth2 token for your user in the AWX UI

Installing the Operator

1. Clone the awx-resource-operator

git clone [email protected]:ansible/awx-resource-operator.git

2. Log in to your cluster.

kubectl login <cluster-url>

3. Run the make target

NAMESPACE=awx IMG=quay.io/ansible/awx-resource-operator:latest make deploy

This will create a deployment named resource-operator-controller-manager in the namespace specified, in this case the awx namespace. The resulting operator pod will reconcile any AnsibleJob or JotTemplate resources created.

Usage

Now that the awx-resource-operator is installed in your cluster, you can try creating a k8s resource to manage your AWX instance.

First, you need create a k8s secret with connection information for your AWX instance though. Below is example yaml that you can save to a file, edit, and apply to your cluster.

Create an AWX Access Token

Create an OAuth2 token for your user in the AWX UI.

1. Navigate to Users
2. Select the username you wish to create a token for
3. Click on tokens, then the green plus icon
4. Application can be left empty, input a description and select the read/write scope.

Alternatively, you can create one at the command-line using the create_oauth2_token manage command (docs)

Create a Connection Secret

To make this connection information available to the operator, create a k8s secret with the host and token value. Here is example yaml for the connection k8s secret.

---
apiVersion: v1
kind: Secret
metadata:
  name: awx-access
  type: Opaque
stringData:
  token: <generated-token>
  host: https://my-awx-host.example.com/

Save this to a file, for example, awx-connection-secret.yml, edit it with your host and token value, the apply it to your cluster.

kubectl create -f awx-connection-secret.yml

AnsibleJob

Launch an automation job on AWX by creating an AnsibleJob resource. Be sure to specify the connection secret on connection_secret as well as the Job Template you wish to launch.

---
apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  generateName: demo-job-1 # generate a unique suffix per 'kubectl create'
spec:
  connection_secret: awx-access
  job_template_name: Demo Job Template

It is possible to configure other things like inventory, extra vars, and time to live for the job.

spec:
  connection_secret: awx-access
  job_template_name: Demo Job Template
  inventory: Demo Inventory                    # Inventory prompt on launch needs to be enabled
  runner_image: quay.io/ansible/awx-resource-runner
  runner_version: latest
  job_ttl: 100

  extra_vars:                                  # Extra variables prompt on launch needs to be enabled
     test_var: test

  job_tags: "provision,install,configuration"  # Specify tags to run
  skip_tags: "configuration,restart"           # Skip tasks with a given tag

Note that prompt on launch needs to be enabled for inventories and extra vars if you are specifying those. In order to enable "PROMPT ON LAUNCH", within the AWX UI:

1. Select Resources -> Templates
2. Within Templates, select your template and click the "PROMPT ON LAUNCH" checkbox next to INVENTORY and EXTRA VARIABLES.

Launching Workflows

Launch a Workflow Job Template with an AnsibleJob object by specifying the workflow_template_name instead of job_template_name.

---
apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  generateName: demo-job-1 # generate a unique suffix per 'kubectl create'
spec:
  connection_secret: awx-access
  workflow_template_name: Demo Workflow Template

JobTemplate

Create a job template on AWX by creating a JobTemplate resource.

---
apiVersion: tower.ansible.com/v1alpha1
kind: JobTemplate
metadata:
  name: jobtemplate-4
spec:
  connection_secret: awx-access
  job_template_name: ExampleJobTemplate4
  job_template_project: Demo Project
  job_template_playbook: hello_world.yml
  job_template_inventory: Demo Inventory

Building and Testing

If you have made changes to the operator and want to test them, you can do the following:

Build the operator image

make docker-build docker-push IMG=<your-registry>:dev

Deploy the operator

make deploy NAMESPACE=awx IMG=<your-registry>:dev

Author

This maintained by the Red Hat Ansible Team.

Code of Conduct

We ask all of our community members and contributors to adhere to the Ansible code of conduct. If you have questions or need assistance, please reach out to our community team at [email protected]

Get Involved

We welcome your feedback and ideas. The AWX resource operator uses the same mailing list and IRC channel as AWX itself. Here's how to reach us with feedback and questions:

• Join the #ansible-awx channel on irc.libera.chat
  • You can also join using Matrix and a client of your choice (Element is recommended) for a better experience (more info here)
• Join the mailing list

awx-resource-operator's Issues

Split up the readme into multiple files

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX Operator is open source software provided for free and that I might not receive a timely response.

Feature Summary

In preparation for creating a docsite, split up the existing readme into multiple smaller files in the /docs folder.

• draft out a proposal for how to split the file up. Use a comment in this github issue to describe the proposed set of new files.
• Once the writers agree, move that information into the agreed list of files.
• Leave everything from Contributing section to the end of the README file in the README file.
• Leave everything BEFORE the Table of contents in the README file.
• Add the new files to the mkdocs.yml and test the docs build

Add verbosity param for AnsibleJob

Users should be able to configure verbosity for their AnsibleJob's.

• https://docs.ansible.com/ansible/latest/collections/awx/awx/job_launch_module.html#parameter-verbosity

This would need to be added in these two places:

• https://github.com/ansible/awx-resource-operator/blob/devel/roles/job_runner/tasks/launch_job.yml#L7
• https://github.com/ansible/awx-resource-operator/blob/devel/roles/job_runner/tasks/launch_workflow.yml#L7

And a new variable would be needed for it on the AnsibleJob CRD schema & form.

Create scaffolding to make the docs locally

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX Operator is open source software provided for free and that I might not receive a timely response.

Feature Summary

Based on MkDocs, create the scaffolding required to build the docs locally.

Use ansible/awx-operator#1359 as an example of how to do this.

NOTE - we may not need everything in that example but it's a good working project to learn from

Create field for workflow and workflow template

Although Ansiblejob can work with workflows, as a user I would like to have the same view for workflows as I do with jobs.

Molecule tests

Figure out a way to reenable molecule testing

Rename job_template_ parameters to be consistent with AWX API

The parameters on the JobTemplate spec are named in a way that is not consistent with the AWX API. For example:

              job_template_inventory:
              job_template_playbook:
              job_template_project:
              job_template_ask_vars:
              job_template_ask_inventory:

I think we should deprecate these parameters and introduce and document new ones that align with the naming used in the AWX API.

job_template_inventory
job_template_playbook
job_template_project
ask_variables_on_launch
ask_inventory_on_launch

The above are just the ones in the CRD, however, there are a lot of "passthrough" JobTemplate parameters which we may want to deprecate and shim for a release in case anyone is relying on them.

job_type: "{{ job_template_type | default('run') }}"
credentials: "{{ job_template_credentials | default(omit) }}"
vault_credential: "{{ job_template_vault_credentials | default(omit) }}"
forks: "{{ job_template_forks | default(omit) }}"
limit: "{{ job_template_limit | default(omit) }}"
verbosity: "{{ job_template_verbosity | default(omit) }}"
extra_vars: "{{ job_template_extra_vars | default(omit) }}"
become_enabled: "{{ job_template_become_enabled | default(omit) }}"
allow_simultaneous: "{{ job_template_allow_simultaneous | default(omit) }}"
state: "{{ tower_resource_state if tower_resource_state is defined else 'present' }}"
ask_credential_on_launch: "{{ job_template_ask_credential | default(omit) }}"
ask_job_type_on_launch: "{{ job_template_ask_job_type | default(omit) }}"
ask_limit_on_launch: "{{ job_template_ask_limit | default(omit) }}"
ask_scm_branch_on_launch: "{{ job_template_ask_scm | default(omit) }}"
ask_skip_tags_on_launch: "{{ job_template_ask_skip_tags | default(omit) }}"
ask_tags_on_launch: "{{ job_template_ask_tags | default(omit) }}"
ask_verbosity_on_launch: "{{ job_template_ask_verbosity | default(omit) }}"
custom_virtualenv: "{{ job_template_venv | default(omit) }}"
force_handlers: "{{ job_template_force_handlers | default('no') }}"
host_config_key: "{{ job_template_host_config_key | default(omit) }}"
instance_groups: "{{ job_template_instance_groups | default(omit) }}"
job_slice_count: "{{ job_template_job_slice_count | default('1') }}"
job_tags: "{{ job_template_job_tags | default(omit) }}"
organization: "{{ job_template_organization | default(omit) }}"
scm_branch: "{{ job_template_scm_branch | default(omit) }}"
skip_tags: "{{ job_template_skip_tags | default(omit) }}"
start_at_task: "{{ job_template_start_at_task | default(omit) }}"
use_fast_cache: "{{ job_template_use_fast_cache | default(omit) }}"
webhook_credential: "{{ job_template_webhook_credential | default(omit) }}"
webhook_service: "{{ job_template_webhook_service | default(omit) }}"

*Backwards Compatibility*

We'll also need to add backwards compatibility shims in the role. Basically a task that:
* preferentially uses extra_vars_prompt_on_launch if set
* falls back to job_template_ask_vars
* If neither are set, omit it or set a default where applicable

### Checklist

- [ ] Deprecate the existing params we wish to change (done in the CRD)
- [ ] Create new entries for the new name
- [ ] Update the clusterServiceVersion UI form spec.descriptors so that the deprecated ones are hidden
- [ ] Also in the CSV base template, add spec.descriptor entries for the new params
- [ ] Add logic in the job_template role to check for the old param name and use it if the preferred (new) on is not set. This can be done [here](https://github.com/ansible/awx-resource-operator/blob/eef331ef0e0cba2002b6419542a1b7f5fc9914d4/roles/jobtemplate/tasks/main.yml#L21-L58) I think.
- [ ] Add documentation in the top level README.md
- [ ] Talk to other teams and customers who may be using this communicate this change and determine if it will be breaking given the changes above.

TTL Controller for Finished Resources causes AnsibleJobs to run repeatedly

In OCP 4.8 https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/ is enabled by default.
This means jobs get deleted after an hour. When the job gets deleted, it seems the AnsibleJob wants to re-run the automation again. This causes the automation to run hourly.

The feature gate for TTL Controller for Finished Resources can be disabled which I think will prevent this from happening. I'm trying that out now. To disable you do this:

Run: oc edit kubeapiservers.operator.openshift.io cluster and then update the unsupportedConfigOverrides

  unsupportedConfigOverrides: 
    apiServerArguments:
      feature-gates:
      - TTLAfterFinished=false

It would be nice if the AnsibleJob status could be used to prevent the re-run to allow job cleanup to work.

Improve error handling when WFJT or JT doesn't exist

We should fail early, and create a k8s event and status with info about why the job failed, specifically when the WFJT or JT doesn't exist).

spec:
  workflow_template_name: "does not exist"

Currently fails silently in operator logs iirc.

set gather_facts to false

@tima mentioned that gather_facts should be set to false for Ansible based operator to see some performance improvement. See operator-framework/operator-sdk-samples#160 for more details.

This issue is to make sure we don't lose track of this improvement.

AnsibleJobs fail at scale due to API timeout

Summary

AnsibleJob resources fail when run at scale because the AWX api/v2/job_templates endpoint can get very slow when there are many concurrent running jobs.

Details

I created 200 ansiblejobs at once and noticed that a number of them failed on this task because of a timeout caused by AWX taking too long to respond:

TASK [job_runner : Register Job result when complete] **************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was an unknown error when trying to connect to https://awx-awx.apps.aap-bugbash.ocp4.testing.ansible.com/api/v2/jobs/251/: timeout The read operation timed out"}

I did a couple GET's to the endpoint in question and the request times from the API were ranging from .3 to 3 seconds.

X-API-Product-Name: AWX
X-API-Product-Version: 23.4.0
X-API-Time: 2.612s

Analysis & Solutions

Problem 1: The job_templates API endpoint can't keep up

We could adjust the timeout on the awx.awx collection invocation in the runner role, or we could try to solve this on the awx api side... But I'm not sure what we could do there to reduce response time.

Problem 2: Operator pod can get OOMKilled

The jobs failing leads to jobs getting backed up and results in more concurrent running AnsibleJob reconciliation loops and pods for the resource operator.

Solutions:

• We could increase the "requests" for the resource-operator's pod because in combination with the pods spun up to run each job, it could exhaust the free memory available on that node, resulting in the OOMKilled operator pod.
• Better solution would be to employ a priority class
  • higher priority class value for the operator pod than the job pods so that the operator pod doesn't get reaped

Supporting information:

When creating 100 AnsibleJob's at once, I see that about half of them fail with the timeout from the jobtemplates API endpoint. Also, ~ 50 ansiblejob related containers end up running concurrently (each retries once, so it doubles the container count per ansiblejob).

$ kubectl top pods | wc -l
53

Total CPU Usage: 3485 millicores (m)
Total Memory Usage: 6157 Mi

So the high number of jobs running, eats up the underlying node's memory, to be point that it starts needing to kill of pods.

Instances groups

Feature Overview
Add the ability to create new instance groups and add new instances to a group via the AAP Operator.

Background, and strategic fit
With this ability to create new instance groups and add new instances we can have Ansible scale mesh with remote execution nodes programatically.

Align params for all new CRD's with AWX API

Similar to:

• #111

We should align the names of all of the new CRD's to be consistent with the AWX API schema. This will help us out later on when adding import/export logic.

• jobtemplate (covered in #111)
• ansiblejob
• ansiblecredentials
• ansibleinstancegroups
• ansibleworkflows
• ansibleprojects
• ansibleschedules

[RFE] Add prompt on launch for extra vars and inventory

Add option to CRD for prompt on launch for extra vars and inventory. Required for use with ACM.

How to pass "ssh_key_unlock" in the input when creating a AnsibleCredentials ?

Hi,

I am running the operator in OpenShift 4.16.

Based on https://docs.ansible.com/ansible/latest/collections/awx/awx/credential_module.html#parameter-inputs, I can set in the "inputs" a key like "ssh_key_unlock" when creating a new AnsibleCredentials. How to use the field "Credential Inputs" in the operator ?

If I read

awx-resource-operator/roles/job_runner/tasks/credentials/create_ssh_credential.yml

Line 9 in 270c217

inputs:

, there is no other parameters available (only ssh_key_data and username) ?

Thanks

SA resource-operator-controller-manager-job cannot list resource ansiblejobs

I have installed the AWX Resource Operator in a Minikube cluster in the same namespace as AWX using the command "IMG=awx-resource:dev make docker-build && NAMESPACE=awx make deploy IMG=awx-resource:dev" to build the operator image locally and use it in Minikube. I am able to deploy a JobTemplate to AWX running in the same namespace but when I try to deploy a Project with the manifest below I see an error in the pod logs that is kicked off to deploy this project.

AnsibleProject manifest

---
apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleProject
metadata:
  name: test-project
spec:
  repo: <URL>
  tower_auth_secret: awx-local
  connection_secret: awx-local
  scm_type: Git
  organization: Default
  name: Test project via resource operator

Pod logs

[WARNING]: Unable to parse /runner/inventory as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [job_runner : Read AnsibleJob Specs] **************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     raise ApiException(http_resp=r)
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 55, in inner\n    resp = func(self, *args, **kwargs)\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 270, in request\n    api_response = self.client.call_api(\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 348, in call_api\n    return self.__call_api(resource_path, method,\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 180, in __call_api\n    response_data = self.request(\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 373, in request\n    return self.rest_client.GET(url,\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py\", line 240, in GET\n    return self.request(\"GET\", url,\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py\", line 234, in request\n    raise ApiException(http_resp=r)\nkubernetes.client.exceptions.ApiException: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'e5b04f58-8fd4-4a2d-bf70-ddd8b214065b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2855c0c0-1dea-47fc-80fb-009b97a84615', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'de09d317-c587-427c-8753-22a05d788475', 'Date': 'Fri, 21 Jul 2023 11:17:10 GMT', 'Content-Length': '387'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"ansiblejobs.tower.ansible.com is forbidden: User \\\\\"system:serviceaccount:awx:resource-operator-controller-manager-job\\\\\" cannot list resource \\\\\"ansiblejobs\\\\\" in API group \\\\\"tower.ansible.com\\\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"group\":\"tower.ansible.com\",\"kind\":\"ansiblejobs\"},\"code\":403}\\n'\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1689938228.168541-58-277705374183620/AnsiballZ_k8s_info.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1689938228.168541-58-277705374183620/AnsiballZ_k8s_info.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1689938228.168541-58-277705374183620/AnsiballZ_k8s_info.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_k8s_info_payload_7g4ywld1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 206, in <module>\n  File \"/tmp/ansible_k8s_info_payload_7g4ywld1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 202, in main\n  File \"/tmp/ansible_k8s_info_payload_7g4ywld1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 173, in execute_module\n  File \"/tmp/ansible_k8s_info_payload_7g4ywld1/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py\", line 324, in kubernetes_facts\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 112, in get\n    return self.request('get', path, **kwargs)\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 57, in inner\n    raise api_exception(e)\nkubernetes.dynamic.exceptions.ForbiddenError: 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'e5b04f58-8fd4-4a2d-bf70-ddd8b214065b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '2855c0c0-1dea-47fc-80fb-009b97a84615', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'de09d317-c587-427c-8753-22a05d788475', 'Date': 'Fri, 21 Jul 2023 11:17:10 GMT', 'Content-Length': '387'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"ansiblejobs.tower.ansible.com is forbidden: User \\\\\"system:serviceaccount:awx:resource-operator-controller-manager-job\\\\\" cannot list resource \\\\\"ansiblejobs\\\\\" in API group \\\\\"tower.ansible.com\\\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"group\":\"tower.ansible.com\",\"kind\":\"ansiblejobs\"},\"code\":403}\\n'\nOriginal traceback: \n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 55, in inner\n    resp = func(self, *args, **kwargs)\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py\", line 270, in request\n    api_response = self.client.call_api(\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 348, in call_api\n    return self.__call_api(resource_path, method,\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 180, in __call_api\n    response_data = self.request(\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py\", line 373, in request\n    return self.rest_client.GET(url,\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py\", line 240, in GET\n    return self.request(\"GET\", url,\n\n  File \"/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py\", line 234, in request\n    raise ApiException(http_resp=r)\n\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[REQUEST] Build operator bundle and catalogs for community-catalog

If I'm not mistaken, the awx-operator uses Github Actions to build and push the awx-operator-bundle and awx-operator-catalog images to quay.io/ansible namespace, and sends a PR to https://github.com/k8s-operatorhub/community-operators/tree/main/operators to add the manifests for the operator for each new release.

I would like the awx-resource-operator to build the bundle and catalog images for itself, as well as send automated PR's to the community catalog more or less in the same way as the awx-operator.

AnsibleCredentialsResult in awx-resource-operator/roles/job_runner/tasks /create_project.yml

awx-resource-operator/roles/job_runner/tasks/create_project.yml

Line 43 in f43c2c0

AnsibleCredentialsResult:

I believe 'AnsibleCredentialsResult' above should be 'AnsibleProjectResult' as it is at:

awx-resource-operator/roles/job_runner/tasks/create_project.yml

Line 56 in f43c2c0

AnsibleProjectResult:

JobTemplate is created appears to invoke the Job by default

Reported by @mdelder It seems like when the Job Template is created, a job is launch by default.
We most likely want that default to be turned off and configurable by a spec value.

Create operator test using Kind and AWX

As a resource operator user I would like to be able to test changes to the operator using kind and the YAMLs defined in the repository.

Test automation

We need some tests for this repo. @matburt @maxamillion any suggestions?

I am thinking we can use:
https://github.com/kubernetes-sigs/kind
https://github.com/ansible/awx
using a script/ansible/gocode

1. launch a kind local cluster
2. deploy awx on it (not sure awx can be deployed to kind) maybe part of the requirement for the test is to have a cluster
3. deploy this operator to the cluster
4. run tests

Generate credentials

As an awx-resource user I would like to generate credentials so that I can use them in my jobs and plays.

How to run jobs from the operator

Recently I deployed Ansible AWX on my Openshift cluster, and it works fine, and I also deployed the Ansible automation platform resource operator from the OperatorHub, Then I used the Ansible JobTemplate API instance, where I mentioned all the informations of the JobTemplate i have already on my AWX.

I want to know how to launch the JobTemplate from the Operator, can anyone help me how it works?!!

Thanks in advance!

This is my JobTemplate yaml.

patchesStrategicMerge in kustomization.yaml is deprecated

patchesStrategicMerge in kustomization.yaml is deprecated.

Using kustomize 5.x causes warning message: Warning: 'patchesStrategicMerge' is deprecated.

$ which kustomize
~/bin/kustomize

$ kustomize version
v5.3.0

$ NAMESPACE=awx IMG=quay.io/ansible/awx-resource-operator:latest make deploy
cd config/manager && /home/kuro/bin/kustomize edit set image controller=quay.io/ansible/awx-resource-operator:latest
# Add a way to inject the runner image
cd config/default && /home/kuro/bin/kustomize edit set namespace awx
/home/kuro/bin/kustomize build config/default | kubectl apply -f -
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/awx unchanged
customresourcedefinition.apiextensions.k8s.io/ansiblecredentials.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansibleinstancegroups.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansibleinventories.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansiblejobs.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansibleprojects.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansibleschedules.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/ansibleworkflows.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/jobtemplates.tower.ansible.com created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.tower.ansible.com created
serviceaccount/resource-operator-controller-manager created
role.rbac.authorization.k8s.io/resource-operator-awx-resource-manager-role created
role.rbac.authorization.k8s.io/resource-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/resource-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/resource-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/resource-operator-awx-resource-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/resource-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/resource-operator-proxy-rolebinding created
configmap/resource-operator-awx-resource-manager-config created
service/resource-operator-controller-manager-metrics-service created
deployment.apps/resource-operator-controller-manager created

Add support for creating Execution Environments

As a user of this operator, I need to be able to define Execution Environments, so that my custom EEs can be declared and later consumed by JobTemplates, etc.

Job templates should run within a kubernetes job

If job templates are launched from a controller and they are incorrect they will be launched in an infinite loop.

How to test.

• Modify the sample job to point to a non-existent project and inventory
• launch job template
• follow controller log

Result.

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Create Job Template] ***************************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21

-------------------------------------------------------------------------------
{"level":"info","ts":1667940073.9394093,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"1106887023490022726","EventData.Name":"jobtemplate : Create Job Template"}
{"level":"error","ts":1667940081.9289565,"logger":"logging_event_handler","msg":"","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"runner_on_failed","job":"1106887023490022726","EventData.Task":"Create Job Template","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/jobtemplate/tasks/main.yml:21","error":"[playbook task failed]"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Create Job Template] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1", "query": {"name": "Demo Inventory243234243"}, "response": {"json": {"count": 0, "next": null, "previous": null, "results": []}, "status_code": 200}, "total_results": 0}

-------------------------------------------------------------------------------
{"level":"error","ts":1667940082.4251635,"logger":"runner","msg":"ansible-playbook 2.9.27\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'actionable', as we already have a stdout callback.\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'counter_enabled', as we already have a stdout callback.\nSkipping callback 'debug', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'full_skip', as we already have a stdout callback.\nSkipping callback 'json', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping callback 'null', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\nSkipping callback 'selective', as we already have a stdout callback.\nSkipping callback 'skippy', as we already have a stdout callback.\nSkipping callback 'stderr', as we already have a stdout callback.\nSkipping callback 'unixy', as we already have a stdout callback.\nSkipping callback 'yaml', as we already have a stdout callback.\n\r\nPLAYBOOK: 5875fcdc1e4240258338761c61d70c25 *************************************\n1 plays in /tmp/ansible-operator/runner/tower.ansible.com/v1alpha1/JobTemplate/ansible-automation-platform/jobtemplate-sample/project/5875fcdc1e4240258338761c61d70c25\n\r\nPLAY [localhost] ***************************************************************\nMETA: ran handlers\n\r\nTASK [jobtemplate : Read Secret Configuration] *********************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:3\nok: [localhost] => {\"api_found\": true, \"changed\": false, \"resources\": [{\"apiVersion\": \"v1\", \"data\": {\"host\": \"aHR0cHM6Ly9teS1hdXRvbWF0aW9uLWNvbnRyb2xsZXItYW5zaWJsZS1hdXRvbWF0aW9uLXBsYXRmb3JtLmFwcHMucm9nZXJvY3AxLmRlbW9yZWRoYXQuY29tLw==\", \"token\": \"ZFJJQ25xWG91QkpjWWFGaXE2TDNnMWF5TWtrYW0w\"}, \"kind\": \"Secret\", \"metadata\": {\"creationTimestamp\": \"2022-11-08T17:03:21Z\", \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:data\": {\".\": {}, \"f:host\": {}, \"f:token\": {}}, \"f:type\": {}}, \"manager\": \"kubectl-create\", \"operation\": \"Update\", \"time\": \"2022-11-08T17:03:21Z\"}], \"name\": \"controller-access-secret\", \"namespace\": \"ansible-automation-platform\", \"resourceVersion\": \"624\", \"uid\": \"336b16cf-13da-44f8-8f8b-c1cb176fa8fd\"}, \"type\": \"Opaque\"}]}\n\r\nTASK [jobtemplate : Validate Secret Exists] ************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:10\nok: [localhost] => {\r\n    \"changed\": false,\r\n    \"msg\": \"All assertions passed\"\r\n}\n\r\nTASK [jobtemplate : Show secret details] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:16\nok: [localhost] => {\"censored\": \"the output has been hidden due to the fact that 'no_log: true' was specified for this result\"}\n\r\nTASK [jobtemplate : Create Job Template] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21\n[WARNING]: You are using the awx version of this collection but connecting to\r\nRed Hat Ansible Automation Platform\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1\", \"query\": {\"name\": \"Demo Inventory243234243\"}, \"response\": {\"json\": {\"count\": 0, \"next\": null, \"previous\": null, \"results\": []}, \"status_code\": 200}, \"total_results\": 0}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   \r\n\n","job":"1106887023490022726","name":"jobtemplate-sample","namespace":"ansible-automation-platform","error":"exit status 2"}

----- Ansible Task Status Event StdOut (tower.ansible.com/v1alpha1, Kind=JobTemplate, jobtemplate-sample/ansible-automation-platform) -----


PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   


----------
{"level":"error","ts":1667940082.437448,"msg":"Reconciler error","controller":"jobtemplate-controller","object":{"name":"jobtemplate-sample","namespace":"ansible-automation-platform"},"namespace":"ansible-automation-platform","name":"jobtemplate-sample","reconcileID":"4abccd23-7f77-4161-9c2b-1fd2f8e52f1f","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":1667940088.1294959,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5277922004219225884","EventData.Name":"jobtemplate : Read Secret Configuration"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Read Secret Configuration] *********************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:3

-------------------------------------------------------------------------------
{"level":"info","ts":1667940090.8530579,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/ansible-automation-platform/secrets/controller-access-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"ansible-automation-platform","Resource":"secrets","Subresource":"","Name":"controller-access-secret","Parts":["secrets","controller-access-secret"]}}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Validate Secret Exists] ************************************
{"level":"info","ts":1667940091.2236872,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5277922004219225884","EventData.Name":"jobtemplate : Validate Secret Exists"}
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:10

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Show secret details] ******************************** 
ok: [localhost] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

-------------------------------------------------------------------------------
{"level":"info","ts":1667940091.9161265,"logger":"logging_event_handler","msg":"[playbook debug]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"runner_on_ok","job":"5277922004219225884","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Create Job Template] ***************************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21

-------------------------------------------------------------------------------
{"level":"info","ts":1667940091.9321482,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5277922004219225884","EventData.Name":"jobtemplate : Create Job Template"}

--------------------------- Ansible Task StdOut -------------------------------

{"level":"error","ts":1667940094.4479342,"logger":"logging_event_handler","msg":"","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"runner_on_failed","job":"5277922004219225884","EventData.Task":"Create Job Template","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/jobtemplate/tasks/main.yml:21","error":"[playbook task failed]"}
 TASK [Create Job Template] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1", "query": {"name": "Demo Inventory243234243"}, "response": {"json": {"count": 0, "next": null, "previous": null, "results": []}, "status_code": 200}, "total_results": 0}

-------------------------------------------------------------------------------

----- Ansible Task Status Event StdOut (tower.ansible.com/v1alpha1, Kind=JobTemplate, jobtemplate-sample/ansible-automation-platform) -----


PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   


----------
{"level":"error","ts":1667940095.0208952,"logger":"runner","msg":"ansible-playbook 2.9.27\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'actionable', as we already have a stdout callback.\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'counter_enabled', as we already have a stdout callback.\nSkipping callback 'debug', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'full_skip', as we already have a stdout callback.\nSkipping callback 'json', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping callback 'null', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\nSkipping callback 'selective', as we already have a stdout callback.\nSkipping callback 'skippy', as we already have a stdout callback.\nSkipping callback 'stderr', as we already have a stdout callback.\nSkipping callback 'unixy', as we already have a stdout callback.\nSkipping callback 'yaml', as we already have a stdout callback.\n\r\nPLAYBOOK: 370c24877df14647a5ea13aefd33e2fd *************************************\n1 plays in /tmp/ansible-operator/runner/tower.ansible.com/v1alpha1/JobTemplate/ansible-automation-platform/jobtemplate-sample/project/370c24877df14647a5ea13aefd33e2fd\n\r\nPLAY [localhost] ***************************************************************\nMETA: ran handlers\n\r\nTASK [jobtemplate : Read Secret Configuration] *********************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:3\nok: [localhost] => {\"api_found\": true, \"changed\": false, \"resources\": [{\"apiVersion\": \"v1\", \"data\": {\"host\": \"aHR0cHM6Ly9teS1hdXRvbWF0aW9uLWNvbnRyb2xsZXItYW5zaWJsZS1hdXRvbWF0aW9uLXBsYXRmb3JtLmFwcHMucm9nZXJvY3AxLmRlbW9yZWRoYXQuY29tLw==\", \"token\": \"ZFJJQ25xWG91QkpjWWFGaXE2TDNnMWF5TWtrYW0w\"}, \"kind\": \"Secret\", \"metadata\": {\"creationTimestamp\": \"2022-11-08T17:03:21Z\", \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:data\": {\".\": {}, \"f:host\": {}, \"f:token\": {}}, \"f:type\": {}}, \"manager\": \"kubectl-create\", \"operation\": \"Update\", \"time\": \"2022-11-08T17:03:21Z\"}], \"name\": \"controller-access-secret\", \"namespace\": \"ansible-automation-platform\", \"resourceVersion\": \"624\", \"uid\": \"336b16cf-13da-44f8-8f8b-c1cb176fa8fd\"}, \"type\": \"Opaque\"}]}\n\r\nTASK [jobtemplate : Validate Secret Exists] ************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:10\nok: [localhost] => {\r\n    \"changed\": false,\r\n    \"msg\": \"All assertions passed\"\r\n}\n\r\nTASK [jobtemplate : Show secret details] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:16\nok: [localhost] => {\"censored\": \"the output has been hidden due to the fact that 'no_log: true' was specified for this result\"}\n\r\nTASK [jobtemplate : Create Job Template] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21\n[WARNING]: You are using the awx version of this collection but connecting to\r\nRed Hat Ansible Automation Platform\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1\", \"query\": {\"name\": \"Demo Inventory243234243\"}, \"response\": {\"json\": {\"count\": 0, \"next\": null, \"previous\": null, \"results\": []}, \"status_code\": 200}, \"total_results\": 0}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   \r\n\n","job":"5277922004219225884","name":"jobtemplate-sample","namespace":"ansible-automation-platform","error":"exit status 2"}
{"level":"error","ts":1667940095.0394337,"msg":"Reconciler error","controller":"jobtemplate-controller","object":{"name":"jobtemplate-sample","namespace":"ansible-automation-platform"},"namespace":"ansible-automation-platform","name":"jobtemplate-sample","reconcileID":"b4597bdd-563a-4f83-a973-a78c827028cc","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":1667940103.224124,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5409160531841725505","EventData.Name":"jobtemplate : Read Secret Configuration"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Read Secret Configuration] *********************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:3

-------------------------------------------------------------------------------
{"level":"info","ts":1667940106.0554104,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/ansible-automation-platform/secrets/controller-access-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"ansible-automation-platform","Resource":"secrets","Subresource":"","Name":"controller-access-secret","Parts":["secrets","controller-access-secret"]}}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Validate Secret Exists] ************************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:10

-------------------------------------------------------------------------------
{"level":"info","ts":1667940106.5177867,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5409160531841725505","EventData.Name":"jobtemplate : Validate Secret Exists"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Show secret details] ******************************** 
ok: [localhost] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

-------------------------------------------------------------------------------
{"level":"info","ts":1667940107.1472943,"logger":"logging_event_handler","msg":"[playbook debug]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"runner_on_ok","job":"5409160531841725505","EventData.TaskArgs":""}
{"level":"info","ts":1667940107.2120945,"logger":"logging_event_handler","msg":"[playbook task start]","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"playbook_on_task_start","job":"5409160531841725505","EventData.Name":"jobtemplate : Create Job Template"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [jobtemplate : Create Job Template] ***************************************
task path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Create Job Template] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1", "query": {"name": "Demo Inventory243234243"}, "response": {"json": {"count": 0, "next": null, "previous": null, "results": []}, "status_code": 200}, "total_results": 0}

-------------------------------------------------------------------------------
{"level":"error","ts":1667940110.1384823,"logger":"logging_event_handler","msg":"","name":"jobtemplate-sample","namespace":"ansible-automation-platform","gvk":"tower.ansible.com/v1alpha1, Kind=JobTemplate","event_type":"runner_on_failed","job":"5409160531841725505","EventData.Task":"Create Job Template","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/jobtemplate/tasks/main.yml:21","error":"[playbook task failed]"}
{"level":"error","ts":1667940110.617833,"logger":"runner","msg":"ansible-playbook 2.9.27\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'actionable', as we already have a stdout callback.\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'counter_enabled', as we already have a stdout callback.\nSkipping callback 'debug', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'full_skip', as we already have a stdout callback.\nSkipping callback 'json', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping callback 'null', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\nSkipping callback 'selective', as we already have a stdout callback.\nSkipping callback 'skippy', as we already have a stdout callback.\nSkipping callback 'stderr', as we already have a stdout callback.\nSkipping callback 'unixy', as we already have a stdout callback.\nSkipping callback 'yaml', as we already have a stdout callback.\n\r\nPLAYBOOK: 406ad59f6b284e5a8c307679b8ac8bd1 *************************************\n1 plays in /tmp/ansible-operator/runner/tower.ansible.com/v1alpha1/JobTemplate/ansible-automation-platform/jobtemplate-sample/project/406ad59f6b284e5a8c307679b8ac8bd1\n\r\nPLAY [localhost] ***************************************************************\nMETA: ran handlers\n\r\nTASK [jobtemplate : Read Secret Configuration] *********************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:3\nok: [localhost] => {\"api_found\": true, \"changed\": false, \"resources\": [{\"apiVersion\": \"v1\", \"data\": {\"host\": \"aHR0cHM6Ly9teS1hdXRvbWF0aW9uLWNvbnRyb2xsZXItYW5zaWJsZS1hdXRvbWF0aW9uLXBsYXRmb3JtLmFwcHMucm9nZXJvY3AxLmRlbW9yZWRoYXQuY29tLw==\", \"token\": \"ZFJJQ25xWG91QkpjWWFGaXE2TDNnMWF5TWtrYW0w\"}, \"kind\": \"Secret\", \"metadata\": {\"creationTimestamp\": \"2022-11-08T17:03:21Z\", \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:data\": {\".\": {}, \"f:host\": {}, \"f:token\": {}}, \"f:type\": {}}, \"manager\": \"kubectl-create\", \"operation\": \"Update\", \"time\": \"2022-11-08T17:03:21Z\"}], \"name\": \"controller-access-secret\", \"namespace\": \"ansible-automation-platform\", \"resourceVersion\": \"624\", \"uid\": \"336b16cf-13da-44f8-8f8b-c1cb176fa8fd\"}, \"type\": \"Opaque\"}]}\n\r\nTASK [jobtemplate : Validate Secret Exists] ************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:10\nok: [localhost] => {\r\n    \"changed\": false,\r\n    \"msg\": \"All assertions passed\"\r\n}\n\r\nTASK [jobtemplate : Show secret details] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:16\nok: [localhost] => {\"censored\": \"the output has been hidden due to the fact that 'no_log: true' was specified for this result\"}\n\r\nTASK [jobtemplate : Create Job Template] ***************************************\r\ntask path: /opt/ansible/roles/jobtemplate/tasks/main.yml:21\n[WARNING]: You are using the awx version of this collection but connecting to\r\nRed Hat Ansible Automation Platform\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Request to api/v2/inventories/?name=Demo+Inventory243234243 returned 0 items, expected 1\", \"query\": {\"name\": \"Demo Inventory243234243\"}, \"response\": {\"json\": {\"count\": 0, \"next\": null, \"previous\": null, \"results\": []}, \"status_code\": 200}, \"total_results\": 0}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   \r\n\n","job":"5409160531841725505","name":"jobtemplate-sample","namespace":"ansible-automation-platform","error":"exit status 2"}

----- Ansible Task Status Event StdOut (tower.ansible.com/v1alpha1, Kind=JobTemplate, jobtemplate-sample/ansible-automation-platform) -----


PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Ad-hoc calls of tower modules

Possibility to define CRDs that call ansible modules.

User Stories:

• Ad-hoc execution of Tower Modules against Tower Cluster(s).
• Configure Tower Cluster(s) as per CRs entries.

Add README.md

Hi,

i am missing an README.md in this repo which explains what the awx-resource-operarator does and how you can run it.
Personally i don't have any clue what this repo does at this moment.

InstanceGroup Requires "OpenShift or Kubernetes credential"

InstanceGroup CRD sets credential_name as required even tho it is not required when specifying the InstanceGroup.
I propose we remove the credential_name from required.

Instance Group CRD required

Moreover, there should be a fieldDependency set in the ClusterServiceVersion, for the InstanceGroup:
For example, the credential_name only exists for containerGroups:

     - displayName: Container Group
        path: is_container_group
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:booleanSwitch
      - displayName: OpenShift or Kubernetes credential
        path: credential_name
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:text
        - urn:alm:descriptor:com.tectonic.ui:fieldDependency:is_container_group:true

While "policy_instance_minimum" and "policy_instance_percentage" only exists for instanceGroups (and not containerGroups):

    - displayName: Policy Instance Percentage
       path: policy_instance_percentage
       x-descriptors:
       - urn:alm:descriptor:com.tectonic.ui:int
       - urn:alm:descriptor:com.tectonic.ui:fieldDependency:is_container_group:false
     - displayName: Policy Instance Minimum
       path: policy_instance_minimum
       x-descriptors:
       - urn:alm:descriptor:com.tectonic.ui:int
       - urn:alm:descriptor:com.tectonic.ui:fieldDependency:is_container_group:false

Status of Ansible job and Ansible Job template condition always stays at a running state.

Description

When creating a Ansible Job from the AWX Operator, when the job gets created and finishes the status reported within the Ansible Job continues to always show state 'Running'

Steps to Reproduce

Create an Ansible Job
Once job completes, verify state and it will show running.

Actual Behavior

shows state Running

Expected Behavior

Expecting correct status like Failed or Complete when done.

Add option to run AWX Workflows

We currently mainly use Workflows composed of multiple jobs in AWX, but there isn't an option to run them in a CR

Allow for Projects to be generated

As a AAP OpenShift user, I would like to be able to create and manage projects from the AAP operartor.

[Epic] Create and publish AWX resource operator docs

Feature Summary

This epic is to use the current docs and readme to create and publish AWX Resource Operator documentation to Read The Docs.

• #120
• #121
• Edit and restructure the docs folder as needed to make a coherent set of guides
• Publish docs to readthedocs docsite

@samccann is running a mini project on the #docs matrix channel on Thursdays to get community help on this effort. We'll use this issue and the sub-issues to coordinate work etc.

Additional Information

Related Epic for awx-operator here:

• ansible/awx-operator#1356

Example for creating a Container Registry Credential

Do you have an example of creating a "Container Registry" credential defined as YAML manifest?
The examples do not cover my approach.

Secret shown in job yaml

When requesting the YAML from the generated job the value of the secret is shown rather than referencing a secret.

Deprecate the tower_auth_secret in favor of a new connection_secret

We should rename the tower_auth_secret to connection_secret. In order to not break existing deployments though, tower_auth_secret should be deprecated, but logic to conditionally use either should be added.

It should preferentially use connection_secret if provided.

Example connection secret does not work as expected

In README.md an example is given for running an ansible job as follows.

apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  generateName: demo-job-1 # generate a unique suffix per 'kubectl create'
spec:
  connection_secret: awx-access
  job_template_name: Demo Job Template

When I tested this it did not work, instead producing the following error:

apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  creationTimestamp: "2023-10-28T23:16:21Z"
  generateName: demo-job-1
  generation: 1
  name: demo-job-1b7n5p
  namespace: awx
  resourceVersion: "204252"
  uid: 07cd628b-bae5-46ae-8089-5f3caafc6b65
spec:
  connection_secret: awx-access
  job_template_name: Demo Job Template
status:
  conditions:
  - lastTransitionTime: "2023-10-28T23:16:23Z"
    message: ""
    reason: ""
    status: "False"
    type: Successful
  - ansibleResult:
      changed: 0
      completion: 2023-10-28T23:16:36.938598
      failures: 1
      ok: 3
      skipped: 5
    lastTransitionTime: "2023-10-28T23:16:37Z"
    message: |
      The task includes an option with an undefined variable. The error was: 'tower_auth_secret' is undefined

      The error appears to be in '/opt/ansible/roles/job/tasks/main.yml': line 77, column 3, but may
      be elsewhere in the file depending on the exact syntax problem.

      The offending line appears to be:


      - name: Read Secret Configuration
        ^ here
    reason: Failed
    status: "False"
    type: Failure
  - lastTransitionTime: "2023-10-28T23:16:37Z"
    message: Running reconciliation
    reason: Running
    status: "True"
    type: Running

Investigating this it appears as though #122 may have introduced a bug.

When I updated my job specification to either of the following they both worked.

apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  generateName: demo-job-1 # generate a unique suffix per 'kubectl create'
spec:
  connection_secret: awx-access
  tower_auth_secret: awx-access
  job_template_name: Demo Job Template

apiVersion: tower.ansible.com/v1alpha1
kind: AnsibleJob
metadata:
  generateName: demo-job-1 # generate a unique suffix per 'kubectl create'
spec:
  tower_auth_secret: awx-access
  job_template_name: Demo Job Template

So it seems tower_auth_secret is still required despite being deprecated. I am happy to raise a PR if someone can provide some gentle guidance on where the change needs to be done.

Job&Pods created by the Resource Operator from an AnsibleJob CR need to be able to specify extra volume and envs mounts

There are two different ways that the AAP2 Controller instance treats spawned Ansible Jobs - the properly configured way via Job Template execution from the AAP2 Controller API/WebUI, and the incomplete way that is done as part of this template: https://github.com/ansible/awx-resource-operator/blob/acdf5bb1a1994fb534418bcc9fe2fd9b629a8c07/roles/job/templates/job_definition.yml.j2

Use Case - Adding the Cluster Root CA Bundle and Proxy Configuration to the AnsibleJob started Job>Pods

For example, in the AAP2 Controller Web UI, in the Settings > Job Settings page, you can enable "Expose host paths for Container Groups" and set "Paths to expose to isolated jobs" to ["/etc/pki/ca-trust:/etc/pki/ca-trust:O"] to pass the Trusted Root CA bundles from the hosts (as set by the Proxy config) into the ephemeral Job container that runs in the namespace the operator is installed into. Alternatively, you can set that path via a custom Pod specification as a volumeMount in the default Instance Group - this works splendidly.

Challenge

However, whenever an AnsibleJob CR is created on the cluster, the AAP2 Operator's Resource Operator Controller does not consume these values. In fact, the template of the Job definition, and thus Pods created by the Resource Operator Controller, starts Ansible Jobs not with any defined Execution Environment, and does not consume the same settings as set in the AAP2 Controller Job Settings.

Workaround

To work around this issue, you need to build a custom container based off the registry.redhat.io/ansible-automation-platform-22/platform-resource-runner-rhel8 image, add your CA root certificates to it, RUN an update-ca-trust, build/push the image to a registry, then use that for the .spec.runner_image + .spec.runner_version in the AnsibleJob CR.

It looks like the Job Runner Role Task uses the awx.awx.tower_job_launch module which is actually a bit limited in what it passes to the instantiated Job so the module would need to be updated, or the Resource Operator needs to be able to compensate for additional container specifications inherited from the cluster/settings.

Potential Solutions

What if the Resource Operator consumed the Root CA Certificate ConfigMap as many other RH Operators do, and then apply to templated Job CRs? The process would be something similar to the following:

• Query for .spec.trustedCA.name, if set then add a ConfigMap with the config.openshift.io/inject-trusted-cabundle: 'true' 'metadata.label
• Attach that ConfigMap to the Job CR templated by the Job Role in the Resource Operator
• Repeat for the other configuration such as Proxy > Container Envs

Alternatively, the Resource Operator could query for the AAP2 Controller Job Configuration Settings and apply those alternatively - or another configuration area could be presented for Runner Configuration Settings.

Thoughts?

Add the ability to define schedules

Schedules are an important part of specific AAP jobs. It would be nice to be able to define those items with the resource operator

Updating a Job Template Resource does not take the changes

Currently if I created a Job Template as such on the first attempt this would create as expected.

apiVersion: tower.ansible.com/v1alpha1
kind: JobTemplate
metadata:
  name: rocketchat-volsync-template
  namespace: aap
spec:
  job_template_inventory: ACM Cluster Inventory
  job_template_name: RocketChatVolSync
  job_template_playbook: volsync/playbook.yml
  job_template_project: VolSync Demo
  tower_auth_secret: aap2-cred
  job_template_credentials:
  - "ACM Hub"

If I wanted to make a change and say remove the job_template_credentials from the Job Template if I went ahead removed this piece and gave another unique metadata.name , when you apply, you don't see the changes to the Job Template where the template credential is removed. It's exactly the same as the initial creation.

apiVersion: tower.ansible.com/v1alpha1
kind: JobTemplate
metadata:
  name: rocketchat-volsync-template
  namespace: aap
spec:
  job_template_inventory: ACM Cluster Inventory
  job_template_name: RocketChatVolSync
  job_template_playbook: volsync/playbook.yml
  job_template_project: VolSync Demo
  tower_auth_secret: aap2-cred

IAAC-Branch: Configure project type to work with more assets besides GIT

Specifically for Project management I would need to work with different SCM types.

Add Support For Inventory Kind, Variables in Ansible Inventories

Ansible Inventory CRD only allows defining:

• connection_secret
• copy_from
• description
• name
• organization
• state

As of now we cannot create an inventory that contain inventory kind and variables.
I propose we add properties for all those properties to the CRD.

Resource Operator or Ansible awx module

Currently I bootstrap awx on top of Azure AKS with LDAP integration at the beginning. Since we adopt gitops, everything must/should be checked in code including awx setting (like LDAP, proxy, ...) and awx resources.

In a perfect world, awx and its core resources: credential/inventory/project/template are able to be provisioned via manifest files at boot and keep updating with SCM. As your work on this repo just covered template and job only for now, do you have any plan for other resources? It is possible to re-use Ansible awx module? Eg:

    - name: Create a valid SCM credential 
      awx.awx.credential:
        name: my_git
        organization: '{{ org }}'
        state: present
        credential_type: Source Control
        inputs:
          username: '{{ git_user }}'
          password: '{{ git_token }}'

    - name: Add DRE project
      awx.awx.project:
        name: my_project
        description: Set of Ansible playbooks developed by XYZ
        organization: '{{ org }}'
        credential: '{{ git_credential }}'
        scm_type: git
        scm_branch: main
        scm_update_on_launch: yes
        scm_url: https://�myGitServer/abc/xyz
        validate_certs: no
        state: present
        wait: no

I'll be crazy happy if these resource can be created with operator rather than running it from my local as an AWX seed playbook.

Thank you,

suggestion: get inspiration at redhat CoP

Please have a look here https://github.com/redhat-cop/infra-ansible/tree/main/roles/ansible/tower
In my opinion, any manage-* and potentially some config-* roles too could become a CRD on itself.
I also created an issue there ansible/awx-operator#700 prior I discover the install and config were decoupled.

Add support for Surveys to JobTemplates

I think we should support adding Surveys to JobTemplates

Here are the related parameters from the awx.awx.job_template module

• https://docs.ansible.com/ansible/latest/collections/awx/awx/job_template_module.html#parameter-survey_enabled
• https://docs.ansible.com/ansible/latest/collections/awx/awx/job_template_module.html#parameter-survey_enabled

Here's an example of what the YAML from RHCOP's controller_configuration looks like:

    survey_enabled: True
    survey_spec: {'name': '', 'description': '', 'spec': [{'question_name': 'Which bastion host?', 'question_description': '', 'required': True, 'type': 'multiplechoice', 'variable': 'remote_host', 'min': 0, 'max': 1024, 'default': 'server.example.com', 'choices': ['server.example.com'], 'new_question': True}, {'question_name': 'Which user on the bastion host?', 'question_description': "This user's home directory will be searched for the ./secrets directories and files", 'required': True, 'type': 'multiplechoice', 'variable': 'username', 'min': 0, 'max': 1024, 'new_question': False, 'default': '', 'choices': ['user1', 'user2', 'user3', 'user4', 'user5']}, {'question_name': 'Which repository?', 'question_description': 'This repository folder in ~/vscode/ will be searched for ./secrets directories and files', 'required': True, 'type': 'multiplechoice', 'variable': 'repo_name', 'min': 0, 'max': 1024, 'new_question': False, 'default': 'repo1', 'choices': ['repo1']}]}

survey_spec is really messy, so maybe we can make the fields look nicer for users like this:

kind: JobTemplate
...
spec:
...
  survey_enabled: true
  survey_spec:
    questions:
      - name: Which thing do you want?
        description: Pick your favorite thing
        required: true
        type: multiplechoice
        choices:
          - thing1
          - thing2
          - thing3
        default: thing1
        variable: thing_var
      - name: Where do you want it?
        description: Pick your favorite place
        required: true
        type: multiplechoice
        choices:
          - place1
          - place2
          - place3
        default: place1
        variable: place_var
      - name: What is your favorite color?
        description: This one is optional
        required: false
        type: text
        variable: color_var

The TowerAPI seems to treat survey_spec as a separate thing that gets related to job_templates. We could create a new JobTemplateSurveySpec API, but it would probably be harder to implement this because it would introduce dependencies on the JobTemplate API and whatever objects may or may not already be defined on a cluster. 🤔

Does anyone have any thoughts or opinions on this?

add an optional job wait timeout field to AnsibleJob CR spec

In the job runner execution, it calls the tower_ job_wait module but currently there is no timeout value given so it might wait indefinitely. For more information: https://docs.ansible.com/ansible/latest/collections/awx/awx/tower_job_wait_module.html#parameter-timeout

I propose we add a new optional field to AnsibleJob CR spec that is int type with the name job_wait_timeout. If this value is given, then it will be passed to the tower_ job_wait module as the timeout value and the AnsibleJob will timeout and update its status accordingly.