AWX Deployment on RKE2 fails on k8s_info

[rmastersbun]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

When deploying the awx-operator on RKE2, the deployment of the operator succeeds but then loops with the following:
2024-10-15T16:17:26.004798609+08:00 TASK [Verify imagePullSecrets] ************************************************* 2024-10-15T16:17:26.004803763+08:00 task path: /opt/ansible/playbooks/awx.yml:10 2024-10-15T16:17:26.004808093+08:00 2024-10-15T16:17:26.004821573+08:00 ------------------------------------------------------------------------------- 2024-10-15T16:17:26.260766825+08:00 {"level":"error","ts":"2024-10-15T08:17:26Z","logger":"logging_event_handler","msg":"","name":"awxtest15","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"7862226185777618437","EventData.Task":"Verify imagePullSecrets","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/playbooks/awx.yml:10","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"} 2024-10-15T16:17:26.260864035+08:00 2024-10-15T16:17:26.260898398+08:00 --------------------------- Ansible Task StdOut ------------------------------- 2024-10-15T16:17:26.260902750+08:00 2024-10-15T16:17:26.260907455+08:00 TASK [Verify imagePullSecrets] ******************************** 2024-10-15T16:17:26.260919623+08:00 fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2024-10-15T16:17:26.260935467+08:00

AWX Operator version

2.19.1

AWX version

v24.6.1

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

RKE2 Kubernetes Version: v1.24.16 +rke2r1

Modifications

no

Steps to reproduce

I am attempting to deploy AWX via the operator using a minimal values configuration.

Start with a bare RKE2 cluster, and follow the steps in https://ansible-community.github.io/awx-operator-helm/ using the following values:

AWX: enabled: true name: awxtest15 postgres: dbName: Unset enabled: false host: Unset password: Unset port: 5678 sslmode: prefer type: unmanaged username: admin spec: admin_user: admin

Expected results

A running AWX instance.

Actual results

The operator container installs and starts, but then loops on
2024-10-15T16:17:26.004798609+08:00 TASK [Verify imagePullSecrets] ************************************************* 2024-10-15T16:17:26.004803763+08:00 task path: /opt/ansible/playbooks/awx.yml:10 2024-10-15T16:17:26.004808093+08:00 2024-10-15T16:17:26.004821573+08:00 ------------------------------------------------------------------------------- 2024-10-15T16:17:26.260766825+08:00 {"level":"error","ts":"2024-10-15T08:17:26Z","logger":"logging_event_handler","msg":"","name":"awxtest15","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"7862226185777618437","EventData.Task":"Verify imagePullSecrets","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/playbooks/awx.yml:10","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"} 2024-10-15T16:17:26.260864035+08:00 2024-10-15T16:17:26.260898398+08:00 --------------------------- Ansible Task StdOut ------------------------------- 2024-10-15T16:17:26.260902750+08:00 2024-10-15T16:17:26.260907455+08:00 TASK [Verify imagePullSecrets] ******************************** 2024-10-15T16:17:26.260919623+08:00 fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2024-10-15T16:17:26.260935467+08:00

Additional information

Running the installer manually within the awx-manager container, setting ansible_python_interpreter: ‘{{ ansible_playbook_python }}’ in vars.yml gives:
`bash-4.4$ ansible-playbook run.yml -e @vars.yml -v
Using /etc/ansible/ansible.cfg as config file
[WARNING]: Found variable using reserved name: no_log

PLAY [localhost] ****************************************************************************************************************************************************************************

TASK [common : Get information about the cluster] *******************************************************************************************************************************************
ok: [localhost] => {"ansible_facts": {"api_groups": ["", "apiregistration.k8s.io", "apps", "events.k8s.io", "authentication.k8s.io", "authorization.k8s.io", "autoscaling", "batch", "certificates.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io", "admissionregistration.k8s.io", "apiextensions.k8s.io", "scheduling.k8s.io", "coordination.k8s.io", "node.k8s.io", "discovery.k8s.io", "flowcontrol.apiserver.k8s.io", "catalog.cattle.io", "crd.projectcalico.org", "helm.cattle.io", "k3s.cattle.io", "operator.tigera.io", "ui.cattle.io", "upgrade.cattle.io", "awx.ansible.com", "cns.vmware.com", "management.cattle.io", "metrics.k8s.io"]}, "changed": false}

TASK [common : Determine the cluster type] **************************************************************************************************************************************************
ok: [localhost] => {"ansible_facts": {"is_k8s": true, "is_openshift": false}, "changed": false}

TASK [common : debug] ***********************************************************************************************************************************************************************
ok: [localhost] => {
"msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

TASK [installer : Check for presence of old awx Deployment] *********************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/bin/sh: line 1: 4158 Killed /usr/bin/python3 /opt/ansible/.ansible/tmp/ansible-tmp-1729214698.536406-4146-47837280803179/AnsiballZ_k8s_info.py\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 137}

PLAY RECAP **********************************************************************************************************************************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

bash-4.4$`

Using -v -v -v gives this at the end:
TASK [installer : Check for presence of old awx Deployment] ********************************************************************************************************************************* task path: /opt/ansible/roles/installer/tasks/main.yml:2 <localhost> ESTABLISH LOCAL CONNECTION FOR USER: ansible <localhost> EXEC /bin/sh -c 'echo ~ansible && sleep 0' <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p " echo /opt/ansible/.ansible/tmp "&& mkdir " echo /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398 " && echo ansible-tmp-1729497208.756366-10930-68667324237398=" echo /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398 " ) && sleep 0' Including module_utils file ansible/__init__.py Including module_utils file ansible/module_utils/__init__.py Including module_utils file ansible/module_utils/basic.py Including module_utils file ansible/module_utils/_text.py Including module_utils file ansible/module_utils/common/_json_compat.py Including module_utils file ansible/module_utils/common/__init__.py Including module_utils file ansible/module_utils/common/_utils.py Including module_utils file ansible/module_utils/common/arg_spec.py Including module_utils file ansible/module_utils/common/file.py Including module_utils file ansible/module_utils/common/locale.py Including module_utils file ansible/module_utils/common/parameters.py Including module_utils file ansible/module_utils/common/collections.py Including module_utils file ansible/module_utils/common/process.py Including module_utils file ansible/module_utils/common/sys_info.py Including module_utils file ansible/module_utils/common/text/converters.py Including module_utils file ansible/module_utils/common/text/__init__.py Including module_utils file ansible/module_utils/common/text/formatters.py Including module_utils file ansible/module_utils/common/validation.py Including module_utils file ansible/module_utils/common/warnings.py Including module_utils file ansible/module_utils/compat/selectors.py Including module_utils file ansible/module_utils/compat/__init__.py Including module_utils file ansible/module_utils/compat/_selectors2.py Including module_utils file ansible/module_utils/compat/selinux.py Including module_utils file ansible/module_utils/distro/__init__.py Including module_utils file ansible/module_utils/distro/_distro.py Including module_utils file ansible/module_utils/errors.py Including module_utils file ansible/module_utils/parsing/convert_bool.py Including module_utils file ansible/module_utils/parsing/__init__.py Including module_utils file ansible/module_utils/pycompat24.py Including module_utils file ansible/module_utils/six/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/ansiblemodule.py Including module_utils file ansible_collections/__init__.py Including module_utils file ansible_collections/kubernetes/__init__.py Including module_utils file ansible_collections/kubernetes/core/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/args_common.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/resource.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/__init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/core.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/exceptions.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py Including module_utils file ansible/module_utils/common/dict_transformations.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/apply.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/common.py Including module_utils file ansible/module_utils/urls.py Including module_utils file ansible/module_utils/compat/typing.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/exceptions.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/hashes.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/waiter.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8sdynamicclient.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/version.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/_version.py Using module file /opt/ansible/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py <localhost> PUT /opt/ansible/.ansible/tmp/ansible-local-109200clhnr2p/tmptihkeltq TO /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py <localhost> EXEC /bin/sh -c 'chmod u+x /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/ /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py && sleep 0' <localhost> EXEC /bin/sh -c '/usr/bin/python3 /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py && sleep 0' <localhost> EXEC /bin/sh -c 'rm -f -r /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/ > /dev/null 2>&1 && sleep 0' fatal: [localhost]: FAILED! => { "changed": false, "invocation": { "api_version": "apps/v1", "kind": "Deployment", "module_args": { "api_version": "apps/v1", "kind": "Deployment", "name": "awx", "namespace": "awx" }, "name": "awx", "namespace": "awx" }, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": -9 }

Running the module itself succeeds:
`export PYTHONPATH=${PYTHONPATH}:/opt/ansible/.ansible/collections/
echo '{ "ANSIBLE_MODULE_ARGS": {
"api_version": "apps/v1",
"kind": "Deployment",
"name": "awx",
"namespace": "awx"
} } ' |python3 /opt/ansible/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py

{"changed": false, "resources": [], "api_found": true, "invocation": {"module_args": {"api_version": "apps/v1", "kind": "Deployment", "name": "awx", "namespace": "awx", "wait": false, "wait_sleep": 5, "wait_timeout": 120, "label_selectors": [], "field_selectors": [], "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "wait_condition": null, "hidden_fields": null}}}`

Operator Logs

No response

[rmastersbun]

Point of clarification: The RKE2 cluster I am deploying on is built on OL9 (RHEL9 equivalent).

[rmastersbun]

To cover all bases, I deployed an RKE2 OL8 cluster and attempted the deployment, and it failed in exactly the same way. So then I tried falling back to the generic RedHat kernel (instead of the UEK kernel), and had the same result.

[rmastersbun]

Eventually identified as being caused by Crowdstrike Falcon Agent.