A Wild "ssh connection failed: Failed to authenticate public key: Access denied for 'keyboard interactive'" has appeared.

[kzersatz]

SUMMARY

When attempting to execute the module 'cisco.nxos.nxos_command' the below error is caught no matter what I attempt to tweak
"changed": false,
"module_stderr": "ssh connection failed: Failed to authenticate public key: Access denied for 'keyboard interactive'. Authentication that can continue: publickey,password,keyboard-interactive",
"module_stdout": "",
"msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error"

This playbook shown is only a snippit of a larger one that was created to on-schedule execute MDS backups to a dedicated FTP server and scheduled via Rundeck (this is reproducable VIA rundeck OR this one-off playbook that I've moved to a basic deployment of Ansible-Core to troubleshoot)

It was running faultlessly for over a year, it began failing last week.
The uptime on the MDS is a frightful 532 days -- There was no update performed here.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/jal/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jal/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /mnt/UnixAppRepo/ansible/collections/ansible_collections
  executable location = /home/jal/.local/bin/ansible
  python version = 3.11.10 (main, Sep 24 2024, 09:33:51) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

Collection             Version
---------------------- -------
ansible.netcommon      7.1.0
ansible.posix          1.3.0
ansible.utils          5.1.2
ansible.windows        2.4.0
cisco.nxos             9.2.1
community.vmware       2.2.0
community.general      9.2.0
community.windows      2.2.0
ibm.power_aix          1.3.1
microsoft.ad           1.6.0
netapp.elementsw       21.7.0
netapp.ontap           22.11.0
purestorage.flasharray 1.26.0
purestorage.flashblade 1.15.0

CONFIGURATION

COLLECTIONS_PATHS(/etc/ansible/ansible.cfg) = ['/mnt/AppRepo/ansible/collections/ansible_collections']
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 5
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/mnt/AppRepo/ansible/storage/hosts']
DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = default
DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = False
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_silent
SYSTEM_WARNINGS(/etc/ansible/ansible.cfg) = False

OS / ENVIRONMENT

Worker node: Red Hat Enterprise Linux release 8.10 (Ootpa)

MDS Version: 
Software
  BIOS:      version 3.8.0
  kickstart: version 9.3(2)
  system:    version 9.3(2)
  BIOS compile time:       04/23/2021
  kickstart image file is: bootflash:///m9700-sf3ek9-kickstart-mz.9.3.2.bin
  kickstart compile time:  10/31/2022 12:00:00 [12/13/2022 12:18:59]
  system image file is:    bootflash:///m9700-sf3ek9-mz.9.3.2.bin
  system compile time:     10/31/2022 12:00:00 [12/13/2022 13:36:19]

Hardware
cisco MDS 9706 (6 Slot) Chassis ("Supervisor Module-3")

STEPS TO REPRODUCE

- hosts: sansw01
  name: "Backup sansw01"
  connection: ansible.netcommon.network_cli
  collections:
    - cisco.nxos
  gather_facts: no
  vars:
    host: sansw01.domain.net
    iusername: ansibleuser
    ansible_user: ansibleuser
    ansible_ssh_pass2: "knowngoodpass"
    ansible_password: "{{ ansible_ssh_pass2 }}"
    ipassword: "{{ ansible_ssh_pass2 }}"
    transport: ssh #cli
    ansible_network_os: cisco.nxos.nxos #nxos
  tasks:
    - name: Include variables
      include_vars: '{{ item }}'
      loop:
        - /mnt/AppRepo/ansible/storage/vault/mds/cfdc/sansw01_creds_vault.yml
    - name: Debug domain value
      debug:
        msg: "{{ ansible_user }} ++ {{ iusername }} "
    - name: Backup running config
      cisco.nxos.nxos_command:
        commands:
          - command: "copy running-config ftp://[email protected]/$(SWITCHNAME)/$(SWITCHNAME)-$(TIMESTAMP)-running-config.backup"
            prompt: 'Password'
            answer: "{{ answer }}"
      register: output

EXPECTED RESULTS

It to connect and execute the command, allowing the FTP to complete.

ACTUAL RESULTS

From the specific play

task path: /mnt/AppRepo/ansible/storage/playbooks/cisco/mds/generic/mds_switch_backups.yml:25
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/netcommon
Loading collection ansible.utils from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/utils
redirecting (type: terminal) ansible.builtin.nxos to cisco.nxos.nxos
redirecting (type: cliconf) ansible.builtin.nxos to cisco.nxos.nxos
<sansw01.columbuschildrens.net> Using network group action cisco.nxos.nxos for cisco.nxos.nxos_command
<sansw01.columbuschildrens.net> attempting to start connection
<sansw01.columbuschildrens.net> using connection plugin ansible.netcommon.network_cli
<sansw01.columbuschildrens.net> local domain socket does not exist, starting it
<sansw01.columbuschildrens.net> control socket path is /home/jal/.ansible/pc/e0d81fac40
<sansw01.columbuschildrens.net> Loading collection ansible.builtin from
<sansw01.columbuschildrens.net> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<sansw01.columbuschildrens.net> Loading collection ansible.netcommon from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/netcommon
<sansw01.columbuschildrens.net> Loading collection ansible.utils from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/utils
<sansw01.columbuschildrens.net> redirecting (type: terminal) ansible.builtin.nxos to cisco.nxos.nxos
<sansw01.columbuschildrens.net> Loading collection cisco.nxos from /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos
<sansw01.columbuschildrens.net> redirecting (type: cliconf) ansible.builtin.nxos to cisco.nxos.nxos
<sansw01.columbuschildrens.net> local domain socket listeners started successfully
<sansw01.columbuschildrens.net> loaded cliconf plugin ansible_collections.cisco.nxos.plugins.cliconf.nxos from path /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos/plugins/cliconf/nxos.py for network_os nxos
<sansw01.columbuschildrens.net> ssh type is set to auto
<sansw01.columbuschildrens.net> autodetecting ssh_type
<sansw01.columbuschildrens.net> ssh type is now set to libssh
<sansw01.columbuschildrens.net> Loading collection ansible.builtin from
<sansw01.columbuschildrens.net> local domain socket path is /home/jal/.ansible/pc/e0d81fac40
<sansw01.columbuschildrens.net> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<sansw01.columbuschildrens.net> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.nxos.nxos_command  at /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos/plugins/modules/nxos_command.py
<sansw01.columbuschildrens.net> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.nxos.nxos_command
<sansw01.columbuschildrens.net> ANSIBLE_NETWORK_IMPORT_MODULES: complete
fatal: [sansw01.columbuschildrens.net]: FAILED! => {
    "changed": false,
    "module_stderr": "ssh connection failed: Failed to authenticate public key: Access denied for 'keyboard interactive'. Authentication that can continue: publickey,password,keyboard-interactive",
    "module_stdout": "",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error"
}