[BUG] Unrealistic 'duration' Values in Salt Jobs for state.highstate Operations

[shiver25]

Description

While running the state.highstate function for managing server configuration, I noticed unusually high duration values in the SaltStack logs. Below are two examples of such logs. The reported durations are over 86,000 seconds, which seems incorrect.

Setup

• on-prem machine
• VM KVM
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

Run the salt-call state.highstate command on the affected minion.
Check the logs in /var/log/salt/minion.
Observe the reported duration values. Note that the issue does not occur consistently and seems to happen randomly.

2025-01-21 12:08:10,457 [salt.state :2312][INFO ][2482] Running state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.457217 2025-01-21 12:08:10,457 [salt.state :2343][INFO ][2482] Executing state file.managed for [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] 2025-01-21 12:08:10,166 [salt.state :320 ][INFO ][2482] File /etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch is in the correct state 2025-01-21 12:08:10,167 [salt.state :2510][INFO ][2482] Completed state [/etc/pki/rpm-gpg/RPM-GPG-KEY-elasticsearch] at time 12:08:10.166998 (**duration_in_ms=86399709.781**)

2025-01-22 12:08:03,815 [salt.state :2312][INFO ][2486] Running state [/etc/backup/client.conf] at time 12:08:03.814996 2025-01-22 12:08:03,815 [salt.state :2343][INFO ][2486] Executing state file.managed for [/etc/backup/client.conf] 2025-01-22 12:08:03,627 [salt.state :320 ][INFO ][2486] File /etc/backup/client.conf is in the correct state 2025-01-22 12:08:03,628 [salt.state :2510][INFO ][2486] Completed state [/etc/backup/client.conf] at time 12:08:03.627972 (**duration_in_ms=86399812.975**)

Expected behavior

The duration values should reflect the actual time taken to execute the tasks. Values such as 86399709.781, 86399812.975 are implausibly high and do not match the operations performed.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

# salt-minion --versions-report
Salt Version:
          Salt: 3006.9
 
Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.34.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Additional context

Logs show duration values that are far too high for the described operations.
Please advise on whether this is a known issue or provide guidance on further debugging steps.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!