[BUG] - Slurm Pods Stuck in Init

[seb-835]

Describe the bug
Giving a try to you project because i love the labelling approach with slurmabler
But running the payload/full and simple example failed, the pods are stuck in

[root@rockynode1 ~]# kubectl get pods
NAME                                 READY   STATUS     RESTARTS       AGE
full-master-01-6d984b798f-j7t4k      0/1     Init:0/1   0              2m20s
full-master-02-8468996d54-gmhgb      0/1     Init:0/1   0              2m20s
full-master-03-74c785ff-gnwvv        0/1     Init:0/1   0              2m20s
full-slurm-toolbox-dd7d7fbcb-lrhl2   0/1     Init:0/1   0              2m19s
full-slurmabler-bnpk2                1/1     Running    0              2m21s
full-slurmabler-vffxr                1/1     Running    1 (109s ago)   2m21s
full-slurmabler-vwkhj                1/1     Running    1 (105s ago)   2m21s
full-slurmctld-85cfd7b58-vgvfh       0/1     Init:0/1   0              2m20s
slik-operator-674d8d4ccc-5ds2n       1/1     Running    0              34m

There is no specific message , each munged init-container give same logs :

[root@rockynode1 ~]# kubectl logs full-master-01-6d984b798f-j7t4k -c munged
munged: Notice: Starting munge-0.5.14 daemon (pid 8)
munged: Info: Set origin address to 10.42.0.128 (eth0)
munged: Info: Seeded PRNG with 128 bytes from getrandom()
munged: Warning: PRNG seed dir is insecure: invalid ownership of "/var/lib/munge"
munged: Info: Enabled PRNG entropy pool enhanced stirring
munged: Warning: Keyfile is insecure: "/etc/munge/munge.key" should not be a symbolic link
munged: Warning: Keyfile is insecure: "/etc/munge/munge.key" should not be readable or writable by group (perms=0644)
munged: Warning: Keyfile is insecure: "/etc/munge/munge.key" should not be readable or writable by other (perms=0644)
munged: Warning: Keyfile is insecure: group-writable permissions without sticky bit set on "/etc/munge"
munged: Info: Updating supplementary group mapping every 3600 seconds
munged: Info: Enabled supplementary group mtime check of "/etc/group"
munged: Warning: Socket is insecure: group-writable permissions without sticky bit set on "/run/munge"
munged: Info: Found 0 users with supplementary groups in 0.000 seconds
munged: Info: Created socket "/run/munge/munge.socket.2"
munged: Warning: PIDfile is insecure: group-writable permissions without sticky bit set on "/run/munge"
munged: Info: Created 2 work threads

but main containers do not start.

[liyingjun]

Are you using a k8s version higher than 1.28? If not, you may have to upgrade to version 1.28 or later, as munge uses the sidecarContainers feature implemented in version 1.28

[seb-835]

Hi, i am using a K8S Version 1.28.12 Cluster so i may support sidecarContainers.
What can i check to analyse this behaviour ?

[liyingjun]

Sidecar containers are not enabled by default in version 1.28. Please check if the SidecarContainers feature gate is enabled in your Kubernetes cluster. For more details, please refer to the link: https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/

[seb-835]

Oh damn,
my bad...i jump to K8S 1.30 and all goes fine.
Thanks for helping.

[happytreees]

We can add Kubernetes v1.28+ to the requirements for this.