No clusters available

[salcedo]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Fresh install Rancher on k3s. Rancher was then used to create 1 additional k3s cluster. After adding a GitRepo, Continuous Delivery > Git Repos shows the repo with a red "Git Updating" badge. In the Clusters Ready column, 0/0 (No clusters available). Under Continuous Delivery > Clusters, the cluster is visible and its detail shows the GitRepo just added.

No solution found after searching existing issues, asking Rancher Users slack, adding a fleet user and GlobalRole with Cluster, GitRepo, and FleetWorkspace permissions per reading one user's possible fix.

Logs from fleet-controller, fleet-agent on the target cluster, did not appear to have any errors except:

time="2023-05-15T16:16:04Z" level=error msg="error syncing 'fleet-default/test-repo': handler sync-repo: failed to delete fleet-default/test-repo-4212c batch/v1, Kind=Job for sync-repo fleet-default/test-repo: jobs.batch \"test-repo-4212c\" not found, requeuing"

This error occurs when there is a commit to the git repository the GitRepo is watching. Could be related but I'm unsure how.

Expected Behavior

We're not doing anything special here. Just installed Rancher per the documentation. After adding a GitRepo, Fleet simply doesn't show any Clusters Ready. No errors in the UI just a yellow caution icon.

Steps To Reproduce

1. Create a k3s cluster.
2. Install Rancher.
3. Using Rancher, create a Custom k3s cluster.
4. In the Rancher UI, add a GitRepo, set it to deploy to the cluster created in step 3.
5. See the cluster listed under Continuous Delivery > Clusters and the repo under Continuous Delivery > Git Repos
6. See 0/0 (No clusters available) under Continuous Delivery > Git Repos

Environment

- Architecture: x86_64
- Fleet Version: 0.6.0
- Cluster:
  - Provider: K3S
  - Options: 3 nodes
  - Kubernetes Version: 1.25

Logs

No response

Anything else?

We have Traefik running in AWS that is linked back to our on-premises clusters over ipsec. Our clusters are unaware of this proxy, and we configure our ingresses with TLS turned off because Traefik handles the termination. This has not caused any issues with Rancher being able to create and manage clusters, but I thought I should mention it.

[kkaempf]

Looks like Fleet cannot reach the git repo.
Please re-check the repo's URL and make sure it's reachable from Fleet (network issues ?)

Further questions

• are you using a git@... or a https://... URL ?
• does the git repo need authentication for access ?

[salcedo]

When I first add a repo, it shows Active. Then after a short time, it gets stuck on Git Updating. The repo url is to an externally facing GitLab instance. The repo has a deploy key with write access and I've confirmed that this key does indeed work. It does not have a passphrase.

If there is a connectivity issue between fleet and this external GitLab, would that cause the Clusters Ready column to show 0/0 and No clusters available?

[salcedo]

Issue resolved after switching to an RSA key instead of ED25519. EC keys generated by openssh have BEGIN OPENSSH PRIVATE KEY. According to Fleet documentation, looks like the key needs to be modified to BEGIN EC PRIVATE KEY, but I saw errors in Rancher UI about it being unable to parse it. RSA key fixed it.

[und-miller]

Issue resolved after switching to an RSA key instead of ED25519. EC keys generated by openssh have BEGIN OPENSSH PRIVATE KEY. According to Fleet documentation, looks like the key needs to be modified to BEGIN EC PRIVATE KEY, but I saw errors in Rancher UI about it being unable to parse it. RSA key fixed it.

Tried this but unfortunately this didn't work.