fix: Too many requests to GET /v1/servers/{id}
#661
Comments
|
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
lukasmetzner
added a commit
that referenced
this issue
Dec 16, 2024
This includes metrics about internal operations from `k8s.io/cloud-provider` like the workqueue depth and requests to the Kubernetes API. This metrics were already exposed on `:8233/metrics` but this was not documented or scraped. This commit now uses the same registry for our metrics and the Kubernetes libraries, and also exposes them on both ports for backwards compatibility. Besides having all data available, this will also help us with debugging #661. Co-authored-by: Lukas Metzner <[email protected]>
lukasmetzner
pushed a commit
that referenced
this issue
Jan 10, 2025
<!-- section-start changelog --> This release includes an extension of our current metrics to also include the internals of `k8s.io/cloud-provider` with respect to the work queue depth and requests to the Kubernetes API. Besides having all data available, this will also help us with debugging [#661](#661). ### Features - **metrics**: add metrics from cloud-provider library (#824) - **load-balancer**: emit warning if unsupported port protocol is configured (#828) - allow arbitrary length API tokens (#752) <!-- section-end changelog --> --- <details> <summary><h4>PR by <a href="https://github.com/apricote/releaser-pleaser">releaser-pleaser</a> 🤖</h4></summary> If you want to modify the proposed release, add you overrides here. You can learn more about the options in the docs. ## Release Notes ### Prefix / Start This will be added to the start of the release notes. ```rp-prefix This release includes an extension of our current metrics to also include the internals of `k8s.io/cloud-provider` with respect to the work queue depth and requests to the Kubernetes API. Besides having all data available, this will also help us with debugging [#661](#661). ``` ### Suffix / End This will be added to the end of the release notes. ```rp-suffix ``` </details> Co-authored-by: releaser-pleaser <>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR
In some situations hcloud-cloud-controller-manager starts to spam
GET /v1/servers/{id}for a subset of the nodes in the cluster every few seconds.Expected behavior
I would expect hccm to only send a single request per
--node-status-update-frequency(defaults to 5 minutes).Observed behavior
We get reports from customers that exhaust the rate limit and we can see that hcloud-cloud-controller-manager sends > 1 req/s for
GET /v1/servers/{id}. The default rate limit is 1 request per second.A restart of the pod fixes the behaviour.
Minimal working example
We are not sure how to reproduce this yet.
Log output
No response
Additional information
Looking at some request logs, this seems to affect even very old versions of HCCM (>2 years).
The text was updated successfully, but these errors were encountered: