Issues with Artifact Upload in Mender 3.7.4

[mehta-akshay-scanomat]

Description:

I am currently using the open-source version of Mender (3.7.4) and encountering persistent issues when attempting to upload artifacts. Specifically, I receive a 5xx error during the upload process.

When using the UI to upload an artifact, I see the message: "Artifact couldn't be generated. Request failed with status code 502." The logs for the mender-deployment service show a corresponding status code of 500:

time="2024-10-21T16:55:48Z" level=error msg="azblob PutObject: failed to upload object to blob: context canceled" caller="view.(*RESTView)[email protected]:72" request_id=66772fb8-f862-451a-83b7-046743424cc2 user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e
time="2024-10-21T16:55:48Z" level=info msg="500 59998118μs POST /api/management/v1/deployments/artifacts/generate HTTP/1.1 - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" byteswritten=78 caller="accesslog.(*AccessLogMiddleware)[email protected]:82" method=POST path=/api/management/v1/deployments/artifacts/generate qs= request_id=66772fb8-f862-451a-83b7-046743424cc2 responsetime=59.998118647 status=500 ts="2024-10-21 16:54:48.518920248 +0000 UTC" type=http user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e

When using curl, I receive a 502 Bad Gateway error:

curl -X POST ${URL}/api/management/v1/deployments/artifacts \
       -H 'Content-Type: multipart/form-data' \
       -H "Authorization: Bearer ${JWT}" \
       -F "artifact=@${ARTIFACT}"

Using the Mender CLI for artifact uploads results in a status code 409.

I have identified a pattern: files that take longer than one minute to upload consistently fail, whereas smaller files (around 200-300 MB) that upload in under a minute succeed with a status code of 201.

Questions:

1. Is there a default size limitation or timeout configuration that could be affecting these uploads?
2. I found some relevant configurations in the Mender Server repository. I attempted to modify these settings by editing the deployment in Kubernetes:

DEPLOYMENTS_STORAGE_DEFAULT:                 azure
DEPLOYMENTS_STORAGE_UPLOAD_EXPIRE_SECONDS:   300
DEPLOYMENTS_STORAGE_MAX_GENERATE_DATA_SIZE:  1073741824

However, the issue persists. Any guidance on resolving this would be greatly appreciated!

Thank you!

[oldgiova]

Hello @mehta-akshay-scanomat , are you using the official Helm Chart? Did you configure Azure Blob Storage as documented? Which Ingress controller are you using?

[mehta-akshay-scanomat]

Hello @mehta-akshay-scanomat , are you using the official Helm Chart? Did you configure Azure Blob Storage as documented? Which Ingress controller are you using?

1. Yes, I'm using the official helm chart of mender.
2. I have configured Azure Blob Storage as given in the mender documentation.
3. I'm using traefik as ingress controller which I think is default ingress controller.

[oldgiova]

It could be an Ingress controller timeout; with this troubleshooting tip, the solution was to increase the proxy body size. Maybe something similar also with Traefik.
You should see some error logs in the Traefik Ingress controller deployment.

[mehta-akshay-scanomat]

@oldgiova
Thanks for reply.
I have added the the below argument as given in the traefik documentation to 300s which default was 60s
--entryPoints.name.transport.respondingTimeouts.readTimeout=300
But still I'm not able to upload artifact. The error in logs of mender-deployment is still same but in UI status code is changed to 499. It says Artifact couldn't be generated. Request failed with status code 499

I tried to upload using mender-cli as well and got the 499 error as shown below:

67.22 MiB / 535.74 MiB [----------------------->__________________________________________________________________________________________________________________________________________________________________] 12.55% 1.11 MiB p/sVERBOSE response: HTTP/1.1 499 status code 499Connection: closeContent-Length: 21Date: Wed, 23 Oct 2024 12:14:13 GMTReferrer-Policy: no-referrerStrict-Transport-Security: max-age=31536000; includeSubDomains; preloadVary: Accept-EncodingX-Content-Type-Options: nosniffX-Xss-Protection: 1; mode=blockClient Closed RequestFAILURE: artifact upload to 'mender.scanomat.com' failed with status 499ERROR: exit status: 1

When I used curl to upload artifact, I got the following response:

curl -v -X POST \  https://mender.scanomat.com/api/management/v1/deployments/artifacts \  
-H "Authorization: Bearer ..." \  
-F "[email protected]"
Note: Unnecessary use of -X or --request, POST is already inferred.* Host mender.scanomat.com:443 was resolved.* 
IPv6: (none)* IPv4: 13.69.133.251*   Trying 13.69.133.251:443...* Connected to mender.scanomat.com (13.69.133.251) 
port 443* ALPN: curl offers h2,http/1.1* (304) (OUT), TLS handshake, Client hello (1):*  CAfile: /etc/ssl/cert.pem*  
CApath: none* (304) (IN), TLS handshake, Server hello (2):* (304) (IN), TLS handshake, Unknown (8):* (304) (IN), TLS 
handshake, Certificate (11):* (304) (IN), TLS handshake, CERT verify (15):* (304) (IN), TLS handshake, Finished (20):* (304) 
(OUT), TLS handshake, Finished (20):* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / 
UNDEF* ALPN: server accepted h2* Server certificate:*  subject: CN=mender.scanomat.com*  
start date: May 17 06:19:53 2024 GMT*  expire date: Jun 18 06:19:53 2025 GMT*  subjectAltName: 
host "mender.scanomat.com" matched cert's "mender.scanomat.com"*  
issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; 
CN=Go Daddy Secure Certificate Authority - G2*  SSL certificate verify ok.* using HTTP/2* [HTTP/2] [1] OPENED stream for https://mender.scanomat.com/api/management/v1/deployments/artifacts* 
[HTTP/2] [1] [:method: POST]* [HTTP/2] [1] [:scheme: https]* [HTTP/2] [1] [:authority: mender.scanomat.com]* 
[HTTP/2] [1] [:path: /api/management/v1/deployments/artifacts]* 
[HTTP/2] [1] [user-agent: curl/8.7.1]* [HTTP/2] [1] 
[accept: */*]* [HTTP/2] [1] [authorization: Bearer ...]* [HTTP/2] [1] [content-length: 561762549]* [HTTP/2] [1]
 [content-type: multipart/form-data; boundary=------------------------qKlT1gdNLonJbvM5ahJJxT]>
 POST /api/management/v1/deployments/artifacts HTTP/2> Host: mender.scanomat.com> User-Agent: curl/8.7.1> Accept: */*> 
Authorization: Bearer ...> Content-Length: 561762549> Content-Type: multipart/form-data; boundary=------------------------qKlT1gdNLonJbvM5ahJJxT> 
< HTTP/2 499 < date: Wed, 23 Oct 2024 12:54:45 GMT< referrer-policy: no-referrer< strict-transport-security: max-age=31536000; includeSubDomains; preload< vary: 
Accept-Encoding< x-content-type-options: nosniff< x-xss-protection: 1; mode=block< content-length: 21< * 
HTTP error before end of send, stop sending* abort upload after having sent 479788000 bytes* Connection #0 to host mender.scanomat.com left intactClient Closed Request%  

These are how my Traefik deployment and mender-deployment looks like

Name:                   traefik
Namespace:              default
CreationTimestamp:      Wed, 23 Oct 2024 10:26:44 +0000
Labels:                 app.kubernetes.io/instance=traefik-default
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=traefik
                        helm.sh/chart=traefik-32.1.1
Annotations:            deployment.kubernetes.io/revision: 3
                        meta.helm.sh/release-name: traefik
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/instance=traefik-default,app.kubernetes.io/name=traefik
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=traefik-default
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=traefik
                    helm.sh/chart=traefik-32.1.1
  Annotations:      prometheus.io/path: /metrics
                    prometheus.io/port: 9100
                    prometheus.io/scrape: true
  Service Account:  traefik
  Containers:
   traefik:
    Image:       docker.io/traefik:v3.1.6
    Ports:       9100/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entryPoints.metrics.address=:9100/tcp
      --entryPoints.traefik.address=:9000/tcp
      --entryPoints.web.address=:8000/tcp
      --entryPoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetescrd.allowEmptyServices=true
      --providers.kubernetesingress
      --providers.kubernetesingress.allowEmptyServices=true
      --entryPoints.websecure.http.tls=true
      --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300
      --log.level=INFO
    Liveness:   http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
  Volumes:
   data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   tmp:
    Type:          EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:        
    SizeLimit:     <unset>
  Node-Selectors:  <none>
  Tolerations:     <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  traefik-6c56784b69 (0/0 replicas created), traefik-7858bc95cb (0/0 replicas created)
NewReplicaSet:   traefik-7d9bf4d54 (1/1 replicas created)
Events:          <none>

Name:                   mender-deployments
Namespace:              default
CreationTimestamp:      Wed, 23 Oct 2024 08:29:05 +0000
Labels:                 app.kubernetes.io/component=deployments
                        app.kubernetes.io/instance=mender
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=mender-deployments
                        app.kubernetes.io/part-of=mender
                        app.kubernetes.io/version=3.7.7
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: mender
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/name=mender-deployments
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=deployments
                    app.kubernetes.io/instance=mender
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=mender-deployments
                    app.kubernetes.io/part-of=mender
                    app.kubernetes.io/version=3.7.7
  Service Account:  default
  Containers:
   deployments:
    Image:      docker.io/mendersoftware/deployments:mender-3.7
    Port:       <none>
    Host Port:  <none>
    Args:
      server
      --automigrate
    Limits:
      cpu:     300m
      memory:  128Mi
    Requests:
      cpu:      300m
      memory:   64Mi
    Liveness:   http-get http://:8080/api/internal/v1/deployments/alive delay=0s timeout=1s period=5s #success=1 #failure=3
    Readiness:  http-get http://:8080/api/internal/v1/deployments/health delay=0s timeout=1s period=15s #success=1 #failure=3
    Startup:    http-get http://:8080/api/internal/v1/deployments/alive delay=0s timeout=1s period=5s #success=1 #failure=36
    Environment Variables from:
      mongodb-common     Secret with prefix 'DEPLOYMENTS_'  Optional: false
      artifacts-storage  Secret with prefix 'DEPLOYMENTS_'  Optional: false
    Environment:
      DEPLOYMENTS_STORAGE_DEFAULT:               azure
      DEPLOYMENTS_MIDDLEWARE:                    prod
      DEPLOYMENTS_AWS_TAG_ARTIFACT:              
      DEPLOYMENTS_STORAGE_ENABLE_DIRECT_UPLOAD:  true
      DEPLOYMENTS_STORAGE_MAX_IMAGE_SIZE:        1073741824
    Mounts:                                      <none>
  Volumes:                                       <none>
  Node-Selectors:                                <none>
  Tolerations:                                   <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  mender-deployments-965b7b49f (0/0 replicas created)
NewReplicaSet:   mender-deployments-8d9dbd4db (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  28m   deployment-controller  Scaled up replica set mender-deployments-8d9dbd4db to 1
  Normal  ScalingReplicaSet  28m   deployment-controller  Scaled down replica set mender-deployments-965b7b49f to 0 from 1

[alfrunes]

Hello @mehta-akshay-scanomat 👋

Hmm... Status code 499 is not a code a client should be able to receive 🤔
It is used internally in the access logs to indicate that the client closed the connection (same as for Nginx). Could you share the logs from the deployments service?

kubectl logs deploy/mender-deployments

[mehta-akshay-scanomat]

Hi @alfrunes,

time="2024-10-29T10:27:07Z" level=error msg="azblob PutObject: failed to upload object to blob: context canceled" caller="view.(*RESTView)[email protected]:72" request_id=38dce04c-c907-485a-9928-2d94cfc84bae user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e
time="2024-10-29T10:27:07Z" level=info msg="500 59997732μs POST /api/management/v1/deployments/artifacts/generate HTTP/1.1 - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" byteswritten=78 caller="accesslog.(*AccessLogMiddleware)[email protected]:82" method=POST path=/api/management/v1/deployments/artifacts/generate qs= request_id=38dce04c-c907-485a-9928-2d94cfc84bae responsetime=59.997732938 status=500 ts="2024-10-29 10:26:07.83761632 +0000 UTC" type=http user_id=753afdfb-ee20-4fd3-985e-85c74fe4c56e

I got only this thing in the logs of mender-deployments

[alfrunes]

Thank you for sharing the logs, so the status code did not change after adjusting the ingress timeouts.
I'm fairly confident that the deployments service returns status code 500 because the client (Traefik) closes the connectionk prematurely (the internal 499 status code is not present in 3.7.4).
I would assume the issue is still with the ingress, or the mender-api-gateway Traefik deployment). Note that Traefik (as Nginx) also treats status code 499 as the internal condition that the client closed the connection (and generally shouldn't be visible to clients).
Could you also check the logs for the ingress controller and the mender-api-gateway controller?

Important

Make sure you redact all sensitive information such as IP addresses and email addresses before posting here.

[mehta-akshay-scanomat]

Thank you @alfrunes for your reply. I was very hopeful that adjusting ingress timeouts will work but no luck. It's still showing same error as still getting timeout after one minute. Below are the logs of ingress and mender-api-gateway respectively:

2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:101 > Traefik version 3.1.6 built on 2024-10-09T13:57:41Z version=3.1.6
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:108 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true},"entryPoints":{"metrics":{"address":":9100/tcp","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"5m0s"}},"udp":{"timeout":"3s"}},"traefik":{"address":":9000/tcp","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"5m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":8000/tcp","forwardedHeaders":{},"http":{},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"5m0s","readTimeout":"5m0s","writeTimeout":"5m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":8443/tcp","forwardedHeaders":{},"http":{"tls":{}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"5m0s","readTimeout":"5m0s","writeTimeout":"5m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true,"sendAnonymousUsage":true},"log":{"format":"common","level":"DEBUG"},"metrics":{"prometheus":{"addEntryPointsLabels":true,"addServicesLabels":true,"buckets":[0.1,0.3,1.2,5],"entryPoint":"metrics"}},"ping":{"entryPoint":"traefik","terminatingStatusCode":503},"providers":{"kubernetesCRD":{"allowEmptyServices":true},"kubernetesIngress":{"allowEmptyServices":true},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConnsPerHost":11},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:611 > Stats collection is enabled.
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:612 > Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:613 > Help us improve Traefik by leaving this feature on :)
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:614 > More details on: https://doc.traefik.io/traefik/contributing/data-collection/
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:502 > Configured Prometheus metrics metricsProviderName=prometheus
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=metrics
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator aggregator.ProviderAggregator
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *ingress.Provider
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *ingress.Provider provider configuration config={"allowEmptyServices":true}
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:94 > ingress label selector is: "" providerName=kubernetes
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:104 > Creating in-cluster Provider client providerName=kubernetes
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *crd.Provider
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *crd.Provider provider configuration config={"allowEmptyServices":true}
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"models":{"websecure":{"tls":{}}},"routers":{"ping":{"entryPoints":["traefik"],"priority":9223372036854775807,"rule":"PathPrefix(`/ping`)","ruleSyntax":"v3","service":"ping@internal"},"prometheus":{"entryPoints":["metrics"],"priority":9223372036854775807,"rule":"PathPrefix(`/metrics`)","ruleSyntax":"v3","service":"prometheus@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":11}},"services":{"api":{},"dashboard":{},"noop":{},"ping":{},"prometheus":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:91 > label selector is: "" providerName=kubernetescrd
2024-10-30T09:46:56Z INF github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:101 > Creating in-cluster Provider client providerName=kubernetescrd
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=kubernetescrd
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"routers":{"default-mender-ingress-mender-scanomat-com":{"entryPoints":["websecure"],"rule":"Host(`mender.scanomat.com`) \u0026\u0026 PathPrefix(`/`)","service":"default-mender-api-gateway-80"}},"services":{"default-mender-api-gateway-80":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.42.0.172:9080"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=kubernetes
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:97 > No store is defined to add the certificate MIIGqDCCBZCgAwIBAgIJALu+IHG+h+n3MA0GCSqGSIb3DQEBCw, it will be added to the default store
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:131 > Adding certificate for domain(s) mender.scanomat.com,www.mender.scanomat.com
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=metrics middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:268 > Creating load-balancer entryPointName=websecure routerName=default-mender-ingress-mender-scanomat-com@kubernetes serviceName=default-mender-api-gateway-80@kubernetes
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:310 > Creating server entryPointName=websecure routerName=default-mender-ingress-mender-scanomat-com@kubernetes serverName=1da80c0d576ab3bf serviceName=default-mender-api-gateway-80@kubernetes target=http://10.42.0.172:9080
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:82 > Creating middleware entryPointName=websecure middlewareName=metrics-service middlewareType=Metrics routerName=default-mender-ingress-mender-scanomat-com@kubernetes serviceName=default-mender-api-gateway-80@kubernetes
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=websecure middlewareName=metrics-service routerName=default-mender-ingress-mender-scanomat-com@kubernetes serviceName=default-mender-api-gateway-80@kubernetes
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=websecure middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=websecure middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:22 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=metrics middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=metrics middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=traefik middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=traefik middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/metrics/metrics.go:50 > Creating middleware entryPointName=web middlewareName=metrics-entrypoint middlewareType=Metrics
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/entrypoint.go:47 > Creating middleware entryPointName=web middlewareName=tracing middlewareType=TracingEntryPoint
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for mender.scanomat.com with TLS options default entryPointName=websecure
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:181 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetes
2024-10-30T09:46:56Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:179 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetescrd
2024-10-30T09:47:05Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:181 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetes
2024-10-30T09:47:05Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:179 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetescrd
2024-10-30T09:47:05Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:179 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetescrd
2024-10-30T09:47:05Z DBG github.com/traefik/traefik/v3/pkg/provider/kubernetes/ingress/kubernetes.go:181 > Skipping Kubernetes event kind *v1.EndpointSlice providerName=kubernetes

{"ClientAddr":"x.x.x.x:56790","ClientHost":"x.x.x.x","ClientPort":"56790","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":499,"Duration":60000461644,"OriginContentSize":21,"OriginDuration":59996670142,"OriginStatus":499,"Overhead":3791502,"RequestAddr":"mender.scanomat.com","RequestContentSize":300825125,"RequestCount":1329752,"RequestHost":"mender.scanomat.com","RequestMethod":"POST","RequestPath":"/api/management/v1/deployments/artifacts/generate","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"http","RetryAttempts":0,"RouterName":"deploymentsMgmt@file","ServiceAddr":"mender-deployments:8080","ServiceName":"deployments@file","ServiceURL":"http://mender-deployments:8080","StartLocal":"2024-10-30T10:46:12.285322954Z","StartUTC":"2024-10-30T10:46:12.285322954Z","entryPointName":"http","level":"info","msg":"","time":"2024-10-30T10:47:12Z"}

[oldgiova]

Hello @mehta-akshay-scanomat . Could you give some more details about your setup?

• The kubernetes version
• The kubernetes provider
• The Helm chart values file (without any secret on it)
• Which artifacts storage are you using

Thanks

[mehta-akshay-scanomat]

Hi @oldgiova, here are the details you asked for:

1. Kubernetes version : v1.30.5+k3s1

2. Kubernetes provider: local Kubernetes cluster with k3s

3. Artifact storage: Azure Blob Storage

4. Helm chart values files:
   Mender

   api_gateway:
     env:
       SSL: false
   device_auth:
     certs:
       key: |-
         RSA PRIVATE KEY
   global:
     azure:
       AUTH_CONNECTION_STRING: Azure-Blob-Storage-Connection-String
       CONTAINER_NAME: mender-artifact-storage
     enterprise: false
      image:
       tag: mender-3.7
     mongodb:
       URL: mongodb://username:[email protected]:27017
     nats:
       URL: ""
     storage: azure
     url: https://mender.scanomat.com
   nats:
     enabled: true
   useradm:
     certs:
       key: |-
         RSA PRIVATE KEY
   

   Treaefik

   - --entrypoints.web.transport.respondingTimeouts.idleTimeout=600s
   - --entryPoints.web.transport.respondingTimeouts.readTimeout=300s
   - --entryPoints.websecure.transport.respondingTimeouts.idleTimeout=600s
   - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s
   - --serversTransport.forwardingTimeouts.dialTimeout=30s
   - --serversTransport.forwardingTimeouts.responseHeaderTimeout=300s
   - --serversTransport.maxIdleConnsPerHost=10
   

[alfrunes]

Inspecting the logs, it seems like the connection between the api-gateway and deployments gets interrupted after 60s. The Traefik access logs is reporting a downstream error status code of 499, yet deployments is actually returning 500 to a terminated connection. It looks like the issue is at the networking level and not in the application.
If you don't mind, could you try to install the same configuration using a different local Kubernetes provider (e.g. minikube or kind)?