Pingora graceful shutdown sequence and app readiness/healthcheck endpoint

[deorus]

I'm using Pingora v0.4.0 to build an HTTP app and I want the readiness probe endpoint handled by the app (/.healthz/ready) to return non-200 status code if its in a period of a graceful shutdown.

However, when the shutdown starts, all listening services' listeners stop immediately:

2025-01-07T23:42:59.993176Z  INFO pingora_core::server: SIGTERM received, gracefully exiting
2025-01-07T23:42:59.993294Z  INFO pingora_core::server: Broadcasting graceful shutdown
2025-01-07T23:42:59.993377Z  INFO pingora_core::server: Graceful shutdown started!
2025-01-07T23:42:59.993396Z  INFO pingora_core::server: Broadcast graceful shutdown complete
2025-01-07T23:42:59.993440Z  INFO pingora_core::server: Graceful shutdown: grace period 60s starts
2025-01-07T23:42:59.993749Z  INFO pingora_core::services::listening: Shutting down 0.0.0.0:3000
2025-01-07T23:42:59.994083Z  INFO pingora_core::server: service exited.

Any suggestions to work around it?

[xushichangdesmond]

ah ok i found it, its there set_read_timeout and set_write_timeout in the HttpSession.

[deorus]

@xushichangdesmond what you seem to refer to is the upstream health check in the load balancer module:

pingora/pingora-load-balancing/src/health_check.rs

Lines 234 to 236 in 42e11c4

	if let Some(read_timeout) = peer.options.read_timeout {
	session.set_read_timeout(read_timeout);
	}

It's a different health check than the one I'm referring to: I want to serve HTTP until the very last second of the process (including during the graceful shutdown period), but that doesn't seem possible as the service listener stops, so no more (TCP) connections are allowed – does that make sense?