Health check and logs for subscribe and publish worker health

[neekolas]

No description provided.

[richardhuaaa]

1. We need to detect if the subscribe or publish worker dies. Even if they die, our API endpoints will continue to function, so just checking the API endpoints is insufficient.
2. Both workers operate via a poll loop on the DB to fetch new records. Each poll has a row limit, so there is a risk that the poll loop starts to fall behind compared to the latest records. We should detect that and scale up if needed - one dumb idea would be to log how many rows were fetched on each poll and alert if that is close to or equal to the page size

[mkysel]

What scenarios could cause a worker to die? Don't they restart themselves?

I can think of:

• catastrophic crash that does not take down the entire process
• shutdown
• ???