Reconnects

[FrankSaxo]

I am currently running a piece of software that has 9 NATS pods for distributing data, no JetStream, just Core publish subscribe. and we run millions of messages per minute.
Currently we have our own connection failover logic, on the subscriber side, but is that really needed, or does the current failover logic ensure that we do not loose messages, when restarting the NATS (lame duck scenario).
I Have found it hard to find any descriptions on how reconnects works for Core pubsub in the NATS client.

[mtmk]

Subscriptions would failover automatically but at the moment there isn't lame duck support as of v2.1.2.

Currently reconnect happens when a socket exception occurs. With back off, we try the next server in the list (or randomized, configurable with options) to connect to and send SUB messages to recreate the subscriptions made and registered in the same NatsConnection object. During this all the messages published would not be received and lost. My understanding is that when the lame duck support implemented this loss should not happen or be very minimal(?).

Also note that NATS core QoS is at-most-once (or fire-and-forget) meaning applications are responsible for recovering lost messages or loss must be acceptable for the use case.

hope it helps. Please feel free to reply if I missed anything or have more questions.

[FrankSaxo]

Yes we do also cater for lost messages.
but it might be that you had made a smart solution, where in case of LameDuck, you started a new connection, received messages on both, and when the new and the old had received the same message, switched over from the old to the new..
That would ensure a seamless looseless reconnect catering for the most common scenarios.
but thanks for the reply.
is there any plans for making LameDuck support ?

[mtmk]

yes you're right, we should be able to switch over to another connection without losing messages.

We have started discussing LDM support but there are no concrete plans yet afaik.

[mskitroot]

@mtmk what's the workaround for LDM? We can capture the event. Should we just start a new connection?

[mskitroot]

I have a similar issue but I am still on the v1.1.6 client and can set the LameDuckModeEventHandler. I could not find anything comparable for V2.

[mtmk]

ah ok. so for v2 we need to implement the event handler and the reconnect feature at lease.

[mskitroot]

I am just restarting my pod. Too complex to re-initialize the subscriptions. It would be good if the client handled the reconnection from LDM.

[mtmk]

currently we don't have a spec yet to implement a proper LDM reconnect so it's left to the application developer to decide what to do (in general for all NATS clients - we will implement the event handler for v2 eventually). Also, when we implement the reconnect method as well, then you can just call that. I'm guessing that should work for the simple reconnect scenario i.e. without handling all the message handover stuff gracefully

[Jarema]

yes, the client reconnect issued by user code when LDM happens is the way to go.