Is it possible to watch realtime subscribers of a certain subject?

[tpoxa]

I need to build something like a scheduler which watches for subscribers of a list of subjects and spawns a new instance of s subscriber automatically.

Is this possible using JetStream?

Thanks.

[derekcollison]

JetStream can do WorkQueue semantics where a job or request is submitted and you can have a horizontally scalable set of workers process them. Would that work or am I misunderstanding the scenario?

[tpoxa]

Hey Derek. Thanks for not giving up :)

I build Flow-based programming system (https://en.wikipedia.org/wiki/Flow-based_programming)

Each worker supports a certain type of graph node (read functionality). So the list of workers installed may vary depending on a user's needs. The Discovery subject is made to understand a palette of functionality available.

When a user adds a desired node to the graph, it becomes an autonomous instance (component.id8206582334.*), which reads its incoming messages, processes them specifically and passes them to the next node.

That's where I am now.

So as a possible simple solution - the manager, from time to time, publishes a ping message which should be answered by all instances in some period of time. The manager gets all of them back, matches them with persistency storage, and now it knows which instances are not functioning (worker was restarted). The manager sends "components.create" message to the available workers to instantiate them.

I just don't like that "time to time"...

[tpoxa]

As an example of a simple graph

[derekcollison]

Would it make sense that when a node is created (with a sort of UUID/NUID) that a NATS KV entry is created that represents all the subjects it should be interested in?

Then when a node restarts, it first gets its state from the KV and listens on those subjects (could be JetStream consumers with backing streams) for any missed messages?

[tpoxa]

Yeah, that's might work out. Thanks. Just need to take care that this node id won't be lost during scale up and scale down.
Thanks, @derekcollison; I think this is the way.

[derekcollison]

Keep us posted!