hcs-sxc-core: message sequences and hashes

[gregscullard]

Detailed Description
HCS-SXC-Core doesn't currently validate incoming HCS notifications for appropriate sequencing and hashing. This could lead to messages being processed out of sequence or malicious messages being injected.

Actual Behavior
If a message arrives out of sequence from mirror node (or a notification is missed leading to a gap in the sequences), hcs-sxc doesn't detect it and sends notifications to applications regardless.
If a message were to be malicious (incorrect hash sequence), hcs-sxc would also pass it onto the application.

Expected Behavior
Out of sequence or malicious messages to be handled as appropriate, proposal below for discussion.

hcs-sxc-core subscribes to notifications from mirror node

On receipt of notification, it persists the message and checks sequence numbers

If out of sequence, waits for a further x messages (or a period of time) before unsubscribing and re-subscribing from the time of the last messages that was in the correct sequence (hope is that the missing message does eventually come through)

If not out of sequence, check the hash

If the hash doesn’t match, the notification is marked as “malicious” in the database and hcs-sxc-core waits for the next message (a non malicious message should eventually arrive).
Malicious messages are never notified to the application (or if notified, they are notified in a way that ensures the app doesn’t process them).

If the hash and sequence do match, notify the application, the application processes the message if it’s not already processing a prior message. Applications should be single threaded (maybe only on a per-topic basis ?) for message processing to ensure two consecutive messages aren’t processed in parallel, leading to data corruption in state (only really likely in the event of fast throughput on topic, but if the throughput is low, serial processing won’t be an issue from a performance point of view, except when catching up).

When the application is done processing the message, it notifies hcs-sxc-core that it is ready for the next message. hcs-sxc-core looks for later messages that may have arrived since the last one (or in the event messages were out of sequence, it likely has some messages left to notify the app)