Consider persisting tombstones

[maackle]

Hey, your friendly neighborhood model checker here. Model checking showed me this seemingly unhandled case and I was wondering if you had considered it:

It looks to me like tombstones don't persist, they just remove an entry and "expire" immediately. If another entry later comes in which predates the tombstone, it is allowed to be stored. This test fails.

If this is true, and the current behavior is desired, it might be good as an extra line in the spec, since the usual meaning of "tombstone" would imply that this test should pass.

[neonphog]

The tombstones are persisted in the gossiped peer stores of the individual nodes. This is why the term tombstone is in the data structure.

We have to sign those and send them anyway. The bootstrap server needs a way of verifying that the same client which published a non-tombstone entry is the one that is requesting a deletion of that entry. Rather than coming up with a totally parallel set of data to sign, we're just using the gossip tombstone entry as the delete mechanism in the bootstrap server.

If it becomes a real problem that legitimate nodes are copying pre-tombstone entries and re-publishing them to grief systems in the short time before they expire we can add some extra cache to maintain tombstones on the server... but that will come with downsides of memory usage. We don't want to push out other non-tombstone entries from the space memory.

I'm happy to merge some docs to this affect as suggested into the spec.

[maackle]

That design decision makes sense. Yeah, a line in the spec sounds good, for completeness, and to keep in mind the potential griefing vector.