Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streams supporting tombstones #10639

Open
sillencem opened this issue Jan 13, 2025 · 0 comments
Open

Streams supporting tombstones #10639

sillencem opened this issue Jan 13, 2025 · 0 comments

Comments

@sillencem
Copy link

Is your feature request related to a problem? Please describe.
The only way to rekey a topic is to use streams
Streams drop tombstones
There is this workaround https://rmoff.net/2020/11/03/kafka-connect-ksqldb-and-kafka-tombstone-messages/ that creates two independent streams but being independent they will presumably suffer from race conditions where one stream could be ahead of the other and a sequence of delete and adds could ultimately end in the wrong final state

Describe the solution you'd like
We need to be able to support rekeying and ideally other lightweight transformations with tombstones

Describe alternatives you've considered
Duplicate streams one handling the normal case and another handling the tonbstone
A sequence of new record followed by tombstone either stream could process the entire stream first resuting in both possible final states.

Additional context
processing everthing with tables beacuse they support tombstones uses more memory and has more overhead than a stimple stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant