Skip to content

Commit

Permalink
fixup! feat(block-scheduler): add tracker to manage commits
Browse files Browse the repository at this point in the history
  • Loading branch information
ashwanthgoli committed Jan 2, 2025
1 parent 909b72a commit 18fd3b1
Showing 1 changed file with 0 additions and 17 deletions.
17 changes: 0 additions & 17 deletions pkg/blockbuilder/scheduler/tracker.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,23 +15,6 @@ type Committer interface {
Commit(ctx context.Context, partition int32, offset int64) error
}

/*
why do we need a commit tracker?
given jobs generated by the planner can be processed parallelly and out of order, we need to keep track of the offset that have been completely consumed before committing them.
if there are gaps in what was consumed, tracker has to wait for those gaps to be consumed before committing.
Start building with the assumption that the scheduler does not lose it's state - does not restart
We can write the completion messages to kafka in a follow-up to sustain restarts.
read completion messages from the last 2h by default to recover the state and commit any missing offsets.
first thing to do before marking ready?
what about the case when there are no commits?
planner and commit tracker should start from the same offset.
to keep things simple, we can assume that consumption start from start of the previous day if there are no commits.
This should ensure all components even across restarts are aligned.
*/

type CommitTracker struct {
mu sync.Mutex

Expand Down

0 comments on commit 18fd3b1

Please sign in to comment.