Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Direct batch and multi last client API and examples #310

Merged
merged 3 commits into from
Dec 12, 2024
Merged

Direct batch and multi last client API and examples #310

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a058edf
Propose strategies to expose for direct batch get
scottf Nov 4, 2024
845871a
multi for api
scottf Nov 5, 2024
f910544
multi for variants with batch
scottf Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions adr/ADR-31.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,30 @@ After the batch is sent a zero length payload message will be sent with the `Nat

When requests are made against servers that do not support `batch` the first response will be received and nothing will follow. Old servers can be detected by the absence of the `Nats-Num-Pending` header in the first reply.

There are 4 viable API calls for a batch. All require a batch amount greater than 0 and a subject which may include a wildcard.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should now be 6* viable API calls.

They also require a start sequence -or- a start time.
There is a server issue under consideration requesting that if neither start sequence nor a start time is supplied that it defaults to start sequence of 1.
For now the client can optionally provide the 2 additional calls which provide the start sequence of 1 for the user.

1. get up to batch number of messages >= than sequence 1
* API: `batch: number, subject: string`
* Request: `{"batch":3,"seq":1,"next_by_subj":"foo.>"}`
1. get up to batch number of messages >= than sequence that has specified subject
* API: `batch: number, sequence: number, subject: string`
* Request: `{"batch":3,"seq":4,"next_by_subj":"foo.>"}`
1. gets up to batch number of messages >= than start time that has specified subject
* API: `batch: number, start time: time, subject: string`
* Request: `{"batch":3,"start_time":"2024-11-04T23:45:02.060192000Z","next_by_subj":"foo.>"}`
1. get up to batch number of messages >= than sequence 1, limited by max bytes
* API: `batch: number, max_bytes: number, sequence: number, subject: string`
* Request: `{"batch":3,"max_bytes":2002,"seq":1,"next_by_subj":"foo.>"}`
1. get up to batch number of messages >= than sequence that has specified subject, limited by max bytes
* API: `batch: number, max_bytes: number, sequence: number, subject: string`
* Request: `{"batch":3,"max_bytes":2002,"seq":4,"next_by_subj":"foo.>"}`
1. gets up to batch number of messages >= than start time that has specified subject, limited by max bytes
* API: `batch: number, max_bytes: number, start time: time, subject: string`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start_time is not supported in batch requests (possibly should be)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have unit tests that say otherwise.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check: https://github.com/nats-io/nats.js/blob/4fb61dc8ad21b7467569758fa10cd6351bda7561/jetstream/tests/jsm_direct_test.ts#L327

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The start time is ignored.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have unit tests for all of these and they all work as expected.

  1. batch only
  2. batch with starting sequence
  3. batch with start time
  4. batch with max bytes
  5. batch with max bytes and starting sequence
  6. batch with max bytes and start time

For multi last I have these tests

  1. just subjects
  2. subjects with up_to_seq
  3. subjects with up_to_time

Here is the server validation code (Stream.go line 4266):

// Check we don't have conflicting options set.
// We do not allow batch mode for lastFor requests.
if (req.Seq > 0 && req.LastFor != _EMPTY_) ||
	(req.Seq > 0 && req.StartTime != nil) ||
	(req.StartTime != nil && req.LastFor != _EMPTY_) ||
	(req.LastFor != _EMPTY_ && req.NextFor != _EMPTY_) ||
	(req.LastFor != _EMPTY_ && req.Batch > 0) ||
	(req.LastFor != _EMPTY_ && len(req.MultiLastFor) > 0) ||
	(req.NextFor != _EMPTY_ && len(req.MultiLastFor) > 0) ||
	(req.UpToSeq > 0 && req.UpToTime != nil) {
	hdr := []byte("NATS/1.0 408 Bad Request\r\n\r\n")
	mset.outq.send(newJSPubMsg(reply, _EMPTY_, _EMPTY_, hdr, nil, nil, 0))
	return
}

Relevant notes on that code

  • seq and last for cannot both be present
  • seq and start time cannot both be present
  • up_to_seq and up_to_time cannot both be present

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that batch works to limit the multi_for response, so I have to add these variants in

* Request: `{"batch":3,"max_bytes":2002,"start_time":"2024-11-04T23:45:02.060192000Z","next_by_subj":"foo.>"}`

#### Multi-subject requests

Multiple subjects can be requested in the same manner that a Batch can be requested. In this mode we support consistent point in time reads by allowing for a group of subjects to be read as they were at a point in time - assuming the stream holds enough historical data.
Expand Down Expand Up @@ -142,6 +166,18 @@ A `batch` parameter can be added to restrict the result set to a certain size, o

When the server cannot send any more data it will respond, like the above Batch, with a zero-length payload message including the `Nats-Num-Pending` and `Nats-Last-Sequence` headers enabling clients to determine if further batch calls are needed. In addition, it would also have the `Status` header set to `204` with the `Description` header being `EOB`. The `Nats-UpTo-Sequence` header will be set indicating the last message in the stream that matched criteria. This number would be used in subsequent requests as the `up_to_seq` value to ensure batches of multi-gets are done around a consistent point in time.

For the multi last API, we can make 3 distinct calls:

1. get the last messages for the subjects specified subject
* API: `subjects: []string`
* Request: `{"multi_last":["foo.A","foo.D"]}`
1. get the last messages for the subjects, where the last message is less than or equal to the up to sequence.
* API: `subjects: []string, up_to_sequence: number`
* Request: `{"multi_last":["foo.A","foo.D"],"up_to_seq":23}`
1. get the last messages for the subjects, where the last message is less than or equal to the up to time.
* API: `subject: []string, up_to_time: time`
* Request: `{"multi_last":["foo.A","foo.D"],"up_to_time":"2024-11-05T00:50:25.248431300Z"}`

#### Response Format

Responses may include these status codes:
Expand Down
Loading