feat(nns): Add pagination to list_neurons API #3358

max-dfinity · 2025-01-07T22:03:33Z

No description provided.

github-actions

If this pull request affects the behavior of any canister owned by
the Governance team, remember to update the corresponding
unreleased_changes.md file(s).

To acknowldge this reminder (and unblock the PR), dismiss this
code review by going to the bottom of the pull request page, and
supply one of the following reasons:

Done.
No canister behavior changes.

max-dfinity · 2025-01-08T00:46:13Z

rs/rosetta-api/icp/src/request_handler/construction_payloads.rs

@@ -435,6 +435,7 @@ fn handle_list_neurons(
        include_neurons_readable_by_caller: true,
        include_empty_neurons_readable_by_caller: None,
        include_public_neurons_in_full_neurons: None,
+        start_from_neuron_id: None,


@dfinity/finint - I am wondering if the API needs to be updated to take advantage of the paging. It's likely going to start at a limit of 500, so inactive neurons won't be returned, which means in practice no users will use the paging. However, if a user had over 500 active neurons, they would not get them all in a single request.

Are you saying that active neurons will always be in earlier pages than inactive neurons? How does that work with start_from_neuron_id? It can't be that inactive neurons always have larger neuron IDs, right?

@max-dfinity IIUC, this ListNeurons request type would need to be modified to also take a start_from_neuron_id value. Again IIUC, what is returned to the Rosetta client is an OperationOutput::ListNeuronsResponse enum variant, which contains the (in this PR) updated ic_nns_governance_api::pb::v1::ListNeuronsResponse with the next_start_from_neuron_id field. It would then be up to the user calling Rosetta to pass this value in a subsequent call to the list neurons construction endpoint. @fsodre probably has a better understanding!

Thanks @mbjorkqvist

@fsodre would that be a good idea - to add the parameter to the Rosetta ListNeurons type? Alternately, we could have the Rosetta API do the subsequent retrievals. This would largely only affect users with lots of old inactive neurons. I'm not aware of any user with more than 500 active neurons, and I don't really understand the use case for the Rosetta API.

rs/nns/governance/src/governance.rs

dskloetd · 2025-01-08T08:43:40Z

rs/nns/governance/api/src/ic_nns_governance.pb.v1.rs

@@ -3350,6 +3350,12 @@ pub struct ListNeurons {
    /// existing (unmigrated) callers.
    #[prost(bool, optional, tag = "4")]
    pub include_public_neurons_in_full_neurons: Option<bool>,
+    /// If this is set, it skips all neuron IDs until this value is reached, then returns


Does the neuron ID have to be an ID that exists?
Or is it allowed to pass lastNeuronIdFromPreviousPage + 1 in order to continue after the previous page?

Also it could be clearer whether the skipped neurons have higher IDs or lower IDs than the passed one. For example when fetching transactions, the newest transactions are in the first page.

@dskloetd it doesn't have to exist, but it would need to be what's returned from the previous request. If the neuron didn't exist, it would have been in the list that was being queried.

Typically the skipped neurons will have lower IDs b/c of the BTreeMap, but that's not a guarantee of the API by design - it's only that if you call it with the same request but with 'start_from_neuron_id' that you got from the last response, it will give you another set.

If I understand the implementation correctly, providing a non-existing ID will make it return an empty result (since it'll skip results until it sees an id that doesn't exist). Would it be possible that the id returned by next_start_from_neuron_id is deleted between getting that value and requesting the new page?

@fsodre Neurons aren't deleted, so that's not possible.

The only way for the neuron to be non-existing is if the user supplies it, but in that case, we will still see it in the list of neuron ids that are skipped b/c it was user provided. Typically the user should only use a value that was in the reply, b/c that will guarantee it exists in the list (assuming they don't change other request parameters).

mbjorkqvist

Thanks for adding this, @max-dfinity!

If a client requests multiple pages of list neurons, I suppose it is possible that the number of neurons matching the query may change in the meantime. Is this something that should be taken into account, or is the current "best effort" approach deemed good enough? I could imagine that if a new neuron is added during the requests, it would have a high ID and end up at the end of the list (in the last request)? I tried to look for more details of the approach, but neither this PR, nor the related JIRA ticket has any description, and the design linked in the epic also doesn't seem to cover this work.

mbjorkqvist · 2025-01-09T09:25:04Z

rs/rosetta-api/icp/src/request_handler/construction_payloads.rs

@@ -435,6 +435,7 @@ fn handle_list_neurons(
        include_neurons_readable_by_caller: true,
        include_empty_neurons_readable_by_caller: None,
        include_public_neurons_in_full_neurons: None,
+        start_from_neuron_id: None,


@max-dfinity IIUC, this ListNeurons request type would need to be modified to also take a start_from_neuron_id value. Again IIUC, what is returned to the Rosetta client is an OperationOutput::ListNeuronsResponse enum variant, which contains the (in this PR) updated ic_nns_governance_api::pb::v1::ListNeuronsResponse with the next_start_from_neuron_id field. It would then be up to the user calling Rosetta to pass this value in a subsequent call to the list neurons construction endpoint. @fsodre probably has a better understanding!

Done

fsodre · 2025-01-13T10:57:23Z

rs/nns/governance/api/src/ic_nns_governance.pb.v1.rs

@@ -3350,6 +3350,12 @@ pub struct ListNeurons {
    /// existing (unmigrated) callers.
    #[prost(bool, optional, tag = "4")]
    pub include_public_neurons_in_full_neurons: Option<bool>,
+    /// If this is set, it skips all neuron IDs until this value is reached, then returns


If I understand the implementation correctly, providing a non-existing ID will make it return an empty result (since it'll skip results until it sees an id that doesn't exist). Would it be possible that the id returned by next_start_from_neuron_id is deleted between getting that value and requesting the new page?

fsodre · 2025-01-13T11:54:33Z

rs/nns/governance/src/governance.rs

+            if count >= MAX_LIST_NEURONS_RESULTS {
+                next_start_from_neuron_id = Some(neuron_id.id);
+                break;
+            }


Correct me if I'm wrong, but current users of this endpoint currently expect all neuron ids to be returned, but here we're breaking that assumption, which can silently cause problems for the users working under that assumption.

Could we consider not limiting the number of results if start_from_neuron_id is not provided? Or introducing pagination in a new endpoint?

@fsodre The reason for the pagination is that without it, some users cannot access their neurons at all b/c the endpoint runs out of instructions or the request is too big (i.e. they have many many neurons).

There is only a small handful of users who this will affect.

The question I'm hoping to get feedback about is whether or not the rosetta api should hide this detail, or pass it along.

We can add pagination data to Rosetta's request and sequentially fetch all pages if that's not provided.

github-actions bot added the feat label Jan 7, 2025

max-dfinity force-pushed the msum/list-neurons-pagination branch 5 times, most recently from a654324 to 2d22844 Compare January 8, 2025 00:32

max-dfinity marked this pull request as ready for review January 8, 2025 00:44

max-dfinity requested review from a team as code owners January 8, 2025 00:44

github-actions bot previously requested changes Jan 8, 2025

View reviewed changes

github-actions bot added @idx @finint @nns-team labels Jan 8, 2025

max-dfinity commented Jan 8, 2025

View reviewed changes

max-dfinity force-pushed the msum/list-neurons-pagination branch from 6804e43 to 4fb4103 Compare January 8, 2025 00:48

jasonz-dfinity reviewed Jan 8, 2025

View reviewed changes

rs/nns/governance/src/governance.rs Show resolved Hide resolved

dskloetd reviewed Jan 8, 2025

View reviewed changes

mbjorkqvist reviewed Jan 9, 2025

View reviewed changes

nmattia approved these changes Jan 9, 2025

View reviewed changes

max-dfinity force-pushed the msum/list-neurons-pagination branch from 4fb4103 to 8e8c4c1 Compare January 10, 2025 01:59

fsodre reviewed Jan 13, 2025

View reviewed changes

max-dfinity added 7 commits January 15, 2025 11:07

Update the types

c7ef00d

Fix did files

2279eec

Adding paging (with deterministic ordering)

437d960

Fix clippy (in rosetta-api)

d6de12e

fix missing field

f287154

clippy

bc0fbd4

Move constant to top of governance file

2710b8b

max-dfinity added 4 commits January 15, 2025 11:07

Add description to unreleased_changelog.md

3a45cbe

Update comment

4ff1818

Fix test

3cbb128

clippy

282aa55

max-dfinity force-pushed the msum/list-neurons-pagination branch from 539e0b2 to 282aa55 Compare January 15, 2025 19:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(nns): Add pagination to list_neurons API #3358

feat(nns): Add pagination to list_neurons API #3358

max-dfinity commented Jan 7, 2025

github-actions bot left a comment

max-dfinity Jan 8, 2025

dskloetd Jan 8, 2025

mbjorkqvist Jan 9, 2025

max-dfinity Jan 10, 2025

dskloetd Jan 8, 2025

max-dfinity Jan 8, 2025

fsodre Jan 13, 2025

max-dfinity Jan 13, 2025

mbjorkqvist left a comment

mbjorkqvist Jan 9, 2025

fsodre Jan 13, 2025

fsodre Jan 13, 2025

max-dfinity Jan 13, 2025

fsodre Jan 14, 2025

feat(nns): Add pagination to list_neurons API #3358

Are you sure you want to change the base?

feat(nns): Add pagination to list_neurons API #3358

Conversation

max-dfinity commented Jan 7, 2025

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbjorkqvist left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment