From 7ef6857395f4e68fcca1dea08e4e9e5077a44659 Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 12:04:05 +0100 Subject: [PATCH 01/30] template --- neps/nep-XXXX.md | 136 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 neps/nep-XXXX.md diff --git a/neps/nep-XXXX.md b/neps/nep-XXXX.md new file mode 100644 index 000000000..47b65bdac --- /dev/null +++ b/neps/nep-XXXX.md @@ -0,0 +1,136 @@ +--- +NEP: 0 +Title: NEP Template +Authors: Todd Codrington III +Status: Approved +DiscussionsTo: https://github.com/nearprotocol/neps/pull/0000 +Type: Developer Tools +Version: 1.1.0 +Created: 2022-03-03 +LastUpdated: 2023-03-07 +--- + +[This is a NEP (NEAR Enhancement Proposal) template, as described in [NEP-0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md). Use this when creating a new NEP. The author should delete or replace all the comments or commented brackets when merging their NEP.] + + + +## Summary + +[Provide a short human-readable (~200 words) description of the proposal. A reader should get from this section a high-level understanding about the issue this NEP is addressing.] + +## Motivation + +[Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.] + +## Specification + +[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] + +## Reference Implementation + +[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: + +* Its interaction with other features is clear. +* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) +* It is reasonably clear how the feature would be implemented. +* Corner cases are dissected by example. +* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] + +## Security Implications + +[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] + +## Alternatives + +[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] + +## Future possibilities + +[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] + +## Consequences + +[This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] + +### Positive + +* p1 + +### Neutral + +* n1 + +### Negative + +* n1 + +### Backwards Compatibility + +[All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] + +## Unresolved Issues (Optional) + +[Explain any issues that warrant further discussion. Considerations + +* What parts of the design do you expect to resolve through the NEP process before this gets merged? +* What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +* What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] + +## Changelog + +[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] + +### 1.0.0 - Initial Version + +> Placeholder for the context about when and who approved this NEP version. + +#### Benefits + +> List of benefits filled by the Subject Matter Experts while reviewing this version: + +* Benefit 1 +* Benefit 2 + +#### Concerns + +> Template for Subject Matter Experts review for this version: +> Status: New | Ongoing | Resolved + +| # | Concern | Resolution | Status | +| --: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 55bf0b3ec6a9ef06ecc9fc968567fcfdf5936123 Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 12:15:55 +0100 Subject: [PATCH 02/30] rename --- neps/{nep-XXXX.md => nep-0568.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename neps/{nep-XXXX.md => nep-0568.md} (100%) diff --git a/neps/nep-XXXX.md b/neps/nep-0568.md similarity index 100% rename from neps/nep-XXXX.md rename to neps/nep-0568.md From 2ac4b05ed2638d0bb9eaed561d7e172cd925adf8 Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 12:16:32 +0100 Subject: [PATCH 03/30] metadata --- neps/nep-0568.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 47b65bdac..ca95a2715 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -1,13 +1,13 @@ --- -NEP: 0 -Title: NEP Template -Authors: Todd Codrington III -Status: Approved -DiscussionsTo: https://github.com/nearprotocol/neps/pull/0000 -Type: Developer Tools -Version: 1.1.0 -Created: 2022-03-03 -LastUpdated: 2023-03-07 +NEP: 568 +Title: Resharding V3 +Authors: Adam Chudas, Aleksandr Logunov, Andrea Spurio, Marcelo Diop-Gonzalez, Shreyan Gupta, Waclaw Banasik +Status: Draft +DiscussionsTo: https://github.com/near/nearcore/issues/11881 +Type: Protocol +Version: 1.0.0 +Created: 2024-10-24 +LastUpdated: 2024-10-24 --- [This is a NEP (NEAR Enhancement Proposal) template, as described in [NEP-0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md). Use this when creating a new NEP. The author should delete or replace all the comments or commented brackets when merging their NEP.] @@ -126,10 +126,10 @@ The section should return to the examples given in the previous section, and exp > Template for Subject Matter Experts review for this version: > Status: New | Ongoing | Resolved -| # | Concern | Resolution | Status | -| --: | :------ | :--------- | -----: | -| 1 | | | | -| 2 | | | | +| # | Concern | Resolution | Status | +| ---: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | ## Copyright From 5ef6b9db6d6e2f2e5cd6aec8806df7a1d74ff659 Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 12:42:19 +0100 Subject: [PATCH 04/30] summary --- neps/nep-0568.md | 52 +++++++++++++++++++----------------------------- 1 file changed, 20 insertions(+), 32 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index ca95a2715..11535ebb2 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -2,7 +2,7 @@ NEP: 568 Title: Resharding V3 Authors: Adam Chudas, Aleksandr Logunov, Andrea Spurio, Marcelo Diop-Gonzalez, Shreyan Gupta, Waclaw Banasik -Status: Draft +Status: New DiscussionsTo: https://github.com/near/nearcore/issues/11881 Type: Protocol Version: 1.0.0 @@ -10,41 +10,23 @@ Created: 2024-10-24 LastUpdated: 2024-10-24 --- -[This is a NEP (NEAR Enhancement Proposal) template, as described in [NEP-0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md). Use this when creating a new NEP. The author should delete or replace all the comments or commented brackets when merging their NEP.] +## Summary - - -## Summary - -[Provide a short human-readable (~200 words) description of the proposal. A reader should get from this section a high-level understanding about the issue this NEP is addressing.] +The primary objective of Resharding V3 is to increase chain capacity by +splitting overutilized shards. A secondary aim is to lay the groundwork for +supporting Dynamic Resharding, Instant Resharding, and Shard Merging in future +updates. ## Motivation @@ -134,3 +116,9 @@ The section should return to the examples given in the previous section, and exp ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + + + + +[NEP-040]: https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md +[NEP-508]: https://github.com/near/NEPs/blob/master/neps/nep-0508.md From 62f127c7a516a5fcfb1ba7fe62df0eaa7733dbe7 Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 13:27:25 +0100 Subject: [PATCH 05/30] specification --- neps/nep-0568.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 11535ebb2..310bdcbd4 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -36,6 +36,56 @@ updates. [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] + +Resharding will be scheduled in advance by the NEAR developer team. The new +shard layout will be hardcoded into the neard binary and linked to the protocol +version. As the protocol upgrade progresses, resharding will be triggered during +the post-processing phase of the last block of the epoch. At this point, the +state of the parent shard will be split between two child shards. From the first +block of the new protocol version onward, the chain will operate with the new +shard layout. + +There are two key dimensions to consider: state storage and protocol features, +along with a few additional details. + +1) State Storage: Currently, the state of a shard is stored in three distinct +formats: the state, the flat state, and the mem-trie. Each of these +representations must be resharded. Logically, resharding is an almost +instantaneous event that occurs before the first block under the new shard +layout. However, in practice, some of this work may be deferred to +post-processing, as long as the chain's view reflects a fully resharded state. + +1) Protocol Features: Several protocol features must integrate smoothly with the + resharding process, including: +* Stateless Validation: Resharding must be validated and proven through + stateless validation mechanisms. +* State Sync: Nodes must be able to sync the states of the child + shards post-resharding. +* Cross-Shard Traffic: Receipts sent to the parent shard may need to be + reassigned to one of the child shards. +* Receipt Handling: Delayed, postponed, buffered, and promise-yield receipts + must be correctly distributed between the child shards. +* ShardId Semantics: The shard identifiers will become abstract identifiers + where today they are number in the 0..num_shards range. + +### State Storage - Mem Trie + +### State Storage - Flat State + +### State Storage - State + +### Stateless Validation + +### State Sync + +### Cross Shard Traffic + +### Receipt Handling - Delayed, Postponed, PromiseYield + +### Receipt Handling - Buffered + +### ShardId Semantics + ## Reference Implementation [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: @@ -60,6 +110,8 @@ The section should return to the examples given in the previous section, and exp [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] + + ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] From bec1ae6665c1d520bcd2e2f9f7a345daf6f0106d Mon Sep 17 00:00:00 2001 From: wacban Date: Thu, 24 Oct 2024 13:42:43 +0100 Subject: [PATCH 06/30] formatting template texts --- neps/nep-0568.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 310bdcbd4..f11cb6bba 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -12,8 +12,6 @@ LastUpdated: 2024-10-24 ## Summary -Summary - This proposal introduces a new resharding implementation and shard layout for production networks. @@ -25,17 +23,20 @@ Tracking, and Mem-Trie. The primary objective of Resharding V3 is to increase chain capacity by splitting overutilized shards. A secondary aim is to lay the groundwork for -supporting Dynamic Resharding, Instant Resharding, and Shard Merging in future +supporting Dynamic Resharding, Instant Resharding and Shard Merging in future updates. ## Motivation +``` [Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.] +``` ## Specification +``` [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] - +``` Resharding will be scheduled in advance by the NEAR developer team. The new shard layout will be hardcoded into the neard binary and linked to the protocol @@ -88,6 +89,7 @@ post-processing, as long as the chain's view reflects a fully resharded state. ## Reference Implementation +``` [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: * Its interaction with other features is clear. @@ -97,24 +99,31 @@ post-processing, as long as the chain's view reflects a fully resharded state. * For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] +``` ## Security Implications +``` [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] +``` ## Alternatives +``` [Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] +``` ## Future possibilities +``` [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] - - +``` ## Consequences +``` [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] +``` ### Positive @@ -130,19 +139,25 @@ The section should return to the examples given in the previous section, and exp ### Backwards Compatibility +``` [All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] +``` ## Unresolved Issues (Optional) +``` [Explain any issues that warrant further discussion. Considerations * What parts of the design do you expect to resolve through the NEP process before this gets merged? * What parts of the design do you expect to resolve through the implementation of this feature before stabilization? * What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] +``` ## Changelog +``` [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] +``` ### 1.0.0 - Initial Version From d72d5096ac54cee832561e1057f27553ff6feee7 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Mon, 28 Oct 2024 13:22:15 +0000 Subject: [PATCH 07/30] fix lints --- neps/nep-0568.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index f11cb6bba..e15fd8e9f 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -28,13 +28,13 @@ updates. ## Motivation -``` +```text [Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.] ``` ## Specification -``` +```text [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] ``` @@ -89,7 +89,7 @@ post-processing, as long as the chain's view reflects a fully resharded state. ## Reference Implementation -``` +```text [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: * Its interaction with other features is clear. @@ -103,25 +103,25 @@ The section should return to the examples given in the previous section, and exp ## Security Implications -``` +```text [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] ``` ## Alternatives -``` +```text [Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] ``` ## Future possibilities -``` +```text [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] ``` ## Consequences -``` +```text [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] ``` @@ -139,13 +139,13 @@ The section should return to the examples given in the previous section, and exp ### Backwards Compatibility -``` +```text [All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] ``` ## Unresolved Issues (Optional) -``` +```text [Explain any issues that warrant further discussion. Considerations * What parts of the design do you expect to resolve through the NEP process before this gets merged? @@ -155,7 +155,7 @@ The section should return to the examples given in the previous section, and exp ## Changelog -``` +```text [The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] ``` From 47f22f9b3bf357c6936a8ba7c0698d19f8e5a9d5 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Tue, 29 Oct 2024 10:29:09 +0000 Subject: [PATCH 08/30] Resharding V3 - added future possibilities (#569) Filling in the future possibilities section - and looking for more ideas :) --- neps/nep-0568.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index e15fd8e9f..d4f9adfff 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -115,9 +115,9 @@ The section should return to the examples given in the previous section, and exp ## Future possibilities -```text -[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] -``` +* Dynamic Resharding - In this proposal, resharding is scheduled in advance and hardcoded within the neard binary. In the future, we aim to enable the chain to dynamically trigger and execute resharding autonomously, allowing it to adjust capacity automatically based on demand. +* Fast Dynamic Resharding - In the Dynamic Resharding extension, the new shard layout is configured for the second upcoming epoch. This means that a full epoch must pass before the chain transitions to the updated shard layout. In the future, our goal is to accelerate this process by finalizing the previous epoch more quickly, allowing the chain to adopt the new layout as soon as possible. +* Shard Merging - In this proposal the only allowed resharding operation is shard splitting. In the future, we aim to enable shard merging, allowing underutilized shards to be combined with neighboring shards. This would allow the chain to free up resources and reallocate them where they are most needed. ## Consequences From 8d8e761f6ea898e2a01802565eae77a5e5cbb091 Mon Sep 17 00:00:00 2001 From: Andrea Date: Tue, 29 Oct 2024 16:34:12 +0100 Subject: [PATCH 09/30] Add flat state specs to resharding NEP (#570) Filling the specification section about flat state. --- neps/nep-0568.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index d4f9adfff..b04a336e4 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -73,6 +73,37 @@ post-processing, as long as the chain's view reflects a fully resharded state. ### State Storage - Flat State +Flat State is a collection of key-value pairs stored on disk and each entry +contains a reference to its ShardId. When splitting a shard, every item inside +its Flat State must be correctly reassigned to either one of the new children; +due to technical limitations such operation can't be completed instantaneously. + +Flat State main purposes are allowing the creation of State Sync snapshots and +the construction of Mem Tries. Fortunately, these two operations can be delayed +until resharding is completed. Note also that with Mem Tries enabled the chain +can move forward even if the current status of Flat State is not in sync with +the latest block. + +For the reason stated above, the chosen strategy is to reshard Flat State in a +long-running background task. The new shards' states must converge with their +Mem Tries representation in a reasonable amount of time. + +Splitting a shard's Flat State is performed in multiple steps: +1) A post-processing 'split' task is created during the last block of the old + shard layout, instantaneously. +2) The 'split' task runs in parallel with the chain for a certain amount of + time. Inside this routine every key-value pair belonging to the shard being + split (also called parent shard) is copied into either the left or the right + child Flat State. Entries linked to receipts are handled in a special way. +3) Once the task is completed, the parent shard Flat State is cleaned up. The + children shards Flat States have their state in sync with last block of the + old shard layout. +4) Children shards must apply the delta changes from the first block of the new + shard layout until the final block of the canonical chain. This operation is + done in another background task to avoid slowdowns while processing blocks. +5) Children shards Flat States are now ready and can be used to take State Sync + snapshots and to reload Mem Tries. + ### State Storage - State ### Stateless Validation From 1ee5a74b24317defd53c3bbd0a1b881033fbc815 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Tue, 29 Oct 2024 15:47:23 +0000 Subject: [PATCH 10/30] fix lints --- neps/nep-0568.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index b04a336e4..07ce68761 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -58,6 +58,7 @@ post-processing, as long as the chain's view reflects a fully resharded state. 1) Protocol Features: Several protocol features must integrate smoothly with the resharding process, including: + * Stateless Validation: Resharding must be validated and proven through stateless validation mechanisms. * State Sync: Nodes must be able to sync the states of the child @@ -89,6 +90,7 @@ long-running background task. The new shards' states must converge with their Mem Tries representation in a reasonable amount of time. Splitting a shard's Flat State is performed in multiple steps: + 1) A post-processing 'split' task is created during the last block of the old shard layout, instantaneously. 2) The 'split' task runs in parallel with the chain for a certain amount of From 6c984415ce0399429ae2f6bcc516acc901ce0220 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Chuda=C5=9B?= <18039094+staffik@users.noreply.github.com> Date: Wed, 6 Nov 2024 16:43:05 +0100 Subject: [PATCH 11/30] State Storage - State (#571) --- neps/nep-0568.md | 106 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 07ce68761..b44878e81 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -107,6 +107,28 @@ Splitting a shard's Flat State is performed in multiple steps: snapshots and to reload Mem Tries. ### State Storage - State +// TODO Describe integration with cold storage once design is ready + +Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. +This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding, +a mapping strategy allows child shards to access ancestor shard data without directly creating new entries. + +A naive approach to resharding would involve copying all `State` entries with a new `ShardUId` for a child shard, effectively duplicating the state. +This method, while straightforward, is not feasible because copying a large state would take too much time. +Resharding needs to appear complete between two blocks, so a direct copy would not allow the process to occur quickly enough. + +To address this, Resharding V3 employs an efficient mapping strategy, using the `DBCol::ShardUIdMapping` column +to link each child shard’s `ShardUId` to the closest ancestor’s `ShardUId` holding the relevant data. +This allows child shards to access and update state data under the ancestor shard’s prefix without duplicating entries. + +Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`, +pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data. +Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected. +For archival nodes, mappings are retained indefinitely to maintain access to the full historical state. + +This mapping strategy enables efficient shard management during resharding events, +supporting smooth transitions without altering storage structures directly. + ### Stateless Validation @@ -134,6 +156,90 @@ Splitting a shard's Flat State is performed in multiple steps: The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] ``` +### State Storage - State mapping + +To enable efficient shard state management during resharding, Resharding V3 uses the `DBCol::ShardUIdMapping` column. +This mapping allows child shards to reference ancestor shard data, avoiding the need for immediate duplication of state entries. + +#### Mapping application in adapters + +The core of the mapping logic is applied in `TrieStoreAdapter` and `TrieStoreUpdateAdapter`, which act as layers over the general `Store` interface. +Here’s a breakdown of the key functions involved: + +- **Key resolution**: + The `get_key_from_shard_uid_and_hash` function is central to determining the correct `ShardUId` for state access. + At a high level, operations use the child shard's `ShardUId`, but within this function, + the `DBCol::ShardUIdMapping` column is checked to determine if an ancestor `ShardUId` should be used instead. + + ```rust + fn get_key_from_shard_uid_and_hash( + store: &Store, + shard_uid: ShardUId, + hash: &CryptoHash, + ) -> [u8; 40] { + let mapped_shard_uid = store + .get_ser::(DBCol::StateShardUIdMapping, &shard_uid.to_bytes()) + .expect("get_key_from_shard_uid_and_hash() failed") + .unwrap_or(shard_uid); + let mut key = [0; 40]; + key[0..8].copy_from_slice(&mapped_shard_uid.to_bytes()); + key[8..].copy_from_slice(hash.as_ref()); + key + } + ``` + + This function first attempts to retrieve a mapped ancestor `ShardUId` from `DBCol::ShardUIdMapping`. + If no mapping exists, it defaults to the provided child `ShardUId`. + This resolved `ShardUId` is then combined with the `node_hash` to form the final key used in `State` column operations. + +- **State access operations**: + The `TrieStoreAdapter` and `TrieStoreUpdateAdapter` use `get_key_from_shard_uid_and_hash` to correctly resolve the key for both reads and writes. + Example methods include: + + ```rust + // In TrieStoreAdapter + pub fn get(&self, shard_uid: ShardUId, hash: &CryptoHash) -> Result, StorageError> { + let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); + self.store.get(DBCol::State, &key) + } + + // In TrieStoreUpdateAdapter + pub fn increment_refcount_by( + &mut self, + shard_uid: ShardUId, + hash: &CryptoHash, + data: &[u8], + increment: NonZero, + ) { + let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); + self.store_update.increment_refcount_by(DBCol::State, key.as_ref(), data, increment); + } + ``` + The `get` function retrieves data using the resolved `ShardUId` and key, while `increment_refcount_by` manages reference counts, + ensuring correct tracking even when accessing data under an ancestor shard. + +#### Mapping retention and cleanup + +Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on an ancestor’s data. +To manage this, the `set_shard_uid_mapping` function in `TrieStoreUpdateAdapter` adds a new mapping during resharding: +```rust +fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: ShardUId) { + self.store_update.set( + DBCol::StateShardUIdMapping, + child_shard_uid.to_bytes().as_ref(), + &borsh::to_vec(&parent_shard_uid).expect("Borsh serialize cannot fail"), + ) +} +``` + +When a node stops tracking all descendants of a shard, the associated mapping entry can be removed, allowing RocksDB to perform garbage collection. +For archival nodes, mappings are retained permanently to ensure access to the historical state of all shards. + +This implementation ensures efficient and scalable shard state transitions, +allowing child shards to use ancestor data without creating redundant entries. + + + ## Security Implications ```text From 08b18342bfb57cad6c83ab6d8f4046c03c1b6e34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Chuda=C5=9B?= Date: Wed, 6 Nov 2024 16:45:07 +0100 Subject: [PATCH 12/30] fix lint --- neps/nep-0568.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index b44878e81..8a27a0c04 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -107,6 +107,7 @@ Splitting a shard's Flat State is performed in multiple steps: snapshots and to reload Mem Tries. ### State Storage - State + // TODO Describe integration with cold storage once design is ready Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. @@ -238,8 +239,6 @@ For archival nodes, mappings are retained permanently to ensure access to the hi This implementation ensures efficient and scalable shard state transitions, allowing child shards to use ancestor data without creating redundant entries. - - ## Security Implications ```text From 6e431ff588f9fce53a4255c8959dbe5865398b45 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Chuda=C5=9B?= Date: Wed, 6 Nov 2024 16:55:21 +0100 Subject: [PATCH 13/30] fix lint --- neps/nep-0568.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 8a27a0c04..285d3524c 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -167,7 +167,7 @@ This mapping allows child shards to reference ancestor shard data, avoiding the The core of the mapping logic is applied in `TrieStoreAdapter` and `TrieStoreUpdateAdapter`, which act as layers over the general `Store` interface. Here’s a breakdown of the key functions involved: -- **Key resolution**: +* **Key resolution**: The `get_key_from_shard_uid_and_hash` function is central to determining the correct `ShardUId` for state access. At a high level, operations use the child shard's `ShardUId`, but within this function, the `DBCol::ShardUIdMapping` column is checked to determine if an ancestor `ShardUId` should be used instead. @@ -193,7 +193,7 @@ Here’s a breakdown of the key functions involved: If no mapping exists, it defaults to the provided child `ShardUId`. This resolved `ShardUId` is then combined with the `node_hash` to form the final key used in `State` column operations. -- **State access operations**: +* **State access operations**: The `TrieStoreAdapter` and `TrieStoreUpdateAdapter` use `get_key_from_shard_uid_and_hash` to correctly resolve the key for both reads and writes. Example methods include: @@ -216,6 +216,7 @@ Here’s a breakdown of the key functions involved: self.store_update.increment_refcount_by(DBCol::State, key.as_ref(), data, increment); } ``` + The `get` function retrieves data using the resolved `ShardUId` and key, while `increment_refcount_by` manages reference counts, ensuring correct tracking even when accessing data under an ancestor shard. @@ -223,6 +224,7 @@ Here’s a breakdown of the key functions involved: Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on an ancestor’s data. To manage this, the `set_shard_uid_mapping` function in `TrieStoreUpdateAdapter` adds a new mapping during resharding: + ```rust fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: ShardUId) { self.store_update.set( From 5368857ee874d82038ba5d406529e191a8dac2f9 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Thu, 7 Nov 2024 11:28:08 +0000 Subject: [PATCH 14/30] ShardId Semantics (#572) --- neps/nep-0568.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 285d3524c..98615acf0 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -143,6 +143,10 @@ supporting smooth transitions without altering storage structures directly. ### ShardId Semantics +Currently, shard IDs are represented as numbers within the range `[0,n)`, where n is the total number of shards. These shard IDs are sorted in the same order as the account ID ranges assigned to them. While this approach is straightforward, it complicates resharding operations, particularly when splitting a shard in the middle of the range. Such a split requires reindexing all subsequent shards with higher IDs, adding complexity to the process. + +In this NEP, we propose updating the ShardId semantics to allow for arbitrary identifiers. Although ShardIds will remain integers, they will no longer be restricted to the `[0,n)` range, and they may appear in any order. The only requirement is that each ShardId must be unique. In practice, during resharding, the ID of a parent shard will be removed from the ShardLayout, and the new child shards will be assigned unique IDs - `max(shard_ids)+1` and `max(shard_ids)+2`. + ## Reference Implementation ```text From a7bbee7b091abe192467cc4b1fe784e8fbf18a18 Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Thu, 14 Nov 2024 22:04:09 +0530 Subject: [PATCH 15/30] ReshardingV3 memtrie (#574) Adding memtrie section to resharding --- neps/assets/nep-0568/NEP-HybridMemTrie.png | Bin 0 -> 20185 bytes neps/assets/nep-0568/NEP-SplitState.png | Bin 0 -> 81729 bytes neps/nep-0568.md | 65 +++++++++++++++++++-- 3 files changed, 61 insertions(+), 4 deletions(-) create mode 100644 neps/assets/nep-0568/NEP-HybridMemTrie.png create mode 100644 neps/assets/nep-0568/NEP-SplitState.png diff --git a/neps/assets/nep-0568/NEP-HybridMemTrie.png b/neps/assets/nep-0568/NEP-HybridMemTrie.png new file mode 100644 index 0000000000000000000000000000000000000000..013c761f8a6722a96c6a34686d9c73f017e3c6f2 GIT binary patch literal 20185 zcmbrmbySq^{w_>|h#(@}pdyljbd01R-6bUgQi2kaLkc1&NP{#8LkLKB3L-5GNSBmD zHw+AO?!o=td;iXP*SpqvU)SOvC!XiI<8xo1D<)i1{n0gII$|s=tZT|j^4eHf*caeW zln4j>k5D7Q6c(0xrn0<@uBX{%+7(gCb>{C}2oL9VH4oOTH;)sjHVcY5-$cvW$=o$z z%gR-HchgD!ljh`et&ddXc6v2xcH4Q+&EcnL@on@#+KMjqDoOsi^x*MsL`I(AsdTKb4r^iFDP;aleqUELBr2{lX){&p>WJ+Z zN%DXKXaeKWwaG2{jyF<$$J^w z_0OBL`W)`7^0bQ<^~IVw*|*#dV(BJ(cvwM>86Vkeg&FTNk>q&f_L?Y1JTO*oHG>hh z?3bUV(}mE>fN^7k!*h@*5sA7lsW;2|leu|kS7}obk2U1SQsbw|l!zt{V^KBOh0HS7 zv+|nZrBdY$*(r|9Iq8hzD#yZW*g{<&E3nb+s|*F!5X3M zP*VGrB^LcgV#eUvS=-`puyolKPC2T)!5~K2YsvScAJ&kUjf(AcKPe2|{QG~NEast{5F`^@Ptr2^;YI@v-Ssk<)R_?IK6!-oj-EJ-ddDH<;J z(s}bjK)Csiu9O^VsEf0mj<|Ee9e#KX zJrRD`{8uSL_doC)TvehjAKLq*!Jq##P|MIeO)!Pn0y)cQ^&YpwPvUD>$nDYwt!q8g znI_}YpHoscF)L*+a$~ZfQHF%dZ;qsQKOD(eO}(q3`z??PrP=t;$cJnm6G)I%Ag&_F zRzCEg%`^U3OZNsm2+)G~QK0+P6VesU9P(=wG;Rs?R-nKW|Cu6b+RLC&|-L2eens6F4apiw*8qt@vf)*pY3|>z@0IuYqy?6Cr8yxbfDW43-fIY zZ%QtfLWYj$L!6U~jXmm`|h3UbWrmcJvCn)N9a%PtFo8+b_4%a7syyF4}`AbxJYPhPTrzSW6tihO@g z&Bn%-rYFIZw0gcM!+ZH9s}V`h)8B>fe2o7{LK-1yxMT2gLPb?bN7wkP5c})M&uU?$ z0=6G+XI&PgHAL91v?p5gg3_$YvM`yFzB;0YnoIgCrfL55Es!`bp3njX8ryiU z;Ucxo8u{yZr`Ry4a1f@{!({&N>SS5&Mb!`5ey9U4JBLR0L_WlsKJb>Lu(G_a+xG6& z#v;W1%bTG)f|riI#&T|I&5?K((kcJQC*SDbFQ(M%ws^Dz`Vo-mX=+RRZT?8wrD!-D z3VCI5a?jA(8p1dxt6N_f^)&@gNV;v|U7Bv3#OqHFW>6bx)bt}{^vUmk4Q+AVo@#?X z#hF+LJTKZ~l(5cd;9;Ib*?x-Y{>1@wCX4c*|ft!bu(&VmJ4?7$}q@|nUIS`VJ z)Ff{d=MLjDiFl!vVOQSI4jFwRWVrHR%BwQ^Woy;nIS-XOtg)X79cRE%4DEp!SH4xA ze!16^lPn#`uN#&!t#bTqX(>FB^-SFiXC~h^W2StYhKAPrZT3{Ww_;9Wmy>a9qsC*! zE%_h8?IweDPR)*si&Fl`jXF&l8-9B&xcA-S*~Y8&fmYPp?KuzUDB`1uwZ) zTbSgmBND!Jk25IFNkSgbX@@VOHwwZaqhRV;Fg)xwFUFc{gk{VcW_ zH_U1F6Rr-n*qaah3Q6~;oTJla_|l-APY%hLUKsYFu>8oiv|?WhKjX;GRvVo5CP2*j zEV9|O#5 z`IQwNXH<2iX`=vBJzl>widw$wV~<#yN>;iXJtyHlDfJB2A(;rB+}II)ZjGJB05T3$ zI8)zo=hoB37|t+t|CjXzVwHg(vQD@CH+nelNKB-ByegClz1uN4&eI@+;tvZ*mhXO7 zB@so13@a&fPyHb}Z%lZ3Ug16!5TqTheQBkDpDeE9zf&%{3B|1#oKc@lq z{F93R;7*Ekw7X${Ttc>-}&aka5aAo1@+SsK3Ts!Ys`4fn@OJ9gSrS+ zFBhiDL|QUzuF&_vavMgSf9`=v2<#Wx?hjPhk**WKWzBJ>fs@hN6-zwBd;4i)==Fj? z`KBZKdYSOXHE}Zguptve3DLGz=C1AB75N_>;Um+TQD2kRi1rC!$-=#fjxip!Sfp6Q zLtY1CjFMgUHx6hk&ShauyR?_Zy{9vRdaBBS@aYXo%0MlDMdLWE(Ye;k1?B0ryNLf* z`E#sqg^ijyL^=>bt$PUx`IMEFS@H^3@treL*0Mr0{w5XGV-*#MA2N=6 zc+@!;;;F2PQ`*Zf^w`rerl8;wrCBcg{vJg~M`u#)m}S3)&`>8D{d2Y~H^hG~-w?G` z9di4!NWCd5wlgt0&$k9F#?5YI$jQL^&g_KlNcej$_Z8=nQ z&)Iih8#Qe`{pTTYJW6Z)|2lG_k!-B2GLIhR7Z$$LLMy167ZJ0MV^&Q8GZO^L&B^Jt zH7&M6tP45AJh(Q*(8S~ni4<_C``6hN?aji@E@xzP+W$$UkMaqCx0pq*WV*k%tfHXM z$^PgAzri_fVa6pqEO3~fnPFsLNUD19&l}TXGBWshcudXA2A;_LbIidRnWcT#e&&3y zcA84!Y9{#C=?7=b%F62M>gqFQ`sd?}FE=NJUcTIwc)i7Mu%ju0@OS$?6t*{#d%kD6 zwcjuiNkDo0$_ygi*06Av@1^0)oJ9rEyD8ZSjlMU>Iq|(AybObEluUO{_gAyz!W0!0 z?i=_<)N8f0xy%$PNGN%A6rRes=HI8|PF;hSYL`?OjO-}al%sKGtK*Yd&0U-G9; z#x+=otQ(>G_+rY+4%AN|iBJJ9Ci^Y3AJa6{Zpbyo$Ee?mWDzC{tCFKZ+)UGyu^DNe z&yVljWuC2|ql*=5tBW-$y_wgR?keyrCSg^>DY)v73Hy9j0sCJ_Ww`n?=TS^d3^nf) zi;@#k_q>Zv89|xa=pPhnGAP1fgMn0=9}A?jt7q!?hYm*@knSPy~hHecff~=JJ79#4&#{GUv zM~|+QuX*NO7n&n2&Ki4mKZb0JO~lBUh>%yF6ClF+vzSs8_zp`N{Yma%58ba-t#Etw zU{y*t5VJIi8a8Gc4D|F4H>Ya)DChDA;DbJ2=6-cZi?O;bl{GVLRY8aD-+ZUx$Ow6H z3&tGRfBFdJkbR!;W+1fFAnF6f;W1)@c7?bsqb{9PbH!zp%pq$?ONO8xFJ@T9#{y#a zsXdDciK_{bd^a0CyIZu{A#JNl`(0H166%B~m^FH`^6_QF#&$f}l|yKu_>l>*@c{~? z2B@6(gdJs!q354|@jZhvlgt|ApY>mD=%A_%L|%Or6M0CeDZwoHkYa^64r!_Xm91}o zD_)f%I55nk^eGHt@xkg_w$ae5n5x}4L*FDi`vsg#IDhM> zq>t~_$nVN^4JmsOgyto1=BXB8AQaVWIIM_-Owc8l3>|70n~;+Gw%RTerx;kj=e3qG;JcaaYL!EKu} zFSR4Ls{TRsThu}0mvE-KySF#?<43QHb1~OWe=QNjYJU!WSVCxRaKD!U>gnaa@wvD- za~2e=m~Ha$@%9$b_dD5n;v1ijpIo7TtBI7ly1MOPx|Hup zSyno%9pu!+@Q19ODy}v%DsjF*|{t}P;!TPV##Q1nMB_$XPMkDs>mKzqs zmaQ-e-GjS#?u1U%Pd<=6tVrdvt*wSbRdA#mibFH_>wM$!^`c#cNtG?_g%7N$MEJ4EgV?K~)V6$D94mGIzFS8Ug~a|0@@VZ*Ge-~sNtAP&x5@%HZ~?M zE^cjYt$U4gW3sA9C6RZC`FyHg-`DGKZ=9IDb?0~3q_eVxWW!JLX6@nO{RDYt?vtY0 zjoW!CKhD;7)2H`caj3&1YS@{1deN8PdVx9{%z(_b`1>Dk&E(|d6gVK`yqajUg+Hyrg566w-#PU8?<#P8Oh0F4rAQfSAzYLu%q|Rj<*}!*Q>3*liE`( zQHPpW42|-!v1z+C!~k-L?(@#rex90w7Wni;!)s?SsZk7KdkY<5ZNY?RCnq;4Db+Hh z84=3CnVBqL(U*$ILXA#@KwqV~LP?uPl{Ij$keQr?ze|LlA0QE*lbr>yfHpQa3kI#Z zI629wsayQf7r#tcTDZ?s!1`4LSc(3bwe|AyveQg`l}XJD%nC$6{4XReS_8(%4UJ8f zsKd-F{x;6!qO7c}TU%Qb6BC-!{uk#C%3arRI>r3oxgPi;K5!-`2f0e0}~G_un24Z(t}(P9sfRp+`|sQMV~M*+^6Kdez$p z*TDn*#bSZ3zq_vS@O76hu6aS^7Y^aKs$hjg69seNnD^_rQZ|q>Apk zbLGmkg6~m4O_8O>e2f)gu(A^vP<&jC>xT86cXo73Yo9EPzKJKMB`HTux!t{P`=Jj6-&!xgm zEJ=Sb!-xz|-G8mr-xk!OZi?D7vU_%adFYR+SZ6K?>RWgWA-}?SHTv6x1fP=i#bHa$ zhD7!6Zd&^c_v^Zlc41J6z*c}27e6b3K}t*;62QNd6y(XSRbhEq*$z0pEA`DicjntB zDr~b{RtM6qQuFeUHT$EbB_uvw9O7afeu*VfPnW#%p7yoKOzlhDdp~O8M}q+Fu=c-- z2#z5--k#&*<<0vAzkQza8L)mU^u-hT=ukNBo0@b%{rm7amHfGtCb8-dg0^R2(R7{a z_cSM}d`dxQ>E=fGhc@MiS$x={62BG-HB>z|n+!f6$z+1cTCL&6>DEVnP05`wp`PSt ziuhbRXi8*TzISmSJk^cu*{1d3&%NMzlDH7z92)5~w6s-@6R*H?7#kb6ppK>r^@_Uh zIXP{9`}VE$lmDrr@~hb4nA<9&+cOP>{-R*3hWnxI{j4b!RGidWq*lEZ%CSVEcl}SNKe67N}BR=y3is zyHnFIHto^X0zE(hyENy#T79~O&vyDDt^1*wde-lhB|ps_)7rji4`Mqp&yb&mYJWga z=R1Mk?16Mnc95{#c}eKT-#LAJ!?@B>0YY$uvwuT0Hs;1=c6$?M!{OoKMbe9WFf!3- zrJ>a=DlOIUbA00MElwlI!4bC4c-Z25dV}C1)gFfK2qUw!v;-*Gd90W=hZY;o$HDQ) z4@R!3uTN0E@(AUYo10thIMH6bytmjH5k|%=;IK{%?bqf*K%l46oJ*-KzsaxP$!Kw& zO>cJCI;$rwW@pV%Po7_VXKsAdu05}6)}wKM3@valUhPD`pUY0f{G{GZSyna(rbE-+ z(_=qb$@yFG78SRl%g($EfG8h7el#;P^Y-?>xVTti>-%c;or?EK_o*?eEMxac=4vLK zEG@O9gtz;c5RoC};}J&62y>7I2T7W+6W!;F!otFME(1Dx`cI!e**<#~ExcopoRC0} z*3L=M_pWl*8m0TAwryI3Y#9iAK>5AtR{P8q zeWik`HMhB}EZQjIfv&~dnNsr>A%`)312{8_gy$PU2Q)T(4zwxDy@}i7U*$l{Gddc- z|K+MQAt}T2nR=lUk=wWNQ?w8q&h)8LK1YOzuo78g;}m2*p{M+bYYrSd;v9e?;H6s} z9CCt-FTnvPdhg^4V*tSJg@yN&{K3u#U8IEz@`4s&u(-IoO4x$q-j-imTdP;3A3?{#EW!ZOveT^YD?g>Igz2pm4BG^XO3o z72(y3Y(NUvh#$inOT^9^n2BX0#<}FY)xVf?MB;ZI+blBcg z+_(XU!$qGQf(n+QOF&2%`VmC&eV>n%BT^%nqK!0&MH z>J&1U3+;=NEZO!Qnb6gQUnu`nZm1Zkh{(kHjaou+{P|Qoom72%^5(6O$)2<1ZJ~x& z?<_fZXZMfOYl$KnqSAsFAVvx)j18XDil`UOZ3RZB_roPDJ~qo8wQEAce0~VryGLx6 z7t5j1{Ca!#mqvR{o%=dfWN`Q0D>4BMueSl3G;i{bugf;Wt~lPDN;Ennn@66Ucxul! z`!&?pi?5m>5)%^%gJW&vr-ii9;S?O*LK)G~*J)yO5Mg?N#g)5|(b1_;j=jV0igVbp zle)pm#@6228W2Z8L7}`V-&O=oVu%*_3|$k8i;pjO6nOd#6p8{h!$n&?z@3{NQrzR> zT5#vQbB6$e-mjKiyJx|GM==Yovo+U}@$$+IyMUyWlo*vIS^P#(|FcbzE`#8!MivH2uL&2P4~d5{DEN6r8HG(?JR`Y{8fbET55OLIKXy zF8P2g;8V(k&sErJti4(z(;`ef(?Yc;Z5L!<z>ArV%CV%?miuVOdO`Z6W{ZUMj%aF6(^XIA3 zekqe7x0@{!=mG(bf#^dPhCizS?jTLv9l65uke|n}!sc3IlEpo>J10BH&=#r_iL8}Q zr4DSxj{*V${Dsyf*GcC)i(#<4e0*eraB_CVx9{GyUrj15%tQ9}^jM6OS{oSrdNET! zDvLTjJuRc8Wz_^n*><7&r00GL{Jk=%{`uafBKXi!gw=iXy!&DOpQj#E)l`O%VP+Tij2F3kqXzHvEKea$s` z;bOBUx=;kY$ctS6JEmSs+2gy#AxfUX@?3gLBcFAuzfPcqavvJcvo?vRO|;* z#pL{s$PY{aMV3nv_Bc5WYliZW_kMIjf}#||c?y8N9Lplkvk!YSGc#uzUjGU_7K1?- zR1ZfrPG)%??v!7gZMS4yXJTTap?U7?OvauFK<`|gE1~h7TeskA39L&Tw>dZn3E+HY zjTR*j>G($ldNESz$CSd1ee)VDg-4HK(UD=v+kMG`QA|?ABib|phDJu8^YaDi*jsKm z&o%4m=~y0grgo-mJ;H z3X1X`k~cP{%elW7i__!-E;MgvUGngHTcN*-Z<2n-MI&#AF2v{6U=6jy>-9yZ3-F+r z)lB*J{rfa&KW{U$SuoZdYALxzMf8k}fK?-9mlqKc;o>R+lqafzdWi#X#aAwjl(KNp z@nHxdv|rW$o&OG}-)yD{N7n0oXn}g`**iwcU*GXG^W*#vDVkqj9r|KYW0A|u#H3DT znF=c?pk-z@*V4)YP^X3S+G$;_>*|NvT2XgFwjh4c54au5^*k03zt=M{KAFeZ^IMO8 z#YcIN*KPlRisH-_uxSVToOcghoBgk9T3dexLtju}_wwaSb#+o2e>hAFU|bR+qSh%{ zs9ENxeCg{nGM)>+xJeG>?rk&gCkrX-88wq@JH^F|i)vJ5mh6{hLEDdCUHhBT@4Hg78PtS? zgnP+}M0>dZdPm{qJ7l5S9~i{4K`RN^&lO_4Q#xpWd(p*f{|vS_3j1lRr>d;W_!ojJ zXX>|?GwVizPv@O_HOg0X4^Il5|aD>ZW>+m0H;5ba`Cd_au^uGzR6K91_A! z6;qiWMfNk>t+#iqgBR)vUw20Dz0L9?-8NF|adBK#e{tK|8}SeovOR-y%Md`inAq6& zk!rVYeTE&2MKiOrv1L!zskrO}q0g_UgCcR4i%VyY8E_1spL&2aM!l)~1gNZ?dnEh6 zQz2EGs1eCoBwv0_38r!cWfvE#KFY6AZ^2g4cba~=q^f0yzn!K|Gus+Inex%W{=Q5} z@OiSJZBl-GXlZFFzyK9gGR^ocHkL5vJ)nM;w)2mY^pTZ&p#AT)7MNTv-|HE=y1IAW z5la-cNd`_y_BNfNn@<-PJ3Bl3VtbdT>s*JNVx3tZMWaK^D|QN*J2cd7I#+Euv?JV0 zFPBK6kn2j{*RP-2F+4)~U3)(?Hf}OUqR(x$M~^sISp(~EWSYUk0{Y9#p+2(Zd@uLV z0(a2JhB(jS=t@{{{CPCEgHf6P%A|APqaERqk?@fbt1-6jW3ag04kO+^IMvl^UFTC( z)A3fe$XMur;MCzO>FQSQjJcX6TH2>{y{8u(aI&*206XnaPoGGe@scT^{~hFskyURl zTPw-Xz%}OgsWKH$grsqZw^aagU;$mWRyYeJ^i-gb)ALKl=*~sjFZT0koDd+b(C~%x zm{dQ1{v6(4Q^(eF0ZTDixyOfR^whf zoK4pK3;GlMfQm)8B~#X4d3<8R%hPj6T`)+O8gK`{kP~mOo2{eojJDttjq<@X`NE5f zi$5?*GohH3db3@2P-M1bB*WmG-Hrsj;yH=mmMM;*JAVEZ`I4V`AF*hGk9y!5vmXh4(6+fad_j6}w{O^Vvo*2H)NUp6^l2gXEP+1$hOp zb^LVN3RE#;Dv(ZhN(#TXCIj%Q`uOqF^w)R=SHQ6Hos^|yXraJSX5Wt=iq3?RlYa#B zSUu`2FwkXDsZe3lvN_0={D%TIte{rsD&jiD2^!doh@8r^Yay^hI3jzDg0A%OD3dZsi6b`ZWJA-n3cf zXGbIF)KVTycvW7vS2UT7Q5$od0zxp;%|@KWfRQY$`mi~+O&4s*cfli$E>gO(5Nw;l zCO7BCnaZ!iN1C|TZb}r$T9!ww0xCmOX*~He@B^R><~dwQ&W<*%0FPV1dYB8|* z3J`8LA#S%O!fSIMNIrNF92TsX>kx>;KqHx*T*c@nb{%nYN|LIKkB^^Ze1A|N2#$x@ zSrD^zYF#M80W~K-kFeDZLW*pj3Vb^vB64>*Vu|O6)|6+E;_nEA=)UNeESavO|06tH zF__07ehuNw$Eq;a4#$R*<(yzx3%sW@N2){z*!(qEhblk1KiY`b2Pb^H6`l6b9H%}E z#b-uquBzh?boQYupKX?%r~GA2rK!ag^35-vj21Opr$CS?=z;lbeQaX%zA$ zebImw*C%Ho?|}Ig$z>{gm81no-N}TE=)`Y;z^RRx0)-}&1;m2cg?&B?_Dy2KGK@+Am38(>f{`rxIrn4R6-pCcn3 z&JHs6c>C76Iy=S?dT`~W?Kj7_cmIrVfF6;Wnp#mR=p1XM7nX-9T%ZWwpX(JdzX56& z=%D}2w$vK_n8Pe?XZ)7c$k)DD6?S1^eKoZ&<>fJ$L-dkA z_upC?Bm6~rd18)#5UGC&rRGEnAX?nl6AXQJrRr8O$1AiFc`#SG3~45q0LS>ZZ%ZTj z8sO1n!JT5(>wYhc*|6qClk9i(=V;^a>@1au2}l0oC?LlJ>PlK#`d{zeDQqdeWtBd7 z6W9l!sd~LDPdHVrbT1ww2*;eqht!LkV#Cb6J;+NZ`R7!6vPKfBV<{!>fw87M39=Hey3W5y+m|~Su z_^)5)4IVrgoob~xcOIZYPC%q-Pwn;Dap%+?a^|;lw9fN(@QYi3u&Mg;hH_sH_S5zAZ&%s zC98;J2^CTNM`5&BdVZ(`lXOr2v8}Q zU!q{36q*1EaB3>Hb9i_dcrol99M;#@=YVvEfER$03D}Wy@0>}wMBM+)OLz+&h=jyt zK;3>{uku8fVth0-PMI;U<+|rShgrJX@1D+V{EE`hv?kiVBTETQH^y671_cht#N_1U zy8Qro0xB7!4hIAT09-dUImsmM_O@DK6Tr@o1i2SnZ|p+Z-3mIl?0Xt}LLbjw154?7 zt@|$8+@pH|%HOfIq@*n@M3-CoE9x~x7BOZbd}9G2*wfQvV`CH33HU7_14{@A#d8C7 zjMuPies0Vcj5j;*`~bVak450**pAx>?)gplt2F9LfTgr9uzR=)a6x}nTh|eeeS|(W zN4x`Vo~sn&TVxi5#!3KB`_-NJjx>gcXYO39Z6%ou6blug!As_4P%^*b6Pn^7Jk-P*~c2O*Yl*N%7GL8-52Z<|aM0Mx{W#r}tTm zg~4h874V;kcPrmGjO~&T4vW?BMCEK0Jxv za?fc{t<`rx?0{eI~ zhH#dlGK`MXl#R%g`or~;y8{4QBzZEaQpKL zg**f-9#HNCYYMIajdZvS*i711XXf(V9yGk+aha-eqSZYb(`I+rbX48@a4P`$BRUnv zEX-=)EQ^yWvjtoobVC}!eXOBmn@yIZzM5!VAHnBX6!>eB<$_}mK{t-tHIg1Jh;Hw; z^rJo*43LVXnOwF1Ns4?~PL5gPr}tUB|6+o|6`ez238~gUuXX>gmk0A<`1|*I zIy%4i6PO4Nv!UGwE?hf@&i)T{woApW3WW3qHCsTk;6ns=Cf55It2tnq{|U;Qj=pLzsfv`)k_&zGuXC*>*U3Ti)!3X&S)LIqJMYd!Di3=RFoK^)u)9pAI|u35&)6 zJ-fF+so&U1!wJ8`f9)2?a>NNOspXMgQ+H?NS9B;-*p8P(yndfDE~6)YcXU*$tK^Gk z(bp38KVfZVny71Rm^!O{_3`7!;6T{l+e_lIS8swb-$-2%el;8bVlIU5N0KSf=qVo- zFdh8kVX`2Iq%{&6(|d5MIeeseHOcc!+9H+IM|cSHPT&hH4}mUg(K_>PxN=l+c1m)? zx*YDBpyVP$UYLoNkChPYg5a_)6)+|zCzqF(FXfz{W;Mgn4uqp~ESJw+b%%{jR!3+5 z(nuVf9arUV8uWWY=3d>vCfu0uVKx6(>VUbwjMyDC&+as-xI zE!jc7>>)z)2LgklL2$}{V=Jo?7}I-A6nDr={bY`&NnoP_F*YwC+kqxxfx}fn=%2Wj zJ9Z?}a)V3|Yvmm}=rUf?%@acd1*tEM%7VKXOIbaOsHmZV!7*s{`Y0hGCD5-&P9VB< z87JYM6g1H?BKXHvY71l;fI(N@F$41Kq8kcgHC_M$Y2D#tuzfzwjCmX5?n*zsnp*$c zw?r%Nw&e``U&+Gi69g{(#EE8)r*6n%vS)BcAUlHgqrb2388CFZ9r_pE9U9&cx726! zE8pd#Re9ZE)>J>#Vncj+>FC5n4o*%ZZSD3ayF4J|5f}f=O*c-cib23XRuB_U>26e0 zSERzF{7*b3S{z_`Wh0}y#>UjTAi_{2Yy5H4XIyv|g8y+4W5pmVE^VdI4Ve!xZF7$Ol1pl@K zznE?0ynQ=9Ev^5_E?Faj;}=pdj`HH4Xp5zSnjNRVL0BdEIXJ-eQ8L~+AfD|L+-q+D$uH1iYwf$99NpdIzg@pWo@v9<@ z=UW58`j!wEN1mR}wD>dM19$@Hz=D?T zAwYn=K!2>IHb^@A!#uw{+=N|BOH+VQ%TCy^P z0%LyUT&YXrQ%Kk4sKwP2@VPZa(VB92QEKn;GElRqu06r zE@u$Y75?%-+Onu9vYhl1XvI(V4y#y9BcXM)8t=Boaz;_)zcKz5pleOT@@L_|1lUI_ zQx2|MRb3t01BdI{+1UYLrlQgdES+H?A>BWI0FhB{js>)?fb3oHVA8a6bOgZw1U4=~ zG`W-;!PBQtEnJcehnPv>B>eFW zA0~hY6Cl;rT`~sI>o!%uRT|n6fc>GoT)+wqB(XUbpum-ul0^~uk<`uW=;;YP>%Km& z;p^K7$exf8JU1Tn7LE+mrl+@eAqs55a5H5CVEpI_i+FKKg{0OU7F!o3Kw^-&$`s73sf z|FRoivekx=WCUMe6KvPKm|-ij;BItXRd;sYzHil~q^#_|G5)bb4YoQ$AX zuJ*ZuL&suoBA*NjG*7u|*4DRm6XN4vLSYx6{8c$k9reU>@yD{9aUjAj#=e-y%YO^RB?Oy9wSf7aaxlCX}mMFgG{|G!E!N*j;Y!Tw%=kotTWl;lBGT*9b=k)uH%@2D-YfS*xq7v7Nxv zdtGGJ;V{@1ST<@l{4bpFDy1#k@koz@21Q0oVX*GAta9K8r;OYJguF8Ej3%|O8=wuu zW$A)PvdrcYVLz^^`T^Iu41${5@H!=>0_NFC{vN3O$%>%GHJ3pJfM)pX*Ih90Qj(IH zYG7p-6&9``2&wqYrb|pEXbR6jC%>A+O<1-gDiULk-&33S{7pL-dbeK= z25noh)kXt10}XE;R$ypINB~z2m}*en*KjTFX&M{%JSU?%Iy^i)I+A|1L{xz7tgVC1 zFW`8B808{8{%Y~iZ*3^cUl`S{%$u7UAFt3v;SLOk`>TVo0^N-DRKTh6?VJ27`U_BD z7GT}RYo_z8-D$nRc^W2=Jbc4q6>t=AwmvIZ7<wl&|$%6?4*MeFc*Jb)DZ`?;nw@2P3F0E)z?f8tIaDBYA4< zylS7`NjE|@0}vXj5H^rjRH)GKpB2XWQJ9krM2SnfC#YQ?O zf1Zmd1|ZBorDpvrzG(a0UbO2~mm>tDQT+|czWoCl(gdqVmG-BEQwR}STPw*iMcHlF{ORPS*%T;SM3nYW2Z;U`p z3a?5O1v%DP^C+2PpD6QyqaiMkh6}=RyV5)M;!QiEzhN>wJl|%SnYSLhD=aMd@tUdk zF*DAa0eh+G72bfLv*|Zf6pUFzIz@w7g1<8PCpujV<L#wf-jL>}f!*%y1J25w>tz zznOe%V)X?k%gCc)?t}BDt>ao&Ol8PNPfdC5o2`FleC!?-PvBUWw%*oEzduDb?7!9> zu45{Ll2fFMRm&6%*up06Noxo!2MNPA-|RV-rl%#ywiC^}^`;Y*b`=FLq|3toRICa+ z9MK;<$b{OLm15G2A50tRf%LSI>u_wga~Hn5|2Hpp%?GjCG(S?WW};0a*-pfu)??+g zA*WxAs}nq7d-^8&@!)9$urF#!eJ@FUs)=^Vx&MsE5IU01K-5RldUgxQQG~`}Gwr_F zNkz~hzv(e4xH%Jcy%*0^3>|7_v;R#CY{8*ESf8+1|Qb9OhMrhdJe+~2CR~JAG4P&wgzdZ5( z7F=G7x$z%A67ex}_@~PRpV=CMn%cQ#|2ZoL*7q!A)88w+cezWzPf%PtB;*TMp2r9) zSEAGegg__qH*=nmdcAO@h8gpbclyjTPFGYeZvgxX#U=RHJx)Py6aKcaryD%Y0%`P) z|B*&tgWa{YmH>`)J4{6Ix~Nq`wa2@6uUts~i}&m=0`(5i0hyiGn0Y)f+dzCB?$TYg%M&? zI=q9iF}PUphbq00ZJF zGt$z%+hYoU09mUk_$+gbCYDue!ED_MCzE!9QVBBd?2T@g$}&^Q;aa-@)EX|*?$$Q~ zS*vl@?JQ!+f8R}?U4n)0+k&4gC#PBGm5O9~G z-JdE=GN^U^fnX$H(zw`t)2DloAOxBH`xEYpXiS*J-w6k!0@<#DzUEl>E8u$DBZH!~ z)6%SsPIQ!3B!iRnteodS%Z}dhYZ&_OpWLz|WA~Pzr#QQkmv_5+ zmD@7Y`}jfr?hPu}i8#!A4J(tQ`^S^3n?UK}P1z=?AnpPb=jHNyBvpt85@TV{-hFau zK)`Ys(fj)*Y~OdqiCm*a@QKBWKg*=@WOgJ)gc-G$wTZ(Ad>jhWAXV?RTY%v6<4Ts^ z;MKgzPmB}vy_)DJ8iaRS9dX5(#s&W~wDnr(@}UV#@^E4*LW{&)Vz z$B7XagFPQ)(f4~?uJ=(|kRQ3O7~Z&P(U?)?a~SMEn5Rm2Eif&F$0f(39gsAb-@|dw zfzIkq_fAkTi~sC-`+Ag*>hH7r6)-E#?fd0R*Tr`>AgHfi{!Ta`3 zsiwQ!M&;nOkgrF;S^J(&h^=I<+*+lY> zmR9R2;-T}DR@K8yNvTqyZsa)1YV*({88QTsb9F+nnAMf$rnknZGx4y)GBT~ES$;(J ziQV-t*dOq_`~803@8^5p&-?v;eNLS{D}T{G4ZZQx6@Xs+NhAyg;~@?8F;)TL@=S$O zjMoprK6Vo zD>JAZ%AA~>w39LAha4;<-jH8)b#dt+p*p0KlOX&wa6z zV!Lw$0x0e2>FKm$R2I|e8N3#|vb?-pDD*1O7%Rqkp4l?J5B!+EvDv`*hA^@F#0KW$(7=D?U_;a_~AjBuP_FI4#efxl7DAZyogzBV(8>I-4b;eaK zT<(?V=nK>%=he`D-SZ{9gy9!}b>*sF+lOk@yGmMspNR&r(;g1z$uc*dLa7PY*V8L{ zwQPr92&5M5zEx{@gOEF%uYqKL{U>vCbJbITJp_`;qV8fyu>ZHrOA8#xdl%F z0Tme;xqdfL?e?p*KkaDB+m$DnAw)V_fVsK_5qm`(%DQF0yW8dYXD$K${^DlBqv2r8 zgmfSy9y%4~9uX=7w&U1q_CDYN4+tU$1VP#w!810vwAHxds;I$4V;>?kPZZ^lEOh>H ziY^_MYEi4bm4va1`o4IatW_-YIK90Y!1=BJHE^ep%@_(R6Ok@yt(hKQDN2g68=jW;=8dFL7}{`8P2>YF+)St1V9O8zLNsb zQ=3wiL@4a+>;w!ND@WVFXoj3KUoGInRPbbIYXGFCwV=SkZ~nABs3~=pXI&ux9LeGG z!#9v)_rl|=yVhsIwasaQ+0ylnKj-_MLp5OVe6J(q@Ho#iRz41qIV#br-CcUJjCooj z`q~zVT{pMGNjm)C`|O#*owdt&V>*2YSOPqGPWNw(jUiThhAXBax8hr}xVQ-V7d~gx zl4{m6oVlv+R99R30E5i(VvpaRFKRNg@xJsMNFXJprGUZ^cOk#{)Ak^re{*#093yT}`(W1u}&jgMFxPDzAl{&y^v6oeq~d&wqZ-JnLi?-WRkU`c;;JvVu3Q*VQ} zq3rpA!}*4(%ZnR%&*J?s2BaAh6aQ%Rphxt^siCpv5)NwN|2Xd3#$Ne!)^FD?2?tp) PvQFm$DafDamr(pKQO?#D literal 0 HcmV?d00001 diff --git a/neps/assets/nep-0568/NEP-SplitState.png b/neps/assets/nep-0568/NEP-SplitState.png new file mode 100644 index 0000000000000000000000000000000000000000..a160a5949b648e3dd00eeecc3d9f834f0b23f068 GIT binary patch literal 81729 zcmZU)19)A*_XQd?xv_2AW@Fp7ZKp{ZJ86=}wr#s{V>DJ{oA0K-zutTKzLPumoZ&t* zd)8iSZiJ$|1Uw853koE_1@YM-W<9X zqP^PxGdItY>U(bU@b_F|5*(d$5?tx(hZ~xv!@k{@?Qx$tMnp?gv|(4qfbT{b&U7SG zjC7IRl&%Gs{8$6I`C78l6sT<99qg)(>G^aWhM(xw;P@^G<=xUtQvOOn6}9x-&@Uv9 zYU3P1xoMgd!+U-o997G3(x)U~qDH&I_?tQIlc|<6eauSDE$u6}RE1rF}p-^OgK#~X&*=iQeR5DcrZ!Y!8f>L)gA|DXSo~kL1;I-^3L=q+> z@skSy90%+YH4Lc`;l|+O-DrSTJz!KLw`=Wk-fpWafzSZc z01(hHD-Z}^3KV$Z051>_@E@TdP{2EN1jBGC`5YSBtn(^XSWmdC`wj={*(!Ptz!)6Vg;2ne4i4=`zG=4wRbX=iKi z!sE$L@-GJuF#UO(k%Z`97FQd75=}WpA`u5?Ga?QKCI%)F0T?18B0gtRa~@?;@&76g z{KijW>FVmp!^r62;lbd+%HZH^!N|YUB;@7iMvrgyGLk*@?>xP*{~JJ$#<$XW37S@R0&LcjfEmy7}3LRFt5f4F3<9 z#9C4>%&EUQVuc=L_Br2p4<4tL4&F8(zqK8rD9g+_ay-zjuC5|oa6)-Zx5Hwr!sl|f z-g*Fs+Y2Ef2wR5w3U+*M1MD>_V&L;n$kWbxR6^ZnkUoOGWu&gy^qCd5GhFk2hYQ=0 zveQJ*_d!^IGoY`$G&z-!PXV?9^+NqOKLHdVoX`{et>@&(Sw&MVynj{LG|3{rxlCP) zHPbGWY0?Ep&kk=Muc5A%0IDk+NAI`r@qFtPT>oM4^T<3r1%>aG!IHo4k{HldSnLoq zxHMO*Wj(u+i$Zi6&2S!*uVB7-olxuFpuj7D;~bDF(HD0TgZ@^3G?IFbKk!ub_cph0 zo>Eo{VwHDQNl`C6GZHUzSC{99lW0|B-h?$l6+twE_ ztf;K1>FMs!)h4SZ>#gF3*QAcb7bx9t8!eZ~MG9N7sr0&lvrzWK@9DyPE{}**73IB% zi|-)Y1bGAH3OSX?sW5By&{ow5B1izUH{0w6pN&mIr*y51`;~2Ao)fCTT{yR>0Am0eNk1eGS!>8tK zB<-}}P>}bu(r(-MZ-41FM$(d6Z{b`p>MWttyOSms3*ou@d0`v#QvdNUJsiCoG;u*X z?E2m}>xZ6!%}LBn?PvOBEd&KYbgrU}3UZCBO{jO=ZTq>)w?Ew_7%S9pO$6=Yxdsq4 zRJq;W*X)^7ydhLMl54mgh$MU@FZ?Bc<3`CxJ&*Qe9Gji{#;j%k-ypy zFN7P$x~Y+5ax}BC)RLe(Wc$Iksd1;`^WnOHAJBl#N6z$f%;z_^v=nq#fwIT2OVY>- zTT9W5S&!GU*-~>%FT+V#YYv%~>Vge>E4KT^^pXkXc6NjPk+%~Glj0@FcuCDqU|IaF z{YS@%>ISJ9-yi#7WHU~Vl>?{3S}#uAP?J+CKx&gDhFEe{MWy zSofoC0EUCEjJ7W)Rv=8KP3mzpl#q0ovVEz^eW!rKo%FYr(g=c2K@;79PhOE1lfR{;&S`%)%MX3J@ zI3ff8Se>mg9FOKzRn3PG#nu)zgm;G@Uc&T;ugFi?ReUzWo8~KJ&Js8~(;{H}spR;vd|_JWfIfUG1$b$sHZo1&iLP*{T|&0gLSh2) zG8MlpUnNDYfP=*Y;>CQ0Ie;t!M6mU-=m9k(8kh~~;*YrH#ZquMH~680)o{8=Q2Rp!0btt%w#h5(vEX$GUjH#@G zM%>S~T_~p4R7kKAzfH~S()AiIxzvZ!#v(N}Be%8Lv|g}uBo7%21FVeP>R0tt|7-3O zji;w(@}1LlajKl7lj~q|K5e8^LEOOYQjjR{#RNgvcH%Ltrlk*S7owFgd$e$?ht>pi znQadC>r#$Bm}_2;Z;|}s4###hXMX;GsULT_Eo6n`@k$ojq=XVNQNbFk^GEW@9K_G9X zHuLI5RmLcsq@mgaW{n#%w`09!4f=c0Uh^WPoMzaDr@x6O-D@2tlLrh*U3d{=^2q%3JSKw=(=Q)X;r9q9Prh^C5ulxsrz zo$p#fmegHc;|>vmhwDmIWp-@?43Xj6Tk@;(mW2bj!3%8s$aE+gbbq{_vN%Tt@%Q;F7bG&(~sEPPNuW3cacQ(H)qy#C;prgLKOl&?IHTMz$pK z@bog__RZ`R=QFF-WX7{QOEYKpU=ik~FqsuTPVE(V6~`@Gp(|%FZcmdJ^%A|)F%p4Q z6l_r*J+w1hxya*uwKSQ!Et7vQwXV!pFP6@Dbr6*J$U|m=16{dO1-S(phfBf2bp#H<);MXA%6we-vsSulfZ$gDvuypE$!20I(4Uxe1gB z_P3EZ*^(#Nn<0Ua$t|!dc?!VzDWqP7xb(U12Gy#X+vKZ~N%=u+SZyf!oCKMxDB7{0 zKIb;n$V^p8eZJ^@6l-fij7dslV-CUmOcq-M7TFz?Eo7clGL0BWGx@eZc~eJ`<{fd`CA8EQem`a*@vN~_8{lH9Vt%=*ufH!KyhVgv{U(&9tS1b zf2Hd#D8hd=_njZ&0*-Zoi!0Wf{)bt48lz7!AAV^o%i&i62uriys%J#IIb%St~K!Z}*i$~1kK;<}*y&)r3dfb!XiH_*87ph}IJb0nI) zFt(UGup6RZek8`oQ`=X*09Vkkuru39o=X|MN7#?q#>}U7C_3Sk&emt|LOq`t3zC$`o|U!JBq7xeJFEc#*LTAZ!S=!w%|!}EdTd-Oq`{o2 zr3rb^>PcK)uvN z{xWdl@(fK;XEpUthV)V}(*P7*Vs*$VC_Fv<7^9l+$-ku;8X8bizWLbBiB(FGE;@oEi?09GitYSVK%iK;`q;<4&Gk#@dY!7`7=|x|}Vu1Gppfr5`Q~ zm#Q!2A*ZqdM7<^Y2c}S8lppe{f^agBsgxbW^tfL$597-3xWz>@VavpWoLy<_nOYS2 ziZP0RvR&z&a9(%oa8>2eWe5xVi^$ly&|~(Xr85X)-n(LA&#tiQ+{*VAA$LMW;P-Wl|Z>KwyUY82k6q(Ao#Ev4^94tnP88^Yb9`# zFqvwYkrr@;b^o=YDzK(r&?L&htsv2xJ|rRXwf(lo%y=;!{s>k(M45qYqF&N6oNfGVDMkI$~gV7E{%t6 z#bF_H8&k46WN9Tk9%|;o%MA&Mn5%b%WG& z9ox|9O+w?A5*&ya%L4^tMh5798R6`qglj7QIh9+9nw~{4Ho1jtRUTMv2**%^SP-Y^ zpeCX0arjEzMV|gOM0=%c>3(2n^$V;B237Wa_TAC+XE5r7lzM49lYew%HU7Od{uoRLvEJLhm4PW#W% z*nq?;RD$deq@&Wv%Z|5?n6;HQPo{4t@oOtn2U$Z667K*VR-aeXUBn}M3AuO%@z0P@ zJj~UVHYw}RX|Jbhg6)UmDlvoRfhM;d`Do{QO~!jQv*z*2`mGc@X_`NwD9Fwx1scwd z&dpIWeraXZU}!14EA^p7KK^>%@gf^oA_^!Wq{^sGu|ee+v%aq0cQ{Co`8mqhMA#e} z`XgcMfMERIktT)(dPJy^$Z-~t0f67k-)Si*#X5@KJlsdVlM4muE^Fn){bO% zR5X%QzAv|vjty=q3EkEA=Bb$Wk!vpWdU3lMmG(!iOO5RnEi%JMu3G1X|2_{FbFWvX2nZqNqh!v#k97pDK9Sy76{q+3n zLqFf^N>$CZ?N(6?$q}WUapM(1RZ1fmEJ4OzM~ie!07ERa_zySXLRe>rJeQ3mBi zc1C4eW%ql258S8Y3ssr((0O5$W4(iNw;c+R+Kgg^w7jwlDy(f}o8A|raBQUuI4D2p zTj+oMiy$deAX2>B)cS^=7U@7)YC>xC_j2K6GK-O9y;jZmV5ox)ClmD*2pE`9#8%kKoC}nUFG* z^1ZArzgC;aq-?fbJ59&NZqibhY&}%G7XXl?nHSeqAoHuEYv%a5xQgEkXg#JWCIBlg zE*`UGB*i3GfP$Y8yM=FQa2@1Lq6ryg zR`0T;a&1BUCYfB^SpXI~D6(=58rkkH?4Oz0#mLoS?N0x9kxr*$xq{{b+E3TW24)fw zuvi#N{#-Ect|NgdQJ!YS`$cU#_gKe=rg-2y{|3>6G;b4#eh@*8?j4)m)OE}HemXIgIvUDPF);HF@3zOdD@xkg zo%G1fOw6->u@~@53z`2fI?S4U+%c?Bc7X{NrnbU+hds2k{EkpSBf_;p*FbF7s^kZaJ1^fy1*OQ z;UDZNpYhX%D%z4A=n3Lr@*7AHrv9du@LezV3QhdI>E(EK1cZ<{fyBf_Kcnr{d*xyB zuG>x)0ukhm?=DNYU*O4e`b=j78_I+zyE{P!rIOO>=uxV8N}38M*pi);mvbGr~9j_ybSjx z#8&dFHL;sytz2bdXJ&xeOmB#-Ow!VAH99(cY%1a;*5sk(F31Z7RO-UdvLPqq81>KT z=7daDP2cB#{zSN^woikT`TWxx-RM<+=cTsAxp(%G{L=ZdpAA`=Th{9zc8(SR6Jr-^ zRrif4d_PYejoIaIxKPF`FW5{RUC+l*rhK?lE3hg4YHh}E8G4wzna)+BeamCLt(R1( zA{C;M9Kk&G6~pMfuZy6fXC62sqoV>KgvGq&f}rLKvd8!X7h|h);TB-5oP;w6tX6Zv zn5L1$A{sajIDmX$u$F^;Q#4kK1<(e)dlG`JN*PvtmM4}PLFfp^b>{QTHyMgNk>mqo z{ntDR{;d*R>Jl}gzJho?Ci$u^5u3k*rG~pz8sn{PMMsqU^0Do93Q_H%6SLSW>2MA0 zl0&Y&HTGEGGf6;NY6D4GEhXzri_3_dDhM;OxCT|yTuL1NQfxJjOIM8hn5`qPPkxf zT(+%u1I6;9kp6P2NFXSz@QNV%79z**aYPCKC0^x)9BGIHhV@Ev?ks+-6;&OpL@+~1 zNrn$Zcgp^UA2C#5tAWO%+xUORCMPGb;7^0nzRhi$ngk*wsy8-6Qh^z3!GY_6Axrq< zIK8^(j{1A;DObj7b2n!#R@p!lbAe&FcBJ95;lh5$Wcj|Sn)YA;1td=5+f?wd`F&%0 ze~}=aaDtzd)D&GG5P}W^V~a+vVPo)~C9~U4Zt&1Vz?wdi=vZ1i`ib}o@Zc;~0)14% z(}^b3lau(hCnpSWo$J0&!x3Vs;)XO_szZt951NpjKtrSCiSaLt9wrFs8Dv_ zEMKE3!Qq^+ciK1QD0FgNI~DdRv`k?MeM-9O(>)4Sf{bDZ#ak|uC7< zd{f_t<_j8)tUBxvdo@y~E97{p_n*xeLs8nQdf$KEn~Tx!|HTT&%CRg=E}DCFC}-)2 zs9lu<=e&{-c$YBjSM@c8m@}&W*X=xq&vOXK%f#=w{{<7g1u9AW0xl{ z%_GI2D}TUoe4Agh^q(Jf$2`1>)_@>;c(9=~7@4x+b$pqSYqeXohTwwIrGz0Y-DK%u zrD3_8*tUq*Q2xX_CCA9KPM`AwbO7lPL*8uQaeF_m_H3Ef(zYTDb7`2UMNpaRfXew} zv6*9@W8TDN4RX+P*l;dduF+7_`mEfD3GsxAz7PSn`^v3e+HQ3QG|dsHK#?}^ffr4@ z=;_*N)3akUQoO|Ma#*lAi?$&>o1Kq`0c&LflhbbX{hz(?xDgB#U;*f#H=@?d?%11}_waNSANgJTTZ!i3-||YU0S@HdA#u}wkp>%2E@TiChn1Y9E-S88iEyB! z6ly?a#sWbg$b+#nNN$bpeXhO)`3QO8L=x5-=CUP`k)(T0AJGkkds zN@{|XT{MLCp6WsQu8E&{e4G+a#GqP!rU@;7~62M;6wpfU`FDNtB3Dc{Q7x}6X#!8B)4 zJVvshktyDSBDrA}!#zm~fAwO<-mGc`l;5F*o#e6@#GCe(D&vBgEA^TJk@U42{go>_jCpGIbZ z?t^RAUev~A_)uSZeU(gH@W4u13Pzv%h6TvvF#Zft7G%fm_S#4bjQ5!ClrS6j`sja2 z71!oo$vyt0xz-{D2~@uOdtRa9O)SSXG21wOR6<#f*5WdSYf|BN1eKV-Osiv;y*r;y zFAsoJ*#TE>zc5N5FRM$; z>3O=gtRpoPR~Dq~4Jb-~dpm8SJeM-LX-?AAZkeqI>DI%U8QVCE zKv4?|!`H;$S;mLfPl1(MGdrZB3-&zw?o0Tn{-cr@AbEBXbe?n9Mb7%d@2AvX z%-s&e>p;2`eQKsk7NeVl=NlzDdEev)`iaRR>2K~!jLDKJ@~smvu4To-KvTX_ zObNA#&@bQpn)Fl;P`N6(0$!%|^iM>tC{Y{}bQD(0;652ZikicDg~_-JsieOXIkz~& zvE>!39;->v4Q4Md&z0e0-@AJUiW68CENE?ob6~kj3(y(j=Glp~`e7Nk*wU%k*I7uF_n?Jda;l_3dn6 zD&NqNRq?Cvn8M7XiCT#8#(9c8v0r#4K8inSt~fy%AJ}V~Ye|^*wi>ywVJ+9osN9s7 zCA-tx1%+{!efx-YH^&Hj8h`Ix)Rz>l;lghtdo47+;$CX{0eR_iBBH>dkvWC@LqZiVtff>mAg zg`_Fo{KGxBIM6#CwA5+Hl?qhN>DI=^vP#2F(&t|oA0#1opd+M4Z16=Bwa&X2W-#P0 z-MWXoc~)>o6)k5fzGob$5~0D9XRnLd)t%F=8jj|b+CSkMT}V)^TItq2*Sf@9n1MU69q(O<98Ph!(vbS*vZ-8VDbpV4G`?~i8f--~ z-IxR+{NGSk5X0DDYO>U8zc)Y-q2fS&)HgJRvah0$WQs&@UYzGBg_IdMs^1_rrM9lF z!}lMPsH0H7qdfDw_sSoFSCjtk2zGw6cr)rGA>!?de1>^0Z<R$Q8o zES~b^iOD$-UXd5zav6f<%rcMcj$|z>H}1^8a^K+{W0Z$2^CK?mS^6YEErnr^c+Zax z&1&Omen!!Lk@DPrywN?CMc$pH<1{^EDhx0FwKf5_@3s$cg=gDC^NSWYoF&PvZ%8)% zv0n^*1O6A0YifQLjc?kEas_K>^EP>|rZji(qUuq)wK^2}|ZC<87jnZSYC90oPxwpq~qPWLnzf=Fl`z{~JR3Lx_*3%Pqyk zZH&svNkxf)n2L@{kFd1{Ul?iT=~;q)dhAg*vkJa*P*l#zs;Ri2`?82j_5NKv-+iUn!w>f{G6 zbk{rGYQ(+EkuoJCBW(-{rU;%YY)X76UpX{jAt5-yUbh7kmW~kCQiquh6)Aog>eC-F4-jaFdZf<*awN177m#!Mucv z;mLVWYc%Uv-o*m;r%1Wx#c7R-{i6rAyS^e{wc8-ge}Xv>fa~+aL9~J@R9B45F55y1 zvFGOJ2^W3SL23lK34-MC3x^&E6y4s3-;-JB-r_W@`T(l3-vew_vgFMb-_%G^^5BT-+f20-nA&Go-kfqS1JH>;3U%)b7>HtO@VX zSI0%?I);}tMXSn{839Z?(Haj=q65Q=dnvCiW| zTZZ))$2*dzx~(fsd!9#I2+m~^TXkRM)(}X448kfH&5W(#aQLdZgi+y433qq-pm2$T zHvL?CaS;x@y9!7_ZM9_n%LHCTR-l^dvz%MME*YSXFNayZQH_dBy>1cH+bY;QwTjSp zF2qJiFfd#i^L8&~rH`HWHR*?bcIhdakBm>fo+=g_ z*|Fl+ulASP*xMr2C5){dT^&MpM?sL{KpGj`gzY>Q$t4#;7FtYB%8z+AlK1LW0QIzE zZ7AgV*4xmE4e7p(WGFTevmcT2Tjc?oIy`;8STYFx=ugsjr&YoQ#l2f6oIqS;M-r9c z=R-HotH&ssKQYAbcASfb%LbG4^UcjIeaV4@{~WigPu}Rp!Z5zolNDqi^STXBw(hH& zvceO#tM7DBaC3Qaw|ay1EreNW4t$Dc=t{i6dMjXAQ=9db1ngz1gOt4s|Ic)~u{X%& zT~jZ3YdkYhayLJZRgR|D@7OfF^UQak>Nq zjg8Hv)GTEy<9T%+ zyTOoVv?wRJyGZM}-zj!(5m0Rl`Y*MOMw%8({}t|WFOyroFx5zZ4;|vsFT83zs*YjG z(@zl;t1#<_YAkUzanQ91yFR0<95BQT3J`sCNbgQ4_VnO$Im9-PCRNBpj?|Pyft}ht zYj7|T+7g~XBuH%<08Ad!08gVqf--7@WmW0c(Ru;p%2?5nn{t-~wI4C)b3bW^eoHno zqo2$JeRtcjELO;_YWM3i_*PE~xDUf(-}o6H^z+xypgGE4W!y3!Gmr(*;_ks;7QS$0 zyWtHj6#c585_I=M*ll2gN>&0{OqSJx2wr;JZbQr~&O`7QxX+Zfe>Kx6*61CPEWuHiwf$@Yr5aRASh5-N(i zLSlNsj=CzJV~>FV6Q;de%dS3rgI$pgI@Su1Z`ch-=mhzS$$J3NBB8IAJqCY1G{F`$ ziuldNUyN2|Lo-(7C@fyCR0ptj3P}i0moXYw?Hjy#`VMdYGBpI$3?;V|Hl42Pf$>103>Rbg8pEL%gU7rGCc>!(}vNin0BiXXbgqoWf_XcnR0v&eE9_fCYEfpP9n_ZVD; zV*Y8)?U4E7y6o`ig6Rd-=*auqJJQ}M(dYDZHBy>OJ`@rbq`mV1Ne4~{W=;=18Txzu zKVb*=;%lh>i1~a4p|!k?=)+r$?3e>6WulzTk zaFaODw~zKW2kCkq84xZvDsV3Uu(Y4P-tFBiyFZ%YJ#7%n@v{>d2YnT^QHF|J3C~ed zL(H545#Sk^FP^|l*4q$=X28U->Mzy72t@}VP z{!}lVzT6LHo5&EEc+5|xE~oDw(S4LR&m7JSSh=mz0(GEd6zAcg*;d`qS=DxIqWZD| z#H>~2rhD-rbv0scREBP@?zvfvUdxidQH{Et4F=R78dU7iOw&wI7ApK2Kx^ zty|q0eH#A-bvA0w$j%zbQQL#*zJ{qTi4J--K3B4r6@)Yfo`qW()Qe@wesNAC8ko(8 z8#kU-vZU=@W0E0*m=uJ-!MA@$a`nkz~(wh)L3FV$8?hxlIQM3uR55qb1%-bah6 zH$`8g`>ackD-kBn5Q;|KRO_C(gf6W1d*~oRGe7eC61@Jhi8*WdD8$qduRa%}r63OG z5)w>cY!|h;visil&^xdZV-@zNRXuz7p7u|IdUhnck_~eVHg-xd-S+Z_QJ)IvK++FW zdY&I{5DuAdyx@d3jw8|Zt@Os^uUT*CACEJ)yPhAf5}LRTI6&GR!TW|PV_{;Jj@-6m zrNcQ0r&`NxPx`X2G-O^WI~L5-4Bzo*|KWL5@3tw^cLKBt9?L1lum~-l^i^1-eT}Fh zWFvqRVqx1ltU8k#L@EskeVEQ%G%tZ8OYC&b>I~x?cKgwoXDAm$`i>={*Aa!?w(jg2KCF z*@^0+7CBs$xSQRr-MSvurQY%TTj1Z)Ol!rA8Q})i(vX~?~C+G@1Bo`D`=Szd(0gZ#a zDKqC-@Kp6LrHCO@yboObQ`eW;zF%y{UG(36 zNf!#1gS%guL-h!uRZP6$U=5>>aFOKM3M{g@U+XvaxsN{^6WDR|ezHT}dx?a)%eLjm z6}8;Q^83#iyQoDZ4`MT8aE)<$#J6 zwxGgf^HP8$cbtrI&O(b*2^BB@RZBty4OpF}#Sk2IBTu)yKx=Vhxfhqz+<5mxi!)n^&2m3|thbEv=p#ZV5028nmV?)FfHF879 z#Ct_1#43T{XO@t>y%3~$=x*PT-<(&L|1JIOS)CtGh5V>YwmcU2;f^` z;fW3rbVVp~RCE~}OsYyvt|w;~o06H+aq5nF%{d<^5;F}UBiSX`65&_j7dH-@MQ%bS zdOxbEs!MLoZBjS7DVO@1_65bWOTNlGELukfq{}=sEGfL{RZ(slyx0L3Jo<4#_{9Td ziolRwL$fa8?;>~+YQtx)qRpD3uk{Des8iW{;K$1$a62eN@&;nhbm~+0_wMZ3?lk6O zSQkbM)7*Xw>?_?%!p8tTCFo%97l@!-EHv_!eBIN&L*K3Hr>yYXDiN&#blVW{M0^W+ztzFL4Fe~#D%EHYGA@6moa(Gl%h8foGho?NSOqB@_fNfOF0Ntg8&rRSg8y6X2T7~!J*DI96 zyVWoMsCtVp>~$!aO(~$mC}AX#I4AnLi?&qZk?7aXLO-tKl10JubCGTK==`l4ialq5 z9HavXbSRJLq!mZO?Z-EW)B|we#59;7{xV@Pc*%iDTX_vEt<6HV)H%L^WTK;Aof`w_ z@*QEy^3r9h;-^vL0gu`D8lvFGwtp*SPgAaIZs`l_QC7~R0uljze&{prV-VTcoF<$wl$k6spcBL+LCXe+9PDp%FV^z@HOQ&; zfN(bvFmY4;70qQs>@tVJdCvxejifJyvMc}St)Hbmc1(V%bq#KfWwZeVEm99_{-+08I$GxeAn_s)gY`5Ws^k#D_ z{Z1?@hh#?d`!Xyxi*%6+XX814V0x*XLKcjE5yxMnQZEisB@NK~Ge2bIEr9Czf#%2z zl{)$Ima>)_;^Seg?g$%FO>(}7f?epEOpT;hm&$3=DgpSyEm6Qg!$J%cgA#s~K00jP zHwtmKS{ekOlXN9jf}v%<_yFJ_8MNrmEX}2e@WZ27tH|&vTX{PL>-RNbvUdo!`g&>8 zI+gd<deYqK}z# z9(b=5ciSUWTnTSlsiJhL9uj2rp|0MA*mM=1s284>-ax;2k`C!UGqPId!k%ZWm|jsK zK`=N2{wGLqRt%bo0gJszU&oEE%Y1TSAR{O$N|bCfVE<#ML}i8qR3gfX1k+|sQ0!H0 zlk0|qx**dLkvk;B!Znv9gD)bZwST_9r*)SS9A98>(@YL^Lsh9gytpyuTL~>mioV8_ z^aZE=YSar-;4hk4zvkBi5B9-l}n+0QF~>`$_O+E}vOCr%vkYG8lvp zNL{81eir-0dr~;)9C8o9rTYI^f8l5J>EO zCn9=qR@Xvc2ai89>I_*S%6o4zPrVJh+y5$Q){+GHLvNuowmWU7!y0(GBiQl6f+S zM2(x+u?BkpZ!^QBRdAf8ycn0A(yR4f>Rjnn#jBdkZOj6;5|TpbvpGrhnWSR05+)yU zo_DJS6=|e1`-%jLqzbX6B#N8nU;D4CqR@3Dbka^Hs_(Dy5WeE9Q4o3&KN~C(LX_C5 zZ-fuZU8DK^aVIR0)y+~mNQ~30DU*}b>U}nxyjoC_#AwMV{DI1~3OuM59OEqdf#w1A z5bT<8MHpGVWS)N;MBJ}omhi;{Gvp|>fcd_pfmY6Ud%o^RBk?aS+XYd7>Nvf6FqBJH zV_1boXdiq~FRoP5EG}D27)?M)cY2Wgd8KYe;9)?LZpfECrWe3zg65^Gsdv=PEv*cH zU+ZW4e5`f6@QHc$gs6VRre##?ygc)H1`vuWqUWeG!VGr&xTmhzR<@36TB!BkYK&j~ zDsa4XI?JH)PX3(U?G;Vo2UAP{mmiZ$ceiCsG)##0xQaEY-%FI|LQ~TP%bs`>glbt1 z2u5X3;SUOivz*Np8FWyYP5c>OF5K8S-auP0 zT!}?i3OTWpE});DGT|;ihPqH)ovnsB`VsR06uJKIT#qVPdZ-GppMe^dLb}+E=u?8_ z6=>CorEc$!J0;Qpj94{|bNx?dn;ZhC&9cYwR!o<+LsqRk0!B57t@J{lX_sOO6*qrr z4hOkv&2`E&XK0Q#XYXMOCO=K~yHh3u@UUOVSco~zcJT<06B5!{2?QOwD*AQ^DI@c^ z3VnC`txWh`R`nv5QaGGyVQhc8@oAw=z#w~NtUfytVauS6WMfz55D61Zi3j&m$2-ZP zSR&2qj_`?PG>HkU1ygP1LdVfH#rJ#X=hE0$1^S>~g{D2DS-Jqg`w4B|_!L{oDDw3z! zy32>RCTX%gl4XXFHGH9QaGODk(hvy&lF1S4XT07n>mpAbGMa9xa1i-z=q^2gCx_V<$>0m9zv~i4iSOoTz-eLFgcDlr_gzlBAZ z`Z`Ih72^i;Y5lclU}+(9^rB=>1#T2dTJSZ$>Fw8V7hL^2W~@DL+J@hC0Pi7QVJI{L zxpyMZ&6x~P_1(wLw7~PuUFfKHlUGR$RY%O-Q3tN-Oj}dS**?FdE+jw~CK|!3?5bSb zcAp2?#|MpBx01&_apy%7sxV-#a(Vm57u zLJ2;2Mn5q`RdRLyNnns+t1v#tG|1aLea*_pLrx5*Z%~i;8V6LpMxz64!3ec#(ZlBA zT&+~T5?1mym5!%!%Qu#;Gw?9v*eG^fAvNTQik##lV>tkRt!l``SUDTPeCs&rNs0&B z{aY25zISyZ_?-@0>HY-5T@`0f9DJeODvQkx{Yj!GD%n*u_uRK}Z(1Qq=1flPq7~1l z!HUQIbA#<_Y>#c_K61XU9qL&jj`~G;vlZM@Ka)3xYTkqmk|fuWOFXc#w0W9FNKdYgQEW9uY5Cc&F$D;-3zM8Ng| z(=5v;S3aTnJQYhq7Q>eOi9(PsMfxEw0(JD2mC|I!6r}XG5fb4)9r`PdW*(u?TiPW| z^LJ+(7v&o?lGadJ#b^@DWSv?UsZ7d*ps|Z(U7br6M$qG(M2OLSrHHm zM;zy?1U_a~rjft6iO&ifmC0=Zea0p5P#2BBL_(mB8PU?|HulcB4^%Rk#}_@Sw3(tg zym_sZafw-81V`ci?WtXq_0AIbegFVK07*naR7?D`|8vj(U}Nk$+#P@CEv{#B%pVRs z%Yfi8l~hJtWgO^MChMMLTokoY6O}k`U3i`ATnGN;qkmB@GQQ5V_i#L)&8<$jb`7*OIun<{rpN))&u*PwsiYBAf>P?AnHf-r;et-sra1w zyD_CH{b+J3@^L2Tyox9kO@=soE51)L?W;*3)8NJ;TFDd&jk}Zf)3o5o^lNvhj7&Ek zHTb`jkV-SWH?vhB-;9%voLDSv45F|ywM1ONOqxUd)OV6QShOq`N!b6+*$>orqP<)k zz5xEtek$VyHM^1GbT&>)(EPT1*K%I=^)nl5R2#8ymtb3va z5lSf!2IoGo`A3Lij6+}9qQp8Vv{1s6Dtw|sjVMJlofk??OC+nH**sQp>5bjzlD;mr2_j9jtU5QebP+40>vxtyBlbqq5 z^m%>j?7PbMRW!ss1=VWCk_bDGC~&4?(LyS(`mPJWi=TB-N;S~$bYnd5@G=HQd4rje zXr4Efy!ExznqE^GFlw4=6?KzvfBKNV>1NaZ%{NkAQ$6yjMXjP~-h`7)w6NZ!VKVI_ zt=rA#_w~2z5-sUgAk84q{`iF@i{xeaN5%6Y6Qj;0-{Zo#rgKwjdyM6QrP<`>C);LcbVSqGuq>S|!~oUXtqbNVrhsY0lmuF$mK$Q-}!+zpD^ zO|++ya6a{gvYV7wZ$%<2zrO5i6b!BL%IQ~7*;vl8b*jlBi{w}p@xK!vRS8Iw zU^K}pvWG!k;szi-PLeiN9z42e1SSIlq6LygR#-Z2i(H8pPi{jf7g0Z27G%K^zD6m% znMo(Z`1KejHUbiuD}w3n_xgZe1Y^_TU4;~h%`QqE1j@H zs$^vlKF`wH5>!${X+vR69^mp4Gxp67@iq#k*=4{xC!n%K714dOq#E#YsX%;qHJ%E0 z#r$wb1bLXi<#Vq6B5FK8y|PQ0_9czyO!|Y#Vj*UJF-aRzTIeE>U7n+aDv6|*I$OwR zx`+$_gAr1N169T=3sninECR0|Gc^ezv6xYutXvR9Q^vPeeUpAIYYgI<>)C(NT8e+n zSC@Q=W_~lga`H85QtdgR&8Yb;q{*E=wWPKYGrI_ll1giQZ{2@NL@E(GhUtF|83xXHWMaTD+ z=bW+ort8xP7=nOMN(smB_RG7}eJiPYMNsr(^!;Fpq|BCiAU>(rO0VJMQ?dtFG@V{8! z87hV9D<|H-Czp3%@3nmliZ0<=cbcy=ifU^-^m8=(FrkE0d6?&tymucpX=ifIF{q#! z_BM9v##I49G^=-Zbn*rbQbNm2-XroH8|&+EHg7j_ zs}I4>VJ&@6H!2ge2se@y;&{dhC7=jrsQ9XgkRrRdn+4$hE?7M^v0~mAfpT@sD#f8H zemc3NSn_i6>NnC~w6OVA`Pg^u08VG0QIpHg+D-}KtrXHLnRmFNz6zJ~li}m+g(s;A zT^F-<$h$2Oydy|~^W{a4!?e>B?@bJ8?UDO_Itd=|S=a0OI^LZEW z>pOojAa7(rbpbxS_%Y68o`r>_nexLGt-0I_QddJ$lQPK!y9Q%@%sOR?F~*p59U6gC zS*P&g(U(!x!VPoKEUb@P4REw&W|y z=T4)j5$K74_;$+rA(p^wTJD7JQ1mGe(zoTl5!3@JdSyNAu$D-XIDkQ6^P3R&p~Nt}LyR96_$N_~#DS#jt0)qxIQ@7q-hmakN?-WIJ-> zN+6J=`uyPH2gt3;QI6L$d}7su+=^N?aiA6xS#xU(BzJvESqiD4D#5Q%Q86F6MN3g! zV}iwzB;Tj@UR*)_-Kf>9JB!~R_+L^uoufMaE^6c+Q9Pn2NMRw9QZ$Q_g-JAcVnP^< za4N$TXhk6XEcGw~Ik9hxT7g^!UP)FMA^5+W>ie5DvfUFcy)+$fpMMt_gpHXkHqY3I4KW*(ws9~}cjKJRG>fmCdtmU;8SR1t#o7nbfF#*WfU(&a8MtoF_SMS!Dw7WV+fHm_fJzpH(Ak*f zJB!*{7nqreD}|jRofa*&jQLdVX_VHNBCjkDUx>R4YogXDOa_CqK2{^pjetC3-sK$M zb#)h_eWP$^?41nYn8wA`>4wF-`*?X5GiZHFy>cPip1d8`3R3Wgdw;J5)t(?t;m@Lh zl%y?*`4_MwX{WL{x!OBZnPabn8=-G%;fEM(n;og$t=&r&o=%H4pO{e5$oWe4?WM#=+PUR{o2AO@srIzR6)+NUJL#b=W;gp94)>(7&lV| zM_UItQ)4V{fW`Fb73E!1OA8WcO$>An#PaZEN~t%H%ft;0kMSSlT1b^0f;dkf%{YQH zxo6aaI3b8MEZ#Fnz2vF1o}yjrE^^^Ys@t&hQVm`@*ns5RRtm0;SUAlT_bqO~0-E%v zc?*RR)2UGy6WWJYPrr_=%1m_u69N)c_M=D&XVb~6CR9<98EH5-aXnM7)ihPBzRM0s zZW=M$i$k~U>Us*kGTvhDb`Pf=#;Kf>N&s3IvJkNp$R)r@X>nz|GUU&<7GmxfPB?LS z$gRr7IX)mP$Pahd%r7WEPKj(Bs7SkITredl>3H@r-o&<8Ozr3lpDvt5kXAVNXjdK# zvRjkL!#uWomV!R_#<(Z$9yRd|X?LK08Tz|CL`7Jao&I*3NQ9Q_)1*(4RD4-Ep)U?v zhM#ZwDaW)xX$@o}P{j$oD`_{5W*=1#?-gOoxG9+q4?8|kxgk-d$9To(x3sR50zv~f z8aX(Ta{|uRE?CbDK;8~sgWePvIyQaWH_n`k^3S;O>XD(Q6oh z7p?fP<(Vc7FXARAfj|wd^`t7?uI{1y@;xz9v1_we7ohzk^a?~L1` zT*|~PYUh>R>bWaHeUaYNu>H!XIGK48b3)^>I&w7voC2tXG3T?re;#+SJyld}NPg9l@piOGq!t!aQ0c z?w@n7vIq`mjW_0<_aUu<&m4FTC$djan@o#kj}qb%foC81Uj{%gM;Yh-TW8-=Np~WA z!e~KVPv7+r#-&&)xyYQpx3XaoWmYkh@*K)IgkoBBD{89fe|x`L-$!$e?lFu(e;l_$ zst6w071`LAascVnCeHMY#iEc!Dl>&OgL`@kcQWdNN>Cu}GV8x~q>Dnz+nH1p9tCK( z_=GoU8Yk^aI&m*&p6hl-5bnfthn~Zkywl2rEkQ@IjEeu_!p9Z8g}a0MEtxejZh zZyQyUxQt)4OqmR<`k#G&U|-%vZA;fAcd&HCgL59n|6BKc1Ty$a?kf>c4`m+0#_1cd zB78;Hrbqc1_dQ-gLB^~!`_v-8o%jx_>Z)*0{5=e&TQI6x-OdSN|d3282w7;Er38f7s z%sBWUR)wz`aML3S6WM&Hm8D_Zsh6l#^~J+;A5y-Xq7V^1hy*hI>nFd&JLlg=PE9sk zt=$mg6{A>RHqYEhOXD2ndiU1(x0LJMR~9{ic;9#>DD@@yb+1)o2@~Pz&2w*J>+G$F z^o+vd-z}%oBb)0xEm>rj@pB4tF+U*<0ymC+gv!to^A<(DUJy575qkTjxH2WueOz-jeFPw<36pGE2rS?n2v6y^MJQ z^RbbFt@!C)SIYEu@;La;>}M?FpdYt%K{R`xT;8s(#C~)8#`;_OAsS!NM8A6KHDnM+B}B6UnV>Zdbt>RX_*egSY#XkXU&Ukd zAH6Q1cKK!XFQ7_aWz2G{_%sE$_$&95OQgh7Nfo$zAMX?1Sa|DthK7I$28T0_;=m@ibd$oI+(}6*=;9Z z#_nr7@zk;>@s$N%QO{%CNgHJXSz=e<56<)^H2Yc?N%z+zMI|ediHxXK6{b( zd_m|u#+Q5*Utahnh=N5@BD2Rz3oc<`JFk6;gwVP8>AD}&$~sf6d;MK&!9#G7vH3aq zQ32(Kza0NF6*YU99At*lO!tS`XpFnRHpZopDpXG=vrpk_$yKFMTt+_%S2{oUb|Er~ zfoFFEpPr|4H?u`;o(+6+JrGJ(B4=84FgK&^BJPGsl!2Z;+%Up?FfFo_qSLXvmD=B`F-_F8cOfo7^o*M zUZNxs?V>0g#JBaC1AkI(Q$6bxfz^S2q9-UE?@il{@7(q+JQ)9Auj_Nq^LqO%Yuq&V zX;>1z7#~qP=wRW5U*Gi$Jh}X9xPR{5DjuRYR+VqR{avJ2rsCIIeyuF5Ls=`uBp&Y{ zk5JDLyng0Y3USul@KI={06QLv#M&%mu-5}~?^PKW(~2{3we%u>xba``<@sOcTJ1EX zZi+mxICK$CG63@$(~n1cM=Q6+@hYf}J0Z1=DrpgLE)`u?sWBHbwWnyUdIRvilfX~k z>dF+gEC%=qGem8okm^Y}%e04lCxWL44>nddbo9QABWXv`(A-3yLRW;jh4V(8L;uT} zm=`pks`PjzpvsO$sE$UuNKj5nX{s8pL_Lacnh2S_jn~k=$ap2_M`*c(&O@B#B`Uoq z->GW1Bn9WG^^74^hw_OVqJY}nFA>~?_}_u`B$Xtohn#Rc_UCTUeeQ7IMQd6}->G71 z;KeU5&?N{F%p&4XXK{3x@jC}SAmBU`h1!wUi6nd(oS;v$ciCn{p zyfq=w(36Im$FZ7#^wjl>-9MO z<>W!2m|VJVCQ^Iu=7T2}J;n9Rd02y!H6xa8bqCTvBV){E?7F;{ZeRzL_Vnh8Wy4=o zpNH?Q{tnhftsnL?&&}2yzND)0(2K?_iEuaigq2!Ka~Ru| z9qNwDJFzf$A+^Fo)6RC+X~?@^CT6Ow+;s}`fM~5ma1|dU(Fn-q=h=gQA{Xv#te!@{ zJtHM3ko!h_sf32>eNwQ9f2#-##<+%tj9VWwF%XbUI!Q%IC}D8N9TZSUq69hzVQKhM z9K3p11p@WyhXQ!AFcZlvkt}#OF-?eTh>ZZ(qxopF!*ab88*U|6|npFv3?ioFbQ%C^9seC zGwvL=fZ9EtV3%P0VD%61v+e(i1!?o?Qn_5Y1a$vCUf#!@kShLtb**)n>oZ49JkeI& zT+U|!@16i=kgn}Zt3aRAPPo}iuKAmfyzy_*u!-A8 z2pJNQnO&Jpw}V3N2xF8>!$9qd6SO$A_te4kc8-ifw8yg3Oe@Vm4yn7M$ygxSH$)&7 zCV>3fe9U5&3lUIztK&xWx`Kw}yKcO0{=NG**^$rjpG{}}vkZDmSG;&4ypCj%K1p%} zp*->G^t|4bU?mRY=eY3+_X)wOh*e$Vy#c}NhhP8Dg|nR96c8;t!7^1Dg` zYEMTp1TZR97mYwS0*#EO7GG`2R%d9DMY*Cud;y_0>jRx9o9#ukJYD0a_bA<&HcVeX z&?)-&35q~rO(FJN+k;ga7kz|8b$Dvw!5#Vyi4Gg_!d|3dRtkT59mrEfpz$ceJQ8Z)y9jkwvdI&VNHmkr*$rNR15ufq@IPf2M_SmzEU(%5Cg6ZZW znxK>9de&ufA|RB5yQ$@jCV#fE1yn&yo}=r?&=MaMM>Z5cY^D&imTWB&P%PY|W)_`% zwlAc`EX?Q(E}}%)5Va1qZ8eNBU_|uiMN~6K8E|N6ZHBp|ij?QPv5WY13*}>8O^%{? zGIsq#sjrj_250ilU(vb;ofSxuIcHXZ$NYjLwLk zWLilYoUNRc52YyujGl$bq>xbpT%w$G7c*Txw1o5-rh!hpY$kAnL0KHlo@7kNy9_sMXI%0a8Ps-E;W;!QYd&?xfN{_jRra!jj=kh@vG! zmDHt+MnDjddm)Qif10@|oaGm95Ge8-%i+DaFL|F5Zbd_HND+4>AF_hfw$v!@V?)l- z$4o2)gz_(sRxm$D!-bYg<q69LeNF2!hbM9A>`#q1ZB6|x-oD2%usU&lbP&^*Y zIH;(m4wLfl68$RsD!A<_P0 zRpjAR(^$(zyH;V{b@o=X2^VQ;;OTUS&@muk=gbSBO5TF96phA%l5|R*2RMC4c2HYFQ-HuDvaU3V>ILT&HcC*|G5_-ZSX(;U*wi17usl`RJKP*9D^iq_Zyw!=Oj1S=1PNM}2FfcXq9 zJ^ZOYNA|AAv`q8bGfT5?`Wnu4Bd_F@fSr*6{>jCYZR@Jj4RJKwG9bpoPTv(6oT@MgPBgw}&@nGv8Bm8qP)-Mq0r z*+=h3?&>5k$Z$PN!JbiqRMEByl_}yhdwTScSfoCsm9D%R*7@oprG>JG;;aUqLI>m^ zC&XoD-xTEdW|VI>lS8|nlY%2%zzSi>GBy2Xu2fi zCujsW3HsaO!D4_A`8ut+1T=LR1SqD6JCBKQvkKk)bVbv0Y>$PdBQi%3Z^j3{GbR7E#H!wK3VMr}g)iHv+t;oC|J9v2AZjEKJnQWOD}1mHjFE z6e{M^$;YfA$h1zbhx829uHVdPcvdZ%UBQuRZ^V1 z%DrH4AuN#WHr$Ub72a!Scs|J*p_`u_>y)Lsn)>_eCBdd{p&s>h?F5P>XlxW#>sq2d z-7T{E2nhOJF!*?tLK|dGTIQF3Y~*9|^th+R1yP6za3_#<$Xa1eTUs!w4N@q%S%vA{ z=n^jm-qRlKg_De=%OPHUB@`T>{s@wwE8kY3FiZiJspTk0Ib*SK>Cjno(c3z=F~9R6 zXTEz{x^GtD!^el57A#wMERM~{UL!$poCx<+gklL#6r6zTCqo~83iYhdQn>(^3xn}| zk%`JuiaJZ@wtoNp9f7tWZ6a#I8Y>ou}m&(_fd@XZ%8gANM0eo~^Zw zqe%Su`Sa&t5kf|j*7Cg?0lMCqK06eNL+mN^zR~l>eDxKuDy9^r$Wh?z=^nOKxE_Xr z=shMIO1oN8%hhw&1K56hiz|fF;eNwqT<`HR+JCh4vJw+uT4xtn4n2?=DXX+LFqjtA z&Q6`xx@&q|bA0y2^;IG1O9iM4Z)V}W=9&@c5omSaVIQfKa-UbL@s&wSr)BtNb}b=Le5eaK3H`J3W3=WC@)Jd&okS0ahaR1@5J88GvA9lZVaJHUViF38&bg=~Y z4AHijl!cZ|wYJ&Sx+q*~M#v0h0=3LEHDhunW&DV(!ej^?+jW%4tRrAYn4|E3Ls=0DSs-sYg-~mSkR;w$#*kQ&o2;GRoWEJ` zd2_aGrtY4wxX}Vt3T$d!i$MY4ScMgBqy}=vg^O)emwrk7k$FcN<2K*|*lKG2#%>x+v+ zCwMTrTR7_W`ufO`BeHkzUcKnalPAlNAwzE20icjb%;etmJvOg_YZCVsX5#zB%wHhs zNMA`WOOwS*AC)w`bN{in=WtIVE5)KAi*=7-);{i+#SmQWDOA%_KqK|WjyHrBRTg6X ze>Cw&`q|q5=uKb}79ZDIZ-7r{FSz`x0fA{=!wUOE@1HYd2(I}zvoLOkF%fZvF>184 zw*Pbkd}CVQY4Szn=kl9PzlCt$S>p5Kk=<^ggnIY0>DlAGCS;NU*;s;FRB2H{Dz!vv zeA%E;$OaI&FN9FzA##Orimxutf=G3Nu+;H7jy7B1VvamPYa3I5{&3O{fmZ~y z&PE}QF~2lV0z3n*2&sUlLD?@s0IofypHHaVhqjB5s@f>7@c!9Th_9?Za|Q;2TCag8 z-p`&pC!}Pux6rb9&)~cDS`9hE*17a#&=c$C;Gt|BwvKPk-`iu)^>Oi*9f*mrRcI-l z88Q>z6DPnsqScT!tKTe{+kdX6?zVOQ1$eJdovu``jWuD|Rz1z}dy5iMd4WTE4$(-& z=$t5}STY39Y|Qgs2u2^$n*wUXhoTyq7|q%YI&y{3cQXs!(EjedkuEV_h&SVBy5yU( z(526;HJ-4Xassn3kjf7NSSK(9>`mLN7S^UpZD;hIZkfEwT-5E0`*pM3CT(S;e>ljX z_bSgT7kO(Sa$W-;)cQa{E&@VA^6BBe;q^Mt7_Zj%@AnAMZO3e9f8O?I&1}QKmF6g< zNvlLua&Hg+=BU24&YL^}RJbX)IRYBTNc|wsAd5ZX8b-o>zKp_k@4+Dt8h=`RC**Eh z(iREx9SqAY=yXu%v7}ciD>p-}skZ13eO>(;q>z(oRJwnB+bNv~Ato5lWers_;st<082dFj3r`ErFe;L?qpboSn z(B5~OwQwM?uumL7%Vp_70F^Q3aA;8%*go3Fl2HmKG+8y1=9-wK8D?FjZ!%Run zROKwPRAqtfpbBp2(SD<@VnaFkJs9~197}Y5l+sSn71UIP0xAWzQ>RbKkkFwTrP)+< zH>4g~Lg^yw(Zx@8r|i~LaKtP(RiRrg2AqJW&(wf!Wu#pTXj8R)E7e1b*6P)(8`b5z z@4j1neSI5M$IUpI8zuuFI<)O5E23A(3u9ks@@|yFYeL+rw{tfc)B}rTYx!P|0KH)I z2Hqi`9sNvxJ@ePia*c$EWdSx406e2<84AJic@$O!%Mkn*rbC8nNK6Vb zuF)`qA51@}8VLnpdtCs{d3!oLc9KEhHe)u8{Yd*pY>8K=TFtdF?NU%Aq3<5l+1oup z`grxZ2HMaG?GE-0Q;ZUc!l@y<+sV6UtzT?6Ha|u-Cv29Xe#6Ai!_RJQ&RV^V{f!oc z`Ge-kJ0O={pRisY8Me5|fboZsRb^%LDw*AHrp{M;-~o-jo?G$!;^Jc2zWqC_Wkark zn8I68NaeM9ckz-Z#yl=dk1Um8J%?$ub3?|!+&QGGSsAku!K3NGR6k|zkk;?MCnG?s z{Am9%^1~I+!z%g!u%?z_Vhz1yuH$Q?*U8dHmR?guAQDSBw(X>8S;OHqqII*W3xmC4 zv*P8osjpqFnrrF^zY+4AwSSPUfK}Z&U}3X(td7Hj+p&#{41oo?B@|k#d%Fe7irBA| zdeW9PwV{1pd1S3@d1<}P;NXVV-9pwCdlM2yLH^!g6~g?IjuRxwI{MdUXUk$FHdU5)Kp4zBl~`#+Gro_%F(;Mkk)B5LgQw4UF5 zBfx;E@qy!{zvlpXbJtsnjcUkSuRkF7n~<`{cv@A4=irBAZMZ#(DOMC1LZf$aQgNm98`R}d&Esu?UL<8yUg$LL(rcHHBi?gKi!bp`dA!rQH?qoy*MFwyi(qLdFU(Upql~= zPeaCr8)aT)p1Po-GGgV&Q-7o`la@5;Ii6GcP6oMuwY&=7vL~e1R64S#{KxdzxVCiSAPHd-#6+q0|EjXb^gshnGw~Gj9est-}@dm zhd;<`(|!iej(>xmIbI9N-S=ngm$$e5pNtC_D}#H&-PBsX_aop1TzAN9J z_*PS-T1)wC?|?9Y#L~k{^vuU%(1di5(C!ZhrTv=jB+h>a*#-8?>j{1#V zZzAbR7g;c9{&lA)o_56j;IKX-f7tQ|`PHmn)_Shst7}Z*-o=bu@9qDGjPoC-#)ih! z*VN~R_=L!;&{?u7c9jOeO+Zkjt(XdX`oJuTZ|!+Y3Mz`^ck_PJVkzs|K!~$W+-J|# zHp4OORt{33{p;|TuzE#Ge9B@uT`@;~d5_5BlK_ggAzt%*MqD4uJ|-XR`w$dF&hpf_ zrx0x(szPrwbh%MgvcJRteTGzs2ht9Jn{%El8oo#q4%kz%UN2lU@rCj7-k$fRy84X# ze9q5cRSCg_ZfCIyo<{=uj}I)BZ3)}u{!w?!iA4$WyN$mWZzmu5 z(X^L<-9pZ1Knhw*>j>By0ScDPiT<};?}(FQM)*WYae-lsr$&pWFV-(t7^ga@zU>=(hKDpn-vYbL|=5Q9w;AO;cgNvZ_YjdGM_U8HbGen(Ms)r2K#1^_~>eZpK`65e|R#%r5opUT!? zCvK)Z-<(C)80Q2CYhQDtjQkQ&q(Sn+q#sC-cgqw|b)2#A(QP!ccR0o;Sl-?Fo=CgP zszsi;SflTjgiVq>pDaq0&yRmD@9+6QPME zAcIehGHj67ko?XRKKZ;^vdF7&3ZCbPtO$93-}`bRHy#4&BJd$D0G*we#q(~h5vqT0 zGcq#d+i$;>va+)4UNtq}aclCkxnua%+)&aZCVk z{=#{1zLvt8QYpFRxghY)k+v5dKyx!0*E~f9T1y+9r7>W;tY=l!x8PPy7Pd?BWu`S{ z(g1g`5S&H+W~QVijaS~OJjpIasz+p@orZCq*>8qEH1%!QQh%E3doE6~IdQW6lw=fTixc>E z7ll8Fl(Xg`70q=Gt@VCCM&KN%ezqoVmA7Eo@6olJrcWiVl)<8Wj+ps|&n4$9-}ziF zi(DoLk`Bt0;3@LzjGw5>vmu++_hSTWuY!dAjfIl@_P4*4u&`SXE|F4{DjQ-q%F(T$J3|?%%x`(y6zoU`fc4O>jiO4&CRV?t%n#kC)*g z2Y{t4kc;?tSuRN%qyZ##1lDmCpu9kmKX1?+)%I=`;ncV(e5YH3X*_c)^5jGw^2e8D zDJgO_GI5kt7avA;fm=snPNjDEX z+wI${_lL@NCY0{-stS;}7lHjGTz)n0br}GLlGai?0vC~|+|*)fIxeI=`|7K&^s7f6 zc|>N6F5c_Jd6S3gSkRE;uwU}H8E>}HBLnEcn3HSJIIfx z{ZK~q9&YZG7WG|=6;g|^P_K?!4}r510>CVtz`t7|^2S<`AI!WgFU>gzTxyl3I3@2X zwF$aAnAs(@I8C-E?9y{)^`B-_Q>YagE}O-`-0Zu>O`ZR!InsKWkbhFv#cTkEq=(!+ zAIfN)g-3u2ahe&j1;V(KOL4Z8n$D0e}ySx6W z7SXrv|I^hmGp?5q2WC_$MvAz9?D*28RS~`wTd2$u=+qG+_O^_tOk$<}g^zJsvl*{@dBwWzRVO=*w4; zaj%y=6n;Mh)j~vVpVjBRixb?q*npXxj%(E$TmMgH4w!~{IqkMlZtNL9OUEo~g74wXE2@PIo(76<}m7Co|z_b9I}SB`WFvi_iT8qhsd z2KE?Wcli>Dfq+;DHx zOEdJ^nHHDufZ(nn%A7<`A!RE^od4M9MbfoXSDQVwWL#)UnZr^iT4baLQ=-KJYen4j z_~5a+aPtCQo`<};6(Vn~Ya>-P9p$GBIwEBo(z|ix#<9jG&k0KF6jOci;i*eyVGIUo zCnyZ(;wBQ8>F4gN$KHs6H~L`foqgdyW{>FD3070rZpdfst>pCG@ZK1!skOGhYqtY< zTRnwtA!cf#`@j2@e9G3&Z;8LrWx6M6zd|McZOU_31yuIcxZbwy+G>^$Dm}zL(d|mQ z7jxsx^rs63-vMiPzHB+M4K9B!5T4i6Ym5UI>|g|wc_6iH1vZYt%5yTV&uENgcg6o2 zbJsY&_5Jo5fgDiZ&|*3%c%r6j_PLVfXSn}Reg2L4$x*?QAV~$uG7c6L3b2+6Hx@=r z)BECzdB<#atf#ene@7q$^kaP>nB|m*D{D)7ah4QIH?2 znI9Wln67(r$V7Ne2M7gLT4yt^B<)#wY_$r&;{(S_7oRS=-ulA8LmX>;t+tKrXJgOj z8u8)2C9-hHe0VU2Lni1T1Az<9D$B+*UajLzxMNh#arL}T4xTLju7ErSP6J1?w;Gs? z44xEd-`xIxka_&%&LMYL!~nC;Tu1V-#}&e(N+S+m^du5$Nn;shljbZfPDs575>#SP zcPH&t05vr{=9*{e>?pE$k^l!OvhuLV06<8dna1>$*A9^vG=qSy{=g6xN&#Nx@HN#4 zSlE@cM-Hb)>LAl%OHFcq*jxxF-i>}}ukBIO{Z+fiP$I7tUYo9FD>YqF0Ca=9MfW&9 zXvu}CvSk0K^2&^tz&_B&$_3Z{WRKwJ4h{9WiBHMTR{TnKpWGvN0AfN7zHXtA>)}E{ z0a#aT4(EZP<^n-w>UJp|IzaGmWMB&G_&%0R@!Md{3Iiq@SXBPIB21c7jtDv7#zd=> zB_WkcWlS!z*cGHGUZ3@h#d*ED`rUGy^Y`(7H|I4`A?|g>DoUb%x_kGh900+1JDJsY zmW0BB=+prU3;s2Ap&~>fnT1);&I7i`OnhaqVlEuC5NjXPb2pDr!^8r7G2y`Q^TX9@ znpT{t0_lB2?^R=fxgg-C9)sqnkAmS&>@Q4hPR|i~7526h zQWe|_(>>kx#G)Dksxg4!kS)gc71^ICauBa614R1P2`Z#e%7-)?7E%VqP}--f*0hwu zrJG%v13{su%pNdP@sx#?#i}Kk9(OB#ra;Dt#f_=%1JE^w;TO8G0q!7 zb{^yt1Z4zSu$sUCoIqp0qQr5gHJ;p;FHCwyJ_B9aSXf7USg}~uwp+V7H2oHFqer!V z?J+3j{rBIm-$}R0(J|Sv*eoX4<@O{s(BHPdtI9kTYIAtzT)1b$rdM)122cVcHaN;pepRK-5 zNbLrfb#KpJ@@d2uh)&!N^Z8I27c^D^y#ks!z^Y7c@({ zF~L0FT!mQK$joDkAEuH)R{#To2K4AxYn5&WjlDkCavJksL+oZ1D4)f{MI!9R6bh#d zf=Vnn2}LMZlO+Ur`a_HNKp@o;d1>kkzyP^6i>0D5{cb(CWsX36UZS+Y1{wg@E9uc} z6?%#osCYE%IH=S{03Uw6%d{gbESI1_#Ui+m5EKSDGA(O|D;5}AZM6R0Is&aD&@x7V z@QKe4f2}bO17JmMDhoIT)j!fE&xswc31w8RoVK)`feOCPbSbg%sq z#Pf?rJqV0ov3vni?8uCx*ByIp%`NvY;?LG1sOsI_A3$?Gi-`3k*#VlR5>Q4pXCa{b z48ZBcX0C@YcqHSPDjJs>lE1czJIech6!&z={-w&_N|NZt76S3tf%4@uC^&S(97sE) zEEErpxKAM_o8z`B?$u&Dh6&Ypw|HRd54XbztU-=+58&vDH@CM8$EK16`nR(Pz%y6q zzPoSu-Eg@MkfWJV@;{sZq5&6l{oZbcED6re&idl#)^hy_Q1Rv(&Y;m6K;iiJjVf%5 zaJ!PKQn0EtE;V<3`b^^$v9k;=GhT!Nwxq)6^=QU%O)owkMzkjJ0AjzUL9pa>84Nd< z}H}k|R)nz)ia0ry^Z1pHWlHN3u?)yiG*lC6y9dQ#sxY!1`$4 zC-4Hkh`?cQ^C#b=Z(D>Vl>+MWqpN@!X(#gs&r`Qo-9^{j_$(M9^7ORJ;s@_-aTOx} zPXQZlq5#sGKp;2qr{@r;VpnX+qF=85I~Vhk#-EB65XJa#D0Ps-T%`TWVBoH|*r9GpFTCY|N>}sq*y1Cjg-D z-Xv|4En+eR(}9R4C1>3~_b!o$w8I)CKzPMO0MWO`?~q(1JE*Ifa?vN{3c>0I_`OLw z)mq6dIRaG7X+b3wj;UhK&4fXn^hz8BlqL>}R_$h5E4VTP4PiAOANW)QW9nW!p-#J# z_enf>e;L`!%tTdK2TTRuy63e1eS3}ot%&ptMWh~vqU{Mj>>?=kCsYYmWIafDM0i>nk_HhHM64^UoTgp(JuxCaZ|el$)C~a>;qqh%aCpI)l?W6qur)>=a_LU zzC7}^JU(Wz67ieX%Lnlyu7#y-8(v3STqnl)a_u-Xvlf3pf9bqluZ}a0qmZd|A4M66 zP1|n$+&Tgr0V3ye&Cp=Q@ zQ*)Q0eqmsF3D7KyboQNZkCQkq_v;U-$Rs`unB|sT3qgZIFYkN)s?x;~=Cm5|D+0 z7I8g1dClE77$uf;`0+UqMW>G&amn+z>E+H zT3<*Oe;Cv>0O>@`aNy`6QAFiWSf0y7%a)MRG5M+$spXT^zfpvShk$nzq;a`Su+FuHSP9 z&eC~FdZE=Z>tuT9RCxe%uerTFOj5y^kiE(K;eFZ;A-O=F9QT+SRmfR=JC?JMcfO5Y ztM2!|Km4`|hD|jNA0h2^SEcoUCLSJ=iv+CyTANx$KQ2a z{jdOx?=u#g+TeO$9>sRL8@@Wa9KlyR1U6J*yG%Xn6xC7uT>Y>C-YucXvSbLU+&DXR zaKXZ6p7x^Nov`x+3e@C`q=G~3pYVqF_k0M<-WJ*Z^5$!_J|4Rb59oUcE(`M=00snC z4Va@9pV^;2-nT@ebE8z?JcsLb!1cCk;~>7c-iN~j}3C?+w4F)qp|-Ap^$zwevDt54*)jps1=?=N;$E0`M_& zet5uo#MHc;8+8k_l5Ii|9-l9!h%T)zQ??G{I0~G!I+5U;>)Zuc{kdLpll$4MS5z^l zg@n&!>s&KN?e=-65y)`esz(fe^WAWz(__dvM1`L51Pfr9r*Y^o0!=r>ZbD*-2?`;r z?}!;5Py-@X+mG+s=1gb1oRdj#JG;QE`RM3}HKXF@xNV?LTdkbF4N>e(+r9wH72Bp! zfgV7tM3l>A&=VFC5l7a-Dt#>LB}=pITxEdRCPf90uhPsogZQ=knN z;=4$BZo+fd2&n8A-KZZOSSml6@gog5;dg{Y{A2G&>f#vJXPhJ!B}mzsV*VtIHEq`( zyt^RY=R>RW4i7s8i)BxKsPtweGgd5BWhjtB3>Hegfvz|R9Tpb{3^?I(9aQM(|E zVLhNnptamSBS6KQ6nlw9CqaGp8LX$?u!MF{VSrwXZosAzLvIvrM5Tzp<%YvG?-Djz zOg$~Mj?%+I;_@TuN9Czce*g@04zdTi!yWgKQAO>VT6dwm`fKZl@9PLKAn6!_hG`jP zNi9iNYC^)cx*>BQ1vTyy^d>T*J_^B0q|>8Cp4{OScnmmHeaDEfv;7c$Nc_%n5bHDN zK60?@n#b&pWBQELgesiZTVO$*7&HzBj}Xo7$3EAMx3ISl#JKK9pb;5;XpkUVk1JAy zYA`XbnO-l#{0L`giG?f&TOg>@xKwv9fP?KOymk*x&r)-ALMBo>~O;?tl|!XO?;PGUYp zygBA@qvJzG0>PL1`xPSp+TTugq@9#A?GDSiOVwAac0ijH4(bXOR}W}`ANW8q?HdVw zixW%WqOOAbxjn3?z2JTxi=c?WYkI8~O)LbBEwmUi(ZPYV2q{S}!b5*X9NM%O|1SOz z@FqY&?%miX8ht(=u2T^HHlN&}6ebKrDTamq5eR{I554o6`Z)&)4rf}@0Jt}dLKq3H zP<((N9rLj4P1!GpviGUQliF&admzSPmRhS!H6!8zxd2)!J`h44AG=sSIq;dbdkFA> z^>G_ynC~!T?3k^KN2`U)(&&BvzUOmE?&6};xp?JC+XAZPZPY@_tTD`1HXC?y=2T}P zE5~A3n3#NmS@;-WLPBp^P09QFW#n?%pSEABFI0lI9GeU*C}c2c3YxX=-{{|7S41Lv zCbId%=KqBddKMN^qzK0vw>NbU6n6*d<>s%M$LM+E-b(>-*}>02A-G8s=#cJ+9>|6i z3RA>WeiHHNm38%uT6A|O?}Ae1B!hbn*3`eu`WFNtD*{lqH@E#wx{zD?avL=UFsP{^ zg@Wl9h(kD>c^GY8gt4X?vO+uAl(0$Dzs~44L$B2kQYnDZf=KVyjV|18xF4~7BWn}W zxPE=~8+q%2zo^n`+NWgfZAC>RJadHkRO9?d$1Vb<_oV#yrvJq~K8t<2Hh$zl z`hIB-d8dE(0cxxo*t5TSt0|a%j>I!xBe0s+|KjB5uF6B))53gfQ?2d*T#b7e<8)Z) z1te7NMrTR9VM1y(ILXLCMpN|R%!6_$bH4<6hR7>ZUtx@ox*@oTn-&V9#F2VHSYoz} zR9G7rFvOQ(?lwK1pL@8AFe}yOD0iv8JXJb7AC;n-y?9U|sN#l@1e?N_xzv?)>uM=~ z)0!IS)f<61;c`4DN-`@lFfN@ietn<`4?lz3>1s}svL}4XY?gd zadE^G-R#1Jw-{PwI|$DdEDvYzhZ55j3ITJ^S2Cc+P`jnL$Xd~ z=y&9>Ek0F*U@jjB{Wt6pwAS6$-)_JNFmQ7MEJhm=R--irh1NE@KLgn1f;gRD6b=j_ zmFptcM7~I;M~Fx%IFCt_x8m3;*_pDVc3m^+JD*u=ij&EqtslR~Bfx!=5#ikt7#x`s zp#qz6xg8-Wx;YP)yN4~*ImPd%!B8|NU=)O=N>!c-Nom*a+Wpu#-_|>Y-a%-au&=zb zEY5U2<@EWsNPOZ0zUn?|Y~q*Pi)k@@IQ)JUxZH6r;eaHIZ%CoC$6)Dwse7gLOcAQ6 zeaPCKa}W_Q)JFV&3wN|zxp;V&!6F-@nxqS?6VxOOP0DDirN4Ut@XX+{U7(1_uguqnn?Xy= z6k)I6)R1c=$v8PsH^*<0!qY|46%YDo2!M10-0Z@Ig~ZdeuwU8lb+O?>``4{+A;=^d zfm8QD&<@b9a-3+%qV<3-reG|V%x7@~nR_S%P-CPZa6bYDzKvcZQF)Pg@I8RnnNjad zWSpbCSKk0jyzS&x$*auP#e7wk5IbKf?v6h4 z*vLil%Ji46S}mwVzcBF`q)CNU6$>2&*D-kR=qly2_rj!S)ukDceGm%%S*@up6dqU1 zS5KE~gRqQtTHkN45nwRP3zL2zuYQv%B{lh1=15x^+C5w`(at9P*%VS}A!`aN8NEZ~ zj-Cap<u=H1G+s=SZjSCnrli`Ik6^dx>qwvL7V+&yni$^^KI;}&pFJm9p+1h8r zr&CXv(s!y_QB4Ib?g`D!1#9V=$SANXb_EP3nTiu7E|_pLa-tdmLH4>8t0Rxrzz^X(ZOOGo;D$jFl-$9%%IBH$2lhNvoc;yY_IadK% zbw&xNiVp(L1AW@ah=3{L<5?oBWB(}$g>mQ;>V+oR4uS&7Y-!o$M5PO)ZlCK^d(-~p zc=7s4G|T`WVnn-QGh)yZqk{Q+a|o)emk&KHrW8~ovyUiUMQFEBY=m8#wOyT8u$VI= z23_%ONdIKaEA&aP7gK;%^&+qyoWJ;8rVxU~kLO0Jrs#!!29B>4Qh5W3`FnWO1F~e_ zGWpf&->QZF5{|!X$O07*STE<{eP9KMSsw*u#!H1-SLa$6ti5Ah4(lm1Pi=v4UkvLP z?+^=JP(`PU6~IoORRak0GRX zb@WCQ{-Y33|4lL7#)WZ{BC)nnh%KK93%^ka3t(W$&sY9Zj^{_}yyBY5jh+F4t)+DY z2y3GuBdq%%*_mDj7MpY8-q8zGPac3nJ%s($b_(8xjw1K-aY18bI|QhtQYvJyqB?nL zoWsH)cQmwGyUwHWfh;W1c`*>!iU2Egh6T4P#$q@WyfC}!Xzqi!#?!mpwrvNA$l50l zE!n3{e*X9uSE({h`#`yhdol68G}h6WLI_8}sbZz5qrgceHW=RN$lMCZN9V=MsXJt* zZeVVht=4JN`t12kyerqC->m&FNiR=BTu-HRbLpvEwqzJF7543^uc5~@NJyoCNMTX& zeHb)aNsAG=T2fuA7v%=CGX+W|)5k?pisua>NSd24F(Qu8;(B!MAwz-ByJOH?4Wu&F z;CioN9~T`A}6v%j;4CZlztCL&x!329P@z=Y$^JABoi|Z6<5mhFcs|JCJ0h5jR#gV_Py;-X zd*VvKg_B+fziVutjdmF^Pon4S@@&QN(t6z;!BOo%4MK1>*%@`NH*t3STvpbFyT3I{C<= z@I~_Sv={4*Bjr410P0`w`;)w~{AUtda9r19Ob_+4@ENYveQKkC*@qjosj#iq(lSPX zLAMWpIVUDB7S_}PSn_)-6Nzc|#pdcMBgVHP2#D}BTC)cA90U&U=Fdb(eCHwhzq(ispndSk==M>N-^*1KK?(Q*%M5ATO*njuE@SHrbF6YhCaY&ogs{9zd<#bh4j zEqY|Q?;X#M0u#b$ZGRYoc}aVNDJ84uSDyzV6-i=^!&HCV)Ee+o*N2CT5MK zl_44(h3kh4-~l(f)JsU^!dne1W*H(*lZ%-bey0+@c7VmxvrAX;h5*4$z660z<)YK0R-vr&M~hhR{B_KXS!bf*mgxI5T8 zSo=UQ@r^7TXVT9wJ4AAE3Npv+(bRDr9Du9o>>+aoPt!EA-$bvF^XJdQCFm+)h|&!2 z6)u5@_9tacLpCI~OC^B!DGB#?OVTz(_(e$v$M)jn>?AWrPnN>cLRk{^p`1Sl;jWVp zQhyGV$$=9gH2W(irg7~SVxFYHJ&tsbS+U~Qsk6GD$Y4NLgDpv001c^BMgaEXi^6iOLz>t-Myr;v`Q*ZS4u_2DPXvQq>o3S z%m@Z2D}+`uQ}Fp{EiG>ZC>&DpVY)?9Mlq1@Eaq4r1c`7xJhZ*`>K~app>I_dYUj;2Z@D_k-KGIXByG?iZvppcw0-W8`@8 zN7BASsr+!-b22M*w)jB6s#}P6jm(aeorpp_axhW`bQ>UZ`_G0zW_8|ePTT^L{1fupx34J$S8i3VgnIXv zIsNA#BhND$>{a)?x_^@okgO5MbB=*Dd5P430bdIGvQ=N@*p09xkw2+y*UIzpoIJ9%t`^Uw4`LR_z)YwY}f|aq~fK>mqWESQCQ=TW&LZ->D=l??CGmY&XxugGf|6k;# zFJ4w|;5*(~C|+)!;868Y55b1Sb?_3LmfVs8nLBuvgm?za{lo6NAsK}ldu^@fH*N&D z57YNTMfXV75d^#rg?H?fTyVRHwW)a(UGks@BBK+PY6ozXhQauwf)&1N&O$|ziEjvX zXoG-S2ENjwTwGbw>>1Z+uljg*gsFwon^LMQ+YRBX$$w3}Lwix3pdfGTsw znyVGp`QVD<8S4_47Y{>9CIY7OV9cpf&rsQ!GT8%4MGDOHs2YWoSM-2pLpkR7lfIH3 zX;W*kacq6xL0ty(nUzA9)ft$dnYMIO{3b*wXQ|nIYVcI)-nqM7LS{x;hAca>OrkQQ zWdf{jOy|dRU(IEqIOytG#pl}p@Ds6bidKk71nWkN+qnDd!zPBL0+LeIPgK~ z3pa!I*;>1tS8yw&Amh$A$5%=QFfXLjx}*O*vkFAresEsW8uj+>w`FI-9=UVSLIgbB zryPLPI;|D9n+D#%FC)K_Ohnw1PHTR@J3z(aY3;PDzY!L&^vF_Ogr){f(@0iY4Xg=_ ze8?CycofRVFV_4@E;(P6$wMc~$k37E=HhmpaC)^dlMfu{9mjS`yVLDtg5LzS8WS7b zTyxT->DG@HMt}=)Mrnq;vF$BnE88J62F{R&hCL+y@GNjX7?2^dQ1DnP>K`2Z0G8Gq zsi`?HKb`RtY$oFre{7)%i(eb5ec#;iHUdzWN^tjJd3wx~>ZRh^%6UOtGqKKWn+ou! zk)JA@kJu;K>E#q$>>0C0zXJwjzp z=p2P4HdW(s`SJ4Eq0g}QCW(`ilX~Jju&&dsUjS+Z>ksTB4OXB?5On0S}l zT!OV~whZk#O!1F)T`dl(2Gv1YSvrW!w=2Q$=-y*w-hjC}Nt$DWC5AH+nA2@=S8q8w zSYngr%TMP!$m@5L(#76h3tX|evL<$|97;Q+3;$h%7pj}r(&Vwf=If8-!_6G<9OnzL z!YqW)HL}-8ixc(w8fw2`P|oVuZ&lkF<~v*-9R7gS34a=zHt+A+gtd}?F&9?7$inCn^SRrvha=d$}G zw90<7Wj?fDcQ7GXnj6ftwg=Yk)$wbjv~96W9X3^hdIZ&5z$~pSl^sz#Bqb$H?&^OR zl$cp+Ewj{aJuj>E@Oq2@m6g>oYh`mHI2c`A<(|R!AV?)lU2v8Za0Y7fS*In1UW?VS zYb4xv1So)}K}qVWLajYr;JUmIsU-h}Ie6~U8M$Z3Ju=pRjKY<;FYjruUt$7EHEF76GpJRFo)S zWmjaYw=x?k(cK`#yg2!JHEdW~vn}0I1M^LM|GT^XE{UZH(l0nvLVJe-D&~f@wJn52 zCuM+P9lZ9WtR%@R%9r$v44EE0O`aU{1lCV4OHEk(o^A75wkB?q_jmmhREMGgN`vtH zlh%pKDaW0bReGII|Nl(YXM_V!x+opy5 zvqPV&OWx1R7g*of5JLNC>;hSl7;Mb`abA!OXlKe!g%DI%pONRGoQ?{xPqrWf0ii=* zMXr#)?|Dy35m(XQ6GHx=nQ}BIQu+i2%D_GYr7Pwe1DiVnCdWd<<|Smo$|%f`)T~s= z%g>iVUSaZYlU`KsV}0(mRUws=idjuQKDb0Wv~MqW4Zc(QdiAw?A+39d85}}V-3SOL z449gMg_*%h_33ckI_v(;J}1HIyZrbHu(Ms1%JO?;XH=*>GNG+J56m`Gr`D&?B)dk#pdV^=k0_ ztv(9@<-}e|PtB0|!SiJ*u(@=pHRW?uckEjK{%!k;j(rJZP%1c`C(5Kg zlQm^Fc`+>&;+4rI%?xPUPHxkrC3lA1scSZcEPD!tB_=KMI~l?&}O< zHCH(AOTc(i4Wj*sj6*P19FY44-v<_zfr{I{#$;3SMsalQDB=CWC7@e? zbOOxAQeZXXPbfgqs1kEBMrz8=$O~hhhv3xX8h5<8ejDHM*-0r*RS#fnK{U9^hsvzJ zvox5QL0ab4VfMGQrc^DT^wNG5@saFHIH-Y>TtAeW4tQ}}g=2RlBN7Vc3Is=$;$BaZ zNr96P59DppaLD<=U~wu6F}X2VJ9f&L-eYBQ&?LRrHicsJohj0pF7<`aUy{O_92q}& ztc)BuQbKx$C>+odoZ#Gf1lS`x9%X~L)Hun^%#s&JzbL_8LAqX=8>_lw(C$A`>V9eb zP2fKTw3-X2VfmaV{eAj3SU_c+1Zxw-UkY49Jfg>lpI|D$`Yh~5-^{&pA`sumEG(y? zp-hDk>?i|ecbo&F&_#v?l5f!5nU=p}6ge}zRG&YMi0pWf;dW9Nx~&^)xyEaM;e;n{ zBmyZzDm-1h_qxAN>M z#X$FltCbWebsqpm(SIBBtyH4j7stPV`v@}sMig2J0z3kw;B-C~jw2eh#>Lt6dRaZO zLMyz-i;7YSs$;rN7o*R^30c$xk=L7EkHK~)BDTgT1)ec02lq=um1zG zoNWS@_@^>2be=A_b?>{a)_K3U9#SyfleSl~O0qQi+6|d^tkv};GU|MK_*1!i_?`09 zsHapwWnK0PsO%S?4F+*Bpr)|80MMudz{B-W=8V>PWb4kgoZo3tKbCz=U9y+kU6h|p z`3Z!UiEt}-x2bSYfbHVws&kN}`rpC>$4q7nDD94Y+1ecB2++z#OX5cdKZ1+8zq~s2 zRTZ+=gUGOetXC)65xu~KoL9D^W0^DGJ0#E`Y&FHU$7-ji|4i)>A);CaJ;*?0B`HnL)V(6dzVRmS%D1bF+g#v zmd-W4w=Yl&IF~u=E>E z>xVKAsXJjzz*v1?0=zm30;OT<776ng>3+RI(Nc9+zo&3bftr}l3}6|3ko%afxuz)8 zL>V-cRt8L__BpIkiU9k>3CBKn>DUDV6e9JDQoyn5ubYz5XST~Av16sjq<=_%=?x8( z=K97IZj!{wa6;w*G@ezEE&aUvDP@>ZZ>^g@0vMMR1klZco2TzJ!n1K)ecrWQ%qu@E zuJJ_)N*5K@BkUUYrQw?4I6pZ2eh8F(Y%ap~^nr^*AOzJY1Vg1Fr!w0mGk_s5a4}j3 z7g%Cdf{YFut(IzAi;=zU(i)ot3{-JhG4MX2n)=RY*Pc3BkE#C%P`NvXMn5Erhc8y4v%U%r*UwPzP$caM zk);uzU`=kPZFBxxYx|dLel0sPx64Db9+VLwBd!RimW3tL>oq=eoeu1deb1?#tcv(n z<5m0-WM#=J$n(;HKmX5lzY~WJZRPbDKZkd9kh<8d_0?$4c;_?U2iYP=`j3)4q(?r0 z^rj3nvK7~`K6Zn=vExmdKXjfvGx}-uK3Wq_jrL66$-C_jqfo!@{bVb=kA>h_?cc2* z=p2D1#Xoa}bffP8m-4y!FT7sp*jhGfP<7MeMe$Sgp{z)#%WuX2^L*NYYSxr&O z)g*mViF=6+W1kto=myVkT#m@WTDq@mn$kX_|JDj871;n75Of+Y3T85*HKnn|uqvjO zq^PT5I3N)AYoUWK9_L&=P85jiOixZ@cGa^5G23gf0>)v0s zZ>jM9x^{AvVE+(FEK86KLQ776jz9Ua(YSF|<|Krmc2a2VJeir5?$O1e~DonsaDv*o}PIzJ7kv z!?TuS)>6Cm8E4kv-q~FS1oa0?z%ivzBL=9oG~WpD{v;x>aarUtc?sO6%;v&uIZaUl zKr!~xUz`4#3Qo%o|4ThyrpK6mX0YIv zgw5b1eMlA#Sg4jd)8Fg!nR~zpzi@f`{30;)#tj%JcTAltf8G8DQnp8${(Q}6!n>e31XFhHS1Tvs&13d zBfnId7Qw#0T%bJcrFMBg2{>m2Kcfig)-u|(bdNY59>~O%QIV;n z_~gdpcdY9`T7+bQYjsM{WSc;F&R;G_+Aih>EuWmvmfGd_w5+F= zo=um~fKatCTdISCIB`21YmW2Ta{%0*Kda9Wufg1T{H`a!$^5m!|6yDO1 zHY}aPz$varWZD=2>k!?&t)=CT0N051;8#9bk|Nd#q z=LUavh8vzxwf#updigTea9FVGKKI-MzXrA4g*NBpk>QUh{fY;g;MU(i^iS~%>?vahjJ>+AnN(Bnk8Fc{@8>U{lRa^Jg~mAs9a}2b&b}*0 zaWWLaB195cVn^zB+_bV&zya<7G5)F(t7H?RTsNQGEL-qOFH5&-Z6qFw1lRd!hnaS0 zrD#lHpUYt#Bp!u9B&H=QLjY112rjkX(h8b|=+-?5F4~Z=K{h6AL>kTG;Q2abwI5u_ z{XpwLD{~G4VH(@#ix2=SL5@qIs4FInwZdq-K>fYDljT zq;cOl^PvM#svxsmI{UNIk0L{IAq^h zC}@{}^?+GchzYV*IFFdwOUD@Vne!tWse75k!dh_zHOFi)Uy!(;6jv;)D;J!Y4vtW8 zI=}^I!JWcmDflO3C19Kn%I2e+WoOJzIh=G@N{~c=^Tbji-X}CLRQ3UfO{{NY=cJ`} zTE7>G&WVwt>U_BAM%pEy@?IGK>={4K56<1w;3Q;%3e&=9{CK2wrdyPon!Tky^I>A2 ziQ)>V4!M9S1htlyI|8||x^0TzC{I9P8Q5)LlXi3rSRzITjFwLTE1|Mpw{VSXhkVY9 zM=WkKfyroNUO!L}x_X15;Ir%2GVWXN@A;>U92_o_!X{fZk{ET>=e?7Yi;NpIPSUDV zF|J#$JC)ffS4ORpr=Sp!PN^viy`3|VAZBIMH$;Pz zd_lQxX2?uUU;C%ce^CQi-NQdW{G~X$caSNAr!)!N)2N^F!I%MKJE`De zkUN};Bwra&+HI|IywAPd{E^ylj(mRP^J_X;2xYzobK2JYHBREXN`C77U=PTIXTcnN zz&O@;rx)UGP+1+!I3SEbZY?cy1i07j1%$}6i)XzARBP*(>6LOGlAoJvcT!QZ=}({| z%2Y@SgZ;c*?4xdCyY{wC8Z<(uaw~Hs2YZE4xPK6Sn0GE$h6WF{=%qD|=eyCa@oxH? zxz9fC-K39SfSfE&RGGjyh8{pda2z{!kkS65jjye~Yn$jZ|JhTf1x*2cQ%4QtI)m66 zqcEzT*${gJvr2e@J;mPZWnH}2Pmg+9@~U!_@0t6SQ5YbWQkf$DJ^jT6LcFbFv}?TE zI^M05yNntz64YUdN|R;jc(b8wrAXOGA%iY?d$4=FP~ty30fbKhgBz=Z9hg6!L$Y7G z&uR6d;M&l_-1Zd2IzxbBe@u&yn@A?)l>><)Q=xPcobBk~q=FMIdlZ7W_|x=m&DzB} znX8__bZVHKytbkNP5V*zGwYz>rqOk1*;=92V+2z$vrPcff_CfF?do^d&ZVH*2^(uM zHcoC<%*|U4<-!#aj z5#DgESSt*GBZnl@NIp60Nwwxvpywuk5_BnqTySn$Dh?owCLT8f3u9$?r8xuBw>SYHQ zBtWQ*dz%RiAKjV1obgMA+*lezu8EUDDf9l0e~2vn?x@tFbbW5geQc|c3w3$SH{e}9 zg8AsJsfO9MbjDbH@NFDnfPl zXVZEpZ_qM4xL25b1Is0G+D1u$@aTr18`Dfj#&4$YyxyGR9EGTv8l4!?-^~rru7$0( zBhtGM4H_zYBKNA-WD)|#S$GYprK#9+?GqBYSH;2WL5eRL`FNiw8(AwfBt01z&%Zx#kokDy=BsDWtdCngi{-D`r%nGSxz(7#Y zq#$7J0*xBUwU!B&J%xyTZ0y88vV+{Ttrc$k#LzLfH3bu7YPWW{)!!;0q+Y@eqC400 zApj&&2KPql$pM0hG?4jYg0kIyh-f8G8!f>5QunC<=`fBj!s`zMa*tQ@SM;~V5^X- zhTfH|@+=vAE)467d6ofN^|b!pd?QeSw5I!#_RA}ie{8c6dRV@J!oox%-2oFx1=Gv? zf%9eIz=c+gLKGmThs==w+3-h^d%kNXsUQVQU>gXMgTDLH!l%Nj0{QE%H}Gq1ya%~K zX|)@JWIoo4JJ#8OaDjgUsv<*3<(_aT`LOg0>n9!CTQ?Y7goiUeGfuLLvn3^muo(mg z`vys9uTaqc`DmJ0OYQr4`%6umEs|c8DU+_~Xp~LJ?f$z3zp-@M)$cRQGvzNk{-OpU z8aUERG9jpvF6xI84DNGlWhS^*>H6mL?t29dS9Gcf0&FW<=9%Jq6>3fYlR`(j*VNLb)^pqG-Rz74w|28W{azp20sRAm516zfH5~KA; z1It3|;1@@~RO^*37gJkK7gz{sV1^9Ss8ZW!T1lCy_`6Q$m5IE)9sw-TwG>nr(GG=Z zYA56lWnfq;ZYnc9#pK4~@1e*tY#EHLW2&!Bhlwng)1=q9sjl_1o5|a9A^l4>@s`J{#y!cGuM8DzQb)`8^qJrwpkmgZ%d2k z?_SUVBM?bREKo<+4vGV&pg^uU;-syWT0zx8GwZP3i&yTIJ=9MO9yykCRDzI3Zwg#6 ziG_*4W__ZB*Gyz#?LP503aYGw#hCwXFb2(8cz^jE)wahgZY@nc0))tL{&cW2nx9G; zQ=d*4)o}m-KmbWZK~(Nd-K!e}qvpwI;nLBz+l*@%y*g@{z>{+UDJ?mt7$aa$q0&k! z9j0j{Bxo1{6D9ZY=O0A)a$zXnR8F@KHkTi4GVr;hLfq$8N9x3bfmMe0p2SKCQY$k#n_&?JSm zeZBi@H5lAG;a(MP30t#P9NgyAXJsNBxz=!*~^;8N>YyIYLWC=<{ z_N|#{Z=j;!^8zcIQRG1!U~y~fgpImuM04~-XDGy&GG zD_2|qilT`02qn|y2E;zlYG8c|lPLfZ?wE|^7o_UWNyuu@BNwsIwqxF&$OicgA1%MBOYezRJtAqq^XW@P3=4orv8P)N6E?l@Erz)Wp z!sW@uf$h3L$RzB*S|R>)NRJ`f*HzIgH95m}WJFyO=(}Z$sJ07XoC19yAQ8Nu z#J~~)KufYcg>VR_j68|WfL?-55Ez^g7-g%_)lP^&FArQVA3j)BKPEMb$()&vfNNZ{O5y%1WFEPU&R-#brw#q<7V%&N`*l^szXq+(|Lu9lDy5LoHs9&yr}u+` zYHMj(Bf!0lUOaj#?J11d?hmWv(uk$<^2#4!|Jbd8w3f0#nbyfQp9+f+oJ(T^t?_lE z&rQ!U{X7EcemNI^^35yqyUo9Y(3GmY*`~)>`b_T-1;#VjZ&|qKV*9ac|6#ugQ{xyB z3Z-x;e71MLrjUCcilO0opvRTo6B+`n6)KXnr1tCHUlj<#)w*N+0}*e+7#7xJt56Z8 z$C3L5?=|;sC+wN_6k2M1V72|rw!g@~Lokknv4x3&?7P=(F-{$>KJ%miyNG`@y4X`_ z-R5|6jA@1EXWj#Q3ZbRU4$8gOTroT9O09%JB2Ltn6izpEZ;#&Uawevfe}}>?O-cdP z)+`uyC`?fEAQco{{*B#SDXdWd<4rY$)N}A|Z8(O=eqd33AlJBFxz}`=^#wkLmR#nJ z-wv*>UT(cKRT#-`Efr223f{afUMC-<#uQF`zNl^45@N_$KoTM5gs;6g9xFK!h@$?+rceoj5xj@AZrD@!oDJ(WR9>F+ zviy4XujTm(&m$N$MXmexoX<6;-IWc(P*{w0&DQyu;3nP;3vM4^?Oy@G`+v;-4>^tm z1f-&~r*L!T+_Sc>*i%pIF-<=LOfyNWJ-N&6Ez?7$$sh0hBdEm&AhF0g`RmTVC`Q!Q z`CLmvLG(`83Fm%!b%oMQl77*i!uj*_8L!E!(|>}s&{>v5d@P?F{=}YBH8_#$H(`V< z^rCfe0t8WmOYN$n!3WV=TXwEYJsGToDNy~~d)gM*uryr2b6X7qov})A+$sS5p*PQ- z^_(fBpaLy$5(**LfxSqxar>?;r?}AV`9w zSVc;tx@23DTO5}pPMet|Ghb%jyqAn8$t2$-j_o8)abi2RBuk1qNj3IPkVGK}qW9i= zZ*Q%Ad8teAmIL5k5~S{?pxe$lyPSRY{>%E;uQ08zT>gdp{kgwa5J5}lQkJ0=lXHu0 zrUib^aaN@~wRG-kfA{h9kwxcuaEv2x$Vf3NWf50LOCj^U(tS_Ruc-x#0BRCT8JGwd)_bCIYOchQdW^YQ} zo2mqH3R{-I*t`}6Cjz)R&QX|zTzKfJp|$(k;1wlQF}0bc;^*lrdvo_#I_&oEe8~ET z`^06@Sp_d~{?JWND}8l;r4ks;(LwQkJPMb^uB2TGYf07yEpvpMJL%8(P?O?}uYRzk z26+St$wvC-C#9~ShccevBJ;k0nvfcN~z`+(Te|k_6uqr`N8GLwXN;ehHdU#{zfIV9Ug6))}pV zsdb3x-Wt0N&zl{OYH6(R_3sXj3{rNJKCq2wDLt@=lM$sLwgANOzm5C4MgHL3KUC|W zC44jY7`0yd>E%Du`3j#fBp7YdTp7y`TkR0Qg3 zuTLR}LJ3Xo#1T%yj9S_xa`+m9jzqE`sN(s)qflVuhQW=*lJLM2NkhbkUI|-t<*u@D zQWF4JxzQHxw>81bp4UQS#pLi%f1s-FuR1K~C&Y{o*Pvno5 z;X&3?FVEo^YY|Y{CJGkk8};>mru!8nb8uUHegQQOPf8UfA&uAz0X%4;cZ zFxHTo?q@|mlXuGBQ2~5$fw4-Fv2;EaZ*t~lMx?7bOxP5k17<(cQfPF!)O1N%Pmfj} zMW)6YKuMCcoT|KSL*qjd2@56*74W&KIh9OC(*;&>(7@<`__D~XP zF3o$hGkK>Hu77m)NAg=oG5LP)q3lZvC>f$CqA(fNs8o=$Oma=AHn<-ci!@3*= zN)awFEEN2re4{1YEmEqQtJQ{R&3bu%$EQHAH8vsXLFuC@<-C_~)1EPy z>93#uYm1mXD%q95v3fdtNh*RTDBv>tO(;Acy}+(BfYMT-6@)>5|M0>;$UlGapXJA2 z_(ypv^GRhTv2-pe-pE=)Z@_Bf1T58M|9$x*KsXw4(WD5|Qphavc*f(3)3ms)M0rMk zcjxcjmGH*cwYZM=A-6jPMDDL1Fs3kR2@M}wN+wcovTm-8l;sd;SMQcw!JE$HQFF?K3%u z1X#c9FGB-NEk zRoNG~O;VUJ6B8FM(xTApz1qT!l`iueofU|bbyXmJOVQd|>-6{V13oqmH|U1^dsB;C zZ2)K-!f8${`AxN6+TPNpt&h+QNEA#zB+S)+iyap-3N>bsNjW z{t{cu3|MCx831+@flvijvDk24@nOCIAv6dSChH=zZVRbwKR0^fO3yT&QTI_Fn6;d~ z89SDX5bHe)3?LyChAB~bygwG44N>`Wy5ppDM0F_sQ>|chWAzz+H#|NJ;kQO&yka5z ze0F~gkFmeE-Uw_<-ily^Ybs)vkPurGy4_b4oi~a&TB~hd=&zyHrNfH zdzWP{d{4+_$=Qjv^ zrsj$V_U=bh_bYch-Sq4;pF=H$35FP4%<eEc8r?%ffv+7=7&ln*(E4bctrvPbUAzK(}>>@Lx_Dofwt&o;JvVX-XzB zcCN`g?FiraR-{H_RtcKyNoT-`wwl5Xous~Jku9UPhi-&SMa#09YZ{)*z?}6_x#BpN zCm&vOgEenf3RCAL-aL1_;U(^R&o(pb(B+oPs?BNO&}7uOwF=#86l|ig*pN=D77BqS zw}Pd*DX`IkS_EzCCbCB4M6L%O^-iuc4_H;#AxCklUz*&kzo`i1EipN0sdpUOSViMa z>F(&3UGcjVxxL|ei~`J$+X$Cr3^GW4s-fI9s#*`TYrbzyjS~-nraQLwB<;@ zx)Ui%4GYtu2D$|TUwBMerVM|x=6h!KAqhCyQtV}IjzAz3h)2`+fq~>r%^pq)J@lk$ z6`1JiXV$;m=sT@q*?>)PouOrv>z|fFH#h}ctpXDRYnkyZ126-9VV%3vc1mSk)$NFX z{cqNM<@)Tg=pkXgpV3VbYPK`w5jk@6i01KTpBYD?b-dXJ(#25$(R)1kT{x`RsSw~! z)Sb{g-h>Je#;A3?Sr4;@riG>57LNJdmZSn1?jHf)Y%4$7r06`ZlknWco4meBA(fMY zS$np`Z;??59;a$gsjLfJM72&%!&;Rffy_S?crSp6o~bM8f@K;sYZcxUPvV=^-| zD;IBEQZBUB^q2H**XzsFIB$#G2FrBXUE{p84ffBg9{~nAW@F7FBM6lfYZuNbQp^Q; zhCtcAQ#eq{ETXepKn{Mrv&`C6tLsq_Cldud4n7vRh;7mU`$aX40E6hrbn;QfM=HCJ zrjwR=5NWP1fv5V(tS9d{GbJQNyqvtj(S5=u$`WM@a+o(mFxms=8hxP`j$X#@PNW-d zEVt--*PpkXS3ZBOAn^}!3{h*U{#@>zo8vYkfx<;8hv06D!u|dN_G?g?n7<$WEoaJb-v-tT`N_4~rGCb3)tZIH}_fo;d)b;Lc^F>8}L|4P!)ZI$c;R9pM zLaG@(ZPB%sV(A3#jpVv)v)=4)sXl$1>3Zh9I#F{{VOIzA4kC%g)k1Hi-X`TG6>Sg3R`ggSa zD5!A;RRA3!C0eyVeSeu{gM{fH-uzgie4^#?tjF$n$PMe~eWTlm%GYaGUk9BEf>U6X zVC19LdU-uAD9k8C%+b+`4?+B&BKu(3AXd!S+)#o!+*ZUge^~mls@K~SL0Mo`&^pfS z@520o(8W&y^?D_&9Km^r-rl92(dD$?a7^s)wT}SrEiu)nK!?^c+^k6QwXP{zYI(Qf zUHK4#$FFVrI=FUY@0v^GTqG6G)wZk36>5$RL`!{!$6SS*|EHh`B9vfDJc16fCfUo{ z905M-Ozm9?dcMBNer)gs%UZuI^@<+T_bc8dW|G`I|7f4$UjSz;q&gxR62RXE4^bC9NiI_X23X|d&?vt68Y#z?jGZKeQHkL(R> zGp&(vz@L42(@Uz&)_frK&8NuA2auK;wyO#*`i{{yosbHg2X8190%PN)Z?hU_wE1G= zMLAPLAk zK-CG!jmVX+Z2q#!%WiINuwS9Tb9uQSt4CX%63ji^-Ng}^Cg}cWy{675-z>+uy|-O% zHdab)U7hTT-6b#Ny?{ti9j%yQAMNjLHv%L#Cq>NPo%jcd2#Qc1A+Ck1DV%pD!2bWu zvF{^)_ls)5HfE8hMaKs&#E)-$BK?zn;CfC|QTV3jjg~fXZ6N{qkI(;DVtwP~OB=ta zDM5&qYuk^p-3T!9oCc&fOWu$`2oFifkY}y(M%nA~UfBoo`v|b&`m(enA%}fKUn5-5 z#})UgwUj09DSvhRFQu`!S^jX(AKaDxlnVXE*bVY_@q3{At4AEgXYme8+qm4%wCKHZ z>=jA(&Xix<{PnxmVltu4NXx4G%yYU4nimM@-hI5pk9J>sa+8$ zOgsstZ~oRa&?F^@D7*l{X&6Mnbn)(nb`%CTvnfHquyMtzTAq|Tf=hgpaNN!9%ui&Hb;N~$E~<;$7(-S>luaC)$FUB50?npR=;r;^4s_Gr;jneASb|NGou8Z`#74JvSa!BJkl z_L?My#N)Y-T-A6}IVBb24^RJ49!q^(?P135AB{=GAzy7QmFr!l*h4auH+yyCJp(1< znEc?xkL1;_zl!lTPXEL@$PjVl>PIprBa#@Nh$JIAYDdHSvYhu!v~#@VxU|-_%fEm1 zrzXc+e@zG6ByH&DTW2trdRc3^G!8aEVA5q)&{bb))px=J3oIamZI zA-Ik~AYwXA60b9r-1g*ciq~zmmXek9m$YBS05g-6o9SaYQ3<vMEU7k%5WrQ9Jj+A#5S|3|5O6a%IXj7)8wiy#R@2i+8HX%K z4uFyRI|WF%M}*{{-Yf*dt)y_y(9Kf>%VvFlolMQ^GGP%wPJxnt>@SIhNr^$49tze0 zu7Q$^oYI+iXjfXOoLp|aqBd*t0W*k;%ncl4RWCmCyKxBX^Q;#XEdFi*vLP;4@n9NT zRJ62(3emNe5~&@mmf`tfnOc|<7grbEIJ>&x-p)ZOK-LkAwF4H$K#X%3#yKx$z0i_j z%+jW%Ui&fD838J8-#_sKVDpa2W7!Af8wKCED_GB5P28ZU%%7}3DgSoi-=GZd6edBT zOP^MCbA`ghTP5$vFOX3tB{E6=XwM%(Xb9K5Ve01dIP!v5%ikkis^grq{HMLY4@`BM zeO{Rx_%-$$*J~yy`N7E_N!NIXd~@qJ6>0xU_AAC-oM}8OuN1v1aefK%TaSFlY)~`j zP*r!e{K=93CFzK^|MJF{f#duv&}lhC#h8S{HxJ9}MgJi$KuM!#${as6Jx?cUPpVX_ zg`2h_?qr(;`F@7u)LJ*MLxaqz($msb*D1fV^SgkH9WZ&iet-NqQq1zX{Z`>ORW!Yx zf86AWMt)$L*P_zszNfM0Qz$lXTzgaUqYLEsP`6>Q*3$F1)?I12D*x@>|1KLg_W_AP7I(eH&t3u-@796!w*GvgO2R?ABXU?B)J3gvA^JK@qmy$sVu4F zp3_rB&=N^ub$fE5Y>L}xcEjMLH7pb)tyw7SQ)r~%+zZQQKgPTh4|I9^4TV`{!UYCY z(2X2}ab`p!1*_E-{RY|rzEO?`6_d&N2^pOoRndyRu6_tm)72Xbx(sU7mNr%`ZQ{bm z2Z))-y0G}NUJ{Gb63;nA0gd$%R~iq71?DMUsgSs|J*<<8K?icQ(`rhOz}dR9&Yt>6)Y(77oLtTKS^y$127-!~N_JCX2*A>~aU?E9Be|ev+G0sc- zZvVXU5okpgsJ}e+my!Zc#M9YNE0-BJSY>rFa#_xC&aE0Kfp3<+DU}^n@&sslUfA#) z*39K?v{X?4%iDhlc#gMxdBe*Ja^ec*jr1|5$~cs9X2CgHc|<+}Y+z&bMr^j@jK+jO`CQNC07ZH1ZCQZJ8T@L^?7C4y%UOIdTd5)um&3r#*dhSz1L z$X{J~T~0Ng62EXC*|+gg3HH;@NHH=oA~kI_a_I)Jo@1`^U-$nJSS{|1Za1vY@IA@( ze|7au7;-9=C+!JvBAXMc4bP>_K;{B69DN8@m;}FM`OfxlDp-Z3J%#uBB0RiQP$w71 z<>iep$(Fb+n)iHKtLZ8J82Hyq^_S$S?5E`EoTn5$qhTA3y{AxpxcrD5zHwL_0vF|A z!9j@(iP9|bjDr)uv8zeWm7kZX-f8)5C6)_u z^e4)u_>H=Z1u(-~TX56YYxr8(Ci2&ZMQXc5{?kE`r}7z`V|aup_xa)S50xeN5G)@Y z915;n=(+KexSE1=eJ`*!)u(`|jKT!mhg`ur0(`@((aJ>uml_6Bq%uVVuR(Xctt`bV z+YKfP!P_5PKdRdIeBLt(x3k*xtcHDC`d-~%ST7aS0MnV%r@+Io2`>rtiD*izBrIP0MmoLlT8Eh5BH zYYkGg)tK0?YkmZ}M!H}P{TVn+uSrsPf}{Y#l7lE+W{hE!JeA;4Fq?3_=%479()Loh zR96IE-EjH#j&CZQo%tHf|N7HQ|1R&By(bA_aZpslWm`g_vW$>#nd=c*H)a-QR2y#g z+?4Zm7i4N;R=&FRWwl9}zqU+{e$Z5DRKn>=702M} z;3fys56C0QJJk86&N22qjXJ-$@=IwS0dFgCoAg3PBU4fWpeV))Q*~E#RVXjSbr@XY z{NuqQ@PurL&0DrHs{yHhUnzN2dgi+%B0NG_RO7=Eg!$l&l~H6js%fj1E1>V29G;S` zQH7}cka}i~9dGWtGj(Sn+@FLO(**{t0wt*CL+LZNPjFx97-^Sth;uns^QpiYq5`lO zj7|9Ca?2~Qz}CPM5)3zeE?9JeU=(Cfm9fHgrv`Bpr=f^2WowX!F2MugZp-DUCp_tg zXa7#xrV&FD5+IR)TSNv$nY3gR9$42=D_5$oNndxLocSC8QQWNWGUS8vh0Y>aWi0tf;@X3)!WR*=uRJhzPZk)-r{&p+(LV8F0r?CoqTS zfQ<$(Tl!8FWV7^ILlZ*)jyFj=tZvM{L03EJoS3DB52+;$bm{e$^O=D4_bH^Jp67F) zvT0q*X_qaI<%3OO@n3EWsZZIW8l%@F_WB>s{400@bKuV1XEn>I(aP*!KJX*Jx!MA& zDl@6wgmskocV}k?0nRF@iZe3dQJ>}Jv_2+Vp2~SrVnLK{jkSVxk3a}M2^T2aP|{oq zXpSrDoB{?inVc!@KL51^P7!e-$oBIHDfDkcK8Fh|#0v3fF^tkWj z+9Y!dE%o%=(1O{57yueqnC+0PDD)hr124bQLMz=_1niRQdt`P*L9+aiM%52VftaS3 z0tw>>ST{X`uJ9DmBS(DcF7Tf+Nx(`A;e4mTxKY|&CWG?>GB!UZ{=R<7pb-%mp$t*I z!+kOi0d*W1`hr|SB-|xZDTmtFmWxI zf>6ov20wxF%EkJ{$E}$yx38-6Gq5}waHtm)tK$(JOZFenPzn?{Z8@8Km<{t$aVqG~YD5S2oFNSrj zf_1@-?sPPfK5-*Le;^nn&g{k-D%uk@i`ZkGs&SXvl;YTxvij>_lbGv^+2qQ?Zp zAaWC2kATTA3CPbcz5@74pM0b6YjEj>NfdHp>z_w!%@x*7fq-;DeEJC|AXf2g?FHcQ zFUnJSPb07n_zcN~oU zYD2^(2(O_kR)qW4YUcwzllzdggb9G^`sxAI>Qh0flZab6a{WUUqml!K-`CAYw#7ao zu|d&_SDGvZt1X0IePP23vTe9fx)2!M+TW(KLowZES7R64;Yb7S5iQ<;#e|241Dca8 zi9rdPT+Qa%#^;Cuxcs9hc^rMM?W&btun9FIDE6Zp#}K%!TNBcQGBDm@YP?C8VjM9W)e)S z1wvX}SGrhJuMMwd&G&>ZkZX-WYx;3!f5^GYT)Yokp`b`h>maS!(A24| zQ%j=@!<41m8e>4XN9S;tOoIu4@0kvciv|@HTfsf7(Qf-SY(D}z1=vC1!-)}31pLVd zR*e*_BcxojW#PKUf3%|TuM_bCoq)@54I>1>UOpcqz_lm?X|P!~#pWxR#|04cpR7KM ztay(jKqpQ7JpF8<&~rau5B3t4i3G6cVDF!s`vnX-L-K{(7huo{7GEe-wkQ<-_?L$? zunJxSyzLNJg$S1-?1pQzxd5Tnv?(29%90D11?#Trs)5yZS{_N-0W0-ZwI^wHZ!f1E zgefv;nPpR4zGA~UT>g;?Zhvn5^Y8@g#@iOg8^%iTFHiLuY&h5t*>{1WObfcR+K*Q+ zcia;pmDs{2WJ;;)Z9qUwmlB#{0)btL$WW8UoY1I7-^Bm=F&>S*BB7Y`2RG=fsY?%~ zAafBWahlv2t*<4ph*k+s=7F(6WtGqg(sy22Z3s%(g}}@I%m}`Bo@u*J#5C6Tfa?@v z+&0**gzMDMWJPO5;`o&+1Ga1N>aI)|(`7IMa9*V#&ok-7oPp!qp0rg3Ij!#Mrq3qE zryAH`(uB1Rv+^G?lOf7Buzh<*#1vMgO^n z@{iU#GJVrPWGeyHus>XX_4QDQx|f9z8m6e$jrFZDw}SOjOCPxRDC6I)dmV^-&9>Z& z;&aOgM$UC>mnGpv*1o*%dJM1$^$k(xUt%*T)DbMa8Y?QVObaTb`V)aW-VMyKyODSr za-BByH!E$5?zW{G1+9;i{wYY)b<9FbYAl3k(jv7Cv?-Ez^}`zfA0vAs{K8dMFyeIB zw@ClURgd#p>^9w+6c*{uqCidy>L9e{Pj4PWW;Dz(8x|@$Q5aWRY}^bd2-ru!J_4&BfqP;}r7)W4mH?q|3<52X_K5CJ~_{3HQ%!1$5P0EiQlN)mNyS9&X#autK*pTSQII-sBt2>~w>B;C|TNH;`) zFmWF1Kw0Gs4QXkiJVN*0TCq@)R*QHWMio~9&>at2ne_0~rOke*J`?6l3o;doc_und!DQg zwl0LKP?>FkH-Hc(D(19&>DNeG*Kfc6+86;EXE+xKpVJraZFJX~i{H>6yP8+-!~DCW zZtSTf=wB=Q>T3FD>fE_o{pRmZfIpHT|6cUh*u7>O$WU-E>StZcEvphz^}$fIYEcke zz(UJG&`&%rc1tan%Wa^k<7W{iJ254aiAer}WI-Nu-I020ClCB?NjVh^fs~Vm7F7z=#LbZGKMDdXc}n@c5`1tW_L=?SA1!GKi0G$aWXl`OI!R`a z%GKDij~s73?o{6%Ted+zZujC{Y3kKHqO@PmH`AC4Xnx5?C(j2Yp)py!ixip&J) z5A3Qf3%#6#YblyEc*eN@`oSxno-Zy<)%i6BP!t0{i${>x0A!vt+4r#qes?TV_A>2y_DREAfLaE?5 zlS(DRACYn>445Le`LV41I0k`JP#jhYQ0IImbb#|_7>Wt;(+zz-sQ0>hkQw+wYx_#C zZ;kn2zsAEe0x>}`@-)_vv*0ZqgO_Pf>Q3Otla)2o*1e>Xehq>Ny+-7NCN&Tl31%QD z*vsc@1h{Wt8Q%@tt$H^k+sI0h=}wl0KhjbiR!L|_-|e-^Zg&RF9q$UUruo|EHM z$hy>9FFR9rs37fEi{6m$?)OxCUT)h4<)Gi6wGZ5HI%R+^8vQuivTe4wHs=+yFa5IUH3*1}O0#e5ZxWw2 zZ^gmKs6ht6J(Y7%DHJ3uXL14JjaOT`5D4|N3%``M;Wj0xvi|ap>!9&Dr-Gt(r9L80 zt$*USkV=Jyo4kHbt+syq`Jb;5Pzf76T$Do3c|>J-xg3Hk9We&^ZGtfJjeq}5A))%? zNN{pMt!uXy)0JM>e%=E&0?ga~G}bi=E>vFjVb9(aw?XaMw(bn0;tBnud_X`yh1389 z{5&jPK~v*L_#k=JH^=A0Ko*33rb32b7_oJULF_~{EVh)c{*Nw95g@!O8epWZcZM^evk>*Yo2Zcza@~;TA3G{tE`%<+1NV30tE_UVH1JVq=}nu4Mx80g_(N@mOEPM zXqlvylmZ}O03^$;L#Bp`&I*NjbOa&pzO;QpA@yDrN02eI?WEEAwCcEw&J5ieXGgh? zba+`9XY$V(TKBjCF;fyB_6Myf0UIum)`52UxbjnE_8LQf=hdR+*5M)P!5NC5jq`_n z&W%}z2kbKo`D*%0rGyyfPpUsfdeZUR^V!AGUDA=4nnEh?g}yL)l8GzAZz*B=@9nSl z5qJ}(O@goLHXlK(=`?ROPa1NfL5f0Bv|3fmM@|1^{r7>@0EgJ`Mmpe#EOMf zZq|{=WI;mNVp#MBC;NfpgR2Ox_+ZZvMOw==XLOP3CmqRb`M26KJRxwq>+q<|F6Den zNTsFLA3`bdqcwdUO1P1JQMsxwH5Mtd;Vqhfj(_*6P~(ciIB&)zzSLAC*2hjRdg@h5Jk;U5(Wlg*`8OzA+vSR9kK;|(_5T~U9wgo zB;@hz$CQP-X=p%qSGN!xBJpUx+=$*TVw7Q$52Xf)56dx9xluT}*Vs?85^RL~r64X}njth(A#jD8J3cT@Ho;xMpsyf)yms|XDFNP-rLnI`d2Q+u971e=6_X6tFi-1C6{g~N@9->Dx_Q|1j2b34W6|k$PbeXcFXZxGAEQfj-~y$Q9Ok>y z?)1*BWGp6;yocjWE_xaX8AHLew{MlauaseTK#9mul!E#m#IhYOU`8fO*_~E0VBHWP@3`dM*U)=b-{%iG077=;}R!h81j4R90n;**H6xOU;r37iC zS#rWRLP*`K6fFu1R0R8wTbxiQrcXQoTst>2dwK9jU>@-o6iWa2gZ~G6*BmhRv+x>D z%gMUq%2*TS8z*1e@?}ML#{Gqb`zPZ?R!ciIgMIgto5$ss*Iq&0^S3$3kJo+*A=zEB z!q@4HSq4HXJ%-HO$5uS3g~?sM_u*dvKG>@SRSJ(M>yAOT0Drkpv^=x^X~~QPRUrgc zj(^|(nIplZLi^{eV$>%H2%$7}-@gVk1V z^Yd3Fq_TBf?5GvxMX#5J{#po3!HPbDpkOr!R_?;hHj!E^y7@_qvObQoBWam*^gyhN zRyex*c{AK$eaj46CtoReS)PM{O3*L`AQlQFOv_3^S|3Sxd(vR7FHG8^j^Xt`t^Py? zCI%qzx~NDvS~htO1FZN=u`cz`hButk%_=CY zwX_k+Pk^FCA?24X3-2)lv$iE{mAA`VF+T!jd&)MYjM=TIwjB9~1_e^Rzpdl#Ruuz+kRahOMeV-flvsku8K30oSbyu zF>r}d|BSxxhecuvn$s-$ZLR*YDf)3HhaD-Fm;;;$5{!@))?sf zcn$WKo95*7gu0j9&?&Hn`GhO1D*H-Tmg=t++o`K}T1B_cz~Zn2o4K2dC%AH>lw!iU ze0!X&Sg-w358nuIv)u};z7N(ReWCJ`7M3P2YfZJ!OD)~JPH~T?577lou;hsn(hqQ`% zC1#}!rg3U_+78*1vI_#S=k3dQI(tceOup=d1z(@GRR1G`eHjz5bRX9L{vftVO66YTa3u}-P@>Xh%qR=}qHY<(Y&TwV*%k`cjSuAgqd50MxO%rdFi-U_yO-)Jj zP&4Wdm&danQ`%y5K;&&K>T(93nA-wiJ}DH@gkIlYFGa0IGBQ6bvx_tM)79&WdOe&y zCCob%7Pn}L_KQ~7E(*0P?JrIE6W|FYp0>8PRz;nU&WuR!OpnAz#fT56zAE}|V4T~O zi#i&@RJ?y2HlYCJN?A@n*TS)+2b-9kQ2njysa80ik=YUHpYE58_zcV|cQvmX1{;*Z z5#k*JH%6M`MJ0XDN(;FFnX#&Wq8~Wsn`%Cf&mlR$^nhfCU<-to3hABd7I$A)2?+5QXCEhV^>Gy+*6Zmgla5p9@35}F{dG#9B(9Yf zPt{)ot;*aO^EZ8en+BTYSj{mx({x7a#_MDV|2WS6!G4mSkSbxJ(1JZ2WMmQbjx?)r zW{}40{JeT72$p^j3gHyAK(`FG%12e7$mu4K9FNzl{yKU)h^L>2^>nOwM zZ{tXlGz~QZ_E8H3WnL*j%M}=Db*A#sjB)-Dlx1g|InE8LUMFuy+-o;gZ!qe0!h9Zr zV$w9yAdQ2lx2qN$a|?+24^$Q?YR2|r9|6M=U=kAsQINiF9vjA{*aG>jo!?Pw(sI@z z!)t1N?+qcDK~xtTiV*xcgiKP&@}Kwqo=TDYAX}=m_TPS-ZUne4QJJ2@o^T1-#ir*b zB|B`rd||_LDyrO;Ry5A>2xM)%*;x%EOSL#VxJqHdHVCNCN)~XL4{M4kkzizZ7d1*blA zzOzKQe`Cc1{aspAcB|n&Y42;7fuVjm z-F{lS$GRjxDj%-uFooS+dd||%thblrj3BgYUBzmgef@nTGa*w~FR&*|3+wG2>Q(*i z8|ahMZKtGTq+JSQ3sn{^VusghVSh=PbGiA7ly;X$<7lJ!2lz>LVz%lpQ^qYVv@o~! zwaU=QkhFHS$(gp(h$8Kj!nm!yr?U=I%xr86TDAjuttIaqb@N!9tL)=9S$-we(JXzCY82PP)6};5Kd; zXpoHXba((FEv|Xi7&Wq(klJQ@(l&Xcv_&N;q3hR{LWwonZNG-~M}RCgq$A7-%aDsL zXO#itNw7uFFhkISwhUWwG05ggEcu83jYrx}EzGZdf|-@~s_-a3}=zM>SWqfPe!0iTo+4 zIHqO*%6aMAy1m*|Ubj;=OG2V&MYrI^^=1H*R*$DQk2fhb2TW}!l&YPmIQUIN^j_@= zR0sUL#or}p+hr|ZO}VB_vCf8sjBFYlr8`n$#=Zjxqm}K_8On6c%%C_e@=4xF#&Isn zjdD7Wl`GuGLtmwPnCEBXORCqFT@;O*zlqUUIL1b6V$iTq%Fc#E)lz^^eu)RpXJir- zV^0U{Q@k2UP%sJd_}U<=eEp4(AjDxSxslLEf{4KB`~$PY3QyrrVOhicQU!4Rt1{05 zWD{ILDzYjvrO?+4&@VMZpQSwx930_yw|@lwZ(R`969p2qjxvgmosFlFex#gbzh^6= zv;Qr;0ng~OKC$;lXi;SaZr+J9CMy7-2XR@OpETO4Kc6A(z$8%RG0{9cD|UfBiQnUl z#Z9OyGKlaO(Rp$HVk*skO4b#o^u4Waxr)5DV`5+IZKKHRat+>;qOyTG^9-#6%zIm+@@R-j?37jrVGFyzuVJ z0mMecr0kp+Nrpt@T|9K7tW~T^+B&pfPX2dFSCL_slfPRDf1>-Qk9~P9uPY#$TU|@Uwpr%K3ubx4M+7@|E%s zi_^rwtm<^+HHLtNN?{;Jau8~5f^1ZikQCG-3euGMm)*I)?MG<)HqB)2Qt2-admgIv zTX-}a$aY-E*Y+sf-rg#n_YN!+_MJCB{u?7H<0d0x6wRLW6eH8jyRl?IoQ6W;8+--$ z>+dBlmiX7%rGl4mS`gSIwrz1Gb28uxmc}^I*K>mF5`w*2hi_Lt-5*}DN{dJ554>Eo zj4il#edWUu!XD=DcWj){rS{GetwPnH1pf8b`5Lu#=XLL{(3bx-qQ% zWATZBw)<^K02AGBbIT|;rq|2y?KR3kO8*N{4uYApwFWmuz+BSFEisdhlj7Dk{LU() zt6KX;`WmOIS(Hxn&JE*Bu)cv=wX?#(t8QzkT8z{CGOEzaY()LOJ0F0NvSj5TZ~@5^ zN75tffm9pM+>@)@`c%Gvpw^F5tH@@s3H1OTg01H3$pxiouzTCT->;;iN*dT7$jrq- zK5fcR@vi&~4SJWN+`_5!%Ql#O&rIEi9^>z{w4-(cYBf#LNu0SV!}+m;CpTE9aw-i2 zaLjK)c)I92^D!R}`gg2w(Rvs!4-T;%`@vrx$7^29=WJC`ct0@GJy2Z>p%M;^Qf(&! zOl*RLqbGEYw7(dLm@HjT=0A|xWV2};$nj~0NZr6%XQd9%O)?&0nSc)@+GtM={$uJK z(7Wa*;F#sbc{&!TAh{bkCEV;=!CS-)_#O9z;rHaono+2HtiF69GDEUa>l5me z1}$3A2aB@ufm|A1ix(Car3B4^(NDxxcOt(y#I)31)JN-z_nEJl$;-lO=_^K3-1r6O z{A5AzF23jil+HZ*Z%&OJce|CAWhrkVo};iJsrqaF+-HX%y6}F_;l(J-XUx=C&_8)k zVcMnrmXuO=nHsH^8W$yWygLsdj>dYUhObxRIN~jP66|q5@f`5uD-UR)9|OS`F;jYP z`p~x;w$}b}m6cZ{eB}P|AihWPTOaYU&w&LW#I8ldt>&afu|=teu6DIjaIn`^{kM@s zvs^Q3bQ?R0q7izPtF-&48p5haE7@+eak$ZXe6<*TdJzEBwNZy3Zd}2 zyI^ZkciZP{4bQ(ip8oYPS^ZFHnG8B~*~Dr|EgT_cNux-Hr#mw=lO-st@u_i)lTvLo zy9^1F09;fxA^^htl&NlmAc-LPxqKUq0Sp8E2vy3_xE^MKbYVXJ+8GzG~StvwY$N2aNSTZWBM^;BJy3I(Vrlh?DChhd( zG7jm$$jnpLyR*PPJsJ)~kn?aec^L|?v?Y^ZMm5@%wc>W_v3`&BgyQRI^ zL4HhABR>sp1Ww&%JA#wCg~~3D*rPgkk*4bO1LE~9V6Ca^mi#(~^rc=zVLv>F>?Q|4 z1-B$%LI<*30lx~3DhLz#2%%A8|GL^EDX`**K9zuuXzn8+C}zmW~q*h*c7aX%8Kd9GQFT=IpHnu!zd_V>RDi=q!D zpppudIrwJM^3xwY097e|f$iv|rl>7I{h>uJlQs>REM zIwFU*BlREb?C;#8$G0>@su&He!q>>GcGmed4TLBeF850r5$oLG`@#FLqV=nu7nfTz z%qy?8ehaLH?I9dkg4Lzs#45Wcf>eeS*%XD`9_aq!pPg10Z0F>VG{|I7vCCokbZ~_Oyzq=5_Z+E%f9J@&A)C3b$F1*&u_fs zx68#-%f9mC(wfeHC%8c&6r3bGvb>YexAC=$2=bb~J*$cUU)?0W#MMMocXL}9$z*!q zq(iRDQd!0;B9rN4IM=J&`w`Ec|hFuQv-e9w35~U*yjW z%0G@^$)U`XFiMZlJ}cxMd`f6QGqilmHUyfHP4VE{H}40<9M?9f9`B86lUc<-dZP}` zVm>6yM8>VjT;)ph@*oNlRYmN-IM=Z>{K@>hZ{4K4+^X5DUr{1pBDCCWVIVYtzDC4` z2hNLB>FL?Rj-hae{!B~TO*dJRM32_{oTq750%rMx+rz&RlCJ{jMdDl^nx!EpRzCYb z$0{dBhcnD%6b_?vY4qXLnhd~@kvj0;>U~qj3CHcS&6N_D447o5w4qZ*B`H>wS18JE z_gCPY`^5|bI8uLt|1GYsfoAyy)?Q*<{vtoDpwss>9?3?@O9ZJ0Ild}O(4A8!{*A{D z2Uhe!!UrfP2BVuQX~$cZgZ2^Vkdq{mlq=#R#YJV6oCWJog>}!?DTd!it9rDZp4EuJ zG~s|AA`{@%H0tSGUw!d|2DLFwUE-QOqz16nh);x5G_n{;CsgaIN6mAfXXB5;jjD8b zxaoav{Smf0n#`7>BA`ueeEHo}nqHnBKs21*Yc_B@tHT7D+ST`_Z;M5P>&33lKIp5t zs7=2Ou%7Cl#Pv~U zH|$b7No*qN?mnh4^kriKb@d{{!5iy;aj<1bOA01P@Ba@A8~!CYKd)Fehz2_U<_Ic-5u_8QJ_ z0Wge%E@Y+clA?{vjjA;MecGLk?ys*e&<$9RKQh}w8Z8O#PZLfIf-7T2P96_NmM+kd_Ome|dX|r|lO~H-u$6Hw#xqwV^ zWvA3LyS;rW{BK#+RXXb|sBb}u(2tICThCW~wc1H~%tHMOhcjYn)U~t$8|C0C-gXv> z-oOE3^TvsPlw4#m9&*2LmuNbH2OLX^keymaNssJCEztQLWj8X~)wMq|drBPvOyB^X zQJ)@fJm4#hAXzj--Q-~ygr0JA+XaF8OfjXCaKy;ePw?1)&0-0u_3?$Sjt6W|6wtf* zAN$fHVIOKnGDsCfH_v@E*AV@~#jh2owb~u3x)WT4oCAaE*QDMM##H$fJH&A;y~#yZ zP^Tt6dlxN$xnA#z_9`-{SEC2w5)^c|z{70hRNXBFdeeNn^4;^*G{FZYjn% ztgC`NNS@P^l9i{sYMKJIz3OS1i_IRV;CfhY^ovA_L9XU+RWb*nM?%Rk|(>@_Ez~zDCjB{2=W|K)eVy#&e?W z8RDhl%o@zj4lvb`0cwJmQ(gl(*XdAY0JRk}h(8ECJ~|4?rQLysmj~Qs-}u?f(0x>Lewx9dZ}=JkIl@J8|8J3OVDh%l-~8e(GPqJ+E24VmWr2{kCG%s84V zcuboul%tQC3;x?=()cny0EEUzzH69U(adq_Sj^E4V$Lu9I##Zyrm&TlyNYy)Zn z69e;63XlyM5SO78;(gi=bi(Dsk~a$B)tY%`m=n{Jh`OmggooZrebxrPFCOG*Y(zhS;Qp{`o>`cTe`QaHF+%F8A% z=830fHBK`MeG3UTVhzf_Q{Ki0>3bM}*4!JIZDR#cvV{I}z{ID(9Mc{%52x^?enEwV ziv&PuX3(NhgZXs%NI(GF3bn^sTD0J|Y)|*#@gMO-kRnT}HRNI=fGqn(m*EV~dMl2# zSL1vIEhIFB9nqx@oPf#Rzr-Z6`iw-43u&I~mt9m0U>5~4TJS9TtnlIjR~HnTXWC<@ z!tJD2>9oY!Os)m|o{5$KnQoR$$ z!mOE}h76Bvj&Gxm{F$y`r%TcLEDnZMG_u;Xk#dKNlXGn01JyQaDxdSZHg^NynFoIt zGAmR8;s0I6L?u*z_O!)jabi6K?by@IvDa1%bAEVTnQHn+lg<@&Ew{7eTBkKAr_|EF4 zp{=jk6}*m{4I#SSwN+wT9H3}Mt8ZE{e5>6d_K)K($|`E7qyh5eqQUp&R}>I$tmvg^ z0(O>V9dMzp@y5^xa#8xPLu$X(0kjh;+KE-_ZdxDJujxGgZf})&g+w?81Y!#BkIQSB zP6OQDkpV`bWd3!h6AzP~SgJq-PzdFGr9z@|U{ouDk>-Cs^9zU9tIw#|(rr?t^mFw; zr*XX@#*=-O6Q)AJaC-=+V2IKz5AH+5cTuXaamaD;LaxlRgLCmSQiwl$DPu1OEf29z z6+8BybsMlGBOkz6J%GH-LE7|Sj||{PPU7y)&-M{yQ=Z+py8G)vH!-cNLzs$Lke831 zQt88 z#hQ%7YF*PDM+bX!-T9PYY^?Y5qYHYuG@hs%Gn^3%%ke%3a`voLM9|6!Eo+-agrzZk zV)ep-tnAqox~L3ly1#`gyZuA5*+LaXeed#vWQ3>>YA@nK7G@jUr&#M-dVDm>^ND+i ze45IUwMGBdef2U(1TNl!|H2}tBSV|VdVFwr_2?_zku2}o=^HE5BQqyc5Q5MAqWEA@ z<>*%KrN0}CS4Z0jf?-7)$KJUXHw`7F;aguQYwlJ*1=RSE!sEe5=$-&>p<5_njm@Uy zidMGyW=3%K7JT?#oUjJO#$T;-hQ%OGe?Z>nnvz;&A4e(I0Bh*ZsHk0H^>9;ob;?k+QKu&)GL_&5*LF4G$$W) zEQUW|E%MZM76GiZ=~iAY9JArSN8AG;^eThTi%NG?mRONIN z8xnV3qij1rJR?(ptX0?wzh_wn`L_(4&b?w<6+AY*vXT}=#P#>GIW^P_Yz>0jc*y`A z;5Byv=xRRS;qpapjSmxL^$_Zr~U$m&1h?tc8 z3PFIO9(Mm`2C9@6S~ZXo*dxq2v7IxL;MogqU-#W+Yh)^-i%Z+&$*?do=N^D0T$J># z=_n~j`RQ%2zy`nL?z3vf>83N$H0~@tE=FdwX9Tcna_EH7P-Le`;vUV!xImu^Jaj|b z+}1jmOT}LRy(7+_R5ux_B*WYUd3erhXarO4m{XyW-zD57i6){mNC%rbko}(Erkr!( z`|Xe5F9jfNLOy48!()d+JXk$o`b_peVJkGilIje5U4#rE0}m~bman} zuqs399vhv8)QD6}dQ1S;H}Dbc>kz)qxETlz2WhRLVa!0GgSpbT-QcL+*eNMVH8VLf z_f* z+Td-Zl0lWgQt`N0E1_2QSWs%| zW_odn@;dUSmw6Dj6t056_te3Z7L;~m#k7QS%LSH|&btk-qgJ6j5!`i>2*IZdwu1W} z0TOA!soGRgX+kwr*3lM7gf6AHe*C;^!M3WH-TZ6&3@{c*OG{Hgr{dMqX{p0baG1HzXb9i*w~ctfiK5Q(9J;SHRPd7AJl)z5fVXm zEfew#r4DS%2yJiB`}%QXjxfRAZJhfcd!#ec;!cAOF^Ex#b+b&t7;}!iLMS6Tqwf_a znPqUbTkZ=uex8$)f^?7Tu|J<8xAb-+Q|1m+v!TiIs$Uf_>!4in(UZW%!CL6ipVuc6 zHsO}@Wg{6Ys(HHR=<&I5zQWN2E-WfodWn0PG)e{sx^z+SsY!$5Z*h^KD$$sT+S%da zK-T0CDc#1-9^~$J-#Px08|u%Jt{hmV=Yf5bQo4n$B8> z!_7kSAlsoE)=2y$T*T=&nh-65EhnY#(Nf~{yHZr{R>nq)ak{g5(l%>vc{ENr5ln06y!n%c2YTG_IPIqYixpDPTS-)^#Aq9++EUah zYRF?m%1f(bLG?z@>Tk! zErInY7mpj_UZ4=EDhiTq1tlK_+CM}UDo!9tR8?qNx;Iz&n~|>kUa;h3?V_D)M)<~4 zMBAQnmW!OXBUM?eu!7z%70U{xTM%+ z;y1*5@zTAGN9Jq~mD-5l7~J>|`A&>;-AC<~KS3-QAY%zbL)AEh?4_ZkO8sjpUI|Hu zuva&y_02n(xc@0;!%D57V#Kv0Lr^pL!MHHX@`mQT?%(Dd}R)Zr=xvBC60Ee10O}Q zuBrrLY5|BV+`$$4}m7HUq{DO-g5z!r)>_L1X_v8#i@eFA$}a zB2`yNM&3IeXP^{^uXnJ>$H)v`3D2d3bp1x0Iq8v}UK6gI+8%&_kAR?bJ~j1gVsjnO z1C>#W(WlWAop5fnuGL!I_xDgEunz-Km(nmfi(G2nV%4mqD>`ZR$la;vNU1kC4ObJL($@yM0ZN9Cj@E{KX<`!% z^h`ArDI+IWu1`r#stL0%`cl$le@%Q4<6m7?d(!Yi2$YQebk0jQ)*`SbPGS+>r)3&Sr- zX@56Tx<0s#3_2c^;AJK~=5Zp!E`KRaKv(Jc)Y99A)5O7j%UD=iC_N%mG$BV)v)0Fa z$+bG(mq$HCOCp?^C`9`ppA4r3u$MK%=pT;XjK>|Nn6}P33-2~Gf+?NW6!@C>azk&f2yCLq(#5_Z3rowAqE5l4A8B2DpEPtsn>*&louE2_^+bk& z_0)eO3zu`e7<4(O_+5w`bSCF#_&I!5BQI}4R-(xlp8%{RwC-hE|L@sc?8!zy7aLkWTi+Ryl09`54$ zcVe{{HADVMR0y+H<*4z5@FZn^u z)9AV!rV=O4{2Q=wY&ts|W60l~7(0@Zgp*Pk^ZU9{=!s$uWI6B@PXD6G=wP0)@}6jG zKS0S)`8N^y7fs2+J0f7*LT^kr+$lCOV^NjbG8#8Y6I=|YG1Ix}vdOXdrII6xa}4D) zl$3#5U9=g;is9#OSJsW}G6}O8P*z@+a*ip|R-K|{r=yjXNzX^p5ZK^Ic}WPOC4gfB zf-3{e4DolIBs^B4QY%wMO?D(llV-m&vGf_KUc$OfnsE79 zNAFq=u*WgAOk+F3YF`oU&#!8=%Yibij5ufal0D|G$yTsc6i~NhwRXm_rKcH= zV9G-up8f6XrH-$%?48A*GE9vB-h~}xY*iN|VhyVy0Z1m324Pr5)sWhFs`;H|En6Z@ zUbTP9HYKyRP}MF&zAvfm#wB|(5LqaKT#v2=i%ckhPDmmEnO6LvdhKe!q!^#xvEI~xA@UN3i@#6sAmq?9u~}+Aj_Dhu zoX>9Zs^zjt78h5~<>uQ(2RpfICWT!V!t&{yBW7|wF1*VDBivfxnnjdA4L zS}cy?_Dkg-aWXR+A}u9y2vmR`B-iE%LN z4w!Cj@e}FFq5k8k8_PZNn_og;Y-u)_xGNN+k9?qR*b!gYLcC`6nqp6LU3Edby7%#%P69!$cWGnzw3gRULRvW)K8%Fq@o%3G zB0QWpWII*k^9+451gR9($TS6t8~d8!UI(MI?Zv+tTZQ^Mt+>vZiMKSWakMC%vtm;* zGr~;QV5;@mb%UTP6BjDulF>`0?wGggA$q(stK}LJs+$-cjp&$!MjiuROZEJ5V)yO2 zRgjE^`+f9sO#|U4J(hsGgVRua)b2*8_Ef9R7Kozp$%hQ(FCLfIAN|3C}{zsb(57W znO!9gkKfTWWE3du`*0<)cu!us36&eFZ7&40d|0eJ`yFnn(TCfcgA!``jrJMT-u=`nH~ys zF}clM&9>SHB#%KhE61KKBxM*e%llYjg8>SZ@HhcA+NX?|GJ%q>oDqK;-c2;gift1O z7LZ{?q#9L##i6dzw%^%(#8YyqU#rF;LA%EZ#Et_q!%-D+mZ7xvTN79d1dOCQ0= znly)_Q^+(B?_3MiA|Da0BU}p?y!X{o1jzU|Btl&@pTHonebpZQqrX~coU3k!2laQ3 z5e}4fgZPDmeyJdN4ht;~s48|!io8^_aHG0BDycCsW=q7&?)%fYF58Y z^qQgCa%elKFca6>6~9EuM@Ux*8bLa1(UYVrv93sQUME%>fxd$<@Chxg!ByKUBH?|C zKxUL2tY$LqvBn##^?BLoWu`0Iz!c0eGW#puS^Hgfphw&2Y&*J^n#k`AcpN3QfmGC9 z?lJ9Gi)xg&9GchyIT!~pk2}>gQw|@g9Ab&pCj{l90mV!Wk zFH+$uQGLuOVxK+MVEFBlw2ziG(o}x<;pw^lh2E}Iz(ba^tNx7B+JM@_5G$p5c<%EE z64+z=v-pcpx6B7z3-(`{oZMc=lq*W;smi6KsqFd9VrW;0iw2?-ehh!Lv%vfD{&&F7 ziIAr$4~(jAA>k=-pY(dJS*|k}c%87T;Z|M-$mrkx?eaG4mWle*-XID?{>?5trMBWp zs+T--F2-Mm9pgi)3!PSsj8y9{5q8G-Y_F<4U;Dx$RC#pth9#b12KRB6l7~N3XJ-t% zgLvQiQt;%uR=&IJ8T&Fgl6YSB29ab6?F=N;D0O*tVZ{5t<6?yLYH(s~A+24xGn;iV zU1t<=poFC)VALq8hu$VY4vG%bH3d0CYEU!*!G`R-%%IQ<>`f3FR3vCn&lDFinC)dm zWd!Rwvh8K4XNX0(LD#M(9ncby#@)7Jo5Uc*=Bd0p;+Ja|_Re)hR1FJ6p9T^pTvt=> z%Onh0uBScj)O47U=L6T^1tM0&bCqS}V=a{WznY9zfn_e9IA$#x@|q)A+ophmo|Fl5 z_LG127HFK;g}4@7*EDaMkgs*&@Q=ePwi%4N@2@)dU=LvMA16ot7z?Fv_9-o86 zs+pjXhFK!75#99;hA=A=%%VNx+60g?lOyY6DPo1yDd-L&!?%F!TsU}JFFUM<|AIVL z<;URzB*f=r+GXx_{D>e86uXgno!>VF1+FIDkxPAi&RpQMLS#gg0XJQIT0_L-RrD3u z74|m=M1Pz<*;`sBa~dPFHYjd)9ftC(|FIKA=fA7nL6{n8;Vv94nO^XB5{Q;%+y=Ae z;9+p$cxAtR_6Z6QTh|>h|K4l~sIM>?k;_AP`Wiu>UUqXS?)k!c_N1HXud-y`=DjQp z!7`4kcv=(v$bUF~F&IURUYiwiRyZ_#Mg7@CE@Yb2@syDL_s9mxPdZb;B@Sbxj+mpa zId?$Gze`0JKf(PPp_0M^8mTU>&R?rGdyoo1sfia9Wk{sg)TjChT#N8kbxGwCyo^;@ zxO5@AG0YT(sJ)Mhcmy@G-X`e-H;D1+*{)YEnds)htQ>0dsYHqYU9BYFz9AmhE&uS< zF)}v?G^%T&mal3vZr$qEqCzeso#8_|Lh(U~tfb&CeHteCe+6Yies znk7$Y2G}FMR%mT`KRY)_2>!9}yt?TMxI02cz(&NQr?M4bq;NpUvS-xBBAgM2qHTM8 ztQVgKQ>?$e2Zsv4zIS|o*7*9lO4GR7ZP~O5P99`YIB7}^>2g0{&jnufKf8XAIijF@ z;ntGsSJ7&u9k3jl}ry-SNe(Ad!>CFvlz$!J8cW4p z4_DoFU`2Rd|E^4NCav!)A52M^F@mIt=8Ud8nh`}$Gj1Oz^@I(B+?-y0X?*;1vR|=~ znGJ6*Qh1~2nQ;ye$s-Ht&*BMcI713b*IDwmf-S z%fPH0mtPJ_xg_juiF;osQS6Av69TUb^Sge#I%+#-k2vB*N^)bK-Fela|@p$9DYM>}VB6_Z|L2P(!UT)KK za-9nC$}Gw%r=`WTQMKsBhvZ>`7ZSD`w}01rVq{qM@+#KvN%9OWlpFi~StTXDtD(>p zYiNMJ9lagNckeFz!-CAik)i$S^5C3CDWxjR3eg5xN9GnuR4QSSVEfDl!q4}JqT;4T z#7h(XxAZ`tv4TSAEt^d%7CzPPcN~-_zO5I3c8f1}wx(*I63ccyNFUbgaKU#_H4r!Y zF|8Q$HSq4Q0nL4KRvTkvNeT3_!M}(Mgy;@0bAK7~%twd3$`=kU5x07GkNU&pqgmPp z&?5(vz9kq<_Q(Y>EW(^0m(ViH_(8q2*dyvAZ@`ORSd4f#YETgGzWJru4#kIc`&89g z=2+$@K~(3yC7BWv@di0BFRl*qAB2U&U|oZE%97=c*6W9(-Mb-KV2_s13hC3N^CI&`wT(c^}!Ie z#xvm2P1=n0BG3iWAu|HB`IHnc;R6Hzmu;K0Xq>(US)DX##hWA$stNk(9?G7nj*UI? zzv>rWp1&-*kWOk8`M!VDf5lx)*NqQ5)ILFZg0xTYG>U2c!Xa{U_$6YlOKeeaKXXa^ zo&@(*ID>j|!1E_eWIKI{lj{v{#m#^~jxA?M-<`il>9ZxxeGQ)FDG-2e#(qB|TZE4T zp);Yx8T8Tymow=#hm>Jv`EKh^uhA88p1h*oN$}54?g6sJAiAEIo;l@hz#uFljF~x{ z7~W&u14l0O2FI_q7YemzD<--X$7}*GzJ9oWq8QhT1411te4Z3%cwit+wEQe2H3YEZ12aGO~n46lB6djQ1LV8K|Zu23hIAUHVq;*ZVu-yW-%qSU})N?}M zibKUKEdWIgrMLpTj-liZqeA(6sK8ZJpxr6cOD)XuW5NN5xlMFb6 zVJiVlgC`*6^TV&7nu>|PUzBzYK?nqH;zepVjc9{8fbQatofg$JDh`qBxKqhG*))EMMKApr$=yVHC7406>(Tr5f@GH6 zz+yIW?tiR`-~TIm8lj@pPqSRzHU)nv{AUgV|G!A{0{dy;6}1@fe;WA3LX{?-k?6Oz z|DWof4U$~Bvkjw4^#ATd0ql>6K$Z#t@cbs)e|4<@c7zY?qI8UJ)b!tD@gv?L0XZvt zgG#%q{;whW5i>sjyp!u0vjY5ohxebs68uu2SQPJZcKbio{;lEySqrVh0J{I3=6`1U c64vt#Jc^Hx!BZ4a3iR_xipq&p3+V^^4<}zHfdBvi literal 0 HcmV?d00001 diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 98615acf0..9f536e9bc 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -69,8 +69,41 @@ post-processing, as long as the chain's view reflects a fully resharded state. must be correctly distributed between the child shards. * ShardId Semantics: The shard identifiers will become abstract identifiers where today they are number in the 0..num_shards range. +* Congestion Info: CongestionInfo in the chunk header would be recalculated for the child + shards at the resharding boundary. Proof must be compatible with Stateless Validation. -### State Storage - Mem Trie +### State Storage - MemTrie + +MemTrie is the in-memory representation of the trie that the runtime uses for all trie accesses. This is kept in sync with the Trie representation in state. + +As of today it isn't mandatory for nodes to have MemTrie feature enabled but going forward, with ReshardingV3, all nodes would require to have MemTrie enabled for resharding to happen successfully. + +For the purposes of resharding, we need an efficient way to split the MemTrie into two child tries based on the boundary account. This splitting happens at the epoch boundary when the new epoch is expected to have the two child shards. The set of requirements around MemTrie splitting are: +* MemTrie splitting needs to be "instant", i.e. happen efficiently within the span of one block. The child tries need to be available for the processing of the next block in the new epoch. +* MemTrie splitting needs to be compatible with stateless validation, i.e. we need to generate a proof that the memtrie split proposed by the chunk producer is correct. +* The proof generated for splitting the MemTrie needs to be compatible with the limits of the size of state witness that we send to all chunk validators. This prevents us from doing things like iterating through all trie keys for delayed receipts etc. + +With ReshardingV3 design, there's no protocol change to the structure of MemTries, however the implementation constraints required us to introduce the concept of a Frozen MemTrie. More details are in the [implementation](#state-storage---memtrie-1) section below. + +Based on the requirements above, we came up with an algorithm to efficiently split the parent trie into two child tries. Trie entries can be divided into three categories based on whether the trie keys have an `account_id` prefix and based on the total number of such trie keys. Splitting of these keys is handled in different ways. + +#### TrieKey with AccountID prefix + +This category includes most of the trie keys like `TrieKey::Account`, `TrieKey::ContractCode`, `TrieKey::PostponedReceipt`, etc. For these keys, we can efficiently split the trie based on the boundary account trie key. Note that we only need to read all the intermediate nodes that form a part of the split key. In the example below, if "pass" is the split key, we access all the nodes along the path of `root` -> `p` -> `a` -> `s` -> `s`, while not needing to touch any of the other intermediate nodes like `o` -> `s` -> `t` in key "post". The accessed nodes form a part of the state witness as those are the only nodes that the validators would need to verify that the resharding split is correct. This limits the size of the witness to effectively O(depth) of trie for each trie key in this category. + +![Splitting Trie diagram](assets/nep-0568/NEP-SplitState.png) + +#### Singleton TrieKey + +This category includes the trie keys `TrieKey::DelayedReceiptIndices`, `TrieKey::PromiseYieldIndices`, `TrieKey::BufferedReceiptIndices`. Notably, these are just a single entry (or O(num_shard) entries) in the trie and hence are small enough to read and modify for the children tries efficiently. + +#### Indexed TrieKey + +This category includes the trie keys `TrieKey::DelayedReceipt`, `TrieKey::PromiseYieldTimeout` and `TrieKey::BufferedReceipt`. The number of entries for these keys can potentially be arbitrarily large and it's not feasible to iterate through all the entries. In pre-stateless validation world, where we didn't care about state witness size limits, for ReshardingV2 we could just iterate over all delayed receipts and split them into the respective child shards. + +For ReshardingV3, these are handled by either of the two strategies +- `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards as each entry could belong to either of the child shards. More details in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. +- `TrieKey::BufferedReceipt` are independent of the account_id and therefore can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details in the [Buffered Receipts](#buffered-receipt-handling) section below. ### State Storage - Flat State @@ -137,9 +170,9 @@ supporting smooth transitions without altering storage structures directly. ### Cross Shard Traffic -### Receipt Handling - Delayed, Postponed, PromiseYield - -### Receipt Handling - Buffered +### Delayed Receipt Handling +### PromiseYield Receipt Handling +### Buffered Receipt Handling ### ShardId Semantics @@ -160,6 +193,30 @@ In this NEP, we propose updating the ShardId semantics to allow for arbitrary id The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] ``` +### State Storage - MemTrie + +The current implementation of MemTrie uses a pool of memory (`STArena`) to allocate and deallocate nodes and internal pointers in this pool to reference child nodes. MemTries, unlike the State representation of Trie, do not work with the hash of the nodes but internal memory pointers directly. Additionally, MemTries are not thread safe and one MemTrie exists per shard. + +As described in [MemTrie](#state-storage---memtrie) section above, we need an efficient way to split the MemTrie into two child MemTries within a span of 1 block. What makes this challenging is that the current implementation of MemTrie is not thread safe and can not be shared across two shards. + +The naive way to create two MemTries for the child shards would be to iterate through all the entries of the parent MemTrie and fill in these values into the child MemTries. This however is prohibitively time consuming. + +The solution to this problem was to introduce the concept of Frozen MemTrie (with a `FrozenArena`) which is a cloneable, read-only, thread-safe snapshot of a MemTrie. We can call the `freeze` method on an existing MemTrie that converts it into a Frozen MemTrie. Note that this process consumes the original MemTrie and we can no longer allocate and deallocate nodes to it. + +Along with `FrozenArena`, we also introduce a `HybridArena` which is effectively a base made of `FrozenArena` with a top layer of `STArena` where we support allocating and deallocating new nodes into the MemTrie. Newly allocated nodes can reference/point to nodes in the `FrozenArena`. We use this Hybrid MemTrie as a temporary MemTrie while the flat storage is being constructed in the background. + +While Frozen MemTries provide the benefits of being compatible with instant resharding, they come at the cost of memory consumption. Once a MemTrie is frozen, since it doesn't support deallocation of memory, it continues to consume as much memory as it did at the time of freezing. In case a node is tracking only one of the child shards, a Frozen MemTrie would continue to use the same amount of memory as the parent trie. Due to this, Hybrid MemTries are only a temporary solution and we rebuild the MemTrie for the children once the post-processing step for Flat Storage is completed. + +Additionally, a node would have to support 2x the memory footprint of a single trie as after resharding, we would have two copies of the trie in memory, one from the temporary Hybrid MemTrie in use for block production, and other from the background MemTrie that would be under construction. Once the background MemTrie is fully constructed and caught up with the latest block, we do an in-place swap of the Hybrid MemTrie with the new child MemTrie and deallocate the memory from the Hybrid MemTrie. + +During a resharding event, at the boundary of the epoch, when we need to split the parent shard into the two child shards, we do the following steps: +1. Freeze the parent MemTrie arena to create a read-only frozen arena that represents a snapshot of the state as of the time of freezing, i.e. after postprocessing last block of epoch. Note that we no longer require the parent MemTrie in runtime going forward. +2. We cheaply clone the Frozen MemTrie for both the child MemTries to use. Note that this doesn't clone the parent arena memory, but just increases the refcount. +3. We then create a new MemTrie with HybridArena for each of the children. The base of the MemTrie is the read-only FrozenArena while all new node allocations happens on a dedicated STArena memory pool for each child MemTrie. This is the temporary MemTrie that we use while Flat Storage is being built in the background. +4. Once the Flat Storage is constructed in the post processing step of resharding, we use that to load a new MemTrie and catchup to the latest block. +5. After the new child MemTrie has caught up to the latest block, we do an in-place swap in Client and discard the Hybrid MemTrie. + +![Hybrid MemTrie diagram](assets/nep-0568/NEP-HybridMemTrie.png) ### State Storage - State mapping From 65ece27c2a0e95398aa367fc5783991b078b6973 Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Fri, 15 Nov 2024 00:10:29 +0530 Subject: [PATCH 16/30] lint --- neps/nep-0568.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 9f536e9bc..7627a75d4 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -79,6 +79,7 @@ MemTrie is the in-memory representation of the trie that the runtime uses for al As of today it isn't mandatory for nodes to have MemTrie feature enabled but going forward, with ReshardingV3, all nodes would require to have MemTrie enabled for resharding to happen successfully. For the purposes of resharding, we need an efficient way to split the MemTrie into two child tries based on the boundary account. This splitting happens at the epoch boundary when the new epoch is expected to have the two child shards. The set of requirements around MemTrie splitting are: + * MemTrie splitting needs to be "instant", i.e. happen efficiently within the span of one block. The child tries need to be available for the processing of the next block in the new epoch. * MemTrie splitting needs to be compatible with stateless validation, i.e. we need to generate a proof that the memtrie split proposed by the chunk producer is correct. * The proof generated for splitting the MemTrie needs to be compatible with the limits of the size of state witness that we send to all chunk validators. This prevents us from doing things like iterating through all trie keys for delayed receipts etc. @@ -102,8 +103,10 @@ This category includes the trie keys `TrieKey::DelayedReceiptIndices`, `TrieKey: This category includes the trie keys `TrieKey::DelayedReceipt`, `TrieKey::PromiseYieldTimeout` and `TrieKey::BufferedReceipt`. The number of entries for these keys can potentially be arbitrarily large and it's not feasible to iterate through all the entries. In pre-stateless validation world, where we didn't care about state witness size limits, for ReshardingV2 we could just iterate over all delayed receipts and split them into the respective child shards. For ReshardingV3, these are handled by either of the two strategies -- `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards as each entry could belong to either of the child shards. More details in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. -- `TrieKey::BufferedReceipt` are independent of the account_id and therefore can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details in the [Buffered Receipts](#buffered-receipt-handling) section below. + +* `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards as each entry could belong to either of the child shards. More details in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. + +* `TrieKey::BufferedReceipt` are independent of the account_id and therefore can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details in the [Buffered Receipts](#buffered-receipt-handling) section below. ### State Storage - Flat State @@ -171,7 +174,9 @@ supporting smooth transitions without altering storage structures directly. ### Cross Shard Traffic ### Delayed Receipt Handling + ### PromiseYield Receipt Handling + ### Buffered Receipt Handling ### ShardId Semantics @@ -193,6 +198,7 @@ In this NEP, we propose updating the ShardId semantics to allow for arbitrary id The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] ``` + ### State Storage - MemTrie The current implementation of MemTrie uses a pool of memory (`STArena`) to allocate and deallocate nodes and internal pointers in this pool to reference child nodes. MemTries, unlike the State representation of Trie, do not work with the hash of the nodes but internal memory pointers directly. Additionally, MemTries are not thread safe and one MemTrie exists per shard. @@ -210,6 +216,7 @@ While Frozen MemTries provide the benefits of being compatible with instant resh Additionally, a node would have to support 2x the memory footprint of a single trie as after resharding, we would have two copies of the trie in memory, one from the temporary Hybrid MemTrie in use for block production, and other from the background MemTrie that would be under construction. Once the background MemTrie is fully constructed and caught up with the latest block, we do an in-place swap of the Hybrid MemTrie with the new child MemTrie and deallocate the memory from the Hybrid MemTrie. During a resharding event, at the boundary of the epoch, when we need to split the parent shard into the two child shards, we do the following steps: + 1. Freeze the parent MemTrie arena to create a read-only frozen arena that represents a snapshot of the state as of the time of freezing, i.e. after postprocessing last block of epoch. Note that we no longer require the parent MemTrie in runtime going forward. 2. We cheaply clone the Frozen MemTrie for both the child MemTries to use. Note that this doesn't clone the parent arena memory, but just increases the refcount. 3. We then create a new MemTrie with HybridArena for each of the children. The base of the MemTrie is the read-only FrozenArena while all new node allocations happens on a dedicated STArena memory pool for each child MemTrie. This is the temporary MemTrie that we use while Flat Storage is being built in the background. From 615a92f3e4f7b05e74762f587ccb889398b40874 Mon Sep 17 00:00:00 2001 From: Andrea Date: Fri, 15 Nov 2024 15:54:34 +0100 Subject: [PATCH 17/30] Add reference implementation for Flat state to resharding NEP (#575) --- neps/nep-0568.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 7627a75d4..720c165c9 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -225,6 +225,91 @@ During a resharding event, at the boundary of the epoch, when we need to split t ![Hybrid MemTrie diagram](assets/nep-0568/NEP-HybridMemTrie.png) +### State Storage - Flat State + +Resharding Flat State is a time consuming operation and it runs in parallel with block processing for several block heights. +Thus, there are a few important aspects to consider during implementation: +- Flat State's own status should be resilient to application crashes. +- The parent shard's Flat State should be split at the correct block height. +- New shards' Flat States should eventually converge to same representation the chain is using to process blocks (MemTries). +- Resharding should work correctly in the presence of chain forks. +- Retired shards are cleaned up. + +Note that the Flat States of the newly created shards won't be available until resharding is completed. This is fine because the temporary MemTries are +built instantly and they can satisfy all block processing needs. + +The main component responsible to carry out resharding on Flat State is the [FlatStorageResharder](https://github.com/near/nearcore/blob/f4e9dd5d6e07089dfc789221ded8ec83bfe5f6e8/chain/chain/src/flat_storage_resharder.rs#L68). + +#### Flat State's status persistence +Every shard Flat State has a status associated to it and stored in the database, called `FlatStorageStatus`. We propose to extend the existing object +by adding the new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is +persisted to disk, which makes the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat +State creation (see `FlatStorageShardCreator`), reducing the code duplication factor. + +`FlatStorageStatus` is updated at every committable step of resharding. The commit points are the following: +- Beginning of resharding or, in other words, the last block of the old shard layout. +- Scheduling of the _"split parent shard"_ task. +- Execution, cancellation or failure of the _"split parent shard"_ task. +- Execution or failure of any _"child catchup"_ task. + +#### Splitting a shard Flat State + +When, at the end of an epoch, the shard layout changes we identify a so called _resharding block_ that corresponds to the last block of the current epoch. +A task to split the parent shard's Flat State is scheduled to happen after the _resharding block_ becomes final. The reason to wait for the finality condition +is to avoid a split on a block that might be excluded from the canonical chain; needless to say, such situation would lock the node +into an erroneous state. + +Inside the split task we iterate over the Flat State and copy each element into either child. This routine is performed in batches in order to lessen the performance +impact on the node. + +Finally, if the split completes successfully, the parent shard Flat State is removed from the database and the children Flat States enter a catch-up phase. + +One current technical limitation is that, upon a node crash or restart, the _"split parent shard"_ task will start copying all elements again from the beginning. + +A reference implementation of splitting a Flat State can be found in [FlatStorageResharder::split_shard_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L295). + +#### Values assignment from parent to child shards +Key-value pairs in the parent shard Flat State are inherited by children according to the rules stated below. + +Elements inherited by the child shard which tracks the `account_id` contained in the key: +- `ACCOUNT` +- `CONTRACT_DATA` +- `CONTRACT_CODE` +- `ACCESS_KEY` +- `RECEIVED_DATA` +- `POSTPONED_RECEIPT_ID` +- `PENDING_DATA_COUNT` +- `POSTPONED_RECEIPT` +- `PROMISE_YIELD_RECEIPT` + +Elements inherited by both children: +- `DELAYED_RECEIPT_OR_INDICES` +- `PROMISE_YIELD_INDICES` +- `PROMISE_YIELD_TIMEOUT` +- `BANDWIDTH_SCHEDULER_STATE` + +Elements inherited only be the lowest index child: +- `BUFFERED_RECEIPT_INDICES ` +- `BUFFERED_RECEIPT` + +#### Bring children shards up to date with the chain's head +Children shards' Flat States build a complete view of their content at the height of the `resharding block` sometime during the new epoch +after resharding. At that point in time many new blocks have been processed already, and these will most likely contain updates for the new shards. A catch-up step is necessary to apply all Flat State deltas accumulated until now. + +This phase of resharding doesn't have to take extra steps to handle chain forks. On one hand, the catch-up task doesn't start until the parent shard +splitting is done, and at such point we know the `resharding block` is final; on the other hand, Flat State deltas are capable of handling forks automatically. + +The catch-up task commits to the database "batches" of Flat State deltas. If the application crashes or restarts the task will resume from where it left. + +Once all Flat State deltas are applied, the child shard's status is changed to `Ready` and clean up of Flat State deltas leftovers is performed. + +A reference implementation of the catch-up task can be found in [FlatStorageResharder::shard_catchup_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L564). + +#### Failure of Flat State resharding + +In the current proposal any failure during Flat State resharding is considered non-recoverable. +`neard` will attempt resharding again on restart, but no automatic recovery is implemented. + ### State Storage - State mapping To enable efficient shard state management during resharding, Resharding V3 uses the `DBCol::ShardUIdMapping` column. From e488308046352d6451f926ee8fe0bb6076979484 Mon Sep 17 00:00:00 2001 From: Trisfald Date: Fri, 15 Nov 2024 16:08:16 +0100 Subject: [PATCH 18/30] fix lint --- neps/nep-0568.md | 56 +++++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 720c165c9..04aacf072 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -229,11 +229,12 @@ During a resharding event, at the boundary of the epoch, when we need to split t Resharding Flat State is a time consuming operation and it runs in parallel with block processing for several block heights. Thus, there are a few important aspects to consider during implementation: -- Flat State's own status should be resilient to application crashes. -- The parent shard's Flat State should be split at the correct block height. -- New shards' Flat States should eventually converge to same representation the chain is using to process blocks (MemTries). -- Resharding should work correctly in the presence of chain forks. -- Retired shards are cleaned up. + +* Flat State's own status should be resilient to application crashes. +* The parent shard's Flat State should be split at the correct block height. +* New shards' Flat States should eventually converge to same representation the chain is using to process blocks (MemTries). +* Resharding should work correctly in the presence of chain forks. +* Retired shards are cleaned up. Note that the Flat States of the newly created shards won't be available until resharding is completed. This is fine because the temporary MemTries are built instantly and they can satisfy all block processing needs. @@ -241,16 +242,18 @@ built instantly and they can satisfy all block processing needs. The main component responsible to carry out resharding on Flat State is the [FlatStorageResharder](https://github.com/near/nearcore/blob/f4e9dd5d6e07089dfc789221ded8ec83bfe5f6e8/chain/chain/src/flat_storage_resharder.rs#L68). #### Flat State's status persistence + Every shard Flat State has a status associated to it and stored in the database, called `FlatStorageStatus`. We propose to extend the existing object by adding the new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is persisted to disk, which makes the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat State creation (see `FlatStorageShardCreator`), reducing the code duplication factor. `FlatStorageStatus` is updated at every committable step of resharding. The commit points are the following: -- Beginning of resharding or, in other words, the last block of the old shard layout. -- Scheduling of the _"split parent shard"_ task. -- Execution, cancellation or failure of the _"split parent shard"_ task. -- Execution or failure of any _"child catchup"_ task. + +* Beginning of resharding or, in other words, the last block of the old shard layout. +* Scheduling of the _"split parent shard"_ task. +* Execution, cancellation or failure of the _"split parent shard"_ task. +* Execution or failure of any _"child catchup"_ task. #### Splitting a shard Flat State @@ -269,30 +272,35 @@ One current technical limitation is that, upon a node crash or restart, the _"sp A reference implementation of splitting a Flat State can be found in [FlatStorageResharder::split_shard_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L295). #### Values assignment from parent to child shards + Key-value pairs in the parent shard Flat State are inherited by children according to the rules stated below. Elements inherited by the child shard which tracks the `account_id` contained in the key: -- `ACCOUNT` -- `CONTRACT_DATA` -- `CONTRACT_CODE` -- `ACCESS_KEY` -- `RECEIVED_DATA` -- `POSTPONED_RECEIPT_ID` -- `PENDING_DATA_COUNT` -- `POSTPONED_RECEIPT` -- `PROMISE_YIELD_RECEIPT` + +* `ACCOUNT` +* `CONTRACT_DATA` +* `CONTRACT_CODE` +* `ACCESS_KEY` +* `RECEIVED_DATA` +* `POSTPONED_RECEIPT_ID` +* `PENDING_DATA_COUNT` +* `POSTPONED_RECEIPT` +* `PROMISE_YIELD_RECEIPT` Elements inherited by both children: -- `DELAYED_RECEIPT_OR_INDICES` -- `PROMISE_YIELD_INDICES` -- `PROMISE_YIELD_TIMEOUT` -- `BANDWIDTH_SCHEDULER_STATE` + +* `DELAYED_RECEIPT_OR_INDICES` +* `PROMISE_YIELD_INDICES` +* `PROMISE_YIELD_TIMEOUT` +* `BANDWIDTH_SCHEDULER_STATE` Elements inherited only be the lowest index child: -- `BUFFERED_RECEIPT_INDICES ` -- `BUFFERED_RECEIPT` + +* `BUFFERED_RECEIPT_INDICES ` +* `BUFFERED_RECEIPT` #### Bring children shards up to date with the chain's head + Children shards' Flat States build a complete view of their content at the height of the `resharding block` sometime during the new epoch after resharding. At that point in time many new blocks have been processed already, and these will most likely contain updates for the new shards. A catch-up step is necessary to apply all Flat State deltas accumulated until now. From 12fd9941bac716f804e082f1b09a0f1618eba986 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Mon, 18 Nov 2024 15:33:54 +0000 Subject: [PATCH 19/30] motivation (#576) --- neps/nep-0568.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 04aacf072..3391794c8 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -28,16 +28,10 @@ updates. ## Motivation -```text -[Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.] -``` +The sharded architecture of the NEAR Protocol is a cornerstone of its design, enabling parallel and distributed execution that significantly boosts overall throughput. Resharding plays a pivotal role in this system, allowing the network to adjust the number of shards to accommodate growth. By increasing the number of shards, resharding ensures the network can scale seamlessly, alleviating existing congestion, managing the rising traffic demands, and welcoming new customers. This adaptability is essential for maintaining the protocol's performance, reliability, and capacity to support a thriving, ever-expanding ecosystem. ## Specification -```text -[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] -``` - Resharding will be scheduled in advance by the NEAR developer team. The new shard layout will be hardcoded into the neard binary and linked to the protocol version. As the protocol upgrade progresses, resharding will be triggered during From 03eeeea2b450b2f247da018bf1510d5fa32432f0 Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Tue, 19 Nov 2024 16:04:46 +0000 Subject: [PATCH 20/30] move some summary to motivation --- neps/nep-0568.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 3391794c8..310de3ec6 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -15,12 +15,6 @@ LastUpdated: 2024-10-24 This proposal introduces a new resharding implementation and shard layout for production networks. -Resharding V3 is a significantly redesigned approach, addressing limitations of -the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508]. -The earlier solutions became obsolete due to major protocol changes since -Resharding V2, including the introduction of Stateless Validation, Single Shard -Tracking, and Mem-Trie. - The primary objective of Resharding V3 is to increase chain capacity by splitting overutilized shards. A secondary aim is to lay the groundwork for supporting Dynamic Resharding, Instant Resharding and Shard Merging in future @@ -30,6 +24,12 @@ updates. The sharded architecture of the NEAR Protocol is a cornerstone of its design, enabling parallel and distributed execution that significantly boosts overall throughput. Resharding plays a pivotal role in this system, allowing the network to adjust the number of shards to accommodate growth. By increasing the number of shards, resharding ensures the network can scale seamlessly, alleviating existing congestion, managing the rising traffic demands, and welcoming new customers. This adaptability is essential for maintaining the protocol's performance, reliability, and capacity to support a thriving, ever-expanding ecosystem. +Resharding V3 is a significantly redesigned approach, addressing limitations of +the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508]. +The earlier solutions became obsolete due to major protocol changes since +Resharding V2, including the introduction of Stateless Validation, Single Shard +Tracking, and Mem-Trie. + ## Specification Resharding will be scheduled in advance by the NEAR developer team. The new From 080ac3cc60c7745aa8473e52a5fe32d612846eb4 Mon Sep 17 00:00:00 2001 From: Marcelo Diop-Gonzalez Date: Wed, 20 Nov 2024 00:19:51 -0500 Subject: [PATCH 21/30] Resharding V3 - add a few state sync details (#573) --- neps/nep-0568.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 310de3ec6..cdf15490d 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -165,6 +165,12 @@ supporting smooth transitions without altering storage structures directly. ### State Sync +Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync intended to make the resharding implementation easier to work with. + +When nodes sync state (either because they've fallen far behind the chain, or because they're going to become a chunk producer for a new shard in a future epoch), they first identify a point in the chain they'd like to sync to. Then they download the tries corresponding to that point in the chain and apply all chunks from that point until they're caught up. Currently, the tries downloaded initially are those corresponding to the `prev_state_root` field of the last new chunk before the current epoch's first block. This means the state downloaded is the state at some point in the previous epoch. + +The change we propose is to move the initial state download point to one in the current epoch rather than the previous. This allows some simplification in the resharding implementation, and reduces the size of the state we need to download. Suppose that the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and that a chunk producer that wasn't tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. Then with the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then when it's done, it would need to perform the resharding that all the other nodes had already done. This isn't a correctness issue, but it simplifies the implementation somewhat if we instead download only the state for shard `S'`, and it allows the node to download only the state belonging to `S'`, which is much smaller. + ### Cross Shard Traffic ### Delayed Receipt Handling @@ -396,6 +402,12 @@ For archival nodes, mappings are retained permanently to ensure access to the hi This implementation ensures efficient and scalable shard state transitions, allowing child shards to use ancestor data without creating redundant entries. +### State Sync + +The state sync algorithm defines a `sync_hash` that is used in many parts of the implementation. This is always the first block of the current epoch, which the node should be aware of once it has synced headers to the current point in the chain. A node performing state sync first makes a request (currently to centralized storage on GCS, but in the future to other nodes in the network) for a `ShardStateSyncResponseHeader` corresponding to that `sync_hash` and the Shard ID of the shard it's interested in. Among other things, this header includes the last new chunk before `sync_hash` in the shard, and a `StateRootNode` with hash equal to that chunk's `prev_state_root` field. Then the node downloads (again from GCS, but in the future it'll be from other nodes) the nodes of the trie with that `StateRootNode` as its root. Afterwards, it applies new chunks in the shard until it's caught up. + + As described above, the state we download is the state in the shard after applying the second to last new chunk before `sync_hash`, which belongs to the previous epoch (since `sync_hash` is the first block of the new epoch). To move the point in the chain of the initial state download to the current epoch, we could either move the `sync_hash` forward or we could change the state sync protocol (perhaps changing the meaning of the `sync_hash` and the fields of the `ShardStateSyncResponseHeader`, or somehow changing these structures more significantly). The former is an easier first implementation, since it would not require any changes to the state sync protocol other than to the expected `sync_hash`. We would just need to move the `sync_hash` to a point far enough along in the chain so that the `StateRootNode` in the `ShardStateSyncResponseHeader` refers to state in the current epoch. Currently we plan on implementing it that way, but we may revisit making more extensive changes to the state sync protocol later. + ## Security Implications ```text From 910cf6c7e3e2b22762bb7312a840261a44717ece Mon Sep 17 00:00:00 2001 From: Aleksandr Logunov Date: Sat, 23 Nov 2024 02:21:26 +0400 Subject: [PATCH 22/30] Resharding V3 - state witness, implementation (#577) Adding State Witness section and filling some empty sections. --- neps/nep-0568.md | 135 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 110 insertions(+), 25 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index cdf15490d..24a4ec221 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -163,6 +163,38 @@ supporting smooth transitions without altering storage structures directly. ### Stateless Validation +As only a fraction of nodes track the split shard, there is a need to prove the transition from state root of parent shard +to new state roots for children shards to other validators. +Otherwise the chunk producers for split shard may collude and provide invalid state roots, +which may compromise the protocol, for example, with minting tokens out of thin air. + +The design allows to generate and check this state transition in the time, negligible compared to the time it takes to apply chunk. +As shown above in [State Storage - MemTrie](#state-storage---memtrie) section, generation and verification logic consists of constant number of trie lookups. +More specifically, we implement `retain_split_shard(boundary_account, RetainMode::{Left, Right})` method for trie, which leaves only keys in trie that +belong to the left or right child shard. +Inside, we implement `retain_multi_range(intervals)` method, where `intervals` is a vector of trie key intervals to retain. +Each interval corresponds to unique trie key type prefix byte (`Account`, `AccessKey`, etc.) and either defines an interval from empty key to `boundary_account` key for left shard, or from `boundary_account` to infinity for right shard. +`retain_multi_range` is recursive. Based on current trie key prefix covered by current node, it either: + +* returns node back, if subtree is fully contained within some interval; +* returns "empty" node, if subtree is outside of all intervals; +* otherwise, descends into all children and constructs new node with children returned by recursive calls. + +Implementation is agnostic to the trie storage used for retrieving nodes, it applies to both memtrie and partial storage (state proof). + +* calling it for memtrie generates a proof and new state root; +* calling it for partial storage generates a new state root. If method doesn't fail with error that node wasn't found in the proof, it means that proof was sufficient, and it remains to compare generated state root with the one proposed by chunk producer. + +### State Witness + +Resharding state transition becomes one of `implicit_transitions` in `ChunkStateWitness`. It must be validated between processing last chunk (potentially missing) in the old epoch and the first chunk (potentially missing) in the new epoch. `ChunkStateTransition` fields also nicely correspond to the resharding state transition: in `block_hash` we store the hash of the last block of the parent shard, in `base_state` we store the resharding proof, and in `post_state_root` we store the proposed state root. + +Note that it leads to **two** state transitions corresponding to the same block hash. On the chunk producer side, the first transition is stored for the `(block_hash, parent_shard_uid)` pair and the second one is stored for the `(block_hash, child_shard_uid)` pair. + +The chunk validator has all the blocks, so it identifies whether implicit transition corresponds to applying missing chunk or resharding independently. This is implemented in `get_state_witness_block_range`, which iterates from `state_witness.chunk_header.prev_block_hash()` to the block with includes last last chunk for the (parent) shard, if it exists. + +Then, on `validate_chunk_state_witness`, if implicit transition corresponds to resharding, chunk validator calls `retain_split_shard` and proves state transition from parent to child shard. + ### State Sync Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync intended to make the resharding implementation easier to work with. @@ -187,18 +219,59 @@ In this NEP, we propose updating the ShardId semantics to allow for arbitrary id ## Reference Implementation -```text -[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: +### Overview + + +1. Any node tracking shard must determine if it should split shard in the last block before the epoch where resharding should happen. + +```pseudocode +should_split_shard(block, shard_id): + shard_layout = epoch_manager.shard_layout(block.epoch_id()) + next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) + if epoch_manager.is_next_block_epoch_start(block) && + shard_layout != next_shard_layout && + next_shard_layout.shard_split_map().contains(shard_id): + return Some(next_shard_layout.split_shard_event(shard_id)) + return None +``` + +2. This logic is triggered on block postprocessing, which means that block is valid and is being persisted to disk. -* Its interaction with other features is clear. -* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) -* It is reasonably clear how the feature would be implemented. -* Corner cases are dissected by example. -* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. +```pseudocode +on chain.postprocess_block(block): + next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) + if let Some(split_shard_event) = should_split_shard(block, shard_id): + resharding_manager.split_shard(split_shard_event) +``` + +3. The event triggers changes in all state storage components. -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] +```pseudocode +on resharding_manager.split_shard(split_shard_event, next_shard_layout): + set State mapping + start FlatState resharding + process MemTrie resharding: + freeze MemTrie, create HybridMemTries + for each child shard: + mem_tries[parent_shard].retain_split_shard(boundary_account) ``` +4. `retain_split_shard` leaves only keys in trie that belong to the left or right child shard. +It retains trie key intervals for left or right child as described above. Simultaneously the proof is generated. +In the end, we get new state root, hybrid memtrie corresponding to child shard, and the proof. +Proof is saved as state transition for pair `(block, new_shard_uid)`. + +5. The proof is sent as one of implicit transitions in ChunkStateWitness. + +6. On chunk validation path, chunk validator understands if resharding is +a part of state transition, using the same `should_split_shard` condition. + +7. It calls `Trie(state_transition_proof).retain_split_shard(boundary_account)` +which should succeed if proof is sufficient and generates new state root. + +8. Finally, it checks that the new state root matches the state root proposed in `ChunkStateWitness`. +If the whole `ChunkStateWitness` is valid, then chunk validator sends endorsement which also endorses the resharding. + ### State Storage - MemTrie The current implementation of MemTrie uses a pool of memory (`STArena`) to allocate and deallocate nodes and internal pointers in this pool to reference child nodes. MemTries, unlike the State representation of Trie, do not work with the hash of the nodes but internal memory pointers directly. Additionally, MemTries are not thread safe and one MemTrie exists per shard. @@ -296,7 +369,7 @@ Elements inherited by both children: Elements inherited only be the lowest index child: -* `BUFFERED_RECEIPT_INDICES ` +* `BUFFERED_RECEIPT_INDICES` * `BUFFERED_RECEIPT` #### Bring children shards up to date with the chain's head @@ -410,15 +483,32 @@ The state sync algorithm defines a `sync_hash` that is used in many parts of the ## Security Implications -```text -[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] -``` +### Fork Handling + +In theory, it can happen that there will be more than one candidate block which finishes the last epoch with old shard layout. For previous implementations it didn't matter because resharding decision was made in the beginning previous epoch. Now, the decision is made on the epoch boundary, so the new implementation handles this case as well. + +### Proof Validation + +With single shard tracking, nodes can't independently validate new state roots after resharding, because they don't have state of shard being split. That's why we generate resharding proofs, whose generation and validation may be a new weak point. However, `retain_split_shard` is equivalent to constant number of lookups in the trie, so its overhead its negligible. Even if proof is invalid, it will only imply that `retain_split_shard` fails early, similarly to other state transitions. ## Alternatives -```text -[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] -``` +In the solution space which would keep blockchain stateful, we also considered an alternative to handle resharding through mechanism of `Receipts`. The workflow would be to: + +* create empty `target_shard`, +* require `source_shard` chunk producers to create special `ReshardingReceipt(source_shard, target_shard, data)` where `data` would be an interval of key-value pairs in `source_shard` alongside with the proof, +* then, `target_shard` trackers and validators would process that receipt, validate the proof and insert the key-value pairs into the new shard. + +However, `data` would occupy most of the whole state witness capacity and introduce overhead of proving every single interval in `source_shard`. Moreover, approach to sync target shard "dynamically" also requires some form of catchup, which makes it much less feasible than chosen approach. + +Another question is whether we should tie resharding to epoch boundaries. This would allow to come from resharding decision to completion much faster. But for that, we would need to: + +* agree if we should reshard in the middle of the epoch or allow "fast epoch completion" which has to be implemented, +* keep chunk producers tracking "spare shards" ready to receive items from split shards, +* on resharding event, implement specific form of state sync, on which source and target chunk producers would agree on new state roots offline, +* then, new state roots would be validated by chunk validators in the same fashion. + +While it is much closer to Dynamic Resharding (below), it requires much more changes to the protocol. And the considered idea works very well as intermediate step to that, if needed. ## Future possibilities @@ -428,27 +518,22 @@ The state sync algorithm defines a `sync_hash` that is used in many parts of the ## Consequences -```text -[This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] -``` - ### Positive -* p1 +* The protocol is able to execute resharding even while only a fraction of nodes track the split shard. +* State for new shard layouts is computed in the matter of minutes instead of hours, thus ecosystem stability during resharding is greatly increased. As before, from the point of view of NEAR users it is instantaneous. ### Neutral -* n1 +N/A ### Negative -* n1 +* The storage components need to handle additional complexity of controlling the shard layout change. ### Backwards Compatibility -```text -[All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] -``` +Resharding is backwards compatible with existing protocol logic. ## Unresolved Issues (Optional) From 8ab6927b9ab387131923f5bad74167832ecca1fc Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Mon, 25 Nov 2024 22:44:35 +0530 Subject: [PATCH 23/30] [resharding] Add sections for receipt handling (#578) This is complete except for section `Handling buffered receipts that target parent shard` which is still being discussed. --- neps/nep-0568.md | 91 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 24a4ec221..5ddb2acbb 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -207,10 +207,101 @@ The change we propose is to move the initial state download point to one in the ### Delayed Receipt Handling +The delayed receipts queue contains all incoming receipts that could not be executed as part of a block due to resource constraints like compute cost, gas limits etc. The entries in the delayed receipt queue can belong to any of the accounts as part of the shard. During a resharding event, we ideally need to split the delayed receipts across both the child shards according to the associated account_id with the receipt. + +The singleton trie key `DelayedReceiptIndices` holds the start_index and end_index associated with the delayed receipt entries for the shard. The trie key `DelayedReceipt { index }` contains the actual delayed receipt associated with some account_id. These are processed in a fifo queue order during chunk execution. + +Note that the delayed receipt trie keys do not have the `account_id` prefix. In ReshardingV2, we followed the trivial solution of iterating through all the delayed receipt queue entries and assigning them to the appropriate child shard, however due to constraints on the state witness size limits and instant resharding, this approach is no longer feasible for ReshardingV3. + +For ReshardingV3, we decided to handle the resharding by duplicating the entries of the delayed receipt queue across both the child shards. This is great from the perspective of state witness size and instant resharding as we only need to access the delayed receipt queue root entry in the trie, however it breaks the assumption that all delayed receipts in a shard belong to the accounts within that shard. + +To resolve this, with the new protocol version, we changed the implementation of runtime to discard executing delayed receipts that don't belong to the account_id on that shard. + +Note that no delayed receipts are lost during resharding as all receipts get executed exactly once based on which of the child shards does the associated account_id belong to. + ### PromiseYield Receipt Handling +Promise Yield were introduced as part of NEP-519 to enable defer replying to caller function while response is being prepared. As part of Promise Yield implementation, it introduced three new trie keys, `PromiseYieldIndices`, `PromiseYieldTimeout` and `PromiseYieldReceipt`. + +* `PromiseYieldIndices`: This is a singleton key that holds the start_index and end_index of the keys in `PromiseYieldTimeout` +* `PromiseYieldTimeout { index }`: Along with the receiver_id and data_id, this stores the expires_at block height till which we need to wait to receive a response. +* `PromiseYieldReceipt { receiver_id, data_id }`: This is the receipt created by the account. + +An account can call the `promise_yield_create` host function that increments the `PromiseYieldIndices` along with adding a new entry into the `PromiseYieldTimeout` and `PromiseYieldReceipt`. + +The `PromiseYieldTimeout` is sorted as per the time of creation and has an increasing value of expires_at block height. In the runtime, we iterate over all the expired receipts and create a blank receipt to resolve the entry in `PromiseYieldReceipt`. + +The account can call the `promise_yield_resume` host function multiple times and if this is called before the expiry period, we use this to resolve the promise yield receipt. Note that the implementation allows for multiple resolution receipts to be created, including the expiry receipt, but only the first one is used for the actual resolution of the promise yield receipt. + +We use this implementation quirk to facilitate resharding implementation. The resharding strategy for the three trie keys are: + +* `PromiseYieldIndices`: Duplicate across both child shards. +* `PromiseYieldTimeout { index }`: Duplicate across both child shards. +* `PromiseYieldReceipt { receiver_id, data_id }`: Since this key has the account_id prefix, we can split the entries across both child shards based on the prefix. + +After duplication of the `PromiseYieldIndices` and `PromiseYieldTimeout`, when the entries of `PromiseYieldTimeout` eventually get dequeued at the expiry height of the following happens: + +* If the promise yield receipt associated with the dequeued entry IS NOT a part of the child trie, we create a timeout resolution receipt and it gets ignored. +* If the promise yield receipt associated with the dequeued entry IS part of the child trie, the promise yield implementation continues to work as expected. + +This means we don't have to make any special changes in the runtime for handling resharding of promise yield receipts. + ### Buffered Receipt Handling +Buffered Receipts were introduced as part of NEP-539, cross-shard congestion control. As part of the implementation, it introduced two new trie keys, `BufferedReceiptIndices` and `BufferedReceipt`. + +* `BufferedReceiptIndices`: This is a singleton key that holds the start_index and end_index of the keys in `BufferedReceipt` for each shard_id. +* `BufferedReceipt { receiving_shard, index }`: This holds the actual buffered receipt that needs to be sent to the receiving_shard. + +Note that the targets of the buffered receipts belong to external shards and during a resharding event, we would need to handle both, the set of buffered receipts in the parent shard, as well as the set of buffered receipts in other shards that target the parent shard. + +#### Handling buffered receipts in parent shard + +Since buffered receipts target external shards, it is fine to assign buffered receipts to either of the child shards. For simplicity, we assign all the buffered receipts to the child shard with the lower index, i.e. copy `BufferedReceiptIndices` and `BufferedReceipt` to the child shard with lower index while keeping `BufferedReceiptIndices` as empty for child shard with higher index. + +#### Handling buffered receipts that target parent shard + +TODO(shreyan) + +### Congestion Control + +Along with having buffered receipts, each chunk also publishes a CongestionInfo to the chunk header that has information about the congestion of the shard as of processing block. + +```rust +pub struct CongestionInfoV1 { + /// Sum of gas in currently delayed receipts. + pub delayed_receipts_gas: u128, + /// Sum of gas in currently buffered receipts. + pub buffered_receipts_gas: u128, + /// Size of borsh serialized receipts stored in state because they + /// were delayed, buffered, postponed, or yielded. + pub receipt_bytes: u64, + /// If fully congested, only this shard can forward receipts. + pub allowed_shard: u16, +} +``` + +After a resharding event, we need to properly initialize the congestion info for the child shards. Here's how we handle each of the fields + +#### delayed_receipts_gas + +Since the resharding strategy for delayed receipts is to duplicate them across both the child shards, we simply copy the value of `delayed_receipts_gas` across both shards. + +#### buffered_receipts_gas + +Since the resharding strategy for buffered receipts is to assign all the buffered receipts to the lower index child, we copy the `buffered_receipts_gas` from parent to lower index child and set `buffered_receipts_gas` to zero for upper index child. + +#### receipt_bytes + +This field is harder to deal with as it contains the information from both delayed receipts and buffered receipts. To calculate this field properly, we would need the distribution of the receipt_bytes across both delayed receipts and buffered receipts. The current solution is to start storing metadata about the total `receipt_bytes` for buffered receipts in the trie. This way we have the following: + +* For child with lower index, receipt_bytes is the sum of both delayed receipts bytes and congestion control bytes, hence `receipt_bytes = parent.receipt_bytes` +* For child with upper index, receipt_bytes is just the bytes from delayed receipts, hence `receipt_bytes = parent.receipt_bytes - buffered_receipt_bytes` + +#### allowed_shard + +This field is calculated by a round-robin mechanism which can be independently calculated for both the child shards. Since we are changing the [ShardId semantics](#shardid-semantics), we need to change implementation to use `ShardIndex` instead of `ShardID` which is just an assignment for each shard_id to the contiguous index `[0, num_shards)`. + ### ShardId Semantics Currently, shard IDs are represented as numbers within the range `[0,n)`, where n is the total number of shards. These shard IDs are sorted in the same order as the account ID ranges assigned to them. While this approach is straightforward, it complicates resharding operations, particularly when splitting a shard in the middle of the range. Such a split requires reindexing all subsequent shards with higher IDs, adding complexity to the process. From 09dc287fc2d5c75727f6d876ff4077627e79a1f7 Mon Sep 17 00:00:00 2001 From: wacban Date: Wed, 4 Dec 2024 17:55:10 +0000 Subject: [PATCH 24/30] cross shard traffic --- neps/nep-0568.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 5ddb2acbb..1cb7fc000 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -205,6 +205,46 @@ The change we propose is to move the initial state download point to one in the ### Cross Shard Traffic +When the shard layout changes, it is crucial to handle cross-shard traffic +correctly, especially in the presence of missing chunks. Care must be taken to +ensure that no receipt is lost or duplicated. There are two important receipt +types that need to be considered - the outgoing receipts and the incoming +receipts. + +Note - this proposal reuses the approach taken by ReshardingV2. + +#### Outgoing Receipts + +Each new chunk in a shard contains a list of outgoing receipts generated during +the processing of the previous chunk in that shard. + +In cases where chunks are missing at the resharding boundary, both child shards +could theoretically include the outgoing receipts from their shared ancestor +chunk. However, this naive approach would lead to the duplication of receipts, +which must be avoided. + +The proposed solution is to reassign the outgoing receipts from the parent chunk +to only one of the child shards. Specifically, the child shard with the lower +shard ID will claim all outgoing receipts from the parent, while the other child +will receive none. This ensures that all receipts are processed exactly once. + +#### Incoming Receipts + +To process a chunk in a shard, it is necessary to gather all outgoing receipts +from other shards that are targeted at this shard. These receipts must then be +included as incoming receipts. + +In the presence of missing chunks, the new chunk must collect receipts from all +previous blocks, spanning the period since the last new chunk in this shard. +This range may cross the resharding boundary. + +When this occurs, the chunk must also consider receipts that were previously +targeted at its parent shard. However, it must filter these receipts to include +only those where the recipient lies within the current shard, discarding those +where the recipient belongs to the sibling shard in the new shard layout. This +filtering process ensures that every receipt is processed exactly once and in +the correct shard. + ### Delayed Receipt Handling The delayed receipts queue contains all incoming receipts that could not be executed as part of a block due to resource constraints like compute cost, gas limits etc. The entries in the delayed receipt queue can belong to any of the accounts as part of the shard. During a resharding event, we ideally need to split the delayed receipts across both the child shards according to the associated account_id with the receipt. From d889993958a8413ec2510fd80341a35e15611be9 Mon Sep 17 00:00:00 2001 From: Marcelo Diop-Gonzalez Date: Fri, 13 Dec 2024 05:48:14 -0500 Subject: [PATCH 25/30] state sync: reword to make clear it's not just for convenience (#579) --- neps/nep-0568.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 1cb7fc000..e127861ad 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -197,11 +197,11 @@ Then, on `validate_chunk_state_witness`, if implicit transition corresponds to r ### State Sync -Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync intended to make the resharding implementation easier to work with. +Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync related to resharding. When nodes sync state (either because they've fallen far behind the chain, or because they're going to become a chunk producer for a new shard in a future epoch), they first identify a point in the chain they'd like to sync to. Then they download the tries corresponding to that point in the chain and apply all chunks from that point until they're caught up. Currently, the tries downloaded initially are those corresponding to the `prev_state_root` field of the last new chunk before the current epoch's first block. This means the state downloaded is the state at some point in the previous epoch. -The change we propose is to move the initial state download point to one in the current epoch rather than the previous. This allows some simplification in the resharding implementation, and reduces the size of the state we need to download. Suppose that the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and that a chunk producer that wasn't tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. Then with the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then when it's done, it would need to perform the resharding that all the other nodes had already done. This isn't a correctness issue, but it simplifies the implementation somewhat if we instead download only the state for shard `S'`, and it allows the node to download only the state belonging to `S'`, which is much smaller. +The change we propose is to move the initial state download point to one in the current epoch rather than the previous. This keeps shard IDs consistent throughout the state sync logic, allows some simplification in the resharding implementation, and reduces the size of the state we need to download. Suppose that the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and that a chunk producer that wasn't tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. Then with the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then when it's done, it would need to perform the resharding that all the other nodes had already done. This isn't a correctness issue, but it simplifies the implementation somewhat if we instead download only the state for shard `S'`, and it allows the node to download only the state belonging to `S'`, which is much smaller. ### Cross Shard Traffic From 0ae4721769a2fcecb2b41024dd64032485073c6e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Chuda=C5=9B?= <18039094+staffik@users.noreply.github.com> Date: Fri, 13 Dec 2024 12:43:31 +0100 Subject: [PATCH 26/30] resharding: cold storage (#580) --- neps/nep-0568.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index e127861ad..bec8dd340 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -138,8 +138,6 @@ Splitting a shard's Flat State is performed in multiple steps: ### State Storage - State -// TODO Describe integration with cold storage once design is ready - Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding, a mapping strategy allows child shards to access ancestor shard data without directly creating new entries. @@ -155,11 +153,18 @@ This allows child shards to access and update state data under the ancestor shar Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`, pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data. Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected. -For archival nodes, mappings are retained indefinitely to maintain access to the full historical state. This mapping strategy enables efficient shard management during resharding events, supporting smooth transitions without altering storage structures directly. +#### Integration with cold storage (archival nodes) + +Cold storage uses the same mapping strategy to manage shard state during resharding: +* When state data is migrated from hot to cold storage, it retains the parent shard’s `ShardUId` prefix, ensuring consistency with the mapping strategy. +* While copying data for the last block of the epoch where resharding occured, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data. +* These mappings are permanent in cold storage, aligning with its role in preserving historical state. + +This approach minimizes complexity while maintaining consistency across hot and cold storage. ### Stateless Validation From cf3174f38c832e126b6a7878d71f8778f240859c Mon Sep 17 00:00:00 2001 From: Waclaw Banasik Date: Fri, 13 Dec 2024 13:11:19 +0100 Subject: [PATCH 27/30] lint --- neps/nep-0568.md | 1 + 1 file changed, 1 insertion(+) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index bec8dd340..6013bdd39 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -160,6 +160,7 @@ supporting smooth transitions without altering storage structures directly. #### Integration with cold storage (archival nodes) Cold storage uses the same mapping strategy to manage shard state during resharding: + * When state data is migrated from hot to cold storage, it retains the parent shard’s `ShardUId` prefix, ensuring consistency with the mapping strategy. * While copying data for the last block of the epoch where resharding occured, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data. * These mappings are permanent in cold storage, aligning with its role in preserving historical state. From c4f5e3051d27eee1073526d20d0ec638300446a7 Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Mon, 16 Dec 2024 16:06:38 +0530 Subject: [PATCH 28/30] Add section on buffered receipt handling --- neps/nep-0568.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 6013bdd39..5491ba349 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -307,7 +307,23 @@ Since buffered receipts target external shards, it is fine to assign buffered re #### Handling buffered receipts that target parent shard -TODO(shreyan) +This scenario is slightly more complex to deal with. At the boundary of resharding, we may have buffered receipts created before the resharding event targeting the parent shard. At the same time, we may also have buffered receipts that are generated after the resharding event that directly target the child shard. The receipts from both parent and child buffered receipts queue need to appropriately sent to the child shard depending on the account_id while respecting the outgoing limits calculated by bandwidth scheduler and congestion control. + +The flow of handling buffered receipts before ReshardingV3 is as follows: + +1. Calculate `outgoing_limit` for each shard. +2. For each shard, try and forward as many in-order receipts as possible from the buffer while respecting `outgoing_limit`. +3. Apply chunk and `try_forward` newly generated receipts. The newly generated receipts are forwarded if we have enough limit else they are put in the buffered queue. + +The solution for ReshardingV3 is to first try draining the parent queue before moving onto draining the child queue. The modified flow would look something like: + +1. Calculate `outgoing_limit` for both the child shards using congestion info from parent. +2. Forwarding receipts + * First try forward as many in-order receipts as possible from parent shard buffer. Stop either when we drain the parent buffer or as soon as we exhaust the `outgoing_limit` of either of the children shards. + * Next try forward as many in-order receipts as possible from child shard buffer. +3. Apply chunk and `try_forward` newly generated receipts remains the same. + +The minor downside to this approach is that we don't have guarantees between order of receipt generation and order of receipt forwarding, but that's anyway the case today with buffered receipts. ### Congestion Control From 1efd6cf18b05973c1e7ff1e67893804f4eec95ac Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Mon, 16 Dec 2024 20:39:07 +0530 Subject: [PATCH 29/30] [resharding] ChatGPT modulation (#581) Some beautification courtesy ChatGPT. I double checked everything, we aren't changing any meaning. --- neps/nep-0568.md | 593 +++++++++++++++++++---------------------------- 1 file changed, 243 insertions(+), 350 deletions(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 5491ba349..6e4d9dd93 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -10,324 +10,244 @@ Created: 2024-10-24 LastUpdated: 2024-10-24 --- -## Summary +## Summary -This proposal introduces a new resharding implementation and shard layout for -production networks. +This proposal introduces a new resharding implementation and shard layout for production networks. -The primary objective of Resharding V3 is to increase chain capacity by -splitting overutilized shards. A secondary aim is to lay the groundwork for -supporting Dynamic Resharding, Instant Resharding and Shard Merging in future -updates. +The primary objective of Resharding V3 is to increase chain capacity by splitting overutilized shards. A secondary aim is to lay the groundwork for supporting Dynamic Resharding, Instant Resharding, and Shard Merging in future updates. ## Motivation -The sharded architecture of the NEAR Protocol is a cornerstone of its design, enabling parallel and distributed execution that significantly boosts overall throughput. Resharding plays a pivotal role in this system, allowing the network to adjust the number of shards to accommodate growth. By increasing the number of shards, resharding ensures the network can scale seamlessly, alleviating existing congestion, managing the rising traffic demands, and welcoming new customers. This adaptability is essential for maintaining the protocol's performance, reliability, and capacity to support a thriving, ever-expanding ecosystem. +The sharded architecture of the NEAR Protocol is a cornerstone of its design, enabling parallel and distributed execution that significantly boosts overall throughput. Resharding plays a pivotal role in this system, allowing the network to adjust the number of shards to accommodate growth. By increasing the number of shards, resharding ensures the network can scale seamlessly, alleviating existing congestion, managing rising traffic demands, and welcoming new participants. This adaptability is essential for maintaining the protocol's performance, reliability, and capacity to support a thriving, ever-expanding ecosystem. -Resharding V3 is a significantly redesigned approach, addressing limitations of -the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508]. -The earlier solutions became obsolete due to major protocol changes since -Resharding V2, including the introduction of Stateless Validation, Single Shard -Tracking, and Mem-Trie. +Resharding V3 is a significantly redesigned approach, addressing limitations of the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508]. The earlier solutions became obsolete due to major protocol changes since Resharding V2, including the introduction of Stateless Validation, Single Shard Tracking, and Mem-Trie. ## Specification -Resharding will be scheduled in advance by the NEAR developer team. The new -shard layout will be hardcoded into the neard binary and linked to the protocol -version. As the protocol upgrade progresses, resharding will be triggered during -the post-processing phase of the last block of the epoch. At this point, the -state of the parent shard will be split between two child shards. From the first -block of the new protocol version onward, the chain will operate with the new -shard layout. - -There are two key dimensions to consider: state storage and protocol features, -along with a few additional details. - -1) State Storage: Currently, the state of a shard is stored in three distinct -formats: the state, the flat state, and the mem-trie. Each of these -representations must be resharded. Logically, resharding is an almost -instantaneous event that occurs before the first block under the new shard -layout. However, in practice, some of this work may be deferred to -post-processing, as long as the chain's view reflects a fully resharded state. - -1) Protocol Features: Several protocol features must integrate smoothly with the - resharding process, including: - -* Stateless Validation: Resharding must be validated and proven through - stateless validation mechanisms. -* State Sync: Nodes must be able to sync the states of the child - shards post-resharding. -* Cross-Shard Traffic: Receipts sent to the parent shard may need to be - reassigned to one of the child shards. -* Receipt Handling: Delayed, postponed, buffered, and promise-yield receipts - must be correctly distributed between the child shards. -* ShardId Semantics: The shard identifiers will become abstract identifiers - where today they are number in the 0..num_shards range. -* Congestion Info: CongestionInfo in the chunk header would be recalculated for the child - shards at the resharding boundary. Proof must be compatible with Stateless Validation. +Resharding will be scheduled in advance by the NEAR developer team. The new shard layout will be hardcoded into the `neard` binary and linked to the protocol version. As the protocol upgrade progresses, resharding will be triggered during the post-processing phase of the last block of the epoch. At this point, the state of the parent shard will be split between two child shards. From the first block of the new protocol version onward, the chain will operate with the new shard layout. + +There are two key dimensions to consider: state storage and protocol features, along with additional details. + +1. **State Storage**: Currently, the state of a shard is stored in three distinct formats: the state, the flat state, and the mem-trie. Each of these representations must be resharded. Logically, resharding is an almost instantaneous event that occurs before the first block under the new shard layout. However, in practice, some of this work may be deferred to post-processing, as long as the chain's view reflects a fully resharded state. + +2. **Protocol Features**: Several protocol features must integrate smoothly with the resharding process, including: + + * **Stateless Validation**: Resharding must be validated and proven through stateless validation mechanisms. + * **State Sync**: Nodes must be able to synchronize the states of the child shards post-resharding. + * **Cross-Shard Traffic**: Receipts sent to the parent shard may need to be reassigned to one of the child shards. + * **Receipt Handling**: Delayed, postponed, buffered, and promise-yield receipts must be correctly distributed between the child shards. + * **ShardId Semantics**: The shard identifiers will become abstract identifiers where today they are numbers in the `0..num_shards` range. + * **Congestion Info**: `CongestionInfo` in the chunk header will be recalculated for the child shards at the resharding boundary. Proof must be compatible with Stateless Validation. ### State Storage - MemTrie -MemTrie is the in-memory representation of the trie that the runtime uses for all trie accesses. This is kept in sync with the Trie representation in state. +MemTrie is the in-memory representation of the trie that the runtime uses for all trie accesses. It is kept in sync with the Trie representation in the state. -As of today it isn't mandatory for nodes to have MemTrie feature enabled but going forward, with ReshardingV3, all nodes would require to have MemTrie enabled for resharding to happen successfully. +Currently, it isn't mandatory for nodes to have the MemTrie feature enabled, but going forward with Resharding V3, all nodes will be required to have MemTrie enabled for resharding to happen successfully. -For the purposes of resharding, we need an efficient way to split the MemTrie into two child tries based on the boundary account. This splitting happens at the epoch boundary when the new epoch is expected to have the two child shards. The set of requirements around MemTrie splitting are: +For resharding, we need an efficient way to split the MemTrie into two child tries based on the boundary account. This splitting happens at the epoch boundary when the new epoch is expected to have the two child shards. The requirements for MemTrie splitting are: -* MemTrie splitting needs to be "instant", i.e. happen efficiently within the span of one block. The child tries need to be available for the processing of the next block in the new epoch. -* MemTrie splitting needs to be compatible with stateless validation, i.e. we need to generate a proof that the memtrie split proposed by the chunk producer is correct. -* The proof generated for splitting the MemTrie needs to be compatible with the limits of the size of state witness that we send to all chunk validators. This prevents us from doing things like iterating through all trie keys for delayed receipts etc. +* **Instantaneous Splitting**: MemTrie splitting needs to happen efficiently within the span of one block. The child tries need to be available for processing the next block in the new epoch. +* **Compatibility with Stateless Validation**: We need to generate a proof that the MemTrie split proposed by the chunk producer is correct. +* **State Witness Size Limits**: The proof generated for splitting the MemTrie needs to comply with the size limits of the state witness sent to all chunk validators. This prevents us from iterating through all trie keys for delayed receipts, etc. -With ReshardingV3 design, there's no protocol change to the structure of MemTries, however the implementation constraints required us to introduce the concept of a Frozen MemTrie. More details are in the [implementation](#state-storage---memtrie-1) section below. +With the Resharding V3 design, there's no protocol change to the structure of MemTries; however, implementation constraints required us to introduce the concept of a Frozen MemTrie. More details are in the [implementation](#state-storage---memtrie-1) section below. -Based on the requirements above, we came up with an algorithm to efficiently split the parent trie into two child tries. Trie entries can be divided into three categories based on whether the trie keys have an `account_id` prefix and based on the total number of such trie keys. Splitting of these keys is handled in different ways. +Based on these requirements, we developed an algorithm to efficiently split the parent trie into two child tries. Trie entries can be divided into three categories based on whether the trie keys have an `account_id` prefix and the total number of such trie keys. Splitting of these keys is handled differently. -#### TrieKey with AccountID prefix +#### TrieKey with AccountID Prefix -This category includes most of the trie keys like `TrieKey::Account`, `TrieKey::ContractCode`, `TrieKey::PostponedReceipt`, etc. For these keys, we can efficiently split the trie based on the boundary account trie key. Note that we only need to read all the intermediate nodes that form a part of the split key. In the example below, if "pass" is the split key, we access all the nodes along the path of `root` -> `p` -> `a` -> `s` -> `s`, while not needing to touch any of the other intermediate nodes like `o` -> `s` -> `t` in key "post". The accessed nodes form a part of the state witness as those are the only nodes that the validators would need to verify that the resharding split is correct. This limits the size of the witness to effectively O(depth) of trie for each trie key in this category. +This category includes most trie keys like `TrieKey::Account`, `TrieKey::ContractCode`, `TrieKey::PostponedReceipt`, etc. For these keys, we can efficiently split the trie based on the boundary account trie key. We only need to read all the intermediate nodes that form part of the split key. In the example below, if "pass" is the split key, we access all the nodes along the path of `root` ➔ `p` ➔ `a` ➔ `s` ➔ `s`, while not needing to touch other intermediate nodes like `o` ➔ `s` ➔ `t` in key "post". The accessed nodes form part of the state witness, as those are the only nodes needed by validators to verify that the resharding split is correct. This limits the size of the witness to effectively O(depth) of the trie for each trie key in this category. ![Splitting Trie diagram](assets/nep-0568/NEP-SplitState.png) #### Singleton TrieKey -This category includes the trie keys `TrieKey::DelayedReceiptIndices`, `TrieKey::PromiseYieldIndices`, `TrieKey::BufferedReceiptIndices`. Notably, these are just a single entry (or O(num_shard) entries) in the trie and hence are small enough to read and modify for the children tries efficiently. +This category includes the trie keys `TrieKey::DelayedReceiptIndices`, `TrieKey::PromiseYieldIndices`, and `TrieKey::BufferedReceiptIndices`. These are just a single entry (or O(num_shard) entries) in the trie and are small enough to read and modify efficiently for the child tries. #### Indexed TrieKey -This category includes the trie keys `TrieKey::DelayedReceipt`, `TrieKey::PromiseYieldTimeout` and `TrieKey::BufferedReceipt`. The number of entries for these keys can potentially be arbitrarily large and it's not feasible to iterate through all the entries. In pre-stateless validation world, where we didn't care about state witness size limits, for ReshardingV2 we could just iterate over all delayed receipts and split them into the respective child shards. - -For ReshardingV3, these are handled by either of the two strategies +This category includes the trie keys `TrieKey::DelayedReceipt`, `TrieKey::PromiseYieldTimeout`, and `TrieKey::BufferedReceipt`. The number of entries for these keys can potentially be arbitrarily large, making it infeasible to iterate through all entries. In the pre-stateless validation world, where we didn't care about state witness size limits, for Resharding V2 we could iterate over all delayed receipts and split them into the respective child shards. -* `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards as each entry could belong to either of the child shards. More details in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. +For Resharding V3, these are handled by one of two strategies: -* `TrieKey::BufferedReceipt` are independent of the account_id and therefore can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details in the [Buffered Receipts](#buffered-receipt-handling) section below. +* **Duplication Across Child Shards**: `TrieKey::DelayedReceipt` and `TrieKey::PromiseYieldTimeout` are handled by duplicating entries across both child shards, as each entry could belong to either child shard. More details are in the [Delayed Receipts](#delayed-receipt-handling) and [Promise Yield](#promiseyield-receipt-handling) sections below. +* **Assignment to Lower Index Child**: `TrieKey::BufferedReceipt` is independent of the `account_id` and can be sent to either of the child shards, but not both. We copy the buffered receipts and the associated metadata to the child shard with the lower index. More details are in the [Buffered Receipts](#buffered-receipt-handling) section below. ### State Storage - Flat State -Flat State is a collection of key-value pairs stored on disk and each entry -contains a reference to its ShardId. When splitting a shard, every item inside -its Flat State must be correctly reassigned to either one of the new children; -due to technical limitations such operation can't be completed instantaneously. +Flat State is a collection of key-value pairs stored on disk, with each entry containing a reference to its `ShardId`. When splitting a shard, every item inside its Flat State must be correctly reassigned to one of the new child shards; however, due to technical limitations, such an operation cannot be completed instantaneously. -Flat State main purposes are allowing the creation of State Sync snapshots and -the construction of Mem Tries. Fortunately, these two operations can be delayed -until resharding is completed. Note also that with Mem Tries enabled the chain -can move forward even if the current status of Flat State is not in sync with -the latest block. +Flat State's main purposes are allowing the creation of State Sync snapshots and the construction of Mem Tries. Fortunately, these two operations can be delayed until resharding is completed. Note also that with Mem Tries enabled, the chain can move forward even if the current status of Flat State is not in sync with the latest block. -For the reason stated above, the chosen strategy is to reshard Flat State in a -long-running background task. The new shards' states must converge with their -Mem Tries representation in a reasonable amount of time. +For these reasons, the chosen strategy is to reshard Flat State in a long-running background task. The new shards' states must converge with their Mem Tries representation in a reasonable amount of time. Splitting a shard's Flat State is performed in multiple steps: -1) A post-processing 'split' task is created during the last block of the old - shard layout, instantaneously. -2) The 'split' task runs in parallel with the chain for a certain amount of - time. Inside this routine every key-value pair belonging to the shard being - split (also called parent shard) is copied into either the left or the right - child Flat State. Entries linked to receipts are handled in a special way. -3) Once the task is completed, the parent shard Flat State is cleaned up. The - children shards Flat States have their state in sync with last block of the - old shard layout. -4) Children shards must apply the delta changes from the first block of the new - shard layout until the final block of the canonical chain. This operation is - done in another background task to avoid slowdowns while processing blocks. -5) Children shards Flat States are now ready and can be used to take State Sync - snapshots and to reload Mem Tries. +1. A post-processing "split" task is created instantaneously during the last block of the old shard layout. +2. The "split" task runs in parallel with the chain for a certain amount of time. Inside this routine, every key-value pair belonging to the shard being split (also called the parent shard) is copied into either the left or the right child Flat State. Entries linked to receipts are handled in a special way. +3. Once the task is completed, the parent shard's Flat State is cleaned up. The child shards' Flat States have their state in sync with the last block of the old shard layout. +4. Child shards must apply the delta changes from the first block of the new shard layout until the final block of the canonical chain. This operation is done in another background task to avoid slowdowns while processing blocks. +5. Child shards' Flat States are now ready and can be used to take State Sync snapshots and to reload Mem Tries. ### State Storage - State -Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. -This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding, -a mapping strategy allows child shards to access ancestor shard data without directly creating new entries. +Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash. This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding, a mapping strategy allows child shards to access ancestor shard data without directly creating new entries. -A naive approach to resharding would involve copying all `State` entries with a new `ShardUId` for a child shard, effectively duplicating the state. -This method, while straightforward, is not feasible because copying a large state would take too much time. -Resharding needs to appear complete between two blocks, so a direct copy would not allow the process to occur quickly enough. +A naive approach to resharding would involve copying all `State` entries with a new `ShardUId` for a child shard, effectively duplicating the state. This method, while straightforward, is not feasible because copying a large state would take too much time. Resharding needs to appear complete between two blocks, so a direct copy would not allow the process to occur quickly enough. -To address this, Resharding V3 employs an efficient mapping strategy, using the `DBCol::ShardUIdMapping` column -to link each child shard’s `ShardUId` to the closest ancestor’s `ShardUId` holding the relevant data. -This allows child shards to access and update state data under the ancestor shard’s prefix without duplicating entries. +To address this, Resharding V3 employs an efficient mapping strategy, using the `DBCol::ShardUIdMapping` column to link each child shard’s `ShardUId` to the closest ancestor’s `ShardUId` holding the relevant data. This allows child shards to access and update state data under the ancestor shard’s prefix without duplicating entries. -Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`, -pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data. -Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected. +Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`, pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data. Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected. -This mapping strategy enables efficient shard management during resharding events, -supporting smooth transitions without altering storage structures directly. +This mapping strategy enables efficient shard management during resharding events, supporting smooth transitions without altering storage structures directly. -#### Integration with cold storage (archival nodes) +#### Integration with Cold Storage (Archival Nodes) Cold storage uses the same mapping strategy to manage shard state during resharding: * When state data is migrated from hot to cold storage, it retains the parent shard’s `ShardUId` prefix, ensuring consistency with the mapping strategy. -* While copying data for the last block of the epoch where resharding occured, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data. +* While copying data for the last block of the epoch where resharding occurred, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data. * These mappings are permanent in cold storage, aligning with its role in preserving historical state. This approach minimizes complexity while maintaining consistency across hot and cold storage. ### Stateless Validation -As only a fraction of nodes track the split shard, there is a need to prove the transition from state root of parent shard -to new state roots for children shards to other validators. -Otherwise the chunk producers for split shard may collude and provide invalid state roots, -which may compromise the protocol, for example, with minting tokens out of thin air. +Since only a fraction of nodes track the split shard, it is necessary to prove the transition from the state root of the parent shard to the new state roots for the child shards to other validators. Without this proof, chunk producers for the split shard could collude and provide invalid state roots, potentially compromising the protocol, such as by minting tokens out of thin air. + +The design ensures that generating and verifying this state transition is negligible in time compared to applying a chunk. As detailed in the [State Storage - MemTrie](#state-storage---memtrie) section, the generation and verification logic involves a constant number of trie lookups. Specifically, we implement the `retain_split_shard(boundary_account, RetainMode::{Left, Right})` method for the trie, which retains only the keys in the trie that belong to the left or right child shard. Internally, this method uses `retain_multi_range(intervals)`, where `intervals` is a vector of trie key intervals to retain. Each interval corresponds to a unique trie key type prefix byte (`Account`, `AccessKey`, etc.) and defines an interval from the empty key to the `boundary_account` key for the left shard, or from the `boundary_account` to infinity for the right shard. -The design allows to generate and check this state transition in the time, negligible compared to the time it takes to apply chunk. -As shown above in [State Storage - MemTrie](#state-storage---memtrie) section, generation and verification logic consists of constant number of trie lookups. -More specifically, we implement `retain_split_shard(boundary_account, RetainMode::{Left, Right})` method for trie, which leaves only keys in trie that -belong to the left or right child shard. -Inside, we implement `retain_multi_range(intervals)` method, where `intervals` is a vector of trie key intervals to retain. -Each interval corresponds to unique trie key type prefix byte (`Account`, `AccessKey`, etc.) and either defines an interval from empty key to `boundary_account` key for left shard, or from `boundary_account` to infinity for right shard. -`retain_multi_range` is recursive. Based on current trie key prefix covered by current node, it either: +The `retain_multi_range` method is recursive. Based on the current trie key prefix covered by the current node, it either: -* returns node back, if subtree is fully contained within some interval; -* returns "empty" node, if subtree is outside of all intervals; -* otherwise, descends into all children and constructs new node with children returned by recursive calls. +* Returns the node if the subtree is fully contained within an interval. +* Returns an "empty" node if the subtree is outside all intervals. +* Descends into all children and constructs a new node with children returned by recursive calls. -Implementation is agnostic to the trie storage used for retrieving nodes, it applies to both memtrie and partial storage (state proof). +This implementation is agnostic to the trie storage used for retrieving nodes and applies to both MemTries and partial storage (state proof). -* calling it for memtrie generates a proof and new state root; -* calling it for partial storage generates a new state root. If method doesn't fail with error that node wasn't found in the proof, it means that proof was sufficient, and it remains to compare generated state root with the one proposed by chunk producer. +* Calling it for MemTrie generates a proof and a new state root. +* Calling it for partial storage generates a new state root. If the method does not fail with an error indicating that a node was not found in the proof, it means the proof was sufficient, and it remains to compare the generated state root with the one proposed by the chunk producer. ### State Witness -Resharding state transition becomes one of `implicit_transitions` in `ChunkStateWitness`. It must be validated between processing last chunk (potentially missing) in the old epoch and the first chunk (potentially missing) in the new epoch. `ChunkStateTransition` fields also nicely correspond to the resharding state transition: in `block_hash` we store the hash of the last block of the parent shard, in `base_state` we store the resharding proof, and in `post_state_root` we store the proposed state root. +The resharding state transition becomes one of the `implicit_transitions` in `ChunkStateWitness`. It must be validated between processing the last chunk (potentially missing) in the old epoch and the first chunk (potentially missing) in the new epoch. The `ChunkStateTransition` fields correspond to the resharding state transition: the `block_hash` stores the hash of the last block of the parent shard, the `base_state` stores the resharding proof, and the `post_state_root` stores the proposed state root. -Note that it leads to **two** state transitions corresponding to the same block hash. On the chunk producer side, the first transition is stored for the `(block_hash, parent_shard_uid)` pair and the second one is stored for the `(block_hash, child_shard_uid)` pair. +This results in **two** state transitions corresponding to the same block hash. On the chunk producer side, the first transition is stored for the `(block_hash, parent_shard_uid)` pair, and the second one is stored for the `(block_hash, child_shard_uid)` pair. -The chunk validator has all the blocks, so it identifies whether implicit transition corresponds to applying missing chunk or resharding independently. This is implemented in `get_state_witness_block_range`, which iterates from `state_witness.chunk_header.prev_block_hash()` to the block with includes last last chunk for the (parent) shard, if it exists. +The chunk validator, having all the blocks, identifies whether the implicit transition corresponds to applying a missing chunk or resharding independently. This is implemented in `get_state_witness_block_range`, which iterates from `state_witness.chunk_header.prev_block_hash()` to the block that includes the last chunk for the (parent) shard, if it exists. -Then, on `validate_chunk_state_witness`, if implicit transition corresponds to resharding, chunk validator calls `retain_split_shard` and proves state transition from parent to child shard. +Then, in `validate_chunk_state_witness`, if the implicit transition corresponds to resharding, the chunk validator calls `retain_split_shard` and proves the state transition from the parent to the child shard. ### State Sync -Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync related to resharding. +Changes to the state sync protocol are not typically considered protocol changes requiring a version bump, as they concern downloading state that is not present locally rather than the rules for executing blocks and chunks. However, it is helpful to outline some planned changes to state sync related to resharding. -When nodes sync state (either because they've fallen far behind the chain, or because they're going to become a chunk producer for a new shard in a future epoch), they first identify a point in the chain they'd like to sync to. Then they download the tries corresponding to that point in the chain and apply all chunks from that point until they're caught up. Currently, the tries downloaded initially are those corresponding to the `prev_state_root` field of the last new chunk before the current epoch's first block. This means the state downloaded is the state at some point in the previous epoch. +When nodes sync state (either because they have fallen far behind the chain or because they will become a chunk producer for a new shard in a future epoch), they first identify a point in the chain to sync to. They then download the tries corresponding to that point in the chain and apply all chunks from that point until they are caught up. Currently, the tries downloaded initially correspond to the `prev_state_root` field of the last new chunk before the first block of the current epoch. This means the state downloaded is from some point in the previous epoch. -The change we propose is to move the initial state download point to one in the current epoch rather than the previous. This keeps shard IDs consistent throughout the state sync logic, allows some simplification in the resharding implementation, and reduces the size of the state we need to download. Suppose that the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and that a chunk producer that wasn't tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. Then with the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then when it's done, it would need to perform the resharding that all the other nodes had already done. This isn't a correctness issue, but it simplifies the implementation somewhat if we instead download only the state for shard `S'`, and it allows the node to download only the state belonging to `S'`, which is much smaller. +The proposed change is to move the initial state download point to one in the current epoch rather than the previous one. This keeps shard IDs consistent throughout the state sync logic, simplifies the resharding implementation, and reduces the size of the state to be downloaded. Suppose the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and a chunk producer that was not tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. With the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then, when it is done, it would need to perform the resharding that all other nodes had already done. While this is not a correctness issue, it simplifies the implementation if we instead download only the state for shard `S'`, allowing the node to download only the state belonging to `S'`, which is much smaller. -### Cross Shard Traffic +### Cross-Shard Traffic -When the shard layout changes, it is crucial to handle cross-shard traffic -correctly, especially in the presence of missing chunks. Care must be taken to -ensure that no receipt is lost or duplicated. There are two important receipt -types that need to be considered - the outgoing receipts and the incoming -receipts. +When the shard layout changes, it is crucial to handle cross-shard traffic correctly, especially in the presence of missing chunks. Care must be taken to ensure that no receipt is lost or duplicated. There are two important receipt types that need to be considered: outgoing receipts and incoming receipts. -Note - this proposal reuses the approach taken by ReshardingV2. +*Note: This proposal reuses the approach taken by Resharding V2.* #### Outgoing Receipts -Each new chunk in a shard contains a list of outgoing receipts generated during -the processing of the previous chunk in that shard. +Each new chunk in a shard contains a list of outgoing receipts generated during the processing of the previous chunk in that shard. -In cases where chunks are missing at the resharding boundary, both child shards -could theoretically include the outgoing receipts from their shared ancestor -chunk. However, this naive approach would lead to the duplication of receipts, -which must be avoided. +In cases where chunks are missing at the resharding boundary, both child shards could theoretically include the outgoing receipts from their shared ancestor chunk. However, this naive approach would lead to the duplication of receipts, which must be avoided. -The proposed solution is to reassign the outgoing receipts from the parent chunk -to only one of the child shards. Specifically, the child shard with the lower -shard ID will claim all outgoing receipts from the parent, while the other child -will receive none. This ensures that all receipts are processed exactly once. +The proposed solution is to reassign the outgoing receipts from the parent chunk to only one of the child shards. Specifically, the child shard with the lower shard ID will claim all outgoing receipts from the parent, while the other child will receive none. This ensures that all receipts are processed exactly once. #### Incoming Receipts -To process a chunk in a shard, it is necessary to gather all outgoing receipts -from other shards that are targeted at this shard. These receipts must then be -included as incoming receipts. +To process a chunk in a shard, it is necessary to gather all outgoing receipts from other shards that are targeted at this shard. These receipts must then be included as incoming receipts. -In the presence of missing chunks, the new chunk must collect receipts from all -previous blocks, spanning the period since the last new chunk in this shard. -This range may cross the resharding boundary. +In the presence of missing chunks, the new chunk must collect receipts from all previous blocks, spanning the period since the last new chunk in this shard. This range may cross the resharding boundary. -When this occurs, the chunk must also consider receipts that were previously -targeted at its parent shard. However, it must filter these receipts to include -only those where the recipient lies within the current shard, discarding those -where the recipient belongs to the sibling shard in the new shard layout. This -filtering process ensures that every receipt is processed exactly once and in -the correct shard. +When this occurs, the chunk must also consider receipts that were previously targeted at its parent shard. However, it must filter these receipts to include only those where the recipient lies within the current shard, discarding those where the recipient belongs to the sibling shard in the new shard layout. This filtering process ensures that every receipt is processed exactly once and in the correct shard. ### Delayed Receipt Handling -The delayed receipts queue contains all incoming receipts that could not be executed as part of a block due to resource constraints like compute cost, gas limits etc. The entries in the delayed receipt queue can belong to any of the accounts as part of the shard. During a resharding event, we ideally need to split the delayed receipts across both the child shards according to the associated account_id with the receipt. +The delayed receipts queue contains all incoming receipts that could not be executed as part of a block due to resource constraints like compute cost, gas limits, etc. The entries in the delayed receipt queue can belong to any of the accounts within the shard. During a resharding event, we ideally need to split the delayed receipts across both child shards according to the associated `account_id` with the receipt. -The singleton trie key `DelayedReceiptIndices` holds the start_index and end_index associated with the delayed receipt entries for the shard. The trie key `DelayedReceipt { index }` contains the actual delayed receipt associated with some account_id. These are processed in a fifo queue order during chunk execution. +The singleton trie key `DelayedReceiptIndices` holds the `start_index` and `end_index` associated with the delayed receipt entries for the shard. The trie key `DelayedReceipt { index }` contains the actual delayed receipt associated with some `account_id`. These are processed in a FIFO queue order during chunk execution. -Note that the delayed receipt trie keys do not have the `account_id` prefix. In ReshardingV2, we followed the trivial solution of iterating through all the delayed receipt queue entries and assigning them to the appropriate child shard, however due to constraints on the state witness size limits and instant resharding, this approach is no longer feasible for ReshardingV3. +Note that the delayed receipt trie keys do not have the `account_id` prefix. In Resharding V2, we followed the trivial solution of iterating through all the delayed receipt queue entries and assigning them to the appropriate child shard. However, due to constraints on the state witness size limits and instant resharding, this approach is no longer feasible for Resharding V3. -For ReshardingV3, we decided to handle the resharding by duplicating the entries of the delayed receipt queue across both the child shards. This is great from the perspective of state witness size and instant resharding as we only need to access the delayed receipt queue root entry in the trie, however it breaks the assumption that all delayed receipts in a shard belong to the accounts within that shard. +For Resharding V3, we decided to handle the resharding by duplicating the entries of the delayed receipt queue across both child shards. This is beneficial from the perspective of state witness size and instant resharding, as we only need to access the delayed receipt queue root entry in the trie. However, it breaks the assumption that all delayed receipts in a shard belong to the accounts within that shard. -To resolve this, with the new protocol version, we changed the implementation of runtime to discard executing delayed receipts that don't belong to the account_id on that shard. +To resolve this, with the new protocol version, we changed the implementation of the runtime to discard executing delayed receipts that don't belong to the `account_id` on that shard. -Note that no delayed receipts are lost during resharding as all receipts get executed exactly once based on which of the child shards does the associated account_id belong to. +Note that no delayed receipts are lost during resharding, as all receipts get executed exactly once based on which of the child shards the associated `account_id` belongs to. ### PromiseYield Receipt Handling -Promise Yield were introduced as part of NEP-519 to enable defer replying to caller function while response is being prepared. As part of Promise Yield implementation, it introduced three new trie keys, `PromiseYieldIndices`, `PromiseYieldTimeout` and `PromiseYieldReceipt`. +Promise Yield was introduced as part of NEP-519 to enable deferring replies to the caller function while the response is being prepared. As part of the Promise Yield implementation, it introduced three new trie keys: `PromiseYieldIndices`, `PromiseYieldTimeout`, and `PromiseYieldReceipt`. -* `PromiseYieldIndices`: This is a singleton key that holds the start_index and end_index of the keys in `PromiseYieldTimeout` -* `PromiseYieldTimeout { index }`: Along with the receiver_id and data_id, this stores the expires_at block height till which we need to wait to receive a response. +* `PromiseYieldIndices`: This is a singleton key that holds the `start_index` and `end_index` of the keys in `PromiseYieldTimeout`. +* `PromiseYieldTimeout { index }`: Along with the `receiver_id` and `data_id`, this stores the `expires_at` block height until which we need to wait to receive a response. * `PromiseYieldReceipt { receiver_id, data_id }`: This is the receipt created by the account. An account can call the `promise_yield_create` host function that increments the `PromiseYieldIndices` along with adding a new entry into the `PromiseYieldTimeout` and `PromiseYieldReceipt`. -The `PromiseYieldTimeout` is sorted as per the time of creation and has an increasing value of expires_at block height. In the runtime, we iterate over all the expired receipts and create a blank receipt to resolve the entry in `PromiseYieldReceipt`. +The `PromiseYieldTimeout` is sorted by time of creation and has an increasing value of `expires_at` block height. In the runtime, we iterate over all the expired receipts and create a blank receipt to resolve the entry in `PromiseYieldReceipt`. -The account can call the `promise_yield_resume` host function multiple times and if this is called before the expiry period, we use this to resolve the promise yield receipt. Note that the implementation allows for multiple resolution receipts to be created, including the expiry receipt, but only the first one is used for the actual resolution of the promise yield receipt. +The account can call the `promise_yield_resume` host function multiple times, and if this is called before the expiry period, we use this to resolve the promise yield receipt. Note that the implementation allows for multiple resolution receipts to be created, including the expiry receipt, but only the first one is used for the actual resolution of the promise yield receipt. -We use this implementation quirk to facilitate resharding implementation. The resharding strategy for the three trie keys are: +We use this implementation quirk to facilitate the resharding implementation. The resharding strategy for the three trie keys is: -* `PromiseYieldIndices`: Duplicate across both child shards. -* `PromiseYieldTimeout { index }`: Duplicate across both child shards. -* `PromiseYieldReceipt { receiver_id, data_id }`: Since this key has the account_id prefix, we can split the entries across both child shards based on the prefix. +* **Duplicate Across Both Child Shards**: + * `PromiseYieldIndices` + * `PromiseYieldTimeout { index }` +* **Split Based on Prefix**: + * `PromiseYieldReceipt { receiver_id, data_id }`: Since this key has the `account_id` prefix, we can split the entries across both child shards based on the prefix. -After duplication of the `PromiseYieldIndices` and `PromiseYieldTimeout`, when the entries of `PromiseYieldTimeout` eventually get dequeued at the expiry height of the following happens: +After duplication of the `PromiseYieldIndices` and `PromiseYieldTimeout`, when the entries of `PromiseYieldTimeout` eventually get dequeued at the expiry height, the following happens: -* If the promise yield receipt associated with the dequeued entry IS NOT a part of the child trie, we create a timeout resolution receipt and it gets ignored. -* If the promise yield receipt associated with the dequeued entry IS part of the child trie, the promise yield implementation continues to work as expected. +* If the promise yield receipt associated with the dequeued entry **is not** part of the child trie, we create a timeout resolution receipt, and it gets ignored. +* If the promise yield receipt associated with the dequeued entry **is** part of the child trie, the promise yield implementation continues to work as expected. -This means we don't have to make any special changes in the runtime for handling resharding of promise yield receipts. +This means we don't have to make any special changes in the runtime for handling the resharding of promise yield receipts. ### Buffered Receipt Handling -Buffered Receipts were introduced as part of NEP-539, cross-shard congestion control. As part of the implementation, it introduced two new trie keys, `BufferedReceiptIndices` and `BufferedReceipt`. +Buffered Receipts were introduced as part of NEP-539 for cross-shard congestion control. As part of the implementation, it introduced two new trie keys: `BufferedReceiptIndices` and `BufferedReceipt`. -* `BufferedReceiptIndices`: This is a singleton key that holds the start_index and end_index of the keys in `BufferedReceipt` for each shard_id. -* `BufferedReceipt { receiving_shard, index }`: This holds the actual buffered receipt that needs to be sent to the receiving_shard. +* `BufferedReceiptIndices`: This is a singleton key that holds the `start_index` and `end_index` of the keys in `BufferedReceipt` for each `shard_id`. +* `BufferedReceipt { receiving_shard, index }`: This holds the actual buffered receipt that needs to be sent to the `receiving_shard`. -Note that the targets of the buffered receipts belong to external shards and during a resharding event, we would need to handle both, the set of buffered receipts in the parent shard, as well as the set of buffered receipts in other shards that target the parent shard. +Note that the targets of the buffered receipts belong to external shards, and during a resharding event, we would need to handle both the set of buffered receipts in the parent shard and the set of buffered receipts in other shards that target the parent shard. -#### Handling buffered receipts in parent shard +#### Handling Buffered Receipts in Parent Shard -Since buffered receipts target external shards, it is fine to assign buffered receipts to either of the child shards. For simplicity, we assign all the buffered receipts to the child shard with the lower index, i.e. copy `BufferedReceiptIndices` and `BufferedReceipt` to the child shard with lower index while keeping `BufferedReceiptIndices` as empty for child shard with higher index. +Since buffered receipts target external shards, it is acceptable to assign buffered receipts to either of the child shards. For simplicity, we assign all the buffered receipts to the child shard with the lower index, i.e., copy `BufferedReceiptIndices` and `BufferedReceipt` to the child shard with the lower index while keeping `BufferedReceiptIndices` empty for the child shard with the higher index. -#### Handling buffered receipts that target parent shard +#### Handling Buffered Receipts that Target Parent Shard -This scenario is slightly more complex to deal with. At the boundary of resharding, we may have buffered receipts created before the resharding event targeting the parent shard. At the same time, we may also have buffered receipts that are generated after the resharding event that directly target the child shard. The receipts from both parent and child buffered receipts queue need to appropriately sent to the child shard depending on the account_id while respecting the outgoing limits calculated by bandwidth scheduler and congestion control. +This scenario is slightly more complex. At the boundary of resharding, we may have buffered receipts created before the resharding event targeting the parent shard. At the same time, we may also have buffered receipts generated after the resharding event that directly target the child shard. The receipts from both the parent and child buffered receipts queue need to be appropriately sent to the child shard depending on the `account_id`, while respecting the outgoing limits calculated by the bandwidth scheduler and congestion control. -The flow of handling buffered receipts before ReshardingV3 is as follows: +The flow of handling buffered receipts before Resharding V3 is as follows: 1. Calculate `outgoing_limit` for each shard. -2. For each shard, try and forward as many in-order receipts as possible from the buffer while respecting `outgoing_limit`. -3. Apply chunk and `try_forward` newly generated receipts. The newly generated receipts are forwarded if we have enough limit else they are put in the buffered queue. +2. For each shard, try to forward as many in-order receipts as possible from the buffer while respecting `outgoing_limit`. +3. Apply chunk and `try_forward` newly generated receipts. The newly generated receipts are forwarded if we have enough limit; otherwise, they are put in the buffered queue. -The solution for ReshardingV3 is to first try draining the parent queue before moving onto draining the child queue. The modified flow would look something like: +The solution for Resharding V3 is to first try draining the parent queue before moving on to draining the child queue. The modified flow would look like this: -1. Calculate `outgoing_limit` for both the child shards using congestion info from parent. -2. Forwarding receipts - * First try forward as many in-order receipts as possible from parent shard buffer. Stop either when we drain the parent buffer or as soon as we exhaust the `outgoing_limit` of either of the children shards. - * Next try forward as many in-order receipts as possible from child shard buffer. -3. Apply chunk and `try_forward` newly generated receipts remains the same. +1. Calculate `outgoing_limit` for both child shards using congestion info from the parent. +2. Forwarding receipts: + * First, try to forward as many in-order receipts as possible from the parent shard buffer. Stop either when we drain the parent buffer or when we exhaust the `outgoing_limit` of either of the child shards. + * Next, try to forward as many in-order receipts as possible from the child shard buffer. +3. Applying chunk and `try_forward` newly generated receipts remains the same. -The minor downside to this approach is that we don't have guarantees between order of receipt generation and order of receipt forwarding, but that's anyway the case today with buffered receipts. +The minor downside to this approach is that we don't have guarantees between the order of receipt generation and the order of receipt forwarding, but that's already the case today with buffered receipts. ### Congestion Control -Along with having buffered receipts, each chunk also publishes a CongestionInfo to the chunk header that has information about the congestion of the shard as of processing block. +Along with having buffered receipts, each chunk also publishes a `CongestionInfo` to the chunk header that contains information about the congestion of the shard during block processing. ```rust pub struct CongestionInfoV1 { @@ -343,165 +263,150 @@ pub struct CongestionInfoV1 { } ``` -After a resharding event, we need to properly initialize the congestion info for the child shards. Here's how we handle each of the fields +After a resharding event, it is essential to properly initialize the congestion info for the child shards. Here is how each field is handled: -#### delayed_receipts_gas +#### `delayed_receipts_gas` -Since the resharding strategy for delayed receipts is to duplicate them across both the child shards, we simply copy the value of `delayed_receipts_gas` across both shards. +Since the resharding strategy for delayed receipts is to duplicate them across both child shards, we simply copy the value of `delayed_receipts_gas` to both shards. -#### buffered_receipts_gas +#### `buffered_receipts_gas` -Since the resharding strategy for buffered receipts is to assign all the buffered receipts to the lower index child, we copy the `buffered_receipts_gas` from parent to lower index child and set `buffered_receipts_gas` to zero for upper index child. +Given that the strategy for buffered receipts is to assign all buffered receipts to the lower index child, we copy the `buffered_receipts_gas` from the parent to the lower index child and set `buffered_receipts_gas` to zero for the higher index child. -#### receipt_bytes +#### `receipt_bytes` -This field is harder to deal with as it contains the information from both delayed receipts and buffered receipts. To calculate this field properly, we would need the distribution of the receipt_bytes across both delayed receipts and buffered receipts. The current solution is to start storing metadata about the total `receipt_bytes` for buffered receipts in the trie. This way we have the following: +This field is more complex as it includes information from both delayed receipts and buffered receipts. To calculate this field accurately, we need to know the distribution of `receipt_bytes` across both delayed receipts and buffered receipts. The current solution is to store metadata about the total `receipt_bytes` for buffered receipts in the trie. This way, we have the following: -* For child with lower index, receipt_bytes is the sum of both delayed receipts bytes and congestion control bytes, hence `receipt_bytes = parent.receipt_bytes` -* For child with upper index, receipt_bytes is just the bytes from delayed receipts, hence `receipt_bytes = parent.receipt_bytes - buffered_receipt_bytes` +* For the child with the lower index, `receipt_bytes` is the sum of both delayed receipts bytes and buffered receipts bytes, hence `receipt_bytes = parent.receipt_bytes`. +* For the child with the higher index, `receipt_bytes` is just the bytes from delayed receipts, hence `receipt_bytes = parent.receipt_bytes - parent.buffered_receipt_bytes`. -#### allowed_shard +#### `allowed_shard` -This field is calculated by a round-robin mechanism which can be independently calculated for both the child shards. Since we are changing the [ShardId semantics](#shardid-semantics), we need to change implementation to use `ShardIndex` instead of `ShardID` which is just an assignment for each shard_id to the contiguous index `[0, num_shards)`. +This field is calculated using a round-robin mechanism, which can be independently determined for both child shards. Since we are changing the [ShardId semantics](#shardid-semantics), we need to update the implementation to use `ShardIndex` instead of `ShardID`, which is simply an assignment for each `shard_id` to the contiguous index `[0, num_shards)`. ### ShardId Semantics -Currently, shard IDs are represented as numbers within the range `[0,n)`, where n is the total number of shards. These shard IDs are sorted in the same order as the account ID ranges assigned to them. While this approach is straightforward, it complicates resharding operations, particularly when splitting a shard in the middle of the range. Such a split requires reindexing all subsequent shards with higher IDs, adding complexity to the process. +Currently, shard IDs are represented as numbers within the range `[0, n)`, where `n` is the total number of shards. These shard IDs are sorted in the same order as the account ID ranges assigned to them. While this approach is straightforward, it complicates resharding operations, particularly when splitting a shard in the middle of the range. Such a split requires reindexing all subsequent shards with higher IDs, adding complexity to the process. -In this NEP, we propose updating the ShardId semantics to allow for arbitrary identifiers. Although ShardIds will remain integers, they will no longer be restricted to the `[0,n)` range, and they may appear in any order. The only requirement is that each ShardId must be unique. In practice, during resharding, the ID of a parent shard will be removed from the ShardLayout, and the new child shards will be assigned unique IDs - `max(shard_ids)+1` and `max(shard_ids)+2`. +In this NEP, we propose updating the ShardId semantics to allow for arbitrary identifiers. Although ShardIds will remain integers, they will no longer be restricted to the `[0, n)` range, and they may appear in any order. The only requirement is that each ShardId must be unique. In practice, during resharding, the ID of a parent shard will be removed from the ShardLayout, and the new child shards will be assigned unique IDs - `max(shard_ids) + 1` and `max(shard_ids) + 2`. ## Reference Implementation ### Overview -1. Any node tracking shard must determine if it should split shard in the last block before the epoch where resharding should happen. +1. Any node tracking a shard must determine if it should split the shard in the last block before the epoch where resharding should happen. ```pseudocode should_split_shard(block, shard_id): - shard_layout = epoch_manager.shard_layout(block.epoch_id()) - next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) - if epoch_manager.is_next_block_epoch_start(block) && - shard_layout != next_shard_layout && - next_shard_layout.shard_split_map().contains(shard_id): - return Some(next_shard_layout.split_shard_event(shard_id)) - return None + shard_layout = epoch_manager.shard_layout(block.epoch_id()) + next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) + if epoch_manager.is_next_block_epoch_start(block) && + shard_layout != next_shard_layout && + next_shard_layout.shard_split_map().contains(shard_id): + return Some(next_shard_layout.split_shard_event(shard_id)) + return None ``` -2. This logic is triggered on block postprocessing, which means that block is valid and is being persisted to disk. +2. This logic is triggered during block post-processing, which means that the block is valid and is being persisted to disk. ```pseudocode on chain.postprocess_block(block): - next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) - if let Some(split_shard_event) = should_split_shard(block, shard_id): - resharding_manager.split_shard(split_shard_event) + next_shard_layout = epoch_manager.shard_layout(block.next_epoch_id()) + if let Some(split_shard_event) = should_split_shard(block, shard_id): + resharding_manager.split_shard(split_shard_event) ``` 3. The event triggers changes in all state storage components. ```pseudocode on resharding_manager.split_shard(split_shard_event, next_shard_layout): - set State mapping - start FlatState resharding - process MemTrie resharding: - freeze MemTrie, create HybridMemTries - for each child shard: - mem_tries[parent_shard].retain_split_shard(boundary_account) + set State mapping + start FlatState resharding + process MemTrie resharding: + freeze MemTrie, create HybridMemTries + for each child shard: + mem_tries[parent_shard].retain_split_shard(boundary_account) ``` -4. `retain_split_shard` leaves only keys in trie that belong to the left or right child shard. -It retains trie key intervals for left or right child as described above. Simultaneously the proof is generated. -In the end, we get new state root, hybrid memtrie corresponding to child shard, and the proof. -Proof is saved as state transition for pair `(block, new_shard_uid)`. +4. `retain_split_shard` leaves only keys in the trie that belong to the left or right child shard. It retains trie key intervals for the left or right child as described above. Simultaneously, the proof is generated. In the end, we get a new state root, hybrid MemTrie corresponding to the child shard, and the proof. Proof is saved as state transition for pair `(block, new_shard_uid)`. -5. The proof is sent as one of implicit transitions in ChunkStateWitness. +5. The proof is sent as one of the implicit transitions in `ChunkStateWitness`. -6. On chunk validation path, chunk validator understands if resharding is -a part of state transition, using the same `should_split_shard` condition. +6. On the chunk validation path, the chunk validator determines if resharding is part of the state transition using the same `should_split_shard` condition. -7. It calls `Trie(state_transition_proof).retain_split_shard(boundary_account)` -which should succeed if proof is sufficient and generates new state root. +7. It calls `Trie(state_transition_proof).retain_split_shard(boundary_account)`, which should succeed if the proof is sufficient and generates a new state root. -8. Finally, it checks that the new state root matches the state root proposed in `ChunkStateWitness`. -If the whole `ChunkStateWitness` is valid, then chunk validator sends endorsement which also endorses the resharding. +8. Finally, it checks that the new state root matches the state root proposed in `ChunkStateWitness`. If the whole `ChunkStateWitness` is valid, then the chunk validator sends an endorsement, which also endorses the resharding. ### State Storage - MemTrie -The current implementation of MemTrie uses a pool of memory (`STArena`) to allocate and deallocate nodes and internal pointers in this pool to reference child nodes. MemTries, unlike the State representation of Trie, do not work with the hash of the nodes but internal memory pointers directly. Additionally, MemTries are not thread safe and one MemTrie exists per shard. +The current implementation of MemTrie uses a memory pool (`STArena`) to allocate and deallocate nodes, with internal pointers in this pool referencing child nodes. Unlike the State representation of the Trie, MemTries do not work with node hashes but with internal memory pointers directly. Additionally, MemTries are not thread-safe, and one MemTrie exists per shard. -As described in [MemTrie](#state-storage---memtrie) section above, we need an efficient way to split the MemTrie into two child MemTries within a span of 1 block. What makes this challenging is that the current implementation of MemTrie is not thread safe and can not be shared across two shards. +As described in the [MemTrie](#state-storage---memtrie) section above, we need an efficient way to split the MemTrie into two child MemTries within the span of one block. The challenge lies in the current implementation of MemTrie, which is not thread-safe and cannot be shared across two shards. -The naive way to create two MemTries for the child shards would be to iterate through all the entries of the parent MemTrie and fill in these values into the child MemTries. This however is prohibitively time consuming. +A naive approach to creating two MemTries for the child shards would involve iterating through all entries of the parent MemTrie and populating these values into the child MemTries. However, this method is prohibitively time-consuming. -The solution to this problem was to introduce the concept of Frozen MemTrie (with a `FrozenArena`) which is a cloneable, read-only, thread-safe snapshot of a MemTrie. We can call the `freeze` method on an existing MemTrie that converts it into a Frozen MemTrie. Note that this process consumes the original MemTrie and we can no longer allocate and deallocate nodes to it. +The solution to this problem is to introduce the concept of a Frozen MemTrie (with a `FrozenArena`), which is a cloneable, read-only, thread-safe snapshot of a MemTrie. By calling the `freeze` method on an existing MemTrie, we convert it into a Frozen MemTrie. This process consumes the original MemTrie, making it no longer available for node allocation and deallocation. -Along with `FrozenArena`, we also introduce a `HybridArena` which is effectively a base made of `FrozenArena` with a top layer of `STArena` where we support allocating and deallocating new nodes into the MemTrie. Newly allocated nodes can reference/point to nodes in the `FrozenArena`. We use this Hybrid MemTrie as a temporary MemTrie while the flat storage is being constructed in the background. +Along with `FrozenArena`, we also introduce a `HybridArena`, which effectively combines a base `FrozenArena` with a top layer of `STArena` that supports allocating and deallocating new nodes into the MemTrie. Newly allocated nodes can reference nodes in the `FrozenArena`. This Hybrid MemTrie serves as a temporary MemTrie while the flat storage is being constructed in the background. -While Frozen MemTries provide the benefits of being compatible with instant resharding, they come at the cost of memory consumption. Once a MemTrie is frozen, since it doesn't support deallocation of memory, it continues to consume as much memory as it did at the time of freezing. In case a node is tracking only one of the child shards, a Frozen MemTrie would continue to use the same amount of memory as the parent trie. Due to this, Hybrid MemTries are only a temporary solution and we rebuild the MemTrie for the children once the post-processing step for Flat Storage is completed. +While Frozen MemTries facilitate instant resharding, they come at the cost of memory consumption. Once a MemTrie is frozen, it continues to consume the same amount of memory as it did at the time of freezing, as it does not support memory deallocation. If a node tracks only one of the child shards, a Frozen MemTrie would continue to use the same amount of memory as the parent trie. Therefore, Hybrid MemTries are only a temporary solution, and we rebuild the MemTrie for the children after resharding is completed. -Additionally, a node would have to support 2x the memory footprint of a single trie as after resharding, we would have two copies of the trie in memory, one from the temporary Hybrid MemTrie in use for block production, and other from the background MemTrie that would be under construction. Once the background MemTrie is fully constructed and caught up with the latest block, we do an in-place swap of the Hybrid MemTrie with the new child MemTrie and deallocate the memory from the Hybrid MemTrie. +Additionally, a node would need to support twice the memory footprint of a single trie. After resharding, there would be two copies of the trie in memory: one from the temporary Hybrid MemTrie used for block production and another from the background MemTrie under construction. Once the background MemTrie is fully constructed and caught up with the latest block, we perform an in-place swap of the Hybrid MemTrie with the new child MemTrie and deallocate the memory from the Hybrid MemTrie. -During a resharding event, at the boundary of the epoch, when we need to split the parent shard into the two child shards, we do the following steps: +During a resharding event at the epoch boundary, when we need to split the parent shard into two child shards, we follow these steps: -1. Freeze the parent MemTrie arena to create a read-only frozen arena that represents a snapshot of the state as of the time of freezing, i.e. after postprocessing last block of epoch. Note that we no longer require the parent MemTrie in runtime going forward. -2. We cheaply clone the Frozen MemTrie for both the child MemTries to use. Note that this doesn't clone the parent arena memory, but just increases the refcount. -3. We then create a new MemTrie with HybridArena for each of the children. The base of the MemTrie is the read-only FrozenArena while all new node allocations happens on a dedicated STArena memory pool for each child MemTrie. This is the temporary MemTrie that we use while Flat Storage is being built in the background. -4. Once the Flat Storage is constructed in the post processing step of resharding, we use that to load a new MemTrie and catchup to the latest block. -5. After the new child MemTrie has caught up to the latest block, we do an in-place swap in Client and discard the Hybrid MemTrie. +1. **Freeze the Parent MemTrie**: Create a read-only frozen arena representing a snapshot of the state at the time of freezing (after post-processing the last block of the epoch). The parent MemTrie is no longer required in runtime going forward. +2. **Clone the Frozen MemTrie**: Clone the Frozen MemTrie cheaply for both child MemTries to use. This does not clone the parent arena's memory but merely increases the reference count. +3. **Create Hybrid MemTries for Each Child**: Create a new MemTrie with `HybridArena` for each child. The base of the MemTrie is the read-only `FrozenArena`, while all new node allocations occur in a dedicated `STArena` memory pool for each child MemTrie. This temporary MemTrie is used while Flat Storage is being built in the background. +4. **Rebuild MemTrie**: Once resharding is completed, we use it to load a new MemTrie and catch up to the latest block. +5. **Swap and Clean Up**: After the new child MemTrie has caught up to the latest block, we perform an in-place swap in the client and discard the Hybrid MemTrie. ![Hybrid MemTrie diagram](assets/nep-0568/NEP-HybridMemTrie.png) ### State Storage - Flat State -Resharding Flat State is a time consuming operation and it runs in parallel with block processing for several block heights. -Thus, there are a few important aspects to consider during implementation: +Resharding the Flat State is a time-consuming operation that runs in parallel with block processing for several block heights. Therefore, several important aspects must be considered during implementation: -* Flat State's own status should be resilient to application crashes. -* The parent shard's Flat State should be split at the correct block height. -* New shards' Flat States should eventually converge to same representation the chain is using to process blocks (MemTries). -* Resharding should work correctly in the presence of chain forks. -* Retired shards are cleaned up. +* **Flat State's Status Persistence**: Flat State's status should be resilient to application crashes. +* **Correct Block Height**: The parent shard's Flat State should be split at the correct block height. +* **Convergence with Mem Trie**: New shards' Flat States should eventually converge to the same representation the chain uses to process blocks (MemTries). +* **Chain Forks Handling**: Resharding should work correctly in the presence of chain forks. +* **Retired Shards Cleanup**: Retired shards should be cleaned up. -Note that the Flat States of the newly created shards won't be available until resharding is completed. This is fine because the temporary MemTries are -built instantly and they can satisfy all block processing needs. +Note that the Flat States of the newly created shards will not be available until resharding is completed. This is acceptable because the temporary MemTries are built instantly and can satisfy all block processing needs. -The main component responsible to carry out resharding on Flat State is the [FlatStorageResharder](https://github.com/near/nearcore/blob/f4e9dd5d6e07089dfc789221ded8ec83bfe5f6e8/chain/chain/src/flat_storage_resharder.rs#L68). +The main component responsible for carrying out resharding on Flat State is the [FlatStorageResharder](https://github.com/near/nearcore/blob/f4e9dd5d6e07089dfc789221ded8ec83bfe5f6e8/chain/chain/src/flat_storage_resharder.rs#L68). -#### Flat State's status persistence +#### Flat State's Status Persistence -Every shard Flat State has a status associated to it and stored in the database, called `FlatStorageStatus`. We propose to extend the existing object -by adding the new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is -persisted to disk, which makes the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat -State creation (see `FlatStorageShardCreator`), reducing the code duplication factor. +Every shard's Flat State has a status associated with it and stored in the database, called `FlatStorageStatus`. We propose extending the existing object by adding a new enum variant named `FlatStorageStatus::Resharding`. This approach has two benefits. First, the progress of any Flat State resharding is persisted to disk, making the operation resilient to a node crash or restart. Second, resuming resharding on node restart shares the same code path as Flat State creation (see `FlatStorageShardCreator`), reducing code duplication. -`FlatStorageStatus` is updated at every committable step of resharding. The commit points are the following: +`FlatStorageStatus` is updated at every committable step of resharding. The commit points are as follows: -* Beginning of resharding or, in other words, the last block of the old shard layout. -* Scheduling of the _"split parent shard"_ task. -* Execution, cancellation or failure of the _"split parent shard"_ task. -* Execution or failure of any _"child catchup"_ task. +* Beginning of resharding, at the last block of the old shard layout. +* Scheduling of the "split parent shard" task. +* Execution, cancellation, or failure of the "split parent shard" task. +* Execution or failure of any "child catchup" task. -#### Splitting a shard Flat State +#### Splitting a Shard's Flat State -When, at the end of an epoch, the shard layout changes we identify a so called _resharding block_ that corresponds to the last block of the current epoch. -A task to split the parent shard's Flat State is scheduled to happen after the _resharding block_ becomes final. The reason to wait for the finality condition -is to avoid a split on a block that might be excluded from the canonical chain; needless to say, such situation would lock the node -into an erroneous state. +When the shard layout changes at the end of an epoch, we identify a **resharding block** corresponding to the last block of the current epoch. A task to split the parent shard's Flat State is scheduled to occur after the resharding block becomes final. The finality condition is necessary to avoid splitting on a block that might be excluded from the canonical chain, which would lock the node into an erroneous state. -Inside the split task we iterate over the Flat State and copy each element into either child. This routine is performed in batches in order to lessen the performance -impact on the node. +Inside the split task, we iterate over the Flat State and copy each element into either child. This routine is performed in batches to lessen the performance impact on the node. -Finally, if the split completes successfully, the parent shard Flat State is removed from the database and the children Flat States enter a catch-up phase. +Finally, if the split completes successfully, the parent shard's Flat State is removed from the database, and the child shards' Flat States enter a catch-up phase. -One current technical limitation is that, upon a node crash or restart, the _"split parent shard"_ task will start copying all elements again from the beginning. +One current technical limitation is that, upon a node crash or restart, the "split parent shard" task will start copying all elements again from the beginning. A reference implementation of splitting a Flat State can be found in [FlatStorageResharder::split_shard_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L295). -#### Values assignment from parent to child shards +#### Assigning Values from Parent to Child Shards -Key-value pairs in the parent shard Flat State are inherited by children according to the rules stated below. +Key-value pairs in the parent shard's Flat State are inherited by children according to the following rules: -Elements inherited by the child shard which tracks the `account_id` contained in the key: +**Elements inherited by the child shard tracking the `account_id` contained in the key:** * `ACCOUNT` * `CONTRACT_DATA` @@ -513,51 +418,45 @@ Elements inherited by the child shard which tracks the `account_id` contained in * `POSTPONED_RECEIPT` * `PROMISE_YIELD_RECEIPT` -Elements inherited by both children: +**Elements inherited by both children:** * `DELAYED_RECEIPT_OR_INDICES` * `PROMISE_YIELD_INDICES` * `PROMISE_YIELD_TIMEOUT` * `BANDWIDTH_SCHEDULER_STATE` -Elements inherited only be the lowest index child: +**Elements inherited only by the lowest index child:** * `BUFFERED_RECEIPT_INDICES` * `BUFFERED_RECEIPT` -#### Bring children shards up to date with the chain's head +#### Bringing Child Shards Up to Date with the Chain's Head -Children shards' Flat States build a complete view of their content at the height of the `resharding block` sometime during the new epoch -after resharding. At that point in time many new blocks have been processed already, and these will most likely contain updates for the new shards. A catch-up step is necessary to apply all Flat State deltas accumulated until now. +Child shards' Flat States build a complete view of their content at the height of the resharding block sometime during the new epoch after resharding. At that point, many new blocks have already been processed, and these will most likely contain updates for the new shards. A catch-up step is necessary to apply all Flat State deltas accumulated until now. -This phase of resharding doesn't have to take extra steps to handle chain forks. On one hand, the catch-up task doesn't start until the parent shard -splitting is done, and at such point we know the `resharding block` is final; on the other hand, Flat State deltas are capable of handling forks automatically. +This phase of resharding does not require extra steps to handle chain forks. The catch-up task does not start until the parent shard splitting is done, ensuring the resharding block is final. Additionally, Flat State deltas can handle forks automatically. -The catch-up task commits to the database "batches" of Flat State deltas. If the application crashes or restarts the task will resume from where it left. +The catch-up task commits batches of Flat State deltas to the database. If the application crashes or restarts, the task will resume from where it left off. -Once all Flat State deltas are applied, the child shard's status is changed to `Ready` and clean up of Flat State deltas leftovers is performed. +Once all Flat State deltas are applied, the child shard's status is changed to `Ready`, and cleanup of Flat State delta leftovers is performed. A reference implementation of the catch-up task can be found in [FlatStorageResharder::shard_catchup_task](https://github.com/near/nearcore/blob/fecce019f0355cf89b63b066ca206a3cdbbdffff/chain/chain/src/flat_storage_resharder.rs#L564). -#### Failure of Flat State resharding +#### Failure of Flat State Resharding -In the current proposal any failure during Flat State resharding is considered non-recoverable. -`neard` will attempt resharding again on restart, but no automatic recovery is implemented. +In the current proposal, any failure during Flat State resharding is considered non-recoverable. `neard` will attempt resharding again on restart, but no automatic recovery is implemented. -### State Storage - State mapping +### State Storage - State Mapping -To enable efficient shard state management during resharding, Resharding V3 uses the `DBCol::ShardUIdMapping` column. -This mapping allows child shards to reference ancestor shard data, avoiding the need for immediate duplication of state entries. +To enable efficient shard state management during resharding, Resharding V3 uses the `DBCol::ShardUIdMapping` column. This mapping allows child shards to reference ancestor shard data, avoiding the need for immediate duplication of state entries. -#### Mapping application in adapters +#### Mapping Application in Adapters -The core of the mapping logic is applied in `TrieStoreAdapter` and `TrieStoreUpdateAdapter`, which act as layers over the general `Store` interface. -Here’s a breakdown of the key functions involved: +The core of the mapping logic is applied in `TrieStoreAdapter` and `TrieStoreUpdateAdapter`, which act as layers over the general `Store` interface. Here’s a breakdown of the key functions involved: -* **Key resolution**: - The `get_key_from_shard_uid_and_hash` function is central to determining the correct `ShardUId` for state access. - At a high level, operations use the child shard's `ShardUId`, but within this function, - the `DBCol::ShardUIdMapping` column is checked to determine if an ancestor `ShardUId` should be used instead. +* **Key Resolution**: + + The `get_key_from_shard_uid_and_hash` function is central to determining the correct `ShardUId` for state access. At a high level, operations use the child shard's `ShardUId`, but within this function, the `DBCol::ShardUIdMapping` column is checked to determine if an ancestor `ShardUId` should be used instead. ```rust fn get_key_from_shard_uid_and_hash( @@ -576,41 +475,37 @@ Here’s a breakdown of the key functions involved: } ``` - This function first attempts to retrieve a mapped ancestor `ShardUId` from `DBCol::ShardUIdMapping`. - If no mapping exists, it defaults to the provided child `ShardUId`. - This resolved `ShardUId` is then combined with the `node_hash` to form the final key used in `State` column operations. + This function first attempts to retrieve a mapped ancestor `ShardUId` from `DBCol::ShardUIdMapping`. If no mapping exists, it defaults to the provided child `ShardUId`. This resolved `ShardUId` is then combined with the `node_hash` to form the final key used in `State` column operations. + +* **State Access Operations**: -* **State access operations**: - The `TrieStoreAdapter` and `TrieStoreUpdateAdapter` use `get_key_from_shard_uid_and_hash` to correctly resolve the key for both reads and writes. - Example methods include: + The `TrieStoreAdapter` and `TrieStoreUpdateAdapter` use `get_key_from_shard_uid_and_hash` to correctly resolve the key for both reads and writes. Example methods include: - ```rust - // In TrieStoreAdapter - pub fn get(&self, shard_uid: ShardUId, hash: &CryptoHash) -> Result, StorageError> { + ```rust + // In TrieStoreAdapter + pub fn get(&self, shard_uid: ShardUId, hash: &CryptoHash) -> Result, StorageError> { let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); self.store.get(DBCol::State, &key) - } + } - // In TrieStoreUpdateAdapter - pub fn increment_refcount_by( + // In TrieStoreUpdateAdapter + pub fn increment_refcount_by( &mut self, shard_uid: ShardUId, hash: &CryptoHash, data: &[u8], increment: NonZero, - ) { + ) { let key = get_key_from_shard_uid_and_hash(self.store, shard_uid, hash); self.store_update.increment_refcount_by(DBCol::State, key.as_ref(), data, increment); - } - ``` + } + ``` - The `get` function retrieves data using the resolved `ShardUId` and key, while `increment_refcount_by` manages reference counts, - ensuring correct tracking even when accessing data under an ancestor shard. + The `get` function retrieves data using the resolved `ShardUId` and key, while `increment_refcount_by` manages reference counts, ensuring correct tracking even when accessing data under an ancestor shard. -#### Mapping retention and cleanup +#### Mapping Retention and Cleanup -Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on an ancestor’s data. -To manage this, the `set_shard_uid_mapping` function in `TrieStoreUpdateAdapter` adds a new mapping during resharding: +Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on an ancestor’s data. To manage this, the `set_shard_uid_mapping` function in `TrieStoreUpdateAdapter` adds a new mapping during resharding: ```rust fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: ShardUId) { @@ -622,59 +517,57 @@ fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: } ``` -When a node stops tracking all descendants of a shard, the associated mapping entry can be removed, allowing RocksDB to perform garbage collection. -For archival nodes, mappings are retained permanently to ensure access to the historical state of all shards. +When a node stops tracking all descendants of a shard, the associated mapping entry can be removed, allowing RocksDB to perform garbage collection. For archival nodes, mappings are retained permanently to ensure access to the historical state of all shards. -This implementation ensures efficient and scalable shard state transitions, -allowing child shards to use ancestor data without creating redundant entries. +This implementation ensures efficient and scalable shard state transitions, allowing child shards to use ancestor data without creating redundant entries. ### State Sync -The state sync algorithm defines a `sync_hash` that is used in many parts of the implementation. This is always the first block of the current epoch, which the node should be aware of once it has synced headers to the current point in the chain. A node performing state sync first makes a request (currently to centralized storage on GCS, but in the future to other nodes in the network) for a `ShardStateSyncResponseHeader` corresponding to that `sync_hash` and the Shard ID of the shard it's interested in. Among other things, this header includes the last new chunk before `sync_hash` in the shard, and a `StateRootNode` with hash equal to that chunk's `prev_state_root` field. Then the node downloads (again from GCS, but in the future it'll be from other nodes) the nodes of the trie with that `StateRootNode` as its root. Afterwards, it applies new chunks in the shard until it's caught up. +The state sync algorithm defines a `sync_hash` used in many parts of the implementation. This is always the first block of the current epoch, which the node should be aware of once it has synced headers to the current point in the chain. A node performing state sync first makes a request for a `ShardStateSyncResponseHeader` corresponding to that `sync_hash` and the Shard ID of the shard it's interested in. Among other things, this header includes the last new chunk before `sync_hash` in the shard and a `StateRootNode` with a hash equal to that chunk's `prev_state_root` field. Then the node downloads the nodes of the trie with that `StateRootNode` as its root. Afterwards, it applies new chunks in the shard until it's caught up. - As described above, the state we download is the state in the shard after applying the second to last new chunk before `sync_hash`, which belongs to the previous epoch (since `sync_hash` is the first block of the new epoch). To move the point in the chain of the initial state download to the current epoch, we could either move the `sync_hash` forward or we could change the state sync protocol (perhaps changing the meaning of the `sync_hash` and the fields of the `ShardStateSyncResponseHeader`, or somehow changing these structures more significantly). The former is an easier first implementation, since it would not require any changes to the state sync protocol other than to the expected `sync_hash`. We would just need to move the `sync_hash` to a point far enough along in the chain so that the `StateRootNode` in the `ShardStateSyncResponseHeader` refers to state in the current epoch. Currently we plan on implementing it that way, but we may revisit making more extensive changes to the state sync protocol later. +As described above, the state we download is the state in the shard after applying the second-to-last new chunk before `sync_hash`, which belongs to the previous epoch (since `sync_hash` is the first block of the new epoch). To move the point in the chain of the initial state download to the current epoch, we could either move the `sync_hash` forward or change the state sync protocol (perhaps changing the meaning of the `sync_hash` and the fields of the `ShardStateSyncResponseHeader`, or somehow changing these structures more significantly). The former is an easier first implementation, as it would not require any changes to the state sync protocol other than to the expected `sync_hash`. We would just need to move the `sync_hash` to a point far enough along in the chain so that the `StateRootNode` in the `ShardStateSyncResponseHeader` refers to the state in the current epoch. Currently, we plan on implementing it that way, but we may revisit making more extensive changes to the state sync protocol later. ## Security Implications ### Fork Handling -In theory, it can happen that there will be more than one candidate block which finishes the last epoch with old shard layout. For previous implementations it didn't matter because resharding decision was made in the beginning previous epoch. Now, the decision is made on the epoch boundary, so the new implementation handles this case as well. +In theory, it's possible that more than one candidate block finishes the last epoch with the old shard layout. For previous implementations, it didn't matter because the resharding decision was made at the beginning of the previous epoch. Now, the decision is made at the epoch boundary, so the new implementation handles this case as well. ### Proof Validation -With single shard tracking, nodes can't independently validate new state roots after resharding, because they don't have state of shard being split. That's why we generate resharding proofs, whose generation and validation may be a new weak point. However, `retain_split_shard` is equivalent to constant number of lookups in the trie, so its overhead its negligible. Even if proof is invalid, it will only imply that `retain_split_shard` fails early, similarly to other state transitions. +With single shard tracking, nodes can't independently validate new state roots after resharding because they don't have the state of the shard being split. That's why we generate resharding proofs, whose generation and validation may be a new weak point. However, `retain_split_shard` is equivalent to a constant number of lookups in the trie, so its overhead is negligible. Even if the proof is invalid, it will only imply that `retain_split_shard` fails early, similar to other state transitions. ## Alternatives -In the solution space which would keep blockchain stateful, we also considered an alternative to handle resharding through mechanism of `Receipts`. The workflow would be to: +In the solution space that would keep the blockchain stateful, we also considered an alternative to handle resharding through the mechanism of `Receipts`. The workflow would be to: -* create empty `target_shard`, -* require `source_shard` chunk producers to create special `ReshardingReceipt(source_shard, target_shard, data)` where `data` would be an interval of key-value pairs in `source_shard` alongside with the proof, -* then, `target_shard` trackers and validators would process that receipt, validate the proof and insert the key-value pairs into the new shard. +* Create an empty `target_shard`. +* Require `source_shard` chunk producers to create special `ReshardingReceipt(source_shard, target_shard, data)` where `data` would be an interval of key-value pairs in `source_shard` along with the proof. +* Then, `target_shard` trackers and validators would process that receipt, validate the proof, and insert the key-value pairs into the new shard. -However, `data` would occupy most of the whole state witness capacity and introduce overhead of proving every single interval in `source_shard`. Moreover, approach to sync target shard "dynamically" also requires some form of catchup, which makes it much less feasible than chosen approach. +However, `data` would occupy most of the state witness capacity and introduce the overhead of proving every single interval in `source_shard`. Moreover, the approach to sync the target shard "dynamically" also requires some form of catch-up, which makes it much less feasible than the chosen approach. -Another question is whether we should tie resharding to epoch boundaries. This would allow to come from resharding decision to completion much faster. But for that, we would need to: +Another question is whether we should tie resharding to epoch boundaries. This would allow us to move from the resharding decision to completion much faster. But for that, we would need to: -* agree if we should reshard in the middle of the epoch or allow "fast epoch completion" which has to be implemented, -* keep chunk producers tracking "spare shards" ready to receive items from split shards, -* on resharding event, implement specific form of state sync, on which source and target chunk producers would agree on new state roots offline, -* then, new state roots would be validated by chunk validators in the same fashion. +* Agree if we should reshard in the middle of the epoch or allow "fast epoch completion," which has to be implemented. +* Keep chunk producers tracking "spare shards" ready to receive items from split shards. +* On resharding event, implement a specific form of state sync, on which source and target chunk producers would agree on new state roots offline. +* Then, new state roots would be validated by chunk validators in the same fashion. -While it is much closer to Dynamic Resharding (below), it requires much more changes to the protocol. And the considered idea works very well as intermediate step to that, if needed. +While it is much closer to Dynamic Resharding (below), it requires many more changes to the protocol. The considered idea works well as an intermediate step toward that, if needed. -## Future possibilities +## Future Possibilities -* Dynamic Resharding - In this proposal, resharding is scheduled in advance and hardcoded within the neard binary. In the future, we aim to enable the chain to dynamically trigger and execute resharding autonomously, allowing it to adjust capacity automatically based on demand. -* Fast Dynamic Resharding - In the Dynamic Resharding extension, the new shard layout is configured for the second upcoming epoch. This means that a full epoch must pass before the chain transitions to the updated shard layout. In the future, our goal is to accelerate this process by finalizing the previous epoch more quickly, allowing the chain to adopt the new layout as soon as possible. -* Shard Merging - In this proposal the only allowed resharding operation is shard splitting. In the future, we aim to enable shard merging, allowing underutilized shards to be combined with neighboring shards. This would allow the chain to free up resources and reallocate them where they are most needed. +* **Dynamic Resharding**: In this proposal, resharding is scheduled in advance and hardcoded within the `neard` binary. In the future, we aim to enable the chain to dynamically trigger and execute resharding autonomously, allowing it to adjust capacity automatically based on demand. +* **Fast Dynamic Resharding**: In the Dynamic Resharding extension, the new shard layout is configured for the second upcoming epoch. This means that a full epoch must pass before the chain transitions to the updated shard layout. In the future, our goal is to accelerate this process by finalizing the previous epoch more quickly, allowing the chain to adopt the new layout as soon as possible. +* **Shard Merging**: In this proposal, the only allowed resharding operation is shard splitting. In the future, we aim to enable shard merging, allowing underutilized shards to be combined with neighboring shards. This would allow the chain to free up resources and reallocate them where they are most needed. ## Consequences ### Positive -* The protocol is able to execute resharding even while only a fraction of nodes track the split shard. -* State for new shard layouts is computed in the matter of minutes instead of hours, thus ecosystem stability during resharding is greatly increased. As before, from the point of view of NEAR users it is instantaneous. +* The protocol can execute resharding even while only a fraction of nodes track the split shard. +* State for new shard layouts is computed in a matter of minutes instead of hours, greatly increasing ecosystem stability during resharding. As before, from the point of view of NEAR users, it is instantaneous. ### Neutral @@ -682,7 +575,7 @@ N/A ### Negative -* The storage components need to handle additional complexity of controlling the shard layout change. +* The storage components need to handle the additional complexity of controlling the shard layout change. ### Backwards Compatibility From 74301d0b612802d123ee503ec73b27ac9b6fafe0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Chuda=C5=9B?= <18039094+staffik@users.noreply.github.com> Date: Fri, 3 Jan 2025 18:19:38 +0100 Subject: [PATCH 30/30] fix(resharding): update pseudocode (#583) --- neps/nep-0568.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/neps/nep-0568.md b/neps/nep-0568.md index 6e4d9dd93..ebdcebc81 100644 --- a/neps/nep-0568.md +++ b/neps/nep-0568.md @@ -509,10 +509,14 @@ Mappings in `DBCol::ShardUIdMapping` persist as long as any descendant relies on ```rust fn set_shard_uid_mapping(&mut self, child_shard_uid: ShardUId, parent_shard_uid: ShardUId) { + let mapped_parent_shard_uid = store + .get_ser::(DBCol::StateShardUIdMapping, &parent_shard_uid.to_bytes()) + .expect("set_shard_uid_mapping() failed") + .unwrap_or(parent_shard_uid); self.store_update.set( DBCol::StateShardUIdMapping, child_shard_uid.to_bytes().as_ref(), - &borsh::to_vec(&parent_shard_uid).expect("Borsh serialize cannot fail"), + &borsh::to_vec(&mapped_parent_shard_uid).expect("Borsh serialize cannot fail"), ) } ```