diff --git a/README.md b/README.md index 2ac517436..dfb1ff58b 100644 --- a/README.md +++ b/README.md @@ -10,29 +10,40 @@ Changes to the protocol specification and standards are called NEAR Enhancement ## NEPs -| NEP # | Title | Author | Status | -| ----------------------------------------------------------------- | ---------------------------------------- | -------------------------------------------- | ---------- | -| [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living | -| [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Deprecated | -| [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack, @robert-zaremba | Final | -| [0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final | -| [0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final | -| [0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final | -| [0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final | -| [0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final | -| [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final | -| [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final | -| [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review | -| [0264](https://github.com/near/NEPs/blob/master/neps/nep-0264.md) | Promise Gas Weights | @austinabell | Final | -| [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final | -| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review | -| [0366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md) | Meta Transactions | @ilblackdragon @e-uleyskiy @fadeevab | Final | -| [0393](https://github.com/near/NEPs/blob/master/neps/nep-0393.md) | Sould Bound Token (SBT) | @robert-zaremba | Final | -| [0399](https://github.com/near/NEPs/blob/master/neps/nep-0399.md) | Flat Storage | @Longarithm @mzhangmzz | Final | -| [0448](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) | Zero-balance Accounts | @bowenwang1996 | Final | -| [0452](https://github.com/near/NEPs/blob/master/neps/nep-0452.md) | Linkdrop Standard | @benkurrek @miyachi | Final | -| [0455](https://github.com/near/NEPs/blob/master/neps/nep-0455.md) | Parameter Compute Costs | @akashin @jakmeier | Final | -| [0514](https://github.com/near/NEPs/blob/master/neps/nep-0514.md) | Fewer Block Producer Seats in `testnet` | @nikurt | Final | +| NEP # | Title | Author | Status | +| ----------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------- | ---------- | +| [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living | +| [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Deprecated | +| [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack, @robert-zaremba | Final | +| [0145](https://github.com/near/NEPs/blob/master/neps/nep-0145.md) | Storage Management | @evgenykuzyakov | Final | +| [0148](https://github.com/near/NEPs/blob/master/neps/nep-0148.md) | Fungible Token Metadata | @robert-zaremba @evgenykuzyakov @oysterpack | Final | +| [0171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) | Non Fungible Token Standard | @mikedotexe @evgenykuzyakov @oysterpack | Final | +| [0177](https://github.com/near/NEPs/blob/master/neps/nep-0177.md) | Non Fungible Token Metadata | @chadoh @mikedotexe | Final | +| [0178](https://github.com/near/NEPs/blob/master/neps/nep-0178.md) | Non Fungible Token Approval Management | @chadoh @thor314 | Final | +| [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final | +| [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final | +| [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Final | +| [0264](https://github.com/near/NEPs/blob/master/neps/nep-0264.md) | Promise Gas Weights | @austinabell | Final | +| [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final | +| [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Final | +| [0364](https://github.com/near/NEPs/blob/master/neps/nep-0364.md) | Efficient signature verification and hashing precompile functions | @blasrodri | Final | +| [0366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md) | Meta Transactions | @ilblackdragon @e-uleyskiy @fadeevab | Final | +| [0393](https://github.com/near/NEPs/blob/master/neps/nep-0393.md) | Sould Bound Token (SBT) | @robert-zaremba | Final | +| [0399](https://github.com/near/NEPs/blob/master/neps/nep-0399.md) | Flat Storage | @Longarithm @mzhangmzz | Final | +| [0413](https://github.com/near/NEPs/blob/master/neps/nep-0413.md) | Near Wallet API - support for signMessage method | @gagdiez @gutsyphilip | Final | +| [0418](https://github.com/near/NEPs/blob/master/neps/nep-0418.md) | Remove attached_deposit view panic | @austinabell | Final | +| [0448](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) | Zero-balance Accounts | @bowenwang1996 | Final | +| [0452](https://github.com/near/NEPs/blob/master/neps/nep-0452.md) | Linkdrop Standard | @benkurrek @miyachi | Final | +| [0455](https://github.com/near/NEPs/blob/master/neps/nep-0455.md) | Parameter Compute Costs | @akashin @jakmeier | Final | +| [0488](https://github.com/near/NEPs/blob/master/neps/nep-0488.md) | Host Functions for BLS12-381 Curve Operations | @olga24912 | Final | +| [0491](https://github.com/near/NEPs/blob/master/neps/nep-0491.md) | Non-Refundable Storage Staking | @jakmeier | Final | +| [0492](https://github.com/near/NEPs/blob/master/neps/nep-0492.md) | Restrict creation of Ethereum Addresses | @bowenwang1996 | Final | +| [0508](https://github.com/near/NEPs/blob/master/neps/nep-0508.md) | Resharding v2 | @wacban @shreyan-gupta @walnut-the-cat | Final | +| [0509](https://github.com/near/NEPs/blob/master/neps/nep-0509.md) | Stateless validation Stage 0 | @robin-near @pugachAG @Longarithm @walnut-the-cat | Final | +| [0514](https://github.com/near/NEPs/blob/master/neps/nep-0514.md) | Fewer Block Producer Seats in `testnet` | @nikurt | Final | +| [0519](https://github.com/near/NEPs/blob/master/neps/nep-0519.md) | Yield Execution | @akhi3030 @saketh-are | Final | +| [0536](https://github.com/near/NEPs/blob/master/neps/nep-0536.md) | Reduce the number of gas refunds | @evgenykuzyakov @bowenwang1996 | Final | +| [0539](https://github.com/near/NEPs/blob/master/neps/nep-0539.md) | Cross-Shard Congestion Control | @wacban @jakmeier | Final | ## Specification diff --git a/neps/nep-0021.md b/neps/nep-0021.md index ab45c49c4..d7a1bb29d 100644 --- a/neps/nep-0021.md +++ b/neps/nep-0021.md @@ -2,8 +2,8 @@ NEP: 21 Title: Fungible Token Standard Author: Evgeny Kuzyakov -DiscussionsTo: https://github.com/near/NEPs/pull/21 Status: Final +DiscussionsTo: https://github.com/near/NEPs/pull/21 Type: Standards Track Category: Contract Created: 29-Oct-2019 diff --git a/neps/nep-0141.md b/neps/nep-0141.md index bdc5d35fb..03e1c570f 100644 --- a/neps/nep-0141.md +++ b/neps/nep-0141.md @@ -2,8 +2,8 @@ NEP: 141 Title: Fungible Token Standard Author: Evgeny Kuzyakov , Robert Zaremba <@robert-zaremba>, @oysterpack -DiscussionsTo: https://github.com/near/NEPs/issues/141 Status: Final +DiscussionsTo: https://github.com/near/NEPs/issues/141 Type: Standards Track Category: Contract Created: 03-Mar-2022 diff --git a/neps/nep-0145.md b/neps/nep-0145.md index c6fd70f74..821c7fda2 100644 --- a/neps/nep-0145.md +++ b/neps/nep-0145.md @@ -2,8 +2,8 @@ NEP: 145 Title: Storage Management Author: Evgeny Kuzyakov , @oysterpack -DiscussionsTo: https://github.com/near/NEPs/discussions/145 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/145 Type: Standards Track Category: Contract Created: 03-Mar-2022 diff --git a/neps/nep-0148.md b/neps/nep-0148.md index 1bcc59692..1534ae1b6 100644 --- a/neps/nep-0148.md +++ b/neps/nep-0148.md @@ -2,8 +2,8 @@ NEP: 148 Title: Fungible Token Metadata Author: Robert Zaremba , Evgeny Kuzyakov , @oysterpack -DiscussionsTo: https://github.com/near/NEPs/discussions/148 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/148 Type: Standards Track Category: Contract Created: 03-Mar-2022 diff --git a/neps/nep-0171.md b/neps/nep-0171.md index 8b0990319..31573ac9c 100644 --- a/neps/nep-0171.md +++ b/neps/nep-0171.md @@ -2,8 +2,8 @@ NEP: 171 Title: Non Fungible Token Standard Author: Mike Purvis , Evgeny Kuzyakov , @oysterpack -DiscussionsTo: https://github.com/near/NEPs/discussions/171 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/171 Type: Standards Track Category: Contract Version: 1.2.0 @@ -391,7 +391,7 @@ Note that the example events above cover two different kinds of events: 2. An event that has a relevant trigger function [NFT Core Standard](https://nomicon.io/Standards/NonFungibleToken/Core.html#nft-interface) (`nft_transfer`) This event standard also applies beyond the events highlighted here, where future events follow the same convention of as the second type. For instance, if an NFT contract uses the [approval management standard](https://nomicon.io/Standards/NonFungibleToken/ApprovalManagement.html), it may emit an event for `nft_approve` if that's deemed as important by the developer community. - + Please feel free to open pull requests for extending the events standard detailed here as needs arise. @@ -441,7 +441,7 @@ The extension NEP-0469 that added Token Metadata Update Event Kind to this NEP-0 #### Concerns | # | Concern | Resolution | Status | -| - | - | - | - | +| - | - | - | - | | 1 | Ecosystem will be split where legacy contracts won't emit these new events, so legacy support will still be needed | In the future, there will be fewer legacy contracts and eventually apps will have support for this type of event | Resolved | | 2 | `nft_update` event name is ambiguous | It was decided to use `nft_metadata_update` name, instead | Resolved | diff --git a/neps/nep-0177.md b/neps/nep-0177.md index 2c3969ff0..66ea8b436 100644 --- a/neps/nep-0177.md +++ b/neps/nep-0177.md @@ -2,8 +2,8 @@ NEP: 177 Title: Non Fungible Token Metadata Author: Chad Ostrowski <@chadoh>, Mike Purvis -DiscussionsTo: https://github.com/near/NEPs/discussions/177 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/177 Type: Standards Track Category: Contract Created: 03-Mar-2022 diff --git a/neps/nep-0178.md b/neps/nep-0178.md index cfbc49398..7021c8f23 100644 --- a/neps/nep-0178.md +++ b/neps/nep-0178.md @@ -2,8 +2,8 @@ NEP: 178 Title: Non Fungible Token Approval Management Author: Chad Ostrowski <@chadoh>, Thor <@thor314> -DiscussionsTo: https://github.com/near/NEPs/discussions/178 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/178 Type: Standards Track Category: Contract Created: 03-Mar-2022 diff --git a/neps/nep-0181.md b/neps/nep-0181.md index 716792500..89e44243f 100644 --- a/neps/nep-0181.md +++ b/neps/nep-0181.md @@ -2,8 +2,8 @@ NEP: 181 Title: Non Fungible Token Enumeration Author: Chad Ostrowski <@chadoh>, Thor <@thor314> -DiscussionsTo: https://github.com/near/NEPs/discussions/181 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/181 Type: Standards Track Category: Contract Created: 03-Mar-2022 @@ -78,7 +78,7 @@ function nft_tokens_for_owner( ## Notes -At the time of this writing, the specialized collections in the `near-sdk` Rust crate are iterable, but not all of them have implemented an `iter_from` solution. There may be efficiency gains for large collections and contract developers are encouraged to test their data structures with a large amount of entries. +At the time of this writing, the specialized collections in the `near-sdk` Rust crate are iterable, but not all of them have implemented an `iter_from` solution. There may be efficiency gains for large collections and contract developers are encouraged to test their data structures with a large amount of entries. ## Reference Implementation diff --git a/neps/nep-0199.md b/neps/nep-0199.md index 2e1322d7a..4d7cdeb4f 100644 --- a/neps/nep-0199.md +++ b/neps/nep-0199.md @@ -2,8 +2,8 @@ NEP: 199 Title: Non Fungible Token Royalties and Payouts Author: Thor <@thor314>, Matt Lockyer <@mattlockyer> -DiscussionsTo: https://github.com/near/NEPs/discussions/199 Status: Final +DiscussionsTo: https://github.com/near/NEPs/discussions/199 Type: Standards Track Category: Contract Created: 03-Mar-2022 @@ -116,7 +116,7 @@ NFT and financial contracts vary in implementation. This means that some extra C ## Drawbacks -There is an introduction of trust that the contract calling `nft_transfer_payout` will indeed pay out to all intended parties. However, since the calling contract will typically be something like a marketplace used by end users, malicious actors might be found out more easily and might have less incentive. +There is an introduction of trust that the contract calling `nft_transfer_payout` will indeed pay out to all intended parties. However, since the calling contract will typically be something like a marketplace used by end users, malicious actors might be found out more easily and might have less incentive. There is an assumption that NFT contracts will understand the limits of gas and not allow for a number of payouts that cannot be achieved. ## Future possibilities diff --git a/neps/nep-0245.md b/neps/nep-0245.md index 82c11c5ae..82d1d3d48 100644 --- a/neps/nep-0245.md +++ b/neps/nep-0245.md @@ -1,9 +1,9 @@ --- -NEP: 245 +NEP: 245 Title: Multi Token Standard Author: Zane Starr , @riqi, @jriemann, @marcos.sun +Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/246 -Status: Review Type: Standards Track Category: Contract Created: 03-Mar-2022 @@ -17,11 +17,11 @@ A standard interface for a multi token standard that supports fungible, semi-fun ## Motivation -In the three years since [ERC-1155] was ratified by the Ethereum Community, Multi Token based contracts have proven themselves valuable assets. Many blockchain projects emulate this standard for representing multiple token assets classes in a single contract. The ability to reduce transaction overhead for marketplaces, video games, DAOs, and exchanges is appealing to the blockchain ecosystem and simplifies transactions for developers. +In the three years since [ERC-1155] was ratified by the Ethereum Community, Multi Token based contracts have proven themselves valuable assets. Many blockchain projects emulate this standard for representing multiple token assets classes in a single contract. The ability to reduce transaction overhead for marketplaces, video games, DAOs, and exchanges is appealing to the blockchain ecosystem and simplifies transactions for developers. Having a single contract represent NFTs, FTs, and tokens that sit inbetween greatly improves efficiency. The standard also introduced the ability to make batch requests with multiple asset classes reducing complexity. This standard allows operations that currently require _many_ transactions to be completed in a single transaction that can transfer not only NFTs and FTs, but any tokens that are a part of same token contract. -With this standard, we have sought to take advantage of the ability of the NEAR blockchain to scale. Its sharded runtime, and [storage staking] model that decouples [gas] fees from storage demand, enables ultra low transaction fees and greater on chain storage (see [Metadata][MT Metadata] extension). +With this standard, we have sought to take advantage of the ability of the NEAR blockchain to scale. Its sharded runtime, and [storage staking] model that decouples [gas] fees from storage demand, enables ultra low transaction fees and greater on chain storage (see [Metadata][MT Metadata] extension). With the aforementioned, it is noteworthy to mention that like the [NFT] standard the Multi Token standard, implements `mt_transfer_call`, which allows, a user to attach many tokens to a call to a separate contract. Additionally, this standard includes an optional [Approval Management] extension. The extension allows marketplaces to trade on behalf of a user, providing additional flexibility for dApps. @@ -39,24 +39,24 @@ Why have another standard, aren't fungible and non-fungible tokens enough? The The standard here introduces a few concepts that evolve the original [ERC-1155] standard to have more utility, while maintaining the original flexibility of the standard. So keeping that in mind, we are defining this as a new token type. It combines two main features of FT and NFT. It allows us to represent many token types in a single contract, and it's possible to store the amount for each token. -The decision to not use FT and NFT as explicit token types was taken to allow the community to define their own standards and meanings through metadata. As standards evolve on other networks, this specification allows the standard to be able to represent tokens across networks accurately, without necessarily restricting the behavior to any preset definition. +The decision to not use FT and NFT as explicit token types was taken to allow the community to define their own standards and meanings through metadata. As standards evolve on other networks, this specification allows the standard to be able to represent tokens across networks accurately, without necessarily restricting the behavior to any preset definition. -The issues with this in general is a problem with defining what metadata means and how is that interpreted. We have chosen to follow the pattern that is currently in use on Ethereum in the [ERC-1155] standard. That pattern relies on people to make extensions or to make signals as to how they want the metadata to be represented for their use case. +The issues with this in general is a problem with defining what metadata means and how is that interpreted. We have chosen to follow the pattern that is currently in use on Ethereum in the [ERC-1155] standard. That pattern relies on people to make extensions or to make signals as to how they want the metadata to be represented for their use case. One of the areas that has broad sweeping implications from the [ERC-1155] standard is the lack of direct access to metadata. With Near's sharding we are able to have a [Metadata Extension][MT Metadata] for the standard that exists on chain. So developers and users are not required to use an indexer to understand, how to interact or interpret tokens, via token identifiers that they receive. -Another extension that we made was to provide an explicit ability for developers and users to group or link together series of NFTs/FTs or any combination of tokens. This provides additional flexibility that the [ERC-1155] standard only has loose guidelines on. This was chosen to make it easy for consumers to understand the relationship between tokens within the contract. +Another extension that we made was to provide an explicit ability for developers and users to group or link together series of NFTs/FTs or any combination of tokens. This provides additional flexibility that the [ERC-1155] standard only has loose guidelines on. This was chosen to make it easy for consumers to understand the relationship between tokens within the contract. -To recap, we choose to create this standard, to improve interoperability, developer ease of use, and to extend token representability beyond what was available directly in the FT or NFT standards. We believe this to be another tool in the developer's toolkit. It makes it possible to represent many types of tokens and to enable exchanges of many tokens within a single `transaction`. +To recap, we choose to create this standard, to improve interoperability, developer ease of use, and to extend token representability beyond what was available directly in the FT or NFT standards. We believe this to be another tool in the developer's toolkit. It makes it possible to represent many types of tokens and to enable exchanges of many tokens within a single `transaction`. -## Specification +## Specification **NOTES**: - All amounts, balances and allowance are limited by `U128` (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. -- The contract must track the change in storage when adding to and removing from collections. This is not included in this core multi token standard but instead in the [Storage Standard][Storage Management]. +- The contract must track the change in storage when adding to and removing from collections. This is not included in this core multi token standard but instead in the [Storage Standard][Storage Management]. - To prevent the deployed contract from being modified or deleted, it should not have any access keys on its account. ### MT Interface @@ -93,11 +93,11 @@ type Token = { // * `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. -// * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. -// `owner_id` is the valid Near account that owns the tokens. +// * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. +// `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management -// standard for full explanation. +// standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer @@ -131,14 +131,14 @@ function mt_transfer( // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like an array of strings, although the numbers will be stored as an array of unsigned integer // with 128 bits. -// * `approvals` (optional): is an array of expected `approval` per `token_ids`. -// If a `token_id` does not have a corresponding `approval` then the entry in the array +// * `approvals` (optional): is an array of expected `approval` per `token_ids`. +// If a `token_id` does not have a corresponding `approval` then the entry in the array // must be marked null. -// `approval` is a tuple of [`owner_id`,`approval_id`]. -// `owner_id` is the valid Near account that owns the tokens. +// `approval` is a tuple of [`owner_id`,`approval_id`]. +// `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management -// standard for full explanation. +// standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer @@ -182,8 +182,8 @@ function mt_batch_transfer( // like a string, although the number will be stored as an unsigned integer // with 128 bits. // * `owner_id`: the valid NEAR account that owns the token -// * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. -// `owner_id` is the valid Near account that owns the tokens. +// * `approval` (optional): is a tuple of [`owner_id`,`approval_id`]. +// `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management // * `memo` (optional): for use cases that may benefit from indexing or @@ -234,15 +234,15 @@ function mt_transfer_call( // * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like an array of string, although the numbers will be stored as an array of -// unsigned integer with 128 bits. -// * `approvals` (optional): is an array of expected `approval` per `token_ids`. -// If a `token_id` does not have a corresponding `approval` then the entry in the array +// unsigned integer with 128 bits. +// * `approvals` (optional): is an array of expected `approval` per `token_ids`. +// If a `token_id` does not have a corresponding `approval` then the entry in the array // must be marked null. -// `approval` is a tuple of [`owner_id`,`approval_id`]. -// `owner_id` is the valid Near account that owns the tokens. +// `approval` is a tuple of [`owner_id`,`approval_id`]. +// `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management -// standard for full explanation. +// standard for full explanation. // * `memo` (optional): for use cases that may benefit from indexing or // providing information for a transfer. // * `msg`: specifies information needed by the receiving contract in @@ -267,16 +267,16 @@ function mt_batch_transfer_call( // Returns the tokens with the given `token_ids` or `null` if no such token. function mt_token(token_ids: string[]) (Token | null)[] -// Returns the balance of an account for the given `token_id`. -// The balance though wrapped in quotes and treated like a string, +// Returns the balance of an account for the given `token_id`. +// The balance though wrapped in quotes and treated like a string, // the number will be stored as an unsigned integer with 128 bits. // Arguments: // * `account_id`: the NEAR account that owns the token. // * `token_id`: the token to retrieve the balance from function mt_balance_of(account_id: string, token_id: string): string -// Returns the balances of an account for the given `token_ids`. -// The balances though wrapped in quotes and treated like strings, +// Returns the balances of an account for the given `token_ids`. +// The balances though wrapped in quotes and treated like strings, // the numbers will be stored as an unsigned integer with 128 bits. // Arguments: // * `account_id`: the NEAR account that owns the tokens. @@ -284,12 +284,12 @@ function mt_balance_of(account_id: string, token_id: string): string function mt_batch_balance_of(account_id: string, token_ids: string[]): string[] // Returns the token supply with the given `token_id` or `null` if no such token exists. -// The supply though wrapped in quotes and treated like a string, the number will be stored +// The supply though wrapped in quotes and treated like a string, the number will be stored // as an unsigned integer with 128 bits. -function mt_supply(token_id: string): string | null +function mt_supply(token_id: string): string | null -// Returns the token supplies with the given `token_ids`, a string value is returned or `null` -// if no such token exists. The supplies though wrapped in quotes and treated like strings, +// Returns the token supplies with the given `token_ids`, a string value is returned or `null` +// if no such token exists. The supplies though wrapped in quotes and treated like strings, // the numbers will be stored as an unsigned integer with 128 bits. function mt_batch_supply(token_ids: string[]): (string | null)[] ``` @@ -297,7 +297,7 @@ function mt_batch_supply(token_ids: string[]): (string | null)[] The following behavior is required, but contract authors may name this function something other than the conventional `mt_resolve_transfer` used here. ```ts -// Finalize an `mt_transfer_call` or `mt_batch_transfer_call` chain of cross-contract calls. Generically +// Finalize an `mt_transfer_call` or `mt_batch_transfer_call` chain of cross-contract calls. Generically // referred to as `mt_transfer_call` as it applies to `mt_batch_transfer_call` as well. // // The `mt_transfer_call` process: @@ -323,19 +323,19 @@ The following behavior is required, but contract authors may name this function // * `approvals (optional)`: if using Approval Management, contract MUST provide // set of original approvals in this argument, and restore the // approved accounts in case of revert. -// `approvals` is an array of expected `approval_list` per `token_ids`. -// If a `token_id` does not have a corresponding `approvals_list` then the entry in the +// `approvals` is an array of expected `approval_list` per `token_ids`. +// If a `token_id` does not have a corresponding `approvals_list` then the entry in the // array must be marked null. -// `approvals_list` is an array of triplets of [`owner_id`,`approval_id`,`amount`]. -// `owner_id` is the valid Near account that owns the tokens. +// `approvals_list` is an array of triplets of [`owner_id`,`approval_id`,`amount`]. +// `owner_id` is the valid Near account that owns the tokens. // `approval_id` is the expected approval ID. A number smaller than // 2^53, and therefore representable as JSON. See Approval Management -// standard for full explanation. +// standard for full explanation. // `amount`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the number will be stored as an unsigned integer // with 128 bits. -// -// +// +// // // Returns total amount spent by the `receiver_id`, corresponding to the `token_id`. // The amounts returned, though wrapped in quotes and treated like strings, @@ -361,7 +361,7 @@ Contracts which want to make use of `mt_transfer_call` and `mt_batch_transfer_ca // Take some action after receiving a multi token // // Requirements: -// * Contract MUST restrict calls to this function to a set of whitelisted +// * Contract MUST restrict calls to this function to a set of whitelisted // contracts // * Contract MUST panic if `token_ids` length does not equals `amounts` // length @@ -379,8 +379,8 @@ Contracts which want to make use of `mt_transfer_call` and `mt_batch_transfer_ca // request. This may include method names and/or arguments. // // Returns the number of unused tokens in string form. For instance, if `amounts` -// is `["10"]` but only 9 are needed, it will return `["1"]`. The amounts returned, -// though wrapped in quotes and treated like strings, the numbers will be stored as +// is `["10"]` but only 9 are needed, it will return `["1"]`. The amounts returned, +// though wrapped in quotes and treated like strings, the numbers will be stored as // an unsigned integer with 128 bits. @@ -393,7 +393,7 @@ function mt_on_transfer( ): Promise; ``` -## Events +## Events NEAR and third-party applications need to track `mint`, `burn`, `transfer` events for all MT-driven apps consistently. This exension addresses that. @@ -418,11 +418,11 @@ interface MtEventLogData { ``` ```ts -// Minting event log. Emitted when a token is minted/created. +// Minting event log. Emitted when a token is minted/created. // Requirements // * Contract MUST emit event when minting a token -// Fields -// * Contract token_ids and amounts MUST be the same length +// Fields +// * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account receiving the minted token // * `token_ids`: the tokens minted // * `amounts`: the number of tokens minted, wrapped in quotes and treated @@ -436,11 +436,11 @@ interface MtMintLog { memo?: string } -// Burning event log. Emitted when a token is burned. +// Burning event log. Emitted when a token is burned. // Requirements // * Contract MUST emit event when minting a token -// Fields -// * Contract token_ids and amounts MUST be the same length +// Fields +// * Contract token_ids and amounts MUST be the same length // * `owner_id`: the account whose token(s) are being burned // * `authorized_id`: approved account_id to burn, if applicable // * `token_ids`: the tokens being burned @@ -456,14 +456,14 @@ interface MtBurnLog { memo?: string } -// Transfer event log. Emitted when a token is transferred. +// Transfer event log. Emitted when a token is transferred. // Requirements // * Contract MUST emit event when transferring a token -// Fields +// Fields // * `authorized_id`: approved account_id to transfer // * `old_owner_id`: the account sending the tokens "sender.near" // * `new_owner_id`: the account receiving the tokens "receiver.near" -// * `token_ids`: the tokens to transfer +// * `token_ids`: the tokens to transfer // * `amounts`: the number of tokens to transfer, wrapped in quotes and treated // like a string, although the numbers will be stored as an unsigned integer // array with 128 bits. diff --git a/neps/nep-0264.md b/neps/nep-0264.md index e239a4b41..cbde6521c 100644 --- a/neps/nep-0264.md +++ b/neps/nep-0264.md @@ -2,8 +2,8 @@ NEP: 264 Title: Utilization of unspent gas for promise function calls Authors: Austin Abell +Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/264 -Status: Approved Type: Protocol Version: 1.0.0 Created: 2021-09-30 @@ -20,7 +20,7 @@ We are proposing this to be able to utilize gas more efficiently but also to imp # Guide-level explanation -This host function is similar to [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526), except with an additional parameter that lets you specify how much of the excess gas should be attached to the function call. This parameter is a weight value that determines how much of the excess gas is attached to each function. +This host function is similar to [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526), except with an additional parameter that lets you specify how much of the excess gas should be attached to the function call. This parameter is a weight value that determines how much of the excess gas is attached to each function. So, for example, if there is 40 gas leftover and three function calls that select weights of 1, 5, and 2, the runtime will add 5, 25, and 10 gas to each function call. A developer can specify whether they want to attach a fixed amount of gas, a weight of remaining gas, or both. If at least one function call uses a weight of remaining gas, then all excess gas will be attached to future calls. This proposal allows developers the ability to utilize prepaid gas more efficiently than currently possible. @@ -71,7 +71,7 @@ This host function definition would look like this (as a Rust consumer): The only difference from the existing API is `gas_weight` added as another parameter, as an unsigned 64-bit integer. -As for calculations, the remaining gas at the end of the transaction can be floor divided by the sum of all the weights tracked. Then, after getting this value, just attach that value multiplied by the weight gas to each function call action. +As for calculations, the remaining gas at the end of the transaction can be floor divided by the sum of all the weights tracked. Then, after getting this value, just attach that value multiplied by the weight gas to each function call action. For example, if there are three weights, `a`, `b`, `c`: @@ -80,7 +80,7 @@ weight_sum = a + b + c a_gas += remaining_gas * a / weight_sum b_gas += remaining_gas * b / weight_sum c_gas += remaining_gas * c / weight_sum -``` +``` Any remaining gas that is not allocated to any of these function calls will be attached to the last function call scheduled. @@ -127,7 +127,7 @@ The primary alternative is using a numerator and denominator to represent a frac Pros: -- Can under-utilize the gas for the current transaction to limit gas allowed for certain functions +- Can under-utilize the gas for the current transaction to limit gas allowed for certain functions - This could take responsibility away from DApp users because they would not have to worry less about attaching too much prepaid gas - Thinking in terms of fractions may be more intuitive for some developers - Might future proof better if we ever need this ability in the future, want to minimize the number of host functions created at all costs @@ -158,7 +158,7 @@ Cons: # Unresolved questions -What needs to be addressed before this gets merged: +What needs to be addressed before this gets merged: ~~- How much refactoring exactly is needed to handle this pattern?~~ ~~- Can we keep a queue of receipt and action indices with their respective weights and update their gas values after the current method is executed? Is there a cleaner way to handle this while keeping order?~~ ~~- Do we want to attach the gas lost due to precision on division to any function?~~ diff --git a/neps/nep-0297.md b/neps/nep-0297.md index b4de6cd37..6b58a42e4 100644 --- a/neps/nep-0297.md +++ b/neps/nep-0297.md @@ -2,8 +2,8 @@ NEP: 297 Title: Events Author: Olga Telezhnaya -DiscussionsTo: https://github.com/near/NEPs/issues/297 Status: Final +DiscussionsTo: https://github.com/near/NEPs/issues/297 Type: Standards Track Category: Contract Created: 03-Mar-2022 @@ -37,7 +37,7 @@ Many apps use different interfaces that represent the same action. This interface standardizes that process by introducing event logs. Events use the standard logs capability of NEAR. -Events are log entries that start with the `EVENT_JSON:` prefix followed by a single valid JSON string. +Events are log entries that start with the `EVENT_JSON:` prefix followed by a single valid JSON string. JSON string may have any number of space characters in the beginning, the middle, or the end of the string. It's guaranteed that space characters do not break its parsing. All the examples below are pretty-formatted for better readability. diff --git a/neps/nep-0330.md b/neps/nep-0330.md index 6615165c4..ffb11993b 100644 --- a/neps/nep-0330.md +++ b/neps/nep-0330.md @@ -2,8 +2,8 @@ NEP: 330 Title: Source Metadata Author: Ben Kurrek , Osman Abdelnasir , Andrey Gruzdev <@canvi>, Alexey Zenin <@alexthebuildr> +Status: Final DiscussionsTo: https://github.com/near/NEPs/discussions/329 -Status: Approved Type: Standards Track Category: Contract Version: 1.2.0 @@ -58,7 +58,7 @@ type Standard { type BuildInfo { build_environment: string, // reference to a reproducible build environment docker image, e.g., "docker.io/sourcescan/cargo-near@sha256:bf488476d9c4e49e36862bbdef2c595f88d34a295fd551cc65dc291553849471" or something else pointing to the build environment. source_code_snapshot: string, // reference to the source code snapshot that was used to build the contract, e.g., "git+https://github.com/near/cargo-near-new-project-template.git#9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420" or "ipfs://". - contract_path: string|null, // relative path to contract crate within the source code, e.g., "contracts/contract-one". Often, it is the root of the repository, so can be omitted. + contract_path: string|null, // relative path to contract crate within the source code, e.g., "contracts/contract-one". Often, it is the root of the repository, so can be omitted. build_command: string[], // the exact command that was used to build the contract, with all the flags, e.g., ["cargo", "near", "build", "--no-abi"]. } ``` @@ -118,7 +118,7 @@ Calling the view function `contract_source_metadata`, the contract would return: link: "https://github.com/near/cargo-near-new-project-template/tree/9c16aaff3c0fe5bda4d8ffb418c4bb2b535eb420", standards: [ { - standard: "nep330", + standard: "nep330", version: "1.1.0" } ], @@ -149,11 +149,11 @@ pub struct Standard { } /// BuildInfo structure -pub struct BuildInfo { +pub struct BuildInfo { pub build_environment: String, pub source_code_snapshot: String, - pub contract_path: Option, - pub build_command: Vec, + pub contract_path: Option, + pub build_command: Vec, } /// Contract metadata structure @@ -201,7 +201,7 @@ The extension NEP-351 that added Contract Metadata to this NEP-330 was approved #### Concerns | # | Concern | Resolution | Status | -| - | - | - | - | +| - | - | - | - | | 1 | Integer field as a standard reference is limiting as third-party projects may want to introduce their own standards without pushing it through the NEP process | Author accepted the proposed string-value standard reference (e.g. “nep123” instead of just 123, and allow “xyz001” as previously it was not possible to express it) | Resolved | | 2 | NEP-330 and NEP-351 should be included in the list of the supported NEPs | There seems to be a general agreement that it is a good default, so NEP was updated | Resolved | | 3 | JSON Event could be beneficial, so tooling can react to the changes in the supported standards | It is outside the scope of this NEP. Also, list of supported standards only changes with contract re-deployment, so tooling can track DEPLOY_CODE events and check the list of supported standards when new code is deployed | Won’t fix | diff --git a/neps/nep-0364.md b/neps/nep-0364.md index 3863cb38f..52e4c5779 100644 --- a/neps/nep-0364.md +++ b/neps/nep-0364.md @@ -2,8 +2,8 @@ NEP: 364 Title: Efficient signature verification and hashing precompile functions Author: Blas Rodriguez Irizar +Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/364 -Status: Draft Type: Runtime Spec Category: Contract Created: 15-Jun-2022 diff --git a/neps/nep-0366.md b/neps/nep-0366.md index 5aada7b0a..c5a203c22 100644 --- a/neps/nep-0366.md +++ b/neps/nep-0366.md @@ -2,8 +2,8 @@ NEP: 366 Title: Meta Transactions Author: Illia Polosukhin , Egor Uleyskiy (egor.ulieiskii@gmail.com), Alexander Fadeev (fadeevab.com@gmail.com) +Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/366 -Status: Approved Type: Protocol Track Category: Runtime Version: 1.1.0 diff --git a/neps/nep-0393.md b/neps/nep-0393.md index c414405ba..82f73ffe7 100644 --- a/neps/nep-0393.md +++ b/neps/nep-0393.md @@ -2,16 +2,14 @@ NEP: 393 Title: Soulbound Token Authors: Robert Zaremba <@robert-zaremba> +Status: Final DiscussionsTo: -Status: Approved Type: Standards Track Category: Contract Created: 12-Sep-2022 Requires: --- -# NEP: Soulbound Token - ## Summary Soulbound Token (SBT) is a form of a non-fungible token which represents an aspect of an account: _soul_. [Transferability](#transferability) is limited only to a case of recoverability or a _soul transfer_. The latter must coordinate with a registry to transfer all SBTs from one account to another, and _banning_ the source account. @@ -300,7 +298,7 @@ pub struct ClassMetadata { pub symbol: Option, /// An URL to an Icon. To protect fellow developers from unintentionally triggering any /// SSRF vulnerabilities with URL parsers, we don't allow to set an image bytes here. - /// If it doesn't start with a scheme (eg: https://) then `IssuerMetadata::base_uri` + /// If it doesn't start with a scheme (eg: https://) then `IssuerMetadata::base_uri` /// should be prepended. pub icon: Option, /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme diff --git a/neps/nep-0399.md b/neps/nep-0399.md index 77256b0e8..6dc21b25b 100644 --- a/neps/nep-0399.md +++ b/neps/nep-0399.md @@ -1,9 +1,9 @@ --- -NEP: 0399 +NEP: 399 Title: Flat Storage Author: Aleksandr Logunov Min Zhang +Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/0399 -Status: Draft Type: Protocol Track Category: Storage Created: 30-Sep-2022 @@ -124,7 +124,7 @@ will never be finalized. As a result, if we use the last final block as the flat FlatStorage needs to process is a descendant of the flat head. To support key value lookups for other blocks that are not the flat head, FlatStorage will -store key value changes(deltas) per block for these blocks. +store key value changes(deltas) per block for these blocks. We call these deltas FlatStorageDelta (FSD). Let’s say the flat storage head is at block h, and we are applying transactions based on block h’. Since h is the last final block, h is an ancestor of h'. To access the state at block h', we need FSDs of all blocks between h and h'. diff --git a/neps/nep-0413.md b/neps/nep-0413.md index ef6230fc5..810bc3a89 100644 --- a/neps/nep-0413.md +++ b/neps/nep-0413.md @@ -2,8 +2,8 @@ NEP: 413 Title: Near Wallet API - support for signMessage method Author: Philip Obosi , Guillermo Gallardo +Status: Final # DiscussionsTo: -Status: Approved Type: Standards Track Category: Wallet Created: 25-Oct-2022 diff --git a/neps/nep-0418.md b/neps/nep-0418.md index 9c6f9ed96..602ee3495 100644 --- a/neps/nep-0418.md +++ b/neps/nep-0418.md @@ -1,9 +1,9 @@ --- -NEP: 0418 +NEP: 418 Title: Remove attached_deposit view panic Author: Austin Abell +Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/418 -Status: Approved Type: Standards Track Category: Tools Version: 1.0.0 @@ -25,7 +25,7 @@ Initial discussion: https://near.zulipchat.com/#narrow/stream/295306-pagoda.2Fco ## Rationale and alternatives -The rationale for assigning `0u128` to the pointer (`u64`) passed into `attached_deposit` is that it's the least breaking change. +The rationale for assigning `0u128` to the pointer (`u64`) passed into `attached_deposit` is that it's the least breaking change. The alternative of returning some special value, say `u128::MAX`, is that it would cause some unintended side effects for view calls using the `attached_deposit`. For example, if `attached_deposit` is called within a function, older versions of a contract that do not check the special value will return a result assuming that the attached deposit is `u128::MAX`. This is not a large concern since it would just be a view call, but it might be a bad UX in some edge cases, where returning 0 wouldn't be an issue. diff --git a/neps/nep-0448.md b/neps/nep-0448.md index d7d2856fd..fb7781487 100644 --- a/neps/nep-0448.md +++ b/neps/nep-0448.md @@ -2,8 +2,8 @@ NEP: 448 Title: Zero-balance Accounts Author: Bowen Wang -DiscussionsTo: https://github.com/nearprotocol/neps/pull/448 Status: Final +DiscussionsTo: https://github.com/nearprotocol/neps/pull/448 Type: Protocol Track Created: 10-Jan-2023 --- diff --git a/neps/nep-0452.md b/neps/nep-0452.md index c5df9f258..7df946b7b 100644 --- a/neps/nep-0452.md +++ b/neps/nep-0452.md @@ -2,8 +2,8 @@ NEP: 452 Title: Linkdrop Standard Author: Ben Kurrek , Ken Miyachi -DiscussionsTo: https://gov.near.org/t/official-linkdrop-standard/32463/1 Status: Final +DiscussionsTo: https://gov.near.org/t/official-linkdrop-standard/32463/1 Type: Standards Track Category: Contract Version: 1.0.0 diff --git a/neps/nep-0455.md b/neps/nep-0455.md index 5cd9bddfc..893c68a75 100644 --- a/neps/nep-0455.md +++ b/neps/nep-0455.md @@ -2,8 +2,8 @@ NEP: 455 Title: Parameter Compute Costs Author: Andrei Kashin , Jakob Meier -DiscussionsTo: https://github.com/nearprotocol/neps/pull/455 Status: Final +DiscussionsTo: https://github.com/nearprotocol/neps/pull/455 Type: Protocol Track Category: Runtime Created: 26-Jan-2023 @@ -259,7 +259,7 @@ Progress on this work is tracked here: https://github.com/near/nearcore/issues/8 #### Benefits - Among the alternatives, this is the easiest to implement. -- It allows us to able to publicly discuss undercharging issues before they are fixed. +- It allows us to able to publicly discuss undercharging issues before they are fixed. #### Concerns diff --git a/neps/nep-0488.md b/neps/nep-0488.md new file mode 100644 index 000000000..16ea66184 --- /dev/null +++ b/neps/nep-0488.md @@ -0,0 +1,1212 @@ +--- +NEP: 488 +Title: Host Functions for BLS12-381 Curve Operations +Authors: Olga Kuniavskaia +Status: Final +DiscussionsTo: https://github.com/nearprotocol/neps/pull/488 +Type: Runtime Spec +Version: 0.0.1 +Created: 2023-07-17 +LastUpdated: 2023-11-21 +--- + +## Summary + +This NEP introduces host functions to perform operations on the BLS12-381 elliptic curve. It is a minimal set of functions needed to efficiently verify BLS signatures and zkSNARKs. + +## Motivation + +The primary aim of this NEP is to enable fast and efficient verification of BLS signatures and zkSNARKs based on the BLS12-381[^1],[^11],[^52] elliptic curve on NEAR. + +To efficiently verify zkSNARKs[^19], host functions for operations on the BN254 +elliptic curve (also known as Alt-BN128)[^9], [^12] have already been implemented on NEAR[^10]. +For instance, the Zeropool[^20] project utilizes these host functions for verifying zkSNARKs on NEAR. +However, recent research shows that the BN254 security level is lower than 100-bit[^13] and it is not recommended for use. +BLS12-381, on the other hand, offers over 120 bits of security[^8] and is widely used[^2],[^3],[^4],[^5],[^6],[^7] as a robust alternative. +Supporting operations for BLS12-381 elliptic curve will significantly enhance the security of projects similar to Zeropool. + +Another crucial objective is the verification of BLS signatures. +Initially, host functions for BN254 on NEAR were designed for zkSNARK verification and +are insufficient for BLS signature verification. +However, even if these host functions were sufficient for BLS signature verification on the BN254 elliptic curve, this would not be enough for compatibility with other projects. +In particular, projects such as ZCash[^2], Ethereum[^3], Tezos[^5], and Filecoin[^6] incorporate BLS12-381 specifically within their protocols. +If we aim for compatibility with these projects, we must also utilize this elliptic curve. +For instance, to create a trustless bridge[^17] between Ethereum and NEAR, +we must efficiently verify BLS signatures based on BLS12-381, as these are the signatures employed within Ethereum's protocol. + +In this NEP, we propose to add the following host functions: + +- ***bls12381_p1_sum —*** computes the sum of signed points from $E(F_p)$ elliptic curve. This function is useful for aggregating public keys or signatures in the BLS signature scheme. It can be employed for simple addition in $E(F_p)$. It is kept separate from the `multiexp` function due to gas cost considerations. +- ***bls12381_p2_sum —*** computes the sum of signed points from $E'(F_{p^2})$ elliptic curve. This function is useful for aggregating signatures or public keys in the BLS signature scheme. +- ***bls12381_g1_multiexp —*** calculates $\sum p_i s_i$ for points $p_i \in G_1 \subset E(F_p)$ and scalars $s_i$. This operation can be used to multiply a group element by a scalar. +- ***bls12381_g2_multiexp —*** calculates $\sum p_i s_i$ for points $p_i \in G_2 \subset E'(F_{p^2})$ and scalars $s_i$. +- ***bls12381_map_fp_to_g1 —*** maps base field elements into $G_1$ points. It does not perform the mapping of byte strings into field elements. +- ***bls12381_map_fp2_to_g2 —*** maps extension field elements into $G_2$ points. This function does not perform the mapping of byte strings into extension field elements, which would be needed to efficiently map a message into a group element. We are not implementing the `hash_to_field`[^60] function because the latter can be executed within a contract and various hashing algorithms can be used within this function. +- ***bls12381_p1_decompress —*** decompresses points from $E(F_p)$ provided in a compressed form. Certain protocols offer points on the curve in a compressed form (e.g., the light client updates in Ethereum 2.0), and decompression is a time-consuming operation. All the other functions in this NEP only accept decompressed points for simplicity and optimized gas consumption. +- ***bls12381_p2_decompress —*** decompresses points from $E'(F_{p^2})$ provided in a compressed form. +- ***bls12381_pairing_check —*** verifies that $\prod e(p_i, q_i) = 1$, where $e$ is a pairing operation and $p_i \in G_1 \land q_i \in G_2$. This function is used to verify BLS signatures or zkSNARKs. + +Functions required for verifying BLS signatures[^59]: + +- bls12381_p1_sum +- bls12381_p2_sum +- bls12381_map_fp2_to_g2 +- bls12381_p1_decompress +- bls12381_p2_decompress +- bls12381_pairing_check + +Functions required for verifying zkSNARKs: + +- bls12381_p1_sum +- bls12381_g1_multiexp +- bls12381_pairing_check + +Both zkSNARKs and BLS signatures can be implemented alternatively by swapping $G_1$ and $G_2$. +Therefore, all functions have been implemented for both $G_1$ and $G_2$. + +An analogous proposal, EIP-2537[^15], exists in Ethereum. +The functions here have been designed with compatibility +with that Ethereum's proposal in mind. This design approach aims +to ensure future ease in supporting corresponding precompiles for Aurora[^24]. + +## Specification + +### BLS12-381 Curve Specification + +#### Elliptic Curve + +**The field $F_p$** for some *prime* $p$ is a set of integer +elements $\textbraceleft 0, 1, \ldots, p - 1 \textbraceright$ with two +operations: multiplication $\cdot$ and addition $+$. +These operations involve standard integer multiplication and addition, +followed by computing the remainder modulo $p$. + +**The elliptic curve $E(F_p)$** is the set of all pairs $(x, y)$ with coordinates in $F_p$ satisfying: + +$$ +y^2 \equiv x^3 + Ax + B \mod p +$$ + +together with an imaginary point at infinity $\mathcal{O}$, where: $A, B \in F_p$, $p$ is a prime $> 3$, and $4A^3 + 27B^2 \not \equiv 0 \mod p$ + +In the case of BLS12-381 the equation is $y^2 \equiv x^3 + 4 \mod p$[^15],[^51],[^14],[^11] + +**Parameters for our case:** + +- $A = 0$ +- $B = 4$ +- $p = \mathtt{0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab}$ + +Let $P \in E(F_q)$ have coordinates $(x, y)$, define **$-P$** as a point on a curve with coordinates $(x, -y)$. + +**The addition operation for Elliptic Curve** is a function $+\colon E(F_p) \times E(F_p) \rightarrow E(F_p)$ defined with following rules: let $P$ and $Q \in E(F_p)$ + +- if $P \ne Q$ and $P \ne -Q$ + - draw a line passing through $P$ and $Q$. This line intersects the curve at a third point $R$. + - reflect the point $R$ across the $x$-axis by changing the sign of the $y$-coordinate. The resulting point is $P+Q$. +- if $P=Q$ + - draw a tangent line through $P$ for an elliptic curve. The line will intersect the curve at the second point $R$. + - reflect the point $R$ across the $x$-axis the same way to get point $2P$ +- $P = -Q$ + - $P + Q = P + (-P) = \mathcal{O}$ — the point on infinity +- $Q = \mathcal{O}$ + - $P + Q = P + \mathcal{O} = P$ + +With the addition operation, Elliptic Curve forms a **group**. + +#### Subgroups + +**Subgroup** H is a subset of the group G with the following properties: + +- $\forall h_1, h_2 \in H\colon h_1 + h_2 \in H$ +- $0 \in H$ +- $\forall h \in H \colon -h \in H$ + +Notation: $H \subseteq G$ + +Group/subgroup **order** is the number of elements in group/subgroup. + +Notation: |G| or #G, where G represents the group. + +For some technical reason (related to the `pairing` operation which we will define later), +we will not operate over the entire $E(F_p)$, +but only over the two subgroups $G_1$ and $G_2$ +having the same **order** $r$. +$G_1$ is a subset of $E(F_p)$, +while $G_2$ is a subgroup of another group that we will define later. +The value of $r$ should be a prime number and $G_1 \ne G_2$ + +For the BLS12-381 Elliptic Curve, **the order r** of $G_1$ and $G_2$[^15],[^51] is given by: + +- $r = \mathtt{0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001}$ + +#### Field extension + +**The field extension $F_{p^k}$ of $F_{p}$** is a set comprising all polynomials of degree < k and coefficients from $F_p$, along with defined operations of multiplication ($\cdot$) and addition ($+$). + +$$ +a_{k - 1}x^{k - 1} + \ldots + a_1x + a_0 = A(x) \in F_{p^k} \vert a_i \in F_p +$$ + +The addition operation ($+$) is defined as regular polynomial addition: + +$$ +A(x) + B(x) = C(x) +$$ + +$$ +\sum a_i x^i + \sum b_i x^i = \sum c_i x^i +$$ + +$$ +c_i = (a_i + b_i) \mod p +$$ + + +The multiplication $\cdot$ is defined as regular polynomial multiplication modulo $M(x)$, +where $M(x)$ is an irreducible polynomial of degree $k$ with coefficients from $F_p$. + +$$ +C(x) = A(x) \cdot B(x)\mod M(x) +$$ + +Notation: $F_{p^k} = F_{p}[x] / M(x)$ + +In BLS12-381, we will require $F_{p^{12}}$. +We'll construct this field not directly as an extension from $F_p$, +but rather through a stepwise process. First, we'll build $F_{p^2}$ +as a quadratic extension of the field $F_p$. +Second, we'll establish $F_{p^6}$ as a cubic extension of $F_{p^2}$. +Finally, we'll create $F_{p^{12}}$ as a quadratic extension of the +field $F_{p^6}$. + +To define these fields, we'll need to set up three irreducible polynomials[^51]: + +- $F_{p^2} = F_p[u] / (u^2 + 1)$ +- $F_{p^6} = F_{p^2}[v] / (v^3 - u - 1)$ +- $F_{p^{12}} = F_{p^6}[w] / (w^2 - v)$ + +The second subgroup we'll utilize has order r and +resides within the same elliptic curve but with elements from $F_{p^{12}}$. +Specifically, $G_2 \subset E(F_{p^{12}})$, where $E: y^2 = x^3 + 4$ + +#### Twist + +Storing elements from $E(F_{p^{12}})$ consumes a significant amount of memory. +The twist operation transforms the original curve $E(F_{p^{12}})$ into another curve within a different space, +denoted as $E'(F_{p^2})$. It is crucial that this new curve also includes a $G'_2$ subgroup with order 'r' +so that we can easily transform it back to the original $G_2$. + +We want to have $\psi \colon E'(F_{p^2}) \rightarrow E(F_{p^{12}})$, such as + +- $\forall a, b \in E'(F_{p^2}) \colon \psi(a + b) = \psi(a) + \psi(b)$ +- $\forall a, b \in E'(F_{p^2}) \colon \psi(a) = \psi(b) \Rightarrow a = b$ + +This is referred to as an injective group homomorphism. + +For BLS12-381, E’ is defined as[^51]: + +$$ +E'\colon y^2 = x^3 + 4(u + 1) +$$ + +In most cases, we will be working with points from $G_2' \subset E'(F_{p^2})$ and will simply use the notation $G_2$ for this subgroup. + +#### Generators + +If there exists an element $g$ in the group $G$ such that $\textbraceleft g, 2 \cdot g, 3 \cdot g, \ldots, |G|g \textbraceright = G$, the group $G$ is called a ***cyclic group*** and $g$ is termed a ***generator*** + +$G_1$ and $G_2$ are cyclic subgroups with the following generators[^15],[^51]: + +$G_1$: + +- $x = \mathtt{0x17f1d3a73197d7942695638c4fa9ac0fc3688c4f9774b905a14e3a3f171bac586c55e83ff97a1aeffb3af00adb22c6bb}$ +- $y = \mathtt{0x08b3f481e3aaa0f1a09e30ed741d8ae4fcf5e095d5d00af600db18cb2c04b3edd03cc744a2888ae40caa232946c5e7e1}$ + +For $(x', y') \in G_2 \subset E'(F_{p^2}):$ +$$x' = x_0 + x_1u$$ + +$$y' = y_0 + y_1u$$ + +$G_2$: + +- $x_0 = \mathtt{0x024aa2b2f08f0a91260805272dc51051c6e47ad4fa403b02b4510b647ae3d1770bac0326a805bbefd48056c8c121bdb8}$ +- $x_1 = \mathtt{0x13e02b6052719f607dacd3a088274f65596bd0d09920b61ab5da61bbdc7f5049334cf11213945d57e5ac7d055d042b7e}$ +- $y_0 = \mathtt{0x0ce5d527727d6e118cc9cdc6da2e351aadfd9baa8cbdd3a76d429a695160d12c923ac9cc3baca289e193548608b82801}$ +- $y_1 = \mathtt{0x0606c4a02ea734cc32acd2b02bc28b99cb3e287e85a763af267492ab572e99ab3f370d275cec1da1aaa9075ff05f79be}$ + + +**Cofactor** is the ratio of the size of the entire group $G$ to the size of the subgroup $H$: + +$$ +|G|/|H| +$$ + +Cofactor $G_1\colon h = |E(F_p)|/r$[^51] + +$$h = \mathtt{0x396c8c005555e1568c00aaab0000aaab}$$ + +Cofactor $G_2\colon h' = |E'(F_{p^2})|/r$[^51] + +$$h' = \mathtt{0x5d543a95414e7f1091d50792876a202cd91de4547085abaa68a205b2e5a7ddfa628f1cb4d9e82ef21537e293a6691ae1616ec6e786f0c70cf1c38e31c7238e5}$$ + +#### Pairing + +Pairing is a necessary operation for the verification of BLS signatures and certain zkSNARKs. It performs the operation $e\colon G_1 \times G_2 \rightarrow G_T$, where $G_T \subset F_{p^{12}}$. + +The main properties of the pairing operation are: + +- $e(P, Q + R) = e(P, Q) \cdot e(P, R)$ +- $e(P + S, R) = e(P, R)\cdot e(S, R)$ + +To compute this function, we utilize an algorithm called Miller Loop. +For an affective implementation of this algorithm, +we require a key parameter for the BLS curve, denoted as $x$: + +$$ x = -\mathtt{0xd201000000010000}$$ + +This parameter can be found in the following sources: + +- [^15] section specification, pairing parameters, Miller loop scalar +- [^51] section 4.2.1 Parameter t +- [^14] section BLS12-381, parameter u +- [^11] section Curve equation and parameters, parameter x + +#### Summary + +The parameters for the BLS12-381 curve are as follows: + +Base field modulus: $p = \mathtt{0x1a0111ea397fe69a4b1ba7b6434bacd764774b84f38512bf6730d2a0f6b0f6241eabfffeb153ffffb9feffffffffaaab}$ + +$$ +E\colon y^2 \equiv x^3 + 4 +$$ + +$$ +E'\colon y^2 \equiv x^3 + 4(u + 1) +$$ + +Main subgroup order: $r = \mathtt{0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001}$ + +$$ +F_{p^2} = F_p[u] / (u^2 + 1) +$$ + +$$ +F_{p^6} = F_{p^2}[v] / (v^3 - u - 1) +$$ + +$$ +F_{p^{12}} = F_{p^6}[w] / (w^2 - v) +$$ + +Generator for $G_1$: + +- $x = \mathtt{0x17f1d3a73197d7942695638c4fa9ac0fc3688c4f9774b905a14e3a3f171bac586c55e83ff97a1aeffb3af00adb22c6bb}$ +- $y = \mathtt{0x08b3f481e3aaa0f1a09e30ed741d8ae4fcf5e095d5d00af600db18cb2c04b3edd03cc744a2888ae40caa232946c5e7e1}$ + +Generator for $G_2$: + +- $x_0 = \mathtt{0x024aa2b2f08f0a91260805272dc51051c6e47ad4fa403b02b4510b647ae3d1770bac0326a805bbefd48056c8c121bdb8}$ +- $x_1 = \mathtt{0x13e02b6052719f607dacd3a088274f65596bd0d09920b61ab5da61bbdc7f5049334cf11213945d57e5ac7d055d042b7e}$ +- $y_0 = \mathtt{0x0ce5d527727d6e118cc9cdc6da2e351aadfd9baa8cbdd3a76d429a695160d12c923ac9cc3baca289e193548608b82801}$ +- $y_1 = \mathtt{0x0606c4a02ea734cc32acd2b02bc28b99cb3e287e85a763af267492ab572e99ab3f370d275cec1da1aaa9075ff05f79be}$ + + +Cofactor for $G_1$: +$$h = \mathtt{0x396c8c005555e1568c00aaab0000aaab}$$ + +Cofactor for $G_2$: +$$h' = \mathtt{0x5d543a95414e7f1091d50792876a202cd91de4547085abaa68a205b2e5a7ddfa628f1cb4d9e82ef21537e293a6691ae1616ec6e786f0c70cf1c38e31c7238e5}$$ + +Key BLS12-381 parameter used in Miller Loop: +$$x = -\mathtt{0xd201000000010000}$$ + +All parameters were sourced from [^15], [^51], and [^14], and they remain consistent across these sources. + +### Map to curve specification + +This section delineates the functionality of the `bls12381_map_fp_to_g1` and `bls12381_map_fp2_to_g2` functions, +operating in accordance with the RFC9380 specification "Hashing to Elliptic Curves"[^62]. + +These functions map field elements in $F_p$ or $F_{p^2}$ +to their corresponding subgroups: $G_1 \subset E(F_p)$ or $G_2 \subset E'(F_{p^2})$. +`bls12381_map_fp_to_g1`/`bls12381_map_fp2_to_g2` combine the functionalities +of `map_to_curve` and `clear_cofactor` from RFC9380[^63]. + +```text +fn bls12381_map_fp_to_g1(u): + let Q = map_to_curve(u); + return clear_cofactor(Q); +``` + +We choose not to implement the `hash_to_field` function as a host function due to potential changes in hashing methods. +Additionally, executing this function within the contract consumes approximately 2 TGas, which is acceptable for our goals. + +Specific implementation parameters for `bls12381_map_fp_to_g1` and `bls12381_map_fp2_to_g2` can be found in RFC9380 +under sections 8.8.1[^64] and 8.8.2[^65], respectively. + +### Curve points encoding + +#### General comments + +The encoding rules for curve points and field elements align with the standards established in zkcrypto[^53] and +the implementation in the milagro lib[^29]. + +For elements from $F_p$ the first three bits will always be $0$, because the first byte of $p$ equals $1$. As a result, +we can use these bits to encode extra information: the encoding format, the point at infinity, and the points' sign. +Read more in sections: Uncompressed/compressed points on curve $E(F_p)$ / $E'(F_{p^2})$. + +#### Sign + +The sign of a point on the elliptic curve is represented as a u8 type in Rust, with two possible values: 0 for a positive sign and 1 for a negative sign. Any other u8 value is considered invalid and should be treated as incorrect. + +#### Scalar + +A scalar value is encoded as little-endian [u8; 32]. All possible byte combinations are allowed. + +#### Fields elements $F_p$ + +Values from $F_p$ are encoded as big-endian [u8; 48]. Only values less than p are permitted. If the value is equal to or greater than p, an error should be returned. + +#### Extension fields elements $F_{p^2}$ + +An element $q \in F_{p^{2}}$ can be expressed as $q = c_0 + c_1 v$, where $c_0, c_1 \in F_p$. +An element from $F_{p^2}$ is encoded in [u8; 96] as the byte concatenation of $c_1$ and $c_0$. The encoding for $c_1$ and $c_0$ follows the rule described in the previous section. + +#### Uncompressed points on curve $E(F_p)$ + +Points on the curve are represented by affine coordinates: $(x: F_p, y: F_p)$. +Elements from $E(F_p)$ are encoded in `[u8; 96]` as the byte concatenation of the x and y point coordinates, where $x, y \in F_p$. +The encoding follows the rules outlined in the section “Fields elements $F_p$”. + +*The second-highest bit* within the encoding serves to signify a point at infinity. +When this bit is set to 1, it designates an infinity point. +In this case, all other bits should be set to 0. + +Encoding the point at infinity: + +```bash +let x: [u8; 96] = [0; 96]; +x[0] = x[0] | 0x40; +``` + +#### Compressed points on curve $E(F_p)$ + +The points on the curve are represented by affine coordinates: $(x: F_p, y: F_p)$. +Elements from $E(F_p)$ in compressed form are encoded as `[u8; 48]`, +with big-endian encoded $x \in F_p$. +The $y$ coordinate is determined by the formula: $y = \pm \sqrt{x^3 + 4}$. + +- The highest bit indicates that the point is encoded in compressed form and thus must always be set to 1. +- The second-highest bit marks the point at infinity (if set to 1). + - For the point at infinity, all bits except the first two should be set to 0; other encodings should be considered as incorrect. +- To represent the sign of $y$, the third-highest bit in the x encoding is utilized. + - If the bit is 0, $y$ is positive; if 1, $y$ is negative. We'll consider the number positive by taking the smallest value between $y$ and $-y$, after reducing them to $[0, p)$. + +The encoding for $x \in F_p$ as `[u8; 48]` bytes follows the rules described in the section "Extension fields elements $F_{p}$". + +Encoding a point on $E(F_p)$ with a negative $y$ coordinate: + +```rust +let x: [u8; 48] = encodeFp(x) +x[0] = x[0] | 0x80; +x[0] = x[0] | 0x20; +``` + +Encoding the point at infinity: + +```rust +let x: [u8; 48] = [0; 48]; +x[0] = x[0] | 0x80; +x[0] = x[0] | 0x40; +``` + +#### Uncompressed points on the twisted curve $E'(F_{p^2})$ + +The points on the curve are represented by affine coordinates: $(x: F_{p^2}, y: F_{p^2})$. +Elements from $E'(F_{p^2})$ are encoded in [u8; 192] as a concatenation of bytes representing x and y coordinates, where $x, y \in F_{p^2}$. +The encoding for $x$ and $y$ follows the rules detailed in the "Extension Fields Elements $F_{p^2}$" section. + +*The second-highest bit* within the encoding serves to signify a point at infinity. +When this bit is set to 1, it designates an infinity point. +In this case, all other bits should be set to 0. + +Encoding the point at infinity: + +```bash +let x: [u8; 192] = [0; 192]; +x[0] = x[0] | 0x40; +``` + +#### Compressed points on twisted curve $E'(F_{p^2})$ + +The points on the curve are represented by affine coordinates: $(x: F_{p^2}, y: F_{p^2})$. +Elements from $E'(F_{p^2})$ in compressed form are encoded as [u8; 96], +with big-endian encoded $x \in F_{p^2}$. +The $y$ coordinate is determined using the formula: $y = \pm \sqrt{x^3 + 4(u + 1)}$. + +- The highest bit indicates if the point is encoded in compressed form and should be set to 1. +- The second-highest bit marks the point at infinity (if set to 1). + - For the point at infinity, all bits except the first two should be set to 0; other encodings should be considered as incorrect. +- To represent the sign of $y$, the third-highest bit in the x encoding is utilized. + - If the bit is 0, $y$ is positive; if 1, $y$ is negative. We'll consider the number positive by taking the smallest value between $y$ and $-y$: first compare $c_1$, then $c_0$, after reduction to $[0, p)$. + +The encoding of $x \in F_{p^2}$ as [u8; 96] bytes follows the rules from the section “Extension Fields Elements $F_{p^2}$”. + +Encoding a point on $E'(F_{p^2})$ with a negative $y$ coordinate: + +```rust +let x: [u8; 96] = encodeFp2(x); +x[0] = x[0] | 0x80; +x[0] = x[0] | 0x20; +``` + +Encoding the point at infinity: + +```rust +let x: [u8; 96] = [0; 96]; +x[0] = x[0] | 0x80; +x[0] = x[0] | 0x40; +``` + +#### ERROR_CODE + +Validating the input for the host functions within the contract can consume significant gas. +For instance, verifying if a point belongs to the subgroup is gas-consuming. +If an error is returned by the near host function, the entire execution is reverted. +To mitigate this, when the input verification is complex, the host function +will successfully complete its work but return an ERROR_CODE. +This enables users to handle error cases independently. It's important to note that host functions +might terminate with an error if it's straightforward to avoid it (e.g., incorrect input size). + +The ERROR_CODE is an u64 and can hold the following values: + +- 0: No error, execution was successful. For `bls12381_pairing_check` function, the pairing result equals the multiplicative identity. +- 1: Execution finished with error due to: + - Incorrect encoding (e.g., incorrectly set compression/decompression bit, coordinate >= p, etc.). + - A point not on the curve (where applicable). + - A point not in the expected subgroup (where applicable). +- 2: Can be returned only in `bls12381_pairing_check`. No error, execution was successful, but the pairing result doesn't equal the multiplicative identity. + +### Host functions + +#### General comments for all functions + +In all functions, the input is fetched from memory, beginning at `value_ptr` and extending to `value_ptr + value_len`. +If `value_len` is `u64::MAX`, input will come from the register with id `value_ptr`. + +Execution ends only if there's an incorrect input length, +input extends beyond memory bounds, or gas limits are reached. +Otherwise, execution completes successfully, providing the `ERROR_CODE`. + +If the `ERROR_CODE` equals 0, the output data will be written to +the register with the `register_id` identifier. +Otherwise, nothing will be written to the register. + +***Gas Estimation:*** + +The algorithms described above exhibit linear complexity concerning the number of elements. Gas estimation can be calculated using the following formula: + +```rust +let k = input_bytes/item_size +let gas_consumed = A + B * k +``` + +Here, A and B denote empirically calculated constants unique to each algorithm. + +For gas estimation, the benchmark vectors outlined in EIP-2537[^46] can be used where applicable. + +***Error cases (execution is terminated):*** + +For all functions, execution will terminate in the following cases: + +- The input length is not divisible by `item_size`. +- The input is beyond memory bounds. + +#### bls12381_p1_sum + +***Description:*** + +The function calculates the sum of signed elements on the BLS12-381 curve. It accepts an arbitrary number of pairs $(s_i, p_i)$, where $p_i \in E(F_p)$ represents a point on the elliptic curve, and $s_i \in {0, 1}$ signifies the point's sign. The output is a single point from $E(F_p)$ equivalent to $\sum (-1)^{s_i}p_i$. + +The operations, including the $E(F_p)$ curve, points on the curve, multiplication by -1, and the addition operation, are detailed in the BLS12-381 Curve Specification section. + +Note: This function accepts points from the entire curve and is not restricted to points in $G_1$. + +***Input:*** + +The sequence of pairs $(s_i, p_i)$, where $p_i \in E(F_p)$ represents a point and $s_i \in {0, 1}$ denotes the sign. Each point is encoded in decompressed form as $(x\colon F_p, y\colon F_p)$, and the sign is encoded in one byte, taking only two allowed values: 0 or 1. Expect 97*k bytes as input, which are interpreted as byte concatenation of k slices, with each slice representing the point sign and the uncompressed point from $E(F_p)$. Further details are available in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: 96 bytes represent one point $\in E(F_p)$ in its decompressed form. In case of an empty input, it outputs a point on infinity (refer to the Curve Points Encoding section for more details). +- ERROR_CODE = 1: + - Points or signs are incorrectly encoded (refer to Curve points encoded section). + - Point is not on the curve. + +***Test cases:*** + +Tests for the sum of two points + +This section aims to verify the correctness of summing up two valid elements on the curve: + +- Utilize points on the curve with known addition results for comparison, such as tests from EIP-2537[^47],[^48]. +- Generate random points on the curve and verify the commutative property: P + Q = Q + P. +- Validate that the sum of random points from $G_1$ remains in $G_1$. +- Generate random points on the curve and use another library to cross-check the results. + +Edge cases: + +- Points not from $G_1$. +- $\mathcal{O} + \mathcal{O} = \mathcal{O}$. +- $P + \mathcal{O} = \mathcal{O} + P = P$. +- $P + (-P) = (-P) + P = \mathcal{O}$. +- P + P (tangent to the curve). +- The sum of two points P and (-(P + P)) (tangent to the curve at point P). + + +Tests for inversion + +This section aims to validate the correctness of point inversion: + +- Generate random points on the curve and verify $P - P = -P + P = \mathcal{O}$. +- Generate random points on the curve and verify -(-P) = P. +- Generate random points from $G_1$ and ensure that -P also belong to $G_1$. +- Utilize an external implementation, generate random points on the curve, and compare results. + +Edge cases: + +- Points not from $G_1$. +- $-\mathcal{O}$ + +Tests for incorrect data + +This section aims to validate the handling of incorrect input data: + +- Incorrect input length. +- Incorrect sign value (not 0 or 1). +- Erroneous coding of field elements: one of the first three bits set up incorrectly. +- Erroneous coding of field elements resulting in a correct element on the curve modulo p. +- Erroneous coding of field elements with an incorrect extra bit in the decompressed encoding. +- Point not on the curve. +- Incorrect encoding of the point at infinity. +- Input is beyond memory bounds. + +Tests for the sum of an arbitrary amount of points + +This section focuses on validating the summation functionality with an arbitrary number of points: + +- Generate random points on the curve and verify that the sum of a random permutation matches. +- Generate random points on the curve and utilize another library to validate results. +- Create points and cross-check the outcome with the `multiexp` function. +- Generate random points from $G_1$ and confirm that the sum is also from $G_1$. + +Edge cases: + +- Empty input +- Sum with the maximum number of elements +- A single point + +***Annotation:*** + +```rust +pub fn bls12381_p1_sum(&mut self, + value_len: u64, + value_ptr: u64, + register_id: u64) -> Result; +``` + +#### bls12381_p2_sum + +***Description:*** + +The function computes the sum of the signed elements on the BLS12-381 curve. It accepts an arbitrary number of pairs $(s_i, p_i)$, where $p_i \in E'(F_{p^2})$ represents a point on the elliptic curve and $s_i \in {0, 1}$ is the point's sign. The output is a single point from $E'(F_{p^2})$ equal to $\sum (-1)^{s_i}p_i$. + +The $E'(F_{p^2})$ curve, the points on the curve, the multiplication by -1, and the addition operation are all defined in the BLS12-381 Curve Specification section. + +Note: The function accepts any points on the curve and is not limited to points in $G_2$. + +***Input:*** + +The sequence of pairs $(s_i, p_i)$, where $p_i \in E'(F_{p^2})$ is point and $s_i \in \textbraceleft 0, 1 \textbraceright$ represents a sign. +Each point is encoded in decompressed form as $(x: F_{p^2}, y: F_{p^2})$, and the sign is encoded in one byte. The expected input size is 193*k bytes, interpreted as a byte concatenation of k slices, +each slice representing the point sign alongside the uncompressed point from $E'(F_{p^2})$. +More details are available in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: 192 bytes represent one point $\in E'(F_{p^2})$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). +- ERROR_CODE = 1: + - Points or signs are incorrectly encoded (refer to Curve points encoded section). + - Point is not on the curve. + +***Test cases:*** + +The test cases are identical to those of `bls12381_p1_sum`, with the only alteration being the substitution of points from $G_1$ and $E(F_p)$ with points from $G_2$ and $E'(F_{p^2})$. + +***Annotation:*** + +```rust +pub fn bls12381_p2_sum(&mut self, + value_len: u64, + value_ptr: u64, + register_id: u64) -> Result; +``` + +#### ***bls12381_g1_multiexp*** + +***Description:*** + +The function accepts a list of pairs $(p_i, s_i)$, where $p_i \in G_1 \subset E(F_p)$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ denotes a scalar. It calculates $\sum s_i \cdot p_i$. + +The scalar multiplication operation signifies the addition of that point a scalar number of times: + +$$ +s \cdot p = \underbrace{p + p + \ldots + p}_{s} +$$ + +The $E(F_p)$ curve, $G_1$ subgroup, points on the curve, and the addition operation are defined in the BLS12-381 Curve Specification section. + +Please note: + +- The function accepts only points from $G_1$. +- The scalar is an arbitrary unsigned integer and can exceed the group order. +- To enhance gas efficiency, the Pippenger’s algorithm[^25] can be utilized. + +***Input:*** The sequence of pairs $(p_i, s_i)$, where $p_i \in G_1 \subset E(F_p)$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ is a scalar. The expected input size is 128*k bytes, interpreted as byte concatenation of k slices. Each slice comprises the concatenation of an uncompressed point from $G_1 \subset E(F_p)$— 96 bytes, along with a scalar— 32 bytes. Further details are available in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: 96 bytes represent one point $\in G_1 \subset E(F_p)$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). +- ERROR_CODE = 1: + - Points are incorrectly encoded (refer to Curve points encoded section). + - Point is not on the curve. + - Point is not from $G_1$ + +***Test cases:*** + +Tests for multiplication + +- Tests with known answers for multiplication from EIP-2537[^47],[^48]. +- Random small scalar n and point P: + - Check results with the sum function: `P + P + P + .. + P = n*P`. + - Compare with results from another library. +- Random scalar n and point P: + - Verify against results from another library. + - Implement multiplication by using the sum function and the double-and-add algorithm[^61]. + +Edge cases: + +- `group_order * P = 0` +- `(scalar + groupt_order) * P = scalar * P` +- `P + P + P .. + P = N*P` +- `0 * P = 0` +- `1 * P = P` +- Scalar is a MAX_INT + +Tests for sum of two points + +These are identical test cases to those in the `bls12381_p1_sum` section, but only with points from $G_1$ subgroup. + +- Generate random points P and Q, then compare the results with the sum function. + +Tests for the sum of an arbitrary amount of points + +- Random number of points, random point values; compare results with the sum function. +- Empty input. +- Input of maximum size. + +Tests for the multiexp of an arbitrary amount of points + +- Tests with known answers from EIP-2537[^47],[^48] +- Random number of points, scalars, and points: + - Check with results from another library. + - Check with raw implementation based on the sum function and the double-and-add algorithm. +- Empty input +- Maximum number of scalars and points. + +Tests for error cases + +- The same test cases as those in the `bls12381_p1_sum` section. +- Points not from $G_1$. + +***Annotation:*** + +```rust +pub fn bls12381_g1_multiexp( + &mut self, + value_len: u64, + value_ptr: u64, + register_id: u64, +) -> Result; +``` + +#### ***bls12381_g2_multiexp*** + +***Description:*** + +The function takes a list of pairs $(p_i, s_i)$ as input, where $p_i \in G_2 \subset E'(F_{p^2})$ represents a point on the curve, and $s_i \in \mathbb{N}_0$ denotes a scalar. The function computes $\sum s_i \cdot p_i$. + +This scalar multiplication operation involves adding the point $p$ to itself a specified number of times: + +$$ +s \cdot p = \underbrace{p + p + \ldots + p}_{s} +$$ + +The $E'(F_{p^2})$ curve, $G_2$ subgroup, points on the curve, and the addition operation are defined in the BLS12-381 Curve Specification section. + +Please note: + +- The function accepts only points from $G_2$. +- The scalar is an arbitrary unsigned integer and can exceed the group order. +- To enhance gas efficiency, the Pippenger’s algorithm[^25] can be utilized. + +***Input:*** the sequence of pairs $(p_i, s_i)$, where $p_i \in G_2 \subset E'(F_{p^2})$ is a point on the curve and $s_i \in \mathbb{N}_0$ is a scalar. + +The expected input size is `224*k` bytes, interpreted as the byte concatenation of `k` slices. Each slice is the concatenation of an uncompressed point from $G_2 \subset E'(F_{p^2})$ — `192` bytes and a scalar — `32` bytes. More details are in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: 192 bytes represent one point $\in G_2 \subset E'(F_{p^2})$ in its decompressed form. In case of an empty input, it outputs the point at infinity (refer to the Curve Points Encoding section for more details). +- ERROR_CODE = 1: + - Points are incorrectly encoded (refer to Curve points encoded section). + - Point is not on the curve. + - Point is not in $G_2$ subgroup. + +***Test cases:*** + +The test cases are identical to those for `bls12381_g1_multiexp`, except that the points from $G_1$ and $E(F_p)$ are replaced with points from $G_2$ and $E'(F_{p^2})$ + +***Annotation:*** + +```rust +pub fn bls12381_g2_multiexp( + &mut self, + value_len: u64, + value_ptr: u64, + register_id: u64, +) -> Result; +``` + +#### bls12381_map_fp_to_g1 + +***Description:*** + +This function takes as input a list of field elements $a_i \in F_p$ and maps them to $G_1 \subset E(F_p)$. +You can find the specification of this mapping function in the section titled 'Map to curve specification.' +Importantly, this function does NOT perform the mapping of the byte string into $F_p$. +The implementation of the mapping to $F_p$ may vary and can be effectively executed within the contract. + +***Input:*** + +The function expects `48*k` bytes as input, representing a list of element from $F_p$ (unsigned integer $< p$). Additional information is available in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: `96*k` bytes - represents a list of points $\in G_1 \subset E(F_p)$ in decompressed format. Further information is available in the Curve Points Encoding section. +- ERROR_CODE = 1: $a_i \ge p$. + +***Test cases:*** + +Tests for general cases + +- Validate the results for known answers from EIP-2537[^47],[^48]. +- Generate a random point $a$ from $F_p$: + - Verify the result using another library. + - Check that the resulting point lies on the curve in $G_1$. + - Compare the results for $a$ and $-a$; they should share the same x-coordinates and have opposite y-coordinates. + +Edge cases: + +- $a = 0$ +- $a = p - 1$ + +Tests for an arbitrary number of elements + +- Empty input +- Maximum number of points. +- Generate a random number of field elements and compare the result with another library. + +Tests for error cases + +- Input length is not divisible by 48: +- Input is beyond memory bounds. +- $a = p$ +- Random number $\ge p$ + +***Annotation:*** + +```rust +pub fn bls12381_map_fp_to_g1( + &mut self, + value_len: u64, + value_ptr: u64, + register_id: u64, +) -> Result; +``` + +#### bls12381_map_fp2_to_g2 + +***Description:*** + +This function takes as input a list of elements $a_i \in F_{p^2}$ and maps them to $G_2 \subset E'(F_{p^2})$. +You can find the mapping function specification in the "Map to Curve Specification" section. It's important to note that this function does NOT map byte strings into $F_{p^2}$. +The implementation of the mapping to $F_{p^2}$ may vary and can be effectively executed within the contract. + +***Input:*** the function takes as input `96*k` bytes — the elements from $F_{p^2}$ (two unsigned integers $< p$). Additional details can be found in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: `192*k` bytes - represents a list of points $\in G_2 \subset E'(F_{p^2})$ in decompressed format. More details are in the Curve Points Encoding section. +- ERROR_CODE = 1: one of the values is not a valid extension field $F_{p^2}$ element + +***Test cases:*** + +Tests for general cases + +- Validate the results for known answers from EIP-2537[^47],[^48] +- Generate a random point $a$ from $F_{p^2}$: + - Verify the result with another library. + - Check that the resulting point lies in $G_2$. + - Compare results for $a$ and $-a$; they should have the same x-coordinates and opposite y-coordinates. + +Edge cases: + +- $a = (0, 0)$ +- $a = (p - 1, p - 1)$ + +Tests for an arbitrary number of elements + +- Empty input +- Maximum number of points. +- Generate a random number of field elements and compare the result with another library. + +Tests for error cases + +- Input length is not divisible by 96. +- Input is beyond memory bounds. +- $a = (0, p)$ +- $a = (p, 0)$ +- (random number $\ge p$, 0) +- (0, random number $\ge p$) + +***Annotation:*** + +```rust +pub fn bls12381_map_fp2_to_g2( + &mut self, + value_len: u64, + value_ptr: u64, + register_id: u64, +) -> Result; +``` + +#### bls12381_pairing_check + +***Description:*** + +The pairing function is a bilinear function $e\colon G_1 \times G_2 \rightarrow G_T$, where $G_T \subset F_{q^{12}}$, +which is used to verify BLS signatures/zkSNARKs. + +This function takes as input the sequence of pairs $(p_i, q_i)$, where $p_i \in G_1 \subset E(F_{p})$ and $q_i \in G_2 \subset E'(F_{p^2})$ and validates: + +$$ +\prod e(p_i, q_i) = 1 +$$ + +We don’t need to calculate the pairing function itself as the result would lie on a huge field, and in all known applications only this validation check is necessary. + +***Input:*** A sequence of pairs $(p_i, q_i)$, where $p_i \in G_1 \subset E(F_{p})$ and $q_i \in G_2 \subset E'(F_{p^2})$. Each point is encoded in decompressed form. An expected input size of 288*k bytes is anticipated, interpreted as byte concatenation of k slices. Each slice comprises the concatenation of an uncompressed point from $G_1 \subset E(F_p)$ (occupying 96 bytes) and a point from $G_2 \subset E'(F_{p^2})$ (occupying 192 bytes). Additional details can be found in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct, the pairing result equals the multiplicative identity. +- ERROR_CODE = 1: + - Points encoded incorrectly (refer to the Curve Points Encoded section). + - Point not on the curve. + - Point not in $G_1/G_2$. +- ERROR_CODE = 2: the input is correct, the pairing result doesn't equal the multiplicative identity. + +***Test cases:*** + +Tests for one pair + +- Generate a random point $P \in G_1$: verify $e(P, \mathcal{O}) = 1$ +- Generate a random point $Q \in G_2$: verify $e(\mathcal{O}, Q) = 1$ +- Generate random points $P \ne \mathcal{O} \in G_1$ and $Q \ne \mathcal{O} \in G_2$: verify $e(P, Q) \ne 1$ + +Tests for two pairs + +- Generate random points $P \in G_1$, $Q \in G_2$ and random scalars $s_1, s_2$: + - $e(P, Q) \cdot e(P, -Q) = 1$ + - $e(P, Q) \cdot e(-P, Q) = 1$ + - $e(s_1P, s_2Q) \cdot e(-s_2P, s_1Q) = 1$ + - $e(s_1P, s_2Q) \cdot e(s_2P, -s_1Q) = 1$ + +- $g_1 \in G_1$, $g_2 \in G_2$ are generators defined in section 'BLS12-381 Curve Specification', r is the order of $G_1$ and $G_2$, and $p_1, p_2, q_1, q_2$ are randomly generated scalars: + - if $p_1 \cdot q_1 + p_2 \cdot q_2 \not \equiv 0 (\mod r)$, verify $e(p_1 g_1, q_1 g_2) \cdot e(p_2 g_1, q_2 g_2) \ne 1$ + - if $p_1 \cdot q_1 + p_2 \cdot q_2 \equiv 0 (\mod r)$, verify $e(p_1 g_1, q_1 g_2) \cdot e(p_2 g_1, q_2 g_2) = 1$ + +Tests for an arbitrary number of pairs + +- Empty input +- Test with the maximum number of pairs +- Tests using known answers from EIP-2537[^47],[^48] +- For all possible values of 'n', generate random scalars $p_1 \cdots p_n$ and $q_1 \cdots q_n$ such that $\sum p_i \cdot q_i \not \equiv 0 (\mod r)$: + - Verify $\prod e(p_i g_1, q_i g_2) \ne 1$ +- For all possible values of 'n', generate random scalars $p_1 \cdots p_{n - 1}$ and $q_1 \cdots q_{n - 1}$: + - Verify $(\prod e(p_i g_1, q_i g_2)) \cdot e(-(\sum p_i q_i) g_1, g_2) = 1$ + - Verify $(\prod e(p_i g_1, q_i g_2)) \cdot e(g_1, -(\sum p_i q_i) g_2) = 1$ + +Tests for error cases + +- The first point is on the curve but not in $G_1$. +- The second point is on the curve but not in $G_2$. +- The input length is not divisible by 288. +- The first point is not on the curve. +- The second point is not on the curve. +- Input length exceeds the memory limit. +- Incorrect encoding of the point at infinity. +- Incorrect encoding of a curve point: + - Incorrect decompression bit. + - Coordinates greater than or equal to 'p'. + +***Annotation:*** + +```rust +pub fn bls12381_pairing_check(&mut self, + value_len: u64, + value_ptr: u64) -> Result; +``` + +#### bls12381_p1_decompress + +***Description:*** The function decompresses compressed points from $E(F_p)$. It takes an arbitrary number of points $p_i \in E(F_p)$ in compressed format as input and outputs the same number of points from $E(F_p)$ in decompressed format. Further details about the decompressed and compressed formats are available in the Curve Points Encoding section. + +***Input:*** A sequence of points $p_i \in E(F_p)$, with each point encoded in compressed form. An expected input size of 48*k bytes is anticipated, interpreted as the byte concatenation of k slices. Each slice represents the compressed point from $E(F_p)$. Additional details can be found in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: The sequence of points $p_i \in E(F_p)$, with each point encoded in decompressed form. An expected output of 96*k bytes, interpreted as the byte concatenation of k slices. Each slice represents the decompressed point from $E(F_p)$. k is the same as in the input. More details are available in the Curve Points Encoding section. +- ERROR_CODE = 1: + - Points are incorrectly encoded (refer to the Curve points encoded section). + - Point is not on the curve. + +***Test cases:*** + +Tests for decompressing a single point + +- Generate random points on the curve from $G_1$ and not from $G_1$: + - Check that the uncompressed point lies on the curve. + - Compare the result with another library. +- Generate random points with a negative y: + - Take the inverse and compare the y-coordinate. + - Compare the result with another library. +- Decompress a point on infinity. + +Tests for decompression of an arbitrary number of points + +- Empty input. +- Maximum number of points. +- Generate a random number of points on the curve and compare the result with another library. + +Tests for error cases + +- The input length is not divisible by 48. +- The input is beyond memory bounds. +- Point is not on the curve. +- Incorrect decompression bit. +- Incorrectly encoded point at infinity. +- Point with a coordinate larger than 'p'. + +***Annotation:*** + +```rust +pub fn bls12381_p1_decompress(&mut self, + value_len: u64, + value_ptr: u64, + register_id: u64) -> Result; +``` + +#### bls12381_p2_decompress + +***Description:*** The function decompresses compressed points from $E'(F_{p^2})$. It takes an arbitrary number of points $p_i \in E'(F_{p^2})$ in compressed format as input and outputs the same number of points from $E'(F_{p^2})$ in decompressed format. For more information about the decompressed and compressed formats, refer to the Curve Points Encoding section. + +***Input:*** A sequence of points $p_i \in E'(F_{p^2})$, with each point encoded in compressed form. The expected input size is `96*k` bytes, interpreted as the byte concatenation of k slices. Each slice represents the compressed point from $E'(F_{p^2})$. Additional details are available in the Curve Points Encoding section. + +***Output:*** + +The ERROR_CODE is returned. + +- ERROR_CODE = 0: the input is correct + - Output: the sequence of point $p_i \in E'(F_{p^2})$, with each point encoded in decompressed form. The expected output is 192*k bytes, interpreted as the byte concatenation of k slices. `k` corresponds to the value specified in the input section. Each slice represents the decompressed point from $E'(F_{p^2})$. For more details, refer to the Curve Points Encoding section. +- ERROR_CODE = 1: + - Points are incorrectly encoded (refer to Curve points encoded section). + - Point is not on the curve. + +***Test cases:*** + +The same test cases as `bls12381_p1_decompress`, but with points from $G_2$, and the input length should be divisible by 96. + +***Annotation:*** + +```rust +pub fn bls12381_p2_decompress(&mut self, + value_len: u64, + value_ptr: u64, + register_id: u64) -> Result; +``` + +## Reference Implementation + +Primarily, concerning integration with nearcore, our interest lies in Rust language libraries. The current implementations of BLS12-381 in Rust are: + +1. ***Milagro Library*** [^29]. +2. ***BLST*** [^30][^31]. +3. ***Matter labs EIP-1962 implementation*** [^32] +4. ***zCash origin implementation*** [^33] +5. ***MCL Library*** [^34] +6. ***FileCoin implementation*** [^35] +7. ***zkCrypto*** [^36] + +To compile the list, we used links from EIP-2537[^43], the pairing-curves specification[^44], and an article containing benchmarks[^45]. This list might be incomplete, but it should encompass the primary BLS12-381 implementations. + +In addition, there are implementations in other languages that are less relevant to us in this context but can serve as references. + +1. C++, ETH2.0 Client, ***Chia library***[^37] +2. Haskell, ***Adjoint Lib***[^38] +3. Go, ***Go-Ethereum***[^39] +4. JavaScript, ***Noble JS***[^40] +5. Go, ***Matter Labs Go EIP-1962 implementation***[^41] +6. C++, ***Matter Labs C++ EIP-1962 implementation***[^42] + +One of the possible libraries to use is the blst library[^30]. +This library exhibits good performance[^45] and has undergone several audits[^55]. +You can find a draft implementation in nearcore, which is based on this library, through this link[^54]. + +## Security Implications + +The implementation's security depends on the chosen library's security, supporting operations with BLS curves. + +Within this NEP, a constant execution time for all operations isn't mandated. All the computations executed by smart contract are entirely public anyway, so there would be no advantage to a constant-time algorithm. + +BLS12-381 offers more security bits compared to the already existing pairing-friendly curve BN254. Consequently, the security of projects requiring a pairing-friendly curve will be enhanced. + +## Alternatives + +In nearcore, host functions for another pairing-friendly curve, BN254, have already been implemented[^10]. Some projects[^20] might consider utilizing the supported curve as an alternative. However, recent research indicates that this curve provides less than 100 bits of security and is not recommended for use[^13]. Furthermore, projects involved in cross-chain interactions, like Rainbow Bridge, are mandated to employ the same curve as the target protocol, which, in the case of Ethereum, is currently BLS12-381[^3]. Consequently, there is no viable alternative to employing a different pairing-friendly curve. + +An alternative approach involves creating a single straightforward host function in nearcore for BLS signature verification. This was the initially proposed solution[^26]. However, this solution lacks flexibility[^28] for several reasons: (1) projects may utilize different hash functions; (2) some projects might employ the $G_1$ subgroup for public keys, while others use $G_2$; (3) the specifications for Ethereum 2.0 remain in draft, subject to potential changes; (4) instead of a more varied and adaptable set of functions (inspired by EIP-2537's precompiles), we are left with a single large function; (5) there will be no support for zkSNARKs verification. + +Another alternative is to perform BLS12-381 operations off-chain. In this scenario, applications utilizing the BLS curve will no longer maintain trustlessness. + +## Future possibilities + +In the future, there might be support for working with various curves beyond just BLS12-381. In Ethereum, prior to EIP-2537[^15], there was a proposal, EIP-1962[^27], to introduce pairing-friendly elliptic curves in a versatile format, accommodating not only BLS curves but numerous others as well. However, this proposal wasn't adopted due to its extensive scope and complexity. Implementing every conceivable curve might not be practical, but it remains a potential extension worth considering. + +Another potential extension could involve supporting `hash_to_field` or `hash_to_curve` operations[^58]. Enabling their support would optimize gas usage for encoding messages into elements on the curve, which could be beneficial to BLS signatures. However, implementing the hash_to_field operation requires supporting multiple hashing algorithms simultaneously and doesn't demand a significant amount of gas for implementation within the contract. Therefore, these functions exceed the scope of this proposal. + +Additionally, a potential expansion might encompass supporting not only affine coordinates but also other coordinate systems, such as homogeneous or Jacobian projective coordinates. + +## Consequences + +### Positive + +- Projects currently utilizing BN254 will have the capability to transition to the BLS12-381 curve, thereby enhancing their security. +- Trustless cross-chain interactions with blockchains employing BLS12-381 in protocols (like Ethereum 2.0) will become feasible. + +### Neutral + +### Negative + +- There emerges a dependency on a library that supports operations with BLS12-381 curves. +- We'll have to continually maintain operations with BLS12-381 curves, even if vulnerabilities are discovered, and it becomes unsafe to use these curves. + +### Backward Compatibility + +There are no backward compatibility questions. + +## Changelog + +The previous NEP for supporting BLS signature based on BLS12-381[^26] + +[^1]: BLS 2002 [https://www.researchgate.net/publication/2894224_Constructing_Elliptic_Curves_with_Prescribed_Embedding_Degrees](https://www.researchgate.net/publication/2894224_Constructing_Elliptic_Curves_with_Prescribed_Embedding_Degrees) +[^2]: ZCash protocol: [https://zips.z.cash/protocol/protocol.pdf](https://zips.z.cash/protocol/protocol.pdf) +[^3]: Ethereum 2 specification: [https://github.com/ethereum/consensus-specs/blob/master/specs/phase0/beacon-chain.md](https://github.com/ethereum/consensus-specs/blob/master/specs/phase0/beacon-chain.md) +[^4]: Dfinity: [https://internetcomputer.org/docs/current/references/ic-interface-spec#certificate](https://internetcomputer.org/docs/current/references/ic-interface-spec#certificate) +[^5]: Tezos: [https://wiki.tezosagora.org/learn/futuredevelopments/layer2#zkchannels](https://wiki.tezosagora.org/learn/futuredevelopments/layer2#zkchannels) +[^6]: Filecoin: [https://spec.filecoin.io/](https://spec.filecoin.io/) +[^7]: Specification of pairing friendly curves with a list of applications in the table: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-adoption-status-of-pairing-](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-adoption-status-of-pairing-) +[^8]: Specification of pairing friendly curves, the security level for BLS12-381: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#section-4.2.1](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#section-4.2.1) +[^9]: BN2005: [https://eprint.iacr.org/2005/133](https://eprint.iacr.org/2005/133) +[^10]: NEP-98 for BN254 host functions on NEAR: [https://github.com/near/NEPs/issues/98](https://github.com/near/NEPs/issues/98) +[^11]: BLS12-381 for the Rest of Us: [https://hackmd.io/@benjaminion/bls12-381](https://hackmd.io/@benjaminion/bls12-381) +[^12]: BN254 for the Rest of Us: [https://hackmd.io/@jpw/bn254](https://hackmd.io/@jpw/bn254) +[^13]: Some analytics of different curve security: [https://www.ietf.org/archive/id/draft-irtf-cfrg-pairing-friendly-curves-02.html#name-for-100-bits-of-security](https://www.ietf.org/archive/id/draft-irtf-cfrg-pairing-friendly-curves-02.html#name-for-100-bits-of-security) +[^14]: ZCash Transfer from bn254 to bls12-381: [https://electriccoin.co/blog/new-snark-curve/](https://electriccoin.co/blog/new-snark-curve/) +[^15]: EIP-2537 Precompiles for Ethereum for BLS12-381: [https://eips.ethereum.org/EIPS/eip-2537](https://eips.ethereum.org/EIPS/eip-2537) +[^17]: Article about Rainbow Bridge [https://near.org/blog/eth-near-rainbow-bridge](https://near.org/blog/eth-near-rainbow-bridge) +[^19]: Intro into zkSNARKs: [https://media.consensys.net/introduction-to-zksnarks-with-examples-3283b554fc3b](https://media.consensys.net/introduction-to-zksnarks-with-examples-3283b554fc3b) +[^20]: Zeropool project: [https://zeropool.network/](https://zeropool.network/) +[^24]: Precompiles on Aurora: [https://doc.aurora.dev/dev-reference/precompiles/](https://doc.aurora.dev/dev-reference/precompiles/) +[^25]: Pippenger Algorithm: [https://github.com/wborgeaud/python-pippenger/blob/master/pippenger.pdf](https://github.com/wborgeaud/python-pippenger/blob/master/pippenger.pdf) +[^26]: NEP-446 proposal for BLS-signature verification: [https://github.com/nearprotocol/neps/pull/446](https://github.com/nearprotocol/neps/pull/446) +[^27]: EIP-1962 EC arithmetic and pairings with runtime definitions: [https://eips.ethereum.org/EIPS/eip-1962](https://eips.ethereum.org/EIPS/eip-1962) +[^28]: Drawbacks of NEP-446: [https://github.com/near/NEPs/pull/446#pullrequestreview-1314601508](https://github.com/near/NEPs/pull/446#pullrequestreview-1314601508) +[^29]: BLS12-381 Milagro: [https://github.com/sigp/incubator-milagro-crypto-rust/tree/057d238936c0cbbe3a59dfae6f2405db1090f474](https://github.com/sigp/incubator-milagro-crypto-rust/tree/057d238936c0cbbe3a59dfae6f2405db1090f474) +[^30]: BLST: [https://github.com/supranational/blst](https://github.com/supranational/blst), +[^31]: BLST EIP-2537 adaptation: [https://github.com/sean-sn/blst_eip2537](https://github.com/sean-sn/blst_eip2537) +[^32]: EIP-1962 implementation matter labs Rust: https://github.com/matter-labs/eip1962 +[^33]: zCash origin rust implementation: [https://github.com/zcash/zcash/tree/master/src/rust/src](https://github.com/zcash/zcash/tree/master/src/rust/src) +[^34]: MCL library: [https://github.com/herumi/bls](https://github.com/herumi/bls) +[^35]: filecoin/bls-signature: [https://github.com/filecoin-project/bls-signatures](https://github.com/filecoin-project/bls-signatures) +[^36]: zkCrypto: [https://github.com/zkcrypto/bls12_381](https://github.com/zkcrypto/bls12_381), [https://github.com/zkcrypto/pairing](https://github.com/zkcrypto/pairing) +[^37]: BLS12-381 code bases for ETH2.0 client Chia library C++: [https://github.com/Chia-Network/bls-signatures](https://github.com/Chia-Network/bls-signatures) +[^38]: Adjoint Lib: [https://github.com/sdiehl/pairing](https://github.com/sdiehl/pairing) +[^39]: Ethereum Go implementation for EIP-2537: [https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles](https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles) +[^40]: Noble JS implementation: [https://github.com/paulmillr/noble-bls12-381](https://github.com/paulmillr/noble-bls12-381) +[^41]: EIP-1962 implementation matter labs Go: https://github.com/kilic/eip2537, +[^42]: EIP-1962 implementation matter labs C++: https://github.com/matter-labs-archive/eip1962_cpp +[^43]: EIP-2537 with links: [https://github.com/matter-labs-forks/EIPs/blob/bls12_381/EIPS/eip-2537.md](https://github.com/matter-labs-forks/EIPs/blob/bls12_381/EIPS/eip-2537.md) +[^44]: Pairing-friendly curves specification, crypto libs: [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-cryptographic-libraries](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-09#name-cryptographic-libraries) +[^45]: Comparing different libs for pairing-friendly curves: [https://hackmd.io/@gnark/eccbench](https://hackmd.io/@gnark/eccbench) +[^46]: Bench vectors from EIP2537: [https://eips.ethereum.org/assets/eip-2537/bench_vectors](https://eips.ethereum.org/assets/eip-2537/bench_vectors) +[^47]: Metter Labs tests for EIP2537: [https://github.com/matter-labs/eip1962/tree/master/src/test/test_vectors/eip2537](https://github.com/matter-labs/eip1962/tree/master/src/test/test_vectors/eip2537) +[^48]: Tests from Go Ethereum implementation: [https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles](https://github.com/ethereum/go-ethereum/tree/master/core/vm/testdata/precompiles) +[^51]: draft-irtf-cfrg-pairing-friendly-curves-11 [https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-11#name-bls-curves-for-the-128-bit-](https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-pairing-friendly-curves-11#name-bls-curves-for-the-128-bit-) +[^52]: Paper with BLS12-381: [https://eprint.iacr.org/2019/403.pdf](https://eprint.iacr.org/2019/403.pdf) +[^53]: Zkcrypto points encoding: [https://github.com/zkcrypto/pairing/blob/0.14.0/src/bls12_381/README.md](https://github.com/zkcrypto/pairing/blob/0.14.0/src/bls12_381/README.md) +[^54]: Draft PR for BLS12-381 operations in nearcore: [https://github.com/near/nearcore/pull/9317](https://github.com/near/nearcore/pull/9317) +[^55]: Audit for BLST library: [https://research.nccgroup.com/wp-content/uploads/2021/01/NCC_Group_EthereumFoundation_ETHF002_Report_2021-01-20_v1.0.pdf](https://research.nccgroup.com/wp-content/uploads/2021/01/NCC_Group_EthereumFoundation_ETHF002_Report_2021-01-20_v1.0.pdf) +[^58]: hash_to_curve and hash_to_field function: [https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio](https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio) +[^59]: Implementation of BLS-signature based on these host functions: [https://github.com/olga24912/bls-signature-verificaion-poc/blob/main/src/lib.rs](https://github.com/olga24912/bls-signature-verificaion-poc/blob/main/src/lib.rs) +[^60]: hash_to_field specification: [https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio](https://datatracker.ietf.org/doc/html/rfc9380#name-hash_to_field-implementatio) +[^61]: double-and-add algorithm: [https://en.wikipedia.org/wiki/Exponentiation_by_squaring](https://en.wikipedia.org/wiki/Exponentiation_by_squaring) +[^62]: RFC 9380 Hashing to Elliptic Curves specification: [https://www.rfc-editor.org/rfc/rfc9380](https://www.rfc-editor.org/rfc/rfc9380) +[^63]: map_to_curve and clear_cofactor functions: [https://datatracker.ietf.org/doc/html/rfc9380#name-encoding-byte-strings-to-el](https://datatracker.ietf.org/doc/html/rfc9380#name-encoding-byte-strings-to-el) +[^64]: Specification of parameters for BLS12-381 G1: [https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g1](https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g1) +[^65]: Specification of parameters for BLS12-381 G2: [https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g2](https://datatracker.ietf.org/doc/html/rfc9380#name-bls12-381-g2) diff --git a/neps/nep-0491.md b/neps/nep-0491.md new file mode 100644 index 000000000..828afdd8d --- /dev/null +++ b/neps/nep-0491.md @@ -0,0 +1,395 @@ +--- +NEP: 491 +Title: Non-Refundable Storage Staking +Authors: Jakob Meier +Status: Final +DiscussionsTo: https://gov.near.org/t/proposal-locking-account-storage-refunds-to-avoid-faucet-draining-attacks/34155 +Type: Protocol Track +Version: 1.0.0 +Created: 2023-07-24 +LastUpdated: 2023-07-26 +--- + +## Summary + +Non-refundable storage allows to create accounts with arbitrary state for users, +without being susceptible to refund abuse. + +This is done by tracking non-refundable balance in a separate field of the +account. This balance is only useful for storage staking and otherwise can be +considered burned. + + +## Motivation + +Creating new accounts on chain costs a gas fee and a storage staking fee. The +more state is added to the account, the higher the storage staking fees. When +deploying a contract on the account, it can quickly go above 1 NEAR per account. + +Some business models are okay with paying that fee for users upfront, just to +get them onboarded. However, if a business does that today, their users can +delete their new accounts and spend the tokens intended for storage staking in +other ways. Since this is free for the user, they are financially incentivized +to repeat this action for as long as the business has funds left in the faucet. + +The protocol should allow to create accounts in a way that is not susceptible to +such refund abuse. This would at least change the incentives such that creating +fake users is no longer profitable. + +Non-refundable storage staking is a further improvement over +[NEP-448](https://github.com/near/NEPs/pull/448) (Zero Balance Accounts) which +addressed the same issue but is limited to 770 bytes per account. By lifting the +limit, sponsored accounts can be used in combination with smart contracts. + +## Specification + +Users can opt-in to nonrefundable storage when creating new accounts. For that, +we use the new action `ReserveStorage`. + +```rust +pub enum Action { + ... + ReserveStorage(ReserveStorageAction), + ... +} +``` + +To create a named account today, the typical pattern is a transaction with +`CreateAccount`, `Transfer`, and `AddKey`. To make the funds nonrefundable, we +can use action `ReserveStorage` like this: + +```json +"Actions": { + "CreateAccount": {}, + "ReserveStorage": { "deposit": "1000000000000000000000000" }, + "AddKey": { "public_key": "...", "access_key": "..." } +} +``` + +Adding a `Transfer` action allows the combination of nonrefundable balance and +refundable balance. This allows the user to make calls where they need to attach +balance, for example an FT transfer which requires 1 yocto NEAR. + +```json +"Actions": { + "CreateAccount": {}, + "ReserveStorage": { "deposit": "1000000000000000000000000" }, + "Transfer": { "deposit": "100" }, + "AddKey": { "public_key": "...", "access_key": "..." } +} +``` + +To create implicit accounts, the current protocol requires a single `Transfer` +action without further actions in the same transaction and this has not changed +with this proposal: + +```json +"Actions": { + "CreateAccount": {}, + "Transfer": { "deposit": "0" }, +} +``` + +If a non-refundable transfer arrives at an account that already exists, it will +fail and the funds are returned to the predecessor. + +Finally, when querying an account for its balance, there will be an additional +field `nonrefundable` in the output. Wallets will need to decide how they want +to show it. They could, for example, add a new field called "non-refundable +storage credits". + +```js +// Account near +{ + "amount": "68844924385676812880674962949", + "block_hash": "3d6SisRc5SuwrkJnLwQb3W5pWitZKCjGhiKZuc6tPpao", + "block_height": 97314513, + "code_hash": "Dmi6UTRYTT3eNirp8ndgDNh8kYk2T9SZ6PJZDUXB1VR3", + "locked": "0", + "storage_paid_at": 0, + "storage_usage": 2511772, + "formattedAmount": "68,844.924385676812880674962949", + // this is new + "nonrefundable": "0" +} +``` + + +## Reference Implementation + +On the protocol side, we need to add new action: + +```rust +enum Action { + CreateAccount(CreateAccountAction), + DeployContract(DeployContractAction), + FunctionCall(FunctionCallAction), + Transfer(TransferAction), + Stake(StakeAction), + AddKey(AddKeyAction), + DeleteKey(DeleteKeyAction), + DeleteAccount(DeleteAccountAction), + Delegate(super::delegate_action::SignedDelegateAction), + // this gets added in the end + ReserveStorage(ReserveStorageAction), +} +``` + +and handle the new action in the `apply_action` call. + +Further, we have to update the account meta data representation in the state +trie to track the non-refundable storage. + +```rust +pub struct Account { + amount: Balance, + locked: Balance, + // this field is new + nonrefundable: Balance, + code_hash: CryptoHash, + storage_usage: StorageUsage, + // the account version will be increased from 1 to 2 + version: AccountVersion, +} +``` + +The field `nonrefundable` must be added to the normal `amount` and the `locked` +balance calculate how much state the account is allowed to use. The new formula +to check storage balance therefore becomes + +```rust +amount + locked + nonrefundable >= storage_usage * storage_amount_per_byte +``` + +For old accounts that don't have the new field, the non-refundable balance is +always zero. Adding non-refundable balance later is not allowed. If a transfer +is made to an account that already existed before the receipt's actions are +applied, execution must fail with +`ActionErrorKind::OnlyReserveStorageOnAccountCreation{ account_id: AccountId }`. + +Conceptually, these are all changes on the protocol level. However, +unfortunately, the account version field is not currently serialized, hence not +included in the on-chain state. + +Therefore, as the last change necessary for this NEP, we also introduce a new +serialization format for new accounts. + +```rust +// new serialization format for `struct Account` + +// new: prefix with a sentinel value to detect V1 accounts, they will have +// a real balance here which is smaller than u128::MAX +writer.serialize(u128::MAX)?; +// new: include version number (u8) for accounts with version 2 or more +writer.serialize(version)?; +writer.serialize(amount)?; +writer.serialize(locked)?; +writer.serialize(code_hash)?; +writer.serialize(storage_usage)?; +// new: this is the field we added, the type is u128 like other balances +writer.serialize(nonrefundable)?; +``` + +Note that we are not migrating old accounts. Accounts created as version 1 will +remain at version 1. + +A proof of concept implementation for nearcore is available in this PR: +https://github.com/near/nearcore/pull/9346 + + +## Security Implications + +We were not able to come up with security relevant implications. + +## Alternatives + +There are small variations in the implementation, and then there are completely +different ways to look at the problem. Let's start with the variations. + +### Variation: Allow adding nonrefundable balance to existing accounts + +Instead of failing when a non-refundable transfer arrives at an existing +account, we could add the balance to the existing non-refundable balance. This +would be more flexible to use. A business could easily add more funds for +storage even after account creation. + +The problems are in the implementation details. It would allow to add +non-refundable storage to existing accounts, which would require some form of +migration of the all accounts in the state trie. This is impractical, as we have +to iterate over all existing accounts and re-merklize. That's infeasible within +a single block time and stopping the chain would be disruptive. + +We could maybe migrate lazily, i.e. read account version 1 and automatically +convert it to version 2. However, that would break the assumption that every +logical value in the merkle trie has a unique borsh representation, as there +would be a account version 1 and a version 2 borsh serialization that both map +to the same logical version 2 value. This could lead to different +representations of the same chunk in memory, which might be used in attacks to +force a double-sign by innocent validators. + +It is not 100% clear to me, the author, if this is a problem we could work +around. However, the complications it would involve do not seem to be worth it, +given that in the feature discussions nobody saw it as critical to add +non-refundable balance to existing accounts. + +### Variation: Allow refunds to original sponsor + +Instead of complete non-refundability, the tokens reserved for storage staking +could be returned to the original account that created the account when an +account is deleted. + +The community discussions ended with the conclusion that this feature would +probably not be used and we should not implement it until there is real demand +for it. + +### Alternative: Don't use smart contracts on user accounts + +Instead of deploying contracts on the user account, one could build a similar +solution that uses zero balance accounts and a single master contract that +performs all smart contract functionality required. This master contract can +implement the [Storage Management] +(https://nomicon.io/Standards/StorageManagement) standard to limit storage usage +per user. + +This solution is not as flexible. The master account cannot make cross-contract +function calls with the user id as the predecessor. + +### Alternative: Move away from storage staking + +We could also abandon the concept of storage staking entirely. However, coming +up with a scalable, sustainable solution that does not suffer from the same +refund problems is hard. + +One proposed design is a combination of zero balance accounts and code sharing +between contracts. Basically, if somehow the deployed code is stored in a way +that does not require storage staking by the user themself, maybe the per-user +state is small enough to fit in the 770 bytes limit of zero balance accounts. +(Questionable for non-trivial use cases.) + +This alternative is much harder to design and implement. The proposal that has +gotten the furthest so far is [Ephemeral +Storage](https://github.com/near/NEPs/pull/485), which is pretty complicated and +does not have community consensus yet. Nobody is currently working on moving it +forward. While we could wait for that to eventually make progress, in the +meantime, the community is held back in their innovation because of the refund +problem. + +### Alternative: Using a proxy account + +As suggested by [@mfornet](https://github.com/near/NEPs/pull/491#discussion_r1349496234) +another alternative is using a proxy account approach where the business creates +an account with a deployed contract that has Regular (user has full access key) +and Restricted mode (user doesn't have full access key and cannot delete +account). + +In restricted mode, the user has a `FunctionCallKey` which allows the user to +call methods of the contract that controls the `FullAccessKey` and allows the +user some functionality but not all, e.g. not allowing account deletion. The +user in restricted mode could also upgrade an account by sending the initial +amount of NEAR deposited by the account creator and will attach a new +`FullAccessKey`. + +The downside of this idea is additional complexity on the tooling side because +actions like adding access keys to the account need to be converted to function +calls instead of being direct actions. And the complexity on the business side +is that it needs to include the proxy logic with their business logic in the +same contract, increasing the complexity of development. + +### Alternative: Granular access key + +Another suggestion is introducing another key type `GranularAccessKey`. This +alternative includes a protocol change that introduces a new kind of access key +which can have granular permissions set on, it e.g. not being able to delete an +account. + +The business side gives this key to the user, and with this key comes a set of +permissions that the user can do. The user can also call `Upgrade` and get +`FullAccessKey` by paying for the initial amount which funded the account +creation. + +The drawback of this approach is that it requires that the business side would +have to handle the logic around `GranularAccessKey` and the `Upgrade` method +making the usage more complex. + +## Future possibilities + +- We might want to add the possibility to make non-refundable balance transfers + from within a smart contract. This would require changes to the WASM smart + contract to host interface. Since removing anything from there is virtually + impossible, we shouldn't be too eager in adding it there but if there is + demand for it, we certainly can do it without much trouble. +- We could later add the possibility to refund the non-refundable tokens to the + account who sent the tokens initially. +- We could allow sending non-refundable balance to existing accounts. +- If (cheap) code sharing between contracts is implemented in the future, this + proposal will most likely work well in combination with that. Per-user data + will still need to be paid for by the user, which could be sponsored as + non-refundable balance without running into refund abuse. + + +## Consequences + + +### Positive + +- Businesses can sponsor new user accounts without the user being able to steal + any tokens. + +### Neutral + +- Non-refundable tokens are removed from the circulating supply, i.e. burnt. + +### Negative + +- Understanding a user's balance become even more complicated than it already + is. Instead of only `amount` and `locked`, there will be a third component. +- There is no incentive anymore to delete an account and its state when the + backing tokens are not refundable. + + +### Backwards Compatibility + +We believe this can be implemented with full backwards compatibility. + +## Unresolved Issues (Optional) + +All of these issues already have a proposed solution above. But nevertheless, +these points are likely to be challenged / discussed: + +- Should we allow adding non-refundable balance to existing accounts? (proposal: + no) +- Should we allow adding more non-refundable balance after account creation? + (proposal: no) +- Should this NEP include a host function to send non-refundable balance from + smart contracts? (proposal: no) +- How should a wallet display non-refundable balances? (proposal: up to wallet + providers, probably a new separate field) + +## Changelog + +### 1.0.0 - Initial Version + +> Placeholder for the context about when and who approved this NEP version. + +#### Benefits + +> List of benefits filled by the Subject Matter Experts while reviewing this +> version: + +- Benefit 1 +- Benefit 2 + +#### Concerns + +> Template for Subject Matter Experts review for this version: Status: New | +> Ongoing | Resolved + +| # | Concern | Resolution | Status | +| --: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | + +## Copyright + +Copyright and related rights waived via +[CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/neps/nep-0508.md b/neps/nep-0508.md index 36fffe5ce..3230691b1 100644 --- a/neps/nep-0508.md +++ b/neps/nep-0508.md @@ -2,7 +2,7 @@ NEP: 508 Title: Resharding v2 Authors: Waclaw Banasik, Shreyan Gupta, Yoon Hong -Status: Draft +Status: Final DiscussionsTo: https://github.com/near/nearcore/issues/8992 Type: Protocol Version: 1.0.0 @@ -12,13 +12,13 @@ LastUpdated: 2023-11-14 ## Summary -This proposal introduces a new implementation for resharding and a new shard layout for the production networks. +This proposal introduces a new implementation for resharding and a new shard layout for the production networks. -In essence, this NEP is an extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which was focused on splitting one shard into multiple shards. +In essence, this NEP is an extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which was focused on splitting one shard into multiple shards. -We are introducing resharding v2, which supports one shard splitting into two within one epoch at a pre-determined split boundary. The NEP includes performance improvement to make resharding feasible under the current state as well as actual resharding in mainnet and testnet (To be specific, splitting the largest shard into two). +We are introducing resharding v2, which supports one shard splitting into two within one epoch at a pre-determined split boundary. The NEP includes performance improvement to make resharding feasible under the current state as well as actual resharding in mainnet and testnet (To be specific, splitting the largest shard into two). -While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseeable future, it does not serve all use cases, such as dynamic resharding. +While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseeable future, it does not serve all use cases, such as dynamic resharding. ## Motivation @@ -58,26 +58,26 @@ A new protocol version will be introduced specifying the new shard layout which ### Required state changes * For the duration of the resharding the node will need to maintain a snapshot of the flat state and related columns. As the main database and the snapshot diverge this will cause some extent of storage overhead. -* For the duration of the epoch before the new shard layout takes effect, the node will need to maintain the state and flat state of shards in the old and new layout at the same time. The State and FlatState columns will grow up to approx 2x the size. The processing overhead should be minimal as the chunks will still be executed only on the parent shards. There will be increased load on the database while applying changes to both the parent and the children shards. -* The total storage overhead is estimated to be on the order of 100GB for mainnet RPC nodes and 2TB for mainnet archival nodes. For testnet the overhead is expected to be much smaller. +* For the duration of the epoch before the new shard layout takes effect, the node will need to maintain the state and flat state of shards in the old and new layout at the same time. The State and FlatState columns will grow up to approx 2x the size. The processing overhead should be minimal as the chunks will still be executed only on the parent shards. There will be increased load on the database while applying changes to both the parent and the children shards. +* The total storage overhead is estimated to be on the order of 100GB for mainnet RPC nodes and 2TB for mainnet archival nodes. For testnet the overhead is expected to be much smaller. ### Resharding flow * The new shard layout will be agreed on offline by the protocol team and hardcoded in the reference implementation. * The first resharding will be scheduled soon after this NEP is merged. The new shard layout boundary accounts will be: ```["aurora", "aurora-0", "kkuuue2akv_1630967379.near", "tge-lockup.sweat"]```. * Subsequent reshardings will be scheduled as needed, without further NEPs, unless significant changes are introduced. -* In epoch T, past the protocol version upgrade date, nodes will vote to switch to the new protocol version. The new protocol version will contain the new shard layout. +* In epoch T, past the protocol version upgrade date, nodes will vote to switch to the new protocol version. The new protocol version will contain the new shard layout. * In epoch T, in the last block of the epoch, the EpochConfig for epoch T+2 will be set. The EpochConfig for epoch T+2 will have the new shard layout. * In epoch T + 1, all nodes will perform the state split. The child shards will be kept up to date with the blockchain up until the epoch end first via catchup, and later as part of block postprocessing state application. -* In epoch T + 2, the chain will switch to the new shard layout. +* In epoch T + 2, the chain will switch to the new shard layout. ## Reference Implementation -The implementation heavily re-uses the implementation from [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md). Below are listed the major differences and additions. +The implementation heavily re-uses the implementation from [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md). Below are listed the major differences and additions. -### Code pointers to the proposed implementation +### Code pointers to the proposed implementation -* [new shard layout](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/core/primitives/src/shard_layout.rs#L161) +* [new shard layout](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/core/primitives/src/shard_layout.rs#L161) * [the main logic for splitting states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/resharding.rs#L280) * [the main logic for applying chunks to split states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/update_shard.rs#L315) * [the main logic for garbage collecting state from parent shard](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/store.rs#L2335) @@ -92,7 +92,7 @@ In order to ensure consistent view of the flat storage while splitting the state ### Handling receipts, gas burnt and balance burnt -When resharding, extra care should be taken when handling receipts in order to ensure that no receipts are lost or duplicated. The gas burnt and balance burnt also need to be correctly handled. The old resharding implementation for handling receipts, gas burnt and balance burnt relied on the fact in the first resharding there was only a single parent shard to begin with. The new implementation will provide a more generic and robust way of reassigning the receipts to the child shards, gas burnt, and balance burnt, that works for arbitrary splitting of shards, regardless of the previous shard layout. +When resharding, extra care should be taken when handling receipts in order to ensure that no receipts are lost or duplicated. The gas burnt and balance burnt also need to be correctly handled. The old resharding implementation for handling receipts, gas burnt and balance burnt relied on the fact in the first resharding there was only a single parent shard to begin with. The new implementation will provide a more generic and robust way of reassigning the receipts to the child shards, gas burnt, and balance burnt, that works for arbitrary splitting of shards, regardless of the previous shard layout. ### New shard layout @@ -100,7 +100,7 @@ The first release of the resharding v2 will contain a new shard layout where one ### Removal of Fixed shards -Fixed shards was a feature of the protocol that allowed for assigning specific accounts and all of their recursive sub accounts to a predetermined shard. This feature was only used for testing and was never used in production. Fixed shards feature unfortunately breaks the contiguity of shards and is not compatible with the new resharding flow. A sub account of a fixed shard account can fall in the middle of account range that belongs to a different shard. This property of fixed shards made it particularly hard to reason about and implement efficient resharding. +Fixed shards was a feature of the protocol that allowed for assigning specific accounts and all of their recursive sub accounts to a predetermined shard. This feature was only used for testing and was never used in production. Fixed shards feature unfortunately breaks the contiguity of shards and is not compatible with the new resharding flow. A sub account of a fixed shard account can fall in the middle of account range that belongs to a different shard. This property of fixed shards made it particularly hard to reason about and implement efficient resharding. For example in a shard layout with boundary accounts [`b`, `d`] the account space is cleanly divided into three shards, each spanning a contiguous range and account ids: @@ -108,11 +108,11 @@ For example in a shard layout with boundary accounts [`b`, `d`] the account spac * 1 - `b:d` * 2 - `d:` -Now if we add a fixed shard `f` to the same shard layout, then any we'll have 4 shards but neither is contiguous. Accounts such as `aaa.f`, `ccc.f`, `eee.f` that would otherwise belong to shards 0, 1 and 2 respectively are now all assigned to the fixed shard and create holes in the shard account ranges. +Now if we add a fixed shard `f` to the same shard layout, then any we'll have 4 shards but neither is contiguous. Accounts such as `aaa.f`, `ccc.f`, `eee.f` that would otherwise belong to shards 0, 1 and 2 respectively are now all assigned to the fixed shard and create holes in the shard account ranges. -It's also worth noting that there is no benefit to having accounts colocated in the same shard. Any transaction or receipt is treated the same way regardless of crossing shard boundary. +It's also worth noting that there is no benefit to having accounts colocated in the same shard. Any transaction or receipt is treated the same way regardless of crossing shard boundary. -This was implemented ahead of this NEP and the fixed shards feature was **removed**. +This was implemented ahead of this NEP and the fixed shards feature was **removed**. ### Garbage collection @@ -120,15 +120,15 @@ In epoch T+2 once resharding is completed, we can delete the trie state and the ### Transaction pool -The transaction pool is sharded i.e. it groups transactions by the shard where each transaction should be converted to a receipt. The transaction pool was previously sharded by the ShardId. Unfortunately ShardId is insufficient to correctly identify a shard across a resharding event as ShardIds change domain. The transaction pool was migrated to group transactions by ShardUId instead, and a transaction pool resharding was implemented to reassign transaction from parent shard to children shards right before the new shard layout takes effect. The ShardUId contains the version of the shard layout which allows differentiating between shards in different shard layouts. +The transaction pool is sharded i.e. it groups transactions by the shard where each transaction should be converted to a receipt. The transaction pool was previously sharded by the ShardId. Unfortunately ShardId is insufficient to correctly identify a shard across a resharding event as ShardIds change domain. The transaction pool was migrated to group transactions by ShardUId instead, and a transaction pool resharding was implemented to reassign transaction from parent shard to children shards right before the new shard layout takes effect. The ShardUId contains the version of the shard layout which allows differentiating between shards in different shard layouts. -This was implemented ahead of this NEP and the transaction pool is now fully **migrated** to ShardUId. +This was implemented ahead of this NEP and the transaction pool is now fully **migrated** to ShardUId. ## Alternatives ### Why is this design the best in the space of possible designs? -This design is simple, robust, safe, and meets all requirements. +This design is simple, robust, safe, and meets all requirements. ### What other designs have been considered and what is the rationale for not choosing them? @@ -138,17 +138,17 @@ This design is simple, robust, safe, and meets all requirements. * Changing the trie structure to have the account id first and type of record later. This change would allow for much faster resharding by only iterating over the nodes on the boundary. This approach has two major drawbacks without providing too many benefits over the previous approach of splitting by each trie record type. 1) It would require a massive migration of trie. 2) We would need to maintain the old and the new trie structure forever. -* Changing the storage structure by having the storage key to have the format of `account_id.node_hash`. This structure would make it much easier to split the trie on storage level because the children shards are simple sub-ranges of the parent shard. Unfortunately we found that the migration would not be feasible. -* Changing the storage structure by having the key format as only node_hash and dropping the ShardUId prefix. This is a feasible approach but it adds complexity to the garbage collection and data deletion, specially when nodes would start tracking only one shard. We opted in for the much simpler one by using the existing scheme of prefixing storage entries by shard uid. +* Changing the storage structure by having the storage key to have the format of `account_id.node_hash`. This structure would make it much easier to split the trie on storage level because the children shards are simple sub-ranges of the parent shard. Unfortunately we found that the migration would not be feasible. +* Changing the storage structure by having the key format as only node_hash and dropping the ShardUId prefix. This is a feasible approach but it adds complexity to the garbage collection and data deletion, specially when nodes would start tracking only one shard. We opted in for the much simpler one by using the existing scheme of prefixing storage entries by shard uid. #### Other considerations * Dynamic Resharding - we have decided to not implement the full dynamic resharding at this time. Instead we hardcode the shard layout and schedule it manually. The reasons are as follows: - * We prefer incremental process of introducing resharding to make sure that it is robust and reliable, as well as give the community the time to adjust. - * Each resharding increases the potential total load on the system. We don't want to allow it to grow until full sharding is in place and we can handle that increase. + * We prefer incremental process of introducing resharding to make sure that it is robust and reliable, as well as give the community the time to adjust. + * Each resharding increases the potential total load on the system. We don't want to allow it to grow until full sharding is in place and we can handle that increase. * Extended shard layout adjustments - we have decided to only implement shard splitting and not implement any other operations. The reasons are as follows: * In this iteration we only want to perform splitting. - * The extended adjustments are currently not justified. Both merging and boundary moving may be useful in the future when the traffic patterns change and some shard become underutilized. In the nearest future we only predict needing to reduce the size of the heaviest shards. + * The extended adjustments are currently not justified. Both merging and boundary moving may be useful in the future when the traffic patterns change and some shard become underutilized. In the nearest future we only predict needing to reduce the size of the heaviest shards. ### What is the impact of not doing this? @@ -159,7 +159,7 @@ We need resharding in order to scale up the system. Without resharding eventuall There are two known issues in the integration of resharding and state sync: * When syncing the state for the first epoch where the new shard layout is used. In this case the node would need to apply the last block of the previous epoch. It cannot be done on the children shard as on chain the block was applied on the parent shards and the trie related gas costs would be different. -* When generating proofs for incoming receipts. The proof for each of the children shards contains only the receipts of the shard but it's generated on the parent shard layout and so may not be verified. +* When generating proofs for incoming receipts. The proof for each of the children shards contains only the receipts of the shard but it's generated on the parent shard layout and so may not be verified. In this NEP we propose that resharding should be rolled out first, before any real dependency on state sync is added. We can then safely roll out the resharding logic and solve the above mentioned issues separately. We believe at least some of the issues can be mitigated by the implementation of new pre-state root and chunk execution design. @@ -167,7 +167,7 @@ In this NEP we propose that resharding should be rolled out first, before any re The Stateless Validation requires that chunk producers provide proof of correctness of the transition function from one state root to another. That proof for the first block after the new shard layout takes place will need to prove that the entire state split was correct as well as the state transition. -In this NEP we propose that resharding should be rolled out first, before stateless validation. We can then safely roll out the resharding logic and solve the above mentioned issues separately. This issue was discussed with the stateless validation experts and we are cautiously optimistic that the integration will be possible. The most concerning part is the proof size and we believe that it should be small enough thanks to the resharding touching relatively small number of trie nodes - on the order of the depth of the trie. +In this NEP we propose that resharding should be rolled out first, before stateless validation. We can then safely roll out the resharding logic and solve the above mentioned issues separately. This issue was discussed with the stateless validation experts and we are cautiously optimistic that the integration will be possible. The most concerning part is the proof size and we believe that it should be small enough thanks to the resharding touching relatively small number of trie nodes - on the order of the depth of the trie. ## Future fast-followups @@ -183,7 +183,7 @@ As mentioned above under 'Integration with Stateless Validation' section, the in ### Further reshardings -This NEP introduces both an implementation of resharding and an actual resharding to be done in the production networks. Further reshardings can also be performed in the future by adding a new shard layout and setting the shard layout for the desired protocol version in the `AllEpochConfig`. +This NEP introduces both an implementation of resharding and an actual resharding to be done in the production networks. Further reshardings can also be performed in the future by adding a new shard layout and setting the shard layout for the desired protocol version in the `AllEpochConfig`. ### Dynamic resharding @@ -194,7 +194,7 @@ As noted above, dynamic resharding is out of scope for this NEP and should be im ### Extended shard layout adjustments -In this NEP we only propose supporting splitting shards. This operation should be more than sufficient for the near future but eventually we may want to add support for more sophisticated adjustments such as: +In this NEP we only propose supporting splitting shards. This operation should be more than sufficient for the near future but eventually we may want to add support for more sophisticated adjustments such as: * Merging shards together * Moving the boundary account between two shards @@ -208,7 +208,7 @@ In the future, we would like to potentially change the schema in a way such that ### Other useful features * Removal of shard uids and introducing globally unique shard ids -* Account colocation for low latency across account call - In case we start considering synchronous execution environment, colocating associated accounts (e.g. cross contract call between them) in the same shard can increase the efficiency +* Account colocation for low latency across account call - In case we start considering synchronous execution environment, colocating associated accounts (e.g. cross contract call between them) in the same shard can increase the efficiency * Shard purchase/reservation - When someone wants to secure entirety of limitation on a single shard (e.g. state size limit), they can 'purchase/reserve' a shard so it can be dedicated for them (similar to how Aurora is set up) ## Consequences @@ -233,7 +233,7 @@ In the future, we would like to potentially change the schema in a way such that ### Backwards Compatibility -Any light clients, tooling or frameworks external to nearcore that have the current shard layout or the current number of shards hardcoded may break and will need to be adjusted in advance. The recommended way for fixing it is querying an RPC node for the shard layout of the relevant epoch and using that information in place of the previously hardcoded shard layout or number of shards. The shard layout can be queried by using the `EXPERIMENTAL_protocol_config` rpc endpoint and reading the `shard_layout` field from the result. A dedicated endpoint may be added in the future as well. +Any light clients, tooling or frameworks external to nearcore that have the current shard layout or the current number of shards hardcoded may break and will need to be adjusted in advance. The recommended way for fixing it is querying an RPC node for the shard layout of the relevant epoch and using that information in place of the previously hardcoded shard layout or number of shards. The shard layout can be queried by using the `EXPERIMENTAL_protocol_config` rpc endpoint and reading the `shard_layout` field from the result. A dedicated endpoint may be added in the future as well. Within nearcore we do not expect anything to break with this change. Yet, shard splitting can introduce additional complexity on replayability. For instance, as target shard of a receipt and belonging shard of an account can change with shard splitting, shard splitting must be replayed along with transactions at the exact epoch boundary. diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 4b10254cc..f32539d0c 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -2,7 +2,7 @@ NEP: 509 Title: Stateless validation Stage 0 Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong -Status: Draft +Status: Final DiscussionsTo: https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing, https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing Type: Protocol Version: 1.0.0 @@ -14,10 +14,10 @@ LastUpdated: 2023-09-19 The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation, instead of the traditionally proposed approach of fraud proof and state rollback. -The fundamental idea is that validators do not need to have state locally to validate chunks. +The fundamental idea is that validators do not need to have state locally to validate chunks. * Under stateless validation, the responsibility of a chunk producer extends to packaging transactions and receipts and annotating them with state witnesses. This extended role will be called "chunk proposers". -* The state witness of a chunk is defined to be a subset of the trie state, alongside its proof of inclusion in the trie, that is needed to execute a chunk. A state witness allows anyone to execute the chunk without having the state of its shard locally. +* The state witness of a chunk is defined to be a subset of the trie state, alongside its proof of inclusion in the trie, that is needed to execute a chunk. A state witness allows anyone to execute the chunk without having the state of its shard locally. * Then, at each block height, validators will be randomly assigned to a shard, to validate the state witness for that shard. Once a validator receives both a chunk and its state witness, it verifies the state transition of the chunk, signs a chunk endorsement and sends it to the block producer. This is similar to, but separate from, block approvals and consensus. * The block producer waits for sufficient chunk endorsements before including a chunk into the block it produces, or omits the chunk if not enough endorsements arrive in time. @@ -62,7 +62,7 @@ And the flow goes on for heights H+1, H+2, etc. The "induction base" is at genes One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: - + ### Design after NEP-509 * Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header that is being produced. @@ -131,7 +131,7 @@ With stateless validation, this structure must change for several reasons: * Chunk production is the most resource consuming activity. * *Only* chunk production needs state in memory while other responsibilities can be completed via acquiring state witness -* Chunk production does not have to be performed by all validators. +* Chunk production does not have to be performed by all validators. Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: @@ -142,7 +142,7 @@ if index(validator) < 100: roles(validator).append("chunk validator") ``` -The more stake validator has, the more **heavy** work it will get assigned. We expect that validators with higher stakes have more powerful hardware. +The more stake validator has, the more **heavy** work it will get assigned. We expect that validators with higher stakes have more powerful hardware. With stateless validation, relative heaviness of the work changes. Comparing to the current order "block production" > "chunk production", the new order is "chunk production" > "block production" > "chunk validation". Shards are equally split among chunk producers: as in Mainnet on 12 Jun 2024 we have 6 shards, each shard would have ~16 chunk producers assigned. @@ -168,10 +168,10 @@ Reward for each validator is defined as `total_epoch_reward * validator_relative So, the actual reward never exceeds total reward, and when everyone does perfect work, they are equal. For the context of the NEP, it is enough to assume that `work_quality_ratio = avg_{role}({role}_quality_ratio)`. So, if node is both a block and chunk producer, we compute quality for each role separately and then take average of them. - + When epoch is finalized, all block headers in it uniquely determine who was expected to produce each block and chunk. Thus, if we define quality ratio for block producer as `produced_blocks/expected_blocks`, everyone is able to compute it. -Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. +Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. Ideally, we could compute quality for chunk validator as `produced_endorsements/expected_endorsements`. Unfortunately, we won't do it in Stage 0 because: @@ -182,11 +182,11 @@ Ideally, we could compute quality for chunk validator as `produced_endorsements/ So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. It has clear drawbacks though: -* chunk validators are not incentivized to validate the chunks, given they will be rewarded the same in either case; +* chunk validators are not incentivized to validate the chunks, given they will be rewarded the same in either case; * if chunks are not produced at all, chunk validators will also be impacted. We plan to address them in the future releases. - + #### Kickouts In addition to that, if node performance is too poor, we want a mechanism to kick it out of the validator list, to ensure healthy protocol performance and validator rotation. @@ -203,7 +203,7 @@ if validator is chunk producer and chunk_producer_quality_ratio < 0.8: For chunk validator, we apply absolutely the same formula. However, because: -* the formula doesn't count endorsements explicitly +* the formula doesn't count endorsements explicitly * for chunk producers it kind of just makes chunk production condition stronger without adding value we apply it to nodes which **only validate chunks**. So, we add this line: @@ -215,12 +215,12 @@ if validator is only chunk validator and chunk_validator_quality_ratio < 0.8: As we pointed out above, current formula `chunk_validator_quality_ratio` is problematic. Here it brings even a bigger issue: if chunk producers don't produce chunks, chunk validators will be kicked out as well, which impacts network stability. -This is another reason to come up with the better formula. +This is another reason to come up with the better formula. #### Shard assignment -As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. -This is a **significant difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. +As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. +This is a **significant difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. The most challenging part of becoming a chunk producer for a shard is to download most recent shard state within previous epoch. This is called "state sync". Unfortunately, as of now, state sync is centralised on published snapshots, which is a major point of failure, until we don't have decentralised state sync. @@ -229,12 +229,12 @@ Because of that, we make additional change: if node was a chunk producer for som This way, we minimise number of required state syncs at each epoch. The exact algorithm needs a thorough description to satisfy different edge cases, so we will just leave a link to full explanation: https://github.com/near/nearcore/issues/11213#issuecomment-2111234940. - -### ChunkStateWitness + +### ChunkStateWitness The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). Let's construct it sequentially together with explaining why every field is needed. Start from simple data: - + ```rust pub struct ChunkStateWitness { pub chunk_producer: AccountId, @@ -246,10 +246,10 @@ pub struct ChunkStateWitness { What is needed to prove `ShardChunkHeader`? -The key function we have in codebase is [validate_chunk_with_chunk_extra_and_receipts_root](https://github.com/near/nearcore/blob/c2d80742187d9b8fc1bb672f16e3d5c144722742/chain/chain/src/validate.rs#L141). +The key function we have in codebase is [validate_chunk_with_chunk_extra_and_receipts_root](https://github.com/near/nearcore/blob/c2d80742187d9b8fc1bb672f16e3d5c144722742/chain/chain/src/validate.rs#L141). The main arguments there are `prev_chunk_extra: &ChunkExtra` which stands for execution result of previous chunk, and `chunk_header`. The most important field for `ShardChunkHeader` is `prev_state_root` - consider latest implementation `ShardChunkHeaderInnerV3`. It stands for state root resulted from updating shard for the previous block, which means applying previous chunk if there is no missing chunks. -So, chunk validator needs some way to run transactions and receipts from the previous chunk. Let's call it a "main state transition" and add two more fields to state witness: +So, chunk validator needs some way to run transactions and receipts from the previous chunk. Let's call it a "main state transition" and add two more fields to state witness: ```rust /// The base state and post-state-root of the main transition where we @@ -287,11 +287,11 @@ Receipts are internal messages, resulting from transaction execution, sent betwe However, each receipt is an execution outcome of some transaction or other parent receipt, executed in some previous chunk. For every chunk, we conveniently store `prev_outgoing_receipts_root` which is a Merkle hash of all receipts sent to other shards resulting by execution of this chunk. So, for every receipt, there is a proof of its generation in some parent chunk. If there are no missing chunk, then it's enough to consider chunks from previous block. -So we add another field: +So we add another field: ```rust /// Non-strict superset of the receipts that must be applied, along with - /// information that allows these receipts to be verifiable against the + /// information that allows these receipts to be verifiable against the /// blockchain history. pub source_receipt_proofs: HashMap, ``` @@ -304,7 +304,7 @@ Unfortunately, production and inclusion of any chunk **cannot be guaranteed**: * chunk validators may not generate 2/3 endorsements; * block producer may not receive enough information to include chunk. -Let's handle this case as well. +Let's handle this case as well. First, each chunk producer needs not just to prove main state transition, but also all state transitions for latest missing chunks: ```rust @@ -318,23 +318,23 @@ First, each chunk producer needs not just to prove main state transition, but al pub implicit_transitions: Vec, ``` -Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. +Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. Field structure doesn't change, but we need to carefully pick range of set of source chunks, so different subsets will cover all source receipts without intersection. -Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the block that contains the last new chunk of shard S before B2. +Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the block that contains the last new chunk of shard S before B2. Then, we will define set of blocks B as the contiguous subsequence of blocks B1 (EXCLUSIVE) to B2 (inclusive) in this chunk's chain (i.e. the linear chain that this chunk's parent block is on). Lastly, source chunks are all chunks included in blocks from B. - + The last caveat is **new** transactions introduced by chunk with `chunk_header`. As chunk header introduces `tx_root` for them, we need to check validity of this field as well. If we don't do it, malicious chunk producer can include invalid transaction, and if it gets its chunk endorsed, nodes which track the shard must either accept invalid transaction or refuse to process chunk, but the latter means that shard will get stuck. To validate new `tx_root`, we also need Merkle partial state to validate sender' balances, access keys, nonces, etc., which leads to two last fields to be added: - + ```rust pub new_transactions: Vec, pub new_transactions_validation_state: PartialState, ``` -The logic to produce `ChunkStateWitness` is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/state_witness_producer.rs#L79). +The logic to produce `ChunkStateWitness` is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/state_witness_producer.rs#L79). Itself, it requires some minor changes to the logic of applying chunks, related to generating `ChunkStateTransition::base_state`. It is controlled by [this line](https://github.com/near/nearcore/blob/dc03a34101f77a17210873c4b5be28ef23443864/chain/chain/src/runtime/mod.rs#L977), which causes all nodes read during applying chunk to be put inside `TrieRecorder`. After applying chunk, its contents are saved to `StateTransitionData`. @@ -377,9 +377,9 @@ Based on target number of mandates and total chunk validators stake, [here](http All the mandates are stored in new version of `EpochInfo` `EpochInfoV4` in [validator_mandates](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L775) field. After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L1224) is called to return `ChunkValidatorStakeAssignment`. It is `Vec>` where s-th element corresponds to s-th shard in the epoch, contains ids of all chunk validator for that height and shard, alongside with its total mandate stake assigned to that shard. -`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. If there are no more than 1/3 malicious validators, then by Chernoff bound the probability that at least one shard is corrupted is small enough. **This is a reason why we can split validators among shards and still rely on basic consensus assumption**. +`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. If there are no more than 1/3 malicious validators, then by Chernoff bound the probability that at least one shard is corrupted is small enough. **This is a reason why we can split validators among shards and still rely on basic consensus assumption**. -This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. +This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. ### Size limits @@ -527,35 +527,35 @@ It is also worth mentioning that large state witness size makes witness distribu ## Alternatives -The only real alternative that was considered is the original nightshade proposal. The full overview of the differences can be found in the revised nightshade whitepaper at https://near.org/papers/nightshade. +The only real alternative that was considered is the original nightshade proposal. The full overview of the differences can be found in the revised nightshade whitepaper at https://near.org/papers/nightshade. ## Future possibilities * Integration with ZK allowing to get rid of large state witness distribution. If we treat state witness as a proof and ZK-ify it, anyone can validate that state witness indeed proves the new chunk header with much lower effort. Complexity of actual proof generation and computation indeed increases, but it can be distributed among chunk producers, and we can have separate concept of finality while allowing generic users to query optimistic chunks. -* Integration with resharding to further increase the number of shards and the total throughput. -* The sharding of non-validating nodes and services. There are a number of services that may benefit from tracking only a subset of shards. Some examples include the RPC, archival and read-RPC nodes. +* Integration with resharding to further increase the number of shards and the total throughput. +* The sharding of non-validating nodes and services. There are a number of services that may benefit from tracking only a subset of shards. Some examples include the RPC, archival and read-RPC nodes. ## Consequences ### Positive -* The validator nodes will need to track at most one shard. +* The validator nodes will need to track at most one shard. * The state will be held in memory making the chunk application much faster. -* The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. -* Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. +* The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. +* Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. ### Neutral -* The current approach to resharding will need to be revised to support generating state witness. -* The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. +* The current approach to resharding will need to be revised to support generating state witness. +* The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. ### Negative * The network bandwidth and memory hardware requirements will increase. - * The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. - * The remaining validators will need to participate in state witness distribution. + * The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. + * The remaining validators will need to participate in state witness distribution. * Additional limits will be put on the size of transactions, receipts and, more generally, cross shard communication. -* The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. +* The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. ### Backwards Compatibility diff --git a/neps/nep-0514.md b/neps/nep-0514.md index 10a3bae80..a5a0fee48 100644 --- a/neps/nep-0514.md +++ b/neps/nep-0514.md @@ -2,7 +2,7 @@ NEP: 514 Title: Reducing the number of Block Producer Seats in `testnet` Authors: Nikolay Kurtov -Status: New +Status: Final DiscussionsTo: https://github.com/nearprotocol/neps/pull/514 Type: Protocol Version: 1.0.0 @@ -10,7 +10,6 @@ Created: 2023-10-25 LastUpdated: 2023-10-25 --- - ## Summary This proposal aims to adjust the number of block producer seats on `testnet` in diff --git a/neps/nep-519-yield-execution.md b/neps/nep-0519.md similarity index 99% rename from neps/nep-519-yield-execution.md rename to neps/nep-0519.md index e636d0df3..6afc9edd4 100644 --- a/neps/nep-519-yield-execution.md +++ b/neps/nep-0519.md @@ -2,7 +2,7 @@ NEP: 519 Title: Yield Execution Authors: Akhi Singhania ; Saketh Are -Status: Draft +Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/519 Type: Protocol Version: 0.0.0 diff --git a/neps/nep-0536.md b/neps/nep-0536.md index 6f94d751a..5bc849a0c 100644 --- a/neps/nep-0536.md +++ b/neps/nep-0536.md @@ -2,7 +2,7 @@ NEP: 536 Title: Reduce the number of gas refunds Authors: Evgeny Kuzyakov , Bowen Wang -Status: New +Status: Final DiscussionsTo: https://github.com/near/NEPs/pull/536 Type: Protocol Version: 1.0.0 diff --git a/neps/nep-0539.md b/neps/nep-0539.md new file mode 100644 index 000000000..3328c253e --- /dev/null +++ b/neps/nep-0539.md @@ -0,0 +1,788 @@ +--- +NEP: 539 +Title: Cross-Shard Congestion Control +Authors: Waclaw Banasik , Jakob Meier +Status: Final +DiscussionsTo: https://github.com/nearprotocol/neps/pull/539 +Type: Protocol +Version: 1.0.0 +Created: 2024-03-21 +LastUpdated: 2024-05-15 +--- + +## Summary + +We propose to limit the transactions and outgoing receipts included in each +chunk. Limits depend on the congestion level of that receiving shard and are to +be enforced by the protocol. + +Congestion is primarily tracked in terms of the total gas of all delayed receipts. +Chunk producers must ensure they stop accepting new transactions if the receiver +account is on a shard with a full delayed receipts queue. + +Forwarding of receipts that are not directly produced by a transaction, namely +cross-contract calls and delegated receipts, is limited by the receiver's overall +congestion level. This includes the gas of delayed receipts and the gas of +receipts that have not been forwarded, yet, due to congestion control +restrictions. Additionally, the memory consumption of both receipt types can +also cause congestion. + +This proposal extends the local congestion control rules already in place. It +keeps the transaction pool size limit as is but replaces the old delayed receipt +count limit with limits on gas and size of the receipts. + +## Motivation + +We want to guarantee the Near Protocol blockchain operates stably even during +congestion. + +Today, when users send transactions at a higher rate than what the network can +process, receipts accumulate without limits. This leads to unlimited memory +consumption on chunk producers' and validators' machines. Furthermore, the delay +for a transaction from when it is accepted to when it finishes execution becomes +larger and larger, as the receipts need to queue behind those already in the +system. + +First attempts to limit the memory consumption have been added without protocol +changes. This is known as "local congestion control" and can be summarized in +two rules: + +- Limit the transaction pool to 100 MB. + https://github.com/near/nearcore/pull/9172 +- Once we have accumulated more than 20k delayed receipts in a shard, + chunk-producers for that shard stop accepting more transactions. + https://github.com/near/nearcore/pull/9222 + +But these rules do not put any limits on what other shards would accept. For +example, when a particular contract is popular, the contract's shard will +eventually stop accepting new transactions, but all other shards will keep +accepting more and more transactions that produce receipts for the popular +contract. Therefore the number of delayed receipts keeps growing indefinitely. + +Cross-shard congestion control addresses this issue by stopping new transactions +at the source and delaying receipt forwarding when the receiving shard has +reached its congestion limit. + +## Specification + +The proposal adds fields to chunks headers, adds a new trie column, changes the +transaction selection rules, and changes the chunk execution flow. After the +concepts section below, the next four sections specify each of these changes in +more detail. + +### Concepts + +Below are high-level description of important concepts to make the following +sections a bit easier to digest. + +**Delayed receipts** are all receipts that were ready for execution in the +previous chunk but were delayed due to gas or compute limits. They are stored in +the delayed receipt queue, which itself is stored in the state of the trie. +There is exactly one delayed receipts queue per shard. + +**Outgoing receipts** are all newly produced receipts as a result of executing +receipts or when converting a transaction to non-local receipts. In the absence +of congestion, they are all stored in the output of applying receipts, as a +simple list of receipts. + +The **outgoing receipts buffer** is a data structure added by cross-shard +congestion control. Each shard has one instance of it in the state trie for +every other shard. We use this buffer to store outgoing receipts temporarily +when reaching receipt forwarding limits. + +**Receipt limits** are measured in gas and size. Gas in this context refers to +the maximum amount of gas that could be burn when applying the receipt. The +receipt size is how much space it takes in memory, measured in bytes. + +The **congestion level** is an indicator between 0 and 1 that determines how +strong congestion prevention measures should be. The maximum congestion measured +is reached at congestion level 1. This value is defined separately for each +shard and computed as the maximum value of the following congestion types. + +- **Incoming congestion** increases linearly with the amount of gas in + delayed receipts. +- **Outgoing congestion** increases linearly with the amount of gas in any of + the outgoing buffers of the shard. +- **Memory congestion** increases linearly with the total size of all delayed + and buffered outgoing receipts. +- **Missed chunk congestion** rises linearly with the number of missed chunks + since the last successful chunk, measure in block height difference. + +### Chunk header changes + +We change the chunk header to include congestion information, adding four +indicators. + +```rust +/// sum of gas of all receipts in the delayed receipts queue, at the end of chunk execution +delayed_receipts_gas: u128, +/// sum of gas of all receipts in all outgoing receipt buffers, at the end of chunk execution +buffered_receipts_gas: u128, +/// sum of all receipt sizes in the delayed receipts queue and all outgoing buffers +receipt_bytes: u64, +/// if the congestion level is 1.0, the only shard that can forward receipts to this shard in the next chunk +/// not relevant if the congestion level is < 1.0 +allowed_shard: u16, +``` + +The exact header structure and reasons for the particular integer sizes are +described in more details in the [reference +implementation](#efficiently-computing-the-congestion-information) section. +Usage of these fields is described in [Changes to chunk +execution](#changes-to-chunk-execution). + +This adds 42 bytes to the chunk header, increasing it from 374 bytes up to 416 +bytes in the borsh representation (assuming no validator proposals are +included.) This in turn increases the block size by 42 bytes *per shard*, as all +chunk headers are fully included in blocks. + +Including all this information in the chunk header enables efficient validation. +Using the previous chunk header (or alternatively, the state), combined with the +list of receipts applied and forwarded, a validator can check that the +congestion rules described in this NEP are fulfilled. + +### Changes to transaction selection + +We change transaction selection to reject new transactions when the system is +congested, to reduce to total workload in the system. + +Today, transactions are taken from the chunk producer's pool until `tx_limit` is +reached, where `tx_limit` is computed as follows. + +```python +# old +tx_limit = 500 Tgas if len(delayed_receipts) < 20_000 else 0 +``` + +We replace the formula for the transaction limit to depend on the +`incoming_congestion` variable (between 0 and 1) computed in the previous chunk +of the same shard: + +```python +# new +MIN_TX_GAS = 20 Tgas +MAX_TX_GAS = 500 Tgas +tx_limit = mix(MAX_TX_GAS, MIN_TX_GAS, incoming_congestion) +``` + +This smoothly limits the acceptance of new work, to prioritize reducing the +backlog of delayed receipts. + +In the pseudo code above, we borrow the [`mix`](https://docs.gl/sl4/mix) +function from GLSL for linear interpolation. + +> `mix(x, y, a)` +> +> `mix` performs a linear interpolation between x and y using a to weight between +> them. The return value is computed as $x \times (1 - a) + y \times a$. + +More importantly, we add a more targeted rule to reject all transactions *targeting* +a shard with a congestion level above a certain threshold. + +```python +def filter_tx(tx): + REJECT_TX_CONGESTION_THRESHOLD = 0.25 + if congestion_level(tx.receiver_shard_id) > REJECT_TX_CONGESTION_THRESHOLD + tx.reject() + else + tx.accept() +``` + +Here, `congestion_level()` is the maximum of incoming, outgoing, memory, and +missed chunk congestion. + +This stops (some) new incoming work at the source, when a shard is using too +much memory to store unprocessed receipts, or if there is already too much work +piled up for that shard. + +Chunk validators must validate that the two rules above are respected in a +produced chunk. + +### Changes to chunk execution + +We change chunk execution to hold back receipt forwarding to congested shards. +This has two effects. + +1. It prevents the memory consumption of the congested shard from growing at the + expense of buffering these pending receipts on the outgoing shards. +2. When user demand is consistently higher than what the system can handle, this + mechanism lets backpressure propagate shard-by-shard until it reaches the shards + responsible for accepting too many receipts and causes transaction + filtering to kick in. + +To accomplish this, we add 3 new steps to chunk execution (enumerated as 1, 2, 6 +below) and modify how outgoing receipts are treated in the transaction +conversion step (3) and in the receipts execution step (4). + +The new chunk execution then follows this order. + +1. (new) Read congestion information for *all* shards from the previous chunk headers. + + ```rust + // congestion info for each shard, as computed in the last included chunk of the shard + { + delayed_receipts_gas: u128, + buffered_receipts_gas: u128, + receipt_bytes: u64, + allowed_shard: u16, + // extended congestion info, as computed from the latest block header + missed_chunks_count: u64 + } + ``` + +2. (new) Compute bandwidth limits to other shards based on the congestion information. + The formula is: + + ```python + for receiver in other_shards: + MAX_CONGESTION_INCOMING_GAS = 20 Pgas + incoming_congestion = delayed_receipts_gas[receiver] / MAX_CONGESTION_INCOMING_GAS + + MAX_CONGESTION_OUTGOING_GAS = 2 Pgas + outgoing_congestion = buffered_receipts_gas[receiver] / MAX_CONGESTION_OUTGOING_GAS + + MAX_CONGESTION_MEMORY_CONSUMPTION = 1000 MB + memory_congestion = receipt_bytes[receiver] / MAX_CONGESTION_MEMORY_CONSUMPTION + + MAX_CONGESTION_MISSED_CHUNKS = 10 + missed_chunk_congestion = missed_chunks_count[receiver] / MAX_CONGESTION_MISSED_CHUNKS + + congestion = max(incoming_congestion, outgoing_congestion, memory_congestion, missed_chunk_congestion) + + if congestion >= 1.0: + # Maximum congestion: reduce to minimum speed + if current_shard == allowed_shard[receiver]: + outgoing_gas_limit[receiver] = 1 Pgas + else: + outgoing_gas_limit[receiver] = 0 + else: + # Green or Amber + # linear interpolation based on congestion level + MIN_GAS_FORWARDING = 1 Pgas + MAX_GAS_FORWARDING = 300 Pgas + outgoing_gas_limit[receiver] + = mix(MAX_GAS_FORWARDING, MIN_GAS_FORWARDING, congestion) + ``` + +3. (new) Drain receipts in the outgoing buffer from the previous round + - Subtract `receipt.gas()` from `outgoing_gas_limit[receipt.receiver]` for + each receipt drained. + - Keep receipts in the buffer if the gas limit would be negative. + - Subtract `receipt.gas()` from `outgoing_congestion` and `receipt.size()` + from `receipt_bytes` for the local shard for every forwarded receipt. + - Add the removed receipts to the outgoing receipts of the new chunk. +4. Convert all transactions to receipts included in the chunk. + - Local receipts, which are receipts where the sender account id is equal to + the receiver id, are set aside as local receipts for step 5. + - Non-local receipts up to `outgoing_gas_limit[receipt.receiver]` for the + respective shard go to the outgoing receipts list of the chunk. + - (new) Non-local receipts above `outgoing_gas_limit[receipt.receiver]` for + the respective shard go to the outgoing receipts buffer. + - (new) For each receipt added to the outgoing buffer, add `receipt.gas()` + to `outgoing_congestion` and `receipt.size()` to `receipt_bytes` for + the local shard. +5. Execute receipts in the order of `local`, `delayed`, `incoming`, `yield-resume time-out`. + - Don't stop before all receipts are executed or more than 1000 Tgas have + been burnt. Burnt gas includes the burnt gas from step 4. + - Outgoing receipts up to what is left in + `outgoing_gas_limit[receipt.receiver]` per shard (after step 3) go to the + outgoing receipts list of the chunk. + - (new) Outgoing receipts above `outgoing_gas_limit[receipt.receiver]` + go to the outgoing receipts buffer. + - (new) For each delayed executed receipt, remove `receipt.gas()` from + `incoming_congestion` and `receipt.size()` from `receipt_bytes`. +6. Remaining local or incoming receipts are added to the end of the `delayed` + receipts queue. + - (new) For each receipt added to the delayed receipts queue, add + `receipt.gas()` to `incoming_congestion` and `receipt.size()` to + `receipt_bytes`. +7. (new) Write own congestion information into the result, to be included in the + next chunk header. + - If the congestion level is >= 1.0, the `allowed_shard` can be chosen + freely by the chunk producer. Selecting the own shard means nobody can + send. The reference implementations uses round-robin assignment of all + other shards. Further optimization can be done without requiring protocol + changes. + - If the congestion level is <= 1.0, the `allowed_shard` value does not + affect congestion control. But chunk producer must set it to the own shard + in this case. + +In the formula above, the receipt gas and the receipt size are defined as: + +```python +def gas(receipt): + return receipt.attached_gas + receipt.exec_gas + +def size(receipt): + return len(borsh(receipt)) +``` + +### Changes to Trie + +We store the outgoing buffered receipts in the trie, similar to delayed receipts +but in their own separate column. But instead of a single queue per shard, we add +one queue for each other shard at the current sharding layout. + +We add two trie columns: + +- `BUFFERED_RECEIPT_INDICES: u8 = 13;` +- `BUFFERED_RECEIPT: u8 = 14` + +The `BUFFERED_RECEIPT_INDICES` column only has one value, which stores a +borsh-serialized instance of `BufferedReceiptIndices` defines as follows: + +```rust +pub struct BufferedReceiptIndices { + pub shard_buffer_indices: BTreeMap, +} + +pub struct ShardBufferedReceiptIndices { + // First inclusive index in the queue. + pub first_index: u64, + // Exclusive end index of the queue + pub next_available_index: u64, +} +``` + +The `BUFFERED_RECEIPT` column stores receipts keyed by +`TrieKey::BufferedReceipt{ receiving_shard: ShardId, index: u64 }`. + +The `BufferedReceiptIndices` map defines which queues exist, which changes +during resharding. For each existing queue, all receipts in the range +`[first_index, next_available_index)` (inclusive start, exclusive end) must +exist under the key with the corresponding shard. + + +### Notes on parameter fine-tuning + +Below are the reasons why each parameter is set to the specific value given above. + +For more details, a spreadsheet with the full analysis can be found here: +https://docs.google.com/spreadsheets/d/1Vt_-sgMdX1ncYleikYY8uFID_aG9RaqJOqaVMLQ37tQ/edit#gid=0 + +#### Queue sizes + +The parameters are chosen to strike a balance between guarantees for short +queues and utilization. 20 PGas delayed receipts means that incoming receipts +have to wait at most 20 chunks to be applied. And it can guarantee 100% +utilization as long as the ratio between burnt and attached gas in receipts is +above 1 to 20. + +A shorter delayed queue would result in lower delays but in our model +simulations, we saw reduced utilization even in simple and balanced workloads. + +The 1 GB of memory is a target value for the control algorithm to try and stay +below. With receipts in the normal range of sizes seen in today's traffic, we +should never even get close to 1 GB. But the protocol allows a single receipt to +be multiple MBs. In those cases, a limit of 1 GB still gives us almost 100% +utilization but prevents queues from growing larger than what validators can +keep in memory. + +#### Receipt forwarding limits + +`MIN_GAS_FORWARDING = 1 Pgas` and `MAX_GAS_FORWARDING = 300 Pgas` give us a +large range to smooth out how much should be forwarded to other shards. +Depending on the burnt to attached gas ratio of the workload, it will settle at +different values for each directed shard pair. This gives the algorithm +adaptability to many workload patterns. + +For the forwarding to work smoothly, we need a bit of an outgoing buffer queue. +We found in simulations that `MAX_CONGESTION_OUTGOING_GAS = 2 Pgas` is enough +for the forwarding limit to settle in the perfect spot before we are restricted +by transaction rejection. Higher values did not yield better results but it does +increase delays in some cases, hence we propose 2 Pgas. + +#### Limiting new transactions + +The remaining parameters work together to adapt how much new workload we accept +in the system, based on how congested the chain is already. + +`REJECT_TX_CONGESTION_THRESHOLD = 0.25` defines how quickly we start rejecting +new workload to a shard. Combined with the 20 PGas limit on the delayed receipts +queue, we only reject new work if we have at least 5 PGas excess workload sent +to that shard already. + +This is more aggressive than other mechanisms simply because rejecting more +workload to known-to-be congested shards is the most effective tool to prevent +the system from accumulating more transactions. The sooner we do it, the shorter +the delays experienced by users who got their transactions accepted. + +`MIN_TX_GAS = 20 Tgas` and `MAX_TX_GAS = 500 Tgas` gives a large range to smooth +out the split between gas spend on new transactions vs delayed receipts. This +only looks at how many delayed receipts the local shard has, not at the +receiving shard. Depending on the workload, it will settle at different values. + +Note that hitting `REJECT_TX_CONGESTION_THRESHOLD`, which looks at the +congestion of the receiving shard, overrules this range and stops all +transactions to the congested shard when it is hit. + +`MIN_TX_GAS = 20 Tgas` guarantees that we can still accept a decent amount of +new transactions to shards that are not congested, even if the local shard +itself is fully congested. This gives fairness properties under certain workloads +which we could not achieve in any other of the tried congestion control +strategies. This is also useful to add transaction priority in +[NEP-541](https://github.com/near/NEPs/pull/541), as we can always auction off +the available space for new transactions without altering the congestion control +algorithm. + + +## Reference Implementation + +A reference implementation is available in this PR against nearcore: +https://github.com/near/nearcore/pull/10918 + +Here are the most important details which are not already described in the +specification above but are defined in the reference implementation. + +### Efficiently computing the congestion information + +The congestion information is computed based on the gas and size of the incoming +queue and the outgoing buffers. A naive implementation would just iterate over all +of the receipts in the queue and buffers and sum up the relevant metrics. This +approach is slow and, in the context of stateless validation, would add too much +to the state witness size. In order to prevent those issues we consider two +alternative optimizations. Both use the same principle of caching the previously +calculated metrics and updating them based on the changes to the incoming queue +and outgoing buffers. + +After applying a chunk, we store detailed information of the shard in the chunk +extra. Unlike the shard header, this is only stored on the shard and not shared +globally. + +The new fields in the chunk extra are included in `ChunkExtraV3`. + +```rust +pub struct ChunkExtraV3 { + + // ...all fields from ChunkExtraV2 + + pub congestion_info: CongestionInfo, +} + +pub struct CongestionInfo { + delayed_receipts_gas: u128, + buffered_receipts_gas: u128, + receipt_bytes: u64, + allowed_shard: u16, +} +``` + +This implementation allows to efficiently update the `StoredReceiptsInfo` during +chunk application by starting with the information of the previous chunk and +applying only the changes. + +Regarding integer sizes, `delayed_receipts_gas` and `buffered_receipts_gas` use +128-bit unsigned integers because 64-bit would not always be enough. `u64::MAX` +would only be enough to store `18_446 Pgas`. This translates to roughly 5 hours +of work, assuming 1 Pgas per second. While the proposed congestion control +strategy should prevent congestion ever reaching such high levels, it is not +possible to rule out completely. + +For `receipt_bytes`, a `u64` is more than enough, we have other problems if we +need to store millions of Terabytes of receipts. + +For the id of the allowed shard, we chose a `u16` which is large enough for +65_535 shards. + +### Bootstrapping + +The previous section explain how the gas and bytes information of unprocessed +receipts is computed based on what it was for the previous chunk. But for the +first chunk with this feature enabled, the information for the previous chunk is +not available. + +In this specific case, we detect that the previous information is not available +and therefore we trigger an iteration of the existing queues to compute the +correct values. + +This computed `StoredReceiptsInfo` only applies locally. But the next value of it +will be shared in the chunk header and other shards will start using it to limit +the transactions they accept and receipts they forward. + +The congestion info of other shards is assumed to be 0 for all values for the +first block with the cross-shard congestion control feature enabled. + +### Missing Chunks + +When a chunk is missing, we use the congestion information of the last available +chunk header for the shard. In practical terms this simply means we take the +chunk header available in the block, even if the included height is not the +latest. + +Additionally, we include the number of missed chunks as part of the congestion +formula, treating a shard with 10 or missed chunks the same way as an otherwise +fully congested shard. This is to prevent sending even more receipts to a shard +that already struggles to produce chunks. + +### Validation Changes + +The following fields in the chunk header must be validated: + +- `receipt_bytes`: must be equal to `receipt_bytes` of the previous chunk, plus + all bytes of new receipts added to delayed or buffered receipts, minus all the + receipts removed of the same types. +- `delayed_receipts_gas` must be equal to `delayed_receipts_gas` of the previous + chunk, plus the gas of receipts added to the delayed receipts queue, minus the + gas of receipts removed from the delayed receipts queue. +- `buffered_receipts_gas` must be equal to `buffered_receipts_gas` of the previous + chunk, plus the gas of receipts added to any of the outgoing receipts buffers, minus the + gas of all forwarded buffered receipts. +- `allowed_shard` must be a valid shard id. +- `allowed_shard` must be equal to the chunk's shard id if congestion is below 1. + +The balance checker also needs to take into account balances stored in buffered receipts. + +## Security Implications + +With cross-shard congestion control enabled, malicious users could try to find +patterns that clog up the system. This could potentially lead to cheaper denial +of service attacks compared to today. + +If such patterns exists, most likely today's implementation would suffer from +different problems, such as validators requiring unbounded amounts of memory. +Therefore, we believe this feature is massive step forward in terms of security, +all things considered. + +## Integration with state sync + +What we described in [Efficiently computing the congestion +information](#efficiently-computing-the-congestion-information) creates a +dependence on the previous block when processing a block. For a fully synced +node this requirement is always fulfilled because we keep at least 3 epochs of +blocks. However in state sync we start processing from an arbitrary place in +chain without access to full history. + +In order to integrate the congestion control and state sync features we will +add extra steps in state sync to download the blocks that may be needed in +order to finalize state sync. + +The blocks that are needed are the `sync hash` block, the `previous block` where +state sync creates chunk extra in order to kick off block sync and the `previous +previous block` that is now needed in order to process the `previous block`. On +top of that we may need to download further blocks to ensure that every shard has +at least one new chunk in the blocks leading up to the sync hash block. + +## Integration with resharding + +Resharding is a process wherin we change the shard layout - the assignment of +accound ids to shards. The centerpiece of resharding is moving the trie / state +records from parent shards to children shards. It's important to preserve the +ability to perform resharding while adding other protocol features such as congestion +control. Below is a short description how resharding and congestion control can be +integrated, in particular how to reshard the new trie columns - the outgoing buffers. + +For simplicity we'll only consider splitting a single parent shard into multiple +children shards which is currently the only supported operation. + +The actual implementation of this integration will be done independently and +outside of this effort. + +Importantly the resharding affects both the shard that is being split and all the +other shards. + +#### Changes to the shard under resharding + +The outgoing buffers of the parent shard can be split among children by iterating +all of the receipts in each buffer and inserting it to appropriate child shard. +The assignment can in theory be arbitrary e.g. all receipts can be reassigned to +a single shard. In practice it would make sense to either split the receipts +equally between children or based on the sender account id of the receipt. + +Special consideration should be given to refund receipts where the sender account +is "system" that may not belong to neither parent nor children shards. +Any assignment of such receipts is fine. + +#### Changes to the other shards + +The other shards, that is all shards that are not under resharding, have an +outgoing buffer to the shard under resharding. This buffer should be split +into one outgoing buffer per child shard. The buffer can be split by iterating +receipts and reassigning each to either of the child shards. Each receipt can +be reassigned based on it's receiver account id and the new shard layout. + +## Alternatives + +A wide range of alternatives has been discussed. It would be too much to list +all suggested variations of all strategies. Instead, here is a list of different +directions that were explored, with a representative strategy for each of them. + +1. Use transaction fees and an open market to reduce workload added to the + system. + + - Problem 1: This does not prevent unbounded memory requirements of + validators, it just makes it more expensive. + - Problem 2: In a sharded environment like Near Protocol, it is hard to + implement this fairly. Because it's impossible to know where a transaction + burns most of its gas, the only simple solution would require all shards to + pay the price for the most congested shard. + +2. Set fixed limits for delayed receipts and drop receipts beyond that. + + - Problem 1: Today, smart contract rely on receipts never being lost. This + network-level failure mode would be completely new. + - Problem 2: We either need to allow resuming with external inputs, or + roll-back changes on all shards to still have consistent states in smart + contracts. Both solutions means we are doing extra work when being + congested, inevitably reducing the available throughput for useful work in + times when the demand is the largest. + +3. Stop accepting transactions globally when any shard has too long of a delayed + receipts queue. ([See this issue](https://github.com/near/nearcore/issues/9228).) + - Problem 1: This gives poor utilization in many workloads, as our model + simulations confirmed. + - Problem 2: A global stop conflicts with plans to add fee based transaction + priorities which should allow sending transactions even under heavy + congestion. + +4. Reduce newly accepted transactions solely based on gas in delayed queues, + without adding new buffers or queues to the system. Gas is tracked per shard + of the transaction signer. ([Zulip Discussion](https://near.zulipchat.com/#narrow/stream/295558-core/topic/congestion.20control/near/429973223) and [idea in code](https://github.com/near/nearcore/pull/10894).) + + - Problem 1: This requires `N` gas numbers in each chunk header, or `N*N` + numbers per block, where `N` is the number of shards. + - Problem 2: We did not have time to simulate it properly. But on paper, it + seems each individual delayed queue can still grow indefinitely as the + number of shards in the system grows. + +5. Smartly track and limit buffer space across different shards. Only accept new + transactions if enough buffer space can be reserved ahead of time. + + - Problem 1: Without knowing which shards a transaction touches and how + large receipts will be, we have to pessimistically reserve more space than + most receipts will actually need. + - Problem 2: If buffer space is shared globally, individual queues can still + grow really large, even indefinitely if we assume the number of shards + grows over time. + - Problem 3: If buffer space is on a per-shard basis, we run into deadlocks + when two shards have no more space left but both need to send to the other + shard to make progress. + +6. Require users to define which shards their transactions will touch and how + much gas is burnt in each. Then use this information for global scheduling + such that congestion is impossible. + + - Problem 1: This requires lots of changes across the infrastructure stack. + It would take too long to implement as we are already facing congestion + problems today. + - Problem 2: This would have a strong impact on usability and it is unclear + if gas usage estimating services could close the gap to make it + acceptable. + +7. An alternative way to what is described in [Efficiently computing the + congestion information](#efficiently-computing-the-congestion-information) would + be to store the total gas and total size of the incoming queue and the outgoing + receipts in the state along the respective queue or buffers. Those values will + be updated as receipts are added or removed from the queue. + - Pro: In this case the CongestionInfo struct can remain small and only + reflect the information needed by other shards. (3 bytes instead of 42 + bytes) + + ```rust + pub struct CongestionInfo { + allowed_shard: u16, + congestion_level: u8, + } + ``` + + - Con: Overall, it would result in more state changes per chunk, since the + congestion value needs to be read before applying receipts anyway. In + light of stateless validation, this would be worse for the state witness + size + + +## Future possibilities + +While this proposal treats all requests the same, it sets the base for a proper +transaction priority implementation. We co-designed this proposal with +[NEP-541](https://github.com/near/NEPs/pull/541), which adds a transaction +priority fee. On a very high level, the fee is used to auction off a part of the +available gas per chunk to the highest bidders. + +We also expect that this proposal alone will not be the final solution for +congestion control. Rather, we just want to build a solid foundation in this NEP +and allow future optimization to take place on top of it. + +For example, estimations on how much gas is burnt on each shard could help with +better load distribution in some cases. + +We also forsee that the round-robin scheduling of shard allowed to forward even +under full congestion is not perfect. It is a key feature to make deadlocks +provably impossible, since every shard is guaranteed to make a minimum progress +after N rounds. But it could be beneficial to allocate more bandwidth to shards +that actually have something to forward, or perhaps it would be better to stop +forwarding anything for a while. The current proposal allows chunk producers to +experiment with this without a protocol version change. + +Lastly, a future optimization could do better transaction rejection for meta +transactions. Instead of looking only at the outer transaction receiver, we +could also look at the receiver of the delegate action, which is most likely +where most gas is going to be burnt, and use this for transaction rejection. + +## Consequences + +### Positive + +- Accepted transaction have lower latencies compared to today under congestion. +- Storage and memory requirements on validator for storing receipts are bounded. + +### Neutral + +- More transactions are rejected at the chunk producer level. + +### Negative + +- Users need to resend transaction more often. + +### Backwards Compatibility + +There are no observable changes on the smart contract, wallet, or API level. +Thus, there are no backwards-compatibility concerns. + +## Unresolved Issues (Optional) + +These congestion problems are out of scope for this proposal: + +- Malicious patterns can still cause queues to grow beyond the parameter limits. +- There is no way to pay for higher priority. + ([NEP-541](https://github.com/near/NEPs/pull/541) adds it.) + +Postponed receipts are also considered to be added to `receipt_bytes`. But at +this point it seems better to not include them to avoid further complications +with potential deadlocks, since postponed receipts can only be executed when +incoming receipts are allowed to come in. + +Following the same logic, yielded receipts are also excluded from the size +limits, as they require incoming receipts to resume. + +A solution that also address the memory space of postponed and yielded receipts +could be added with future proposals but is not considered necessary for this +first iteration of cross-shard congestion control. + +## Changelog + +[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] + +### 1.0.0 - Initial Version + +> Placeholder for the context about when and who approved this NEP version. + +#### Benefits + +> List of benefits filled by the Subject Matter Experts while reviewing this version: + +- Benefit 1 +- Benefit 2 + +#### Concerns + +> Template for Subject Matter Experts review for this version: +> Status: New | Ongoing | Resolved + +| # | Concern | Resolution | Status | +| --: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).