From 4886221074103d82c421555379c56d5cc1f7567b Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Fri, 2 Jun 2023 18:06:39 +0200 Subject: [PATCH 01/12] update NEP-455: storage compute costs for v61 (#477) Tracks the newly introduced compute costs for protocol version 61 --- neps/nep-0455.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/neps/nep-0455.md b/neps/nep-0455.md index 062589530..9699f1458 100644 --- a/neps/nep-0455.md +++ b/neps/nep-0455.md @@ -231,6 +231,31 @@ This NEP was approved by Protocol Working Group members on March 16, 2023 ([meet - [Marcelo's vote](https://github.com/near/NEPs/pull/455#pullrequestreview-1340887413) - [Marcin's vote](https://github.com/near/NEPs/pull/455#issuecomment-1471882639) +### 1.0.1 - Storage Related Compute Costs + +Add five compute cost values for protocol version 61 and above. + +- wasm_touching_trie_node +- wasm_storage_write_base +- wasm_storage_remove_base +- wasm_storage_read_base +- wasm_storage_has_key_base + +For the exact values, please refer to the table at the bottom. + +The intention behind these increased compute costs is to address the issue of +storage accesses taking longer than the allocated gas costs, particularly in +cases where RocksDB, the underlying storage system, is too slow. These values +have been chosen to ensure that validators with recommended hardware can meet +the required timing constraints. +([Analysis Report](https://github.com/near/nearcore/issues/8006)) + +The protocol team at Pagoda is actively working on optimizing the nearcore +client storage implementation. This should eventually allow to lower the compute +costs parameters again. + +Progress on this work is tracked here: https://github.com/near/nearcore/issues/8938. + #### Benefits - Among the alternatives, this is the easiest to implement. @@ -250,3 +275,13 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public - https://gov.near.org/t/proposal-gas-weights-to-fight-instability-to-due-to-undercharging/30919 - https://github.com/near/nearcore/issues/8032 + +## Live Compute Costs Tracking + +Parameter Name | Compute / Gas factor | First version | Last version | Tracking issue | +-------------- | -------------------- | ------------- | ------------ | -------------- | +wasm_touching_trie_node | 6.83 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) +wasm_storage_write_base | 3.12 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) +wasm_storage_remove_base | 3.74 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) +wasm_storage_read_base | 3.55 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) +wasm_storage_has_key_base | 3.70 | 61 | *TBD* | [nearcore#8938](https://github.com/near/nearcore/issues/8938) From 2e0a9298168c5a87a2a818513c1f0c436e0d36bc Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Sun, 30 Jul 2023 16:57:47 +0200 Subject: [PATCH 02/12] fix: Remove reference to storage rent (#489) Storage rent has been superseded by storage staking a long time ago. This commit updates the account specification accordingly. --- specs/DataStructures/Account.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/DataStructures/Account.md b/specs/DataStructures/Account.md index a8cc3c2e8..91b0c537e 100644 --- a/specs/DataStructures/Account.md +++ b/specs/DataStructures/Account.md @@ -151,7 +151,9 @@ Every account has its own storage. It's a persistent key-value trie. Keys are or The storage can only be modified by the contract on the account. Current implementation on Runtime only allows your account's contract to read from the storage, but this might change in the future and other accounts's contracts will be able to read from your storage. -NOTE: Accounts are charged recurrent rent for the total storage. This includes storage of the account itself, contract code, contract storage and all access keys. +NOTE: To pay for blockchain storage, the protocol locks a token amount per account proportional to its state size. +This includes storage of the account itself, contract code, contract storage and all access keys. +See [Storage Staking](https://docs.near.org/concepts/storage/storage-staking) in the docs. #### Access Keys From ccbd94d8a978e8249723e2f0f8a775997631a0f5 Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Fri, 4 Aug 2023 18:46:25 +0200 Subject: [PATCH 03/12] fix: replace dead link with wayback machine link (#494) The link to FT docs on figment.io is dead, let's replace it with an archived version of the site. Also includes the following linter fixes in Core.md; - remove extra indentation of lists - add blank lines around lists - replace emphasis with real headings --- neps/nep-0141.md | 2 +- specs/Standards/Tokens/FungibleToken/Core.md | 34 ++++++++++++-------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/neps/nep-0141.md b/neps/nep-0141.md index 2cec6e96c..bdc5d35fb 100644 --- a/neps/nep-0141.md +++ b/neps/nep-0141.md @@ -38,7 +38,7 @@ Prior art: Learn about NEP-141: -- [Figment Learning Pathway](https://learn.figment.io/tutorials/stake-fungible-token) +- [Figment Learning Pathway](https://web.archive.org/web/20220621055335/https://learn.figment.io/tutorials/stake-fungible-token) ## Specification diff --git a/specs/Standards/Tokens/FungibleToken/Core.md b/specs/Standards/Tokens/FungibleToken/Core.md index 736633c7a..3d44307cf 100644 --- a/specs/Standards/Tokens/FungibleToken/Core.md +++ b/specs/Standards/Tokens/FungibleToken/Core.md @@ -12,8 +12,9 @@ The [fungible token metadata standard](Metadata.md) provides the fields needed f ## Motivation NEAR Protocol uses an asynchronous, sharded runtime. This means the following: - - Storage for different contracts and accounts can be located on the different shards. - - Two contracts can be executed at the same time in different shards. + +- Storage for different contracts and accounts can be located on the different shards. +- Two contracts can be executed at the same time in different shards. While this increases the transaction throughput linearly with the number of shards, it also creates some challenges for cross-contract development. For example, if one contract wants to query some information from the state of another contract (e.g. current balance), by the time the first contract receives the balance the real balance can change. In such an async system, a contract can't rely on the state of another contract and assume it's not going to change. @@ -30,11 +31,12 @@ Prior art: Learn about NEP-141: -- [Figment Learning Pathway](https://learn.figment.io/tutorials/stake-fungible-token) +- [Figment Learning Pathway](https://web.archive.org/web/20220621055335/https://learn.figment.io/tutorials/stake-fungible-token) ## Guide-level explanation We should be able to do the following: + - Initialize contract once. The given total supply will be owned by the given account ID. - Get the total supply. - Transfer tokens to a new user. @@ -43,6 +45,7 @@ We should be able to do the following: - Remove state for the key/value pair corresponding with a user's account, withdrawing a nominal balance of Ⓝ that was used for storage. There are a few concepts in the scenarios above: + - **Total supply**: the total number of tokens in circulation. - **Balance owner**: an account ID that owns some amount of tokens. - **Balance**: an amount of tokens. @@ -60,7 +63,7 @@ Given that multiple users will use a Fungible Token contract, and their activity Alice wants to send 5 wBTC tokens to Bob. -**Assumptions** +##### Assumptions - The wBTC token contract is `wbtc`. - Alice's account is `alice`. @@ -68,11 +71,11 @@ Alice wants to send 5 wBTC tokens to Bob. - The precision ("decimals" in the metadata standard) on wBTC contract is `10^8`. - The 5 tokens is `5 * 10^8` or as a number is `500000000`. -**High-level explanation** +##### High-level explanation Alice needs to issue one transaction to wBTC contract to transfer 5 tokens (multiplied by precision) to Bob. -**Technical calls** +##### Technical calls 1. `alice` calls `wbtc::ft_transfer({"receiver_id": "bob", "amount": "500000000"})`. @@ -80,7 +83,7 @@ Alice needs to issue one transaction to wBTC contract to transfer 5 tokens (mult Alice wants to deposit 1000 DAI tokens to a compound interest contract to earn extra tokens. -**Assumptions** +##### Assumptions - The DAI token contract is `dai`. - Alice's account is `alice`. @@ -92,14 +95,15 @@ Alice wants to deposit 1000 DAI tokens to a compound interest contract to earn e
For this example, you may expand this section to see how a previous fungible token standard using escrows would deal with the scenario. -**High-level explanation** (NEP-21 standard) +##### High-level explanation (NEP-21 standard) Alice needs to issue 2 transactions. The first one to `dai` to set an allowance for `compound` to be able to withdraw tokens from `alice`. The second transaction is to the `compound` to start the deposit process. Compound will check that the DAI tokens are supported and will try to withdraw the desired amount of DAI from `alice`. + - If transfer succeeded, `compound` can increase local ownership for `alice` to 1000 DAI - If transfer fails, `compound` doesn't need to do anything in current example, but maybe can notify `alice` of unsuccessful transfer. -**Technical calls** (NEP-21 standard) +##### Technical calls (NEP-21 standard) 1. `alice` calls `dai::set_allowance({"escrow_account_id": "compound", "allowance": "1000000000000000000000"})`. 2. `alice` calls `compound::deposit({"token_contract": "dai", "amount": "1000000000000000000000"})`. During the `deposit` call, `compound` does the following: @@ -108,11 +112,11 @@ The second transaction is to the `compound` to start the deposit process. Compou
-**High-level explanation** +##### High-level explanation Alice needs to issue 1 transaction, as opposed to 2 with a typical escrow workflow. -**Technical calls** +##### Technical calls 1. `alice` calls `dai::ft_transfer_call({"receiver_id": "compound", "amount": "1000000000000000000000", "msg": "invest"})`. During the `ft_transfer_call` call, `dai` does the following: 1. makes async call `compound::ft_on_transfer({"sender_id": "alice", "amount": "1000000000000000000000", "msg": "invest"})`. @@ -124,7 +128,7 @@ Alice needs to issue 1 transaction, as opposed to 2 with a typical escrow workfl Alice wants to swap 5 wrapped NEAR (wNEAR) for BNNA tokens at current market rate, with less than 2% slippage. -**Assumptions** +##### Assumptions - The wNEAR token contract is `wnear`. - Alice's account is `alice`. @@ -133,7 +137,7 @@ Alice wants to swap 5 wrapped NEAR (wNEAR) for BNNA tokens at current market rat - The precision ("decimals" in the metadata standard) on wNEAR contract is `10^24`. - The 5 tokens is `5 * 10^24` or as a number is `5000000000000000000000000`. -**High-level explanation** +##### High-level explanation Alice needs to issue one transaction to wNEAR contract to transfer 5 tokens (multiplied by precision) to `amm`, specifying her desired action (swap), her destination token (BNNA) & maximum slippage (<2%) in `msg`. @@ -143,7 +147,7 @@ Alice needs to attach one yoctoNEAR. This will result in her seeing a confirmati Altogether then, Alice may take two steps, though the first may be a background detail of the app she uses. -**Technical calls** +##### Technical calls 1. View `amm::ft_data_to_msg({ action: "swap", destination_token: "bnna", max_slip: 2 })`. Using [NEAR CLI](https://docs.near.org/docs/tools/near-cli): @@ -179,6 +183,7 @@ Altogether then, Alice may take two steps, though the first may be a background ## Reference-level explanation **NOTES**: + - All amounts, balances and allowance are limited by `U128` (max value `2**128 - 1`). - Token standard uses JSON for serialization of arguments and results. - Amounts in arguments and results have are serialized as Base-10 strings, e.g. `"100"`. This is done to avoid JSON limitation of max integer value of `2**53`. @@ -322,6 +327,7 @@ function ft_resolve_transfer( ## History See also the discussions: + - [Fungible token core](https://github.com/near/NEPs/discussions/146#discussioncomment-298943) - [Fungible token metadata](https://github.com/near/NEPs/discussions/148) - [Storage standard](https://github.com/near/NEPs/discussions/145) From c781b791aa0c95f369142e208a7c2e54098d5537 Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Fri, 4 Aug 2023 18:47:00 +0200 Subject: [PATCH 04/12] fix: linter issues in specs/DataStructures/Account (#493) - remove unused link declarations - surround titles and code with blank lines - give fenced code a language (using c to format comments) --- specs/DataStructures/Account.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/specs/DataStructures/Account.md b/specs/DataStructures/Account.md index 91b0c537e..b01306bbf 100644 --- a/specs/DataStructures/Account.md +++ b/specs/DataStructures/Account.md @@ -2,8 +2,6 @@ ## Account ID -[account_id]: #account_id - NEAR Protocol has an account names system. Account ID is similar to a username. Account IDs have to follow the rules. ### Account ID Rules @@ -26,6 +24,7 @@ Regex for a full account ID, without checking for length: ```regex ^(([a-z\d]+[\-_])*[a-z\d]+\.)*([a-z\d]+[\-_])*[a-z\d]+$ ``` + ### Top Level Accounts | Name | Value | @@ -52,7 +51,7 @@ def action_create_account(predecessor_id, account_id): Valid accounts: -``` +```c ok bowen ek-2 @@ -73,7 +72,7 @@ bro.a Invalid accounts: -``` +```c not ok // Whitespace characters are not allowed a // Too short 100- // Suffix separator @@ -91,6 +90,7 @@ abcdefghijklmnopqrstuvwxyz.abcdefghijklmnopqrstuvwxyz.abcdefghijklmnopqrstuvwxyz ``` ## System account + `system` is a special account that is only used to identify refund receipts. For refund receipts, we set the predecessor_id to be `system` to indicate that it is a refund receipt. Users cannot create or access the `system` account. In fact, this account does not exist as part of the state. ## Implicit account IDs @@ -107,6 +107,7 @@ The corresponding secret key allows you to sign transactions on behalf of this a ### Implicit account creation An account with implicit account ID can only be created by sending a transaction/receipt with a single `Transfer` action to the implicit account ID receiver: + - The account will be created with the account ID. - The account will have a new full access key with the ED25519-curve public key of `decode_hex(account_id)` and nonce `0`. - The account balance will have a transfer balance deposited to it. @@ -117,8 +118,6 @@ Once an implicit account is created it acts as a regular account until it's dele ## Account -[account]: #account - Data for an single account is collocated in one shard. The account data consists of the following: - Balance From 4290e6f76f112b68fa6ffddef4a85a3841e76533 Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Fri, 4 Aug 2023 18:48:45 +0200 Subject: [PATCH 05/12] doc: add delegate action to actions specification (#495) closes #486 --- specs/RuntimeSpec/Actions.md | 151 ++++++++++++++++++++++++++++++++++- 1 file changed, 149 insertions(+), 2 deletions(-) diff --git a/specs/RuntimeSpec/Actions.md b/specs/RuntimeSpec/Actions.md index 7c5622533..1c8cd58dd 100644 --- a/specs/RuntimeSpec/Actions.md +++ b/specs/RuntimeSpec/Actions.md @@ -12,6 +12,7 @@ pub enum Action { AddKey(AddKeyAction), DeleteKey(DeleteKeyAction), DeleteAccount(DeleteAccountAction), + Delegate(SignedDelegateAction), } ``` @@ -20,6 +21,7 @@ converted to receipts when they are processed, we will mostly concern ourselves processing. For the following actions, `predecessor_id` and `receiver_id` are required to be equal: + - `DeployContract` - `Stake` - `AddKey` @@ -38,14 +40,17 @@ pub struct CreateAccountAction {} If `receiver_id` has length == 64, this account id is considered to be `hex(public_key)`, meaning creation of account only succeeds if followed up with `AddKey(public_key)` action. **Outcome**: + - creates an account with `id` = `receiver_id` - sets Account `storage_usage` to `account_cost` (genesis config) ### Errors **Execution Error**: + - If the action tries to create a top level account whose length is no greater than 32 characters, and `predecessor_id` is not `registrar_account_id`, which is defined by the protocol, the following error will be returned + ```rust /// A top-level account ID can only be created by registrar. CreateAccountOnlyByRegistrar { @@ -57,6 +62,7 @@ CreateAccountOnlyByRegistrar { - If the action tries to create an account that is neither a top-level account or a subaccount of `predecessor_id`, the following error will be returned + ```rust /// A newly created account must be under a namespace of the creator account CreateAccountNotAllowed { account_id: AccountId, predecessor_id: AccountId }, @@ -71,18 +77,22 @@ pub struct DeployContractAction { ``` **Outcome**: + - sets the contract code for account ### Errors **Validation Error**: + - if the length of `code` exceeds `max_contract_size`, which is a genesis parameter, the following error will be returned: + ```rust /// The size of the contract code exceeded the limit in a DeployContract action. ContractSizeExceeded { size: u64, limit: u64 }, ``` **Execution Error**: + - If state or storage is corrupted, it may return `StorageError`. ## FunctionCallAction @@ -112,11 +122,13 @@ pub struct TransferAction { ``` **Outcome**: + - transfers amount specified in `deposit` from `predecessor_id` to a `receiver_id` account ### Errors **Execution Error**: + - If the deposit amount plus the existing amount on the receiver account exceeds `u128::MAX`, a `StorageInconsistentState("Account balance integer overflow")` error will be returned. @@ -132,26 +144,32 @@ pub struct StakeAction { ``` **Outcome**: + - A validator proposal that contains the staking public key and the staking amount is generated and will be included in the next block. ### Errors **Validation Error**: + - If the `public_key` is not an ristretto compatible ed25519 key, the following error will be returned: + ```rust /// An attempt to stake with a public key that is not convertible to ristretto. UnsuitableStakingKey { public_key: PublicKey }, ``` **Execution Error**: + - If an account has not staked but it tries to unstake, the following error will be returned: + ```rust /// Account is not yet staked, but tries to unstake TriesToUnstake { account_id: AccountId }, ``` - If an account tries to stake more than the amount of tokens it has, the following error will be returned: + ```rust /// The account doesn't have enough balance to increase the stake. TriesToStake { @@ -163,6 +181,7 @@ TriesToStake { ``` - If the staked amount is below the minimum stake threshold, the following error will be returned: + ```rust InsufficientStake { account_id: AccountId, @@ -170,6 +189,7 @@ InsufficientStake { minimum_stake: Balance, } ``` + The minimum stake is determined by `last_epoch_seat_price / minimum_stake_divisor` where `last_epoch_seat_price` is the seat price determined at the end of last epoch and `minimum_stake_divisor` is a genesis config parameter and its current value is 10. @@ -184,14 +204,17 @@ pub struct AddKeyAction { ``` **Outcome**: + - Adds a new [AccessKey](/DataStructures/AccessKey.md) to the receiver's account and associates it with a `public_key` provided. -### Errors: +### Errors **Validation Error**: If the access key is of type `FunctionCallPermission`, the following errors can happen + - If `receiver_id` in `access_key` is not a valid account id, the following error will be returned + ```rust /// Invalid account ID. InvalidAccountId { account_id: AccountId }, @@ -199,6 +222,7 @@ InvalidAccountId { account_id: AccountId }, - If the length of some method name exceed `max_length_method_name`, which is a genesis parameter (current value is 256), the following error will be returned + ```rust /// The length of some method name exceeded the limit in a Add Key action. AddKeyMethodNameLengthExceeded { length: u64, limit: u64 }, @@ -206,17 +230,21 @@ AddKeyMethodNameLengthExceeded { length: u64, limit: u64 }, - If the sum of length of method names (with 1 extra character for every method name) exceeds `max_number_bytes_method_names`, which is a genesis parameter (current value is 2000), the following error will be returned + ```rust /// The total number of bytes of the method names exceeded the limit in a Add Key action. AddKeyMethodNamesNumberOfBytesExceeded { total_number_of_bytes: u64, limit: u64 } ``` **Execution Error**: + - If an account tries to add an access key with a given public key, but an existing access key with this public key already exists, the following error will be returned + ```rust /// The public key is already used for an existing access key AddKeyAlreadyExists { account_id: AccountId, public_key: PublicKey } ``` + - If state or storage is corrupted, a `StorageError` will be returned. ## DeleteKeyAction @@ -228,6 +256,7 @@ pub struct DeleteKeyAction { ``` **Outcome**: + - Deletes the [AccessKey](/DataStructures/AccessKey.md) associated with `public_key`. ### Errors @@ -235,10 +264,12 @@ pub struct DeleteKeyAction { **Execution Error**: - When an account tries to delete an access key that doesn't exist, the following error is returned + ```rust /// Account tries to remove an access key that doesn't exist DeleteKeyDoesNotExist { account_id: AccountId, public_key: PublicKey } ``` + - `StorageError` is returned if state or storage is corrupted. ## DeleteAccountAction @@ -251,28 +282,144 @@ pub struct DeleteAccountAction { ``` **Outcomes**: + - The account, as well as all the data stored under the account, is deleted and the tokens are transferred to `beneficiary_id`. ### Errors -**Validation Error** +**Validation Error**: + - If `beneficiary_id` is not a valid account id, the following error will be returned + ```rust /// Invalid account ID. InvalidAccountId { account_id: AccountId }, ``` - If this action is not the last action in the action list of a receipt, the following error will be returned + ```rust /// The delete action must be a final action in transaction DeleteActionMustBeFinal ``` - If the account still has locked balance due to staking, the following error will be returned + ```rust /// Account is staking and can not be deleted DeleteAccountStaking { account_id: AccountId } ``` **Execution Error**: + - If state or storage is corrupted, a `StorageError` is returned. + +## Delegate Actions + +Introduced with [NEP-366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md) to enable meta transactions. + +In summary, a delegate action is an indirect submission of a transaction. +It allows a relayer to do the payment (gas and token costs) for a transaction authored by a user. + +```rust +/// The struct contained in transactions and receipts, inside `Action::Delegate(_)``. +struct SignedDelegateAction { + /// The actual action, see below. + pub delegate_action: DelegateAction, + /// NEP-483 proposal compliant signature + pub signature: Signature, +} +``` + +Note that the signature follows a scheme which is proposed to be standardized in [NEP-483](https://github.com/near/NEPs/pull/483). + + +```rust +/// The struct a user creates and signs to create a meta transaction. +struct DelegateAction { + /// Signer of the delegated actions + pub sender_id: AccountId, + /// Receiver of the delegated actions. + pub receiver_id: AccountId, + /// List of actions to be executed. + /// + /// With the meta transactions MVP defined in NEP-366, nested + /// DelegateActions are not allowed. A separate type is used to enforce it. + pub actions: Vec, + /// Nonce to ensure that the same delegate action is not sent twice by a + /// relayer and should match for given account's `public_key`. + /// After this action is processed it will increment. + pub nonce: Nonce, + /// The maximal height of the block in the blockchain below which the given DelegateAction is valid. + pub max_block_height: BlockHeight, + /// Public key used to sign this delegated action. + pub public_key: PublicKey, +} +``` + +### Outcomes + +- All actions inside `delegate_action.actions` are submitted with the `delegate_action.sender_id` as the predecessor, `delegate_action.receiver_id` as the receiver, and the relayer (predecessor of `DelegateAction`) as the signer. +- All gas and balance costs for submitting `delegate_action.actions` are subtracted from the relayer. + +### Errors + +**Validation Error**: + +- If the list of Transaction actions contains several `DelegateAction` + +```rust +/// There should be the only one DelegateAction +DelegateActionMustBeOnlyOne +``` + +**Execution Error**: + +- If the Sender's account doesn't exist + +```rust +/// Happens when TX receiver_id doesn't exist +AccountDoesNotExist +``` + +- If the `signature` does not match the data and the `public_key` of the given key, then the following error will be returned + +```rust +/// Signature does not match the provided actions and given signer public key. +DelegateActionInvalidSignature +``` + +- If the `sender_id` doesn't match the `tx.receiver_id` + +```rust +/// Receiver of the transaction doesn't match Sender of the delegate action +DelegateActionSenderDoesNotMatchTxReceiver +``` + +- If the current block is equal or greater than `max_block_height` + +```rust +/// Delegate action has expired +DelegateActionExpired +``` + +- If the `public_key` does not exist for Sender account + +```rust +/// The given public key doesn't exist for Sender account +DelegateActionAccessKeyError +``` + +- If the `nonce` does match the `public_key` for the `sender_id` + +```rust +/// Nonce must be greater sender[public_key].nonce +DelegateActionInvalidNonce +``` + +- If `nonce` is too large + +```rust +/// DelegateAction nonce is larger than the upper bound given by the block height (block_height * 1e6) +DelegateActionNonceTooLarge +``` From 9cb94e0cc98f76549e6e814f008d9c4793135078 Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Fri, 4 Aug 2023 18:52:57 +0200 Subject: [PATCH 06/12] fix: inaccuracies in delegate action errors (#496) While updating the specs for delegate actions, I noticed that there are two small mistakes in the error description of the NEP. Or I guess it would be more accurate to say the nearcore implementation is not compliant with the NEP. 1) `DelegateActionCantContainNestedOne` is impossible to happen because the chosen serialization format does not allow nesting in the first place. 2) The name for `DelegateActionSenderDoesNotMatchReceiver` somehow managed to get even longer in the real implementation and now spells `DelegateActionSenderDoesNotMatchTxReceiver` I know this isn't ideal. But since this is now de-facto part of the protocol, I suggest we update the NEP to reflect it. Alternatively, we could change it in nearcore. This would be a breaking change in some primitives crates, so I tend towards the first option. --- neps/nep-0366.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/neps/nep-0366.md b/neps/nep-0366.md index e419c9851..5aada7b0a 100644 --- a/neps/nep-0366.md +++ b/neps/nep-0366.md @@ -3,10 +3,12 @@ NEP: 366 Title: Meta Transactions Author: Illia Polosukhin , Egor Uleyskiy (egor.ulieiskii@gmail.com), Alexander Fadeev (fadeevab.com@gmail.com) DiscussionsTo: https://github.com/nearprotocol/neps/pull/366 -Status: Draft +Status: Approved Type: Protocol Track Category: Runtime +Version: 1.1.0 Created: 19-Oct-2022 +LastUpdated: 03-Aug-2023 --- ## Summary @@ -149,7 +151,7 @@ DelegateActionInvalidSignature ```rust /// Receiver of the transaction doesn't match Sender of the delegate action -DelegateActionSenderDoesNotMatchReceiver +DelegateActionSenderDoesNotMatchTxReceiver ``` - If the current block is equal or greater than `max_block_height` @@ -180,13 +182,6 @@ DelegateActionInvalidNonce DelegateActionNonceTooLarge ``` -- If the list of delegated actions contains another `DelegateAction` - -```rust -/// DelegateAction actions contain another DelegateAction -DelegateActionCantContainNestedOne -``` - - If the list of Transaction actions contains several `DelegateAction` ```rust @@ -214,6 +209,13 @@ See the **_Validation_** section in [DelegateAction specification](/specs/Runtim Supporting ZK proofs instead of just signatures can allow for anonymous transactions, which pay fees to relayers anonymously. +## Changelog + +### 1.1.0 - Adjust errors to reflect deployed reality (03-Aug`2023) + +- Remove the error variant `DelegateActionCantContainNestedOne` because this would already fail in the parsing stage. +- Rename the error variant `DelegateActionSenderDoesNotMatchReceiver` to `DelegateActionSenderDoesNotMatchTxReceiver` to reflect published types in [near_primitives](https://docs.rs/near-primitives/0.17.0/near_primitives/errors/enum.ActionErrorKind.html#variant.DelegateActionSenderDoesNotMatchTxReceiver). + ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 3786d9b7e1c057b53b8a66fdd3f30880a2aa2695 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Tue, 12 Sep 2023 18:03:40 +0200 Subject: [PATCH 07/12] feat: adding soulbound token (#393) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Soulbound Tokens (SBT) are non transferrable NFTs. Even though tranferability is not available, we define a recoverability mechanism. SBTs are well suited of carrying proof-of-attendence NFTs, proof-of-unique-human "stamps" and other similar credibility-carriers. --- ## NEP Status *(Updated by NEP moderators)* SME reviews: * [x] @KazanderDad (NDC GWG): https://github.com/near/NEPs/pull/393#pullrequestreview-1380240545 * [x] @alexastrum (Tenamint): https://github.com/near/NEPs/pull/393/#issuecomment-1541004902 Contract Standards WG voting indications (❔ | 👍 | :-1: ): * 👍 @frol: https://github.com/near/NEPs/pull/393#pullrequestreview-1499578010 * 👍 @fadeevab: https://github.com/near/NEPs/pull/393#pullrequestreview-1506223006 * -- @robert-zaremba (can't vote myself) Wallet Standards WG voting indications: * ❔ @Cameron-Banyan * ❔ @MaximusHaximus * ❔ @esaminu ## Concerns | # | Concern | Resolution | Status | | --- | --- | -- | -- | | 1 | [Rober] Should we Emit NEP-171 Mint and NEP-171 Burn by the SBT contract (in addition to SBT native events emitted by the registry)? If the events will be emitted by registry, then we need new events to include the contract address. | [Robert] Don't emit NFT events. SBT is not NFT. Support: @alexastrum | open | | 2 | [Robert] remove `memo` in events. The `memo` is already part of the transaction, and should not be needed to identify transactions. Processes looking for events, can easily track transaction through event and recover `memo` if needed. | currently removed, consequently also removed from registry transactions . Support: @alexastrum | open | | 3 | [Token Spam](https://github.com/near/NEPs/pull/393/#discussion_r1163938750) | [Robert]: we have a `Burn` event. Added example `sbt_burn` function, but keeping it not as a part of required interface. Event should be enough. | open | | 4 | [Multiple registries](https://github.com/near/NEPs/pull/393/#discussion_r1163951624). Registry source of truth [comment](https://github.com/near/NEPs/pull/393/#issuecomment-1531766643) | Robert: this is a part of the design: permissionless approach. [Justification for registry](https://github.com/near/NEPs/pull/393/#issuecomment-1540621077) | open | | 5 | [Robert] Approve the proposed multi-token | Support: @alexastrum | open | | 6 | [Robert] Use of milliseconds as a time unit. | [Robert] Currently the standard uses milliseconds. | open | | 7 | Should a `burn` function be part of a standard or a recommendation? | [Robert] We already have the Burn event. IMHO a function should not be part of the standard inteface (similarly to FT and NFT). | open | | 8 | [Robert] Don't include `sbt_soul_transfer` in the standard interface, [comment](https://github.com/near/NEPs/pull/393#issuecomment-1506969996). | [Robert] moving outside of the required interface. | open | | 9 | [Privacy](https://github.com/near/NEPs/pull/393/#issuecomment-1504309947) | [Robert] Concerns have been addressed: [comment-1](https://github.com/near/NEPs/pull/393/#issuecomment-1504485420) and [comment2](https://github.com/near/NEPs/pull/393/#issuecomment-1505958549) | open | | 10 | x | resolution | open | --------- Co-authored-by: Vlad Frolov --- neps/nep-0393.md | 665 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 665 insertions(+) create mode 100644 neps/nep-0393.md diff --git a/neps/nep-0393.md b/neps/nep-0393.md new file mode 100644 index 000000000..67b3361c2 --- /dev/null +++ b/neps/nep-0393.md @@ -0,0 +1,665 @@ +--- +NEP: 393 +Title: Soulbound Token +Authors: Robert Zaremba <@robert-zaremba> +DiscussionsTo: +Status: Approved +Type: Standards Track +Category: Contract +Created: 12-Sep-2022 +Requires: +--- + +# NEP: Soulbound Token + +## Summary + +Soulbound Token (SBT) is a form of a non-fungible token which represents an aspect of an account: _soul_. [Transferability](#transferability) is limited only to a case of recoverability or a _soul transfer_. The latter must coordinate with a registry to transfer all SBTs from one account to another, and _banning_ the source account. + +SBTs are well suited for carrying proof-of-attendance, proof-of-unique-human "stamps" and other similar credibility-carriers. + +## Motivation + +Recent [Decentralized Society](https://www.bankless.com/decentralized-society-desoc-explained) trends open a new area of Web3 research to model various aspects of what characterizes humans. Economic and governance value is generated by humans and their relationship. SBTs can represent the commitments, credentials, and affiliations of “Souls” that encode the trust networks of the real economy to establish provenance and reputation. + +> More importantly, SBTs enable other applications of increasing ambition, such as community wallet recovery, Sybil-resistant governance, mechanisms for decentralization, and novel markets with decomposable, shared rights. We call this richer, pluralistic ecosystem “Decentralized Society” (DeSoc)—a co-determined sociality, where Souls and communities come together bottom-up, as emergent properties of each other to co-create plural network goods and intelligences, at a range of scales. + +Creating strong primitives is necessary to model new innovative systems and decentralized societies. Examples include reputation protocols, non-transferrable certificates, non-transferrable rights, undercollateralized lending, proof-of-personhood, proof-of-attendance, proof-of-skill, one-person-one-vote, fair airdrops & ICOs, universal basic income, non KYC identity systems, Human DAOs and methods for Sybil attack resistance. + +We propose an SBT standard to model protocols described above. + +_Verifiable Credentials_ (VC) could be seen as subset of SBT. However there is an important distinction: VC require set of claims and privacy protocols. It would make more sense to model VC with relation to W3 DID standard. SBT is different, it doesn't require a [resolver](https://www.w3.org/TR/did-core/#dfn-did-resolvers) nor [method](https://www.w3.org/TR/did-core/#dfn-did-methods) registry. For SBT, we need something more elastic than VC. + +## Specification + +Main requirement for Soulbound tokens is to bound an account to a human. A **Soul** is an account with SBTs, which are used to define account identity. +Often non transferrable NFT (an NFT token with a no-op transfer function) is used to implement SBTs. However, such model is rather shortsighted. Transferability is required to allow users to either recover their SBTs or merge between the accounts they own. At the same time, we need to limit transferability to assure that an SBTs is kept bound to the same _soul_. +We also need an efficient on way to make composed ownership queries (for example: check if an account owns SBT of class C1, C2 and C3 issued by issuer I1, I2 and I3 respectively) - this is needed to model emergent properties discussed above. + +We introduce a **soul transfer**: an ability for user to move ALL SBT tokens from one account to another in a [semi atomic](#soul-transfer) way, while keeping the SBT bounded to the same _soul_. This happens when a user needs to merge his accounts (e.g. they started with a few different accounts but later decides to merge them to increase an account reputation). Soul transfer is different than a token transfer. The standard forbids a traditional token transfer functionality, where a user can transfer individual tokens. That being said, a registry can have extension functions for more advanced scenarios, which could require a governance approval. + +SBT standard separates the token issuer concept from the token registry in order to meet the requirements listed above. +In the following sections we discuss the functionality of an issuer and registry. + +### SBT Registry + +Traditional Token model in NEAR blockchain assumes that each token has it's own balance book and implements the authorization and issuance mechanism in the same smart contract. Such model prevents atomic _soul transfer_ in the current NEAR runtime. When token balance is kept separately in each SBT smart contract, synchronizing transfer calls to all such contracts to assure atomicity is not possible at scale. We need an additional contract, the `SBT Registry`, to provide atomic transfer of all user SBTs and efficient way to ban accounts in relation to the Ban event discussed below. +This, and efficient cross-issuer queries are the main reasons SBT standards separates that token registry and token issuance concerns. + +Issuer is an entity which issues new tokens and potentially can update the tokens (for example execute renewal). All standard modification options are discussed in the sections below. + +Registry is a smart contract, where issuers register the tokens. Registry provides a balance book of all associated SBTs. Registry must ensure that each issuer has it's own "sandbox" and issuers won't overwrite each other. A registry provides an efficient way to query multiple tokens for a single user. This will allow implementation of use cases such us: + +- SBT based identities (main use case of the `i-am-human` protocol); +- SBT classes; +- decentralized societies. + +```mermaid +graph TB + Issuer1--uses--> Registry + Issuer2--uses--> Registry + Issuer3--uses--> Registry +``` + +We can have multiple competing registries (with different purpose or different management scheme). An SBT issuer SHOULD opt-in to a registry before being able to use registry. Registries may develop different opt-in mechanisms (they could differ by the approval mechanism, be fully permissioned etc..). One SBT smart contract can opt-in to: + +- many registries: it MUST relay all state change functions to all registries. +- or to no registry. We should think about it as a single token registry, and it MUST strictly implement all SBT Registry query functions by itself. The contract address must be part of the arguments, and it must check that it equals to the deployed account address (`require!(ctr == env::current_account_id())`). It also MUST emit related events by itself. + +We recommend that each issuer will use only one registry to avoid complex reconciliation and assure single source of truth. + +The registry fills a central and important role. But it is **not centralized**, as anyone can create their own registry and SBT issuers can choose which registry to use. It's also not too powerful, as almost all of the power (mint, revoke, burn, recover, etc) still remains with the SBT issuer and not with the registry. + +#### Issuer authorization + +A registry can limit which issuers can use registry to mint SBTs by implementing a custom issuer whitelist methods (for example simple access control list managed by a DAO) or keep it fully open (allowing any issuer minting withing the registry). + +Example: an `SBT_1 Issuer` wants to mint tokens using the `SBT_Registry`. The `SBT_Registry` has a DAO which votes on adding a new issuer: + +```mermaid +sequenceDiagram + actor Issuer1 as SBT_1 Issuer + actor DAO + + participant SBT_Registry + + Note over Issuer1,DAO: Issuer1 connects with the DAO
to be whitelisted. + + Issuer1-->>DAO: request whitelist + DAO->>SBT_Registry: whitelist(SBT_1 Issuer) +``` + +#### Personal Identifiable Information + +Issuers must not include any PII into any SBT. + +### Account Ban + +`Ban` is an event emitted by a registry signaling that the account is banned, and can't own any SBT. Registry must return zero for every SBT supply query of a banned account. Operations which trigger soul transfer must emit Ban. + +A registry can emit a `Ban` for use cases not discussed in this standard. Handling it depends on the registry governance. One example is to use social governance to identify fake accounts (like bots) - in that case the registry should allow to emit `Ban` and block a scam soul and block future transfers. +NOTE: an SBT Issuer can have it's own list of blocked accounts or allowed only accounts. + +### Minting + +Minting is done by issuer calling `registry.sbt_mint(tokens_to_mint)` method. Standard doesn't specify how a registry authorizes an issuer. A classical approach is a whitelist of issuers: any whitelisted issuer can mint any amount of new tokens. Registry must keep the balances and assign token IDs to newly minted tokens. + +Example: Alice has two accounts: `alice1` and `alice2` which she used to mint tokens. She is getting tokens from 2 issuers that use the same registry. Alice uses her `alice1` account to interact with `SBT_1 Issuer` and receives an SBT with token ID = 238: + +```mermaid +sequenceDiagram + actor Alice + actor Issuer1 as SBT_1 Issuer + + participant SBT1 as SBT_1 Contract + participant SBT_Registry + + Issuer1->>SBT1: sbt_mint(alice1, metadata) + activate SBT1 + SBT1-)SBT_Registry: sbt_mint([[alice1, [metadata]]]) + SBT_Registry->>SBT_Registry: emit Mint(SBT_1_Contract, alice1, [238]) + SBT_Registry-)SBT1: [238] + deactivate SBT1 + + Note over Alice,SBT_Registry: now Alice can query registry to check her SBT + + Alice-->>SBT_Registry: sbt(SBT_1_Contract, 238) + SBT_Registry-->>Alice: {token: 238, owner: alice1, metadata} +``` + +With `SBT_2 Issuer`, Alice uses her `alice2` account. Note that `SBT_2 Contract` has different mint function (can mint many tokens at once), and validates a proof prior to requesting the registry to mint the tokens. + +```mermaid +sequenceDiagram + actor Alice + actor Issuer2 as SBT_2 Issuer + + participant SBT2 as SBT_2 Contract + participant SBT_Registry + + Issuer2->>SBT2: sbt_mint_multi([[alice2, metadata2], [alice2, metadata3]], proof) + activate SBT2 + SBT2-)SBT_Registry: sbt_mint([[alice2, [metadata2, metadata3]]]) + SBT_Registry->>SBT_Registry: emit Mint(SBT_2_Contract, alice2, [7991, 7992]) + SBT_Registry-)SBT2: [7991, 1992] + deactivate SBT2 + + Note over Alice,SBT_Registry: Alice queries one of her new tokens + Alice-->>SBT_Registry: sbt(SBT_2_Contract, 7991) + SBT_Registry-->>Alice: {token: 7991, owner: alice2, metadata: metadata2} +``` + +### Transferability + +Safeguards are set against misuse of SBT transfer and keep the _soul bound_ property. SBT transfer from one account to another should be strictly limited to: + +- **revocation** allows issuer to invalidate or burn an SBT in case a token issuance should be reverted (for example the recipient is a Sybil account, that is an account controlled by an entity trying to create the false appearance); +- **recoverability** in case a user's private key is compromised due to extortion, loss, etc. Users cannot recover an SBT only by themselves. Users must connect with issuer to request recoverability or use more advanced mechanism (like social recoverability). The recovery function provides additional economical cost preventing account trading: user should always be able to recover his SBT, and move to another, not banned account. +- **soul transfer** - moving all SBT tokens from a source account (issued by all issuers) to a destination account. During such transfer, SBT registry emits `SoulTransfer` and `Ban` events. The latter signals that the account can't host nor receive any SBT in the future, effectively burning the identity of the source account. This creates an inherit cost for the source account: it's identity can't be used any more. Registry can have extension functions for more advanced scenarios, which could require a governance mechanism. `SoulTransfer` event can also trigger similar actions in other registries (specification for this is out of the scope of this NEP). + +This becomes especially important for proof-of-human stamps that can only be issued once per user. + +#### Revocation + +An issuer can revoke SBTs by calling `registry.sbt_revoke(tokens_to_revoke, burn)`. Example: when a related certificate or membership should be revoked, when an issuer finds out that there was an abuse or a scam, etc...). Registry, when receiving `sbt_revoke` request from an issuer must always emit the `Revoke` event. Registry must only accept revoke requests from a valid issuer, and only revoke tokens from that issuer. If `burn=true` is set in the request, then the token should be burned and `Burn` event must be emitted. Otherwise (when `burn=false`) the registry must update token metadata and set expire date to a time in the past. Registry must not ban nor emit `Ban` event when revoking a contract. That would create an attack vector, when a malicious registry would thread the registry by banning accounts. + +#### Recoverability + +Standard defines issuer recoverability. At minimum, the standard registry exposes `sbt_recover` method, which allows issuer to reassign a token issued by him from one account to another. + +SBT recovery MUST not trigger `SoulTransfer` nor `Ban` event: malicious issuer could compromise the system by faking the token recovery and take over all other SBTs from a user. Only the owner of the account can make a Soul Transfer transaction and merge 2 accounts they owns. + +#### Recoverability within an SBT Registry + +SBT registry can define it's own mechanism to atomically recover all tokens related to one account and execute soul transfer to another account, without going one by one through each SBT issuer (sometimes that might be even not possible). Below we list few ideas a Registry can use to implement recovery: + +- KYC based recovery +- Social recovery + +![Social Recovery, Image via “Decentralized Society”](https://bankless.ghost.io/content/images/public/images/cdb1fc23-6179-44f0-9bfe-e5e5831492f7_1399x680.png) + +SBT Registry based recovery is not part of this specification. + +#### Soul Transfer + +The basic use case is described above. Registry MUST provide a permissionless method to allow any user to execute soul transfer. It is essential part of the standard, but the exact interface of the method is not part of the standard, because registries may adopt different mechanism and require different arguments. + +Soul transfers must be _semi atomic_. That is, the holder account must be non operational (in terms of SBT supply) until the soul transfer is completed. Given the nature of NEAR blockchain, where transactions are limited by gas, big registries may require to implement the Soul Transfer operation in stages. Source and destination accounts should act as a non-soul accounts while the soul transfer operation is ongoing. For example, in the first call, contract can lock the account, and do maximum X amount of transfers. If the list of to be transferred SBTs has not been exhausted, the contract should keep locking the account and remember the last transferred SBT. Subsequent calls by the same user will resume the operation until the list is exhausted. +Soul Transfer must emit the `SoulTransfer` event. + +Example: Alice has two accounts: `alice1` and `alice2` which she used to mint tokens (see [mint diagram](#minting)). She decides to merge the accounts by doing Soul Transfer. + +```mermaid +sequenceDiagram + actor Alice + participant SBT_Registry + + Alice->>SBT_Registry: sbt_soul_transfer(alice1) --accountId alice2 + + Alice-->>+SBT_Registry: sbt_tokens_by_owner(alice2) + SBT_Registry-->>-Alice: [] + + Alice-->>+SBT_Registry: sbt_tokens_by_owner(alice1) + SBT_Registry-->>-Alice: [[SBT_1_Contract, [238]], [SBT_2_Contract, [7991, 7992]]]} +``` + +Implementation Notes: + +- There is a risk of conflict. The standard requires that one account can't have more than one SBT of the same (issuer, class) pair. +- When both `alice1` and `alice2` have SBT of the same (issuer, class) pair, then the transfer should fail. One of the accounts should burn conflicting tokens to be able to continue the soul transfer. +- Soul transfer may require extra confirmation before executing a transfer. For example, if `alice1` wants to do a soul transfer to `alice2`, the contract my require `alice2` approval before continuing the transfer. +- Other techniques may be used to enforce that the source account will be deleted. + +### Renewal + +Soulbound tokens can have an _expire date_. It is useful for tokens which are related to real world certificates with expire time, or social mechanisms (e.g. community membership). Such tokens SHOULD have an option to be renewable. Examples include mandatory renewal with a frequency to check that the owner is still alive, or renew membership to a DAO that uses SBTs as membership gating. +Registry defines `sbt_renew` method allowing issuers to update the token expire date. The issuer can set a the _expire date_ in the past. This is useful if an issuer wants to invalidate the token without removing it. + +### Burning tokens + +Registry MAY expose a mechanism to allow an account to burn an unwanted token. The exact mechanism is not part of the standard and it will depend on the registry implementation. We only define a standard `Burn` event, which must be emitted each time a token is removed from existence. Some registries may forbid accounts to burn their tokens in order to preserve specific claims. Ultimately, NEAR is a public blockchain, and even if a token is burned, it's trace will be preserved. + +### Token Class (multitoken approach) + +SBT tokens can't be fractionized. Also, by definition, there should be only one SBT per token class per user. Examples: user should not be able to receive few badges of the same class, or few proof of attendance to the same event. +However, we identify a need for having to support token classes (aka multitoken interface) in a single contract: + +- badges: one contract. Each badge will have a class (community lead, OG...), and each token will belong to a specific class; +- certificates: one issuer can create certificates of a different class (eg school department can create diplomas for each major and each graduation year). + +We also see a trend in the NFT community and demand for market places to support multi token contracts. + +- In Ethereum community many projects are using [ERC-1155 Multi Token Standard](https://eips.ethereum.org/EIPS/eip-1155). NFT projects are using it for fraction ownership: each token id can have many fungible fractions. +- NEAR [NEP-245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) has elaborated similar interface for both bridge compatibility with EVM chains as well as flexibility to define different token types with different behavior in a single contract. [DevGovGigs Board](https://near.social/#/mob.near/widget/MainPage.Post.Page?accountId=devgovgigs.near&blockHeight=87938945) recently also shows growing interest to move NEP-245 adoption forward. +- [NEP-454](https://github.com/near/NEPs/pull/454) proposes royalties support for multi token contracts. + +We propose that the SBT Standard will support the multi-token idea from the get go. This won't increase the complexity of the contract (in a traditional case, where one contract will only issue tokens of the single class, the `class` argument is simply ignored in the state, and in the functions it's required to be of a constant value, eg `1`) but will unify the interface. +It's up to the smart contract design how the token classes is managed. A smart contract can expose an admin function (example: `sbt_new_class() -> ClassId`) or hard code the pre-registered classes. + +Finally, we require that each token ID is unique within the smart contract. This will allow us to query token only by token ID, without knowing it's class. + +## Smart contract interface + +For the Token ID type we propose `u64` rather than `U128`. `u64` capacity is more than 1e19. If we will mint 10'000 SBTs per second, then it will take us 58'494'241 years to fill the capacity. +Today, the JS integer limit is `2^53-1 ~ 9e15`. Similarly, when minting 10'000 SBTs per second, it will take us 28'561 years to reach the limit. So, we don't need u128 nor the String type. However, if for some reason, we will need to get u64 support for JS, then we can always add another set of methods which will return String, so making it compatible with NFT standard (which is using `U128`, which is a string). Also, it's worth to note, that in 28'000 years JS (if it will still exists) will be completely different. +The number of combinations for a single issuer is much higher in fact: the token standard uses classes. So technically that makes the number of all possible combinations for a single issuer equal `(2^64)^2 ~ 1e38`. For "today" JS it is `(2^53-1)^2 ~ 1e31`. + +Token IDs MUST be created in a sequence to make sure the ID space is not exhausted locally (eg if a registry would decide to introduce segments, it would potentially get into a trap where one of the segments is filled up very quickly). + +```rust +// TokenId and ClassId must be positive (0 is not a valid ID) +pub type TokenId = u64; +pub type ClassId = u64; + +pub struct Token { + pub token: TokenId, + pub owner: AccountId, + pub metadata: TokenMetadata, +} +``` + +The Soulbound Token follows the NFT [NEP-171](https://github.com/near/NEPs/blob/master/neps/nep-0171.md) interface, with few differences: + +- token ID is `u64` (as discussed above). +- token class is `u64`, it's required when minting and it's part of the token metadata. +- `TokenMetadata` doesn't have `title`, `description`, `media`, `media_hash`, `copies`, `extra`, `starts_at` nor `updated_at`. All that attributes except the `updated_at` can be part of the document stored at `reference`. `updated_at` can be tracked easily by indexers. +- We don't have traditional transferability. +- We propose to use more targeted events, to better reflect the event nature. Moreover events are emitted by the registry, so we need to include issuer contract address in the event. + +All time related attributes are defined in milliseconds (as per NEP-171). + +```rust +/// ContractMetadata defines contract wide attributes, which describes the whole contract. +pub struct ContractMetadata { + /// Version with namespace, example: "sbt-1.0.0". Required. + pub spec: String, + /// Issuer Name, required, ex. "Mosaics" + pub name: String, + /// Issuer symbol which can be used as a token symbol, eg Ⓝ, ₿, BTC, MOSAIC ... + pub symbol: String, + /// Icon content (SVG) or a link to an Icon. If it doesn't start with a scheme (eg: https://) + /// then `base_uri` should be prepended. + pub icon: Option, + /// URI prefix which will be prepended to other links which don't start with a scheme + /// (eg: ipfs:// or https:// ...). + pub base_uri: Option, + /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme + /// (eg: https://) then base_uri should be prepended. + pub reference: Option, + /// Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. + pub reference_hash: Option, +} + +/// TokenMetadata defines attributes for each SBT token. +pub struct TokenMetadata { + pub class: ClassId, // token class. Required. Must be non zero. + pub issued_at: Option, // When token was issued or minted, Unix time in milliseconds + pub expires_at: Option, // When token expires, Unix time in milliseconds + /// JSON or an URL to a JSON file with more info. If it doesn't start with a scheme + /// (eg: https://) then base_uri should be prepended. + pub reference: Option, + /// Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included. + pub reference_hash: Option, +} + + +trait SBTRegistry { + /********** + * QUERIES + **********/ + + /// Get the information about specific token ID issued by `issuer` SBT contract. + fn sbt(&self, issuer: AccountId, token: TokenId) -> Option; + + /// Get the information about list of token IDs issued by the `issuer` SBT contract. + /// If token ID is not found `None` is set in the specific return index. + fn sbts(&self, issuer: AccountId, token: Vec) -> Vec>; + + /// Query class ID for each token ID issued by the SBT `issuer`. + /// If token ID is not found, `None` is set in the specific return index. + fn sbt_classes(&self, issuer: AccountId, tokens: Vec) -> Vec>; + + /// Returns total amount of tokens issued by `issuer` SBT contract, including expired + /// tokens. If a revoke removes a token, it must not be included in the supply. + fn sbt_supply(&self, issuer: AccountId) -> u64; + + /// Returns total amount of tokens of given class minted by `issuer`. See `sbt_supply` for + /// information about revoked tokens. + fn sbt_supply_by_class(&self, issuer: AccountId, class: ClassId) -> u64; + + /// Returns total supply of SBTs for a given owner. See `sbt_supply` for information about + /// revoked tokens. + /// If `class` is specified, returns only owner supply of the given class (either 0 or 1). + fn sbt_supply_by_owner( + &self, + account: AccountId, + issuer: AccountId, + class: Option, + ) -> u64; + + /// Query sbt tokens issued by a given contract. + /// `limit` specifies the upper limit of how many tokens we want to return. + /// If `from_token` is not specified, then `from_token` should be assumed + /// to be the first valid token id. If `with_expired` is set to `true` then all the tokens are returned + /// including expired ones otherwise only non-expired tokens are returned. + fn sbt_tokens( + &self, + issuer: AccountId, + from_token: Option, + limit: Option, + with_expired: bool, + ) -> Vec; + + /// Query SBT tokens by owner. + /// `limit` specifies the upper limit of how many tokens we want to return. + /// If `from_class` is not specified, then `from_class` should be assumed to be the first + /// valid class id. If `with_expired` is set to `true` then all the tokens are returned + /// including expired ones otherwise only non-expired tokens are returned. + /// Returns list of pairs: `(Contract address, list of token IDs)`. + fn sbt_tokens_by_owner( + &self, + account: AccountId, + issuer: Option, + from_class: Option, + limit: Option, + with_expired: bool, + ) -> Vec<(AccountId, Vec)>; + + /// checks if an `account` was banned by the registry. + fn is_banned(&self, account: AccountId) -> bool; + + /************* + * Transactions + *************/ + + /// Creates a new, unique token and assigns it to the `receiver`. + /// `token_spec` is a vector of pairs: owner AccountId and TokenMetadata. + /// Each TokenMetadata must have non zero `class`. + /// Must be called by an SBT contract. + /// Must emit `Mint` event. + /// Must provide enough NEAR to cover registry storage cost. + // #[payable] + fn sbt_mint(&mut self, token_spec: Vec<(AccountId, Vec)>) -> Vec; + + /// sbt_recover reassigns all tokens issued by the caller, from the old owner to a new owner. + /// Must be called by a valid SBT issuer. + /// Must emit `Recover` event once all the tokens have been recovered. + /// Requires attaching enough tokens to cover the storage growth. + /// Returns the amount of tokens recovered and a boolean: `true` if the whole + /// process has finished, `false` when the process has not finished and should be + /// continued by a subsequent call. User must keep calling the `sbt_recover` until `true` + /// is returned. + // #[payable] + fn sbt_recover(&mut self, from: AccountId, to: AccountId) -> (u32, bool); + + /// sbt_renew will update the expire time of provided tokens. + /// `expires_at` is a unix timestamp (in miliseconds). + /// Must be called by an SBT contract. + /// Must emit `Renew` event. + fn sbt_renew(&mut self, tokens: Vec, expires_at: u64); + + /// Revokes SBT by burning the token or updating its expire time. + /// Must be called by an SBT contract. + /// Must emit `Revoke` event. + /// Must also emit `Burn` event if the SBT tokens are burned (removed). + fn sbt_revoke(&mut self, tokens: Vec, burn: bool); + + /// Similar to `sbt_revoke`, but revokes all `owner`s tokens issued by the caller. + fn sbt_revoke_by_owner(&mut self, owner: AccountId, burn: bool); + + /// Allows issuer to update token metadata reference and reference_hash. + /// * `updates` is a list of triples: (token ID, reference, reference base64-encoded sha256 hash). + /// Must emit `token_reference` event. + /// Panics if any of the token IDs don't exist. + fn sbt_update_token_references( + &mut self, + updates: Vec<(TokenId, Option, Option)>, + ); +} +``` + +Example **Soul Transfer** interface: + +```rust + /// Transfers atomically all SBT tokens from one account to another account. + /// The caller must be an SBT holder and the `recipient` must not be a banned account. + /// Returns the amount of tokens transferred and a boolean: `true` if the whole + /// process has finished, `false` when the process has not finished and should be + /// continued by a subsequent call. + /// Emits `Ban` event for the caller at the beginning of the process. + /// Emits `SoulTransfer` event only once all the tokens from the caller were transferred + /// and at least one token was trasnfered (caller had at least 1 sbt). + /// + User must keep calling the `sbt_soul_transfer` until `true` is returned. + /// + If caller does not have any tokens, nothing will be transfered, the caller + /// will be banned and `Ban` event will be emitted. + #[payable] + fn sbt_soul_transfer( + &mut self, + recipient: AccountId, + ) -> (u32, bool); +``` + +### SBT Issuer interface + +SBTContract is the minimum required interface to be implemented by issuer. Other methods, such as a mint function, which requests the registry to proceed with token minting, is specific to an Issuer implementation (similarly, mint is not part of the FT standard). + +```rust +pub trait SBTContract { + /// returns contract metadata + fn sbt_metadata(&self) -> ContractMetadata; +} +``` + +SBT issuer smart contracts may implement NFT query interface to make it compatible with NFT tools. In that case, the contract should proxy the calls to the related registry. Note, we use U64 type rather than U128. However, SBT issuer must not emit NFT related events. + +```rust +trait SBTNFT { + fn nft_total_supply(&self) -> U64; + // here we index by token id instead of by class id (as done in `sbt_tokens_by_owner`) + fn nft_tokens_for_owner(&self, account_id: AccountId, from_index: Option, limit: Option) -> Vec; + fn nft_supply_for_owner(&self, account_id: AccountId) -> U64; +} +``` + +### Events + +Event design principles: + +- Events don't need to repeat all function arguments - these are easy to retrieve by indexer (events are consumed by indexers anyway). +- Events must include fields necessary to identify subject matters related to use case. +- When possible, events should contain aggregated data, with respect to the standard function related to the event. + +```typescript +// only valid integer numbers (without rounding errors). +type u64 = number; + +type Nep393Event { + standard: "nep393"; + version: "1.0.0"; + event: "mint" | "recover" | "renew" | "revoke" | "burn" | "ban" | "soul_transfer" | "token_reference" ; + data: Mint | Recover | Renew | Revoke | Burn | Ban[] | SoulTransfer | TokenReference; +} + +/// An event emitted by the Registry when new SBT is created. +type Mint { + issuer: AccountId; // SBT Contract minting the tokens + tokens: (AccountId, u64[])[]; // list of pairs (token owner, TokenId[]) +} + +/// An event emitted when a recovery process succeeded to reassign SBTs, usually due to account +/// access loss. This action is usually requested by the owner, but executed by an issuer, +/// and doesn't trigger Soul Transfer. Registry reassigns all tokens assigned to `old_owner` +/// that were ONLY issued by the `ctr` SBT Contract (hence we don't need to enumerate the +/// token IDs). +/// Must be emitted by an SBT registry. +type Recover { + issuer: AccountId // SBT Contract recovering the tokens + old_owner: AccountId; // current holder of the SBT + new_owner: AccountId; // destination account. +} + +/// An event emitted when existing tokens are renewed. +/// Must be emitted by an SBT registry. +type Renew { + issuer: AccountId; // SBT Contract renewing the tokens + tokens: u64[]; // list of token ids. +} + +/// An event emitted when existing tokens are revoked. +/// Revoked tokens will continue to be listed by the registry but they should not be listed in +/// a wallet. See also `Burn` event. +/// Must be emitted by an SBT registry. +type Revoke { + issuer: AccountId; // SBT Contract revoking the tokens + tokens: u64[]; // list of token ids. +} + +/// An event emitted when existing tokens are burned and removed from the registry. +/// Must be emitted by an SBT registry. +type Burn { + issuer: AccountId; // SBT Contract burning the tokens + tokens: u64[]; // list of token ids. +} + +/// An event emitted when an account is banned within the emitting registry. +/// Registry must add the `account` to a list of accounts that are not allowed to get any SBT +/// in the future. +/// Must be emitted by an SBT registry. +type Ban = AccountId; + +/// An event emitted when soul transfer is happening: all SBTs owned by `from` are transferred +/// to `to`, and the `from` account is banned (can't receive any new SBT). +/// Must be emitted by an SBT registry. +/// Registry MUST also emit `Ban` whenever the soul transfer happens. +type SoulTransfer { + from: AccountId; + to: AccountId; +} + +/// An event emitted when an issuer updates token metadata reference of existing SBTs. +/// Must be emitted by an SBT registry. +type TokenReference { + issuer: AccountId; // Issuer account + tokens: u64[]; // list of token ids. +} + +/// An event emitted when existing token metadata references are updated. +type TokenReference = u64[]; // list of token ids. +``` + +Whenever a recovery is made in a way that an existing SBT is burned, the `Burn` event MUST be emitted. If `Revoke` burns token then `Burn` event MUST be emitted instead of `Revoke`. + +### Example SBT Contract functions + +Although the transaction functions below are not part of the SBT smart contract standard (depending on a use case, they may have different parameters), we present here an example interface for SBT issuance and we also provide a reference implementation. +These functions must relay calls to an SBT registry, which will emit appropriate events. +We recommend that all functions related to an event will take an optional `memo: Option` argument for accounting purposes. + +```rust +trait SBT { + /// Must provide enough NEAR to cover registry storage cost. + // #[payable] + fn sbt_mint( + &mut self, + account: AccountId, + metadata: TokenMetadata, + memo: Option, + ) -> TokenId; + + /// Creates a new, unique token and assigns it to the `receiver`. + /// `token_spec` is a vector of pairs: owner AccountId and TokenMetadata. + /// Must provide enough NEAR to cover registry storage cost. + // #[payable] + fn sbt_mint_multi( + &mut self, + token_spec: Vec<(AccountId, TokenMetadata)>, + memo: Option, + ) -> Vec; + + // #[payable] + fn sbt_recover(&mut self, from: AccountId, to: AccountId, memo: Option); + + fn sbt_renew(&mut self, tokens: Vec, expires_at: u64, memo: Option); + + fn sbt_revoke(token: Vec, memo: Option) -> bool; +} +``` + +## Reference Implementation + +- Common [type definitions](https://github.com/near-ndc/i-am-human/tree/main/contracts/sbt) (events, traits). +- [I Am Human](https://github.com/near-ndc/i-am-human) registry and issuers. + +## Consequences + +Being fully compatible with NFT standard is a desirable. However, given the requirements related to _soul transfer_ we didn't find an applaudable solution. Also we decided to use u64 as a Token ID, diverging further from the NFT NEP-171 standard. + +Given that our requirements are much striker, we had to reconsider the level of compatibility with NEP-171 NFT. +There are many examples where NFT standards are improperly implemented. Adding another standard with different functionality but equal naming will cause lots of problems and misclassifications between NFT and SBT. + +### Positive + +- Template and set of guidelines for creating SBT tokens. +- Ability to create SBT aggregators. +- An SBT standard with recoverability mechanism provides a unified model for multiple primitives such as non KYC identities, badges, certificates etc... +- SBT can be further used for "lego" protocols, like: Proof of Humanity (discussed for NDC Governance), undercollateralized lending, role based authentication systems, innovative economic and social applications... +- Standard recoverability mechanism. +- SBT are considered as a basic primitive for Decentralized Societies. +- new way to implement Sybil attack resistance. + +### Neutral + +- The API partially follows the NEP-171 (NFT) standard. The proposed design is to have native SBT API and make it possible for issuer contracts to support NFT based queries if needed (such contract will have a limitation of only issuing SBTs with one `ClassId` only). + +### Negative + +- New set of events to be handled by the indexer and wallets. +- Complexity of integration with a registry: all SBT related transactions must go through Registry. + +### Privacy Notes + +> Blockchain-based systems are public by default. Any relationship that is recorded on-chain is immediately visible not just to the participants, but also to anyone in the entire world. Some privacy can be retained by having multiple pseudonyms: a family Soul, a medical Soul, a professional Soul, a political Soul each carrying different SBTs. But done naively, it could be very easy to correlate these Souls to each other. +> The consequences of this lack of privacy are serious. Indeed, without explicit measures taken to protect privacy, the “naive” vision of simply putting all SBTs on-chain may well make too much information public for many applications. + +-- Decentralized Society + +There are multiple ways how an identity can be doxxed using chain data. SBT, indeed provides more data about account. +The standard allows for few anonymization methods: + +- not providing any data in the token metadata (reference...) or encrypt the reference. +- anonymize issuers (standard allows to have many issues for the same entity) and mix it with different class ids. These are just a numbers. + +Perfect privacy can only be done with solid ZKP, not off-chain walls. + +Implementations must not store any personal information on chain. + +## Changelog + +### v1.0.0 + +The Contract Standards Working Group members approved this NEP on June 30, 2023 ([meeting recording](https://youtu.be/S1An5CDG154)). + +#### Benefits + +- SBTs as any other kind of a token are essential primitive to represent real world use cases. This standards provides a model and a guideline for developers to build SBT based solutions. +- Token standards are key for composability. +- Wallet and tools needs a common interface to operate tokens. + +#### Concerns + +| # | Concern | Resolution | Status | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | +| 1 | [Robert] Should we Emit NEP-171 Mint and NEP-171 Burn by the SBT contract (in addition to SBT native events emitted by the registry)? If the events will be emitted by registry, then we need new events to include the contract address. | Don't emit NFT events. SBT is not NFT. Support: @alexastrum | resolved | +| 2 | [Robert] remove `memo` in events. The `memo` is already part of the transaction, and should not be needed to identify transactions. Processes looking for events, can easily track transaction through event and recover `memo` if needed. | Removed, consequently also removed from registry transactions . Support: @alexastrum | resolved | +| 3 | [Token Spam](https://github.com/near/NEPs/pull/393/#discussion_r1163938750) | We have a `Burn` event. Added example `sbt_burn` function, but keeping it not as a part of required interface. Event should be enough. | resolved | +| 4 | [Multiple registries](https://github.com/near/NEPs/pull/393/#discussion_r1163951624). Registry source of truth [comment](https://github.com/near/NEPs/pull/393/#issuecomment-1531766643) | This is a part of the design: permissionless approach. [Justification for registry](https://github.com/near/NEPs/pull/393/#issuecomment-1540621077) | resolved | +| 5 | [Robert] Approve the proposed multi-token | Support: @alexastrum | resolved | +| 6 | [Robert] Use of milliseconds as a time unit. | Use milliseconds. | resolved | +| 7 | Should a `burn` function be part of a standard or a recommendation? | We already have the Burn event. A call method should not be part of the standard interface (similarly to FT and NFT). | resolved | +| 8 | [Robert] Don't include `sbt_soul_transfer` in the standard interface, [comment](https://github.com/near/NEPs/pull/393#issuecomment-1506969996). | Moved outside of the required interface. | resolved | +| 9 | [Privacy](https://github.com/near/NEPs/pull/393/#issuecomment-1504309947) | Concerns have been addressed: [comment-1](https://github.com/near/NEPs/pull/393/#issuecomment-1504485420) and [comment2](https://github.com/near/NEPs/pull/393/#issuecomment-1505958549) | resolved | +| 10 | @frol [suggested](https://github.com/near/NEPs/pull/393/#discussion_r1247879778) to use a struct in `sbt_recover` and `sbt_soul_transfer`. | Motivation to use pair `(number, bool)` rather than follow a common Iterator Pattern. Rust uses `Option` type for that, that works perfectly for languages with native Option type, but creates a "null" problem for anything else. Other common way to implement Iterator is the presented pair, which doesn't require extra type definition and reduces code size. | new | + +## Copyright + +[Creative Commons Attribution 4.0 International Public License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) From 6eee9dc14ff64dd2eac24e4ae704e0b8fbb18ba9 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Sun, 22 Oct 2023 16:13:21 +0300 Subject: [PATCH 08/12] nep-393: add bool return to sbt_revoke_by_owner (#510) In the [reference](https://github.com/near-ndc/i-am-human/blob/cfc92b1/contracts/registry/src/registry.rs#L362) implementation we return `bool` in the `registry.sbt_revoke_by_owner` to signal if the operation should continue or it's finished. We forgot to push this update to the standard. This update also makes the API consistent with `sbt_recover` and `sbt_soul_transfer` --- neps/nep-0393.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/neps/nep-0393.md b/neps/nep-0393.md index 67b3361c2..9c10b0469 100644 --- a/neps/nep-0393.md +++ b/neps/nep-0393.md @@ -404,8 +404,15 @@ trait SBTRegistry { /// Must also emit `Burn` event if the SBT tokens are burned (removed). fn sbt_revoke(&mut self, tokens: Vec, burn: bool); - /// Similar to `sbt_revoke`, but revokes all `owner`s tokens issued by the caller. - fn sbt_revoke_by_owner(&mut self, owner: AccountId, burn: bool); + /// Similar to `sbt_revoke`. Allows SBT issuer to revoke all tokens by holder either by + /// burning or updating their expire time. When an owner has many tokens from the issuer, + /// the issuer may need to call this function multiple times, until all tokens are revoked. + /// Retuns true if all the tokens were revoked, false otherwise. + /// If false is returned issuer must call the method until true is returned + /// Must be called by an SBT contract. + /// Must emit `Revoke` event. + /// Must also emit `Burn` event if the SBT tokens are burned (removed). + fn sbt_revoke_by_owner(&mut self, owner: AccountId, burn: bool) -> bool; /// Allows issuer to update token metadata reference and reference_hash. /// * `updates` is a list of triples: (token ID, reference, reference base64-encoded sha256 hash). From d49bf8fe0cd7813e76a3a08993c1213faacc07c4 Mon Sep 17 00:00:00 2001 From: Jakob Meier Date: Sun, 22 Oct 2023 18:25:34 +0200 Subject: [PATCH 09/12] chore: move nep-0264_promise_gas_ratio to NEPs (#506) --- README.md | 1 + .../nep-0264.md | 36 +++++++++++-------- 2 files changed, 23 insertions(+), 14 deletions(-) rename specs/Proposals/0264-promise_gas_ratio.md => neps/nep-0264.md (95%) diff --git a/README.md b/README.md index 6b81d1867..beb040f80 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ Changes to the protocol specification and standards are called NEAR Enhancement | [0181](https://github.com/near/NEPs/blob/master/neps/nep-0181.md) | Non Fungible Token Enumeration | @chadoh @thor314 | Final | | [0199](https://github.com/near/NEPs/blob/master/neps/nep-0199.md) | Non Fungible Token Royalties and Payouts | @thor314 @mattlockyer | Final | | [0245](https://github.com/near/NEPs/blob/master/neps/nep-0245.md) | Multi Token Standard | @zcstarr @riqi @jriemann @marcos.sun | Review | +| [0264](https://github.com/near/NEPs/blob/master/neps/nep-0264.md) | Promise Gas Weights | @austinabell | Final | | [0297](https://github.com/near/NEPs/blob/master/neps/nep-0297.md) | Events Standard | @telezhnaya | Final | | [0330](https://github.com/near/NEPs/blob/master/neps/nep-0330.md) | Source Metadata | @BenKurrek | Review | | [0366](https://github.com/near/NEPs/blob/master/neps/nep-0366.md) | Meta Transactions | @ilblackdragon @e-uleyskiy @fadeevab | Draft | diff --git a/specs/Proposals/0264-promise_gas_ratio.md b/neps/nep-0264.md similarity index 95% rename from specs/Proposals/0264-promise_gas_ratio.md rename to neps/nep-0264.md index 17306bbfa..e239a4b41 100644 --- a/specs/Proposals/0264-promise_gas_ratio.md +++ b/neps/nep-0264.md @@ -1,31 +1,35 @@ -- Proposal Name: `promise_gas_weight` -- Start Date: 2021-09-30 -- NEP PR: [nearprotocol/neps#264](https://github.com/nearprotocol/neps/pull/264) -- Issue(s): https://github.com/near/near-sdk-rs/issues/526 +--- +NEP: 264 +Title: Utilization of unspent gas for promise function calls +Authors: Austin Abell +DiscussionsTo: https://github.com/near/NEPs/pull/264 +Status: Approved +Type: Protocol +Version: 1.0.0 +Created: 2021-09-30 +LastUpdated: 2022-05-26 +--- # Summary -[summary]: #summary This proposal is to introduce a new host function on the NEAR runtime that allows for scheduling cross-contract function calls using a percentage/weight of the remaining gas in addition to the statically defined amount. This will enable async promise execution to use the remaining gas more efficiently by utilizing unspent gas from the current transaction. # Motivation -[motivation]: #motivation We are proposing this to be able to utilize gas more efficiently but also to improve the devX of cross-contract calls. Currently, developers must guess how much gas will remain after the current transaction finishes and if this value is too little, the transaction will fail, and if it is too large, gas will be wasted. Therefore, these cross-contract calls need a reasonable default of splitting unused gas efficiently for basic cases without sacrificing the ability to configure the gas amount attached at a granular level. Currently, gas is allocated very inefficiently, requiring more prepaid gas or failed transactions when the allocations are imprecise. # Guide-level explanation -[guide-level-explanation]: #guide-level-explanation This host function is similar to [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526), except with an additional parameter that lets you specify how much of the excess gas should be attached to the function call. This parameter is a weight value that determines how much of the excess gas is attached to each function. So, for example, if there is 40 gas leftover and three function calls that select weights of 1, 5, and 2, the runtime will add 5, 25, and 10 gas to each function call. A developer can specify whether they want to attach a fixed amount of gas, a weight of remaining gas, or both. If at least one function call uses a weight of remaining gas, then all excess gas will be attached to future calls. This proposal allows developers the ability to utilize prepaid gas more efficiently than currently possible. # Reference-level explanation -[reference-level-explanation]: #reference-level-explanation This host function would need to be implemented in `nearcore` and parallel [`promise_batch_action_function_call`](https://github.com/near/nearcore/blob/7d15bbc996282c8ae8f15b8f49d110fc901b84d8/runtime/near-vm-logic/src/logic.rs#L1526). Most details of these functions will be consistent, except that there will be additional bookkeeping for keeping track of which functions specified a weight for unused gas. This will not affect or replace any existing host functions, but this will likely require a slightly higher gas cost than the original `promise_batch_action_function_call` host function due to this additional overhead. This host function definition would look like this (as a Rust consumer): + ```rust /// Appends `FunctionCall` action to the batch of actions for the given promise pointed by /// `promise_idx`. This function allows not specifying a specific gas value and allowing the @@ -70,7 +74,8 @@ The only difference from the existing API is `gas_weight` added as another param As for calculations, the remaining gas at the end of the transaction can be floor divided by the sum of all the weights tracked. Then, after getting this value, just attach that value multiplied by the weight gas to each function call action. For example, if there are three weights, `a`, `b`, `c`: -``` + +```rust weight_sum = a + b + c a_gas += remaining_gas * a / weight_sum b_gas += remaining_gas * b / weight_sum @@ -84,6 +89,7 @@ Any remaining gas that is not allocated to any of these function calls will be a This protocol change will allow cross-contract calls to provide a fixed amount of gas and/or adjust the weight of unused gas to use. If neither is provided, it will default to using a weight of 1 for each and no static amount of gas. If no function modifies this weight, the runtime will split the unused gas evenly among all function calls. Currently, the API for a cross-contract call looks like: + ```rust let contract_account_id: AccountId = todo!(); ext::some_method(/* parameters */, contract_account_id, 0 /* deposit amount */, 5_000_000_000_000 /* static amount of gas to attach */) @@ -105,7 +111,6 @@ cross_contract::ext(contract_account_id) At a basic level, a developer has only to include the parameters for the function call and specify the account id of the contract being called. Currently, only the amount can be optional because there is no way to set a reasonable default for the amount of gas to use for each function call. # Drawbacks -[drawbacks]: #drawbacks - Complexity in refactoring to handle assigning remaining gas at the end of a transaction - Complexity in extra calculations for assigning gas will make the host function slightly more expensive than the base one. It is not easy to create an API on the SDK level that can decide which host function to call if dynamic gas assigning is needed or not. If both are used, the size of the wasm binary is trivially larger by including both host functions @@ -116,18 +121,19 @@ At a basic level, a developer has only to include the parameters for the functio - Keep in mind that it will also be positive because transactions will generally succeed more often due to gas more efficiently # Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives Alternative 1 (fraction parameters): The primary alternative is using a numerator and denominator to represent a fraction instead of a weight. This alternative would be equivalent to the one listed above except for two u64 additional parameters instead of just the one for weight. I'll list the tradeoff as pros and cons: Pros: + - Can under-utilize the gas for the current transaction to limit gas allowed for certain functions - This could take responsibility away from DApp users because they would not have to worry less about attaching too much prepaid gas - Thinking in terms of fractions may be more intuitive for some developers - Might future proof better if we ever need this ability in the future, want to minimize the number of host functions created at all costs Cons: + - More complicated logic/edge cases to handle to make sure the percentages don't sum to greater than 100% (or adjusting if they do) - Precision loss from dividing integers may lead to unexpected results - To get closer to expected, we could use floats for the division, but this gets messy @@ -139,30 +145,32 @@ Alternative 2 (handle within contract/SDK): The other alternative is to handle all of this logic on the contract side, as seen by [this PR](https://github.com/near/near-sdk-rs/pull/523). This is much less feasible/accurate because there is only so much information available within the runtime, and gas costs and internal functionality may not always be the same. As discussed on [the respective issue](https://github.com/near/near-sdk-rs/issues/526), this alternative seems to be very infeasible. Pros: + - No protocol change is needed - Can still have improved API as with protocol change Cons: + - Additional bloat to every contract, even ones that don't use the pattern (~5kb in PoC, even with simple estimation logic) - Still inaccurate gas estimations, because at the point of calculation, we cannot know how much gas will be used for assigning gas values as well as gas consumed after the transaction ends - This leads to either underutilizing or having transactions fail when using too much gas if trying to estimate how much gas will be left - Prone to breaking existing contracts on protocol changes that affect gas usage or logic of runtime # Unresolved questions -[unresolved-questions]: #unresolved-questions What needs to be addressed before this gets merged: ~~- How much refactoring exactly is needed to handle this pattern?~~ ~~- Can we keep a queue of receipt and action indices with their respective weights and update their gas values after the current method is executed? Is there a cleaner way to handle this while keeping order?~~ ~~- Do we want to attach the gas lost due to precision on division to any function?~~ - - The remaining gas is now attached to the last function call + +- The remaining gas is now attached to the last function call What would be addressed in future independently of the solution: + - How many users would expect the ability to refund part of the gas after the initial transaction? (is this worth considering the API difference of using fractions rather than weights) - Will weights be an intuitive experience for developers? # Future possibilities -[future-possibilities]: #future-possibilities The future change that would extend from this being implemented is a much cleaner API for the SDKs. As mentioned previously in the alternatives section, the API changes from [the changes tested on the SDK](https://github.com/near/near-sdk-rs/pull/523) will remain, but without the overhead from implementing this on the contract level. Thus, not only can this be implemented in Rust, but it will also allow a consistent API for existing and future SDK languages to build on. From b9c3d15e70fe19ec23fe3aadbf323bbdd81fa069 Mon Sep 17 00:00:00 2001 From: nikurt <86772482+nikurt@users.noreply.github.com> Date: Fri, 3 Nov 2023 23:02:30 +0100 Subject: [PATCH 10/12] NEP-514: Reducing the number of Block Producer Seats in `testnet` (#514) --- README.md | 4 +- nep-0000-template.md | 26 +++---- neps/nep-0514.md | 166 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 182 insertions(+), 14 deletions(-) create mode 100644 neps/nep-0514.md diff --git a/README.md b/README.md index beb040f80..ecdaf7baa 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Changes to the protocol specification and standards are called NEAR Enhancement ## NEPs | NEP # | Title | Author | Status | -| ----------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------- | ------ | +| ----------------------------------------------------------------- |------------------------------------------|---------------------------------------------|--------| | [0001](https://github.com/near/NEPs/blob/master/neps/nep-0001.md) | NEP Purpose and Guidelines | @jlogelin | Living | | [0021](https://github.com/near/NEPs/blob/master/neps/nep-0021.md) | Fungible Token Standard (Deprecated) | @evgenykuzyakov | Final | | [0141](https://github.com/near/NEPs/blob/master/neps/nep-0141.md) | Fungible Token Standard | @evgenykuzyakov @oysterpack | Final | @@ -30,6 +30,8 @@ Changes to the protocol specification and standards are called NEAR Enhancement | [0399](https://github.com/near/NEPs/blob/master/neps/nep-0399.md) | Flat Storage | @Longarithm @mzhangmzz | Review | | [0448](https://github.com/near/NEPs/blob/master/neps/nep-0448.md) | Zero-balance Accounts | @bowenwang1996 | Final | | [0455](https://github.com/near/NEPs/blob/master/neps/nep-0455.md) | Parameter Compute Costs | @akashin @jakmeier | Final | +| [0514](https://github.com/near/NEPs/blob/master/neps/nep-0514.md) | Fewer Block Producer Seats in `testnet` | @nikurt | Final | + ## Specification diff --git a/nep-0000-template.md b/nep-0000-template.md index 772979545..47b65bdac 100644 --- a/nep-0000-template.md +++ b/nep-0000-template.md @@ -58,11 +58,11 @@ See example above --> [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: -- Its interaction with other features is clear. -- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. -- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. +* Its interaction with other features is clear. +* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) +* It is reasonably clear how the feature would be implemented. +* Corner cases are dissected by example. +* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] @@ -84,15 +84,15 @@ The section should return to the examples given in the previous section, and exp ### Positive -- p1 +* p1 ### Neutral -- n1 +* n1 ### Negative -- n1 +* n1 ### Backwards Compatibility @@ -102,9 +102,9 @@ The section should return to the examples given in the previous section, and exp [Explain any issues that warrant further discussion. Considerations -- What parts of the design do you expect to resolve through the NEP process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] +* What parts of the design do you expect to resolve through the NEP process before this gets merged? +* What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +* What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] ## Changelog @@ -118,8 +118,8 @@ The section should return to the examples given in the previous section, and exp > List of benefits filled by the Subject Matter Experts while reviewing this version: -- Benefit 1 -- Benefit 2 +* Benefit 1 +* Benefit 2 #### Concerns diff --git a/neps/nep-0514.md b/neps/nep-0514.md new file mode 100644 index 000000000..10a3bae80 --- /dev/null +++ b/neps/nep-0514.md @@ -0,0 +1,166 @@ +--- +NEP: 514 +Title: Reducing the number of Block Producer Seats in `testnet` +Authors: Nikolay Kurtov +Status: New +DiscussionsTo: https://github.com/nearprotocol/neps/pull/514 +Type: Protocol +Version: 1.0.0 +Created: 2023-10-25 +LastUpdated: 2023-10-25 +--- + + +## Summary + +This proposal aims to adjust the number of block producer seats on `testnet` in +order to ensure a positive number of chunk-only producers present in `testnet` +at all times. + +## Motivation + +The problem is that important code paths are not exercised in `testnet`. This +makes `mainnet` releases more risky than they have to be, and greatly slows +down development of features related to chunk-only producers, such as State +Sync. + +That is because `testnet` has fewer validating nodes than the number of block +producer seats configured. + +The number of validating nodes on `testnet` is somewhere in the range of +[26, 46], which means that all validating nodes are block producers and none of +them are chunk-only producers. [Grafana](https://nearinc.grafana.net/goto/7Kh81P7IR?orgId=1). + +`testnet` configuration is currently the following: + +* `"num_block_producer_seats": 100,` +* `"num_block_producer_seats_per_shard": [ 100, 100, 100, 100 ],` +* `"num_chunk_only_producer_seats": 200,` + +It's evident that the 100 block producer seats significantly outnumber the +validating nodes in `testnet`. + +An alternative solution to the problem stated above can be the following: + +1. Encourage the community to run more `testnet` validating nodes +1. Release owners or developers of features start a lot of validating nodes to +1. ensure `testnet` gets some chunk-only producing nodes. +1. Exercise the unique code paths in a separate chain, a-la `localnet`. + +Let's consider each of these options. + +### More community nodes + +This would be the ideal perfect situation. More nodes joining will make +`testnet` more similar to `mainnet`, which will have various positive effects +for protocol developers and dApp developers. + +However, this option is expensive, because running a validating node costs +money, and most community members can't afford spending that amount of money for +the good of the network. + +### More protocol developer nodes + +While this option may seem viable, it poses significant financial challenges for +protocol development. The associated computational expenses are exorbitantly +high, making it an impractical choice for sustainable development. + +### Test in separate chains + +That is the current solution, and it has significant drawbacks: + +* Separate chains are short-lived and may miss events critical to the unique + code paths of chunk-only producers +* Separate chains need special attention to be configured in a way that + accommodates for chunk-only producers. Most test cases are not concerned about + them, and don't exercise the unique code paths. +* Separate chains can't process real transaction traffic. The traffic must + either be synthetic or "inspired" by real traffic. +* Each such test has a significant cost of running multiple nodes, in some + cases, tens of nodes. + +## Specification + +The proposal suggests altering the number of block producer seats to ensure that +a portion of the `testnet` validating nodes become chunk-only producers. + +The desired `testnet` configuration is the following: + +* `"num_block_producer_seats": 20,` +* `"num_block_producer_seats_per_shard": [ 20, 20, 20, 20 ],` +* `"num_chunk_only_producer_seats": 100,` + +I suggest to implement the change for all networks that are not `mainnet` and +have `use_production_config` in the genesis file. `use_production_config` is a +sneaky parameter in `GenesisConfig` that lets protocol upgrades to change +network's `GenesisConfig`. + +I don't have a solid argument for lowering the number of chunk producer seats, +but that reflects the reality that we don't expect a lot of nodes joining +`testnet`. It also makes it easier to test the case of too many validating nodes +willing to join a network. + +## Reference Implementation + +[#9563](https://github.com/near/nearcore/pull/9563) + +If `use_production_config`, check whether `chain_id` is eligible, then change +the configuration as specified above. + +## Security Implications + +The block production in `testnet` becomes more centralized. It's not a new +concern as 50% of stake is already owned by nodes operated by the protocol +developers. + +## Alternatives + +See above. + +## Future possibilities + +Adjust the number of block and chunk producer seats according to the development +of the number of `testnet` validating nodes. + +## Consequences + +### Positive + +* Chunk-only production gets tested in `testnet` +* Development of State Sync and other features related to chunk-only producers accelerates + +### Neutral + +* `testnet` block production becomes more centralized + +### Negative + +* Any? + +### Backwards Compatibility + +During the protocol upgrade, some nodes will become chunk-only producers. + +The piece of code that updates `testnet` configuration value will need to be +kept in the database in case somebody wants to generate `EpochInfo` compatible +with the protocol versions containing the implementation of this NEP. + +## Changelog + +### 1.0.0 - Initial Version + +The Protocol Working Group members approved this NEP on Oct 26, 2023. + +[Zulip link](https://near.zulipchat.com/#narrow/stream/297873-pagoda.2Fnode/topic/How.20to.20test.20a.20chunk-only.20producer.20node.20in.20testnet.3F/near/396090090) + +#### Benefits + +See [Consequences](#consequences). + +#### Concerns + +See [Consequences](#consequences). + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From a5bb580a037cde12017d594a6fa0a817c748de32 Mon Sep 17 00:00:00 2001 From: Bowen Wang Date: Thu, 16 Nov 2023 22:51:48 +0300 Subject: [PATCH 11/12] NEP-0492: Restrict creation of Ethereum Addresses (#492) Proposal to restrict the creation of Ethereum addresses on NEAR to prevent loss of funds and enable future possibilities for Ethereum wallets to support NEAR transactions. Implementation: https://github.com/near/nearcore/pull/9365 --- neps/nep-0492.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 neps/nep-0492.md diff --git a/neps/nep-0492.md b/neps/nep-0492.md new file mode 100644 index 000000000..a6dfd434f --- /dev/null +++ b/neps/nep-0492.md @@ -0,0 +1,74 @@ +--- +NEP: 492 +Title: Restrict creation of Ethereum Addresses +Authors: Bowen Wang +Status: Final +DiscussionsTo: https://github.com/near/NEPs/pull/492 +Type: Protocol +Version: 0.0.0 +Created: 2023-07-27 +LastUpdated: 2023-07-27 +--- + +## Summary + +This proposal aims to restrict the creation of top level accounts (other than implicit accounts) on NEAR to both prevent loss of funds due to careless user behaviors and scams +and create possibilities for future interopability solutions. + +## Motivation + +Today an [Ethereum address](https://ethereum.org/en/developers/docs/accounts/) such as "0x32400084c286cf3e17e7b677ea9583e60a000324" is a valid account on NEAR and because it is longer than 32 characters, +anyone can create such an account. This has unfortunately caused a few incidents where users lose their funds due to either a scam or careless behaviors. +For example, when a user withdraw USDT from an exchange to their NEAR account, it is possible that they think they withdraw to Ethereum and therefore enter their Eth address. +If this address exists on NEAR, then the user would lose their fund. A malicious actor could exploit this can create known Eth smart contract addresses on NEAR to trick users to send tokens to those addresses. With the proliferation of BOS gateways, including Ethereum ones, such exploits may become more common as users switch between NEAR wallets and Ethereum wallets (mainly metamask). + +In addition to prevent loss of funds for users, this change allows the possibility of Ethereum wallets supporting NEAR transactions, which could enable much more adoption of NEAR. The exact details of how that would be done is outside the scope of this proposal. + +There are currently ~5000 Ethereum addresses already created on NEAR. It is also outside the scope of this proposal to discuss what to do with them. + +## Specification + +The proposed change is quite simple. Only the protocol registrar account can create top-level accounts that are not implicit accounts + +## Reference Implementation + +The implementation roughly looks as follows: + +```Rust +fn action_create_account(...) { + ... + if account_id.is_top_level() && !account_id.is_implicit() + && predecessor_id != &account_creation_config.registrar_account_id + { + // Top level accounts that are not implicit can only be created by registrar + result.result = Err(ActionErrorKind::CreateAccountOnlyByRegistrar { + account_id: account_id.clone(), + registrar_account_id: account_creation_config.registrar_account_id.clone(), + predecessor_id: predecessor_id.clone(), + } + .into()); + return; + } + ... +} +``` + +## Alternatives + +There does not appear to be a good alternative for this problem. + +## Future possibilities + +Ethereum wallets such as Metamask could potentially support NEAR transactions through meta transactions. + +## Consequences + +In the short term, no new top-level accounts would be allowed to be created, but this change would not create any problem for users. + +### Backwards Compatibility + +For Ethereum addresses specifically, there are ~5000 existing ones, but this proposal per se do not deal with existing accounts. + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From f2f981a8c22bde3b10df536e7e34ce9dc37a90b3 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 22 Jan 2024 10:53:55 -0800 Subject: [PATCH 12/12] NEP-508: Resharding v2 (#508) Draft for NEP-508, which introduces a new version of resharding implementation. --- neps/nep-0508.md | 267 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 267 insertions(+) create mode 100644 neps/nep-0508.md diff --git a/neps/nep-0508.md b/neps/nep-0508.md new file mode 100644 index 000000000..36fffe5ce --- /dev/null +++ b/neps/nep-0508.md @@ -0,0 +1,267 @@ +--- +NEP: 508 +Title: Resharding v2 +Authors: Waclaw Banasik, Shreyan Gupta, Yoon Hong +Status: Draft +DiscussionsTo: https://github.com/near/nearcore/issues/8992 +Type: Protocol +Version: 1.0.0 +Created: 2023-09-19 +LastUpdated: 2023-11-14 +--- + +## Summary + +This proposal introduces a new implementation for resharding and a new shard layout for the production networks. + +In essence, this NEP is an extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which was focused on splitting one shard into multiple shards. + +We are introducing resharding v2, which supports one shard splitting into two within one epoch at a pre-determined split boundary. The NEP includes performance improvement to make resharding feasible under the current state as well as actual resharding in mainnet and testnet (To be specific, splitting the largest shard into two). + +While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseeable future, it does not serve all use cases, such as dynamic resharding. + +## Motivation + +Currently, NEAR protocol has four shards. With more partners onboarding, we started seeing that some shards occasionally become over-crowded with respect to total state size and number of transactions. In addition, with state sync and stateless validation, validators will not need to track all shards and validator hardware requirements can be greatly reduced with smaller shard size. With future in-memory tries, it's also important to limit the size of individual shards. + +## Specification + +### High level assumptions + +* Flat storage is enabled. +* Shard split boundary is predetermined and hardcoded. In other words, necessity of shard splitting is manually decided. +* For the time being resharding as an event is only going to happen once but we would still like to have the infrastructure in place to handle future resharding events with ease. +* Merkle Patricia Trie is the underlying data structure for the protocol state. +* Epoch is at least 6 hrs long for resharding to complete. + +### High level requirements + +* Resharding must be fast enough so that both state sync and resharding can happen within one epoch. +* Resharding should work efficiently within the limits of the current hardware requirements for nodes. +* Potential failures in resharding may require intervention from node operator to recover. +* No transaction or receipt must be lost during resharding. +* Resharding must work regardless of number of existing shards. +* No apps, tools or code should hardcode the number of shards to 4. + +### Out of scope + +* Dynamic resharding + * automatically scheduling resharding based on shard usage/capacity + * automatically determining the shard layout +* Merging shards or boundary adjustments +* Shard reshuffling + +### Required protocol changes + +A new protocol version will be introduced specifying the new shard layout which would be picked up by the resharding logic to split the shard. + +### Required state changes + +* For the duration of the resharding the node will need to maintain a snapshot of the flat state and related columns. As the main database and the snapshot diverge this will cause some extent of storage overhead. +* For the duration of the epoch before the new shard layout takes effect, the node will need to maintain the state and flat state of shards in the old and new layout at the same time. The State and FlatState columns will grow up to approx 2x the size. The processing overhead should be minimal as the chunks will still be executed only on the parent shards. There will be increased load on the database while applying changes to both the parent and the children shards. +* The total storage overhead is estimated to be on the order of 100GB for mainnet RPC nodes and 2TB for mainnet archival nodes. For testnet the overhead is expected to be much smaller. + +### Resharding flow + +* The new shard layout will be agreed on offline by the protocol team and hardcoded in the reference implementation. + * The first resharding will be scheduled soon after this NEP is merged. The new shard layout boundary accounts will be: ```["aurora", "aurora-0", "kkuuue2akv_1630967379.near", "tge-lockup.sweat"]```. + * Subsequent reshardings will be scheduled as needed, without further NEPs, unless significant changes are introduced. +* In epoch T, past the protocol version upgrade date, nodes will vote to switch to the new protocol version. The new protocol version will contain the new shard layout. +* In epoch T, in the last block of the epoch, the EpochConfig for epoch T+2 will be set. The EpochConfig for epoch T+2 will have the new shard layout. +* In epoch T + 1, all nodes will perform the state split. The child shards will be kept up to date with the blockchain up until the epoch end first via catchup, and later as part of block postprocessing state application. +* In epoch T + 2, the chain will switch to the new shard layout. + +## Reference Implementation + +The implementation heavily re-uses the implementation from [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md). Below are listed the major differences and additions. + +### Code pointers to the proposed implementation + +* [new shard layout](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/core/primitives/src/shard_layout.rs#L161) +* [the main logic for splitting states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/resharding.rs#L280) +* [the main logic for applying chunks to split states](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/update_shard.rs#L315) +* [the main logic for garbage collecting state from parent shard](https://github.com/near/nearcore/blob/c9836ab5b05c229da933d451fe8198d781f40509/chain/chain/src/store.rs#L2335) + +### Flat Storage + +The old implementation of resharding relied on iterating over the full trie state of the parent shard in order to build the state for the children shards. This implementation was suitable at the time but since then the state has grown considerably and this implementation is now too slow to fit within a single epoch. The new implementation relies on iterating through the flat storage in order to build the children shards quicker. Based on benchmarks, splitting the largest shard by using flat storage can take around 15 min without throttling and around 3 hours with throttling to maintain the block production rate. + +The new implementation will also propagate the flat storage for the children shards and keep it up to date with the chain until the switch to the new shard layout in the next epoch. The old implementation didn't handle this case because the flat storage didn't exist back then. + +In order to ensure consistent view of the flat storage while splitting the state the node will maintain a snapshot of the flat state and related columns as of the last block of the epoch prior to resharding. The existing implementation of flat state snapshots used in State Sync will be used for this purpose. + +### Handling receipts, gas burnt and balance burnt + +When resharding, extra care should be taken when handling receipts in order to ensure that no receipts are lost or duplicated. The gas burnt and balance burnt also need to be correctly handled. The old resharding implementation for handling receipts, gas burnt and balance burnt relied on the fact in the first resharding there was only a single parent shard to begin with. The new implementation will provide a more generic and robust way of reassigning the receipts to the child shards, gas burnt, and balance burnt, that works for arbitrary splitting of shards, regardless of the previous shard layout. + +### New shard layout + +The first release of the resharding v2 will contain a new shard layout where one of the existing shards will be split into two smaller shards. Furthermore additional reshardings can be scheduled in subsequent releases without additional NEPs unless the need for it arises. A new shard layout can be determined and will be scheduled and executed with the next protocol upgrade. Resharding will typically happen by splitting one of the existing shards into two smaller shards. The new shard layout will be created by adding a new boundary account that will be determined by analysing the storage and gas usage metrics within the shard and selecting a point that will divide the shard roughly in half in accordance to the mentioned metrics. Other metrics can also be used based on requirements. + +### Removal of Fixed shards + +Fixed shards was a feature of the protocol that allowed for assigning specific accounts and all of their recursive sub accounts to a predetermined shard. This feature was only used for testing and was never used in production. Fixed shards feature unfortunately breaks the contiguity of shards and is not compatible with the new resharding flow. A sub account of a fixed shard account can fall in the middle of account range that belongs to a different shard. This property of fixed shards made it particularly hard to reason about and implement efficient resharding. + +For example in a shard layout with boundary accounts [`b`, `d`] the account space is cleanly divided into three shards, each spanning a contiguous range and account ids: + +* 0 - `:b` +* 1 - `b:d` +* 2 - `d:` + +Now if we add a fixed shard `f` to the same shard layout, then any we'll have 4 shards but neither is contiguous. Accounts such as `aaa.f`, `ccc.f`, `eee.f` that would otherwise belong to shards 0, 1 and 2 respectively are now all assigned to the fixed shard and create holes in the shard account ranges. + +It's also worth noting that there is no benefit to having accounts colocated in the same shard. Any transaction or receipt is treated the same way regardless of crossing shard boundary. + +This was implemented ahead of this NEP and the fixed shards feature was **removed**. + +### Garbage collection + +In epoch T+2 once resharding is completed, we can delete the trie state and the flat state related to the parent shard. In practice, this is handled as part of the garbage collection code. While garbage collecting the last block of epoch T+1, we go ahead and clear all the data associated with the parent shard from the trie cache, flat storage, and RocksDB state associated with trie state and flat storage. + +### Transaction pool + +The transaction pool is sharded i.e. it groups transactions by the shard where each transaction should be converted to a receipt. The transaction pool was previously sharded by the ShardId. Unfortunately ShardId is insufficient to correctly identify a shard across a resharding event as ShardIds change domain. The transaction pool was migrated to group transactions by ShardUId instead, and a transaction pool resharding was implemented to reassign transaction from parent shard to children shards right before the new shard layout takes effect. The ShardUId contains the version of the shard layout which allows differentiating between shards in different shard layouts. + +This was implemented ahead of this NEP and the transaction pool is now fully **migrated** to ShardUId. + +## Alternatives + +### Why is this design the best in the space of possible designs? + +This design is simple, robust, safe, and meets all requirements. + +### What other designs have been considered and what is the rationale for not choosing them? + +#### Alternative implementations + +* Splitting the trie by iterating over the boundaries between children shards for each trie record type. This implementation has the potential to be faster but it is more complex and it would take longer to implement. We opted in for the much simpler one using flat storage given it is already quite performant. +* Changing the trie structure to have the account id first and type of record later. This change would allow for much faster resharding by only iterating over the nodes on the boundary. This approach has two major drawbacks without providing too many benefits over the previous approach of splitting by each trie record type. + 1) It would require a massive migration of trie. + 2) We would need to maintain the old and the new trie structure forever. +* Changing the storage structure by having the storage key to have the format of `account_id.node_hash`. This structure would make it much easier to split the trie on storage level because the children shards are simple sub-ranges of the parent shard. Unfortunately we found that the migration would not be feasible. +* Changing the storage structure by having the key format as only node_hash and dropping the ShardUId prefix. This is a feasible approach but it adds complexity to the garbage collection and data deletion, specially when nodes would start tracking only one shard. We opted in for the much simpler one by using the existing scheme of prefixing storage entries by shard uid. + +#### Other considerations + +* Dynamic Resharding - we have decided to not implement the full dynamic resharding at this time. Instead we hardcode the shard layout and schedule it manually. The reasons are as follows: + * We prefer incremental process of introducing resharding to make sure that it is robust and reliable, as well as give the community the time to adjust. + * Each resharding increases the potential total load on the system. We don't want to allow it to grow until full sharding is in place and we can handle that increase. +* Extended shard layout adjustments - we have decided to only implement shard splitting and not implement any other operations. The reasons are as follows: + * In this iteration we only want to perform splitting. + * The extended adjustments are currently not justified. Both merging and boundary moving may be useful in the future when the traffic patterns change and some shard become underutilized. In the nearest future we only predict needing to reduce the size of the heaviest shards. + +### What is the impact of not doing this? + +We need resharding in order to scale up the system. Without resharding eventually shards would grow so big (in either storage or cpu usage) that a single node would not be able to handle it. Additionally, this clears up the path to implement in-memory tries as we need to store the whole trie structure in limited RAM. In the future smaller shard size would lead to faster syncing of shard data when nodes start tracking just one shard. + +## Integration with State Sync + +There are two known issues in the integration of resharding and state sync: + +* When syncing the state for the first epoch where the new shard layout is used. In this case the node would need to apply the last block of the previous epoch. It cannot be done on the children shard as on chain the block was applied on the parent shards and the trie related gas costs would be different. +* When generating proofs for incoming receipts. The proof for each of the children shards contains only the receipts of the shard but it's generated on the parent shard layout and so may not be verified. + +In this NEP we propose that resharding should be rolled out first, before any real dependency on state sync is added. We can then safely roll out the resharding logic and solve the above mentioned issues separately. We believe at least some of the issues can be mitigated by the implementation of new pre-state root and chunk execution design. + +## Integration with Stateless Validation + +The Stateless Validation requires that chunk producers provide proof of correctness of the transition function from one state root to another. That proof for the first block after the new shard layout takes place will need to prove that the entire state split was correct as well as the state transition. + +In this NEP we propose that resharding should be rolled out first, before stateless validation. We can then safely roll out the resharding logic and solve the above mentioned issues separately. This issue was discussed with the stateless validation experts and we are cautiously optimistic that the integration will be possible. The most concerning part is the proof size and we believe that it should be small enough thanks to the resharding touching relatively small number of trie nodes - on the order of the depth of the trie. + +## Future fast-followups + +### Resharding should work even when validators stop tracking all shards + +As mentioned above under 'Integration with State Sync' section, initial release of resharding v2 will happen before the full implementation of state sync and we plan to tackle the integration between resharding and state sync after the next shard split (Won't need a separate NEP as the integration does not require protocol change.) + +### Resharding should work after stateless validation is enabled + +As mentioned above under 'Integration with Stateless Validation' section, the initial release of resharding v2 will happen before the full implementation of stateless validation and we plan to tackle the integration between resharding and stateless validation after the next shard split (May need a separate NEP depending on implementation detail.) + +## Future possibilities + +### Further reshardings + +This NEP introduces both an implementation of resharding and an actual resharding to be done in the production networks. Further reshardings can also be performed in the future by adding a new shard layout and setting the shard layout for the desired protocol version in the `AllEpochConfig`. + +### Dynamic resharding + +As noted above, dynamic resharding is out of scope for this NEP and should be implemented in the future. Dynamic resharding includes the following but not limited to: + +* Automatic determination of split boundary based on parameters like traffic, gas usage, state size, etc. +* Automatic scheduling of resharding events + +### Extended shard layout adjustments + +In this NEP we only propose supporting splitting shards. This operation should be more than sufficient for the near future but eventually we may want to add support for more sophisticated adjustments such as: + +* Merging shards together +* Moving the boundary account between two shards + +### Localization of resharding event to specific shard + +As of today, at the RocksDB storage layer, we have the ShardUId, i.e. the ShardId along with the ShardVersion, as a prefix in the key of trie state and flat state. During a resharding event, we increment the ShardVersion by one, and effectively remap all the current parent shards to new child shards. This implies we can't use the same underlying key value pairs for store and instead would need to duplicate the values with the new ShardUId prefix, even if a shard is unaffected and not split. + +In the future, we would like to potentially change the schema in a way such that only the shard that is splitting is impacted by a resharding event, so as to avoid additonal work done by nodes tracking other shards. + +### Other useful features + +* Removal of shard uids and introducing globally unique shard ids +* Account colocation for low latency across account call - In case we start considering synchronous execution environment, colocating associated accounts (e.g. cross contract call between them) in the same shard can increase the efficiency +* Shard purchase/reservation - When someone wants to secure entirety of limitation on a single shard (e.g. state size limit), they can 'purchase/reserve' a shard so it can be dedicated for them (similar to how Aurora is set up) + +## Consequences + +### Positive + +* Workload across shards will be more evenly distributed. +* Required space to maintain state (either in memory or in persistent disk) will be smaller. This is useful for in-memory tries. +* State sync overhead will be smaller with smaller state size. + +### Neutral + +* Number of shards would increase. +* Underlying trie structure and data structure are not going to change. +* Resharding will create dependency on flat state snapshots. +* The resharding process, as of now, is not fully automated. Analyzing shard data, determining the split boundary, and triggering an actual shard split all need to be manually curated and tracked. + +### Negative + +* During resharding, a node is expected to require more resources as it will first need to copy state data from the parent shard to the child shard, and then will have to apply trie and flat state changes twice, once for the parent shard and once for the child shards. +* Increased potential for apps and tools to break without proper shard layout change handling. + +### Backwards Compatibility + +Any light clients, tooling or frameworks external to nearcore that have the current shard layout or the current number of shards hardcoded may break and will need to be adjusted in advance. The recommended way for fixing it is querying an RPC node for the shard layout of the relevant epoch and using that information in place of the previously hardcoded shard layout or number of shards. The shard layout can be queried by using the `EXPERIMENTAL_protocol_config` rpc endpoint and reading the `shard_layout` field from the result. A dedicated endpoint may be added in the future as well. + +Within nearcore we do not expect anything to break with this change. Yet, shard splitting can introduce additional complexity on replayability. For instance, as target shard of a receipt and belonging shard of an account can change with shard splitting, shard splitting must be replayed along with transactions at the exact epoch boundary. + +## Changelog + +[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] + +### 1.0.0 - Initial Version + +> Placeholder for the context about when and who approved this NEP version. + +#### Benefits + +> List of benefits filled by the Subject Matter Experts while reviewing this version: + +* Benefit 1 +* Benefit 2 + +#### Concerns + +> Template for Subject Matter Experts review for this version: +> Status: New | Ongoing | Resolved + +| # | Concern | Resolution | Status | +| --: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).