Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adds draft design doc for new block explorer REST API endpoints #3342

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

konstantinabl
Copy link
Collaborator

@konstantinabl konstantinabl commented Dec 18, 2024

Description:

Creates a design document for the new block explorer REST API endpoints

Related issue(s):

Fixes #3340

Notes for reviewer:

@konstantinabl konstantinabl linked an issue Dec 18, 2024 that may be closed by this pull request
@konstantinabl konstantinabl self-assigned this Dec 18, 2024
@konstantinabl konstantinabl added the design Design, pilot and prototyping exploration work label Dec 18, 2024
@konstantinabl konstantinabl added this to the 0.63.0 milestone Dec 18, 2024
Signed-off-by: Konstantina Blazhukova <[email protected]>
@konstantinabl konstantinabl force-pushed the 3340-create-design-doc-to-support-rest-api branch from 7088d1c to b840444 Compare December 18, 2024 17:40
Copy link

github-actions bot commented Dec 18, 2024

Test Results

 23 files   -   3  352 suites   - 55   47m 8s ⏱️ - 23m 38s
616 tests ±  0  602 ✅ + 11  4 💤 ±0  10 ❌  - 11 
891 runs   - 220  877 ✅  - 197  4 💤  - 4  10 ❌  - 19 

For more details on these failures, see this check.

Results for commit 17b8254. ± Comparison against base commit 7b12e74.

This pull request removes 4 and adds 4 tests. Note that renamed tests count towards both.
"after all" hook in "RPC Server Acceptance Tests" ‑ RPC Server Acceptance Tests "after all" hook in "RPC Server Acceptance Tests"
"before all" hook for "emits an approval event" ‑ RPC Server Acceptance Tests Acceptance tests @erc20 Acceptance Tests HTS token should behave like erc20 transfer from when the token owner is not the zero address when the recipient is not the zero address when the spender has enough allowance "before all" hook for "emits an approval event"
"before all" hook for "reverts" ‑ RPC Server Acceptance Tests Acceptance tests @erc20 Acceptance Tests HTS token should behave like erc20 transfer from when the token owner is not the zero address when the recipient is not the zero address when the spender does not have enough allowance when the token owner has enough balance "before all" hook for "reverts"
"before each" hook for "reverts" ‑ RPC Server Acceptance Tests Acceptance tests @erc20 Acceptance Tests HTS token should behave like erc20 transfer from when the token owner is not the zero address when the recipient is not the zero address when the spender does not have enough allowance "before each" hook for "reverts"
"before all" hook for "should execute "eth_getCode" for hts token" ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-2 RPC Server Acceptance Tests eth_getCode "before all" hook for "should execute "eth_getCode" for hts token"
"before each" hook for "@release should execute "eth_estimateGas" for contract call" ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-2 RPC Server Acceptance Tests "before each" hook for "@release should execute "eth_estimateGas" for contract call"
"before each" hook for "reverts" ‑ RPC Server Acceptance Tests Acceptance tests @erc20 Acceptance Tests ERC20 Contract should behave like erc20 transfer from when the token owner is not the zero address when the recipient is the zero address "before each" hook for "reverts"
"before each" hook for "should execute "eth_getStorageAt" request to get current state changes" ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-2 RPC Server Acceptance Tests "before each" hook for "should execute "eth_getStorageAt" request to get current state changes"

♻️ This comment has been updated with latest results.

@konstantinabl konstantinabl marked this pull request as ready for review December 18, 2024 22:41
@konstantinabl konstantinabl requested review from a team as code owners December 18, 2024 22:41
Copy link
Collaborator

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good start.
I think we have to think through the way in which this would deploy and scale a bit more.

docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
# Design Document: ERC20/ERC721/ERC1155 Token Transfer Events API, Tokens Owned by an Address

## Overview
- This document outlines the design for implementing new endpoints allowing exposure of more EVM centric data, similar to Etherscan's and BlockScout's APIs.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- This document outlines the design for implementing new endpoints allowing exposure of more EVM centric data, similar to Etherscan's and BlockScout's APIs.
- This document outlines the design for implementing new endpoints allowing exposure of more EVM centric data that support Block Explorer (e.g. Etherscan, BlockScout) API needs.

docs/design/additional-endpoints.md Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
### Overview

Introduce two new endpoints:
- `getTokenTransfers` - to fetch token transfer events
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: what are the params?
You should move the params explanation lower to here.
Also note what's mandatory

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So i think in the overview its enough to state the format of the endpoint and the detailed explanation down where it is, do you think thats fine or should I move it all to this section?
Also I have chosen this classic REST format, which differentiates from the way etherscan and blockscout has implemented their endpoints, entirely with query parameters, not sure if thats gonna affect the user. Do they expect the exactly same format
In etherscan the ednpoints look like this:

GET /api
   ?module=account
   &action=tokentx
   &contractaddress=0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2
   &address=0x4e83362442b8d1bec281594cea3050c8eb01311c
   &page=1
   &offset=100
   &startblock=0
   &endblock=27025780
   &sort=asc
   &apikey=YourApiKeyToken

- Test address matching
- Test token standard detection

2. **Acceptance Tests**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acceptance tests should be in teh form of E2E features a User would go through with a focus on the input and the outputs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edited them : )


### Dependencies

- ERC registry
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How so, please expand

Copy link
Member

@quiet-node quiet-node Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on this. Please expand on how ERC Registry is involved.

Copy link
Collaborator

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good start waiting on a bit more structure

@quiet-node quiet-node modified the milestones: 0.63.0, 0.64.0 Dec 26, 2024
Copy link
Member

@quiet-node quiet-node left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great process! Many questions and suggestions

docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved

### Dependencies

- ERC registry
Copy link
Member

@quiet-node quiet-node Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on this. Please expand on how ERC Registry is involved.

@konstantinabl konstantinabl force-pushed the 3340-create-design-doc-to-support-rest-api branch from bc9bfcc to 441d39b Compare January 7, 2025 16:22
Signed-off-by: Konstantina Blazhukova <[email protected]>
@konstantinabl konstantinabl force-pushed the 3340-create-design-doc-to-support-rest-api branch from 441d39b to f81f789 Compare January 7, 2025 16:35
Signed-off-by: Konstantina Blazhukova <[email protected]>
@konstantinabl konstantinabl requested review from quiet-node, Nana-EC and a team January 7, 2025 21:07
@Nana-EC Nana-EC requested a review from acuarica January 8, 2025 05:22
Copy link
Collaborator

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start, looking forward to discussing more


#### 4. CacheService
Redis-based caching service:
- Caches frequent queries
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call out the cached objects. I'd like to ensure that we only cache what is needed e.g. in some cases the relay caches the full account or token object but we really only need a few fields from the mirror node response.
In this case do we need to cache a full log or can we cache only a subset of them?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we dont need to cache the full object, so I changed and specified what we will need


#### 4. CacheService
Redis-based caching service:
- Caches frequent queries
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please suggest an TTL for these caches components.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


### Performance Considerations

1. Possibly restrict the number of logs returned by the MN to a maximum of 10000
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently have a max on the logs endpoint we can likely match that

Copy link
Collaborator Author

@konstantinabl konstantinabl Jan 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, investigating this. Seems like its 100 so added this as a value @Nana-EC

packages/
rest-api/
└── src/
├── config/ # Configuration for REST API
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a section for this, could you add one that calls out which config fields you plan to add and what their defaults might be?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added one :) Its a starting point tho, maybe something comes up later during implementation

### Package Structure
```
packages/
rest-api/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, another approach would be rest-server package with the controllers under there and the services under the relay package or this new rest-api package.
In this way the service implementation could be change in the future a transparent way or the underlying server logic - both packages decoupled from each other.
Let's chat about it, the above is food for thought and might hint towards the final form we want to get to in a tentative refactor of the repo

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add it as a point to discuss in the meeting


### Dependencies

* ERC registry - As explained in the _Flow_ section, the TokenService will need to fetch the ERC registry to get the token standard (ERC20/721/ 1155) contracts and filter the response by only those matching the standard, wanted in the request. E.g if the request is for ERC20, the TokenService will need to fetch the ERC registry to get the ERC20 contracts and filter the response from the MN by only those matching the standard.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should note that initially we'll use the static outputs from the smart contract repo. We should be able to import that package soon but int eh early stage we can copy over the files as needed

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Signed-off-by: Konstantina Blazhukova <[email protected]>
Signed-off-by: Konstantina Blazhukova <[email protected]>
@konstantinabl konstantinabl requested review from Nana-EC and a team January 10, 2025 16:14
Copy link
Collaborator

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestions @konstantinabl.
To confirm after some thought we should indeed position this from the get go as an independently deployable instance.
Should be something like http-explorer-server containing the specific etherscan focused controllers and server logic.
Initially we can colocate the api logic including services in the relay package then revisit when we pick up the repo refactor


| Endpoint | Description |
|----------|-------------|
| `GET /api?module=account&action=tokentx&address={address}&contractaddress={contractaddress}&startblock={startblock}&endblock={endblock}&page={page}&offset={offset}&sort={asc\|desc}` | Get ERC20 token transfer events |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should make the etherscan nature explicit
Somethings like what snowtrace does https://testnet.snowtrace.io/documentation/api-swagger

Suggested change
| `GET /api?module=account&action=tokentx&address={address}&contractaddress={contractaddress}&startblock={startblock}&endblock={endblock}&page={page}&offset={offset}&sort={asc\|desc}` | Get ERC20 token transfer events |
| `GET /api/v1/etherscan?module=account&action=tokentx&address={address}&contractaddress={contractaddress}&startblock={startblock}&endblock={endblock}&page={page}&offset={offset}&sort={asc\|desc}` | Get ERC20 token transfer events |

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nana-EC I get what you mean, but I think their design is like this because they have different networks available. And will we have something else than etherscan, imo it may be confusing to have it like this, since we may also want to redesign the endpoint format in the future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why having a unique oath would be preferred. I that way the etherscan support approach can be left alone and a future version that we designed with better REST principles could be added with my conflict

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree with Nana.


Currently MN doesn't support EVM centric queries, like token transfer events for specific standards like ERC20, ERC721, ERC1155 or balance of an address for a specific token.

## Goals
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please take a look at
https://api-testnet.snowtrace.io/api?module=contract&action=getsourcecode&address=0x5425890298aed601595a70AB815c96711a31Bc65, a user may use this when looking for ABI
I think this might be another goal to capture. P2 in nature for this story though.
This is very interesting as it may require calling the verification service.

@acuarica do you see any other option to retrieve such information?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the verification service is the source of truth when getting the ABI.

Copy link
Member

@quiet-node quiet-node left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Love how meticulous this design doc is! Left comments and quetions around

docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
docs/design/additional-endpoints.md Outdated Show resolved Hide resolved
@konstantinabl konstantinabl force-pushed the 3340-create-design-doc-to-support-rest-api branch from 3c2b0ca to 75de0a2 Compare January 13, 2025 18:00
Signed-off-by: Konstantina Blazhukova <[email protected]>
@konstantinabl konstantinabl force-pushed the 3340-create-design-doc-to-support-rest-api branch from 75de0a2 to 17b8254 Compare January 13, 2025 18:00
Copy link
Member

@quiet-node quiet-node left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the great work!

@quiet-node quiet-node added the enhancement New feature or request label Jan 13, 2025
Copy link

@Ferparishuertas Ferparishuertas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job. I left some comments


### Main Implementation Components

#### 1. TokenController

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it makes sense to call it Etherscan token Controller?
an alternative is to create it inside folder called etherscan. There could be other proxy apis in the future

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I name it token controller, since I plan it will be responsible for all the endpoints from the token module but i may put it into etherscan folder

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine we implement etherscan and snowflake. Same controlles for both, or separated ( folder/ naming)? what will the approach?
Personally Iam used to split to find easierm group things. Whats the usual approach here?

### Performance Considerations

1. Possibly restrict the number of logs returned by the MN to a maximum of 100 (as is the MN limit)
2. Possibly restrict the block range to a maximum of 10000 blocks

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the restrictions at etherscan side?

they say "This API endpoint returns a maximum of 10000 records only." which is lees, because we are talking a about transactions (records)

I suggest to have the same limits if possible

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure what you mean. In etherscan 10000 records limit is for this endpoint, which we are not going to implement. The limit i mentioned here is for this one,
https://docs.blockscout.com/devs/apis/rpc/account#get-token-transfer-events-by-address, where they have limit for 10000 blocks

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you see the limit in code? I am not able to see it on the doc. Only
Up to a maximum of 10,000 token transfer events. Also available through the GraphQL token_transfers query.

no really seeing 10k blocks limit

docs/design/additional-endpoints.md Show resolved Hide resolved

| Endpoint | Description |
|----------|-------------|
| `GET /api?module=account&action=tokentx&address={address}&contractaddress={contractaddress}&startblock={startblock}&endblock={endblock}&page={page}&offset={offset}&sort={asc\|desc}` | Get ERC20 token transfer events |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree with Nana.

docs/design/additional-endpoints.md Show resolved Hide resolved
@quiet-node quiet-node changed the title feat: Adds draft design doc for new endpoints feat: Adds draft design doc for new block explorer REST API endpoints Jan 14, 2025
Copy link

codecov bot commented Jan 14, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.71%. Comparing base (13d09e9) to head (17b8254).
Report is 13 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (13d09e9) and HEAD (17b8254). Click for more details.

HEAD has 21 uploads less than BASE
Flag BASE (13d09e9) HEAD (17b8254)
config-service 2 0
relay 2 0
32 17
server 1 0
ws-server 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #3342       +/-   ##
===========================================
- Coverage   85.30%   62.71%   -22.60%     
===========================================
  Files          69       65        -4     
  Lines        4688     4535      -153     
  Branches     1050     1040       -10     
===========================================
- Hits         3999     2844     -1155     
- Misses        374     1194      +820     
- Partials      315      497      +182     
Flag Coverage Δ
config-service ?
relay ?
server ?
ws-server ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/relay/src/lib/eth.ts 50.16% <ø> (-36.53%) ⬇️

... and 51 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design Design, pilot and prototyping exploration work enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create design doc to support REST API
5 participants