contracts/results/ and contracts/results/logs returns nullable and empty fields if requested too early

[quiet-node]

Problem

The Relay is encountering a significant number of 500 Internal Server Errors, with approximately 60,000 related log entries due to this issue.

The endpoints contracts/results/ and contracts/results/logs are intermittently returning nullable and empty fields under certain conditions, potentially when requests are made too quickly or prematurely.

The following issues have been identified:
• block_number, transaction_index, and log index fields are returned as null.
• block_hash is returned as an empty byte (0x).

This issue is difficult to reproduce as it appears to occur only at specific timestamps when requests might be made prematurely. Additionally, it is observed exclusively on mainnet, with no corresponding results in testnet logs.

Attached below are two examples from the Relay logs on Mainnet

• contracts:results:logs-logs-20250109-122423.json
• contracts:results-logs-20250109-122541.json

Let’s examine the first log file for the contracts/results/logs endpoint. In the log file, if you search for Contract result log contains undefined transaction_index, block_number, index, or block_hash is an empty hex, you’ll find that:
• The block_number is returned as null.
• The block_hash is set to an empty hash (0x).
In this case, only block_number and block_hash are falsy. transaction_index and index have also been reported as falsy in other cases.

This log entry was logged immediately after the log records were retrieved from MN. Above this log entry, you can see the exact contracts/results/logs?... endpoint the Relay sent to MN:
contracts/results/logs?timestamp=gte:1736446204.069548875&timestamp=lte:1736446209.764348710&limit=100&order=asc.

If you query this endpoint now on mainnet (which is obviously long after the specific timestamp) https://mainnet.mirrornode.hedera.com/api/v1/contracts/results/logs?timestamp=gte:1736446204.069548875&timestamp=lte:1736446209.764348710&limit=100&order=asc , it returns complete log information with no null or empty fields. This observation indicates that, at the time the Relay made that specific request, the log fields were still populated with nullable values.

The Relay also employs a retry mechanism for such cases. When nullable or empty fields are detected, the mechanism waits for 500ms before retrying. However, if you search for Log entry is missing required fields: block_number, index, or block_hash is an empty hex (0x) in the log file, you’ll see that even after the 500ms retry, the log records still contain nullable or empty fields. This ultimately causes the Relay to reject the request with a 500 error.

Alternatives

No response

[quiet-node]

Update

The retry mechanism described in the ticket has been updated to a polling mechanism that continuously polls the Mirror Node (MN) until the records are fully mature. To prevent indefinite looping, the process is capped at 10 retries with a 2000ms wait between each, for a maximum duration of 20,000ms.

The provided log file for an eth_getTransactionReceipt request includes the entry: Contract result contains nullable transaction_index or block_number, or block_hash is an empty hex (0x). This demonstrates the polling mechanism is working as intended, resubmitting requests every 2000ms. Each log entry shows the contract result details from MN, in which you can see block_number is null, and block_hash is an empty hex (0x).

In this case (and others observed on mainnet), the polling completed after 20,000ms, but the record still contained null and empty fields. However, manually querying the same URL later (e.g., https://mainnet.mirrornode.com/api/v1/contracts/results/0xa4b8d044524b2d34eabbc55bb4a8a9eb97a93fb30f89a1c9b96c4b0e4d36494d) returned a fully complete result, suggesting the MN data was not synchronized during the Relay’s polling attempts.

The main point I wanna bring up is that even after a long period of 20,000ms (20 seconds), the records returned by the MN remain incomplete in many cases, highlighting a significant delay in data synchronization that affects reliability and consistency.

Log file: downloaded-logs-20250110-134831.json