Forcing miners to retain ephemeral data with probabilistic checks

[adambor]

PMTs (proof merkle trees) make up ephemeral data allowing anyone to construct a proof that their transaction was included in a block. For constructions such as lightning we outlined the need for Watchtowers - specialized network participants who retain few days/weeks of ephemeral data and provide it to clients on request. This has to be in place to make sure that a lightning node which was offline is able to handle potential force-close transactions.

However I feel like it would be nicer if we can get rid of such Watchtowers in the network and rely purely on miners for storing this ephemeral data for some pre-determined time T (could be a week or a month). The trick to do this is to somehow force the miners to retain this data. For this purpose we will use random probabilistic checks for ephemeral data in blocks:

• the miner first mines a block (solves a PoW challenge).
• the blockhash of this block is used as a seed to random number generator that generates N numbers (each number corresponds to one random transaction included in a block in range between now and now-T).
• miner now needs to put txIds of those randomly selected transaction along with their merkle proofs into the block in the clear.

To verify a block, one reconstructs the N random numbers from blockhash and checks if correct merkle proofs were provided.

Some calculations for such a solution:

Miner storage requirements

Worst case ephemeral data size: 24.5MB
Blocks per day: 144
Data storage required to retain ephemeral data for a week: ~25GB
Data storage required to retain ephemeral data for a month: ~100GB

Block size increase due to probabilistic checks

Let's consider setting number of probablistic checks N=256.
Single proof size:

• Normal tx: 20 + (28*log2(txns_in_block)) = 20 + 28*20 = 580
• PTLC tx: 56 + (28*log2(txns_in_block)) = 56 + 28*20 = 616

Total number of proof data in block (considering all PTLCs): 157696 bytes = ~154kB

So overall, this puts more storage requirements on miners, but this data they store is still ephemeral in a sense that they can remove it as soon as week/month passes, so this requirement is static O(1) compared to blockchain, which still grows O(n). For comparison the Abraxas blockchain grows by 130GB per year, considering full blocks.

Using ~154kB more data in a block is also I think very reasonable, considering the block size is already 2.5-3 MB.

[cryptoquick]

Instead of including the txid, would it make sense to hash the random number + the tx preimage? Then, to verify, the old txs are needed by other nodes, which shouldn't be too many. The old txs won't need to be stored by nodes, only needed to verify the block.

[adambor]

Instead of including the txid, would it make sense to hash the random number + the tx preimage?

Why do you think this is better? The randomly generated number uniquely identifies a single merkle leaf inside the merkle tree, miner has to provide a txId stored at that exact position of the merkle tree along with a merkle proof. No ephemeral data has to be stored by the full nodes (as miner provides that piece of ephemeral data - txId in the block directly), only mining nodes need to store this ephemeral data (because they need it to pass the block's challenge).

Ephemeral data is actually just txIds (not full txns), so there won't be any tx preimage stored by anyone (except the party who sent it).

Then, to verify, the old txs are needed by other nodes, which shouldn't be too many.

This is actually not correct, it really is too many! A single transaction is around 1kB, with 1M of them in a block, this is already 1GB, and now imagine you have to store a week of that data - that would be 1TB of data that all full nodes need to store!

Also, how would you go about IBC (initial block download)? If that ephemeral data from 2 years ago is already long gone then you don't have old txns to check the block.

The old txs won't need to be stored by nodes, only needed to verify the block.

Well, but how does a node verify a block if you say it doesn't need to store old txns? Or you just mean that nodes only have to keep last week/month of those txns?

[cryptoquick]

I think all your concerns are correct, that approach is no good.

I have a feeling that incentivized DA solves both of these problems. If we can come up with a proper incentives model, we solve for miners releasing the data, and also for storing it.