Block Stream Bucket Uploader

[netopyr]

To make block streams publicly available as soon as possible, we want to upload them to buckets initially. This requires the support of a new BlockStreamWriterMode. In addition to sending blocks via gRPC and writing them to disk, we also must be able to send them to buckets.

Requirements

• We need a third BlockStreamWriterMode, which sends the blocks to one or more buckets.
• The buckets must be configurable.
• Both the writer mode and the addresses of the buckets must be configurable during runtime (i.e., the node must not be restarted to take the changes into effect).
• If an error occurs, we want to keep a copy of the block on the local disk. It probably makes sense to write the block to a local file initially and upload it once done (instead of writing directly to the bucket).
• We want to upload only one instance of a block. All consensus nodes will try to upload the same block to a bucket, and therefore only the first one can succeed.
• If a consensus node notices the block is already available, it should check if the blocks are the same (via MD5 hash). If they are different, the block should be kept on the local disk, and an alert must be triggered.
• If uploading a block fails, it should be kept on the local disk, and an alert needs to be triggered.
• If the upload was successful, the local copy can be removed after a few hours.

Questions

• How should this be configured? Is uploading file 0.0.121 sufficient?
• What is the best way to trigger an alert? An error message in the log?

Tasks

Preview Give feedback

1. Find a library for bucket access with minimal dependencies #16769
   +0
   
2. Design of Block Stream Bucket Uploader #16770
   +0
   
3. Core Configuration for Block Stream Bucket Uploader #16972
   +0
   
4. Cloud Storage Interface Implementation #16973
   +0
   
5. Block Upload Manager Implementation #16974
   +0
   
6. Block Retention Manager Implementation #16975
   +0
   
7. Block Reupload Manager Implementation #16976
   +0
   
8. Block Stream Bucket Uploader Metrics #16977
   Estimated +1
   
9. Block Stream Bucket Uploader Integration Testing #16978
   +0
10. Block Stream Bucket Uploader End-to-End Testing with Cloud Providers #16979
    +0
11. Update Block Stream Bucket Uploader Design Documentation #16980
    Estimated +1