Blob p2p distribution using Reactor API

[fridrik01]

issue: #2665
depends on cometbft blob PR: berachain/cometbft#26

Context
Currently, blobs are stored as as CometBFT transaction so a blob will never be cleaned up and will take up wasted space since they have a finite lifespan.

This PR addresses this by implementing a complete p2p blob distribution system using the cometbft Reactor API which fetches missing blobs from other peers. It introduces a BlobConsensusEnableHeight configuration parameter that defines when the chain transitions from storing blobs as transactions to using P2P distribution which requires a hard fork.

Components

• BlobReactor (in da/blobreactor/) that handles p2p blob requests/responses with timeout handling and makes sure received blobs pass verification (from potential byzantine peers).
• BlobFetcher background service (in beacon/blockchain/) that manages a persistent filesystem based retry queue. When a block arrives with missing blobs, the request is queued and the background worker attempts to fetch from peers at regular interval with a max retry count. The queue is crash-safe with atomic writes, includes request deduplication, and automatically cleans up requests that exceed retry limits or fall outside the DA availability window.

Consensus flow (after blob enable height reached)

• PrepareProposal: Blobs included in the block are now returned in PrepareProposalResponse in a separate Blob field instead of as CometBFT transactions. This prevents blobs from being persisted in CometBFTs block store.
• ProcessProposal: Blobs are now included in the ProcessProposalRequest in a separate Blob field and are validated synchronously. Proposals are rejected if blobs are missing or invalid. Validated blobs are cached with the state for FinalizeBlock.
• FinalizeBlock: If node is in consensus mode, then blobs come from the ProcessProposal cache and are processed immediately. Otherwise, if the node is syncing (catching up) and comes upon a block that should have blobs the node will make async fetch request to BlobFetcher. FinalizeBlock returns immediately without blocking, allowing the chain to continue syncing while blobs are fetched in the background.

TODO:

• Add peer scoring to avoid bad peers (implemented in see Blob p2p distribution using Reactor API #2938)
• Evaluate if throttling is needed (implemented in see Blob p2p distribution using Reactor API #2938)
• Update RPC endpoint to return "blobs pending" if user requests blobs for a block that are still queued in blobreactor
• Update cometbft dependency
• Evaluate if we need to split blobs into parts like cometbft does
• Evaluate if we need to enable/disable Blobreactor when not needed

[codecov]

Codecov Report

❌ Patch coverage is 62.65173% with 400 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.24%. Comparing base (e293ced) to head (935e407).

Files with missing lines	Patch %	Lines
da/blobreactor/reactor.go	76.27%	74 Missing and 19 partials ⚠️
consensus/cometbft/service/finalize_block.go	13.55%	46 Missing and 5 partials ⚠️
beacon/blockchain/blob_queue.go	58.87%	31 Missing and 13 partials ⚠️
consensus/cometbft/service/encoding/encoding.go	18.91%	22 Missing and 8 partials ⚠️
beacon/blockchain/blob_fetcher.go	73.87%	26 Missing and 3 partials ⚠️
da/blobreactor/messages.go	64.00%	25 Missing and 2 partials ⚠️
beacon/blockchain/finalize_block.go	40.90%	21 Missing and 5 partials ⚠️
consensus/cometbft/service/process_proposal.go	8.33%	20 Missing and 2 partials ⚠️
consensus/cometbft/service/service.go	21.73%	17 Missing and 1 partial ⚠️
chain/spec.go	50.00%	7 Missing and 2 partials ⚠️
... and 13 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2938      +/-   ##
==========================================
+ Coverage   63.10%   63.24%   +0.13%     
==========================================
  Files         353      367      +14     
  Lines       17049    18014     +965     
==========================================
+ Hits        10759    11393     +634     
- Misses       5447     5713     +266     
- Partials      843      908      +65     

Files with missing lines	Coverage Δ
beacon/blockchain/process_proposal.go	69.02% <100.00%> (ø)
beacon/blockchain/service.go	86.84% <100.00%> (+2.46%)	⬆️
...on/blockchain/testhelpers/blob_processor_simple.go	100.00% <100.00%> (ø)
config/config.go	81.81% <100.00%> (+0.42%)	⬆️
config/spec/devnet.go	100.00% <100.00%> (ø)
config/spec/mainnet.go	100.00% <100.00%> (ø)
config/spec/testnet.go	100.00% <100.00%> (ø)
consensus/cometbft/service/blobreactor/config.go	100.00% <100.00%> (ø)
consensus/cometbft/service/cache/cache.go	84.21% <ø> (ø)
da/blobreactor/config.go	100.00% <100.00%> (ø)
... and 26 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[abi87]

I wonder if there should be a cap here. What happens at cap? A node gives up on syncing?

[fridrik01]

If cap is reached, the blob request is deleted from disk, meaning that blob won't exist on this node. The node always continues syncing regardless and other blob requests are processed independently. If/when this happens, we record a blob_fetcher_requests_expired_total{reason="max_retries"}) metric so operators can detect if this happens frequently. Alternatively we can keep this uncapped and instead do something like double retryInterval (exponential backoff) each time it fails. Eventually it will pass the DA window be cleaned up.

[abi87]

unlike Stop, this is not idempotent. Should it be?

[fridrik01]

Probably not, kept it simple since depositCatchupFetcher isn't idempotent either and is called in the same Service.Start() method (https://github.com/berachain/beacon-kit/blob/blobreactor/beacon/blockchain/service.go#L113-L116). Happy to add sync.Once to both if we think it's worth it.

[abi87]

a few questions but overall LGTM