Add e2e preconf load test + remove blutgang/nginx + save logs on err + more by fridrik01 · Pull Request #3042 · berachain/beacon-kit

fridrik01 · 2026-02-28T20:56:17Z

This PR adds preconf load test with a new e2e_preconf_load_test.go that sends eth transfers through the preconf RPC node, measures flashblock inclusion latency (p50/p99), and verifies state consistency against the standard RPC under sustained load with background mempool pressure.

Thie PR is slightly larger than expected as it made some optimizations/changes to our kurtosis env to make sure we can sustain this load, including:

Removed blutgang/nginx as the blutgang load balancer was consuming ~99% CPU, starving consensus and EL nodes. Replaced with direct EL node RPC access.
Added DialWithPooling for high-concurrency RPC clients to avoid ephemeral port exhaustion under load.
The e2e_preconf_load_test test uses block-scanning collector to track when tx receipts are received (to limit rate limiting on EL under heavy load)
Removed seed node (to mirror devnet-core setup) and instead added explicit admin_addPeer calls to ensure that tx gossip does maximum one hop between sequencer and RPC/full nodes when no seed nodes are configured.
TearDownSuite now writes all service logs to e2e-logs/<testname> when any test fails, for easier post-mortem debugging.
Bump kurtosis API from v1.13.2 to v1.16.4 + switch CI to download package instead using package manager (it has outdated versions in package manager)

Note: this PR overrides the bera-reth tag to use latest commit on this PR berachain/bera-reth#214.

Test plan

Run the e2e_preconf_load_test test:

go test -v -timeout 20m -tags e2e_preconf,bls12381,test -run TestPreconfLoadE2ESuite ./testing/e2e/...
=== RUN   TestPreconfLoadE2ESuite
    setup.go:91: <nil> INF Destroying any existing enclave...
    setup.go:96: <nil> INF Creating enclave...
    setup.go:100: <nil> INF Spinning up enclave... num_full_nodes=1 num_validators=2
time="2026-02-28T20:40:16Z" level=info msg="Compressing package 'github.com/berachain/beacon-kit/kurtosis' at '../../kurtosis' for upload"
time="2026-02-28T20:40:16Z" level=info msg="Uploading and executing package 'github.com/berachain/beacon-kit/kurtosis'"
    setup.go:117: <nil> INF Enclave spun up successfully
    setup.go:119: <nil> INF Setting up consensus clients
    setup.go:124: <nil> INF Setting up RPC client
    setup.go:195: <nil> INF Setting up RPC client service=el-full-reth-0
    setup.go:128: <nil> INF Waiting for nodes to get ready...
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=0 target=3
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=0 target=3
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=0 target=3
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=2 target=3
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=3 target=3
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=3 target=3
    setup.go:223: <nil> INF Chain-id is chain_id=80087
    setup.go:269: <nil> INF Funding transaction submitted, waiting for confirmation... account=testAccount1 nonce=2 tx_hash=0x8b0a47afd37d24122a92342940621318042928eaba64d8f36f45f6a73afd5337 value=2000000000000000000000000000
    setup.go:269: <nil> INF Funding transaction submitted, waiting for confirmation... account=testAccount0 nonce=0 tx_hash=0x9b1612c708d4d2c399c9c86566ab6e05f3090098ca8cdab3637f38e05539d75e value=2000000000000000000000000000
    setup.go:269: <nil> INF Funding transaction submitted, waiting for confirmation... account=testAccount2 nonce=1 tx_hash=0x3c9684a6adfc72bb3ccb2930b54e3bf6cf9acbdb2d5b54a481a77a311b3ad9f8 value=2000000000000000000000000000
    setup.go:283: <nil> INF Funding transaction confirmed account=testAccount0 tx_hash=0x9b1612c708d4d2c399c9c86566ab6e05f3090098ca8cdab3637f38e05539d75e
    setup.go:283: <nil> INF Funding transaction confirmed account=testAccount1 tx_hash=0x8b0a47afd37d24122a92342940621318042928eaba64d8f36f45f6a73afd5337
    setup.go:283: <nil> INF Funding transaction confirmed account=testAccount2 tx_hash=0x3c9684a6adfc72bb3ccb2930b54e3bf6cf9acbdb2d5b54a481a77a311b3ad9f8
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=5 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=6 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=7 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=7 target=7
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=7 target=7
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=7 target=7
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=7 target=7
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=7 target=7
    e2e_preconf_load_test.go:113: Preconf RPC EL URL: http://0.0.0.0:62268
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=7 target=8
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=8 target=8
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=8 target=8
=== RUN   TestPreconfLoadE2ESuite/TestAccounts
=== RUN   TestPreconfLoadE2ESuite/TestPreconfTransactions
    e2e_preconf_load_test.go:402: Collector: 40 unseen, 1920 collected, lastScanned=13
    e2e_preconf_load_test.go:402: Collector: 40 unseen, 3920 collected, lastScanned=18
    e2e_preconf_load_test.go:402: Collector: 62 unseen, 5938 collected, lastScanned=23
    e2e_preconf_load_test.go:402: Collector: 40 unseen, 7960 collected, lastScanned=29
    e2e_preconf_load_test.go:402: Collector: 40 unseen, 10000 collected, lastScanned=34
    e2e_preconf_load_test.go:386: Collector: send phase done, 56 unseen, 11944 collected, lastScanned=39
    e2e_preconf_load_test.go:228: Background spammer sent 6005 txs total
    e2e_preconf_load_test.go:248: Preconf latency: avg=198.831379ms p50=193.321958ms p99=607.9265ms min=34.523166ms max=656.713542ms (n=12000)
    e2e_preconf_load_test.go:275: Waiting for full node to reach block 40
=== NAME  TestPreconfLoadE2ESuite
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=39 target=40
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=39 target=40
    setup.go:343: <nil> INF Waiting for finalized block number to reach target finalized=40 target=40
    setup.go:359: <nil> INF Finalized block number reached target 🎉 finalized=40 target=40
=== NAME  TestPreconfLoadE2ESuite/TestPreconfTransactions
    e2e_preconf_load_test.go:289: All 6005 spam tx receipts verified in 2.851186875s
    e2e_preconf_load_test.go:332: State consistency verified at block 40
    e2e_preconf_load_test.go:335: Eth transactions count in block:
    e2e_preconf_load_test.go:342:   block 0: 0 txs
    e2e_preconf_load_test.go:342:   block 1: 1 txs
    e2e_preconf_load_test.go:342:   block 2: 1 txs
    e2e_preconf_load_test.go:342:   block 3: 1 txs
    e2e_preconf_load_test.go:342:   block 4: 1 txs
    e2e_preconf_load_test.go:342:   block 5: 4 txs
    e2e_preconf_load_test.go:342:   block 6: 1 txs
    e2e_preconf_load_test.go:342:   block 7: 1 txs
    e2e_preconf_load_test.go:342:   block 8: 1 txs
    e2e_preconf_load_test.go:342:   block 9: 1 txs
    e2e_preconf_load_test.go:342:   block 10: 499 txs
    e2e_preconf_load_test.go:342:   block 11: 593 txs
    e2e_preconf_load_test.go:342:   block 12: 609 txs
    e2e_preconf_load_test.go:342:   block 13: 600 txs
    e2e_preconf_load_test.go:342:   block 14: 603 txs
    e2e_preconf_load_test.go:342:   block 15: 590 txs
    e2e_preconf_load_test.go:342:   block 16: 630 txs
    e2e_preconf_load_test.go:342:   block 17: 581 txs
    e2e_preconf_load_test.go:342:   block 18: 601 txs
    e2e_preconf_load_test.go:342:   block 19: 602 txs
    e2e_preconf_load_test.go:342:   block 20: 591 txs
    e2e_preconf_load_test.go:342:   block 21: 612 txs
    e2e_preconf_load_test.go:342:   block 22: 601 txs
    e2e_preconf_load_test.go:342:   block 23: 597 txs
    e2e_preconf_load_test.go:342:   block 24: 625 txs
    e2e_preconf_load_test.go:342:   block 25: 577 txs
    e2e_preconf_load_test.go:342:   block 26: 574 txs
    e2e_preconf_load_test.go:342:   block 27: 622 txs
    e2e_preconf_load_test.go:342:   block 28: 600 txs
    e2e_preconf_load_test.go:342:   block 29: 550 txs
    e2e_preconf_load_test.go:342:   block 30: 633 txs
    e2e_preconf_load_test.go:342:   block 31: 615 txs
    e2e_preconf_load_test.go:342:   block 32: 597 txs
    e2e_preconf_load_test.go:342:   block 33: 603 txs
    e2e_preconf_load_test.go:342:   block 34: 555 txs
    e2e_preconf_load_test.go:342:   block 35: 602 txs
    e2e_preconf_load_test.go:342:   block 36: 600 txs
    e2e_preconf_load_test.go:342:   block 37: 628 txs
    e2e_preconf_load_test.go:342:   block 38: 578 txs
    e2e_preconf_load_test.go:342:   block 39: 596 txs
    e2e_preconf_load_test.go:342:   block 40: 173 txs
=== NAME  TestPreconfLoadE2ESuite
    setup.go:391: <nil> INF Destroying enclave...
--- PASS: TestPreconfLoadE2ESuite (119.47s)
    --- PASS: TestPreconfLoadE2ESuite/TestAccounts (0.00s)
    --- PASS: TestPreconfLoadE2ESuite/TestPreconfTransactions (66.60s)
PASS
ok      github.com/berachain/beacon-kit/testing/e2e     120.533s
?       github.com/berachain/beacon-kit/testing/e2e/config      [no test files]
?       github.com/berachain/beacon-kit/testing/e2e/suite       [no test files]
testing: warning: no tests to run
PASS
ok      github.com/berachain/beacon-kit/testing/e2e/suite/types (cached) [no tests to run]
?       github.com/berachain/beacon-kit/testing/e2e/suite/types/tx      [no test files]

…ad test

codecov · 2026-03-01T09:09:24Z

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 61.49%. Comparing base (3e32cca) to head (735d488).

Files with missing lines	Patch %	Lines
beacon/preconf/client.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                   Coverage Diff                   @@
##           add-preconf-support    #3042      +/-   ##
=======================================================
- Coverage                61.50%   61.49%   -0.01%     
=======================================================
  Files                      366      366              
  Lines                    17862    17862              
=======================================================
- Hits                     10986    10985       -1     
- Misses                    5998     5999       +1     
  Partials                   878      878

Files with missing lines	Coverage Δ
consensus/cometbft/service/configs.go	`59.45% <100.00%> (-0.37%)`	⬇️
node-core/node/node.go	`42.42% <100.00%> (ø)`
beacon/preconf/client.go	`0.00% <0.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fridrik01 self-assigned this Feb 28, 2026

fridrik01 had a problem deploying to test-e2e February 28, 2026 20:56 — with GitHub Actions Error

fridrik01 mentioned this pull request Feb 28, 2026

Remove blutgang from kurtosis devnet #3024

Open

fridrik01 force-pushed the add-preconf-e2e-load-test branch from a2068a3 to 36b9093 Compare February 28, 2026 21:07

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:07 — with GitHub Actions Failure

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 36b9093 to 3a05e9b Compare February 28, 2026 21:13

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:13 — with GitHub Actions Error

fridrik01 changed the title ~~remove blutgang/nginx load balancer, add direct EL RPC and preconf load test~~ Add e2e preconf load test + remove blutgang/nginx + save logs on err + more Feb 28, 2026

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 3a05e9b to 2d79a9e Compare February 28, 2026 21:20

fridrik01 temporarily deployed to test-e2e February 28, 2026 21:21 — with GitHub Actions Inactive

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 2d79a9e to 7a78c2b Compare February 28, 2026 21:33

fridrik01 temporarily deployed to test-e2e February 28, 2026 21:33 — with GitHub Actions Inactive

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:33 — with GitHub Actions Failure

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:41 — with GitHub Actions Error

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 7a78c2b to 275fca9 Compare February 28, 2026 21:44

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:44 — with GitHub Actions Failure

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:44 — with GitHub Actions Error

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 275fca9 to 92a94fc Compare February 28, 2026 21:59

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:59 — with GitHub Actions Failure

fridrik01 had a problem deploying to test-e2e February 28, 2026 21:59 — with GitHub Actions Error

remove blutgang/nginx load balancer, add direct EL RPC and preconf lo…

735d488

…ad test

fridrik01 force-pushed the add-preconf-e2e-load-test branch from 92a94fc to 735d488 Compare March 1, 2026 09:00

fridrik01 temporarily deployed to test-e2e March 1, 2026 09:00 — with GitHub Actions Inactive

fridrik01 marked this pull request as ready for review March 1, 2026 09:15

fridrik01 requested a review from a team as a code owner March 1, 2026 09:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e preconf load test + remove blutgang/nginx + save logs on err + more#3042

Add e2e preconf load test + remove blutgang/nginx + save logs on err + more#3042
fridrik01 wants to merge 1 commit intoadd-preconf-supportfrom
add-preconf-e2e-load-test

fridrik01 commented Feb 28, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fridrik01 commented Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fridrik01 commented Feb 28, 2026 •

edited

Loading

codecov bot commented Mar 1, 2026 •

edited

Loading