receipt root mismatch

[InoMurko]

Describe the bug

We regularly get state root mismatches on syncing chains from the genesis:

2026-03-31T16:46:15.745838Z  INFO Received new payload from consensus engine number=66876 hash=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe
2026-03-31T16:46:15.750509Z  INFO Canonical chain committed number=66876 hash=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe elapsed=14.006µs
2026-03-31T16:46:17.008866Z  INFO New payload job created id=0x01e583239bc4b3cf parent=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe
2026-03-31T16:46:17.875725Z  INFO Received new payload from consensus engine number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5
2026-03-31T16:46:20.131771Z  WARN Invalid block error on new payload invalid_hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 invalid_number=66877 validation_err=receipt root mismatch: got 0x6dbb885b5405fe05304e5ef604be520f9183b2a64afaee6c81b2ced815945b12, expected 0x0d19b7a9c229fb742a7f1346ab73055e91b6d5a99b896e7d19fa087c36dba2a5
2026-03-31T16:46:20.199017Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe, block: NumHash { number: 66877, hash: 0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 } }
2026-03-31T16:46:20.695121Z  WARN Encountered invalid block number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5
2026-03-31T16:46:20.695349Z  INFO Block added to canonical chain number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 peers=0 txs=1 gas_used=52.49Kgas gas_throughput=763.46Kgas/second gas_limit=30.00Mgas full=0.2% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=68.75117ms
2026-03-31T16:47:02.980320Z  INFO Status connected_peers=0 latest_block=66876

once we restart it continues from the previous block and moves on for a few minutes, then stops at a random block. We could have a statefulset of replicas syncing and they would stop at random blocks (not even the same blocks, even though they started syncing in paralel). Which probably means there's a race condition of sorts?

Steps to reproduce

ConstellationCrypto/replica-guide-manta-mainnet#2

Node logs

We regularly get state root mismatches on syncing chains from the genesis:

2026-03-31T16:46:15.745838Z  INFO Received new payload from consensus engine number=66876 hash=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe
2026-03-31T16:46:15.750509Z  INFO Canonical chain committed number=66876 hash=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe elapsed=14.006µs
2026-03-31T16:46:17.008866Z  INFO New payload job created id=0x01e583239bc4b3cf parent=0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe
2026-03-31T16:46:17.875725Z  INFO Received new payload from consensus engine number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5
2026-03-31T16:46:20.131771Z  WARN Invalid block error on new payload invalid_hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 invalid_number=66877 validation_err=receipt root mismatch: got 0x6dbb885b5405fe05304e5ef604be520f9183b2a64afaee6c81b2ced815945b12, expected 0x0d19b7a9c229fb742a7f1346ab73055e91b6d5a99b896e7d19fa087c36dba2a5
2026-03-31T16:46:20.199017Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0xed5644e30b4527267e940c546a9f87df0d23cc28c8e0f582dacf28a5fdbe19fe, block: NumHash { number: 66877, hash: 0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 } }
2026-03-31T16:46:20.695121Z  WARN Encountered invalid block number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5
2026-03-31T16:46:20.695349Z  INFO Block added to canonical chain number=66877 hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 peers=0 txs=1 gas_used=52.49Kgas gas_throughput=763.46Kgas/second gas_limit=30.00Mgas full=0.2% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=68.75117ms
2026-03-31T16:47:02.980320Z  INFO Status connected_peers=0 latest_block=66876

Platform(s)

No response

Container Type

Kubernetes

What version/commit are you on?

1.11.3

What database version are you on?

1

Which chain / network are you on?

manta pacific

What type of node are you running?

Archive (default)

What prune config do you use, if any?

none

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[InoMurko]

genesis.json

{
  "config": {
    "chainId": 3441006,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0,
    "istanbulBlock": 0,
    "muirGlacierBlock": 0,
    "berlinBlock": 0,
    "londonBlock": 0,
    "arrowGlacierBlock": 0,
    "grayGlacierBlock": 0,
    "mergeNetsplitBlock": 0,
    "bedrockBlock": 0,
    "regolithTime": 0,
    "canyonTime": 1726610400,
    "ecotoneTime": 1726610400,
    "terminalTotalDifficulty": 0,
    "terminalTotalDifficultyPassed": true,
    "optimism": {
      "eip1559Elasticity": 10,
      "eip1559Denominator": 50,
      "eip1559DenominatorCanyon": 0
    }
  },
  "nonce": "0x0",

op-reth node --chain=/config/genesis.json --datadir=/root/datadir
              --rollup.disable-tx-pool-gossip
              --rollup.sequencer=http://op-geth:8545 --metrics=0.0.0.0:9001
              --authrpc.addr=0.0.0.0 --authrpc.port=8551
              --authrpc.jwtsecret=/config/jwt.txt --http --http.addr=0.0.0.0
              --http.port=8545 --http.api=web3,debug,eth,txpool,net
              --http.corsdomain="*" --ws --ws.addr=0.0.0.0 --ws.port=8546
              --ws.origins="*" --ipcdisable

[InoMurko]

2026-03-31T20:08:48.302338Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/3441006
2026-03-31T20:08:48.303622Z  INFO Starting Reth version="1.11.3-dev (d6324d6)"
2026-03-31T20:08:48.303787Z  INFO Opening database path="/root/datadir/db"
2026-03-31T20:08:48.325354Z  INFO Launching node
2026-03-31T20:08:48.326981Z  INFO Configuration loaded path="/root/datadir/reth.toml"
2026-03-31T20:08:48.330293Z  INFO check_consistency: Healing static file inconsistencies.
2026-03-31T20:08:48.373810Z  INFO check_consistency:check_consistency{read_only=false}: Verifying storage consistency.
2026-03-31T20:08:48.377182Z  INFO Database opened
2026-03-31T20:08:48.377349Z  INFO Starting metrics endpoint at 0.0.0.0:9001
2026-03-31T20:08:48.378797Z  INFO
Pre-merge hard forks (block based):
- Bedrock                          @0
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Canyon                           @1726610400
- Ecotone                          @1726610400
2026-03-31T20:08:48.378841Z  INFO Loaded storage settings settings=StorageSettings { storage_v2: false }
2026-03-31T20:08:48.397363Z  INFO Transaction pool initialized
2026-03-31T20:08:49.618900Z  INFO P2P networking initialized enode=enode://d3f78009e6b5e37650745ba6d19847faf30f6811a6f842afed66be14013f7208e2b388d28e20fca711dc0e9f8ed047b4daf7de561a1ade26bc771df23ea97243@127.0.0.1:30303?discport=9200
2026-03-31T20:08:49.619217Z  INFO StaticFileProducer initialized
2026-03-31T20:08:49.620199Z  INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: None, transaction_lookup: None, receipts: None, account_history: None, storage_history: None, bodies_history: None, receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2026-03-31T20:08:49.621040Z  INFO Consensus engine initialized
2026-03-31T20:08:49.633845Z  INFO Engine API handler initialized
2026-03-31T20:08:49.649145Z  INFO RPC HTTP server started url=0.0.0.0:8545
2026-03-31T20:08:49.649173Z  INFO RPC WS server started url=0.0.0.0:8546
2026-03-31T20:08:49.649203Z  INFO RPC auth server started url=0.0.0.0:8551
2026-03-31T20:08:49.649416Z  INFO Starting consensus engine
2026-03-31T20:08:52.621766Z  INFO Status connected_peers=0 latest_block=177085
2026-03-31T20:10:07.679142Z  INFO Status connected_peers=0 latest_block=177085

[InoMurko]

A commonality between two rollups that end up with receipt mismatch is that they both have only these HFs:

(reverse-i-search)`p': logs op-reth-node-84b57d4fd5-npbgz rollu^Cbedrock-towns-devnet-zxq91
InoM4:~ inomurko$ logs reth-replica-0 rollup-bedrock-towns-devnet-zxq91
Defaulted container "op-reth-replica" out of: op-reth-replica, download-configs (init), dependency-health (init)
2026-04-01T07:41:34.991085Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/6524490
2026-04-01T07:41:34.992329Z  INFO Starting Reth version="1.11.3-dev (d6324d6)"
2026-04-01T07:41:34.992370Z  INFO Opening database path="/root/datadir/db"
2026-04-01T07:41:35.012458Z  INFO Launching node
2026-04-01T07:41:35.014319Z  INFO Configuration loaded path="/root/datadir/reth.toml"
2026-04-01T07:41:35.018039Z  INFO check_consistency: Healing static file inconsistencies.
2026-04-01T07:41:35.066073Z  INFO check_consistency:check_consistency{read_only=false}: Verifying storage consistency.
2026-04-01T07:41:35.069309Z  INFO Database opened
2026-04-01T07:41:35.069591Z  INFO Starting metrics endpoint at 0.0.0.0:9001
2026-04-01T07:41:35.071445Z  INFO
Pre-merge hard forks (block based):
- Bedrock                          @0
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Canyon                           @1712890200
- Ecotone                          @1712890200

[mattsse]

hmm

2026-03-31T16:46:20.131771Z WARN Invalid block error on new payload invalid_hash=0x1a92b10c8d34b560aaf0f7e06385f48c0d46ecbb9da614016fa45294121ae5f5 invalid_number=66877 validation_err=receipt root mismatch: got 0x6dbb885b5405fe05304e5ef604be520f9183b2a64afaee6c81b2ced815945b12, expected 0x0d19b7a9c229fb742a7f1346ab73055e91b6d5a99b896e7d19fa087c36dba2a5

this usually indicates some kind of gas mismatch.
to rule out any caching issues, could you try to repro this by --engine.disable-state-cache?

cc @theochap in case you've seen anything like this

@InoMurko do you have any additional contracts in genesis? maybe this is some kind of stateclear issue

[InoMurko]

Trying that right away (engine.disable-state-cache).

Both genesis files are available here, they come with genesis contracts from optimism (those on the range of 4200000000000000000000000000000000000000 -> 4200000000000000000000000000000000000800 and also
c0d3c0d3c0d3c0d3c0d3c0d3c0d3c0d3c0d30000 etc...
manta_sepolia_genesis.json
towns_devnet_genesis.json

[InoMurko]

Additionally, I have the exact same setup for b3 testnet. And has synced

2026-04-01T11:09:37.205706Z  INFO Received new payload from consensus engine number=3558465 hash=0xb350f44639354450ccf76435626b738c3ad61f644b825fae5ce08a1112004720
2026-04-01T11:09:37.207049Z  INFO Canonical chain committed number=3558465 hash=0xb350f44639354450ccf76435626b738c3ad61f644b825fae5ce08a1112004720 elapsed=11.334µs

without any issues thus far. Hasn't stopped from receipt root mismatch.
It's head genesis:

head -n 55 /config/genesis.json
{
  "config": {
    "chainId": 1993,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0,
    "istanbulBlock": 0,
    "muirGlacierBlock": 0,
    "berlinBlock": 0,
    "londonBlock": 0,
    "arrowGlacierBlock": 0,
    "grayGlacierBlock": 0,
    "mergeNetsplitBlock": 0,
    "shanghaiTime": 0,
    "cancunTime": 0,
    "bedrockBlock": 0,
    "regolithTime": 0,
    "canyonTime": 0,
    "ecotoneTime": 0,
    "terminalTotalDifficulty": 0,
    "terminalTotalDifficultyPassed": true,
    "optimism": {
      "eip1559Elasticity": 6,
      "eip1559Denominator": 250,
      "eip1559DenominatorCanyon": 250
    }
  },
  "nonce": "0x0",
  "timestamp": "0x662c3820",
  "extraData": "0x424544524f434b",
  "gasLimit": "0x1c9c380",
  "difficulty": "0x0",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x4200000000000000000000000000000000000011",
  "alloc": {
    "0000000000000000000000000000000000000000": {
      "balance": "0x1",
      "nonce": "0x2"
    },
    "0000000000000000000000000000000000000001": {
      "balance": "0x1"
    },
    "0000000000000000000000000000000000000002": {
      "balance": "0x1"
    },
    "0000000000000000000000000000000000000003": {
      "balance": "0x1"
    },
    "0000000000000000000000000000000000000004": {
      "balance": "0x1"
    },

[InoMurko]

Restarted the chain towns devnet and it ran into the same issue (with engine.disable-state-cache):

2026-04-01T11:32:11.330535Z  INFO New payload job created id=0x01b730cb0027f87a parent=0xb4a9395a693a69ca019ebea6eab9e0d8aed872d7efa374dc4a3fe26fb2c0edf2
2026-04-01T11:32:11.332760Z  INFO Received new payload from consensus engine number=376128 hash=0xf5ca76dd518ed9254889089df474d47975b7f6eba49e69f1a0b4b0f22f99d7ef
2026-04-01T11:32:11.337479Z  WARN Invalid block error on new payload invalid_hash=0xf5ca76dd518ed9254889089df474d47975b7f6eba49e69f1a0b4b0f22f99d7ef invalid_number=376128 validation_err=receipt root mismatch: got 0x1eed5b7f1fa7dd528809ab5e9766f411a52573b848b31e75a9af1561364e621b, expected 0x55401423176a0c7407f559ed2a7f9b298bd5836180a80538fed192b82a98af02
2026-04-01T11:32:11.337503Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0xb4a9395a693a69ca019ebea6eab9e0d8aed872d7efa374dc4a3fe26fb2c0edf2, block: NumHash { number: 376128, hash: 0xf5ca76dd518ed9254889089df474d47975b7f6eba49e69f1a0b4b0f22f99d7ef } }
2026-04-01T11:32:11.337554Z  WARN Encountered invalid block number=376128 hash=0xf5ca76dd518ed9254889089df474d47975b7f6eba49e69f1a0b4b0f22f99d7ef
2026-04-01T11:32:11.337630Z  INFO Block added to canonical chain number=376128 hash=0xf5ca76dd518ed9254889089df474d47975b7f6eba49e69f1a0b4b0f22f99d7ef peers=0 txs=1 gas_used=64.00Kgas gas_throughput=4.01Ggas/second gas_limit=30.00Mgas full=0.2% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=15.966µs

              op-reth node --chain=/config/genesis.json --datadir=/root/datadir
              --rollup.disable-tx-pool-gossip
              --rollup.sequencer=http://op-geth:8545 --metrics=0.0.0.0:9001
              --authrpc.addr=0.0.0.0 --authrpc.port=8551
              --authrpc.jwtsecret=/config/jwt.txt --http --http.addr=0.0.0.0
              --http.port=8545 --http.api=web3,debug,eth,txpool,net,miner
              --http.corsdomain="*" --ws --ws.addr=0.0.0.0 --ws.port=8546
              --ws.origins="*" --ipcdisable --engine.disable-state-cache

Same with manta sepolia:

2026-04-01T11:33:44.281491Z  INFO Received new payload from consensus engine number=211222 hash=0xe737b72c0f89cadf670c1dca16aae3340b9344153e53a77ca30703d23dc80455
2026-04-01T11:33:44.287740Z  WARN Invalid block error on new payload invalid_hash=0xe737b72c0f89cadf670c1dca16aae3340b9344153e53a77ca30703d23dc80455 invalid_number=211222 validation_err=receipt root mismatch: got 0x3dcd2b38f23fa9e09fda0e7b2d2834f3900d3ccf01f12e23da130257f3bb6d87, expected 0xc6a424d317ede0d9b8c12077d59fd26897051ffe8370197314e73a18abd99171
2026-04-01T11:33:44.288445Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0xf814cacde682b3067512c49fb7aa92a990d0668374aa49785dcc10f4850d9d94, block: NumHash { number: 211222, hash: 0xe737b72c0f89cadf670c1dca16aae3340b9344153e53a77ca30703d23dc80455 } }
2026-04-01T11:33:44.289354Z  WARN Encountered invalid block number=211222 hash=0xe737b72c0f89cadf670c1dca16aae3340b9344153e53a77ca30703d23dc80455
2026-04-01T11:33:44.289501Z  INFO Block added to canonical chain number=211222 hash=0xe737b72c0f89cadf670c1dca16aae3340b9344153e53a77ca30703d23dc80455 peers=0 txs=1 gas_used=50.49Kgas gas_throughput=2.50Ggas/second gas_limit=30.00Mgas full=0.2% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=20.183µs
2026-04-01T11:34:17.503857Z  INFO Status connected_peers=0 latest_block=211221

              op-reth node --chain=/config/genesis.json --datadir=/root/datadir
              --rollup.disable-tx-pool-gossip
              --rollup.sequencer=http://op-geth:8545 --metrics=0.0.0.0:9001
              --authrpc.addr=0.0.0.0 --authrpc.port=8551
              --authrpc.jwtsecret=/config/jwt.txt --http --http.addr=0.0.0.0
              --http.port=8545 --http.api=web3,debug,eth,txpool,net,miner
              --http.corsdomain="*" --ws --ws.addr=0.0.0.0 --ws.port=8546
              --ws.origins="*" --ipcdisable
              --engine.disable-parallel-sparse-trie
              --engine.disable-state-cache

[theochap]

We have not noticed anything like that since v1.11.3. What are you using for your consensus layer node @InoMurko ?

[InoMurko]

We have not noticed anything like that since v1.11.3. What are you using for your consensus layer node @InoMurko ?

Op-node 1.16.1.

[theochap]

And are you using EL or CL sync?

[InoMurko]

And are you using EL or CL sync?

CL