Make witness generation conform to draft specs by jsign · Pull Request #8 · paradigmxyz/stateless

jsign · 2026-02-17T21:00:18Z

For context, please see paradigmxyz/reth#22289 PR description.

This PR contains some extra fixes that now live in this repo. It temporarily adds a patch in crates to use that Reth branch.

I'll create PR comments to explain better the changes.

Trying to get some feelings about this PR plus the Reth mentioned one!

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

…ince sorting is done at generation

.github/workflows/ci.yml

crates/tries/src/default.rs

testing/ef-tests/src/cases/blockchain_test.rs

testing/ef-tests/src/models.rs

.github/workflows/ci.yml

Signed-off-by: jsign <jsign.uy@gmail.com>

testing/ef-tests/src/models.rs

Cargo.toml

Resolve merge conflicts between jsign/witness-generation-testing and main: - Cargo.toml: merge workspace members (add crates/tries, crates/zeth-mpt, testing/runner) - Cargo.lock: regenerate with all workspace members - Auto-merged: two-pass trie logic applied to crates/tries/src/default.rs, FixtureExecutionWitness + witness comparison in models.rs, TestCaseFailed error wrapping + witness assertion in blockchain_test.rs Amp-Thread-ID: https://ampcode.com/threads/T-019d42a6-d255-707f-9cc5-d228f1236413 Co-authored-by: Amp <amp@ampcode.com>

Amp-Thread-ID: https://ampcode.com/threads/T-019d42a6-d255-707f-9cc5-d228f1236413 Co-authored-by: Amp <amp@ampcode.com>

Signed-off-by: jsign <jsign.uy@gmail.com>

jsign · 2026-03-31T12:51:02Z

@mattsse, @figtracer, after this original PR creation we worked on official specs and tests.

The official specs and test live in execution-spec@projects/zkevm. Now there are also official tests released too, so I think after the Reth PR gets merged, we should probably switch to using these official releases.

I'm not sure yet if you have some opinion on how this can be managed in Reth, since the releases we are doing are basically rebasing the project/zkevm branch onto the latest BAL-devnet EEST branch. This is to make life of ELs easier so they can use the same working branch for the latest devnet to run zkVM tests. Maybe stateless@main should not have any CI against these fixtures (since it is targeting reth@mainnet), and is worth keeping a long running separate PR of stateless on top of reth latest devnet branch with the CI activated there? Leaving this more up to whatever you think would be best.

BTW, we can create a TG group if helps to discuss/chat about these things.

figtracer · 2026-04-04T19:42:23Z

@jsign thanks for the update.

we've created a long-running branch https://github.com/paradigmxyz/stateless/tree/bal-devnet-3 pointing to reth@bal-devnet-3, which runs all adjacent fixtures.

regarding the Reth PR, since part of the trie was completely rewritten ever since, you would need to rebase it paradigmxyz/reth#22289 (comment) then we can just add these improvements on top of stateless later.

+i've also added the missing ForkSpec variants on main, thank you for the heads up! will be actively waiting for more developments

jsign · 2026-04-05T13:21:54Z

@figtracer, thanks! I'll work soon on rebasing paradigmxyz/reth#22289, and coming back to this PR to see how things look and we can keep iterating to move this forward.

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

figtracer · 2026-04-06T00:04:46Z

@jsign bumped reth to main, you can rebase now

Signed-off-by: jsign <jsign.uy@gmail.com>

jsign

@figtracer, I think this is ready for some eyes again:

I resolved previous comments so you see new comments as relevant ones.
Added some extra flag helpers in ef-runners debug stuff.

I think we still need to wait for paradigmxyz/reth#22289 to do a last Cargo.toml update and consider merging.

jsign · 2026-04-06T13:07:49Z

testing/ef-tests/src/case.rs

+    fn test_names(&self) -> Vec<&str>;
+
+    /// Keep only the test cases whose name contains the given substring.
+    fn filter_by_name(&mut self, filter: &str);


@figtracer, I added a new --filter to the ef-runner so it is easier to potentially see if a particular test is problematic.

Example: EF_TEST_TRIE=default cargo run -p ef-test-runner --release -- ./fixtures --filter 'test_abcd[fork_PragueToOsakaAtTime15k-..]

jsign · 2026-04-06T13:13:01Z

Cargo.toml

-reth-ethereum-primitives = { git = "https://github.com/paradigmxyz/reth", rev = "adc960162f", default-features = false }
-reth-evm = { git = "https://github.com/paradigmxyz/reth", rev = "adc960162f", default-features = false }
-reth-evm-ethereum = { git = "https://github.com/paradigmxyz/reth", rev = "adc960162f" }
+reth-chainspec = { git = "https://github.com/paradigmxyz/reth", rev = "6dac6974b4337822e2ce0dfe6d06fcc6c2b30a64", default-features = false }


This is pointing to the latest commit in paradigmxyz/reth#22289

But actually this still pointing to paradigmyxz/reth was a mistake, not sure how this works. Looks like my fork branch commit is observable somehow as a detached commit in the repo.

jsign · 2026-04-06T13:53:09Z

.github/workflows/ci.yml

+    name: Execution Witness tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      RUST_LOG: info
+      RUST_BACKTRACE: 1
+      EF_TEST_TRIE: default
+      EXECUTION_WITNESS_TESTS_URL: https://github.com/jsign/artifacts/releases/download/execution_witness_v0.0.1/execution_witness_tests_409fae8.tar.gz
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+      - name: Run Execution Witness tests
+        run: cargo run -p ef-test-runner --release -- --skip testing/runner/execution-witness-skip.txt $EXECUTION_WITNESS_TESTS_URL


As I mentioned in a comment, there are now official spec and tests.

But since they are meant to run on top of hte latest BAL devnet EL branches, for this PR I decided to keep these older execution witness tests, just temporarily.

Not all the tests pass though, and the reason why is explained in the execution-witness-skip.txt file. I believe teh reason for it will also be a problem whenever the official tests are run.

I would say to:

Merge this PR with this CI as is, since it can provide guarantees on a lot of passing tests until official releases are used.

I can create a issue on this repo to explain in more detail that using official spec tests should be done, and provide more details that can help do that work.

we're running these official fixtures at stateless@bal-devnet-3
main...bal-devnet-3#diff-e723b0fac5c1b72ac28debd4e719da6375ed298123f05186e64f5088677235fbR33

still some things to do regarding the ability to debug the tests properly (some don't pass) but most of the reth@bal-devnet-3 patching has been done

Merge this PR with this CI as is, since it can provide guarantees on a lot of passing tests until official releases are used.

we should wait for upstream first imo, the detached commit problem would also be solved

I can create a issue on this repo to explain in more detail that using official spec tests should be done, and provide more details that can help do that work.

would be very appreciated :)

Nice, sounds great you're doing progress in that branch 🔥

I'll create the issue to explain things in more detail later/soon today.

jsign · 2026-04-06T13:55:03Z

testing/runner/execution-witness-skip.txt

+# The following tests are skipped since they all fail with a message:
+#   > Witness generation mismatch: expected X nodes, generated X+1
+# Note that `X` is a concrete number for each case.
+#
+# As far as investigated, the reason for this is that the new ProofNodeV2 in Reth 
+# bundles ExtensionNodes with their BranchNode children in a single virtual node.
+# When the witness is generated, this encoded node always include both tnodes. 
+#
+# This is not correct for witness generation, since if a proof of absence terminal node
+# is the ExtensionNode, then the proof of absence should finish there and not include the
+# BranchNode since isn't required.
+#
+# This makes any fixture that has a proof of absence ending in an ExtensionNode to fail since
+# it is adding this +1 BranchNode when it shouldn't.


@mediocregopher, I dug a bit into Reth new V2 stuff, and I believe this is the reason for the failures. Trying to help a bit, having some line of thought whenever work continues to happen in making the tests pass.

Prob you have the inner workings more clear in your mind and can have a sense if this makes sense to you.

Hi @jsign yes this makes sense and is correct. In proof V2 we did made an optimization to always reveal the extension's branch in the sparse trie, as we do sometimes need it for certain cases to update our DB properly, and this saves us from having to make another round-trip to the proof workers in those cases. Even in the old proof code we were always fully calculating the branch even if we didn't retain it, so this was simply a net-win for performance.

From the witness's perspective I understand it's not ideal though. Here's my thoughts on a possible solution, lmk if I'm missing something.

For the remove phase this situation shouldn't arise, because it's not possible that an account/slot was removed in a block without it being there in the first place, so an exclusion proof isn't relevant.

For the update phase it should be possible to do a post-processing on the proof nodes which were calculated for that phase, such branches whose short key is splitting a target only retain the extension with that short key, skipping the branch itself.

jsign · 2026-04-06T14:50:27Z

@figtracer, @mediocregopher, reg the failing CI can be quickly reproduced by:

RAYON_NUM_THREADS=1 EF_TEST_TRIE=default cargo run -p ef-test-runner --release -- testing/ef-tests/execution-spec-tests --filter 'test_zero_amount'
RAYON_NUM_THREADS=1 EF_TEST_TRIE=default cargo run -p ef-test-runner --release -- testing/ef-tests/execution-spec-tests --filter 'amount_0'
RAYON_NUM_THREADS=1 EF_TEST_TRIE=default cargo run -p ef-test-runner --release -- testing/ef-tests/execution-spec-tests --filter 'amount_1'

Looks like all these cases are a particular limitation of the default trie verification of the witness, when execution_witness.state is empty. I think the way things work for this trie is that it always require the root node RLP in the witness, even if the block never really required any state for execution. (Note how all the failures say got: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, so it is related only for this border case)/ For the Zeth trie, this CI is passing since it doesn't require this.

I'm not sure there is an easy fix here. Do you have any suggestions?

mediocregopher · 2026-04-07T11:00:56Z

@jsign @figtracer if you all want to start a group in TG feel free, my username is the same there

jsign added 7 commits February 16, 2026 18:22

feat: add ef-test-runner for executing test suites

762348a

feat: add Prague to Osaka fork specification in ForkSpec enum

4c5f7d3

all fixes

65594e9

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

refactor: simplify assert_matches method in FixtureExecutionWitness s…

b4c467c

…ince sorting is done at generation

testing/runner: add support for remote .tar.gz bundle

78ae37e

ci: add execution witness test check

7fe7493

cargo: add temp reth patching

fd4e9d1

jsign commented Feb 17, 2026

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

update to latest reth pr commit

5a2e472

Signed-off-by: jsign <jsign.uy@gmail.com>

jsign marked this pull request as ready for review February 18, 2026 00:22

figtracer reviewed Mar 31, 2026

View reviewed changes

testing/ef-tests/src/models.rs Outdated Show resolved Hide resolved

figtracer reviewed Mar 31, 2026

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

figtracer and others added 3 commits March 31, 2026 09:17

ci: set EF_TEST_TRIE env var for execution witness tests

1b0e9ec

Amp-Thread-ID: https://ampcode.com/threads/T-019d42a6-d255-707f-9cc5-d228f1236413 Co-authored-by: Amp <amp@ampcode.com>

move func to assert.rs

f0edf76

Signed-off-by: jsign <jsign.uy@gmail.com>

jsign added 3 commits April 5, 2026 12:08

refactor: update assert functions import and visibility in models

eb4423f

Merge branch 'main' into jsign/witness-generation-testing

d227eb7

rebase fix

c52abfb

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign mentioned this pull request Apr 5, 2026

chore: bump reth deps to latest main #17

Merged

jsign added 3 commits April 6, 2026 09:53

improvements

444944d

Merge branch 'main' into jsign/witness-generation-testing-rebased

402659c

update reth

6406a96

jsign marked this pull request as draft April 6, 2026 13:05

jsign added 2 commits April 6, 2026 10:47

allow test skip list

969b7fe

add skip lits

95174b5

Signed-off-by: jsign <jsign.uy@gmail.com>

jsign commented Apr 6, 2026

View reviewed changes

jsign marked this pull request as ready for review April 6, 2026 13:57

jsign added 3 commits April 6, 2026 10:58

fmt

315fcda

lint

4a69b22

lint

6846e66

jsign mentioned this pull request Apr 7, 2026

Run execution witness tests against official releases (& debugging tips) #18

Open

Conversation

jsign commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsign commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

figtracer commented Apr 4, 2026

Uh oh!

jsign commented Apr 5, 2026

Uh oh!

figtracer commented Apr 6, 2026

Uh oh!

jsign left a comment

Choose a reason for hiding this comment

Uh oh!

jsign Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

jsign Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

jsign Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

figtracer Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

jsign Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

jsign Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

mediocregopher Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

jsign commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mediocregopher commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jsign commented Feb 17, 2026 •

edited

Loading

jsign commented Mar 31, 2026 •

edited

Loading

jsign commented Apr 6, 2026 •

edited

Loading