Add WebGL browser tests to vello_sparse_tests

[ajakubowicz-canva]

Note: This PR currently branches off #1021 as there is a lint error on main and I wanted to ensure CI fully passed.

Tip

This is ready to review and passes all tests in CI.

Context

This follows #1011 and is the testing portion. In order to ensure renderer parity and correctness, I've forked the vello_hybrid tests to also include a _hybrid_webgl variant that use: #[wasm_bindgen_test::wasm_bindgen_test] and run via the browser.

As a demo of this PR, I locally introduced a "fake error" that reflects the image. After running `wasm-pack test --chrome --features webgl` from the `vello_sparse_tests` directory, I got the following very long webpage. (unfurl details section by clicking this text, and scroll to bottom of test results)

Features

• Add headless and headed browser testing of the webgl backend.
• Because the browser can't read from the file system easily, reference images are inlined into the binary via the proc macro and provided to the check_refs function.
• Existing _hybrid tests are unchanged. I've added a _hybrid_webgl test.
• For ease of debugging the diff'd images are appended to the document after tests have run. This makes it much easier to understand the failure.
• Add check vello_hybrid webgl backend passes vello_sparse_tests suite CI step that uses headless browser.

These tests could be fancier, but they do the job right now. Debugging a failure isn't easy, however the diff can be manually retrieved and compared by a human by spinning up a browser.

The goal of having webgl tests automatically match the vello_hybrid tests is to keep the wgpu and webgl renderer backends in sync.

Test plan

This PR changes no business logic and is the test.

Note: Browser WebGL backend tests currently pass all of the same tests as the vello_hybrid wgpu renderer. Can be seen in CI here: https://github.com/linebender/vello/actions/runs/15211363959/job/42786078261?pr=1020

I tested the functionality of this change by introducing intentional breakages and ensuring that headless tests failed, and diff images could be found on the browser when run without headless.

Risks

I'm not sure of any risks as this isn't user facing.

Manually testing this change

Pull branch down locally. Then navigate the vello_sparse_tests and run: wasm-pack test --chrome --features webgl. Navigate to the browser URL and see 65 tests succeed.

Followup work

#1026
#1025

[ajakubowicz-canva]

This ensures that we're testing the vello_hybrid WebGL2 rendering backend by activating the webgl feature.

[ajakubowicz-canva]

Added the vello hybrid webgl tests and LFS. Without LFS the build system can't find any images to compile into the WASM test binary.

[ajakubowicz-canva]

cc/ @LaurenzV (no action required from you), just wanted to let you know that your test framework was very readable and easy to extend to support browser tests.

[LaurenzV]

I'm not sure what changed, but when I run cargo test in the sparse_strips folder, it doesn't run the tests at all anymore. See also the CI run: https://github.com/linebender/vello/actions/runs/15211363959/job/42788302054?pr=1020, they seem to be ignored on any platform other than WASM. :p

[LaurenzV]

This is probably not for you to figure out, but at least on my system running the tests withwasm-pack doesn't work:

   Compiling crossbeam-channel v0.5.15
warning: [email protected]: error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
warning: [email protected]: 1 error generated.
error: failed to run custom build command for `libdeflate-sys v1.24.0`

Caused by:
  process didn't exit successfully: `/Users/lstampfl/Programming/GitHub/vello/target/debug/build/libdeflate-sys-e5920edec3afac9f/build-script-build` (exit status: 1)
  --- stdout
  OPT_LEVEL = Some(0)
  TARGET = Some(wasm32-unknown-unknown)
  HOST = Some(aarch64-apple-darwin)
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  CARGO_ENCODED_RUSTFLAGS = Some(--cfggetrandom_backend="wasm_js")
  cargo:warning=error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
  cargo:warning=1 error generated.

  --- stderr


  error occurred in cc-rs: command did not execute successfully (status code exit status: 1): LC_ALL="C" "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fno-exceptions" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "libdeflate" "-ffreestanding" "-nostdlib" "-DFREESTANDING" "-o" "/Users/lstampfl/Programming/GitHub/vello/target/wasm32-unknown-unknown/debug/build/libdeflate-sys-41f84d30b488f0a0/out/lib/c7b4838539265e38-cpu_features.o" "-c" "libdeflate/lib/arm/cpu_features.c"


warning: build failed, waiting for other jobs to finish...
Error: Compilation of your program failed
Caused by: Compilation of your program failed
Caused by: failed to execute `cargo build`: exited with exit status: 101
  full command: cd "/Users/lstampfl/Programming/GitHub/vello/sparse_strips/vello_sparse_tests" && "cargo" "build" "--tests" "--target" "wasm32-unknown-unknown" "--features" "webgl"

Which surprises me, because we do have the freestanding feature activated for oxipng... So not sure what's going on here.

[AndrewJakubowicz]

I'm not sure what changed, but when I run cargo test in the sparse_strips folder, it doesn't run the tests at all anymore. See also the CI run: https://github.com/linebender/vello/actions/runs/15211363959/job/42788302054?pr=1020, they seem to be ignored on any platform other than WASM. :p

I’ll definitely address that! Sorry and thank you for picking that up. It’s awkward that the easiest way to pass all the tests is to have zero tests :p

Also that’s extremely strange that you are getting an error regarding wasm32.

[ajakubowicz-canva]

I'm not sure what changed, but when I run cargo test in the sparse_strips folder, it doesn't run the tests at all anymore. See also the CI run: https://github.com/linebender/vello/actions/runs/15211363959/job/42788302054?pr=1020, they seem to be ignored on any platform other than WASM. :p

Fixed in 5009e2c

[ajakubowicz-canva]

Thanks @LaurenzV for the initial very great review.
I've fixed all issues raised.

• build.rs conditionally builds the images into the wasm binary via environment variable check (because cfg with target_arch doesn't work in build.rs).
• Running cargo test from vello_sparse_tests runs the tests.
• Running wasm-pack test --headless --firefox --features webgl passes the webgl tests headless. (or chrome in CI)

[ajakubowicz-canva]

This is probably not for you to figure out, but at least on my system running the tests withwasm-pack doesn't work

Do other existing examples work? For example from within sparse_strips/vello_hybrid/examples/wgpu_webgl does wasm-pack test --chrome work?

[LaurenzV]

Do other existing examples work? For example from within sparse_strips/vello_hybrid/examples/wgpu_webgl does wasm-pack test --chrome work?

Yes, this test works fine for me.

[LaurenzV]

Seems fine to me overall, but I'll leave the final review to someone more knowledgeable in WebGL. Perhaps @DJMcNab also wants to take a look at the CI side of things.

[DJMcNab]

A few threads of discussion:

1. A potential way to avoid the build script (resolution of this is from my perspective blocking - but resolution could mean "we decide to make no change here)
2. A tweak for how the CI job is named
3. Thoughts about running even more tests on CI (CPU only tests should be trivial, whereas the "wgpu" hybrid tests depend on ChromeDriver having WebGPU support). This can be deferred.

[DJMcNab]

This pretty much works as I'd expect - leaving final review to others who have more expertise for the web platform.

[taj-p]

This pretty much works as I'd expect - leaving final review to others who have more expertise for the web platform.

Web platform changes LGTM.

I left a couple small comments - none blocking. Perhaps check in with @grebmeg to check whether Alex has any blocking outstanding comments before merge 🙏 .

[taj-p]

Have we given much thought to the pattern we use for target specific implementations? We're currently using the below approach:

#[cfg(TARGET_1)]
pub fn check_ref(...) { ... }

#[cfg(TARGET_2)]
pub fn check_ref(...) { ... }

I suspect that, for better organisation and discovering all target implementations, something like the below may be better when we have separate function implementations. It's more code, but I think it might ease the addition of later targets (e.g. cpu webgl) and improves discoverability.

pub(crate) fn check_ref(
    ...
) {
    #[cfg(TARGET_1)]
    check_ref_wgpu(...)
    #[cfg(TARGET_2]
    check_ref_webgl(...)
}

#[cfg(TARGET_1)]
fn check_ref_wgpu(...) { ... }

#[cfg(TARGET_2)]
fn check_ref_webgl(...) { ... }

I think when we have sufficiently complex scenarios, it might be worth using a module splitting approach. But perhaps for small functions, the above is better?

[ajakubowicz-canva]

Tracking this great refactor as followup work that I'll include in the PR resolving this issue: #1025 (comment)

[grebmeg]

Great work, @AndrewJakubowicz! 🎉 I don’t have any blocking comments. The issue with headless mode in Chrome seems related to something in my local configuration, so I think we can address it later.