Add .NET on WebAssembly workload #73

maraf · 2025-05-23T12:59:18Z

Add benchmarks for .NET runtime on WebAssembly.
This PR adds 2 benchmarks that test characteristic WebAssembly code for .NET. Benchmarks don't contain UI manipulations and javascript interop except for the necessary startup.

Flavors

There are two flavors of the same benchmark code.

1. Interpreter

The default mode for running .NET (Mono runtime) on WebAssembly is using an interpreter. In this mode .NET assemblies are loaded into WebAssembly memory and IL interpreter interprets the code in a loop.

2. AOT

The second mode Mono runtime on WebAssembly operates in is mixed AOT. It is a mixture of AOTed code and fallbacks to interpreter for scenarios that are not supported by the AOT compiler.

Code samples

These code samples are combined into a single test for each .NET flavor.

RayTracer

Source code at https://github.com/pavelsavara/dotnet-wasm-raytracer. It computes a 3D scene in memory. It is a computation heavy algorithm that stretches the performance of interpreter loop. The UI update part is removed.

Exception handling

.NET uses a two-phase exception handling that is built on top of WasmExceptionHandling.

String

Very common and often used type in .NET codebase.

JSON serialization & deserialization

Very common and often used types in .NET codebase.

Fixes dotnet/runtime#109953

netlify · 2025-05-23T12:59:30Z

✅ Deploy Preview for webkit-jetstream-preview ready!

Name	Link
🔨 Latest commit	`5aa9dad`
🔍 Latest deploy log	https://app.netlify.com/projects/webkit-jetstream-preview/deploys/6877a36ba557be000827bb25
😎 Deploy Preview	https://deploy-preview-73--webkit-jetstream-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

JetStreamDriver.js

eqrion · 2025-05-30T16:43:34Z

@maraf Thanks for this!

A couple of questions:

What is the default or recommended compilation mode for .NET on wasm? Is it the interpreter or AOT?
Does exception handling happen in performance sensitive .NET code? The wasm exception handling feature has been mostly designed with the expectation it's for cold non performance sensitive code. So adding a benchmark just testing it's performance would not be ideal. It's useful to have one for looking for easy optimizations we could make, but to really get good performance would require some techniques that could slow down normal code and we'd prefer not to do that.
Can the size of the iterations on these benchmarks be reduced? I'm getting about a minute and a half to just run 'dotnet-aot-string' in Chrome on a pretty powerful laptop. For reference, the zlib-wasm test is running in about 3 seconds.

I think we should probably reduce the number or combine some of these tests so that we're not weighting dotnet too heavily. Right now there are 8 added, which makes it 8 times more important than any other wasm test. I think we've talked about having experimental tests that are not in the main score by default, we could potentially do that here.

maraf · 2025-06-02T07:50:20Z

@eqrion Thank you for feedback!

There isn't one recommendation, and the usage is like 50:50. Both modes have some specific characteristics.
.NET has two pass exception handling (virtual unwind to find matching handler docs and actual unwind) which made us implementation some layers on top of wasm exception handling. It is pretty common to use exceptions in .NET to control the non-success path in code execution.
I was targeting <= 10s per iteration on my machine, I'll work on further reduction.

maraf · 2025-06-18T15:37:01Z

I have combined all the tests to single benchmark. We still have two flavors (interpreter & aot) since those can't be combined together. I have reduced the iterations we do to reduce time to run the benchmark

issackjohn · 2025-06-25T16:19:34Z

Here are the new times that I've observed.

Takes ~12 seconds for dotnet-aot on my machine now and ~28 seconds for dotnet-interp.
Takes my machine about 2 minutes and 56 seconds to run the entire benchmark suite in it's configurations as defined in this PR.
2.5 seconds for zlib-wasm

Using https://deploy-preview-73--webkit-jetstream-preview.netlify.app/

danleh · 2025-07-08T16:30:54Z

Running in shells (e.g., via path/to/d8 cli.js -- dotnet-aot or path/to/spidermonkey/js cli.js dotnet-aot) fails with JetStream3 failed: ReferenceError: dotnetRuntimeUrl is not defined. Could you fix that?

danleh · 2025-07-08T16:44:36Z

@maraf and @issackjohn I left a first round of comments (sorry for the late look). Overall, very happy to have another source language/toolchain combination, thanks for the contribution! :) See the comments above for more detailed issues.

Once it runs in shells (d8, jsc, js), I'll take another look at a CPU profile/flamegraph, compiler tiering behavior etc.

maraf · 2025-07-09T15:03:16Z

@danleh Thank you!

I have updated the benchmark for work without preloading which seems to be implemented only for browser case.
The number of code iterations is reduced

PS JetStream> v8.cmd cli.js -- dotnet
Starting JetStream3
Running dotnet-interp:
    Startup: 1.130
    Worst Case: 1.137
    Average: 1.137
    Score: 1.135
    Wall time: 0:11.575
dotnet-interp
Running dotnet-aot:
    Startup: 4.250
    Worst Case: 4.262
    Average: 4.265
    Score: 4.259
    Wall time: 0:05.508
dotnet-aot


First: 2.192
Worst: 2.201
Average: 2.202

Total Score:  2.198

maraf · 2025-07-09T16:13:19Z

Measurements after scene size reduction for interpreter

PS JetStream> v8.cmd cli.js -- dotnet
Starting JetStream3
Running dotnet-interp:
    Startup: 2.019
    Worst Case: 2.077
    Average: 2.077
    Score: 2.058
    Wall time: 0:06.660
dotnet-interp
Running dotnet-aot:
    Startup: 3.815
    Worst Case: 3.776
    Average: 3.864
    Score: 3.818
    Wall time: 0:06.072
dotnet-aot


First: 2.775
Worst: 2.801
Average: 2.833

Total Score:  2.803

issackjohn · 2025-07-09T16:36:44Z

New times for me.

PS JetStream> v8.cmd cli.js -- dotnet
Starting JetStream3
Running dotnet-interp:
    Startup: 1.779
    Worst Case: 1.814
    Average: 1.814
    Score: 1.802
    Wall time: 0:07.551
dotnet-interp
Running dotnet-aot:
    Startup: 4.512
    Worst Case: 4.515
    Average: 4.525
    Score: 4.517
    Wall time: 0:05.264
dotnet-aot


First: 2.833
Worst: 2.862
Average: 2.865

Total Score:  2.853

danleh

(Oops, I had those comments already typed out but didn't hit submit, sorry! 🙈)
Mostly clarification questions, minor suggestions. Could you please take a look @maraf @issackjohn ?

JetStreamDriver.js

wasm/dotnet/benchmark.js

wasm/dotnet/src/dotnet/Benchmarks/BenchTask.cs

wasm/dotnet/src/dotnet/Program.cs

JetStreamDriver.js

wasm/dotnet/src/dotnet/Benchmarks/BenchTask.cs

JetStreamDriver.js

danleh · 2025-07-10T12:47:05Z

I also briefly looked at the flamegraphs/CPU profiles of the current state of the workloads, but it's hard to dig deeper since we don't have debug info/source names for the Wasm functions.

@maraf Is there some way with Blazor to generate some form of function name map off-to-the-side (similar to -g --emit-symbol-map for Emscripten, see e.g.

JetStream/wasm/zlib/build.sh

Line 22 in 9163dcb

-g1 --emit-symbol-map \

and resulting https://github.com/WebKit/JetStream/blob/main/wasm/zlib/build/zlib.js.symbols)?

maraf · 2025-07-10T16:37:10Z

I also briefly looked at the flamegraphs/CPU profiles of the current state of the workloads, but it's hard to dig deeper since we don't have debug info/source names for the Wasm functions.

@maraf Is there some way with Blazor to generate some form of function name map off-to-the-side (similar to -g --emit-symbol-map for Emscripten, see e.g.

Yes, I'll include symbols

kmiller68 · 2025-07-10T20:03:27Z

Looks like this fails in the SM and JSC shells on my machine. Although it seems like they fail for different reasons: SM fails since it's missing setTimeout but I think that's known and they might be adding that? JSC fails with

Starting JetStream3
Running dotnet-aot:
Error: MONO_WASM: undefined is not an object (evaluating 't.indexOf')
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:31566
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32558
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32558
pt@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32564
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32600
bt@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:34882
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39877
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39889
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39889
create@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39933
@
JetStream3 failed: TypeError: undefined is not an object (evaluating 't.indexOf')
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:31566
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32558
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32558
pt@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32564
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:32600
bt@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:34882
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39877
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39889
@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39889
create@/.../JetStream/wasm/dotnet/build-aot/wwwroot/_framework/dotnet.js:3:39933
@

maraf · 2025-07-15T11:37:34Z

We have an unguarded use of import.meta.url in dotnet.js, that's what is failing on jsc. I have removed it from the app build output.

Anyway, for unknown reason running dotnet-aot on jsc sometimes fails with cryptic Aborted; sometimes it finishes. When it aborts, it aborts inside of the runIteration

$ jsc cli.js -- dotnet-aot
Starting JetStream3
Running dotnet-aot:
Aborted

$ jsc cli.js -- dotnet-aot
Starting JetStream3
Running dotnet-aot:
    Startup: 6.264
    Worst Case: 8.665
    Average: 8.930
    Score: 7.855
    Wall time: 0:09.019


First: 6.264
Worst: 8.665
Average: 8.930

Total Score:  7.855

danleh · 2025-07-15T13:23:43Z

Thanks for the changes!

Regarding the jsc failure: IIUC, you manually patched dotnet.js to remove the import.meta.url usage, right? I'd like to avoid manual edit steps after a build, since this is bound to get lost/forgotten when we rebuild at a later point in time. Could we either:
A) @kmiller68 Fix jsc shell to have import.meta.url available?
B) somehow make this automatic during the build, e.g., by simply applying a patch at the end of the build.sh script and submitting that patch to the repo, i.e.,

# rebuild from sources with build.sh, stage the unmodified build output
# manually patch out import.meta.url
git diff build-aot/wwwroot/_framework/dotnet.js > import-meta-url.patch
# commit the patch file and add this line to build.sh:
patch import-meta-url.patch

If this patch becomes too large (because dotnet.js is minified and essentially everything in one line), one could also prepend something like import.meta.url ??= "" to dotnet.js instead of the current edit deep inside that line.

wasm/dotnet/build.sh

maraf · 2025-07-16T13:06:18Z

If this patch becomes too large (because dotnet.js is minified and essentially everything in one line), one could also prepend something like import.meta.url ??= "" to dotnet.js instead of the current edit deep inside that line.

I have added a command to prepend it to dotnet.js to build.sh

danleh

Thanks for the changes, LGTM now :)

Friendly ping @eqrion and @kmiller68 for feedback from the Mozilla/Apple side.

eqrion · 2025-07-17T18:18:49Z

Overall, this looks good to me now. Thanks for the contribution!

@kmiller68

SM fails since it's missing setTimeout

That should be fixed now in the latest SM nightly.

@maraf @danleh

@maraf Is there some way with Blazor to generate some form of function name map off-to-the-side (similar to -g --emit-symbol-map for Emscripten, see e.g.

Yes, I'll include symbols

Where are these symbols being generated? I don't see anything in the wasm name section for dotnet.native.wasm, nor any names in profiles (which would use the name section).

maraf · 2025-07-17T18:55:57Z

Where are these symbols being generated? I don't see anything in the wasm name section for dotnet.native.wasm, nor any names in profiles (which would use the name section).

dotnet.native.js.symbols

eqrion · 2025-07-17T19:18:05Z

Where are these symbols being generated? I don't see anything in the wasm name section for dotnet.native.wasm, nor any names in profiles (which would use the name section).

dotnet.native.js.symbols

Oh interesting. Is this an emscripten or blazor generated file? I've not seen this before, I only really knew of the wasm name section.

maraf · 2025-07-17T19:22:38Z

It is generated by emscripten --emit-symbol-map

https://bugs.webkit.org/show_bug.cgi?id=296132 rdar://156071047 Reviewed by NOBODY (OOPS!). This patch adds import.meta.url to the JSC CLI and API module import.meta objects. This matches the web's behavior and is desired for WebKit/JetStream#73.

kmiller68 · 2025-07-17T20:03:41Z

A) @kmiller68 Fix jsc shell to have import.meta.url available?

WebKit/WebKit#48196

For the future. Although modules are keyed with a URL so this will end up being a file:// URL in the CLI, which seems to differ from V8/SM, but is technically more accurate for the name.

kmiller68 · 2025-07-17T20:38:24Z

Oh interesting. Is this an emscripten or blazor generated file? I've not seen this before, I only really knew of the wasm name section.

It is generated by emscripten --emit-symbol-map

Yeah, AFAIK, none of the wasm tests emit a name section and just emit the symbol map. Since most apps don't ship with the name section we figured it was a bit more realistic to have it on the side, even if a bit inconvenient.

eqrion · 2025-07-17T20:44:51Z

It is generated by emscripten --emit-symbol-map

I wasn't aware that was a thing. TIL

Yeah, AFAIK, none of the wasm tests emit a name section and just emit the symbol map. Since most apps don't ship with the name section we figured it was a bit more realistic to have it on the side, even if a bit inconvenient.

Ah, I had been seeing function names in the Dart and Kotlin subtests. I don't see them in argon2-wasm, and so I'm guessing all the emscripten one's don't have the name section.

kmiller68

Overall LGTM, two main thoughts:

Is there a reason .NET doesn't forward to the browser's JS internationalization and instead provides your own ICU data?
If we wanted to fiddle numbers around to e.g. add more iterations or reduce the total runtime of these line items, which numbers are the most/least important for the validity of the benchmark overall?

https://bugs.webkit.org/show_bug.cgi?id=296132 rdar://156071047 Reviewed by Yusuke Suzuki. This patch adds import.meta.url to the JSC CLI and API module import.meta objects. This matches the web's behavior and is desired for WebKit/JetStream#73. Canonical link: https://commits.webkit.org/297565@main

maraf · 2025-07-19T16:53:31Z

Is there a reason .NET doesn't forward to the browser's JS internationalization and instead provides your own ICU data?

It's simple as JavaScript API doesn't cover all the APIs .NET have. For some time we have been experimenting with hybrid approach, but the download size savings weren't worth it. .NET is using ICU for other platforms and so reusing it for wasm was easy.

If we wanted to fiddle numbers around to e.g. add more iterations or reduce the total runtime of these line items, which numbers are the most/least important for the validity of the benchmark overall?

For raytracer part the most important is the size of scene, controlled from benchmark.js
For the rest of the tasks, there is a BatchSize * InitialSamples overridable by each "measurement.

maraf · 2025-07-24T17:28:58Z

Thank you all for feedback!

Implement .NET benchmarks

dd055ee

Add JSON

c11b328

maraf marked this pull request as ready for review May 28, 2025 13:36

camillobruni reviewed May 28, 2025

View reviewed changes

JetStreamDriver.js Outdated Show resolved Hide resolved

maraf added 2 commits May 29, 2025 14:56

Deduplicate preloads

ba05a96

Add JSON to in-depth.html

8cba497

maraf requested a review from camillobruni May 30, 2025 08:22

maraf added 3 commits June 18, 2025 14:21

Combine all tests

7735bfe

Update in-depth.html

0f265ab

Reduce iterations

ba35bc3

maraf added 2 commits July 9, 2025 15:03

Further iteration reduction

1f9bc2b

Make it work on cli without preloading

b719e8f

maraf added 2 commits July 9, 2025 17:16

Update to 9.0.7

8ce2c23

Smaller scene size for interp

5623e4c

danleh requested changes Jul 10, 2025

View reviewed changes

maraf added 2 commits July 11, 2025 14:36

Merge remote-tracking branch 'upstream/main' into DotnetWasm

a22b110

Use getBinary

79df4d0

maraf added 2 commits July 15, 2025 13:47

.NET SDK info

ded3bbd

README

b0353a6

danleh reviewed Jul 15, 2025

View reviewed changes

wasm/dotnet/build.sh Show resolved Hide resolved

danleh reviewed Jul 15, 2025

View reviewed changes

wasm/dotnet/build.sh Outdated Show resolved Hide resolved

maraf added 5 commits July 15, 2025 20:33

Description of .NET build output

37e1621

chmod +x build.sh

a1d0702

Exclude additional files from git

7a39202

Prepend import.meta.url and copy symbols from obj

295ae1a

Readme

5aa9dad

maraf requested a review from danleh July 16, 2025 13:06

danleh approved these changes Jul 16, 2025

View reviewed changes

kmiller68 approved these changes Jul 17, 2025

View reviewed changes

webkit-commit-queue mentioned this pull request Jul 18, 2025

add support for import.meta.url to jsc CLI WebKit/WebKit#48196

Merged

kmiller68 merged commit e30a452 into WebKit:main Jul 23, 2025
4 checks passed

maraf deleted the DotnetWasm branch July 23, 2025 17:53

Add .NET on WebAssembly workload #73

Add .NET on WebAssembly workload #73

Uh oh!

Conversation

maraf commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Flavors

1. Interpreter

2. AOT

Code samples

RayTracer

Exception handling

String

JSON serialization & deserialization

Uh oh!

netlify bot commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for webkit-jetstream-preview ready!

Uh oh!

Uh oh!

eqrion commented May 30, 2025

Uh oh!

maraf commented Jun 2, 2025

Uh oh!

maraf commented Jun 18, 2025

Uh oh!

issackjohn commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danleh commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danleh commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maraf commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maraf commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

issackjohn commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danleh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danleh commented Jul 10, 2025

Uh oh!

maraf commented Jul 10, 2025

Uh oh!

kmiller68 commented Jul 10, 2025

Uh oh!

maraf commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danleh commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

maraf commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danleh left a comment

Choose a reason for hiding this comment

Uh oh!

eqrion commented Jul 17, 2025

Uh oh!

maraf commented Jul 17, 2025

maraf commented May 23, 2025 •

edited

Loading

netlify bot commented May 23, 2025 •

edited

Loading

issackjohn commented Jun 25, 2025 •

edited

Loading

danleh commented Jul 8, 2025 •

edited

Loading

danleh commented Jul 8, 2025 •

edited

Loading

maraf commented Jul 9, 2025 •

edited

Loading

maraf commented Jul 9, 2025 •

edited

Loading

issackjohn commented Jul 9, 2025 •

edited

Loading

danleh left a comment •

edited

Loading

maraf commented Jul 15, 2025 •

edited

Loading

maraf commented Jul 16, 2025 •

edited

Loading

maraf commented Jul 17, 2025 •

edited

Loading

kmiller68 commented Jul 17, 2025 •

edited

Loading

kmiller68 commented Jul 17, 2025 •

edited

Loading