PVF: consider spawning a new process per job

[mrcnski]

Currently we spawn worker processes for PVF jobs (prepare/execute) and each incoming job gets its own thread within that process.

This works fine but there are potential security issues with this as described in paritytech/polkadot#7580 (comment). Namely, we can't fully sandbox the process because we have to have an allow-exception for the entire PVF artifact cache directory.

We should investigate the overhead of spawning a whole separate process for each job as opposed to a thread. It may be less overhead now that we are spawning processes from smaller worker binaries instead of polkadot. If the cost is low enough, we can switch to one-process-per-job which allows us to sandbox the process better.

This would probably also come with changes to the execution queue logic (most likely simplifications).

[mrcnski]

I'm going to first try sending the artifact bytes over the socket, avoiding the need for any sandbox allowances at all. I was under the impression that it would take too long but it's worth measuring.

[burdges]

In principle SPREEs could be used by many paracahins, so then we'd benefit from the final artifact, presumably an ELF .so file, being dlopened. Not a concern right now.

[rphmeier]

Numbers on PVF overhead (per job, per byte of data in PoV) would be really helpful to track as they'd inform higher-level protocol things like bundling.

i.e. the lower we can get these overheads, the smaller we can make candidates, and probably the higher utilization we can get out of approval cores.

[mrcnski]

I'm going to first try sending the artifact bytes over the socket, avoiding the need for any sandbox allowances at all. I was under the impression that it would take too long but it's worth measuring.

Just measured on local zombienet. The result is what I expected, but worth a try I guess.

Artifact of size 233768 bytes took 7364ms to transfer

Next I'll benchmark a new process per job.

[rphmeier]

Artifact of size 233768 bytes took 7364ms to transfer

Am I reading this right? Is this meant to be microseconds or milliseconds?

[mrcnski]

Artifact of size 233768 bytes took 7364ms to transfer

Am I reading this right? Is this meant to be microseconds or milliseconds?

Duration::as_millis, so it's indeed pretty bad. I also saw bad numbers in my internet research but wanted to quickly confirm myself.

Shared memory is supposed to be much faster, but it would require a bigger rewrite to test it, and I fear it wouldn't fit well with Rust's ownership scheme and require unsafe. Also, the executor already has optimizations around working with file data efficiently, which we would lose if we read the whole file in the host and wrote it to shared memory.

Maybe worth doing a quick PoC or extending the benchmarks we have, but first I'll try one-process-per-job as it's much simpler to test.

[mrcnski]

Locally on M1 Mac the spawning takes 4-12ms. That's already how long execution by itself takes. When I tried on a beefy Linux machine it was worse, at 20ms. In the grand scheme of things, maybe it's acceptable? It still comfortably fits within the backing timeout. cc @eskimor @s0me0ne-unkn0wn

I tried the socket measurements again, this time on Linux, since my numbers on Mac were even worse than stack overflow's socket / shared memory numbers. On Linux I got 645us (Duration::as_micros) for 233768 bytes. Extrapolating for a 20mb artifact (which is pretty small) it would be 60ms, so still not too good.

[s0me0ne-unkn0wn]

When I tried on a beefy Linux machine it was worse, at 20ms.

Have you tried spawning from memory yet? It could somewhat reduce the overhead.

On Linux I got 645us (Duration::as_micros) for 233768 bytes. Extrapolating for a 20mb artifact (which is pretty small) it would be 60ms, so still not too good.

The question is, how bad is that compared to reading the artifact from the fs? For the case if we read them only once (or never read them at all, preparing and keeping everything in memory)

[rphmeier]

The question is, how bad is that compared to reading the artifact from the fs? For the case if we read them only once (or never read them at all, preparing and keeping everything in memory)

We probably can't keep all artifacts in memory while we scale to tens of thousands of registered parachains. We can give heads-up notifications to pre-load specific artifacts when we assign to a parachain in a approval-checking, but it'd probably only give a few hundred ms of lead time (probably enough).

These are the numbers we're interested in:

• Overhead of spawning a process
• Overhead of starting the execution job (including getting the artifact)
• Overhead of passing in the PoV

Together it's our total overhead per validation job. Ideally we can get this as lean as possible on linux/1st-class platforms for validators. (relates to #607)

[eskimor]

20MB smallish ...? @s0me0ne-unkn0wn is this something we can easily bring down in the future?

I tried the socket measurements again, this time on Linux, since my numbers on Mac were even worse than stack overflow's socket / shared memory numbers. On Linux I got 645us (Duration::as_micros) for 233768 bytes. Extrapolating for a 20mb artifact (which is pretty small) it would be 60ms, so still not too good.

Hmm, shouldn't it be 6ms?

[s0me0ne-unkn0wn]

20MB smallish ...? @s0me0ne-unkn0wn is this something we can easily bring down in the future?

Checked with Moonbeam runtime (7.7 Mb uncompressed Wasm bytecode). Wasmtime produced a 28 Mb artifact. It's an unstripped ELF, .text segment is ~13.5 Mb, and a lot of other things used by Wasmtime runtime which are not obvious to me (e.g., ~10 Mb section called wasmtime.addrmap).

If I strip it, it doesn't make much difference, ~300 Kb are gone, but that's it.

My compiler produces 22.7 Mb artifact from the same bytecode, so the value is fair enough.

Just for comparison, the adder runtime, 126 Kb Wasm bytecode, produces 149 Kb unstripped (143 Kb stripped) artifact with Wasmtime. My compiler produces 102 Kb artifact.