Introduce Ractor benchmarks and harness

[luke-gruber]

This PR introduces a set of benchmarks for assessing the performance of Ractors. The main parts of this work are

1. Introduce a set of Ractor specific performance benchmarks

A new harness harness-ractor.rb has been introduced. This harness will run the benchmark code inside a number of Ractors concurrently and collect timing information. By default this is 0 (ractors disabled), 1, 2, 4 and 8.

A number of Ractor specific benchmarks designed to test specific scenarios of multi-ractor mode have been introduced into the benchmarks-ractor directory. They've also been included in benchmarks.yml with the name prefix ractor/.

In order to run these benchmarks with the new harness a new category ractor-only has been introduced. The ractor only benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor-only

2. Adapt existing benchmarks to be Ractor compatible

Existing benchmarks in the benchmarks directory have been updated (where it's possible to do so in a simple manner) to be multi-Ractor safe (mostly making sure any common state is made shareable using Ractor.make_shareable). Benchmarks that have been updated in this way have been marked inside benchmarks.yml with ractor: true.

These tests can be run with either the normal benchmark harness or the Ractor harness, they should be expected to work with both.

A new category ractor has been introduced that will run all of these tests with the Ractor harness alongside the Ractor only tests mentioned above.

All Ractor benchmarks can be run as follows:

./run_benchmarks.rb --category=ractor

Any benchmark can be run with the Ractor harness using the run_once.sh script in conjunction with the HARNESS environment variable

HARNESS=harness-ractor ./run_once.sh benchmarks/activerecord/benchmark.rb

Or a category can be run using the --harness argument to the main benchmark runner

./run_benchmarks.rb --harness=harness-ractor --category=headline

But be aware that running like this will ignore the ractor key in benchmarks.yml and will run all the chosen benchmarks with the ractor harness, regardless of compatibility. Crashes are likely.

3. Harness chain.

For convenience and as an aid to profiling with ractors, this PR also implements the ability to chain harnesses together. For example, running the ractor-only benchmarks through the Vernier profiler can be done as follows:

HARNESS_CHAIN="vernier,ractor" ./run_benchmarks.rb --harness=harness-chain --category=ractor-only

[tenderlove]

This looks good to me, but I feel I'm missing something. Are the bechmarks just copied fully in to benchmarks-ractor and then made to be Ractor compatible? Also it seems like there were a bunch of images copied, should they have been?

[luke-gruber]

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel. I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

[tenderlove]

Some benchmark directories were copied into benchmarks-ractor, then made to work with ractors using the ractor harness. Others required just a few small changes so were not copied, but were made to work with the ractor harness. The ractor harness takes care of creating the ractors and running the run_benchmark block in parallel.

Got it, makes sense.

I believe the images are all from copying optcarrot into benchmarks-ractor. We can remove them if you want.

If they're not buying us anything, I think we should remove them. They look like possible benchmark results? I'd hate for people to be confused by them.

[eregon]

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

That seems like a nice way to make OptCarrot Ractor-compatible without copying many files.
That specific file could be modified to not run the benchmark (e.g. if $0 != __FILE__) and required by benchmarks/optcarrot/benchmark.rb if using the Ractor harness (like for other benchmarks which were not copied). Or e.g. just copying the .freeze calls in that file.

I think it's best to avoid large copies in the repo, and benchmarks-ractor/optcarrot seems the only offender in this PR.

[k0kubun]

Looks good to me.

Make 30k_methods benchmark Ractor compatible

What was the change? The diff is so large that GitHub doesn't show the diff.

Regarding OptCarrot, have you seen https://github.com/mame/optcarrot/blob/master/bin/optcarrot-bench-parallel-on-ractor ?

If this is possible, I guess we could let benchmarks-ractor/optcarrot/benchmark.rb load optcarrot from ../../benchmarks/optcarrot, use shims like optcarrot-bench-parallel-on-ractor, and run the benchmark.