Continuous Benchmarking

[sebffischer]

In his talk about data.table @tdhock showed us how they use the atime R package to do continuous benchmarks using GitHub actions.
The files that define these benchmarks are here: https://github.com/Rdatatable/data.table/blob/master/.ci/atime/tests.R and an example how the results look is here: Rdatatable/data.table#6993 (comment)
It would be nice if we could do something similar for mlr3.

One issue is that our benchmarks would probably take considerably more time than the data.table ones.

[tdhock]

If you tell me what kind of operation would be important to benchmark, I could give you advice on writing the atime could which could hopefully let you see regressions.
It would be most helpful if you have an example from your project history of something that was slower than expected/desired, and we could use that as a positive control to see if the regression is visible on the atime plot.
Here is an example of atime on mlr3pipelines: https://tdhock.github.io/blog/2025/linear-vs-quadratic-pipe-ops/

[tdhock]

Doris_Amoakohene__atime_for_asymptotic_Timings.pdf

Here is a PDF pre-print of our paper about atime which we submitted for peer review at R Journal.
If you have time, please read and suggest improvements.

[sebffischer]

There is also: https://github.com/lorenzwalthert/touchstone

[tdhock]

There is a comparison between touchstone and atime in our paper:

Features of atime and touchstone for performance testing

The most similar R package to atime is touchstone, which also provides functions for performance
testing, but without asymptotic measurement (for variable data size N).
Both touchstone and atime use relative performance testing, meaning that each test case is run
using different versions of the R package. Touchstone allows specifying two branches, corresponding
to the HEAD of a PR, and its base branch. In addition to those branches, atime supports other branches
that are important in the context of a PR (merge-base and CRAN), as well as user-defined commits
that represent historical Fast/Slow versions.