k-server-bench keeps both a released non-legacy evaluator and a released legacy evaluator because they serve different candidate interfaces.
Relevant code lives under:
In general, prefer the non-legacy evaluator.
The main reason is that it is less restrictive and operationally more robust. The legacy path depends on Ray-backed evaluation patterns, and in practice both "one shared Ray cluster for parallel evaluations" and "multiple Ray clusters per run" tend to be fragile. The non-legacy path avoids baking that orchestration into the benchmark contract and instead puts the responsibility for parallelization inside the candidate search procedure.
Historical note: the k=3 experiments reported in the paper were run with the legacy evaluator. That is useful for reproducing the paper setup, but it should not be read as a recommendation for new experiments.
The non-legacy evaluator expects a candidate module with a Potential class and usually a main(args) search procedure.
The typical flow is:
- run the candidate program as a subprocess
- read the candidate JSON payload
- extract
potential_kwargs - instantiate
Potential(context, **potential_kwargs) - score the resulting potential on the requested metrics
This model is simpler when you want the search procedure and the final potential definition in one file.
It is also the recommended model for new work:
- it exposes fewer benchmark-imposed structural constraints
- it leaves parallelization strategy to the agent or candidate implementation
- it avoids the more brittle Ray-cluster management assumptions used by the legacy path
The legacy evaluator preserves the older three-component split:
PotentialFamilyPotentialSearchEvaluator
That interface is more structured but also more cumbersome. It remains useful for reproducing older experiment styles and for methods already written against the legacy contract.
Use it mainly for backward compatibility and reproduction, not as the default choice for new experiments.
Regardless of evaluator family, candidate code is expected to:
- stay within the provided time and CPU budget
- return JSON-serializable outputs
- preserve a valid best-so-far result near interruption boundaries
- avoid modifying the benchmark infrastructure itself
The benchmark is designed so the evaluator owns the final scoring logic. Candidate code is responsible for proposing the object to score, not redefining the scoring rule.
Both evaluator families also expose MCP server wrappers for agent-driven workflows. Those wrappers live next to the evaluator entrypoints and are documented further under agents/docs/.