Skip to content

Conversation

@caetanosauer
Copy link
Contributor

@caetanosauer caetanosauer commented Nov 11, 2025

A recent commit aimed at "Prevent cheating by Munich-based databases" (4a2f3fa) incorrectly modified Hyper's query.py scripts to restart the database server on each iteration, essentially making every run a cold run. This unfairly brought our results down by a large margin.

This commit reverts the previous behavior which was already correct and there was absolutely no cheating involved.

A recent commit aimed at "Prevent cheating by Munich-based databases"
(4a2f3fa) incorrectly modified Hyper's `query.py` scripts to restart the
database server on each iteration, essentially making every run a cold
run. This unfairly brought our results down by a large margin.

This commit reverts the previous behavior which was already correct and
there was absolutely no cheating involved.
@rschu1ze rschu1ze self-assigned this Nov 13, 2025
@rschu1ze
Copy link
Member

rschu1ze commented Nov 13, 2025

@caetanosauer There was some discussion about what "cold" runs really mean in #667. I pushed #692 in response. Tl;dr is that today's Hyper submission is fine - now we'll only need to migrate all other submissions one-by-one from lukewarm to cold runs for fair comparison. Allow me to close this PR - thanks.

PS: I removed "cheating" from the message of #659. This term was unnecessarily dramatic.

@rschu1ze rschu1ze closed this Nov 13, 2025
@caetanosauer
Copy link
Contributor Author

@rschu1ze Thanks for your response and the update on the lukewarm definitions.

The problem I mentioned in this PR still persists, though: There are no hot runs being measured for Hyper. Last time I measured, Hyper was second place behind Umbra in the "Hot Run" category. Now we are way down below, because all 3 repetitions are run on a freshly restarted Hyper.

@rschu1ze
Copy link
Member

rschu1ze commented Nov 16, 2025

@caetanosauer I see, let's re-open the PR.

However, I like to clarify with you if the first run is cold or lukewarm (see the PR / issue I linked above).

In

with HyperProcess(telemetry=Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU) as hyper:
    with Connection(hyper.endpoint, 'hits.hyper', CreateMode.NONE) as connection:
        for _ in range(3):
            start = timeit.default_timer()
            try:
                connection.execute_list_query(query)

it is not clear to me if the Hyper process survives between queries (or to be more precise: survives after three runs of a query).

We should physically terminate the database to ensure true cold runs.

@rschu1ze rschu1ze reopened this Nov 16, 2025
@caetanosauer
Copy link
Contributor Author

The with HyperProcess guard starts a new process and terminates it at the end. Since the script query.py is invoked for each query from run.sh after clearing the filesystem cache, the first iteration of each query will be truly cold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants