Skip to content

Fix SingleStream and MultiStream reciprocal score comparison for TEST04 #1491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Fix SingleStream and MultiStream reciprocal score comparison for TEST04 #1491

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0c96f8
Fix SingleStream and MultiStream reciprocal score comparison for TEST04.
sahelib25 Sep 11, 2023
b40a42c
Merge branch 'master' into fix_test04
sahelib25 Sep 11, 2023
2a0aeb2
Merge branch 'master' into fix_test04
sahelib25 Dec 9, 2024
7694c10
[Automated Commit] Format Codebase
mlcommons-bot Dec 9, 2024
c92fc10
Merge branch 'master' into fix_test04
sahelib25 Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions compliance/nvidia/TEST04/verify_performance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,11 @@ def main():
if ref_mode == "SingleStream":
if re.match(".*Early stopping 90th percentile estimate", line):
ref_score = line.split(": ",1)[1].strip()
ref_score = 1e9 / float(ref_score)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change can potentially break the below check as the check assumes throughput metric for all the scenarios.
https://github.com/mlcommons/inference/pull/1491/files#diff-5c101fd75c9062a7dec72722d5f4aafe66e7d55fbc32f97378b74c08034c272cR141

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially, but why a similar approach (of not using reciprocals) does not cause any issue for TEST01 and TEST05?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TEST01 is doing comparison like this - both upper and lower bounds are checked and so we don't need reciprocal but the check is stricter than required.

TEST05 also had a similar check but was removed in a PR before 3.1 and so is currently broken for the offline scenario. This PR should fix that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the TEST05-related PR has been merged, should we do TEST04 in the same way?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks fine to me. But I don't know the full reasoning behind taking this reciprocal. @nv-ananjappa any suggestions here?

continue

if ref_mode == "MultiStream":
if re.match(".*Early stopping 99th percentile estimate", line):
ref_score = line.split(": ",1)[1].strip()
ref_score = 1e9 / float(ref_score)
continue

if ref_mode == "Server":
Expand Down Expand Up @@ -92,13 +90,11 @@ def main():
if test_mode == "SingleStream":
if re.match(".*Early stopping 90th percentile estimate", line):
test_score = line.split(": ",1)[1].strip()
test_score = 1e9 / float(test_score)
continue

if test_mode == "MultiStream":
if re.match(".*Early stopping 99th percentile estimate", line):
test_score = line.split(": ",1)[1].strip()
test_score = 1e9 / float(test_score)
continue

if test_mode == "Server":
Expand Down