Skip to content

[Backend Tester] Add CSV report generation #12741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: gh/GregoryComer/87/head
Choose a base branch
from
Open

[Backend Tester] Add CSV report generation #12741

wants to merge 9 commits into from
+198 −15

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Jul 23, 2025 •
edited
Loading

Add some initial CSV report generation, detailing results and parameters for each individual test. Delegation statistics and such will come next. I've also added a basic test for the report generation, which I will expand upon in this stack.

Here's some sample output from running add tests for XNNPACK:

Test ID,Test Case,Backend,Flow,Result,Dtype
test_add_dtype_float32_xnnpack,test_add_dtype,xnnpack,xnnpack,Success (Delegated),torch.float32
test_add_dtype_float32_xnnpack_static_int8,test_add_dtype,xnnpack,xnnpack_static_int8,Success (Delegated),torch.float32
test_add_f32_alpha_xnnpack,test_add_f32_alpha,xnnpack,xnnpack,Fail (Quantize),
test_add_f32_alpha_xnnpack_static_int8,test_add_f32_alpha,xnnpack,xnnpack_static_int8,Fail (Quantize),
test_add_f32_bcast_first_xnnpack,test_add_f32_bcast_first,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_first_xnnpack_static_int8,test_add_f32_bcast_first,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_second_xnnpack,test_add_f32_bcast_second,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_second_xnnpack_static_int8,test_add_f32_bcast_second,xnnpack,xnnpack_static_int8,Success (Delegated),
test_add_f32_bcast_unary_xnnpack,test_add_f32_bcast_unary,xnnpack,xnnpack,Success (Delegated),
test_add_f32_bcast_unary_xnnpack_static_int8,test_add_f32_bcast_unary,xnnpack,xnnpack_static_int8,Success (Delegated),

@GregoryComer
Update
19760fc 
[ghstack-poisoned]
@GregoryComer
Copy link
Member Author

GregoryComer commented Jul 23, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

@GregoryComer GregoryComer requested a review from cccclai as a code owner July 23, 2025 01:28
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 23, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12741

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5cc4941 with merge base 4fd2079 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 23, 2025
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
391b923 
ghstack-source-id: ec13121
ghstack-comment-id: 3105325555
Pull-Request: #12741
This was referenced Jul 23, 2025
Merged
Closed
Merged
Merged
Merged
Open
@GregoryComer GregoryComer requested a review from digantdesai July 23, 2025 01:30
digantdesai
digantdesai reviewed Jul 23, 2025
View reviewed changes
backends/test/suite/tests/test_reporting.py Outdated
@@ -0,0 +1,101 @@
import torch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

digantdesai
digantdesai reviewed Jul 23, 2025
View reviewed changes
backends/test/suite/tests/test_reporting.py
]

class Reporting(unittest.TestCase):
def test_csv_report_simple(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: tsv works better usually

@GregoryComer
Update
5d437b1 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
60fc879 
ghstack-source-id: f67c54b
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
e2ea2a3 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
af471be 
ghstack-source-id: 058e3ca
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
3976629 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
21208f3 
ghstack-source-id: 8dd5a49
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
cdd15c1 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
625c942 
ghstack-source-id: 4f76de1
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
bcb697c 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Jul 23, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
42f383f 
ghstack-source-id: 17e6147
ghstack-comment-id: 3105325555
Pull-Request: #12741
@GregoryComer
Update
de21ac2 
[ghstack-poisoned]
@GregoryComer GregoryComer mentioned this pull request Jul 24, 2025
Open
@GregoryComer
Update
32f54b0 
[ghstack-poisoned]
@GregoryComer
Update
5cc4941 
[ghstack-poisoned]
This was referenced Jul 24, 2025
Open
Draft
GregoryComer added a commit to GregoryComer/executorch that referenced this pull request Jul 31, 2025
@GregoryComer
[Backend Tester] Add CSV report generation
a006ab2 
ghstack-source-id: 6264a78
ghstack-comment-id: 3105325555
Pull-Request: pytorch#12741
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai approved these changes

@cccclai cccclai Awaiting requested review from cccclai cccclai is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GregoryComer @digantdesai @facebook-github-bot