Skip to content

[Backend Tester] Add ARM TOSA flow #14190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: gh/GregoryComer/159/head
Choose a base branch
from
Open

[Backend Tester] Add ARM TOSA flow #14190

wants to merge 8 commits into from
+88 −5

Conversation

GregoryComer
Copy link
Member

@GregoryComer GregoryComer commented Sep 11, 2025 •
edited
Loading

Wire up the Arm tester with the backend test suites. This is an initial implementation, which is missing a few things. Most notably, output accuracy stats aren't available. I intend to take this as a follow-up.

@GregoryComer
Update
5a156c9 
[ghstack-poisoned]
@GregoryComer
Copy link
Member Author

GregoryComer commented Sep 11, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Sep 11, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14190

Note: Links to docs will display an error until the docs builds have been completed.

❌ 86 Cancelled Jobs, 3 Unrelated Failures

As of commit 48ee73d with merge base ed0ab94 (image):

CANCELLED JOBS - The following jobs were cancelled. Please retry:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 11, 2025
GregoryComer added a commit that referenced this pull request Sep 11, 2025
@GregoryComer
[Backend Tester] Add ARM TOSA flow
7d14a4e 
ghstack-source-id: 599e2da
ghstack-comment-id: 3277010040
Pull-Request: #14190
This was referenced Sep 11, 2025
Open
Open
Open
Open
Open
Open
@GregoryComer
Update
4e35585 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Sep 11, 2025
@GregoryComer
[Backend Tester] Add ARM TOSA flow
3baf2c0 
ghstack-source-id: 7540d11
ghstack-comment-id: 3277010040
Pull-Request: #14190
@GregoryComer
Update
cfadc8b 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Sep 11, 2025
@GregoryComer
[Backend Tester] Add ARM TOSA flow
a472bcb 
ghstack-source-id: d61a7ea
ghstack-comment-id: 3277010040
Pull-Request: #14190
@GregoryComer GregoryComer marked this pull request as ready for review September 11, 2025 05:26
@GregoryComer
Update
d8bde7d 
[ghstack-poisoned]
GregoryComer added a commit that referenced this pull request Sep 11, 2025
@GregoryComer
[Backend Tester] Add ARM TOSA flow
766aebc 
ghstack-source-id: 804999d
ghstack-comment-id: 3277010040
Pull-Request: #14190
GregoryComer
GregoryComer commented Sep 11, 2025
View reviewed changes
backends/test/suite/runner.py
extra_stats["pte_size_bytes"] = len(tester.get_artifact())
tester.to_executorch()

if flow.supports_serialize:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARM tester requires a compile spec giving a location to dump the artifact. I'm just skipping this for initial integration, but we can wire this up as a follow-up. I'm not sure how meaningful the binary size for the TOSA flow, as well. Is that something we should report?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah we can ignore binary size for TOSA-only flows

@GregoryComer
Update
3cd1021 
[ghstack-poisoned]
@GregoryComer GregoryComer mentioned this pull request Sep 11, 2025
Open
@GregoryComer
Update
1233680 
[ghstack-poisoned]
@zingo zingo added the module: arm Issues related to arm backend label Sep 11, 2025
@GregoryComer
Update
8f42f77 
[ghstack-poisoned]
digantdesai
digantdesai reviewed Sep 12, 2025
View reviewed changes
backends/test/suite/flows/arm.py

def _create_tosa_flow() -> TestFlow:
return TestFlow(
"arm_tosa",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"arm_tosa",
"arm_tosa_fp",

@digantdesai
Copy link
Contributor

cc @zingo , @per as FYI

@zingo
Copy link
Collaborator

zingo commented Sep 12, 2025

This will be nice to see and play with!
It might be good to look into Ethos-U/VGF FLOWs down the line and this takes us quite long way to fullfill that, thanks!

@GregoryComer
Update
48ee73d 
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai approved these changes

@cccclai cccclai Awaiting requested review from cccclai cccclai is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GregoryComer @digantdesai @zingo