[Backend Tester] Add quantized test flows for XNNPACK and Core ML #12733
Conversation
GregoryComer
requested review from
larryliu0820,
kirklandsign,
JacobSzwejbka,
lucylq,
swolchok,
shoumikhin and
cccclai
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12733
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 77 PendingAs of commit 27cd171 with merge base 29c52b5 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
|
||
| import coremltools as ct |
There was a problem hiding this comment.
come on :p
| import coremltools as ct | |
| import coremltools |
| def _get_static_int8_qconfig(): | ||
| return ct.optimize.torch.quantization.LinearQuantizerConfig( | ||
| global_config=ct.optimize.torch.quantization.ModuleLinearQuantizerConfig( | ||
| quantization_scheme="symmetric", |
There was a problem hiding this comment.
is this the main int8 schema we should be testing for Linear @metascroy
There was a problem hiding this comment.
FYI This is pulled directly from our docs at https://docs.pytorch.org/executorch/main/backends-coreml.html#bit-quantization-using-the-pt2e-flow. Would be good to sanity check with Scott, though.
backends/test/suite/flow.py
Outdated
| logger.info("Skipping XNNPACK flow registration due to import failure.") | ||
| return None | ||
|
|
||
| from executorch.backends.test.suite.flows.xnnpack import XNNPACK_TEST_FLOW, XNNPACK_STATIC_INT8_TEST_FLOW |
There was a problem hiding this comment.
I'm going to clean up flow registration slightly in the stack, so I'll take this as a follow-up there.
backends/test/suite/flow.py
Outdated
| quantize: bool = field(default=False) | ||
| """ Whether to tester should run the quantize stage on the model. """ | ||
|
|
||
| quantize_stage_factory: Callable[..., Quantize] | None = None | ||
| """ A factory function which instantiates a Quantize stage. Can be None to use the tester's default. """ |
There was a problem hiding this comment.
The specific reason is that if quantize_stage_factory isn't provided, it will use the default Quantize stage from the tester. I could maybe just always require the caller to provide quantize_stage_factory.
| @@ -14,23 +14,26 @@ class TestResult(IntEnum): | |||
|
|
|||
| EAGER_FAIL = 2 | |||
| """ The test failed due to the model failing to run in eager mode. """ | |||
|
|
|||
| QUANTIZE_FAIL = 3 | |||
There was a problem hiding this comment.
Do we want to distinguish export_for_trainint (old) before quantize and export after quantize?
There was a problem hiding this comment.
If I'm following correctly, are you asking about catching errors in the quant export vs the quantization itself?
It would be nice to have this, though we'd need to refactor the tester a bit to split it.
There was a problem hiding this comment.
yeah finer grained. We can do this later, hence the stamp :)
|
Closes #12494 |
GregoryComer
requested review from
jackzhxng,
metascroy and
mergennachin
as code owners
|
Updated to address comments and lints. There are a few minor follow-ups (see specific comment chains), but I'm intending to land and take follows-up after to keep things moving. |
|
Unit test failures are pre-existing on main (pytest coverage error). CI was green before rebasing. Merging. |
…torch#12733) Add quantized test flows core static int8 quantization for XNNPACK and Core ML backends. I also ended up doing some light refactoring on the test signature to pass the TestFlow class into the individual tests. This is done to allow for passing quantization parameters into the inner test.
Add quantized test flows core static int8 quantization for XNNPACK and Core ML backends. I also ended up doing some light refactoring on the test signature to pass the TestFlow class into the individual tests. This is done to allow for passing quantization parameters into the inner test.