Fix recipe logic to propagate quantized graph in the pipeline (fixes #12659) #12661
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12661
Note: Links to docs will display an error until the docs builds have been completed. ❌ 5 New FailuresAs of commit e35fd86 with merge base bc0ad82 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D78585588 |
This PR needs a
|
| @@ -74,7 +74,7 @@ def test_int8_dynamic_quant_recipe(self) -> None: | |||
| torch.allclose( | |||
| session.run_method("forward", example_inputs[0])[0], | |||
| m_eager(*example_inputs[0]), | |||
| atol=1e-3, | |||
| atol=1e-1, | |||
There was a problem hiding this comment.
Typically I would set rtol along with atol, ∣input−other∣≤atol+rtol×∣other∣
There was a problem hiding this comment.
Also do we need it to be this high for two linears?
There was a problem hiding this comment.
Yes, it is consistent with my offline run without the recipes, i had to choose high tolerance. Can share more offline.
| @@ -189,6 +189,7 @@ def _test_model_with_factory(self, model_name: str) -> None: | |||
| atol=1e-3, | |||
| ) | |||
|
|
|||
| @unittest.skip("T187799178: Debugging Numerical Issues with Calibration") | |||
There was a problem hiding this comment.
Nit: we should make a GH Issue perhaps?
…ytorch#12659) (pytorch#12661) Summary: I've found couple of issues with the original export recipes logic has incomplete functionality: 1. The output of quantize stage is not getting propagated to next stages 2. When quantize stage is run, we should re-export the model before we lower to edge. This diff adds support for both. After this change the quantization flow revealed few gaps with xnnpack quantization and after which i've disable few tests due to the accuracy issues and an issue with dynamic per tensor quantization. Changes: 1. Adds support for above gaps 2. This gap could've avoided with few unittests and this adds comprehensive tests for export recipe pipeline and stages 3. Includes tests in pytest for oss to run (fixes pytorch#12659) Differential Revision: D78585588
abhinaykukkadapu
force-pushed
the
export-D78585588
branch
from
89cc549 to
e35fd86
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78585588 |
Preventing the test added in #12661 from running since it's broken.
…ytorch#12659) (pytorch#12661) Summary: I've found couple of issues with the original export recipes logic has incomplete functionality: 1. The output of quantize stage is not getting propagated to next stages 2. When quantize stage is run, we should re-export the model before we lower to edge. This diff adds support for both. After this change the quantization flow revealed few gaps with xnnpack quantization and after which i've disable few tests due to the accuracy issues and an issue with dynamic per tensor quantization. Changes: 1. Adds support for above gaps 2. This gap could've avoided with few unittests and this ads comprehensive tests for export recipe pipeline and stages 3. Includes tests in pytest for oss to run (fixes pytorch#12659) Rollback Plan: Differential Revision: D78585588
Summary:
I've found couple of issues with the original export recipes logic has incomplete functionality:
This diff adds support for both. After this change the quantization flow revealed few gaps with xnnpack quantization and after which i've disable few tests due to the accuracy issues and an issue with dynamic per tensor quantization.
Changes:
(fixes #12659)
Rollback Plan:
Differential Revision: D78585588