Switch ANE llama model to use to_edge_transform_and_lower + torchao quantization APIs #12665
Conversation
metascroy
requested review from
JacobSzwejbka,
larryliu0820,
cccclai and
shoumikhin
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12665
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit 505f023 with merge base 727937e ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
This PR needs a
|
metascroy
force-pushed
the
switch-ane-model-to-edge-transform-lower
branch
from
38c31c1 to
be95023
Compare
| @@ -28,6 +28,6 @@ pushd $EXECUTORCH_ROOT/examples/apple/coreml/llama | |||
| # Download stories llama110m artifacts | |||
| download_stories_model_artifacts | |||
|
|
|||
| python export.py -n model.pte -p params.json -c stories110M.pt --seq_length 32 --max_seq_length 64 --dtype fp16 --coreml-quantize c4w | |||
| python export.py -n model.pte -p params.json -c stories110M.pt --seq_length 32 --max_seq_length 64 --dtype fp16 --coreml-quantize c4w --embedding-quantize 4,32 | |||
There was a problem hiding this comment.
Is the pr for supporting embedding quantization?
There was a problem hiding this comment.
The PR just changes to using to_edge_transform_and_lower and torchao APIs for quantize_. It does not add embedding quant support.
Embedding quant support existed, but wasn't being tested before, so I also enabled it in the CI test.
|
|
||
| from executorch.exir.backend.utils import format_delegated_graph | ||
| from executorch.exir.capture._config import EdgeCompileConfig, ExecutorchBackendConfig | ||
| from executorch.exir.passes import MemoryPlanningPass | ||
| from executorch.exir.passes.quant_fusion_pass import QuantFusionPass | ||
| from executorch.exir.passes.sym_shape_eval_pass import ConstraintBasedSymShapeEvalPass | ||
| from executorch.exir.program._program import to_edge | ||
| from executorch.exir.program._program import to_edge_transform_and_lower |
There was a problem hiding this comment.
Should we do from executorch.exir import to_edge_transform_and_lower
| "quantized_decomposed.embedding_4bit.dtype", | ||
| "aten.embedding.default", | ||
| ], | ||
| skip_ops_for_coreml_delegation=[], |
There was a problem hiding this comment.
oh wait, I remember they had the error, did you switch to run cpu 4bit embedding?
There was a problem hiding this comment.
By using torchao APIs, we get around the error.
There was a problem hiding this comment.
Any chance you know the reason? I just remember it errors out, but not sure the reason
This switches the ANE model to use to_edge_transform_and_lower and torchao quantization APIs.
To use to_edge_transform_and_lower, we first need to land: #12629
To use torchao quant APIs, we first need to land #12648 and #12664.
This PR contains all of the changes from those PRs because it is rebased on them. I will rebase on main once those PRs land to make this easier to review.