fixing GPTQ #148
fixing GPTQ #148
Conversation
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
facebook-github-bot
added
the
CLA Signed
Summary: trying to fix the issue with kv_cache update by changing tracing into a tensor subclass. However it seems we have less success than the fx tracer. The fx tracer breaks due k_out[:,:, input_pos] = k_val getting traced as new_var = torch.ops.aten.index_put_(k_out, [None, None, input_pos], k_val) with new var never being accessed afterward. new_var becomes hte correct multiInput value, but then is lost. The subclass ont he other hand, tries to use the func "<slot wrapper '__setitem__' of 'torch._C.TensorBase' objects>" which seems to not want to mutate k_out and so the attempt to make it a multiTensor fails. Test Plan: sh run.sh Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
Summary: trying to fix the issue with kv_cache update by changing tracing into a tensor subclass. However it seems we have less success than the fx tracer. The fx tracer breaks due k_out[:,:, input_pos] = k_val getting traced as new_var = torch.ops.aten.index_put_(k_out, [None, None, input_pos], k_val) with new var never being accessed afterward. new_var becomes hte correct multiInput value, but then is lost. The subclass ont he other hand, tries to use the func "<slot wrapper '__setitem__' of 'torch._C.TensorBase' objects>" which seems to not want to mutate k_out and so the attempt to make it a multiTensor fails. Test Plan: sh run.sh Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 9ed1621 Pull Request resolved: #148
|
Hi @HDCharles! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
Stack from ghstack (oldest at bottom):
Summary:
trying to fix the issue with kv_cache update by changing tracing into a
tensor subclass. However it seems we have less success than the fx
tracer. The fx tracer breaks due
k_out[:,:, input_pos] = k_val
getting traced as
new_var = torch.ops.aten.index_put_(k_out, [None, None,
input_pos], k_val)
with new var never being accessed afterward. new_var becomes hte correct
multiInput value, but then is lost.
The subclass ont he other hand, tries to use the func "<slot wrapper 'setitem' of 'torch._C.TensorBase' objects>"
which seems to not want to mutate k_out and so the attempt to make it a
multiTensor fails.
Test Plan: sh run.sh
Reviewers:
Subscribers:
Tasks:
Tags: