Conversation
pcuenca
left a comment
pcuenca
left a comment
There was a problem hiding this comment.
Getting 18 tps with my changes, on two M3 Ultra over Ethernet.
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
…tize_model and dequantize_model based on usage
|
Thanks Pedro I'm getting 32 tok/s over thunderbolt! 
|
|
Awesome! 🔥 |
| model.shard(tensor_group) | ||
|
|
||
| mx.eval(model.language_model.parameters()) | ||
| mx.eval(model.parameters()) |
There was a problem hiding this comment.
This is still not working for me, did you revert the model.language_model change?
There was a problem hiding this comment.
To be clear: mx.eval(model.language_model.parameters()) is what works for me.
There was a problem hiding this comment.
I tried with evaluating the whole model and it works. Just like in the normal path.
What's was missing is model.eval()
My biggest issue is tight right now is that I'm highly limited in my setup to fully test things
There was a problem hiding this comment.
In my setup, execution never reaches the new model.eval() you added.
2 participants
No description provided.