How do I run the new DeepSeek R1 0528?

[mtcl]

I've got the Q4_k_m downloaded and it's not working with the standard instructions. Any tips on which optimize roles to use?

I have 512 GB of RAM and 2x4090. Intel AMX processor but I can work with avx512

[shinchou]

Same issue.

[mtcl]

I got it working.

Here is the implementation:

https://youtu.be/Xui3_bA26LE

[wqshmzh]

Greeings! Is there anyway to run DeepSeek-R1-0528 in unsloth's IQ1 and IQ2 quants using KT? I only have a desktop computer with 256GB DDR5 memory. Thank you very much if you could reply very soon.

[wqshmzh]

Is there anyway to run unsloth's DeepSeek 0528 in IQ1, IQ2 quants? I have tried the DeepSeek-R1-0528-IQ2_XXS and it only consumes 100+GB of my memory. It can only run without cuda graph and output complete garbage (I mean messy words) after I say "hello" to it. If I turn on cuda graph, the KT will raise an error:
File "/home/wangqs/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 66, in batch_embeddings
self.model.embed_tokens(tokens.to(torch.device('cpu')))
File "/home/wangqs/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wangqs/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wangqs/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/sparse.py", line 192, in forward
return F.embedding(
^^^^^^^^^^^^
File "/home/wangqs/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/functional.py", line 2546, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index out of range in self

I completely have no idea how to fix this.