occurred error when running on Mac min M4

[qinghai5060]

OS：MacOS 15.5 (24F74)
Python version：3.12.3

% python demo.py 
Using device: mps
Available models: ['ABRAHAM', 'BASE', 'COHOST']
Loading model COHOST...
Compiling model COHOST...
Warming up model COHOST...
/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/amp/autocast_mode.py:266: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
  warnings.warn(
codes torch.Size([1, 9, 215])
Traceback (most recent call last):
  File "/Users/xxx/audio/vui/demo.py", line 69, in <module>
    model = load_and_warm_model(default_model)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/demo.py", line 52, in load_and_warm_model
    render(
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/inference.py", line 326, in render
    codes = generate(
            ^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/inference.py", line 231, in generate
    out = self.decoder(embeddings, input_pos)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/model.py", line 332, in forward
    x = block(x, freqs_cis=freqs_cis, input_pos=input_pos, attn_mask=attn_mask)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/model.py", line 231, in forward
    x = x + self.attn(
            ^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/vui/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/model.py", line 124, in forward
    k, v = self.kv_cache.update(input_pos, k, v)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/audio/vui/src/vui/model.py", line 41, in update
    k_out[:, :, input_pos] = k_val
    ~~~~~^^^^^^^^^^^^^^^^^
RuntimeError: Index put requires the source and destination dtypes match, got BFloat16 for the destination and Float for the source.

[mogwai]

Yeah I've seen this happen on some machines, you can move the k_val and v_val to the dtype of k_out before assigning them. I had to do this in the gradio demo:

https://huggingface.co/spaces/fluxions/vui-space/blob/main/vui/model.py#L42

[mogwai]

I've added the fix for this issue, let me know if it works