Skip to content

Commit 2f6e6b3

Browse files
authored
[Bugfix] Fix isinstance check for tensor types in _load_prompt_embeds to use dtype comparison (#21612)
Signed-off-by: Alexandre Juan <[email protected]>
1 parent a55c950 commit 2f6e6b3

File tree

1 file changed

+5
-3
lines changed

1 file changed

+5
-3
lines changed

vllm/entrypoints/openai/serving_engine.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -957,9 +957,11 @@ def _load_prompt_embeds(
957957
def _load_and_validate_embed(embed: bytes) -> EmbedsPrompt:
958958
tensor = torch.load(io.BytesIO(base64.b64decode(embed)),
959959
weights_only=True)
960-
assert isinstance(
961-
tensor,
962-
(torch.FloatTensor, torch.BFloat16Tensor, torch.HalfTensor))
960+
assert isinstance(tensor, torch.Tensor) and tensor.dtype in (
961+
torch.float32,
962+
torch.bfloat16,
963+
torch.float16,
964+
)
963965
if tensor.dim() > 2:
964966
tensor = tensor.squeeze(0)
965967
assert tensor.dim() == 2

0 commit comments

Comments
 (0)