[Bugfix] Decode Tokenized IDs to Strings for hf_processor in llm.chat() with model_impl=transformers

[ariG23498]

This PR adds a check in the llm.chat() processing logic to decode tokenized IDs back to strings when using model_impl="transformers". This ensures compatibility with the Hugging Face processor (hf_processor), which expects string inputs and does not support raw token IDs.

To verify the issue without this fix (or confirm the fix works), use the following code:

from vllm import LLM

model_id = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
vlm = LLM(
    model=model_id,
    model_impl="transformers",
    disable_mm_preprocessor_cache=True,
    enable_prefix_caching=False,
    enable_chunked_prefill=False
)

image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": image_url}},
            {"type": "text", "text": "What is the content of this image?"}
        ],
    },
]

# Perform inference and log output.
outputs = vlm.chat(conversation)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

• Without fix: Expect errors due to tokenized IDs being passed to hf_processor.
• With fix: Inference should succeed, printing a description of the image (e.g., cats on a couch).

CC: @hmellor @zucchini-nlp

[gemini-code-assist]

Code Review

This PR addresses an issue where tokenized IDs were passed to a Hugging Face processor that expects string inputs. The fix correctly decodes the IDs to strings. My review focuses on improving the robustness of the implementation by avoiding in-place modification of input data, which could lead to unexpected side effects or errors.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ariG23498]

@DarkLight1337 I have modified the file as suggested.

[DarkLight1337]

Thanks for the fix! Can you also add a test case in tests/models/multimodal/processing to avoid regression? cc @Isotr0py @hmellor

[Isotr0py]

LGTM too as long as regression test added!

[ariG23498]

I am pretty sure this does not fit the bill for the processing only dir. Any pointers would be great.

[DarkLight1337]

You can create a new file called test_transformers.py

[hmellor]

Shouldn't the test case for this PR test that the MultiModalProcessor in vllm/model_executor/models/transformers.py still works when token IDs are provided to apply?

[DarkLight1337]

We are testing the multi-modal processor here rather than the model implementation, so the test should be under processing directory. The other tests under processing directory also only check the multi-modal processor

[DarkLight1337]

And like @hmellor said, we should be unit testing the multi-modal processor without instantiating vLLM engine

[hmellor]

The fix LGTM, but I think we should change the test to be more specific to the problem we're fixing. Instantiating a full LLM is quite an expensive way to check that the processor can consume token IDs

[hmellor]

Shouldn't the test case for this PR test that the MultiModalProcessor in vllm/model_executor/models/transformers.py still works when token IDs are provided to apply?

[DarkLight1337]

Please fix pre-commit