Skip to content

truncation is hardcoded and cannot be controlled externally #64

@msdsm

Description

@msdsm

Currently, truncation is hardcoded to True within the Qwen3VLEmbedder class.
This can cause the following ValueError when processing long texts or videos, preventing the process from completing successfully.

ValueError: Mismatch in video token count between text and input_ids. Got ids=[8075] and text=[9360]. Likely due to truncation='max_length'. Please disable truncation or increase max_length.

To resolve this, I propose allowing truncation to be specified as an external parameter. This would enable users to set truncation=False when needed to avoid this error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions