Correct type hint for load? Processor or Union[PreTrainedTokenizer, PreTrainedTokenizerFast]?

[jrp2014]

The load function in mlx_vlm/utils.py uses AutoProcessor.from_pretrained(...) internally (returning a Processor), but is explicitly type-hinted to return Union[PreTrainedTokenizer, PreTrainedTokenizerFast].

What is the correct annotation, please?

[Blaizzy]

Could you send a reproducible example?

[Blaizzy]

The type for most if not all processors is ProcessorMixin

Sorry, it's about bit late 👌🏽

[jrp2014]

OK, so should the load function, etc, be annotated with a ProcessorMixin type hint instead??

[jrp2014]

Currently, my ci produces things like the following:

ERROR Argument PythonBackend | TokenizersBackend is not assignable to parameter processor with type PythonBackend in function mlx_vlm.generate.generate [bad-argument-type] → src/check_models.py:5901:23

This suggests that the processor argument you are forwarding to mlx_vlm.generate.generate could be either a PythonBackend or TokenizersBackend, but the generate function expects only a PythonBackend.

Solution: Locate the call to generate in src/check_models.py at or near line 5901. Ensure that the processor argument you supply is strictly of type PythonBackend.

If you have code like:

generate( ..., processor=processor, ... )

Make sure processor is explicitly a PythonBackend. For example, you may need to add a type check or cast before the call:

if isinstance(processor, PythonBackend): output = generate(..., processor=processor, ...) else: # Handle or convert to PythonBackend, or raise an error raise TypeError("processor must be a PythonBackend")

Or, refactor your code to guarantee only PythonBackend is passed at this call site.

This adjustment should resolve the type error and allow the check to pass.

[Blaizzy]

Now,

Could you send a reproducible example?

Because it's not very clear to me what the issue is.

[jrp2014]

As per my first message, is the type annotation of the load function in mlx_vlm/utils.py, which uses AutoProcessor.from_pretrained(...) consistent with the function return result of Union[PreTrainedTokenizer, PreTrainedTokenizerFast] with the new transformers 5?

[Blaizzy]

maybe, I haven't checked and nothing jumped out to me on the logs

[jrp2014]

My bot tells me that I need to make changes to my code such as:

-    from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast
+    from transformers.tokenization_python import PythonBackend
+    from transformers.tokenization_utils_tokenizers import TokenizersBackend


 def _load_model(
     params: ProcessImageParams,
-) -> tuple[Module, PreTrainedTokenizer | PreTrainedTokenizerFast, Any | None]:
+) -> tuple[Module, PythonBackend | TokenizersBackend, Any | None]:

(for transformers 5)

[Blaizzy]

Thanks for the update!

I will make the updates before the next release since that will support official v5.0.0👌🏽