Can I use locale tokenizer ?

[edisonchan]

I have try to set tokenizer use local dir in the benchmark_visualization.ipynb:
tokenizer="/home/xfusion/Downloads/vllm/models/Qwen3.5-122B-A10B-int4-AutoRound",

but the output is:
Initializing Corpus...
Error loading tokenizer: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/xfusion/Downloads/vllm/models/Qwen3.5-122B-A10B-int4-AutoRound'. Use repo_type argument if needed.
Falling back to 'gpt2' tokenizer as approximation.

[eugr]

I can add it in a future version, but in the meantime, you can just use HF model name so it could download the tokenizer automatically. If it fails for you, try to run the version from this repo (not the release - I bumped transformers version).
I will publish the new release later today though.

[eugr]

Published a version that requires transformers 5