Does the output of sequence_attributions support other encoding formats (such as UTF-8)?

[niewangwen]

Question

My input is in Chinese, and I found that during the model inference process, the outputs are all in the correct format. However, in the results displayed by out.show(), everything appears as garbled text. How should I resolve this issue?

Additional context

Checklist

• [ ✅] I've searched the project's issues.

[gsarti]

Hi @niewangwen, thanks for reaching out! The problem is not the encoding of the characters: you can see from the minimal example below that Chinese characters are shown just fine in the output table:

import inseq

model = inseq.load_model("Helsinki-NLP/opus-mt-en-zh", "saliency")
out = model.attribute("Hello world! This is a translation to Chinese.")
out.show()

The problem is likely due to byte splitting, which produces the UTF-8 byte components of each Chinese character instead (see e.g. huggingface/transformers#30937). Please provide a minimal code for reproducing the issue with Inseq, and I will have a look asap!

[niewangwen]

Hi @gsarti , Thank u for your reply, this problem happens when I'm using the Qwen model to do the text generation task.
And I already solve the problem, it is because the sequence_attributions.source and sequence_attributions.target stores the original token in tokenizer.json of Qwen3. the vocabs in tokenizer is unreaderable, so it needs decode like:

toker = AutoTokenizer.from_pretrained(
    model_name)
print("\n before source tokens:")
print([t.token for t in out.sequence_attributions[0].source])

print("\n before target tokens:")
print([t.token for t in out.sequence_attributions[0].target])

for attr in out.sequence_attributions:
    for item in attr.source:
        decode_token = toker.decode(item.id)
        item.token = decode_token
    for item in attr.target:
        decode_token = toker.decode(item.id)
        item.token = decode_token

print("\n after source tokens:")
print([t.token for t in out.sequence_attributions[0].source])

print("\n after target tokens:")
print([t.token for t in out.sequence_attributions[0].target])

and the output is: