What if the visual transformer does not have a class token?

[Tgaaly]

I see the code in the VITAttentionGradRollout code requires a class token. What if the model architecture does not have a class token?

If for example my attention layer is 196x196 (corresponding to 14x14 spatial resolution), can one take the mean of all other patches w.r.t. to each patch, as follows: mask = result[0].mean(0)? I've tried this and I didn't get very meaningful results. Is there another way to deal with transformers without class tokens?

[SohamMazumder]

You could something like this. This will compare the attention of each patch to all other patch (including with itself)

for pos in range(result.size(1)):
        mask = result[0, pos, :]
        width = int(mask.size(-1)**0.5)
        mask = mask.reshape(width, width).numpy()
        mask = mask / np.max(mask)
        masks.append(mask)
return mask