Wav2Vec2Encoder::is_encoded returns true for unencoded features of length <size of hidden layer> samples

[plan9better]

For a 2B wav2vec2 with hidden layer size 1920 passing an input of 1920 samples (~0.12s at 16khz) with "return_hidden": true in config.json the hueristic check of dimensions returns true for unencoded data.

import ctranslate2
import numpy as np
converted_model_path = "ctranslate2_model"
converter = ctranslate2.converters.TransformersConverter("model")
converter.convert(converted_model_path)

model = ctranslate2.models.Wav2Vec2(
        converted_model_path,
        device="cpu",
        device_index=[0],
        compute_type="float32",
        intra_threads=0,
        inter_threads=1,
    )


# 1920 samples = 0.12 seconds at 16khz
array = np.random.uniform(low=-1, high=1, size=(1, 1, 1920)).astype(np.float32)
inputs = ctranslate2.StorageView.from_array(array)

output = model.encode(inputs)
print("Input: ", inputs)
print("Output: ", output)
print("The same: ", np.array_equal(inputs, output))

produces the output

Input:   -0.549616 0.54811 0.688297 ... -0.316497 -0.0261584 -0.179784
[cpu:0 float32 storage viewed as 1x1x1920]
Output:   -0.549616 0.54811 0.688297 ... -0.316497 -0.0261584 -0.179784
[cpu:0 float32 storage viewed as 1x1x1920]
The same:  True

The only place this function is called from is here and seems copy pasted between Whisper, Wav2Vec2 and Wav2Vec2Bart.

To be honest I don't understand the decision to include this since i couldn't find any recursive calls for which this might be the base case which is what this initially reminded me of, and it seems unlikely that a user will pass in already encoded data (in which case failing seems like a sane thing to do). And including it breaks the above mentioned edge case.

I'll be happy to open a PR with the deletion of it from all 3 models if someone confirms.
Cheers!

[a2d8a4v]

Hi @plan9better ,
I think it's not a problem, cause

CTranslate2/include/ctranslate2/layers/wav2vec2.h

Line 83 in 91d46ec

bool is_encoded(const StorageView& features) const {

has shown that if those three condition matches

1. a 3-dimension storageview object passed to the model
2. the 3rd dimension is equal to the model output size (your 23 w2v2 model is set 1920, and you also pass a feature of 1920 size in the final dimension)
3. the 2nd dimension is not equal to the input size (it will be computed via feature extractor, which including downsampling, etc.)

it will be considered as an encoded representation.

I may preprocess the audio sample first outside the CT2 library, I think it's better to processed independently outside this library.

[plan9better]

Hi @a2d8a4v thanks for replying.

I don't see how preprocessing is relevant in this case since even if you preprocess outside of ctranslate2 you still want to run the inference on that data, which doesn't happen if is_encoded returns true. As checked here:

    StorageView Wav2Vec2Replica::maybe_encode(StorageView features) {
      const Device device = _model->device();
      const DataType dtype = _encoder->output_type();

      features.move_to(device, dtype);

      if (_encoder->is_encoded(features))
        return features;

      StorageView encoder_output(dtype, device);
      (*_encoder)(features, encoder_output);
      return encoder_output;
    }

I believe the original author copy pasted the code from whisper where the logic leading up to the maybe_encode call is different so it may be of use there. If you take a look at the implementation you will see it's the same code between this and whisper. It may have went unnoticed because at first the model didn't support returning the hidden layer just the lm_head output where you would have to pass VOCAB_SIZE samples which is comically low. But it starts to pose a problem after a later introduction of returning the last hidden layer of the model instead of the lm head logits where the amount of samples you need to pass to encounter this rises to your hidden layer size 1920 in my case.