Fix BatchEncoding.to() for nested elements

[eginhard]

What does this PR do?

Extend BatchEncoding.to() to also work for nested elements.

When using voice presets in Bark, the processor returns a BatchEncoding of

• { "input_ids": torch.Tensor, "attention_mask": torch.Tensor, "history_prompt": BatchFeature}

Currently, only tensor elements are moved, so running on cuda the following code fails with RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select):

import scipy
import torch

from transformers import AutoProcessor
from transformers import BarkModel

model = BarkModel.from_pretrained("suno/bark-small")
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to(device)

sampling_rate = model.generation_config.sample_rate
processor = AutoProcessor.from_pretrained("suno/bark-small")
voice_preset = "v2/en_speaker_6"

# prepare the inputs
text_prompt = "Let's try generating speech, with Bark, a text-to-speech model"
inputs = processor(text_prompt, voice_preset=voice_preset)

# generate speech
speech_output = model.generate(**inputs.to(device))
scipy.io.wavfile.write("bark_out.wav", rate=sampling_rate, data=speech_output[0].cpu().numpy())

A workaround was to manually do inputs["history_prompt"].to(device). This PR fixes this by moving all nested elements with a callable to().

Fixes #34634

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case: BarkProcessor voice_preset doesn't work #34634 (comment)
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@Rocketknight1

[Rocketknight1]

Yes, LGTM! Thank you for the PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kylesayrs]

Neat

[Rocketknight1]

cc @itazap for tokenizers, feel free to merge it if you're happy

[ebezzam]

@Rocketknight1, @itazap any update on this PR?

I can confirm it would address #34634 and a Bark test mentioned in #39478 🙂

[Rocketknight1]

Good point - I took another look and I think this is safe, so I'm going to merge! If it breaks anything, anyone finding this PR can yell at me 😅