[MVEB] PE-AV Model, Kinetics400 Dataset, RavdessAV Dataset by AdnanElAssadi56 · Pull Request #4199 · embeddings-benchmark/mteb

AdnanElAssadi56 · 2026-03-05T07:26:38Z

(From closed PR)

Adds the following:
mteb/kinetics-400
mteb/RAVDESS_AV
PE-AV (Facebook) Close #3797

Also includes some remaining components from the parallel video integration work we accidently did.

mteb/abstasks/classification.py

mteb/models/model_implementations/pe_av_models.py

AdnanElAssadi56 · 2026-03-05T09:13:44Z

@Samoed I edited the collator to handle one video item.

Samoed · 2026-03-05T09:27:46Z

Yeah, I forgot to update collator after changing to one video

AdnanElAssadi56 · 2026-03-05T11:05:47Z

Results from "facebook/pe-av-small":

RAVDESS_AVClustering.json

Samoed · 2026-03-05T11:14:28Z

This is hard to match results for ravdess. I think you can run one of these tasks (from https://arxiv.org/pdf/2512.19687)

mteb/_create_dataloaders.py

AdnanElAssadi56 · 2026-03-05T21:41:11Z

MSR-VTT:
Some discrepancy but maybe because we are using audio as well.
MSRVTTV2T.json

AdnanElAssadi56 · 2026-03-07T09:01:43Z

@Samoed Anything else here?

Samoed · 2026-03-07T10:10:07Z

Can you resolve my comments and make CI green?

AdnanElAssadi56 · 2026-03-07T10:29:38Z

I think tests are failing because of n_embedding_parameters. It can't be calculated by the method in Model Meta:
Could not calculate embedding parameters for facebook/pe-av-base-16-frame as config.json could not be loaded

Samoed · 2026-03-07T10:59:41Z

Strange. I get it without problems

import mteb

meta = mteb.models.ModelMeta.from_hub("facebook/pe-av-base-16-frame")
meta.n_embedding_parameters
# 51576832

mteb/_create_dataloaders.py

AdnanElAssadi56 · 2026-03-07T21:00:33Z

@Samoed Tests resolved here

isaac-chung

Got a few non-blocking questions. Can be addressed in a separate PR.

mteb/abstasks/task_metadata.py

mteb/tasks/video/classification/eng/kinetics400_classification.py

AdnanElAssadi56 · 2026-03-09T08:41:25Z

@Samoed @isaac-chung Changed input_column to list.

AdnanElAssadi56 · 2026-03-09T08:56:59Z

lint is giving error because list is mutable

isaac-chung · 2026-03-09T09:00:14Z

It's looking for something like this I think:

from typing import ClassVar

input_column_name: ClassVar[list[str]] = ["video", "audio"]

mteb/abstasks/classification.py

mteb/tasks/video/classification/eng/kinetics400_classification.py

mteb/abstasks/clustering.py

mteb/tasks/video/clustering/eng/ravdess_av_clustering.py

mteb/_create_dataloaders.py

mteb/tasks/video/retrieval/eng/msr_vtt.py

AdnanElAssadi56 · 2026-03-12T17:57:11Z

@Samoed Can you give a look here when you have the time?

mteb/abstasks/classification.py

mteb/_create_dataloaders.py

isaac-chung · 2026-03-13T16:13:26Z

How do you tell VA2C and V2C tasks apart? Is it that: only in VA2C tasks, we process the audio, regardless if it's from the video or in a separate column?

AdnanElAssadi56 · 2026-03-17T17:41:04Z

@Samoed @isaac-chung @KennethEnevoldsen
This is somewhat of a blocker right now. Can we discuss the approach here if you are available?

isaac-chung · 2026-03-17T20:15:52Z

One main thing we should clarify is how to handle video with and without audio + separate audio

mteb/_create_dataloaders.py

AdnanElAssadi56 · 2026-04-07T14:11:26Z

@AdnanElAssadi56 so how does a dataset with video (incl. audio) look on HF and how is it different if it also includes audio.

It basically looks like this, with audio col being present when the video has audio.

video	audio	label
VideoDecoder_0	{"array": [...], "sr": 16000}	0
VideoDecoder_1	{"array": [...], "sr": 16000}	3

AdnanElAssadi56 · 2026-04-10T17:15:34Z

Results from pe-av-small
RAVDESSAVClustering.json

Do i merge @KennethEnevoldsen @Samoed @isaac-chung ?

Samoed · 2026-04-10T17:20:12Z

No, please address comments from our reviews

AdnanElAssadi56 · 2026-04-10T17:24:13Z

No, please address comments from our reviews

Is there anything pending?

Samoed · 2026-04-10T17:31:44Z

Comments that unresolved

mteb/abstasks/task_metadata.py

mteb/types/_encoder_io.py

mteb/_create_dataloaders.py

…x collator output - Revert input_column_name from Mapping[str, str] to str | Sequence[str] - Remove VideoInputItem wrapper, pass frames tensor directly - Make VideoCollator return BatchedInput (consistent with AudioCollator) - MultimodalCollator uses static methods instead of chaining collators

AdnanElAssadi56 · 2026-04-11T19:20:19Z

@Samoed Any more points?

mteb/_evaluators/clustering_evaluator.py

mteb/abstasks/classification.py

mteb/abstasks/clustering.py

mteb/_evaluators/zeroshot_classification_evaluator.py

KennethEnevoldsen

alright it sounds like we can keep it as a sequence, but we need to documents the limitations.

I don't see why we don't want to support text+video in classification - it seems like we are avoiding creating a general solution. We will have to deal with this regardless at some point.

mteb/tasks/video/classification/eng/kinetics400_classification.py

mteb/tasks/video/clustering/eng/ravdess_av_clustering.py

mteb/tasks/video/retrieval/eng/msr_vtt.py

mteb/_create_dataloaders.py

mteb/types/_encoder_io.py

mteb/tasks/video/retrieval/eng/msr_vtt.py

mteb/models/model_implementations/pe_av_models.py

mteb/abstasks/classification.py

mteb/abstasks/_data_filter/task_pipelines.py

mteb/_evaluators/zeroshot_classification_evaluator.py

…ations - Rename VideoCollator -> FramesCollator, MultimodalCollator -> VideoCollator - Update VideoInput docstring to clarify frames-only, audio in AudioInput - Update input_column_name docs in classification/clustering base classes - Use ClassVar[Sequence[str]] for video task input_column_name - Extract isinstance check to top of zeroshot evaluator __call__ - Improve task_pipelines.py skip comment for multi-column tasks - Add TODO for MSR-VTT dataset reupload

…umn_name

AdnanElAssadi56 · 2026-04-14T20:46:36Z

@KennethEnevoldsen @Samoed

Added relevant issues from discussions above and resolved convos (commented the issue links here as well).

Samoed added new model Questions related to adding a new model to the benchmark new dataset Issues related to adding a new task or dataset video video extension labels Mar 5, 2026

Samoed reviewed Mar 5, 2026

View reviewed changes

mteb/models/model_implementations/pe_av_models.py Show resolved Hide resolved

Samoed reviewed Mar 5, 2026

View reviewed changes

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

KennethEnevoldsen reviewed Mar 7, 2026

View reviewed changes

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

isaac-chung reviewed Mar 8, 2026

View reviewed changes

mteb/abstasks/task_metadata.py Show resolved Hide resolved

mteb/tasks/video/classification/eng/kinetics400_classification.py Outdated Show resolved Hide resolved

Samoed reviewed Mar 9, 2026

View reviewed changes

Samoed reviewed Mar 10, 2026

View reviewed changes

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

Samoed reviewed Mar 10, 2026

View reviewed changes

mteb/tasks/video/retrieval/eng/msr_vtt.py Show resolved Hide resolved

Samoed reviewed Mar 13, 2026

View reviewed changes

mteb/abstasks/classification.py Show resolved Hide resolved

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

Samoed requested a review from KennethEnevoldsen March 13, 2026 16:19

Samoed reviewed Apr 7, 2026

View reviewed changes

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

AdnanElAssadi56 added 3 commits April 7, 2026 10:55

review fixes

4fa7f8f

lint

f89a06b

type hins fix

e3ed885

Samoed reviewed Apr 11, 2026

View reviewed changes

mteb/abstasks/task_metadata.py Show resolved Hide resolved

mteb/types/_encoder_io.py Outdated Show resolved Hide resolved

mteb/_create_dataloaders.py Outdated Show resolved Hide resolved

AdnanElAssadi56 added 3 commits April 11, 2026 13:00

fix: update clustering_evaluator to use Sequence instead of Mapping

069867a

fix: handle Sequence input_column_name in second create_dataloader call

05674f9

Samoed reviewed Apr 12, 2026

View reviewed changes

mteb/_evaluators/clustering_evaluator.py Show resolved Hide resolved

mteb/abstasks/classification.py Outdated Show resolved Hide resolved

mteb/abstasks/clustering.py Show resolved Hide resolved

AdnanElAssadi56 added 2 commits April 12, 2026 13:07

fix: skip statistics and text cleaning for multi-column video tasks

6b45a86

fix: pass explicit None for TypedDict fields in multi-column statistics

ce39fdb

KennethEnevoldsen reviewed Apr 13, 2026

View reviewed changes

mteb/_evaluators/zeroshot_classification_evaluator.py Outdated Show resolved Hide resolved

KennethEnevoldsen mentioned this pull request Apr 14, 2026

MVEB Overview #4130

Open

64 tasks

KennethEnevoldsen reviewed Apr 14, 2026

View reviewed changes

KennethEnevoldsen mentioned this pull request Apr 14, 2026

Add tests to ensure that input_column works for all modalities #4372

Open

AdnanElAssadi56 added 4 commits April 14, 2026 11:50

docs: link to encoder I/O types for default column names in input_col…

f51f9ef

…umn_name

fix: raise NotImplementedError for multi-column task cleaning

769855e

refactor: use tuples for input_column_name to avoid ClassVar

f1d234f

Conversation

AdnanElAssadi56 commented Mar 5, 2026 • edited by Samoed Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 5, 2026

Uh oh!

Samoed commented Mar 5, 2026

Uh oh!

AdnanElAssadi56 commented Mar 5, 2026

Uh oh!

Samoed commented Mar 5, 2026

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 5, 2026

Uh oh!

AdnanElAssadi56 commented Mar 7, 2026

Uh oh!

Samoed commented Mar 7, 2026

Uh oh!

AdnanElAssadi56 commented Mar 7, 2026

Uh oh!

Samoed commented Mar 7, 2026

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isaac-chung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 9, 2026

Uh oh!

AdnanElAssadi56 commented Mar 9, 2026

Uh oh!

isaac-chung commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 12, 2026

Uh oh!

Uh oh!

Uh oh!

isaac-chung commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdnanElAssadi56 commented Mar 17, 2026

Uh oh!

isaac-chung commented Mar 17, 2026

Uh oh!

Uh oh!

Uh oh!

AdnanElAssadi56 commented Apr 7, 2026

Uh oh!

AdnanElAssadi56 commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Samoed commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdnanElAssadi56 commented Apr 10, 2026

Uh oh!

Samoed commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdnanElAssadi56 commented Apr 11, 2026

Uh oh!

AdnanElAssadi56 commented Mar 5, 2026 •

edited by Samoed

Loading

AdnanElAssadi56 commented Mar 7, 2026 •

edited

Loading

isaac-chung commented Mar 13, 2026 •

edited

Loading

AdnanElAssadi56 commented Apr 10, 2026 •

edited

Loading

Samoed commented Apr 10, 2026 •

edited

Loading

Samoed commented Apr 10, 2026 •

edited

Loading