MVEB Overview

[AdnanElAssadi56]

Classification/ZeroshotClassifcation:

• mteb/kinetics-400
• mteb/HMDB51
• mteb/Breakfast
• mteb/kinetics-700-2020
• mteb/UCF101-51VA
• mteb/SomethingSomethingV2
• mteb/kinetics-600
• mteb/VGGSound
• mteb/AVE-Dataset
• mteb/Human-Animal-Cartoon
• mteb/RAVDESS_AV
• mteb/MUSIC-AVQA_cls-preprocessed
• mteb/MELD
• mteb/AVMeme-Exam
• mteb/WorldSense_1min

Pair Classification:

• mteb/Human-Animal-Cartoon
• mteb/AVE-Dataset
• mteb/RAVDESS_AV
• mteb/MELD
• mteb/MUSIC-AVQA_cls-preprocessed

Clustering:

• mteb/WorldSense_1min
• mteb/AVE-Dataset
• mteb/RAVDESS_AV
• mteb/MELD
• mteb/MUSIC-AVQA_cls-preprocessed
• VideoMME by domain?

Retrieval:

• mteb/MSR-VTT Task Implemented. Pending: Reupload MSR-VTT dataset in standard retrieval format #4375
• mteb/MSVD
• mteb/DiDeMo
• mteb/TUNA-Bench_1K
• mteb/ActivityNet_Captions_val2
• mteb/YouCook2_val
• mteb/VATEX_test_1k
• mteb/Shot2Story20K_test
• mteb/VGGSound_AV_RETRIEVAL
• mteb/VALOR-32K
• mteb/AudioCaps_AV
• mteb/panda-70m
• mteb/AVMeme-Exam

Video Question Answering:

• mteb/worldqa
• mteb/EgoSchema_subset
• mteb/NExT-QA
• mteb/PerceptionTest_val
• mteb/star_bench_val
• mteb/AV-SpeakerBench
• mteb/WorldSense_1min
• mteb/Daily-Omni
• mteb/Video-MME_short
• mteb/OmniVideoBench_subset
• mteb/AVQA_val
• mteb/AVMeme-Exam
• mteb/MVBench

Models:

• ImageBind
• LanguageBind
• LCO
• PE-AV (Facebook) Add model: PE-AV #3797
• Omni-Embed-Nemotron
• Tevatron/OmniEmbed-v0.1
• Qwen2.5-Omni
• e5-omni
• nvidia/omnivinci
• Seed-1.6-Embedding

Requires GitHub repo cloning:

• ONE-PEACE
• OmniBind

[Muennighoff]

is this still the best overview issue? ; cc @shriyasudhakar who's interested in joining!

[AdnanElAssadi56]

is this still the best overview issue? ; cc @shriyasudhakar who's interested in joining!

Great! @KennethEnevoldsen can you add her to the slack channel?

[shriyasudhakar]

Hi! Super excited to get started! Niklas sent me an invite but I seem to be having issues joining with my Stanford email (I'm an incoming student so it's possible my email doesn't have full privileges, submitted a ticket regarding this). Please do try resending the invite if possible. I can also provide my personal email if it's possible to invite external guests

[KennethEnevoldsen]

hi @shriyasudhakar! I sadly do not have you stanford mail - if you send me a mail on kenevoldsen@pm.me then I will try to add you

[manandey]

@Muennighoff / @KennethEnevoldsen - are there any contribution guidelines or docs available? I’d be happy to contribute. Thanks!

[Samoed]

We have a documentation about how to add new dataset here https://embeddings-benchmark.github.io/mteb/contributing/adding_a_dataset/

[manandey]

@Samoed I meant about the MVEB initiative. Is it the same?

[Samoed]

Yes, the same

[KennethEnevoldsen]

@manandey do be aware that the code that allows you to add a video dataset is being added in #4199

[dschechter27875]

Hi! I’ve started experimenting a bit with the MSR-VTT retrieval direction and built a small prototype (CLIP + frame sampling + mean pooling) that seems to work on small subsets.

I saw MSR-VTT is listed here and also mentioned in #4375 — would that be the best place to share more detailed experiments / contribute, or is there another thread I should follow for the retrieval side?