MMMU test dataset can be evaluated locally now

[ywwynm]

Checklist

• I have searched for similar feature requests before opening this one.

Category

Evaluation framework

Feature Description

According to official page, the test split of MMMU dataset can be evaluated locally now. However, in the latest version of lmms-eval, we still generate a result xslx file for the submission of evaluation. Please support evaluating it locally.

Motivation

MMMU dataset is a well-known and widely-used dataset for evaluating LMMs. Now it supports being evaluated locally for its test split, and it is important to add this feature in this repository for the LMM community.

Alternatives Considered

No response

Additional Context

No response

[ywwynm]

Similarly, the test split of MMBench can also be evaluated locally now.