How to evaluate on my own benchmark?

[prabhant]

Hi,

Thank you for benchbench, i have a question on how can i use benchbench for my own multitask benchmark?

Also would there be any changes requires if my evaluation scores are also pearson/kendall scores instead of accuracies?

Thank you!

[ghzhang233]

Hi.

Thanks for your interest in our work.

To use it for your own benchmark, simply format your data as a pd.DataFrame, where each row represents a model and each column represents a task. You'll also need to specify the list of task names as cols.

from benchbench.measures.cardinal import get_diversity, get_sensitivity

data, cols = YOURCODEHERE
diversity = get_diversity(data, cols)
sensitivity = get_sensitivity(data, cols)

If the evaluation metric is not accuracy, the diversity should still work, but sensitivity might need to be redefined depending on the specific context.

Let me know if there's anything else that I can help with.