[EVAL] Adding PHARE

[clefourrier]

Evaluation short description

Bias/hallucination eval

Evaluation metadata

Provide all available

• Paper url: https://www.giskard.ai/knowledge/good-answers-are-not-necessarily-factual-answers-an-analysis-of-hallucination-in-leading-llms
• Dataset url: https://huggingface.co/datasets/giskardai/phare

[nsbg]

Hi, I'm interested in this task. Is it still open?

[clefourrier]

Hi @nsbg !
Yes, if you want to work on it please feel free to, do you want me to assign you this feature?

[nsbg]

Great! Could you explain a bit more clearly what exactly I need to do and do I need GPU?

[clefourrier]

Yes of course!
So the goal is to add the above task (phare) in lighteval, as a new task.
The documentation to add custom new tasks is here, and you might have to add a custom metric as well.
You can take inspiration from the community contributed tasks here: https://github.com/huggingface/lighteval/tree/main/community_tasks.

I would suggest first looking at the dataset, then at how the evaluation is implemented, and feel free to ask us to take a look at things once you get started if you need some help :)

[clefourrier]

I don't think you will need GPUs to start: you can do the first comparisons of implementation quite easily using a very small model on just 10 samples

[nsbg]

Okay I'll try it and ask if I have any question. Thanks!