fix: complete the eval metrics Truth_Ratio calculation mentioned in the paper #163
Conversation
molereddy
left a comment
molereddy
left a comment
There was a problem hiding this comment.
Thank you for adding this. Few small comments to clean things up and clarify some comments, and we can merge this in.
|
|
||
| handler: truth_ratio | ||
| aggregator: closer_to_1_better | ||
| aggregator: prob_mean |
There was a problem hiding this comment.
This edit can be removed. The prob_mean is best added only as an option as it is not the way truth ratio is usually used in the literature, which follows how the TOFU paper implemented it.
src/evals/metrics/memorization.py
Outdated
| # Original definition: wrong / correct | ||
| truth_ratios = wrong_prob / (correct_prob + 1e-10) | ||
| else: | ||
| # New definition: correct / (correct + wrong) |
There was a problem hiding this comment.
better call this as the definition from OpenUnlearning than "new"
src/evals/metrics/memorization.py
Outdated
| truth_ratios = wrong_prob / (correct_prob + 1e-10) | ||
|
|
||
| if use_original_ratio: | ||
| # Original definition: wrong / correct |
There was a problem hiding this comment.
better call this as the definition from TOFU paper for clarity
|
Also, make sure the lint tests pass. Instructions for formatter setup here: https://github.com/locuslab/open-unlearning/blob/main/docs/contributing.md#create-a-pull-request |
|
Thank you for your guidance. I have changed and commited again. It has "make quality" in my own computer. |
2 participants
What does this PR do?
Fixes #160
Now, the "aggregator" is expanded to three types: "closer_to_1_better", "true_better", "prob_mean".
You can modify the “aggregator” in the file "configs/eval/tofu_metrics/forget_Truth_Ratio.yaml" to change the method to calculate the Truth_Ratio.
Before submitting