Development and Evaluation of Wikimpacts V2

[liniiiiii]

1. information extraction splits to two parts, L1 (same as V1), and L3 (merge L2 and L3 in V1), from V1, we know that in L2, most of the impact records are in a single country, so in V2, we merge L2 and L3 information and use one prompt to get the impact information. For single-country impact information, the location field is left NULL.
2. using reasoning model o1 and o3mini for V2 development, the model is required to return the source section containing header names and sentences where it finds the information.
3. L1 evaluation is the same as V1, but in the new L3, we first match the administrative area and GID, either match will continue the further fields (num, time, location) matching, if not, will assign the lowest matching score, for location matching, using both GID and names, and get the higher matching score for the best matches computation. For time and num, are the same as V1.
4. the new L3 evaluation results are shown below, setting 1 is all the weights are the same as 1, and for setting2 is that for the time information, day is 0.1, month 0.15, year 0,2, inflation adjust 0.2 and inflation year 0.2
5. initial error analysis for L3:

• missing information from gold (keywords setup limitation), such as in flood, the text mentions "people drowned", which is not annotated in the gold data, but the model captures it
• the model is poor in catching when text mentions "no deaths reported"
• fine location mis-geocoded issue, some tiny locations are not well recognized by openstreetmap, an approach will adopt here is using llm to extract the finest GID which convert this location in post-processing for the final database not for the evaluation
• in the same location (where time is missing), sometimes, the text mentions several times about the impact information, in the gold, there is one confirmed data entry, but the model will capture all of them

	o1 Weight as Setting 1	o1 Weight as Setting 2	o3mini Weight as Setting 1	o3mini Weight as Setting 2
deaths	0.51	0.27	0.55	0.27
injuries	0.43	0.23	0.45	0.23
homeless	0.49	0.41	0.44	0.37
displaced	0.49	0.33	0.47	0.31
affected	0.51	0.44	0.50	0.43
buildings damaged	0.56	0.35	0.52	0.29
insured damage	0.19	0.17	0.18	0.17
damage	0.44	0.26	0.47	0.25

[i-be-snek]

Thank you for this great work, @liniiiiii :D

I'm curious about this:

the new L3 evaluation results are shown below, setting 1 is all the weights are the same as 1, and for setting2 is that for the time information, day is 0.1, month 0.15, year 0,2, inflation adjust 0.2 and inflation year 0.2

On what basis are these selected? Is it some kind of "importance" scheme? I ask because the differences between 0.1 and 0.15 or 0.2 may not be very high, so you may even get very similar scores if the day, month, and year all have their weight set to 0.1 or 0.2. Maybe that's something worth trying to understand the scores better. I feel like the year is quite significant, so maybe it deserves a higher weight?

[liniiiiii]

indeed, it's basically based on the kind of "importance", do you have any suggestions about the weights, I just pick these values for testing, I mean it will just shape the weighted score

[i-be-snek]

I think it could take some experimentation to create "steps" or "degrees" of importance, because like I said 0.1 and 0.15 are not that far apart. Maybe a simple scale could be better?

• 1.0: essential, for years
• 0.75: important, for months and the inflation year and inflation adjustment
• 0.50: nice-to-know, maybe for months
• 0.25: insignificant, for days

This is just a proposed scheme and it may take some playing around to land on the best one.

[koffiworou]

Hi @liniiiiii , thanks for the great work!

I have two comments:

1. For the evaluations, it would be helpful to compare the V1 and V2 settings for L1 information using the same model, GPT-4o, before moving on with other OpenAI models . You can apply similar weights as in V1, allowing us to assess the impact of our new level information extraction.
2. I find the weight assignments quite subjective. We could start by setting all attributes to 1 and later refine them based on importance, perhaps following @i-be-snek's suggestions.