Skip to content

Fix: Parse CSV Single-Turn Golden ToolCalls as JSON objects#2565

Merged
penguine-ip merged 5 commits intoconfident-ai:mainfrom
seankelley-dt:2564-fix-expected-tools-csv-parsing
Apr 9, 2026
Merged

Fix: Parse CSV Single-Turn Golden ToolCalls as JSON objects#2565
penguine-ip merged 5 commits intoconfident-ai:mainfrom
seankelley-dt:2564-fix-expected-tools-csv-parsing

Conversation

@seankelley-dt
Copy link
Copy Markdown
Contributor

Summary

This PR fixes #2564 by altering the parsing logic for loading goldens from CSV.

Context

Issue #2564 shows full repro code for this error.

Changes

  • deepeval/dataset/dataset.py:EvaluationDataset.add_goldens_from_csv_file: Instead of simply splitting the row value string by the provided delimiter, parse the entire row value string using the same logic as EvaluationDataset.add_test_cases_from_csv_file since EvaluationDataset.save_as serialises the ToolCall objects in this way.

Tests

  • Updated tests/test_core/test_datasets/goldens.csv to include example values for tools_called and expected_tools columns

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 19, 2026

@seankelley-dt is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@seankelley-dt seankelley-dt changed the title Fix: Parse CSV Single-Turn Goldens as JSON objects Fix: Parse CSV Single-Turn Goldens ToolCalls as JSON objects Mar 19, 2026
@seankelley-dt seankelley-dt changed the title Fix: Parse CSV Single-Turn Goldens ToolCalls as JSON objects Fix: Parse CSV Single-Turn Golden ToolCalls as JSON objects Mar 19, 2026
@seankelley-dt
Copy link
Copy Markdown
Contributor Author

looks right — the old delimiter-based split couldn't handle serialized ToolCall objects. two things: (1) the CSV fixture update removes the old format entirely, but users who have existing CSV files with the old delimiter format will now get json.loads errors. might want a fallback that tries JSON first, then falls back to delimiter split for backward compat. (2) what happens when tools_called column contains an empty string vs "[]"? the get_column_data default changed from "" to "[]", but existing CSVs without the column would have "" from pandas fillna.

This is a great point @themavik. I've just updated the changes to fallback to the original functionality if the JSON decode fails.

On the point about the get_column_data default changing from "" to "[]", this will allow for the same functionality as before where, if the csv does not contain expected_tools_col_name, ["[]"] * len(df) is returned from get_column_data, the json decode succeeds and the subsequent list comprehension for tool in trimAndLoadJson(expected_tools_str) never runs.

Empty strings in the dataframe column won't be an issue as there's checks for empty strings before the JSON decode.

@seankelley-dt
Copy link
Copy Markdown
Contributor Author

Actually on second thought, I have changed the get_column_data default back to "" to avoid lots of unnecessary JSON decode calls. If the column is missing, the empty string check will already skip any parsing.

@penguine-ip
Copy link
Copy Markdown
Contributor

@seankelley-dt thank you!

@penguine-ip penguine-ip merged commit ef5b104 into confident-ai:main Apr 9, 2026
8 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV Single-Turn Goldens do not properly parse expected_tools

2 participants