You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
deepeval/dataset/dataset.py:EvaluationDataset.add_goldens_from_csv_file: Instead of simply splitting the row value string by the provided delimiter, parse the entire row value string using the same logic as EvaluationDataset.add_test_cases_from_csv_file since EvaluationDataset.save_as serialises the ToolCall objects in this way.
Tests
Updated tests/test_core/test_datasets/goldens.csv to include example values for tools_called and expected_tools columns
seankelley-dt
changed the title
Fix: Parse CSV Single-Turn Goldens as JSON objects
Fix: Parse CSV Single-Turn Goldens ToolCalls as JSON objects
Mar 19, 2026
seankelley-dt
changed the title
Fix: Parse CSV Single-Turn Goldens ToolCalls as JSON objects
Fix: Parse CSV Single-Turn Golden ToolCalls as JSON objects
Mar 19, 2026
looks right — the old delimiter-based split couldn't handle serialized ToolCall objects. two things: (1) the CSV fixture update removes the old format entirely, but users who have existing CSV files with the old delimiter format will now get json.loads errors. might want a fallback that tries JSON first, then falls back to delimiter split for backward compat. (2) what happens when tools_called column contains an empty string vs "[]"? the get_column_data default changed from "" to "[]", but existing CSVs without the column would have "" from pandas fillna.
This is a great point @themavik. I've just updated the changes to fallback to the original functionality if the JSON decode fails.
On the point about the get_column_data default changing from "" to "[]", this will allow for the same functionality as before where, if the csv does not contain expected_tools_col_name, ["[]"] * len(df) is returned from get_column_data, the json decode succeeds and the subsequent list comprehension for tool in trimAndLoadJson(expected_tools_str) never runs.
Empty strings in the dataframe column won't be an issue as there's checks for empty strings before the JSON decode.
Actually on second thought, I have changed the get_column_data default back to "" to avoid lots of unnecessary JSON decode calls. If the column is missing, the empty string check will already skip any parsing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes #2564 by altering the parsing logic for loading goldens from CSV.
Context
Issue #2564 shows full repro code for this error.
Changes
deepeval/dataset/dataset.py:EvaluationDataset.add_goldens_from_csv_file: Instead of simply splitting the row value string by the provided delimiter, parse the entire row value string using the same logic asEvaluationDataset.add_test_cases_from_csv_filesinceEvaluationDataset.save_asserialises theToolCallobjects in this way.Tests
tests/test_core/test_datasets/goldens.csvto include example values fortools_calledandexpected_toolscolumns