Skip to content

fix: batched upload permanently truncates in-memory test run#2597

Open
aerosta wants to merge 1 commit intoconfident-ai:mainfrom
aerosta:fix/upload-truncates-run
Open

fix: batched upload permanently truncates in-memory test run#2597
aerosta wants to merge 1 commit intoconfident-ai:mainfrom
aerosta:fix/upload-truncates-run

Conversation

@aerosta
Copy link
Copy Markdown
Contributor

@aerosta aerosta commented Apr 4, 2026

Summary

Large test runs are uploaded in batches, but post_test_run() mutates the caller's in-memory TestRun while doing so. It replaces test_run.test_cases or test_run.conversational_test_cases with only the first batch and clears test_run.prompts, but never restores that state afterward.

This means any code that reads the same TestRun object after upload can see a partial run instead of the full evaluated result.

This change snapshots the original state before batching and restores it in a finally block, so the full in-memory run is preserved whether upload succeeds or fails.

Changes

  • deepeval/test_run/test_run.py: preserve and restore the original run state during batched upload
  • tests/test_core/test_run/test_run_manager.py: add regression coverage for successful batched upload and mid-upload failure

Test plan

DEEPEVAL_TELEMETRY_OPT_OUT=1 python -m pytest tests/test_core/test_run/test_run_manager.py -q --tb=short

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 4, 2026

@aerosta is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@aerosta aerosta force-pushed the fix/upload-truncates-run branch from 53d9a72 to 5bb5168 Compare April 4, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant