test(python): add e2e tests for llm.call for multiple image formats#2529
Open
samyak1729 wants to merge 9 commits intoMirascope:mainfrom
Open
test(python): add e2e tests for llm.call for multiple image formats#2529samyak1729 wants to merge 9 commits intoMirascope:mainfrom
samyak1729 wants to merge 9 commits intoMirascope:mainfrom
Conversation
samyak1729
changed the title
- Regenerated all Anthropic and OpenAI cassettes with valid API keys - Skip OpenAI image URL tests (external URLs blocked by OpenAI) - Added missing snapshots for all providers and image formats Co-authored-by: samyak <samyak5jan@gmail.com>
teamdandelion
approved these changes
Feb 10, 2026
…nations - Replace pytest.skip() calls with custom parametrization that excludes unsupported model/mime-type combinations entirely - This prevents the snapshot fixture from importing empty snapshot modules that cause inline_snapshot AssertionError at import time - Delete empty snapshot files for unsupported combinations - Delete corresponding VCR cassette files for unsupported combinations - Fixes all 'ERROR at setup' failures in Python e2e image tests
samyak1729
commented
Feb 13, 2026
Author
samyak1729
left a comment
samyak1729
left a comment
There was a problem hiding this comment.
Tested locally with
CI=true uv run pytest --cov --cov-config=.coveragerc --cov-report=xml --cov-report=term-missing.
Everything looks great and tests are passing with 100% coverage. Thanks for the fix!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add E2E tests for
llm.callwith multiple image formatsSummary
Expands the existing image E2E tests (
test_call_with_image_contentandtest_call_with_image_url) from a single PNG-only test to parametrized tests covering all supported MIME types: PNG, JPEG, WebP, GIF, HEIC, and HEIF.Changes
test_call_with_image.py: Parametrize bothtest_call_with_image_contentandtest_call_with_image_urloverMIME_TYPES, with per-provider skip logic for unsupported formats (e.g., HEIC/HEIF on OpenAI/Anthropic, GIF on Google). Added test asset paths and URLs for each format.conftest.py: Generalized the cassette/snapshot path generation to support arbitrary extra parametrized segments (mime_type,formatting_mode) via_get_extra_segments, replacing the previousformatting_mode-only logic.test.jpg,test.webp,test.gif,test.heic,test.heif) and renamedwikipedia.png→test.png.Testing
I only had access to a Google API key while working on this, so VCR cassettes and inline snapshots have been generated for
google/gemini-2.5-flash. Everything is passing on that end.However, the Anthropic and OpenAI cassettes still need to be recorded. If maintainers with those keys could run the following, that should take care of it:
The test logic itself is provider-agnostic, so I don't expect any surprises.