feat(bedrock): S3 URL pass-through for multimodal content blocks

[ryanh-ai]

Summary

When users provide s3:// URLs in OpenAI-format messages, LiteLLM now maps them directly to Bedrock's native s3Location content blocks — no downloading or base64-encoding required.

This enables efficient multimodal workflows where large images, documents, and videos are already in S3.

Closes #21746

Changes

Core (factory.py)

• BedrockImageProcessor.process_image_sync/async — new first branch: s3:// → _create_s3_bedrock_block()
• _create_s3_bedrock_block() — routes to ImageBlock/DocumentBlock/VideoBlock based on file extension
• _extract_video_url() — helper for video_url content type
• Model guard: checks supports_s3_input flag; raises clear ValueError for unsupported models
• video_url content type handling in both sync and async message transformation paths
• All existing base64/HTTPS paths are completely untouched

Types (bedrock.py, utils.py)

• S3Location TypedDict with required uri and optional bucketOwner
• SourceBlock changed to total=False for mutual exclusivity (bytes vs s3Location)
• supports_s3_input field on ModelInfo

Model config (model_prices_and_context_window.json)

• supports_s3_input: true added to 25 vision-capable Amazon Nova models
• Config-driven approach per AGENTS.md guidelines (no hardcoded model checks)

Utility (utils.py)

• supports_s3_input() helper following existing _supports_factory pattern

Tests

• 41 unit tests in tests/test_litellm/llms/bedrock/chat/test_s3_content_blocks.py — all block types, edge cases, async, model guard, HTTPS regression
• 13 integration tests in tests/llm_translation/test_bedrock_s3_content.py — real Bedrock requests with Nova Lite/Pro/2-Lite (properly skipped without AWS creds)

Usage

# Image from S3
resp = litellm.completion(
    model="bedrock/us.amazon.nova-lite-v1:0",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image"},
            {"type": "image_url", "image_url": {"url": "s3://my-bucket/photo.jpg"}},
        ],
    }],
)

# Document from S3
resp = litellm.completion(
    model="bedrock/us.amazon.nova-pro-v1:0",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Summarize this document"},
            {"type": "file", "file": {"file_data": "s3://my-bucket/report.pdf"}},
        ],
    }],
)

# Video from S3
resp = litellm.completion(
    model="bedrock/us.amazon.nova-lite-v1:0",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this video"},
            {"type": "video_url", "video_url": {"url": "s3://my-bucket/demo.mp4"}},
        ],
    }],
)

Test results

• 460 bedrock unit tests pass
• 41 S3 unit tests pass
• 13 integration tests pass (live Bedrock + S3, Nova Lite/Pro/2-Lite)
• Pre-commit clean: pyright (0 errors), isort, flake8, black, ruff

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Mar 21, 2026 4:52am

[ryanh-ai]

@greptileai

[greptile-apps]

Greptile Summary

This PR adds S3 URI pass-through support for Bedrock's Converse API, mapping s3:// URLs in OpenAI-format messages directly to Bedrock s3Location content blocks for images, documents, and videos — avoiding the need to download and base64-encode large files. The implementation is config-driven (flag in model_prices_and_context_window.json) and follows the established _supports_factory pattern.

Key changes:

• BedrockImageProcessor.process_image_sync/async — new s3:// branch with per-model guard via supports_s3_input()
• _create_s3_bedrock_block() — routes to ImageBlock / DocumentBlock / VideoBlock by file extension; re-uses existing AmazonConverseConfig format lists
• S3Location TypedDict added to litellm/types/llms/bedrock.py; SourceBlock changed to total=False
• supports_s3_input: true added to 25 Bedrock-Converse Nova model entries; Nova Micro and non-Bedrock providers (amazon_nova, vercel_ai_gateway) are correctly excluded
• Issue: Three existing call sites — tool-result message processing (~line 4047), async assistant content (~line 4684), and sync assistant content (~line 5080) — invoke process_image_sync/process_image_async without model or custom_llm_provider. Because the guard is conditioned on if model and ..., an s3:// URL in these paths bypasses the check and produces a raw s3Location block regardless of the model in use. For unsupported models this silently passes an invalid block to Bedrock rather than raising the intended ValueError.
• Minor: MIME-type subtype extraction in _create_s3_bedrock_block does not map non-trivial subtypes (e.g. video/quicktime → "quicktime" rather than "mov"), which would raise a misleading ValueError for valid MOV content passed with its canonical MIME type.
• Integration tests are correctly placed under tests/local_testing/ and guarded by skipif; the autouse=True fixture that patches supports_s3_input is no longer necessary now that the JSON flag ships in this same PR.

Confidence Score: 3/5

• Core happy-path is solid but the model guard has gaps in tool-result and assistant-content paths that can produce confusing Bedrock API errors for unsupported models.
• The new S3 branch in BedrockImageProcessor is well-implemented and tested for user-message paths. However, three pre-existing call sites that now reach the S3 code path do not thread model info through, silently bypassing the guard. For a user on an unsupported model who sends an s3:// URL inside a tool result or assistant turn, they will receive a raw Bedrock API error rather than LiteLLM's clear validation message. The unit tests only cover user-message paths, leaving the inconsistency untested. The MIME subtype mapping gap is a lesser concern but could confuse users who pass canonical video MIME types as the format override.
• litellm/litellm_core_utils/prompt_templates/factory.py — specifically the three call sites at ~line 4047 (tool results), ~line 4684 (async assistant content), and ~line 5080 (sync assistant content) that invoke process_image_sync/async without model context.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/prompt_templates/factory.py	Core S3 pass-through logic added to BedrockImageProcessor. New helpers (_extract_video_url, _get_bedrock_format_from_extension, _create_s3_bedrock_block) and model guard are wired into user-message paths, but tool-result and assistant-content call sites at lines ~4047, ~4684, ~5080 still call process_image_sync/async without model info, silently bypassing the guard for unsupported models.
litellm/types/llms/bedrock.py	Adds S3Location TypedDict and makes SourceBlock total=False to allow mutual exclusivity between bytes and s3Location fields. Clean type additions with appropriate Required annotations.
model_prices_and_context_window.json	Adds supports_s3_input: true to 25 Bedrock-provider Nova model entries (bedrock_converse provider only). Nova Micro models and non-Bedrock providers (amazon_nova, vercel_ai_gateway) are correctly excluded. Also removes a duplicate vertex_ai/gemini-embedding-2-preview key.
tests/local_testing/test_bedrock_s3_content.py	Integration tests correctly placed in tests/local_testing/ and guarded by @pytest.mark.skipif requiring AWS credentials. However, the autouse=True _ensure_s3_input_flag fixture patches supports_s3_input globally for all tests, which can mask regressions in the real JSON-backed implementation since the JSON flag is already in this PR.

Sequence Diagram

sequenceDiagram
    participant User as Caller
    participant BCMP as _bedrock_converse_messages_pt
    participant BIP as BedrockImageProcessor
    participant Guard as supports_s3_input
    participant Builder as _create_s3_bedrock_block
    participant AWS as Bedrock Converse API

    User->>BCMP: messages with s3:// image_url
    BCMP->>BIP: process_image_sync(url, model, provider)

    alt url starts with s3://
        BIP->>Guard: supports_s3_input(model, provider)
        Guard-->>BIP: true or false from JSON config
        alt unsupported model
            BIP-->>BCMP: ValueError model does not support s3
        else supported Nova model
            BIP->>Builder: _create_s3_bedrock_block(s3_url, format)
            Builder-->>BIP: BedrockContentBlock with s3Location
            BIP-->>BCMP: BedrockContentBlock
            BCMP-->>AWS: Converse request with s3Location block
        end
    else base64 or https url
        BIP->>BIP: existing download and encode path unchanged
        BIP-->>BCMP: BedrockContentBlock with inline bytes
        BCMP-->>AWS: Converse request with inline bytes
    end

    Note over BIP: tool_result path at line 4047 and assistant<br/>content paths at lines 4684 and 5080 call<br/>process_image_sync with model=None bypassing the guard

Loading

Comments Outside Diff (3)

1. litellm/litellm_core_utils/prompt_templates/factory.py, line 4047-4050 (link)

   Model guard silently bypassed in tool-result and assistant message paths

   process_image_sync and process_image_async are called here (and at the assistant-content paths at lines ~4684 and ~5080) without passing model or custom_llm_provider. Because the guard inside those methods is gated on if model and not supports_s3_input(...), a None model skips the check entirely.

   This creates an inconsistency:

   • An s3:// URL in a user message → raises a clear ValueError for an unsupported model (e.g. Claude 3 on Bedrock).
   • The same s3:// URL in a tool result (here) or assistant content (lines ~4684, ~5080) → silently builds an s3Location block that Bedrock will reject with a cryptic API-level error.

   All three call sites need the model information threaded through:

   # line ~4047 — tool result path
   _block: BedrockContentBlock = BedrockImageProcessor.process_image_sync(
       image_url=image_url,
       format=format,
       model=model,              # add
       custom_llm_provider=llm_provider,  # add
   )

   # line ~4684 — async assistant content
   assistants_part = await BedrockImageProcessor.process_image_async(
       image_url=image_url,
       model=model,              # add
       custom_llm_provider=llm_provider,  # add
   )

   # line ~5080 — sync assistant content
   assistants_part = BedrockImageProcessor.process_image_sync(  # type: ignore
       image_url=image_url,
       model=model,              # add
       custom_llm_provider=llm_provider,  # add
   )

   This requires checking that model and llm_provider are in scope at each call site (they are in the functions that contain these blocks), and updating the function signatures if needed to accept and forward them.

2. litellm/litellm_core_utils/prompt_templates/factory.py, line 3731-3740 (link)

   S3 URL format dispatch relies on incomplete MIME-type subtype mapping

   When users pass an explicit format override that is a full MIME type (e.g. "video/quicktime" for .mov files), the code strips the subtype:

   raw_format = format.split("/")[-1] if "/" in format else format
   bedrock_format = cls._get_bedrock_format_from_extension(raw_format.lower())

   This produces "quicktime" for video/quicktime, which is not in get_supported_video_types() (["mp4", "mov", "mkv", "webm", ...]). As a result:

   • s3://bucket/clip.mov with format="video/quicktime" raises ValueError("Unsupported file format 'quicktime'"), even though .mov files work fine via extension detection.
   • Similarly, video/x-msvideo → "x-msvideo" (not "avi"), etc.

   _get_bedrock_format_from_extension only maps "jpg"→"jpeg" and "3gpp"→"3gp". A small MIME-subtype-to-Bedrock-format lookup table would close these gaps for common video MIME types:

   _MIME_SUBTYPE_TO_BEDROCK = {
       "quicktime": "mov",
       "x-msvideo": "avi",
       "x-matroska": "mkv",
       "mpeg": "mpeg",
   }

   Rule Used: What: Avoid writing provider-specific code outside... (source)

3. tests/local_testing/test_bedrock_s3_content.py, line 900-920 (link)

   Autouse fixture with hardcoded "nova" string-match may mask future regressions

   The _ensure_s3_input_flag fixture is autouse=True and patches supports_s3_input with a hardcoded lambda that matches "nova" by substring. Because the fixture applies to all tests in the file, including test_should_reject_s3_url_for_unsupported_model (which verifies Claude 3 is rejected), any future refactor that breaks the real supports_s3_input() lookup would still pass locally because the mock completely replaces the function.

   Additionally, now that this PR adds supports_s3_input: true to all the relevant Nova model entries in the JSON, the real function will resolve correctly from the bundled config at test time. The fixture is therefore unnecessary for tests run against the shipped codebase and should be removed, or at minimum scoped away from tests that deliberately test model rejection:

   @pytest.fixture  # remove autouse=True
   def _ensure_s3_input_flag(monkeypatch):
       ...

   Rule Used: What: prevent any tests from being added here that... (source)

_{Last reviewed commit: "feat(bedrock): S3 UR..."}

[ryanh-ai]

Additional research shows that Nova models are only models of chat type that support s3location argument, so we need a guard to prevent the s3 image_url for non nova models (and also guard for Nova Micro which doesn't support MM content).

[ryanh-ai]

@greptileai review

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing ryanh-ai:bedrock-s3-file-upload (b8aa8de) with main (d8e4fc4)}

[ryanh-ai]

Addressed both issues from the Greptile review:

1. HTTPS video_url fallthrough — _extract_video_url() now raises a clear ValueError for non-S3 video URLs: "Bedrock Converse API only supports s3:// URLs for video content." This prevents silent multi-GB downloads that would fail at the Bedrock API anyway.

2. Integration test placement — Note that tests/llm_translation/ already contains real-network Bedrock integration tests (e.g. test_bedrock_completion.py with litellm.completion() calls). Our tests follow the same pattern with proper @pytest.mark.skipif guards for missing AWS credentials. Happy to move them if maintainers prefer a different location.

Additional: 43 unit tests now (was 41), all passing. Full bedrock suite: 463 passed.

@greptileai review

[ryanh-ai]

@greptileai review

[krrishdholakia]

@greptile can you re-review this please

[ryanh-ai]

@greptile can you re-review this please

[ryanh-ai]

@greptile can you please review this and make sure to give the score you usually do? I need to see if the score has improved as I've addressed your feedback. Thanks!