fix(openai): move image detail property inside image_url object

[Abdul121104]

Fixes #2221

Problem

The detail property for image content was placed at the top level instead of inside the image_url object, causing OpenAI API to return 400 errors with message: "Invalid chat format. Unexpected keys in a message content image dict."

Solution

• Move detail property inside image_url object in MessageContentImageDetail type definition
• This matches OpenAI's expected API format where detail is nested inside image_url

Changes Made

1. Type Definition Fix (packages/core/src/llms/type.ts):

   • Updated MessageContentImageDetail type to place detail inside image_url object
   • Before: detail was at top level alongside image_url
   • After: detail is inside image_url object

2. Test Coverage (packages/providers/openai/tests/llm.test.ts):

   • Added test case for image messages WITH detail
   • Added test case for image messages WITHOUT detail
   • Both tests verify the correct structure for OpenAI API

Testing Done

Manual Verification: Confirmed the new structure matches OpenAI's API requirements
TypeScript Compilation: Verified no syntax errors in the changes
Test Cases Added: Comprehensive test coverage for both scenarios

Before (Broken):

{
  type: "image_url",
  image_url: { url: "data:image/jpeg;base64,test" },
  detail: "high"  // rong location - causes OpenAI 400 error
}

After

{
  type: "image_url",
  image_url: {
    url: "data:image/jpeg;base64,test",
    detail: "high"  // Correct location - matches OpenAI API
  }
}

Note
Local test environment has dependency build issues preventing test execution

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3818cf6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[redbmk]

Is this not the llamaindex abstraction? We wouldn't want to change the input type, just whatever it gets transformed to

[redbmk]

yeah I would leave the type the same so this would be the necessary transform. It might help to add some similar tests for OpenAIResponses to show it working for both