Skip to content

Conversation

@windreamer
Copy link
Collaborator

@windreamer windreamer commented Oct 29, 2025

Motivation

OpenAI-compatible response_format={"type": "json_object"} is still part of the official spec and is explicitly requested by users.
lmdeploy currently rejects this value with
ValueError: unsupported format type: json_object.
We need to accept it and force the model to emit any valid JSON object without requiring the caller to supply a schema.

Modification

  1. GuidedDecodingManager

    • Treat json_object as a built-in schema:
      '{"type": "object", "additionalProperties": true}'
    • Dispatch it to the existing JSON-schema processor.
  2. TurboMind backend

    • Add json_object branch when extracting the schema from response_format.
    • Compile the same permissive object schema with the grammar compiler.
  3. OpenAI-API server

    • Remove the outdated TODO comment; no extra logic is needed here after the core change.
  4. Tests

    • Extend the guided-decoding matrix to cover json_object, json_schema, regex_schema, and the un-guided case.
    • Validate json_object responses with the same permissive schema used in the engine.

With this patch, clients can use
response_format={"type": "json_object"} and receive guaranteed valid JSON objects.

@windreamer windreamer force-pushed the support_json_object_format branch 7 times, most recently from c83865e to ad577b3 Compare October 29, 2025 11:29
@windreamer windreamer force-pushed the support_json_object_format branch from ad577b3 to 11a0c11 Compare October 30, 2025 01:28
@windreamer windreamer linked an issue Oct 30, 2025 that may be closed by this pull request
3 tasks
@windreamer windreamer marked this pull request as ready for review October 30, 2025 01:33
@lvhan028 lvhan028 requested a review from grimoire November 1, 2025 03:14
Copy link
Collaborator

@grimoire grimoire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lvhan028 lvhan028 merged commit 2636336 into InternLM:main Nov 1, 2025
5 checks passed
@windreamer windreamer deleted the support_json_object_format branch November 1, 2025 23:41
@jingyibo123
Copy link
Contributor

@windreamer How does this relate to the usage of openai.client.beta.chat.completions.parse along with pydantic model for structured output?
Will that work out of the box after this PR, or does that require additional adaptation?

@windreamer
Copy link
Collaborator Author

openai.client.beta.chat.completions.parse

I haven't dug into the implementation details of OpenAI SDK. But from my point of view, the parse API is just a grammar-sugar on top of structure output (aka guided decoding). This should be working out of box with this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] response_format 不支持 {"type": "json_object"}

4 participants