Skip to content

Commit 3f1b56a

Browse files
heisenbergyeheisenbergye
authored
feat: support Claude 4 Interleaved thinking (beta) (#164)
1 parent 76a3614 commit 3f1b56a

File tree

5 files changed

+115
-3
lines changed

5 files changed

+115
-3
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ If you find this GitHub repository useful, please consider giving it a free star
2828
- [x] Support Cross-Region Inference
2929
- [x] Support Application Inference Profiles (**new**)
3030
- [x] Support Reasoning (**new**)
31+
- [x] Support Interleaved thinking (**new**)
3132

3233
Please check [Usage Guide](./docs/Usage.md) for more details about how to use the new APIs.
3334

‎docs/Usage.md

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ export OPENAI_BASE_URL=<API base url>
1515
- [Multimodal API](#multimodal-api)
1616
- [Tool Call](#tool-call)
1717
- [Reasoning](#reasoning)
18+
- [Interleaved thinking (beta)](#Interleaved thinking (beta))
1819

1920
## Models API
2021

@@ -135,6 +136,7 @@ print(doc_result[0][:5])
135136
**Example Request**
136137

137138
```bash
139+
curl $OPENAI_BASE_URL/chat/completions \
138140
curl $OPENAI_BASE_URL/chat/completions \
139141
-H "Content-Type: application/json" \
140142
-H "Authorization: Bearer $OPENAI_API_KEY" \
@@ -340,7 +342,6 @@ curl $OPENAI_BASE_URL/chat/completions \
340342
-d '{
341343
"model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
342344
"messages": [
343-
{
344345
"role": "user",
345346
"content": "which one is bigger, 3.9 or 3.11?"
346347
}
@@ -441,4 +442,55 @@ for chunk in response:
441442
reasoning_content += chunk.choices[0].delta.reasoning_content
442443
elif chunk.choices[0].delta.content:
443444
content += chunk.choices[0].delta.content
444-
```
445+
```
446+
447+
## Interleaved thinking (beta)
448+
449+
**Important Notice**: Please carefully review the following points before using reasoning mode for Chat completion API.
450+
451+
Extended thinking with tool use in Claude 4 models supports [interleaved thinking](https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html#claude-messages-extended-thinking-tool-use-interleaved) enables Claude 4 models to think between tool calls and run more sophisticated reasoning after receiving tool results. which is helpful for more complex agentic interactions.
452+
With interleaved thinking, the `budget_tokens` can exceed the `max_tokens` parameter because it represents the total budget across all thinking blocks within one assistant turn.
453+
454+
455+
**Example Request**
456+
457+
- Non-Streaming
458+
459+
```bash
460+
curl http://127.0.0.1:8000/api/v1/chat/completions \
461+
-H "Content-Type: application/json" \
462+
-H "Authorization: Bearer bedrock" \
463+
-d '{
464+
"model": "us.anthropic.claude-sonnet-4-20250514-v1:0",
465+
"max_tokens": 2048,
466+
"messages": [{
467+
"role": "user",
468+
"content": "有一天，一个女孩参加数学考试只得了 38 分。她心里对父亲的惩罚充满恐惧，于是偷偷把分数改成了 88 分。她的父亲看到试卷后，怒发冲冠，狠狠地给了她一巴掌，怒吼道：“你这 8 怎么一半是绿的一半是红的，你以为我是傻子吗？”女孩被打后，委屈地哭了起来，什么也没说。过了一会儿，父亲突然崩溃了。请问这位父亲为什么过一会崩溃了？"
469+
}],
470+
"extra_body": {
471+
"anthropic_beta": ["interleaved-thinking-2025-05-14"],
472+
"thinking": {"type": "enabled", "budget_tokens": 4096}
473+
}
474+
}'
475+
```
476+
477+
- Streaming
478+
479+
```bash
480+
curl http://127.0.0.1:8000/api/v1/chat/completions \
481+
-H "Content-Type: application/json" \
482+
-H "Authorization: Bearer bedrock" \
483+
-d '{
484+
"model": "us.anthropic.claude-sonnet-4-20250514-v1:0",
485+
"max_tokens": 2048,
486+
"messages": [{
487+
"role": "user",
488+
"content": "有一天，一个女孩参加数学考试只得了 38 分。她心里对父亲的惩罚充满恐惧，于是偷偷把分数改成了 88 分。她的父亲看到试卷后，怒发冲冠，狠狠地给了她一巴掌，怒吼道：“你这 8 怎么一半是绿的一半是红的，你以为我是傻子吗？”女孩被打后，委屈地哭了起来，什么也没说。过了一会儿，父亲突然崩溃了。请问这位父亲为什么过一会崩溃了？"
489+
}],
490+
"stream": true,
491+
"extra_body": {
492+
"anthropic_beta": ["interleaved-thinking-2025-05-14"],
493+
"thinking": {"type": "enabled", "budget_tokens": 4096}
494+
}
495+
}'
496+
```

‎docs/Usage_CN.md

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ export OPENAI_BASE_URL=<API base url>
1515
- [Multimodal API](#multimodal-api)
1616
- [Tool Call](#tool-call)
1717
- [Reasoning](#reasoning)
18+
- [Interleaved thinking (beta)](#Interleaved thinking (beta))
19+
1820

1921
## Models API
2022

@@ -440,4 +442,56 @@ for chunk in response:
440442
reasoning_content += chunk.choices[0].delta.reasoning_content
441443
elif chunk.choices[0].delta.content:
442444
content += chunk.choices[0].delta.content
443-
```
445+
```
446+
447+
## Interleaved thinking (beta)
448+
449+
**重要提示**：在使用 Chat Completion API 的推理模式（reasoning mode）前，请务必仔细阅读以下内容。
450+
451+
Claude 4 模型支持借助工具使用的扩展思维功能（Extended Thinking），其中包含交错思考（[interleaved thinking](https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html#claude-messages-extended-thinking-tool-use-interleaved) ）。该功能使 Claude 4 可以在多次调用工具之间进行思考，并在收到工具结果后执行更复杂的推理，这对处理更复杂的 Agentic AI 交互非常有帮助。
452+
453+
在交错思考模式下，budget_tokens 可以超过 max_tokens 参数，因为它代表一次助手回合中所有思考块的总 Token 预算。
454+
455+
456+
**Request 示例**
457+
458+
- Non-Streaming
459+
460+
```bash
461+
curl http://127.0.0.1:8000/api/v1/chat/completions \
462+
-H "Content-Type: application/json" \
463+
-H "Authorization: Bearer bedrock" \
464+
-d '{
465+
"model": "us.anthropic.claude-sonnet-4-20250514-v1:0",
466+
"max_tokens": 2048,
467+
"messages": [{
468+
"role": "user",
469+
"content": "有一天，一个女孩参加数学考试只得了 38 分。她心里对父亲的惩罚充满恐惧，于是偷偷把分数改成了 88 分。她的父亲看到试卷后，怒发冲冠，狠狠地给了她一巴掌，怒吼道：“你这 8 怎么一半是绿的一半是红的，你以为我是傻子吗？”女孩被打后，委屈地哭了起来，什么也没说。过了一会儿，父亲突然崩溃了。请问这位父亲为什么过一会崩溃了？"
470+
}],
471+
"extra_body": {
472+
"anthropic_beta": ["interleaved-thinking-2025-05-14"],
473+
"thinking": {"type": "enabled", "budget_tokens": 4096}
474+
}
475+
}'
476+
```
477+
478+
- Streaming
479+
480+
```bash
481+
curl http://127.0.0.1:8000/api/v1/chat/completions \
482+
-H "Content-Type: application/json" \
483+
-H "Authorization: Bearer bedrock" \
484+
-d '{
485+
"model": "us.anthropic.claude-sonnet-4-20250514-v1:0",
486+
"max_tokens": 2048,
487+
"messages": [{
488+
"role": "user",
489+
"content": "有一天，一个女孩参加数学考试只得了 38 分。她心里对父亲的惩罚充满恐惧，于是偷偷把分数改成了 88 分。她的父亲看到试卷后，怒发冲冠，狠狠地给了她一巴掌，怒吼道：“你这 8 怎么一半是绿的一半是红的，你以为我是傻子吗？”女孩被打后，委屈地哭了起来，什么也没说。过了一会儿，父亲突然崩溃了。请问这位父亲为什么过一会崩溃了？"
490+
}],
491+
"stream": true,
492+
"extra_body": {
493+
"anthropic_beta": ["interleaved-thinking-2025-05-14"],
494+
"thinking": {"type": "enabled", "budget_tokens": 4096}
495+
}
496+
}'
497+
```

‎src/api/models/bedrock.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -567,6 +567,10 @@ def _parse_request(self, chat_request: ChatRequest) -> dict:
567567
assert "function" in chat_request.tool_choice
568568
tool_config["toolChoice"] = {"tool": {"name": chat_request.tool_choice["function"].get("name", "")}}
569569
args["toolConfig"] = tool_config
570+
# add Additional fields to enable extend thinking
571+
if chat_request.extra_body:
572+
# reasoning_config will not be used
573+
args["additionalModelRequestFields"] = chat_request.extra_body
570574
return args
571575

572576
def _create_response(

‎src/api/schema.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ class ChatRequest(BaseModel):
107107
tools: list[Tool] | None = None
108108
tool_choice: str | object = "auto"
109109
stop: list[str] | str | None = None
110+
extra_body: dict | None = None
110111

111112

112113
class Usage(BaseModel):

0 commit comments

Comments
 (0)