Skip to content

Fix SSE parser to handle delta.reasoning tokens from reasoning models - #594

Closed
flexwang wants to merge 1 commit into
kubernetes-sigs:mainfrom
flexwang:fix-reasoning-token-parsing
Closed

Fix SSE parser to handle delta.reasoning tokens from reasoning models#594
flexwang wants to merge 1 commit into
kubernetes-sigs:mainfrom
flexwang:fix-reasoning-token-parsing

Conversation

@flexwang

@flexwang flexwang commented Jul 2, 2026

Copy link
Copy Markdown

Summary

  • The extract_content lambda in the chat SSE parser only read delta.content, causing reasoning models (GLM-5.2, DeepSeek-R1) to report output_len≈0 and meaningless TPOT/ITL/TTFT metrics
  • Updated the lambda to fall through content → reasoning → reasoning_content so all generated tokens are counted
  • Added tests covering both delta.reasoning and delta.reasoning_content field names

Fixes #592

Test plan

  • Existing streaming parser tests pass
  • New test: test_parse_sse_stream_reasoning_tokens — verifies delta.reasoning tokens are extracted
  • New test: test_parse_sse_stream_reasoning_content_field — verifies delta.reasoning_content (DeepSeek-R1 style) tokens are extracted
  • Manual validation against a reasoning model endpoint (GLM-5.2 or DeepSeek-R1)

🤖 Generated with Claude Code

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jul 2, 2026
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: flexwang
Once this PR has been reviewed and has the lgtm label, please assign wangchen615 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown

Welcome @flexwang!

It looks like this is your first PR to kubernetes-sigs/inference-perf 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/inference-perf has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 2, 2026

Copy link
Copy Markdown

CLA Not Signed

@kubernetes-prow kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 2, 2026
The content extraction in the chat SSE parser only read delta.content,
causing reasoning models (GLM-5.2, DeepSeek-R1) to report output_len≈0
and meaningless TPOT/ITL metrics. Now also extracts delta.reasoning and
delta.reasoning_content so all generated tokens are counted.

Related: kubernetes-sigs#592
@flexwang
flexwang force-pushed the fix-reasoning-token-parsing branch from 1da0a37 to 1906e9e Compare July 2, 2026 18:38
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jul 2, 2026
@Bslabe123

Copy link
Copy Markdown
Contributor

@flexwang Is this one ready for review?

@Bslabe123

Copy link
Copy Markdown
Contributor

Closing due to inactivity

@Bslabe123 Bslabe123 closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: SSE parser doesn't handle delta.reasoning tokens from reasoning models

2 participants