Fix SSE parser to handle delta.reasoning tokens from reasoning models - #594
Fix SSE parser to handle delta.reasoning tokens from reasoning models#594flexwang wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: flexwang The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @flexwang! |
|
The content extraction in the chat SSE parser only read delta.content, causing reasoning models (GLM-5.2, DeepSeek-R1) to report output_len≈0 and meaningless TPOT/ITL metrics. Now also extracts delta.reasoning and delta.reasoning_content so all generated tokens are counted. Related: kubernetes-sigs#592
1da0a37 to
1906e9e
Compare
|
@flexwang Is this one ready for review? |
|
Closing due to inactivity |
Summary
extract_contentlambda in the chat SSE parser only readdelta.content, causing reasoning models (GLM-5.2, DeepSeek-R1) to reportoutput_len≈0and meaningless TPOT/ITL/TTFT metricscontent → reasoning → reasoning_contentso all generated tokens are counteddelta.reasoninganddelta.reasoning_contentfield namesFixes #592
Test plan
test_parse_sse_stream_reasoning_tokens— verifiesdelta.reasoningtokens are extractedtest_parse_sse_stream_reasoning_content_field— verifiesdelta.reasoning_content(DeepSeek-R1 style) tokens are extracted🤖 Generated with Claude Code