Skip to content

Commit 591668c

Browse files
committed
Fix streaming response token count report
1 parent eb846e5 commit 591668c

File tree

3 files changed

+87
-6
lines changed

3 files changed

+87
-6
lines changed

lib/ruby_llm/providers/ollama/chat.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,9 @@ def parse_completion_response(response)
2828
Message.new(
2929
role: :assistant,
3030
content: data.dig('message', 'content'),
31+
model_id: data['model'],
3132
input_tokens: data['prompt_eval_count'].to_i,
3233
output_tokens: data['eval_count'].to_i,
33-
model_id: data['model'],
3434
tool_calls: parse_tool_calls(data.dig('message', 'tool_calls'))
3535
)
3636
end

lib/ruby_llm/providers/ollama/streaming.rb

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,9 @@ def build_chunk(data)
3131
role: :assistant,
3232
content: data.dig('message', 'content'),
3333
model_id: data['model'],
34-
tool_calls: parse_tool_calls(data.dig('message', 'tool_calls')),
35-
36-
# NOTE: unavailable in the response - https://ollama.readthedocs.io/en/api/#streaming-responses
37-
input_tokens: nil,
38-
output_tokens: nil
34+
input_tokens: data['prompt_eval_count'],
35+
output_tokens: data['eval_count'],
36+
tool_calls: parse_tool_calls(data.dig('message', 'tool_calls'))
3937
)
4038
end
4139

spec/fixtures/vcr_cassettes/chat_streaming_responses_ollama_llama3_1_8b_reports_consistent_token_counts_compared_to_non-streaming.yml

Lines changed: 83 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)