Show token usage

[sp1r1t]

When using LLMs through an API I always have to keep an eye on the cost produced. To get a better feel I would like to see the amount of tokens used by a request.

At the moment I can set a token maximum, which is very nice to cap the costs of a request, but it does not give me actual usage information.

To my knowledge, at least for openai, the API returns the token counts, so it should be possible to display it.

This could be shown in the gptel-mode status bar or added at the end of the the LLM response. E.g. In: 10, Out: 100, Total: 110.

It could be switched on in gptel-menu by t for token, or by the the variable gptel-show-token-usage.

I've not encountered this feature in any gpt client so far, but I would love to see it.

[karthink]

at least for openai, the API returns the token counts

Yes, most backends return token counts, they just do it differently. For example, Claude will return something like this with a response:

{
  "usage": {
    "input_tokens": 7213,
    "cache_creation_input_tokens": 7864,
    "cache_read_input_tokens": 14340,
    "output_tokens": 2336
  }
}

As you can see, this is not easy to interpret, and harder still to display in the UI. I need to figure out how to present this information in a meaningful way that isn't confusing to the user.

[puhsu]

@karthink

As you can see, this is not easy to interpret, and harder still to display in the UI. I need to figure out how to present this information in a meaningful way that isn't confusing to the user.

Maybe something displaying just input and output tokens is an ok baseline. [Usage: ↑5.7k ↓623], or something along these lines. Maybe also add [Cache Read: 50.2k/nil] to monitor cache reads too (in practice I look at this much less often).

I am not deeply familiar with gptel internals, especially the ui parts. I've seen that you recently refactored some of the gptel-request code to support usage tracking in the future, thats one of the features I really miss right now from gptel, very eager to test and help here.

If the ui header change is too broad and not for everyone I can imagine using some bespoke function that just queries these aggregated stats for a particular session may be sufficient for my use-cases (could you point to where do I look for the usage info, or where would I in the future?)

[karthink]

Maybe something displaying just input and output tokens is an ok baseline. [Usage: ↑5.7k ↓623], or something along these lines. Maybe also add [Cache Read: 50.2k/nil] to monitor cache reads too (in practice I look at this much less often).

I am thinking this can go where the status readout (Ready/Waiting/Error) does after a successful request. In any case, adding it to the header-line is not a problem. We'll also have to think about the case of gptel-use-header-line set to nil, and some way to optionally indicate this info outside of gptel-mode.

I am not deeply familiar with gptel internals, especially the ui parts. I've seen that you recently refactored some of the gptel-request code to support usage tracking in the future, thats one of the features I really miss right now from gptel, very eager to test and help here.

I'm looking into capturing this information in the request context when the request ends. Here's a progress update (not yet pushed):

• Anthropic backend: implemented.
• OpenAI backend: Token information is not provided with streaming responses, not sure what to do about that. Capturing from non-streaming responses is implemented.
• Gemini backend: Caching information is not provided, and I'm still trying to understand how to interpret the counts it provides.
• Ollama backend: Not yet
• Custom OpenAI compatible backends (like Openrouter, llama-cpp, vLLM, Deepseek): Not yet
• OpenAI responses backend: This backend itself is not yet supported by gptel, there are open PRs.

I can push the current state if you're interested in experimenting with the implemented backends and the UI.

If the ui header change is too broad and not for everyone I can imagine using some bespoke function that just queries these aggregated stats for a particular session may be sufficient for my use-cases (could you point to where do I look for the usage info, or where would I in the future?)

In the current implementation, no running total for the "session" or buffer exists. After each call to gptel-send, you can extract the tokens from the last completed request via

(plist-get (gptel-fsm-info gptel--fsm-last) :tokens)

It will be a plist of the form (:input 2249 :output 445 :cached 0 :cache 1004). :cache is the cache-creation token count, :cached is cache-read tokens. Not all backends will supply the latter two.

You or gptel will have to maintain a session-local running total separately. Session-local is usually the same as buffer-local, which is very straightforward, but there might be exceptions depending on how you use gptel (with response input/output redirection, etc).

gptel--fsm-last is the state machine from the last completed multi-turn gptel-send command in each buffer. If there are two or more simultaneous requests in a buffer/session, the above method will not be accurate any more (and the running total might be completely off). We'll have to find another way to access the state machine for each request from the Emacs buffer.

Token costs for a model are available via (get gptel-model :input-cost) and (get gptel-model :output-cost) as the cost in USD per 1M tokens. There is no :cached-input-cost property yet, that metadata will have to be added to model specs that support it.

---

I'll work on the backend stuff to capture this information. Happy to accept a PR for integration into the UI. If you do raise one, please remember to account for gptel-use-header-line and non-gptel-mode usage.

[karthink]

@pushu update: I've pushed the token tracking code for the Anthropic backend (only) in a4cc9a1.

[karthink]

Update: I've pushed the token tracking code for the Gemini backend in fadfaf8.

[puhsu]

Nice! I'll try this out during the weekend and try experimenting with the ui, thanks!

The multiple request tracking does seem more challenging, I do find myself using multiple concurrent requests with different follow ups in one chat buffer sometimes. Is some accumulation here possible?

[karthink]

The multiple request tracking does seem more challenging, I do find myself using multiple concurrent requests with different follow ups in one chat buffer sometimes. Is some accumulation here possible?

Sure. The accumulation will have to run as part of the request pipeline at the end of each request, instead of being a separate function that runs at top-level (at which point you've lost the request context except for whatever gptel--fsm-last retains). You can add a handler to gptel-send--handlers for the DONE and ERRS states to integrate it into the pipeline. This handler can add the token counts to the buffer-local token/cost counting variable. See the other handlers in gptel-send--handlers to understand how to extract this information.

[karthink]

@pushu Added token tracking code for OpenAI, OpenAI-compatible backends like Openrouter/llamacpp, Bedrock and Kagi.

That covers all backends.

[puhsu]

@karthink
I've started experimenting with the UI and vibe-coded some initial prototype, seems to work even with concurrent requests - for now it just accumulates all inputs outputs and cache counts.

I'll take time to understand how it fits into gptel and make it correct(-ish) and fix the UI a bit (use a different face at least). Then I can make a PR for the usage tracking.

Also thinking about how this should fit with the #1262 (some agent scaffolds combine both the usage estimates and some heuristic context yet to be send estimation). What do you think is best in gptel for this?

[karthink]

That's a good start!

Re: concurrent requests: the token counts will be wrong.

(some agent scaffolds combine both the usage estimates and some heuristic context yet to be send estimation). What do you think is best in gptel for this?

If the UI has enough room for the running token count and the estimated count for the next request to be displayed independently without confusion, I would prefer that. I'm not sure how to interpret a number that combines the two. How do you read that indicator?

[puhsu]

@karthink

Re: concurrent requests: the token counts will be wrong.

In my current implementation I just track cummulative total usage in a buffer local variable. I think that total usage counts for the buffer would be correct, am I missing somethinhg?

I'm not sure how to interpret a number that combines the two. How do you read that indicator?

Yeah, the combined number does not exist I believe (I was thinking about usage stats being used as a way to better estimate the context size for the buffer). pi by default reports two stats: total session usage and the context percentage. The total usage I think is what's enabled by your commits (+ some additional tracking logic).

The context percentage is harder to implement in gptel (as per #1258, gptel is much more dynamic). The more "linear" agent harness use-case like pi combines usage reports with a heuristic estimate just for the last message. They also have some extensibility and hooks that may alter prompts I think, idk how they handle the estimates wrt that.

Do you think its ok to just support the usage for now in some clean and easy way. Or should the context size estimate be combined with the usage tracking logic? (Is there some abstraction that could better track "session history" here?)

Another question I have about the correct total usage estimate implementation is to which object should I attribute and accumulate the total usage to? It's easy for a separate gptel-mode buffer, should it be supported only there? Do you have ideas on how to make it more general? (I dont use gptel outside of the gptel-mode buffer much, so have almost no idea what persists and where when its invoked anywhere else, is the answer always a buffer-local variable?)