Context window clean up strategy

[alexewerlof]

Does the standard require a context overflow strategy?

Eg, is it a rolling window or are key points retained regardless of their position in session history?

Background:
I convert HTML to Markdown to add it to the context. But even that doesn't help for large pages and the Prompt API simply throws a quota exception.

My mitigation strategy is to break the text and feed to API in smaller chunks. But the question is if this method is pointless because if the LLM throws old stuff away, my mitigation is as good as just feeding it the last piece.

[tomayac]

Please refrain from using this repo dedicated to spec issues for support questions.

Here's your answer, quoted straight from the explainer:

It's possible to send a prompt that causes the context window to overflow. That is, consider a case where session.measureInputUsage(promptString) > session.inputQuota - session.inputUsage before calling session.prompt(promptString), and then the web developer calls session.prompt(promptString)` anyway. In such cases, the initial portions of the conversation with the language model will be removed, one prompt/response pair at a time, until enough tokens are available to process the new prompt. The exception is the system prompt, which is never removed.

Such overflows can be detected by listening for the "quotaoverflow" event on the session:

session.addEventListener("quotaoverflow", () => { console.log("We've gone past the quota, and some inputs will be dropped!"); });

[tomayac]

One mitigation strategy could be to use the Summarizer API to summarize the conversation and feed the summary of each prompt/response pair back into a new session via the initialPrompts option. Some loss may occur of course if the summary isn't detailed enough.

[alexewerlof]

Doesn't the summarization API have the same context length?
In my experience adding an entire page that's converted to Markdown can fail even an empty Prompt API session with 9.6k token quota.
I expect Summarization API to fail since it uses the same model underneath and has the same quota.

[tomayac]

They have different quota. Create sessions with both and check inputQuota. But also see https://issues.chromium.org/issues/456164614.