|
| 1 | +# contextkit |
| 2 | + |
| 3 | +**Never hit context window limits again.** |
| 4 | + |
| 5 | +[](https://www.npmjs.com/package/contextkit) |
| 6 | +[](https://opensource.org/licenses/MIT) |
| 7 | +[](https://www.typescriptlang.org/) |
| 8 | + |
| 9 | +Intelligent conversation compaction for LLM applications. When your conversation history exceeds the context window, contextkit automatically summarizes old messages while preserving critical context — so your agent keeps working seamlessly. |
| 10 | + |
| 11 | +Extracted from battle-tested patterns powering production AI systems serving millions of users. |
| 12 | + |
| 13 | +``` |
| 14 | +Messages grow... ██████████████████████████████░░░░ 85% full |
| 15 | + │ |
| 16 | + contextkit |
| 17 | + │ |
| 18 | +After compaction ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 18% full |
| 19 | + ↑ summary ↑ preserved context |
| 20 | +``` |
| 21 | + |
| 22 | +## The Problem |
| 23 | + |
| 24 | +Every LLM app hits this wall: |
| 25 | + |
| 26 | +``` |
| 27 | +Error: This request would exceed the model's context window (200,000 tokens). |
| 28 | +``` |
| 29 | + |
| 30 | +Current solutions are terrible: |
| 31 | +- **Truncate oldest messages** → Agent forgets what it was doing |
| 32 | +- **Sliding window** → Same amnesia problem |
| 33 | +- **Crash/restart** → User loses all progress |
| 34 | +- **Hope conversations stay short** → They never do |
| 35 | + |
| 36 | +## The Solution |
| 37 | + |
| 38 | +contextkit uses a **3-tier compaction strategy** extracted from production AI systems: |
| 39 | + |
| 40 | +1. **Micro-compact** (free) — Trim old tool results without calling the LLM |
| 41 | +2. **Auto-compact** (smart) — Summarize old messages when approaching the limit |
| 42 | +3. **Circuit breaker** — Stop retrying after consecutive failures |
| 43 | + |
| 44 | +## Install |
| 45 | + |
| 46 | +```bash |
| 47 | +npm install contextkit |
| 48 | +``` |
| 49 | + |
| 50 | +**Zero dependencies.** Works with any LLM provider. |
| 51 | + |
| 52 | +## Quick Start |
| 53 | + |
| 54 | +```typescript |
| 55 | +import { createContextKit } from 'contextkit' |
| 56 | + |
| 57 | +const ctx = createContextKit({ |
| 58 | + contextWindowSize: 200_000, // Your model's context window |
| 59 | + |
| 60 | + // Plug in ANY LLM as the summarizer |
| 61 | + summarize: async (messages, prompt) => { |
| 62 | + const resp = await openai.chat.completions.create({ |
| 63 | + model: 'gpt-4o', |
| 64 | + messages: [ |
| 65 | + ...messages.map(m => ({ role: m.role, content: m.content as string })), |
| 66 | + { role: 'user', content: prompt }, |
| 67 | + ], |
| 68 | + }) |
| 69 | + return resp.choices[0].message.content |
| 70 | + }, |
| 71 | +}) |
| 72 | + |
| 73 | +// After every LLM response, check if compaction is needed: |
| 74 | +const { messages, compacted } = await ctx.autoCompact(conversationHistory) |
| 75 | +if (compacted) { |
| 76 | + conversationHistory = messages // Seamlessly replaced |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +## Provider Examples |
| 81 | + |
| 82 | +### OpenAI / GPT-4 |
| 83 | + |
| 84 | +```typescript |
| 85 | +const ctx = createContextKit({ |
| 86 | + contextWindowSize: 128_000, |
| 87 | + summarize: async (messages, prompt) => { |
| 88 | + const resp = await openai.chat.completions.create({ |
| 89 | + model: 'gpt-4o-mini', // Use a cheap model for summarization |
| 90 | + messages: [ |
| 91 | + ...messages.map(m => ({ role: m.role, content: m.content as string })), |
| 92 | + { role: 'user', content: prompt }, |
| 93 | + ], |
| 94 | + max_tokens: 20_000, |
| 95 | + }) |
| 96 | + return resp.choices[0].message.content ?? '' |
| 97 | + }, |
| 98 | +}) |
| 99 | +``` |
| 100 | + |
| 101 | +### Anthropic / Claude |
| 102 | + |
| 103 | +```typescript |
| 104 | +const ctx = createContextKit({ |
| 105 | + contextWindowSize: 200_000, |
| 106 | + summarize: async (messages, prompt) => { |
| 107 | + const resp = await anthropic.messages.create({ |
| 108 | + model: 'claude-haiku-4-5', // Use Haiku for cheap summarization |
| 109 | + max_tokens: 20_000, |
| 110 | + messages: [ |
| 111 | + ...messages.map(m => ({ role: m.role as 'user' | 'assistant', content: m.content as string })), |
| 112 | + { role: 'user', content: prompt }, |
| 113 | + ], |
| 114 | + }) |
| 115 | + return resp.content[0].type === 'text' ? resp.content[0].text : '' |
| 116 | + }, |
| 117 | +}) |
| 118 | +``` |
| 119 | + |
| 120 | +### Google Gemini |
| 121 | + |
| 122 | +```typescript |
| 123 | +const ctx = createContextKit({ |
| 124 | + contextWindowSize: 1_000_000, // Gemini's 1M context |
| 125 | + summarize: async (messages, prompt) => { |
| 126 | + const chat = model.startChat({ history: messages.map(m => ({ |
| 127 | + role: m.role === 'assistant' ? 'model' : 'user', |
| 128 | + parts: [{ text: m.content as string }], |
| 129 | + }))}) |
| 130 | + const result = await chat.sendMessage(prompt) |
| 131 | + return result.response.text() |
| 132 | + }, |
| 133 | +}) |
| 134 | +``` |
| 135 | + |
| 136 | +### Local Models (Ollama) |
| 137 | + |
| 138 | +```typescript |
| 139 | +const ctx = createContextKit({ |
| 140 | + contextWindowSize: 8_000, // Smaller window = compaction even more important |
| 141 | + summarize: async (messages, prompt) => { |
| 142 | + const resp = await fetch('http://localhost:11434/api/chat', { |
| 143 | + method: 'POST', |
| 144 | + body: JSON.stringify({ |
| 145 | + model: 'llama3', |
| 146 | + messages: [ |
| 147 | + ...messages.map(m => ({ role: m.role, content: m.content })), |
| 148 | + { role: 'user', content: prompt }, |
| 149 | + ], |
| 150 | + }), |
| 151 | + }) |
| 152 | + const data = await resp.json() |
| 153 | + return data.message.content |
| 154 | + }, |
| 155 | +}) |
| 156 | +``` |
| 157 | + |
| 158 | +## API Reference |
| 159 | + |
| 160 | +### `createContextKit(config)` |
| 161 | + |
| 162 | +| Config | Type | Default | Description | |
| 163 | +|--------|------|---------|-------------| |
| 164 | +| `contextWindowSize` | `number` | **required** | Model's context window in tokens | |
| 165 | +| `summarize` | `SummarizeFn` | — | LLM function for full compaction | |
| 166 | +| `maxOutputTokens` | `number` | 32000 | Max output tokens for the model | |
| 167 | +| `autoCompactBuffer` | `number` | 13000 | Buffer before auto-compact triggers | |
| 168 | +| `warningBuffer` | `number` | 20000 | Buffer for warning state | |
| 169 | +| `maxConsecutiveFailures` | `number` | 3 | Circuit breaker threshold | |
| 170 | +| `summaryMaxTokens` | `number` | 20000 | Max tokens for the summary | |
| 171 | +| `microCompactKeepRecent` | `number` | 5 | Recent tool results to keep | |
| 172 | +| `estimateTokens` | `(text: string) => number` | ~1 tok/4 chars | Custom token estimator | |
| 173 | +| `onAutoCompact` | `(result) => void` | — | Callback on compaction | |
| 174 | +| `onWarningStateChange` | `(state) => void` | — | Callback on warning change | |
| 175 | + |
| 176 | +### Methods |
| 177 | + |
| 178 | +| Method | Description | |
| 179 | +|--------|-------------| |
| 180 | +| `autoCompact(messages)` | Auto-detect and compact if needed. Call after every LLM response. | |
| 181 | +| `compact(messages, options?)` | Force full compaction with LLM summarization. | |
| 182 | +| `microCompact(messages)` | Free compaction: trim old tool results, no LLM call. | |
| 183 | +| `shouldCompact(messages)` | Check if messages exceed the auto-compact threshold. | |
| 184 | +| `estimateTokens(messages)` | Estimate token count for messages. | |
| 185 | +| `getWarningState(messages)` | Get context warning state: `ok` / `warning` / `error` / `critical` | |
| 186 | +| `getStats()` | Get current engine stats (thresholds, circuit breaker state). | |
| 187 | + |
| 188 | +### Standalone Utilities |
| 189 | + |
| 190 | +```typescript |
| 191 | +import { |
| 192 | + estimateTokens, // Estimate tokens for a string |
| 193 | + estimateMessageTokens, // Estimate tokens for a message |
| 194 | + estimateConversationTokens, // Estimate tokens for full conversation |
| 195 | + groupMessagesByRound, // Group messages by API round-trip |
| 196 | + microCompact, // Standalone micro-compaction |
| 197 | + stripImages, // Remove images from messages |
| 198 | + buildCompactPrompt, // Build the summarization prompt |
| 199 | +} from 'contextkit' |
| 200 | +``` |
| 201 | + |
| 202 | +## How It Works |
| 203 | + |
| 204 | +### Token Estimation |
| 205 | + |
| 206 | +contextkit uses a hybrid approach: |
| 207 | +- **API-reported tokens** when available (100% accurate) |
| 208 | +- **Character-based estimation** as fallback (~1 token per 4 chars, with 33% safety buffer) |
| 209 | + |
| 210 | +### The 9-Section Summary Prompt |
| 211 | + |
| 212 | +When compacting, contextkit instructs the LLM to produce a structured summary covering: |
| 213 | + |
| 214 | +1. **Primary Request and Intent** — What the user wants |
| 215 | +2. **Key Technical Concepts** — Domain knowledge established |
| 216 | +3. **Files and Code Sections** — Specific files, functions, code snippets |
| 217 | +4. **Errors and Fixes** — Problems encountered and solutions |
| 218 | +5. **Problem Solving** — Decisions made and reasoning |
| 219 | +6. **User Messages** — Every user request and correction |
| 220 | +7. **Pending Tasks** — Work still to do |
| 221 | +8. **Current Work** — Exact state right now |
| 222 | +9. **Next Step** — What should happen next |
| 223 | + |
| 224 | +This produces summaries that preserve enough context for seamless continuation. |
| 225 | + |
| 226 | +### Auto-Compact Flow |
| 227 | + |
| 228 | +``` |
| 229 | +After every LLM response: |
| 230 | + │ |
| 231 | + ├─ estimateTokens(messages) < threshold? → do nothing |
| 232 | + │ |
| 233 | + ├─ Try micro-compact (free) → enough space freed? → done |
| 234 | + │ |
| 235 | + ├─ Try full compact (LLM call) → success? → done |
| 236 | + │ └─ Prompt too long? → truncate oldest rounds, retry (max 3x) |
| 237 | + │ |
| 238 | + └─ All failed → circuit breaker increments |
| 239 | + └─ 3 consecutive failures → stop trying until reset |
| 240 | +``` |
| 241 | + |
| 242 | +## Dependencies |
| 243 | + |
| 244 | +**Zero runtime dependencies.** 4.8KB gzipped. |
| 245 | + |
| 246 | +## License |
| 247 | + |
| 248 | +[MIT](./LICENSE) |
0 commit comments