Skip to content

Commit ccc190e

Browse files
committed
Add native batch scanner acceleration
1 parent 1c084b4 commit ccc190e

9 files changed

Lines changed: 1247 additions & 276 deletions

File tree

README.md

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ histories, that is the multi-GB failure mode `cdxusage` avoids by streaming
2727
JSONL files, indexing compact per-file summaries, and reusing a small local
2828
cache.
2929

30+
On Linux hosts with GNU-compatible `perl` and `xargs -r`, cold full scans also
31+
use a native batch prefilter. Unsupported hosts and native helper failures fall
32+
back to the Node scanner automatically.
33+
3034
No SQLite. No daemon. No provider catalogs. No background service.
3135

3236
## Quick Start
@@ -72,6 +76,7 @@ global install, `npx -y github:Krablante/cdxusage`, or
7276
- pretty terminal tables and JSON output
7377
- date filters, timezone, locale, sorting, compact tables
7478
- automatic Codex home discovery on Linux, macOS, Windows, and WSL
79+
- Linux/GNU native batch prefilter with Node fallback
7580
- OpenAI/Codex pricing only, with missing non-OpenAI model prices reported
7681
- offline pricing fallback and disposable local caches
7782
- portable folder build with Linux/macOS shell, Windows CMD, and PowerShell launchers
@@ -149,25 +154,26 @@ npm run benchmark -- --since 2026-05-01 --upstream-timeout 25 --cdxusage-timeout
149154
The helper resolves `@ccusage/codex@latest` and times the actual
150155
`ccusage-codex` binary so RAM reflects the scanner, not the `npx` wrapper.
151156

152-
Recent local sanity check on a large Codex history:
157+
Recent local sanity check on a large Codex history, comparing this rollout with
158+
the previous public commit `1c084b4`:
153159

154160
| Tool | Scenario | Time | RAM | Result |
155161
| --- | --- | ---: | ---: | --- |
156-
| `@ccusage/codex@18.0.11` | `--since 2026-05-01`, 45s limit | `>45.03s` | `2.38 GB` before timeout | timed out |
157-
| `cdxusage` | same filter, cold full scan | `31.61s` | `0.37 GB` | complete |
158-
| `cdxusage` | same filter, warm cached | `0.41s` | `0.16 GB` | complete |
162+
| `cdxusage` pre-native baseline | cold app cache | `27.80s` | `0.350 GB` | complete |
163+
| `cdxusage` native auto scanner | cold app cache | `10.46s` | `0.180 GB` | complete |
164+
| `cdxusage` pre-native baseline | warm app cache | `0.40s` | `0.166 GB` | complete |
165+
| `cdxusage` native auto scanner | warm app cache | `0.30s` | `0.143 GB` | complete |
159166

160167
See [docs/benchmark-2026-05-16.md](docs/benchmark-2026-05-16.md) for the
161168
same-run command output behind this table.
162169

163170
Cold scans read every matching JSONL file for correctness, including resumed
164171
long-lived sessions whose recent events may live in older session files. After
165-
the cache is built, the same report is dramatically faster: in this run, the
166-
warm cached path was at least 99.1% faster than the upstream timeout window.
172+
the cache is built, the same report is dramatically faster.
167173

168-
The timeout keeps the upstream run from reaching its worst failure mode. The
169-
upstream path reads and sorts a large archive-shaped set of token events in
170-
memory; `cdxusage` keeps memory bounded and predictable by avoiding that shape.
174+
The native prefilter reduces the candidate-line byte volume delivered into Node
175+
processing; it does not claim lower physical disk reads. `bytesRead` in
176+
`--include-stats` remains the logical source byte count.
171177

172178
## Portable Folder
173179

@@ -219,6 +225,8 @@ npm run check
219225
npm run lint
220226
npm run typecheck
221227
npm test
228+
npm run test:node
229+
npm run test:native
222230
npm run smoke
223231
npm run portable:smoke
224232
npm pack --dry-run --json

docs/benchmark-2026-05-16.md

Lines changed: 53 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,66 @@
11
# Benchmark Evidence: 2026-05-16
22

3-
This is a local sanity check on a large Codex history. It is not a universal
4-
benchmark claim; run `npm run benchmark` on your own archive for local numbers.
3+
This is dated local evidence from a large Codex history plus static synthetic
4+
fixtures. It is not a universal benchmark claim; run `npm run benchmark` on
5+
your own archive for local numbers.
56

6-
The public benchmark helper resolves `@ccusage/codex@latest`, finds the actual
7-
`ccusage-codex` binary, and measures that process directly. This avoids
8-
underreporting RAM by timing only the `npx` wrapper.
7+
## Real Profile
98

10-
Command:
9+
Command shape:
1110

1211
```bash
13-
npm run benchmark -- --since 2026-05-01 --upstream-timeout 45 --cdxusage-timeout 90
12+
node ./bin/cdxusage.mjs daily \
13+
--offline --since 2026-05-01 \
14+
--include-stats --json
1415
```
1516

16-
Raw output:
17+
The comparison uses the previous public commit `1c084b4` as the baseline and
18+
the current native-auto scanner worktree as the selected run. Runs are
19+
application-cache-cold/page-cache-warm on an actively changing local archive, so
20+
use the synthetic fixtures below for strict accuracy checks.
1721

18-
```text
19-
| Tool | Time | RAM | Result |
20-
| --- | ---: | ---: | --- |
21-
| cdxusage cold | 31.61s | 0.37 GB | complete |
22-
| cdxusage warm | 0.41s | 0.16 GB | complete |
23-
| @ccusage/codex@18.0.11 | >45.03s | 2.38 GB | timed out (124) |
24-
```
22+
| Tool | Scenario | Wall | CPU | RAM | Result |
23+
| --- | --- | ---: | ---: | ---: | --- |
24+
| `cdxusage` pre-native baseline | cold app cache | `27.80s` | `45.36s` | `0.350 GB` | complete |
25+
| `cdxusage` native auto scanner | cold app cache | `10.46s` | `16.82s` | `0.180 GB` | complete |
26+
| `cdxusage` pre-native baseline | warm app cache | `0.40s` | `0.57s` | `0.166 GB` | complete |
27+
| `cdxusage` native auto scanner | warm app cache | `0.30s` | `0.46s` | `0.143 GB` | complete |
28+
29+
Cold native auto versus pre-native baseline:
30+
31+
- 62.4% less wall time
32+
- 62.9% less CPU time
33+
- 48.5% less RAM
34+
- 4,219 JSONL files, about 9.24GB logical source bytes
35+
- about 0.65GB candidate bytes delivered into Node candidate-line processing
36+
- about 93.0% less candidate data delivered from full-scan source bytes into
37+
Node candidate-line processing
38+
39+
The live archive changed between sequential real runs, so token totals are not
40+
used as strict accuracy evidence here.
41+
42+
## Synthetic Fixtures
43+
44+
Static fixture run comparing public commit `1c084b4` with the native-auto
45+
worktree:
46+
47+
| Scenario | Cold wall saved | Cold CPU saved | RAM saved | Accuracy |
48+
| --- | ---: | ---: | ---: | --- |
49+
| small | 18.2% | 8.3% | 1.1% | match |
50+
| medium | 42.5% | 28.3% | -1.2% | match |
51+
| large | 59.1% | 44.1% | 14.2% | match |
52+
| huge | 60.9% | 47.7% | 29.7% | match |
53+
| adversarial | 27.8% | 14.3% | 3.5% | match |
2554

26-
Interpretation:
55+
## Notes
2756

2857
- Cold `cdxusage` scans read the archive for correctness, including resumed
2958
sessions whose recent activity can live in older session files.
30-
- Warm `cdxusage` uses the compact file cache/index and completed at least
31-
99.1% faster than the upstream timeout window in this run.
32-
- The upstream run was stopped at 45 seconds before completion; RAM shown is
33-
the measured maximum before timeout, not a completed-run peak.
59+
- Warm `cdxusage` uses the compact file cache/index and should normally be
60+
dominated by changed or appended files.
61+
- Native acceleration depends on Linux/GNU-compatible tools. Other platforms or
62+
native failures use the Node scanner.
63+
- `nativeOutputBytes` is candidate byte volume delivered into Node processing;
64+
`bytesRead` remains the logical source byte count.
65+
- The public benchmark helper still resolves and times `@ccusage/codex@latest`
66+
directly when you want an upstream comparison on your own machine.

docs/compatibility.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,29 @@ public ccusage Codex guide.
5353
- `--sort <auto|date|month|lastActivity|tokens|cost|input|output|session|directory>`
5454
- `--order <asc|desc>`
5555

56+
## Scanner Diagnostics
57+
58+
Default scanner selection is `auto`: on Linux hosts with working `perl` and
59+
GNU-compatible `xargs -r`, `cdxusage` uses a native batch prefilter for cold
60+
full scans and falls back to the Node scanner when native tooling is
61+
unavailable or fails. Other platforms use the Node scanner unless explicitly
62+
forced for diagnostics. Tail reads and cached files keep the normal cache
63+
semantics.
64+
65+
Internal diagnostic override:
66+
67+
```bash
68+
CDXUSAGE_SCAN_MODE=node cdxusage daily
69+
CDXUSAGE_SCAN_MODE=grep-batch cdxusage daily
70+
```
71+
72+
This is not an upstream compatibility surface. With `--include-stats`,
73+
`scannerModes` reports aggregate scanner counts, `linesSeen` is physical JSONL
74+
lines scanned, `candidateLinesSeen` is the subset containing `turn_context` or
75+
`token_count`, and `nativeOutputBytes` is candidate byte volume delivered into
76+
Node processing. Cache files and stats output can include absolute local paths,
77+
model names, token volumes, and estimated cost metadata.
78+
5679
## JSON Output
5780

5881
`daily --json`:

package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,12 @@
1111
},
1212
"scripts": {
1313
"start": "node ./bin/cdxusage.mjs",
14-
"check": "node --check ./bin/cdxusage.mjs && node --check ./src/cli.mjs && node --check ./src/codex-home.mjs && node --check ./src/engine.mjs && node --check ./src/format.mjs && node --check ./src/pricing.mjs && node --check ./src/table.mjs && node --check ./scripts/build-portable.mjs && node --check ./scripts/benchmark-local.mjs && node --check ./test/smoke.mjs",
14+
"check": "node --check ./bin/cdxusage.mjs && node --check ./src/cli.mjs && node --check ./src/codex-home.mjs && node --check ./src/discovery.mjs && node --check ./src/engine.mjs && node --check ./src/format.mjs && node --check ./src/pricing.mjs && node --check ./src/table.mjs && node --check ./scripts/build-portable.mjs && node --check ./scripts/benchmark-local.mjs && node --check ./test/smoke.mjs",
1515
"lint": "npm run check",
1616
"typecheck": "npm run check",
1717
"test": "node ./test/codex-home.test.mjs && node ./test/engine.test.mjs && node ./test/cli.test.mjs && node ./test/pricing.test.mjs",
18+
"test:node": "CDXUSAGE_SCAN_MODE=node npm test",
19+
"test:native": "CDXUSAGE_SCAN_MODE=grep-batch npm test",
1820
"smoke": "node ./test/smoke.mjs",
1921
"portable:build": "node ./scripts/build-portable.mjs",
2022
"portable:smoke": "npm run portable:build && sh ./portable/cdxusage --version && node -e \"const fs=require('fs'); for (const file of ['portable/LICENSE','portable/cdxusage.cmd','portable/cdxusage.ps1','portable/src/codex-home.mjs']) if (!fs.existsSync(file)) process.exit(1)\" && node ./test/smoke.mjs --portable",

src/cli.mjs

Lines changed: 46 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -65,30 +65,36 @@ export async function main(argv = process.argv.slice(2), io = { stdout: process.
6565
}
6666
const locale = args.locale ?? DEFAULT_LOCALE;
6767
const mode = args.command === 'sessions' ? 'session' : args.command;
68-
const dataPaths = await resolveCodexDataPaths({
69-
codexHome: args.codexHome,
70-
sessionsDir: args.sessionsDir,
71-
});
72-
const pricingMode = await resolvePricingMode(args, dataPaths.codexHome);
73-
const report = await collectUsage({
74-
dataPaths,
75-
since,
76-
until,
77-
timezone,
78-
cacheFile: args.cacheFile,
79-
pricingCacheFile: args.pricingCacheFile,
80-
pricingOffline: args.offline,
81-
pricingTtlMs: args.pricingTtlHours != null ? Number(args.pricingTtlHours) * 60 * 60 * 1000 : undefined,
82-
pricingFetchTimeoutMs: args.pricingFetchTimeoutMs != null ? Number(args.pricingFetchTimeoutMs) : undefined,
83-
pricingTier: pricingMode.tier,
84-
pricingPriorityModels: pricingMode.priorityModels,
85-
maxCacheBytes: args.maxCacheBytes != null ? Number(args.maxCacheBytes) : undefined,
86-
useCache: !args.noCache,
87-
clearCache: args.clearCache,
88-
saveCache: !args.noSaveCache,
89-
discoveryMode: args.discovery,
90-
includePricing: !args.noPricing,
91-
});
68+
let report;
69+
try {
70+
const dataPaths = await resolveCodexDataPaths({
71+
codexHome: args.codexHome,
72+
sessionsDir: args.sessionsDir,
73+
});
74+
const pricingMode = await resolvePricingMode(args, dataPaths.codexHome);
75+
report = await collectUsage({
76+
dataPaths,
77+
since,
78+
until,
79+
timezone,
80+
cacheFile: args.cacheFile,
81+
pricingCacheFile: args.pricingCacheFile,
82+
pricingOffline: args.offline,
83+
pricingTtlMs: args.pricingTtlHours != null ? Number(args.pricingTtlHours) * 60 * 60 * 1000 : undefined,
84+
pricingFetchTimeoutMs: args.pricingFetchTimeoutMs != null ? Number(args.pricingFetchTimeoutMs) : undefined,
85+
pricingTier: pricingMode.tier,
86+
pricingPriorityModels: pricingMode.priorityModels,
87+
maxCacheBytes: args.maxCacheBytes != null ? Number(args.maxCacheBytes) : undefined,
88+
useCache: !args.noCache,
89+
clearCache: args.clearCache,
90+
saveCache: !args.noSaveCache,
91+
discoveryMode: args.discovery,
92+
includePricing: !args.noPricing,
93+
});
94+
} catch (error) {
95+
io.stderr.write(`${error?.message ?? String(error)}\n`);
96+
return 1;
97+
}
9298

9399
const rows = rowsForMode(report, mode, { locale, sort: args.sort, order: args.order });
94100
if (args.json) {
@@ -248,6 +254,11 @@ export function parseArgs(argv) {
248254
if (token.startsWith('--')) {
249255
throw new Error(`Unknown option: ${token}`);
250256
}
257+
if (COMMANDS.has(token) && !args.commandSpecified) {
258+
args.command = token;
259+
args.commandSpecified = true;
260+
break;
261+
}
251262
throw new Error(`Unknown command or argument: ${token}`);
252263
}
253264
}
@@ -309,11 +320,21 @@ function compareRows(a, b, sort, mode) {
309320

310321
function compareDefault(a, b, mode) {
311322
if (mode === 'session') {
312-
return String(a.lastActivity ?? '').localeCompare(String(b.lastActivity ?? ''));
323+
return compareTimestamp(a.lastActivity, b.lastActivity);
313324
}
314325
return String(a.key ?? '').localeCompare(String(b.key ?? ''));
315326
}
316327

328+
function compareTimestamp(left, right) {
329+
return toSortableTimestamp(left).localeCompare(toSortableTimestamp(right));
330+
}
331+
332+
function toSortableTimestamp(value) {
333+
const text = String(value ?? '');
334+
const ms = Date.parse(text);
335+
return Number.isFinite(ms) ? new Date(ms).toISOString() : text;
336+
}
337+
317338
function compareNumber(a, b) {
318339
return (Number(a) || 0) - (Number(b) || 0);
319340
}

0 commit comments

Comments
 (0)