Improve agent-friendly-docs (afdocs.dev) score - #59
Conversation
Build-time fixes: - Add agent-friendly-docs plugin (en only) injecting a hidden HTML directive and <head> alternate link pointing agents to /llms.txt and noting .md URLs. - Add scripts/llms-postprocess.js (run after build): prepend an llms.txt blockquote to every generated .md file, and split the 203K llms.txt into a ~9.5K root index plus per-section llms.txt files (recursive, each <50K). Honors SITE_URL for staging vs prod. - Wire the post-process step into build.sh and both deploy workflows. - Set Content-Type: text/markdown on .md files during S3 sync so content negotiation returns a readable body, not a download. Infra deliverables (applied manually in CloudFront): - cloudfront/viewer-request-markdown.js: CloudFront Function for content negotiation on Accept: text/markdown. - cloudfront/README.md: apply guide for the function and custom error responses to return real 404s instead of soft 404s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Improves AI-agent discoverability and usability of llms.txt and generated Markdown outputs (per afdocs.dev), by adding build-time directives and post-processing, plus providing CloudFront guidance for content negotiation and real 404s.
Changes:
- Add a Docusaurus plugin to inject HTML discovery directives pointing agents to
/llms.txtand.mdURL variants (en locale only). - Add a post-build script to prepend an
llms.txtdirective to generated.mdfiles and split oversizedllms.txtinto section indexes. - Update build/deploy automation and CloudFront docs/function to support Markdown content negotiation and correct
Content-Typefor.md.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/agent-friendly-docs.js | Injects HTML directives into pages to advertise llms.txt + Markdown availability. |
| scripts/llms-postprocess.js | Post-build script to add Markdown directives and split llms.txt into smaller section files. |
| docusaurus.config.js | Registers the new plugin for the default locale (en). |
| cloudfront/viewer-request-markdown.js | CloudFront Function to rewrite requests to .md when Accept: text/markdown. |
| cloudfront/README.md | Manual CDN-layer instructions (content negotiation + real 404s). |
| build.sh | Runs the post-process script after yarn build. |
| .github/workflows/stage_deploy_docusaurus.yml | Runs post-process + re-uploads .md with text/markdown Content-Type. |
| .github/workflows/prod_deploy_docusaurus.yml | Runs post-process + re-uploads .md with text/markdown Content-Type. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- agent-friendly-docs plugin: make the hidden directive accessible — keep the llms.txt anchor for detectors but set tabindex="-1" (non-focusable) and aria-hidden="true" on the container so keyboard and screen-reader users aren't affected. - llms-postprocess: measure section/root sizes with Buffer.byteLength (UTF-8 bytes) instead of string length so the 50K budget holds for non-ASCII content; fix log wording (bytes, not chars). - CloudFront function: only rewrite /docs/ routes that actually have a .md twin. Skip older versions (/docs/4.0/, /docs/3.5/, ...) and the navigation/index routes excluded from Markdown generation, so agents get HTML instead of a 404 on a missing .md. Added a routing smoke test (13/13) during review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@copilot please review the latest commit and make sure all of the issues you raised are taken care of. |
All issues from the review are addressed in commit 1b17980:
|
Investigated the markdown-content-parity failures (14/47 pages, avg ~13% "missing", worst 73%). Root cause was NOT missing content — the generated .md is complete — but two comparison artifacts: 1. HTML minification: the SWC HTML minifier strips optional closing tags (</td>, </tr>, </body>, ...). afdocs parses HTML with node-html-parser, which doesn't implement HTML5 implicit tag closing, so it flattens <body>/<main>/<article>/<table> into a flat node list, fails to find a content container, and falls back to comparing the entire raw document (doctype + head scripts + delimiter-less table cells) against the .md — producing large false "missing %" on short, table-heavy pages. Fix: swcHtmlMinimizer:false, so Docusaurus's Terser minifier (which keeps optional closing tags) is used. The emitted HTML then parses correctly in node-html-parser and other lenient parsers agents rely on. 2. Escaped underscores: the llms-txt plugin escapes underscores (\_) in prose/ headings. afdocs doesn't unescape them, so identifiers like mv_refresh_total_success_jobs (HTML) don't match mv\_refresh\_total\_... (md). Fix: unescape intra-word underscores in generated .md — render-safe (intra-word _ is never emphasis in CommonMark) and better for agents reading raw markdown. Verified end-to-end with the real afdocs tool against a local Terser build: the 14 previously-failing pages now show avg ~2% missing, worst 8% (WARN), none at the 20% FAIL threshold — the check moves from FAIL to PASS/WARN. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Raises the afdocs.dev score of docs.starrocks.io (was 71/100) by making the generated
llms.txt/ Markdown output discoverable and appropriately sized for AI agents.Build-time fixes (in this repo)
src/plugins/agent-friendly-docs.js(en locale only) injects a visually-hidden directive near the top of<body>plus a<head><link rel="alternate" type="text/markdown">, both pointing agents at/llms.txtand noting.mdavailability. → fixesllms-txt-directive-html.scripts/llms-postprocess.jsprepends a> ...llms.txtblockquote to every generated.mdfile (idempotent). → fixesllms-txt-directive-md.llms.txtinto a ~9.5K root index linking to per-sectionllms.txtfiles (e.g./docs/sql-reference/llms.txt), splitting large sections recursively so every file stays under 50K (largest 39.8K). Descriptions preserved; follows the spec's progressive-disclosure structure. → fixesllms-txt-size.yarn build(Docusaurus runspostBuildhooks in parallel, so it can't be a plugin hook) — wired intobuild.shand both deploy workflows. HonorsSITE_URLfor staging vs prod..mdfiles astext/markdown; charset=utf-8after the mainaws s3 syncso content negotiation returns a readable body, not a download.Infra deliverables (applied manually in CloudFront — not deployable from CI)
cloudfront/viewer-request-markdown.js— CloudFront Function for content negotiation onAccept: text/markdown. → fixescontent-negotiation.cloudfront/README.md— apply guide for that function and custom error responses to return real 404s instead of soft 404s. → fixeshttp-status-codes.Verification
.mdfiles; idempotent on re-run.llms.txt: root 9.5K, 40 section files, largest 39.8K, zero over 50K.Not changed (intentional)
llms-txt-coverage(~15%) —llms.txtcovers only the current version (4.1) while the sitemap counts all 7 versions; indexing 7 versions would be worse for agents.🤖 Generated with Claude Code
Running the check after building staging with the changes in this PR brings the score from 70 to 91. Some of the low scores will be fixed with the CloudFront changes documented in the cloudfront/README.md.
Interim result:
https://docs-stage.starrocks.io/ · 7/17/2026, 10:08:10 PM Overall Score: 90 / 100 (A) Category Scores: Content Discoverability 100 / 100 (A+) Markdown Availability 64 / 100 (D) Page Size and Truncation Risk 99 / 100 (A) Content Structure 100 / 100 (A+) URL Stability and Redirects 36 / 100 (F) Observability and Content Health 80 / 100 (B) Authentication and Access 99 / 100 (A)