Use Markdown for Agents for all docs markdown generation#28921
Use Markdown for Agents for all docs markdown generation#28921mvvmm merged 20 commits intoproductionfrom
Conversation
|
This PR requires additional review attention because it affects the following areas: RedirectsThis PR changes current filenames or deletes current files. Make sure you have redirects set up to cover the following paths:
PartialsThis PR updates partial files, which are pieces of content used across multiple files in our Render component.
|
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
|
/bonk please review this PR for any regressions or issues |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
|
MEDIUM — Missing redirects for deleted fixture pages. The bot flagged this too: |
|
Review posted on PR #28921. Here's a summary of what I flagged: MEDIUM (1 item):
LOW (4 items):
|
|
/bonk please review this PR for any regressions or issues otherwise |
|
Review posted on PR #28921 with 2 inline suggestions: LOW (2 items):
The previous review's concern about the |
Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>
Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>
3 participants
Summary
This PR replaces the custom rehype-based HTML-to-Markdown pipeline with Cloudflare's Markdown for Agents (MFA) network-layer feature, which converts any HTML page to Markdown when requested with an
Accept: text/markdownheader.The
/path/index.mdURL pattern is preserved for backwards compatibility. Requests to/path/index.mdare handled by a Cloudflare Transform Rule that rewrites the URL to/path/and a Request Header Transform Rule that addsAccept: text/markdown— triggering MFA to return Markdown instead of HTML.What changed
Removed the custom Markdown pipeline:
src/util/markdown.ts— the customhtmlToMarkdown()function and rehype pipelinesrc/content/docs/style-guide/fixtures/— test fixture pages used by the old pipelineindex.mdhandler fromworker/index.tsandsrc/middleware/index.tsUpdated documentation:
src/content/docs/style-guide/how-we-docs/ai-consumability.mdx— updated the "Turning our components into Markdownable HTML" section to describe MFA instead of the rehype pluginsrc/content/partials/style-guide/llms-txt.mdx— updated to reflect MFA as the underlying mechanism, withAccept: text/markdownas the primary recommendationUpdated
llms.txtgeneration:src/pages/[product]/llms.txt.tswithsrc/pages/[...product]/llms.txt.ts— a catch-all route that derives each product's llms.txt URL fromentry.data.entry.urlinstead of the YAML filename (entry.id). This fixes broken and mismatched llms.txt URLs for products whose canonical URL path differs from their entry ID (e.g.cloudflare-for-saas→/cloudflare-for-platforms/cloudflare-for-saas/llms.txt).llms.txtnow links to each product using its canonical URL (entry.data.entry.url) rather than/${entry.id}/, fixing the same class of broken links.llms.txtno longer lists sub-products that are fully covered by their parent's llms.txt (e.g. Cache Rules, Logpush, Access), eliminating ~58 duplicate entries and preventing the same pages from appearing twice when enumerating from root. Sub-product llms.txt files still exist at their canonical URLs.llms.txtadds a new ## Other section listing ungrouped directory entries (e.g.style-guide,security,use-cases) that have docs pages but no product group.sidebar.orderfrom each section'sindex.mdxfrontmatter, matching the visible sidebar structure. Products without sidebar ordering fall back to a flat list.index.mdURLs (e.g./workers/get-started/guide/index.md) so agents and parsers receive Markdown immediately without needing to set headers or rewrite URLs.llms.txtandllms-full.txtin the Developer Platform section are updated to use canonical URL paths.llms-full.txtpointer.