Add guidance on blocking crawlers with robots.txt and fix broken FAQ anchor links

- Add new 'Blocking crawlers with robots.txt' section to robots-txt.mdx with three examples: block all bots, block only the /crawl endpoint, and block specific paths
- Add cross-reference from crawl-endpoint.mdx to the new blocking guidance
- Fix broken anchor link in automatic-request-headers.mdx (#how-do-i-allowlist-browser-rendering → #can-i-allowlist-browser-rendering-on-my-own-website)
- Fix broken anchor link in robots-txt.mdx Related resources (#will-browser-rendering-bypass-cloudflares-bot-protection → #will-browser-rendering-be-detected-by-bot-management)
- Add Automatic request headers to Related resources in robots-txt.mdx

Author: kathayl

src/content/docs/browser-rendering/reference/automatic-request-headers.mdx

@@ -43,4 +43,4 @@ The `Signature` headers use an authentication method called [Web Bot Auth](/bots
 
 ### Bot detection
 
-The bot detection ID for Browser Rendering is `128292352`. If you are attempting to scan your own zone and want Browser Rendering to access your website freely without your bot protection configuration interfering, you can create a WAF skip rule to [allowlist Browser Rendering](/browser-rendering/faq/#how-do-i-allowlist-browser-rendering).
+The bot detection ID for Browser Rendering is `128292352`. If you are attempting to scan your own zone and want Browser Rendering to access your website freely without your bot protection configuration interfering, you can create a WAF skip rule to [allowlist Browser Rendering](/browser-rendering/faq/#can-i-allowlist-browser-rendering-on-my-own-website).

src/content/docs/browser-rendering/reference/robots-txt.mdx

@@ -55,6 +55,47 @@ Sitemap: https://example.com/sitemap.xml
 
 The value is in seconds. A `crawl-delay` of 2 means the crawler waits two seconds between requests.
 
+## Blocking crawlers with robots.txt
+
+If you want to prevent Browser Rendering (or other crawlers) from accessing your site, you can configure your `robots.txt` to restrict access.
+
+### Block all bots from your entire site
+
+To prevent all crawlers from accessing any page on your site:
+
+```txt title="robots.txt"
+User-agent: *
+Disallow: /
+```
+
+This is the most restrictive configuration and blocks all compliant bots, not just Browser Rendering.
+
+### Block only the /crawl endpoint
+
+The [`/crawl` endpoint](/browser-rendering/rest-api/crawl-endpoint/) identifies itself with the User-Agent `CloudflareBrowserRenderingCrawler/1.0`. To block the `/crawl` endpoint while allowing all other traffic (including other Browser Rendering [REST API](/browser-rendering/rest-api/) endpoints, which use a [different User-Agent](/browser-rendering/reference/automatic-request-headers/#user-agent)):
+
+```txt title="robots.txt"
+User-agent: CloudflareBrowserRenderingCrawler
+Disallow: /
+
+User-agent: *
+Allow: /
+```
+
+### Block specific paths
+
+To allow crawling of your site but block specific sections:
+
+```txt title="robots.txt"
+User-agent: CloudflareBrowserRenderingCrawler
+Disallow: /admin/
+Disallow: /private/
+Allow: /
+
+User-agent: *
+Allow: /
+```
+
 ## Best practices for sitemaps
 
 Structure your sitemap to help crawlers process your site efficiently:
@@ -113,4 +154,5 @@ Browser Rendering periodically refetches sitemaps to keep content fresh. Serve y
 
 ## Related resources
 
-- [FAQ: Will Browser Rendering bypass Cloudflare's Bot Protection?](/browser-rendering/faq/#will-browser-rendering-bypass-cloudflares-bot-protection) — Instructions for creating a WAF skip rule
+- [FAQ: Will Browser Rendering be detected by Bot Management?](/browser-rendering/faq/#will-browser-rendering-be-detected-by-bot-management) — How Browser Rendering interacts with bot protection and how to create a WAF skip rule
+- [Automatic request headers](/browser-rendering/reference/automatic-request-headers/) — User-Agent strings and non-configurable headers used by Browser Rendering

src/content/docs/browser-rendering/rest-api/crawl-endpoint.mdx

@@ -425,7 +425,7 @@ Use the `source` parameter to customize which sources the crawler uses. The avai
 
 ### robots.txt and bot protection
 
-The `/crawl` endpoint respects the directives of `robots.txt` files, including `crawl-delay`. All URLs that `/crawl` is directed not to crawl are listed in the response with `"status": "disallowed"`. For guidance on configuring `robots.txt` and sitemaps for sites you plan to crawl, refer to [robots.txt and sitemaps](/browser-rendering/reference/robots-txt/).
+The `/crawl` endpoint respects the directives of `robots.txt` files, including `crawl-delay`. All URLs that `/crawl` is directed not to crawl are listed in the response with `"status": "disallowed"`. For guidance on configuring `robots.txt` and sitemaps for sites you plan to crawl, refer to [robots.txt and sitemaps](/browser-rendering/reference/robots-txt/). If you want to block the `/crawl` endpoint from accessing your site, refer to [Blocking crawlers with robots.txt](/browser-rendering/reference/robots-txt/#blocking-crawlers-with-robotstxt).
 
 :::caution[Bot protection may block crawling]
 Browser Rendering does not bypass CAPTCHAs, Turnstile challenges, or any other bot protection mechanisms. If a target site uses Cloudflare products that control or restrict bot traffic such as [Bot Management](/bots/), [Web Application Firewall (WAF)](/waf/), or [Turnstile](/turnstile/), the same rules will apply to the Browser Rendering crawler.