You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/changelog/browser-rendering/2026-03-10-br-crawl-endpoint.mdx
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@ products:
6
6
date: 2026-03-10
7
7
---
8
8
9
-
You can now crawl an entire website with a single API call using [Browser Rendering](/browser-rendering/)'s new [`/crawl` endpoint](/browser-rendering/rest-api/crawl-endpoint/), now available in open beta. Submit a starting URL and the endpoint automatically discovers linked pages, renders them in a headless browser, and returns content as HTML, Markdown, or structured JSON.
9
+
You can now crawl an entire website with a single API call using [Browser Rendering](/browser-rendering/)'s new [`/crawl` endpoint](/browser-rendering/rest-api/crawl-endpoint/), available in open beta. Submit a starting URL and the endpoint automatically discovers linked pages, renders them in a headless browser, and returns content as HTML, Markdown, or structured JSON.
10
10
11
-
Crawl jobs run asynchronously. You submit a URL, receive a job ID, and check back for results as pages are processed:
11
+
Crawl jobs run asynchronously. You submit a URL, receive a job ID, and check back for results as pages are processed. Here is how it works:
12
12
13
13
```sh
14
14
# Initiate a crawl
@@ -31,8 +31,9 @@ Key features:
31
31
-**Automatic page discovery** - Discovers URLs from sitemaps, page links, or both
32
32
-**Incremental crawling** - Use `modifiedSince` and `maxAge` to skip pages that haven't changed or were recently fetched, saving time and cost on repeated crawls
33
33
-**Static mode** - Set `render: false` to fetch static HTML without spinning up a browser, for faster crawling of static sites
34
-
-**Well-behaved bot** - Honors `robots.txt` directives including `crawl-delay`
34
+
-**Well-behaved bot** - Honors `robots.txt` directives, including `crawl-delay`
35
35
36
36
Available on both the Workers Free and Paid plans.
37
37
38
-
To get started, refer to the [crawl endpoint documentation](/browser-rendering/rest-api/crawl-endpoint/). If you are setting up your own site to be crawled, review the [robots.txt and sitemaps best practices](/browser-rendering/reference/robots-txt/).
38
+
To get started, refer to the [crawl endpoint documentation](/browser-rendering/rest-api/crawl-endpoint/).
39
+
If you are setting up your own site to be crawled, review the [robots.txt and sitemaps best practices](/browser-rendering/reference/robots-txt/).
0 commit comments