You can connect a website you own as a data source for your AI Search instance. AI Search crawls and indexes the pages automatically.
You can only crawl domains that you have onboarded onto the same Cloudflare account. Refer to Onboard a domain for more information on adding a domain to your Cloudflare account.
You can connect a website when creating a new instance through the dashboard, the REST API, or Wrangler. Website is an optional data source that you can add alongside built-in storage.
AI Search finds the pages on your site, fetches each one, converts it to Markdown, splits it into chunks, and adds it to the index. The parse type controls how pages are found:
- Sitemap (default): reads the XML sitemaps your site publishes.
- Discover: starts at the source URL and, by default, uses both your sitemaps and the links it finds on the pages it crawls.
Refer to Parse types for how each type finds pages, how each one handles sitemaps and syncing, and which settings apply to only one of them.
Crawled pages are stored in built-in storage automatically.
To see the items parsed from your website, list the instance's items with the Items API, or open the Items tab in the dashboard (AI > AI Search > your instance > Items).
Configure these options during onboarding, or later in your instance settings under Parser options.
| Option | Description |
|---|---|
| Path filtering | Include and exclude URL patterns that decide which pages are crawled. |
| Parse type | How the crawler finds pages. Sitemap reads your XML sitemaps. Discover starts at the source URL and, by default, uses both your sitemaps and the links it finds. |
| Specific sitemap | Crawl a set of sitemap URLs that you choose instead of the ones AI Search discovers. Up to five URLs. Applies to the sitemap parse type only. |
| Discover options | Discovery source, page limit, crawl depth, cache age, and whether to follow external links and subdomains. Applies to the discover parse type only. |
| Rendering mode | Whether pages are downloaded as raw HTML or loaded in a headless browser first. |
| Authentication headers | Custom HTTP headers sent with each request, so the crawler can reach pages behind authentication. Up to five headers. |
| Content selectors | Restrict indexing to the elements that a CSS selector matches, so you skip navigation, sidebars, and footers. |
| Custom metadata | Values extracted from the <meta> tags in each page's <head>, stored alongside the indexed content. |
You can control which pages get indexed by defining include and exclude rules for URL paths. Use this to limit indexing to specific sections of your site or to exclude content you do not want searchable.
For example, to index only blog posts while excluding drafts:
- Include:
**/blog/** - Exclude:
**/blog/drafts/**
Refer to Path filtering for pattern syntax, filtering behavior, and more examples.
For supported file types and size limits, refer to Data source.
You can choose how pages are parsed during crawling:
- Static sites: Downloads the raw HTML for each page.
- Rendered sites: Loads pages with a headless browser and downloads the fully rendered version, including dynamic JavaScript content.
If you have Security rules configured to block bot activity on your own site, you can add a rule to allowlist the AI Search bot. Refer to AI Search in the Cloudflare Radar bot directory ↗ for its verified identity and user agent.
-
In the Cloudflare dashboard, go to the Security rules page.
Go to Security rules ↗ -
To create a new empty rule, select Create rule > Custom rules.
-
Enter a descriptive name for the rule in Rule name, such as
Allow AI Search. -
Under When incoming requests match, use the Field drop-down list to choose Bot Detection ID. For Operator, select equals. For Value, enter
122933950. -
Under Then take action, in the Choose action dropdown, choose Skip.
-
Under Place at, select the order of the rule in the Select order dropdown to be First. Setting the order as First allows this rule to be applied before subsequent rules.
-
To save and deploy your rule, select Deploy.
The regular AI Search limits apply when using the Website data source. The following limits apply to website data sources on both Workers plans:
| Limit | Value |
|---|---|
Pages per crawl, discover parse type |
100,000 |
Crawl depth, discover parse type |
100,000 link hops (defaults to 5) |
Cached page age, discover parse type |
604,800 seconds (7 days) |
| Specific sitemap URLs | 5 per instance |
| Authentication headers | 5 per instance |
| Content selector entries | 10 per instance |
| Content selector path pattern and selector length | 200 characters each |
The files per instance limit also applies, so the effective cap is whichever value is lower. The crawler indexes the first pages it visits until it reaches that cap, and any file it downloads that exceeds the maximum file size is not indexed.