--- description: Choose how AI Search finds pages on a website data source. title: Parse types image: https://developers.cloudflare.com/og-docs.png --- [Skip to content](#main-content) > Documentation Index > Fetch the complete documentation index at: https://developers.cloudflare.com/ai-search/llms.txt > Use this file to discover all available pages before exploring further. # Parse types Last updated Aug 6, 2026|Copy as Markdown|[View as Markdown](https://developers.cloudflare.com/ai-search/configuration/data-source/website/parse-types/index.md)|[Agent setup](https://developers.cloudflare.com/agent-setup/) The parse type controls how AI Search finds the pages to index on a [website data source](https://developers.cloudflare.com/ai-search/configuration/data-source/website/). AI Search supports two parse types. | Parse type | Page discovery | Use when | | ----------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | sitemap (default) | Reads the XML sitemaps declared in robots.txt, or the sitemap URLs you configure. Does not follow links. | Your site publishes a complete and current sitemap. | | discover | Starts at the source URL and, by default, uses both your sitemaps and the links it finds on the pages it crawls. | Your site has no sitemap, or its sitemap does not cover every page you need. | Set the parse type in `source_params.web_crawler.parse_type`. If you do not set it, AI Search uses `sitemap`. Both parse types require a source URL on a domain that you have onboarded onto the same Cloudflare account. Refer to [Onboard a domain](https://developers.cloudflare.com/fundamentals/manage-domains/add-site/) if your domain is not on Cloudflare yet. ## Which parse type to use Both parse types can read your sitemaps, so the choice is not simply "sitemap or links". `discover` defaults to a [discovery source](#discovery-source) of `all`, which means it reads your sitemaps **and** follows links. What differs is how thoroughly each one uses the sitemap. If your site publishes a sitemap that covers the pages you want indexed, prefer `sitemap`. It is the more reliable of the two, because: * **Updates are driven by your sitemap.** `sitemap` re-crawls a page when its `` date changes, so edits are picked up on the next sync. `discover` ignores `` and `` entirely and re-fetches on a fixed [cache age](#cache-age) instead. * **Nothing is cut off.** `discover` stops at the configured [page limit and depth](#page-limit-and-depth), so pages that sit deep in your link graph, or past the limit, can be skipped. `sitemap` indexes everything the sitemap lists. * **You control the order.** `sitemap` indexes pages by their `` value, so your most important pages are indexed first if you hit an instance limit. * **You can narrow the crawl.** [Specific sitemap](#specific-sitemap) is not supported with `discover`. Choose `discover` when a sitemap is missing, incomplete, or stale. Caution Changing the parse type on an existing instance triggers a full resync. AI Search rediscovers and reindexes every page. ## Sitemap The `sitemap` parse type is the default. AI Search reads the XML sitemaps your site publishes to decide which pages to index, and re-crawls a page when its `` date changes. ### How pages are found When you connect a domain, the crawler looks for your website's sitemap to determine which pages to visit: 1. If you configure one or more custom sitemap URLs in the dashboard under **Parser options** \> **Specific sitemap**, AI Search crawls only those sitemap URLs. 2. Otherwise, the crawler checks `robots.txt` for listed sitemaps. 3. If no `robots.txt` is found, the crawler checks for a sitemap at `/sitemap.xml`. 4. If no sitemap is available, the domain cannot be crawled with the `sitemap` parse type. Use [discover](#discover) instead. ### Indexing order If your sitemaps include `` attributes, AI Search reads all sitemaps and indexes pages based on each page's priority value, regardless of which sitemap the page is in. If no `` is specified, pages are indexed in the order the sitemaps are provided, either from the configured custom sitemap URLs or from `robots.txt` from top to bottom. AI Search supports `.gz` compressed sitemaps. Both `robots.txt` and sitemaps can use partial URLs. ### Sync and updates During scheduled or manual [sync jobs](https://developers.cloudflare.com/ai-search/configuration/indexing/syncing/), the crawler will check for changes to the `` attribute in your sitemap. If it has been changed to a date occurring after the last sync date, then the page is crawled, the updated version is stored, and the page is automatically reindexed so that your search results always reflect the latest content. If the `` attribute is not defined, AI Search uses the `` attribute to determine how often to re-crawl the URL. If neither `` nor `` is defined, AI Search automatically crawls each link once a day. ### Specific sitemap By default, AI Search crawls all sitemaps listed in your `robots.txt` in the order they appear (top to bottom). If you do not want the crawler to index everything, or if your sitemap is hosted at a non-standard path, you can configure custom sitemap URLs in the dashboard under **Parser options** \> **Specific sitemap**. When custom sitemap URLs are configured, AI Search uses those sitemap URLs instead of auto-discovering sitemaps from `robots.txt` or `/sitemap.xml`. You can add up to five sitemap URLs. ### robots.txt The AI Search crawler uses the user agent `Cloudflare-AI-Search`. Your `robots.txt` file should reference your sitemap and allow the crawler: ```txt User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` You can list multiple sitemaps or use a sitemap index file: ```txt User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/blog-sitemap.xml Sitemap: https://example.com/sitemap.xml.gz ``` To block all other crawlers but allow only AI Search: ```txt User-agent: * Disallow: / User-agent: Cloudflare-AI-Search Allow: / Sitemap: https://example.com/sitemap.xml ``` ### Sitemap structure Structure your sitemap to give AI Search the information it needs to crawl efficiently: ```xml https://example.com/important-page 2026-01-15 weekly 1.0 https://example.com/other-page 2026-01-10 monthly 0.5 ``` Use these attributes to control crawling behavior: | Attribute | Purpose | Recommendation | | ------------ | ----------------------------- | --------------------------------------------------------------------------------------------------- | | | URL of the page | Required. Use full or partial URLs. | | | Last modification date | Include to enable change detection. AI Search re-crawls pages when this date changes. | | | Expected change frequency | Use when is not available. Values: always, hourly, daily, weekly, monthly, yearly, never. | | | Relative importance (0.0-1.0) | Set higher values for important pages. AI Search indexes pages in priority order. | You can also use a Sitemap Index to bundle other domain-specific sitemaps: ```xml https://www.example.com/sitemap-blog.xml 2024-08-15T10:00:00+00:00 https://www.example.com/sitemap-docs.xml 2024-08-10T12:00:00+00:00 ``` When parsing a Sitemap Index, AI Search collects all child sitemaps and then crawls them recursively, collecting all relevant URLs present in your sitemaps. ### Sitemap recommendations * Include `` on all URLs to enable efficient change detection during syncs. * Set `` to control indexing order. Pages with higher priority are indexed first. * Use `` as a fallback when `` is not available. * Use sitemap index files for large sites with multiple sitemaps. * Compress large sitemaps using `.gz` format to reduce bandwidth. * Keep sitemaps under 50MB and 50,000 URLs per file (standard sitemap limits). ## Discover The `discover` parse type delegates page discovery to the Browser Run [/crawl endpoint](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/). AI Search starts a crawl job at your source URL and stores every page it fetches in [built-in storage](https://developers.cloudflare.com/ai-search/configuration/data-source/built-in-storage/). By default, `discover` collects candidate URLs from both your sitemaps and the links on the pages it crawls. Use the [discovery source](#discovery-source) option to restrict it to one or the other. Because `discover` does not depend on a sitemap, it reaches pages that a sitemap omits. ### How discovery works 1. AI Search starts a crawl job at the source URL. 2. The crawler collects candidate URLs from sitemaps, from links on crawled pages, or from both, depending on the discovery source. 3. The crawler follows links up to the configured depth and stops once it reaches the page limit. 4. Each fetched page is stored in built-in storage, converted to Markdown, chunked, and indexed. The crawler identifies itself as `Cloudflare-AI-Search` on domains in your Cloudflare account. If you enable [external links or subdomains](#external-links-and-subdomains) and the crawl reaches a domain outside your account, it identifies itself as `Cloudflare-AI-Search-External`. Pages that `robots.txt` disallows are recorded with the `blocked_by_robots_txt` [indexing error code](https://developers.cloudflare.com/ai-search/troubleshooting/indexing-error-codes/) instead of being indexed. Note The crawler declares the `search` and `ai-input` crawl purposes. A site that sets either purpose to `no` in a `robots.txt` [Content Signals](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/#content-signals) directive rejects the crawl. ### Configure in the dashboard 1. In the Cloudflare dashboard, go to the **AI Search** page. [Go to **AI Search** ↗](https://dash.cloudflare.com/?to=/:account/ai/ai-search) 2. Select **Create**, then select **Website** as your data source. 3. Under **Crawl target**, enter the website URL. 4. Under **Parse type**, select **Discover**. 5. Under **Crawl options**, adjust **Crawl source**, **Page limit**, **Crawl depth**, **Max cache age**, **Include external links**, and **Include subdomains**. 6. Complete the setup process to create your instance. To change these options later, select your instance, open the **Settings** tab, and edit **Crawl options** under **Parser options**. Saving the change starts a new indexing job that reindexes every item. ### Configure with the API Set `parse_type` to `discover`, then pass any non-default settings in `discover_options`. ```bash curl -X POST "https://api.cloudflare.com/client/v4/accounts//ai-search/instances" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "id": "my-ai-search", "type": "web-crawler", "source": "example.com", "source_params": { "web_crawler": { "parse_type": "discover", "discover_options": { "source": "links", "limit": 5000, "depth": 3, "max_age": 86400, "include_subdomains": true } } } }' ``` ### Discover options The following options apply only when `parse_type` is `discover`. All of them are optional. | Option | Type | Default | Range or values | Description | | ------------------------ | ------- | ------- | -------------------- | --------------------------------------------------------------------- | | source | string | all | all, sitemaps, links | Where the crawler looks for candidate URLs. | | limit | number | 100000 | 1 to 100,000 | Maximum number of pages to crawl. | | depth | number | 5 | 1 to 100,000 | Maximum number of link hops to follow from the source URL. | | max\_age | number | 86400 | 0 to 604,800 seconds | How long the crawler reuses cached page content before it re-fetches. | | include\_external\_links | boolean | false | true, false | Whether to follow links that point to other domains. | | include\_subdomains | boolean | false | true, false | Whether to follow links that point to subdomains of the source URL. | #### Discovery source The `source` option selects where candidate URLs come from: * `all`: Uses both sitemaps and links found on crawled pages. * `sitemaps`: Uses only URLs listed in sitemaps. * `links`: Uses only links found on crawled pages. Use `links` when your sitemap is missing or unreliable. Use `sitemaps` when you want sitemap coverage without following in-page links. #### Page limit and depth The `limit` option caps the number of pages the crawler indexes, up to a maximum of `100000`. Values above that maximum are rejected when you create or update the instance. Your [instance object limit](https://developers.cloudflare.com/ai-search/platform/limits-pricing/) still applies, so the effective cap is whichever value is lower. The `depth` option caps how far the crawler travels from the source URL. A depth of `1` crawls only the pages linked directly from the source URL. #### Cache age The `max_age` option sets the maximum age, in seconds, of cached page content that the crawler accepts before it re-fetches the page from your origin. Set it to `0` to always fetch from the origin. In the dashboard, **Max cache age** offers **No cache**, **1 hour**, **1 day**, **3 days**, and **7 days**. #### External links and subdomains By default, the crawler stays on the host of the source URL. Enable `include_subdomains` to follow links into subdomains, and `include_external_links` to follow links to other domains. Caution Either option can grow a crawl well beyond your own site. Pair them with `limit`, `depth`, and [path filtering](https://developers.cloudflare.com/ai-search/configuration/indexing/path-filtering/). ### Sync behavior Each [sync job](https://developers.cloudflare.com/ai-search/configuration/indexing/syncing/) starts a new crawl job. AI Search reindexes the pages the crawl returns, and removes pages it can no longer reach. If a crawl does not finish, or if it stops at the page limit, AI Search keeps the pages it indexed previously rather than deleting the pages that crawl did not reach. `discover` does not read the `` or `` sitemap attributes. Use `max_age` to control how often the crawler returns to your origin. ### Unsupported options `parse_options.specific_sitemaps` is only valid when `parse_type` is `sitemap`. Sending it with `discover` returns a validation error. ## Options that apply to both parse types The following website settings apply whichever parse type you choose: * [Path filtering](https://developers.cloudflare.com/ai-search/configuration/indexing/path-filtering/): Include and exclude URL patterns. Excluded pages are never fetched. * [Content selectors](https://developers.cloudflare.com/ai-search/configuration/data-source/website/content-selectors/): Restrict indexing to the elements a CSS selector matches. * [Authentication headers](https://developers.cloudflare.com/ai-search/configuration/data-source/website/authentication-headers/): Custom HTTP headers sent with each request. * [Custom metadata](https://developers.cloudflare.com/ai-search/configuration/data-source/website/custom-metadata/): Metadata extracted from `` tags. * [Rendering mode](https://developers.cloudflare.com/ai-search/configuration/data-source/website/#rendering-mode): Whether pages load in a headless browser. ## Limits A `discover` crawl indexes at most 100,000 pages and follows at most 100,000 link hops from the source URL, set with [limit and depth](#page-limit-and-depth). The [files per instance](https://developers.cloudflare.com/ai-search/platform/limits-pricing/#limits) limit also applies, so the effective cap is whichever value is lower. For every limit that applies to website data sources, refer to [Website](https://developers.cloudflare.com/ai-search/configuration/data-source/website/#limits). Was this helpful? YesNo ## On this page [![](https://developers.cloudflare.com/_astro/logo.te5VL_aD.svg)Docs](https://developers.cloudflare.com/) ```json {"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/ai-search/configuration/data-source/website/parse-types/#page","headline":"Parse types · Cloudflare AI Search docs","description":"Choose how AI Search finds pages on a website data source.","url":"https://developers.cloudflare.com/ai-search/configuration/data-source/website/parse-types/","inLanguage":"en","image":"https://developers.cloudflare.com/og-docs.png","dateModified":"2026-08-06","publisher":{"@type":"Organization","name":"Cloudflare","url":"https://www.cloudflare.com/"},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}} ```