Summary

In this MR we have updated Ai Crawl Control Docs.
Adds a new Advanced configuration reference page for Pay Per Crawl, documenting two new capabilities:

• Disable Pay Per Crawl by URI pattern — use Configuration Rules to offer free access to specific pages (homepages, public sections, functional endpoints) while charging for others.
• Dynamic pricing — have your origin or a Worker return a crawler-price response header to set per-request prices, with the cf-pay-per-crawl request header indicating the pricing mode in effect.

Documentation checklist

• Is there a changelog entry (guidelines)? If you don't add one for something awesome and new (however small) — how will our customers find out? Changelogs are automatically posted to RSS feeds, the Discord, and X.
• The change adheres to the documentation style guide.