Total number of URLs that have been crawled so far.
records: array of object {status, url, html, 3 more}
List of crawl job records.
status: "queued" or "errored" or "completed" or 3 more
Current status of the crawled URL.
One of the following:
"queued"
"errored"
"completed"
"disallowed"
"skipped"
"cancelled"
url: string
Crawled URL.
html: optional string
HTML content of the crawled URL.
json: optional map[unknown]
JSON of the content of the crawled URL.
markdown: optional string
Markdown of the content of the crawled URL.
metadata: optional object {status, url, title}
Absent for urls that never reached a fetch.
status: number
HTTP status code of the crawled page.
url: string
Final URL of the crawled page.
title: optional string
Title of the crawled page.
skipped: number
Total number of URLs that were skipped due to include/exclude/subdomain filters. Skipped URLs are included in records but are not counted toward total/finished.
