-
Radar now offers expanded insights into web crawlers, giving you greater visibility into aggregated trends in crawl and refer activity.
We have introduced the following endpoints:
/bots/crawlers/summary/{dimension}: Returns an overview of crawler HTTP request distributions across key dimensions./bots/crawlers/timeseries_groups/{dimension}: Provides time-series data on crawler request distributions across the same dimensions.
These endpoints allow analysis across the following dimensions:
user_agent: Parsed data from theUser-Agentheader.referer: Parsed data from theRefererheader.crawl_refer_ratio: Ratio of HTML page crawl requests to HTML page referrals by platform.
In addition to crawler-specific insights, Radar now provides a broader set of bot endpoints:
/bots/: Lists all bots./bots/{bot_slug}: Returns detailed metadata for a specific bot./bots/timeseries: Time-series data for bot activity./bots/summary/{dimension}: Returns an overview of bot HTTP request distributions across key dimensions./bots/timeseries_groups/{dimension}: Provides time-series data on bot request distributions across the same dimensions.
These endpoints support filtering and breakdowns by:
bot: Bot name.bot_operator: The organization or entity operating the bot.bot_category: Classification of bot type.
The previously available
verified_botsendpoints have now been deprecated in favor of this set of bot insights APIs. While current data still focuses on verified bots, we plan to expand support for unverified bot traffic in the future.Learn more about the new Radar bot and crawler insights in our blog post ↗.
Radar has expanded its security insights, providing visibility into aggregate trends in authentication requests, including the detection of leaked credentials through leaked credentials detection scans.
We have now introduced the following endpoints:
/leaked_credential_checks/summary/{dimension}: Retrieves summaries of HTTP authentication requests distribution across two different dimensions./leaked_credential_checks/timeseries_groups/{dimension}: Retrieves timeseries data for HTTP authentication requests distribution across two different dimensions.
The following dimensions are available, displaying the distribution of HTTP authentication requests based on:
compromised: Credential status (clean vs. compromised).bot_class: Bot class (human vs. bot).
Dive deeper into leaked credential detection in this blog post ↗ and learn more about the expanded Radar security insights in our blog post ↗.
Radar has expanded its DNS insights, providing visibility into aggregated traffic and usage trends observed by our 1.1.1.1 DNS resolver. In addition to global, location, and ASN traffic trends, we are also providing perspectives on protocol usage, query/response characteristics, and DNSSEC usage.
Previously limited to the
toplocations and ASes endpoints, we have now introduced the following endpoints:/dns/timeseries: Retrieves DNS query volume over time./dns/summary/{dimension}: Retrieves summaries of DNS query distribution across ten different dimensions./dns/timeseries_groups/{dimension}: Retrieves timeseries data for DNS query distribution across ten different dimensions.
For the
summaryandtimeseries_groupsendpoints, the following dimensions are available, displaying the distribution of DNS queries based on:cache_hit: Cache status (hit vs. miss).dnsssec: DNSSEC support status (secure, insecure, invalid or other).dnsssec_aware: DNSSEC client awareness (aware vs. not-aware).dnsssec_e2e: End-to-end security (secure vs. insecure).ip_version: IP version (IPv4 vs. IPv6).matching_answer: Matching answer status (match vs. no-match).protocol: Transport protocol (UDP, TLS, HTTPS or TCP).query_type: Query type (A,AAAA,PTR, etc.).response_code: Response code (NOERROR,NXDOMAIN,REFUSED, etc.).response_ttl: Response TTL.
Learn more about the new Radar DNS insights in our blog post ↗, and check out the new Radar page ↗.
Radar has expanded its AI insights with new API endpoints for Internet services rankings, robots.txt analysis, and AI inference data.
Radar now provides rankings for Internet services, including Generative AI platforms, based on anonymized 1.1.1.1 resolver data. Previously limited to the annual Year in Review, these insights are now available daily via the API, through the following endpoints:
/ranking/internet_services/topshow service popularity at a specific date./ranking/internet_services/timeseries_groupstrack ranking trends over time.
Radar now analyzes robots.txt files from the top 10,000 domains, identifying AI bot access rules. AI-focused user agents from ai.robots.txt ↗ are categorized as:
- Fully allowed/disallowed if directives apply to all paths (
*). - Partially allowed/disallowed if restrictions apply to specific paths.
These insights are now available weekly via the API, through the following endpoints:
/robots_txt/top/user_agents/directiveto get the top AI user agents by directive./robots_txt/top/domain_categoriesto get the top domain categories by robots.txt files.
Radar now provides insights into public AI inference models from Workers AI, tracking usage trends across models and tasks. These insights are now available via the API, through the following endpoints:
/ai/inference/summary/{dimension}to view aggregatedmodelandtaskpopularity./ai/inference/timeseries_groups/{dimension}to track changes over time formodelortask.
Learn more about the new Radar AI insights in our blog post ↗.