AI Crawler & Search Access Review
Review crawler-specific robots rules alongside search-readiness signals
Review crawler-specific robots rules alongside search-readiness signals. Follow the request examples below and review the observation limits before adding the workflow to your release or audit process.
Implementation example by the EdgeDNS team · Updated
The Challenge
A robots policy can allow search engines while restricting selected AI crawlers, and broad rules may have unintended effects.
The Workflow
Inspect robots analysis for crawler-specific rule resolution, then compare search readiness and page metadata with your intended policy. Save responses before and after a robots change and recheck after the relevant cache window. Robots rules are advisory: they do not enforce access, prove crawler compliance, control every AI use, or guarantee search indexing or AI citations.
Endpoints Used
Combine these EdgeDNS endpoints to build this solution.
/v1/domain/robotsAPI referenceRobots.txt: Parse robots.txt rules and sitemaps
/v1/domain/search-readinessAPI referenceSearch Readiness: Evaluate search engine optimization readiness
/v1/domain/metaAPI referenceMeta Tags: Extract SEO and social meta tags
What this workflow helps you do
Review crawler-specific access
Compare the analyzer results for supported search and AI crawler user agents.
Check adjacent indexing signals
Review metadata and search-readiness findings alongside robots policy.
Document policy intent
Keep dated findings and verify actual crawler requests in your own server logs.
Code Example
Review crawler-specific robots rules alongside search-readiness signals
const headers = { Authorization: 'Bearer YOUR_API_KEY' };
const checks = [
{
"path": "/v1/domain/robots",
"params": {
"domain": "https://example.com"
}
},
{
"path": "/v1/domain/search-readiness",
"params": {
"domain": "example.com"
}
},
{
"path": "/v1/domain/meta",
"params": {
"domain": "https://example.com"
}
}
];
// Run sequentially to keep concurrency bounded. API tier restrictions apply.
for (const { path, params } of checks) {
const url = new URL(path, 'https://api.edgedns.dev');
url.search = new URLSearchParams(params).toString();
const response = await fetch(url, { headers });
const result = await response.json();
if (!response.ok) {
throw new Error(`${path}: ${response.status} ${result.error?.message ?? 'Request failed'}`);
}
// Keep the response metadata and any partial/unknown findings for review.
console.log(path, result);
}Learn More
Explore industry standards and best practices related to this use case.
Ready to build AI Crawler & Search Access Review?
Get started with 200 free API requests per month. No credit card required.
Other Use Cases
Technical SEO Audit & Optimization
Prioritize technical SEO findings on representative public pages
SEO Specialist / Web DeveloperWebsite Migration SEO Validation
Compare crawl and indexing signals before and after a migration
Frontend Developer / Performance EngineerWebsite Speed & Performance Optimization
Review delivery configuration and observed website performance