Skip to main content
Publishing|Technical SEO Lead / Content Platform Engineer

AI Crawler & Search Access Review

Review crawler-specific robots rules alongside search-readiness signals

Review crawler-specific robots rules alongside search-readiness signals. Follow the request examples below and review the observation limits before adding the workflow to your release or audit process.

Implementation example by the EdgeDNS team · Updated

The Challenge

A robots policy can allow search engines while restricting selected AI crawlers, and broad rules may have unintended effects.

The Workflow

Inspect robots analysis for crawler-specific rule resolution, then compare search readiness and page metadata with your intended policy. Save responses before and after a robots change and recheck after the relevant cache window. Robots rules are advisory: they do not enforce access, prove crawler compliance, control every AI use, or guarantee search indexing or AI citations.

Endpoints Used

Combine these EdgeDNS endpoints to build this solution.

GET
/v1/domain/robotsAPI reference

Robots.txt: Parse robots.txt rules and sitemaps

GET
/v1/domain/search-readinessAPI reference

Search Readiness: Evaluate search engine optimization readiness

GET
/v1/domain/metaAPI reference

Meta Tags: Extract SEO and social meta tags

What this workflow helps you do

Review crawler-specific access

Compare the analyzer results for supported search and AI crawler user agents.

Check adjacent indexing signals

Review metadata and search-readiness findings alongside robots policy.

Document policy intent

Keep dated findings and verify actual crawler requests in your own server logs.

Code Example

Review crawler-specific robots rules alongside search-readiness signals

javascript
const headers = { Authorization: 'Bearer YOUR_API_KEY' };
const checks = [
  {
    "path": "/v1/domain/robots",
    "params": {
      "domain": "https://example.com"
    }
  },
  {
    "path": "/v1/domain/search-readiness",
    "params": {
      "domain": "example.com"
    }
  },
  {
    "path": "/v1/domain/meta",
    "params": {
      "domain": "https://example.com"
    }
  }
];

// Run sequentially to keep concurrency bounded. API tier restrictions apply.
for (const { path, params } of checks) {
  const url = new URL(path, 'https://api.edgedns.dev');
  url.search = new URLSearchParams(params).toString();
  const response = await fetch(url, { headers });
  const result = await response.json();
  if (!response.ok) {
    throw new Error(`${path}: ${response.status} ${result.error?.message ?? 'Request failed'}`);
  }
  // Keep the response metadata and any partial/unknown findings for review.
  console.log(path, result);
}

Learn More

Explore industry standards and best practices related to this use case.

Ready to build AI Crawler & Search Access Review?

Get started with 200 free API requests per month. No credit card required.

Other Use Cases

AI Crawler & Search Access Review | EdgeDNS