AI Agents Scanner AI Agent Readiness & Discovery Audit
Scan any website for AI agent readiness. Inspect robots.txt, sitemaps, MCP configuration, OpenAPI specs, and metadata for AI crawler optimization.
How AI Agent Readiness is Evaluated
Autonomous search agents, LLM browser assistants, and web crawlers query a sequence of machine-readable endpoints to safely understand and cite web content.
Discovery Endpoints
AI agents check /llms.txt, /robots.txt, and /sitemap.xml to map available content without parsing megabytes of HTML boilerplate.
Protocols & MCP
Agents look for Model Context Protocol servers (/.well-known/mcp.json) or OpenAPI catalogs to execute programmatic tool calls and dynamic actions.
Schema & Semantics
Embedded Schema.org JSON-LD and semantic tags allow neural models to extract structured entities, pricing, authorship, and canonical sources with zero ambiguity.
Signal Weights & Readiness Bands
The composite readiness score evaluates 9 core signals totaling 100 points, plus optional protocol checks.
Optimized for AI search engines, crawler discovery, and machine-readable citations.
Basic SEO present but missing dedicated AI artifacts (like /llms.txt or AI bot permissions).
Inaccessible to LLM crawlers, missing structured schema or blocked by generic directives.
Key AI Signals Reference
llms.txt Standard
18 pts (High)Curated markdown map at /llms.txt serving structured, LLM-optimized summary of your site.
Why it matters: Provides instant site overview for AI agents, cutting down crawling overhead and hallucination.
robots.txt AI Bot Rules
18 pts (High)Explicit permissions and rules for AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and CCBot.
Why it matters: Controls and optimizes crawler access so search models index your key knowledge without rate limits.
Schema.org JSON-LD
14 pts (High)Structured JSON-LD schemas describing Organization, WebSite, SoftwareApplication, and FAQs.
Why it matters: Machine-readable entity graph allows AI engines to identify authority, author, and content semantics.
Sitemap XML
12 pts (Medium)Reachable sitemap.xml declared at root or referenced in robots.txt.
Why it matters: Essential for complete URL discovery and fresh content ingestion by automated agents.
Document Title & Meta
10 pts (Medium)Clean <title> and descriptive <meta name="description"> tags.
Why it matters: Primary context anchors used by retrieval models to summarize target pages.
llms-full.txt Extended Docs
8 pts (Medium)Extended full-text documentation companion at /llms-full.txt.
Why it matters: Allows deep context retrieval for multi-turn reasoning and agentic workflow integrations.
Canonical URL Tag
8 pts (Medium)Authoritative <link rel="canonical"> pointing to the primary URL.
Why it matters: Prevents duplicate content confusion when AI systems ingest syndicated or parameterized pages.
Open Graph Metadata
6 pts (Low)og:title, og:description, og:image social cards.
Why it matters: Provides rich preview data for AI chat apps and snippet cards.
Indexable Page Directives
6 pts (Low)Ensures the page has no unintended noindex or nofollow robot tags.
Why it matters: Guarantees AI bots and search agents can index and cite your pages.
Frequently Asked Questions
What is an AI Agents Scanner and why does it matter?
+
Modern search and workflow engines (like Perplexity, ChatGPT Search, Claude, and Gemini) rely on machine-readable discovery standards. An AI Agent Audit checks if your domain provides structured guidance, permissions, and metadata so AI crawlers index, cite, and integrate your service accurately.
What is /llms.txt?
+
llms.txt is an open standard proposed by Jeremy Howard and adopted across the AI ecosystem. It is a concise markdown file hosted at the root of a domain that guides Large Language Models to the most relevant information and documentation on your site.
How does the Model Context Protocol (MCP) fit in?
+
MCP (Model Context Protocol) is Anthropic’s open standard connecting AI models with local or remote tool servers. Publishing an MCP server card at /.well-known/mcp.json lets autonomous AI agents discover and call your APIs natively.
How is the AI Readiness Score calculated?
+
The readiness score is a weighted composite (0-100) based on 9 scored signals, including llms.txt availability (18%), AI crawler rules in robots.txt (18%), Schema.org JSON-LD (14%), sitemap reachable (12%), meta context (10%), llms-full.txt (8%), canonical tags (8%), Open Graph (6%), and indexability (6%). Protocol endpoints like MCP and UCP are audited as informational capabilities.