Website Checklist for Competing in AI Search Results
Check AI search readiness from access to trust: GEO, AEO, LLMO, AAO, RAG, E-E-A-T, llms.txt, sitemaps and the technical foundation.
Introduction: Why Traditional SEO Is Not Enough
Search engines no longer rely solely on inverted text indexes and keyword frequencies. Systems like ChatGPT Search, Perplexity, Claude, and Google Gemini synthesize answers using Retrieval-Augmented Generation (RAG). If your technical infrastructure blocks, obfuscates, or fragments content, generative models either fail to retrieve your site or hallucinate incorrect details.
Pillar 1: Explicit AI Crawler Governance (RFC 9309)
Modern crawlers must be managed intentionally in robots.txt. Do not rely on a generic User-agent: * rule, which often inadvertently blocks specialized search user-agents while intending only to stop bulk scrapers.
# Recommended robots.txt configuration
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /private/
Allow: /Pillar 2: Clean HTTP Edge Responses & HSTS
AI crawlers operate with strict latency budgets. Multi-hop 301 redirects or slow TTFB (>1.5s) frequently cause retrieval agents to drop candidate documents. Ensure HSTS is enabled with max-age=31536000; includeSubDomains; preload to prevent insecure protocol downgrades.
Pillar 3: The Single Primary H1 and Semantic Heading Hierarchy
Documents with multiple H1 tags or arbitrary heading jumps (e.g. H1 directly to H4) confuse semantic parsers that extract page outlines. Every indexable page must feature exactly one H1 expressing the primary subject, followed by cleanly nested H2 and H3 elements.
Pillar 4: JSON-LD Entity Graphs and Authorship
Unambiguous structured data connects your content to the global knowledge graph. Always declare the Organization, Author, and WebSite entities using JSON-LD rather than deprecated Microdata formats.
Pillar 5: Implementation of llms.txt (v2 Specification)
Place a clean, curated /llms.txt file in your domain root. Use markdown headings, blockquote descriptions, and clean URLs linking directly to authoritative documentation.
Pillar 6: Accurate Canonical Tags & Self-Referential Integrity
Every indexable page must contain a self-referential canonical tag matching its exact scheme, apex or www host, and path. Conflicting canonicals cause AI crawlers to discard extracted metadata.
Pillar 7: Programmatic Accessibility & Form Labels
Neural models understand accessible HTML with greater fidelity. Provide explicit aria-label or <label for="..."> associations on all interactive controls, and maintain valid <html lang="..."> attributes.
Pillar 8: Security Baseline & Clickjacking Defenses
Modern AI agents verifying site trust check for baseline HTTP headers: X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, and sensible Content-Security-Policy rules.
Pillar 9: Elimination of 404s and Broken Internal Links
AI agents exploring internal link graphs stop crawling when encountering repeated dead ends. Probe internal links regularly and eliminate circular redirects.
Pillar 10: Rel=Describedby and Markdown Alternates
Advertise machine-readable documentation directly within HTTP Link headers or HTML <link rel="describedby" href="/llms.txt"> elements so AI bots discover structured documentation immediately.
To resolve detected blockers with root cause analysis, acceptance tests and rollbacks for your engineering team, $99 Full Site Fix Mandate is available.