All posts
SEO / AEO / GEO7 min read2026-10-20

Is your website ready for AI agents? A practical checklist

AI agents read your site differently than a browser. Here's the checklist for making your website discoverable and readable by AI, from robots.txt to llms.txt to markdown.

Why AI agents read your site differently

A browser renders a page. An AI agent tries to extract meaning from it. That sounds like a small difference, but it changes what your website needs to do.

When ChatGPT, Perplexity, or an AI search engine visits your site, it's not looking at a designed page. It's looking for text it can understand, answers it can lift out, and clues about who you serve. If your site hides the important text behind scripts or expects a browser to render it, the agent sees nothing useful.

This matters because more people get their answers from AI, not from browsing ten search results. If an AI can't read your site, it can't cite you. If it can't cite you, it doesn't bring you customers.

The checklist for AI readiness

Making a website readable by AI comes down to a handful of concrete, testable things. Most are quick to set up.

  • Let AI crawlers reach you. Your robots.txt is the front door. Check that it doesn't block the AI crawlers you want, like GPTBot, OAI-SearchBot, or ClaudeBot. Many sites block them by accident while trying to stop scrapers.
  • Declare your content policy. Content Signals in robots.txt (the ai-train, search, and ai-input directives) tell AI systems what they may do with your content. It's a preference, not a law, but well-behaved systems respect it.
  • Keep your sitemap current. It tells crawlers which pages exist. If it's stale, AI and search engines crawl a site that doesn't match what you actually have.
  • Publish an llms.txt file. It's a plain text file at your domain root that gives an AI a map of your most important pages, a short summary of what you are, and links to what matters.
  • Serve markdown, not just HTML. AI agents parse clean text far more easily than HTML full of navigation and styling. Serve a markdown version of your pages, either at a predictable URL or when an agent asks for it.
  • Use semantic HTML. Real headings, meaningful links, and proper structure help AI (and everyone else) understand your page. If your content is locked in images or rendered only by JavaScript, AI can't read it.

Readiness is about being ready for something

Here's the part most checklists leave out. Agent readiness is not a single score that says you did it. It's a question of being ready for the right kind of AI.

A content site and a web app need different things. A store needs agents to be able to transact. A web app may need agents to log in. A content site mostly needs agents to be able to read and cite its pages.

There are tools that scan a site and score its agent readiness. Cloudflare's scanner (at isitagentready.com) is one. When you scan, it asks what kind of site you are. Choose content site and the score reflects what matters for a content business, which is whether AI can read and trust what you publish.

The usefulness of the score depends on picking the right lens. A content site scoring perfectly on an app checklist tells you nothing real.

What this gets you

A site that AI can read is a site that AI can cite. Picture a homeowner asking ChatGPT which plumbing company handles tankless water heaters in their area. The answer comes from sources the AI can actually read. If your site is one of them, you're the answer. If it's not, you're invisible to that entire conversation.

What we do at Pistevo

We built our own site to be readable by AI, and we run the same checks for the businesses we work with. We look at your robots.txt, your llms.txt, your markdown, and how your content reads to a machine. Then we fix what's standing between you and an AI citation. No vague promises of rankings. Just a site that AI can actually read and recommend.

Ready to put this into practice?

We help businesses of every size get found, get chosen, and run on autopilot.

Talk to us