AI Agent Readiness Checker
An AI agent readiness check tests whether AI crawlers and agents can reach, read and use a web page. This free checker tests robots.txt rules for 11 AI crawlers, firewall blocks, server-rendered content, structured data, llms.txt, labelled forms and buttons, and WebMCP tools, following Chrome Lighthouse’s Agentic Browsing audit.
How to make your website ready for AI agents
- Check a page. Enter a page URL (your homepage or an important landing page) and run the check.
- Unblock AI search crawlers. If OAI-SearchBot, Claude-SearchBot or PerplexityBot are blocked, add the suggested robots.txt group and check your CDN or firewall bot settings.
- Make the content readable without JavaScript. Make sure the main text, title and description are in the HTML the server sends, not added later by scripts.
- Fix forms and buttons. Give every form field a label and every icon-only button an accessible name, so agents know what each control does.
- Add optional AI guidance. Publish an llms.txt file and, if you want to experiment, annotate key forms with WebMCP toolname and tooldescription attributes.
- Re-check and confirm in Lighthouse. Run this checker again, then run Chrome Lighthouse’s Agentic Browsing category for a full JavaScript-rendered audit.
What does the AI agent readiness checker test?
| Area | What we test | Why it matters |
|---|---|---|
| Access | robots.txt rules per AI crawler, robots.txt errors, firewall blocks, noindex, HTTPS | A blocked or erroring site can’t be read or cited at all. |
| Readability | Text in the initial HTML, title, meta description, one H1, language, canonical URL | AI crawlers read the HTML the server sends; most don’t run JavaScript. |
| Guidance | Valid JSON-LD, llms.txt (H1, Markdown links, no server errors), XML sitemap | Machine-readable facts and maps reduce guessing. |
| Usability | Labelled fields, named buttons and links, no clickable elements inside aria-hidden, a main landmark | Agents act through the accessibility tree, the same names screen readers use. |
| WebMCP | toolname/tooldescription on forms, document.modelContext.registerTool in inline scripts | Lets agents call site actions as typed tools instead of clicking (experimental). |
| Stability | Images without width and height, plus a link to measure real CLS | Shifting layouts make agents click the wrong element. |
Which AI crawlers should you allow?
AI companies run different bots for search, for fetching pages on a user’s request, and for training. Blocking a search bot removes you from that assistant’s answers; blocking a training bot does not.
| User agent | Operator | Type | What it does |
|---|---|---|---|
OAI-SearchBot |
OpenAI | AI search | Shows your pages in ChatGPT search results |
Claude-SearchBot |
Anthropic | AI search | Indexes pages to improve Claude’s search answers |
PerplexityBot |
Perplexity | AI search | Surfaces and links your pages in Perplexity answers |
ChatGPT-User |
OpenAI | User request | Opens a page when a ChatGPT user asks about it |
Claude-User |
Anthropic | User request | Opens a page when a Claude user asks about it |
Perplexity-User |
Perplexity | User request | Opens a page when a Perplexity user asks about it |
GPTBot |
OpenAI | Training | Collects content to train OpenAI models |
ClaudeBot |
Anthropic | Training | Collects content to train Claude models |
Google-Extended |
Training | Controls use in Gemini training and grounding; no effect on Google Search | |
Applebot-Extended |
Apple | Training | Controls use in Apple’s AI model training |
CCBot |
Common Crawl | Training | Open web archive widely used to train AI models |
What the data says about AI crawlers and agents
| Finding | Source |
|---|---|
| Chrome Lighthouse added an Agentic Browsing category in May 2026. It checks the accessibility tree, layout stability (CLS), llms.txt and WebMCP tools, and reports a pass ratio instead of a 0–100 score. | Chrome for Developers |
| ChatGPT’s and Claude’s crawlers download JavaScript files but don’t execute them, so content rendered only in the browser is invisible to them. | Vercel and MERJ crawler study |
| OpenAI runs separate crawlers: GPTBot collects training data, OAI-SearchBot powers ChatGPT search. Each is controlled by its own robots.txt rules. | OpenAI crawler documentation |
| Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude’s search answers. | Anthropic support |
| Perplexity-User fetches pages when a user asks and generally ignores robots.txt, because the request is user-initiated. | Perplexity documentation |
| Google-Extended controls Gemini training and grounding only; it does not affect inclusion or ranking in Google Search. | Google Search Central |
Our take: treat AI search crawlers like search engines and let them in; decide about training crawlers separately. Then fix what every agent depends on: content in the HTML, labelled forms and named buttons. llms.txt and WebMCP are worthwhile extras, not the foundation.
How is this different from Lighthouse’s Agentic Browsing audit?
| This checker | Lighthouse Agentic Browsing | |
|---|---|---|
| How it reads the page | Initial HTML, like AI crawlers | Fully rendered in Chrome |
| AI crawler robots.txt rules | Yes, 11 crawlers | No |
| Firewall / bot-block test | Yes | No |
| llms.txt | Yes, same criteria | Yes |
| WebMCP tools | Declarative and inline scripts | All, including JavaScript-registered |
| Accessibility tree | Approximated from markup | Computed by Chrome |
| Layout shift (CLS) | Image-size hint and PageSpeed link | Measured |
| Fixes | Copy-paste code | Guidance links |
Frequently asked questions
What does it mean for a website to be AI agent ready?
An AI-agent-ready website lets AI crawlers reach its pages, serves its content in plain HTML they can read, and exposes forms and buttons with clear names so agents can act on them. Optional extras such as llms.txt and WebMCP tools make the site easier for agents to navigate and use.
Should I block GPTBot?
That is a business choice. GPTBot only collects training data; OpenAI’s search crawler is a separate bot, OAI-SearchBot. Blocking GPTBot keeps your content out of future training while still allowing your pages to appear in ChatGPT search, as long as OAI-SearchBot is allowed.
Does blocking AI crawlers affect Google Search or AI Overviews?
Blocking Google-Extended does not: Google says it has no effect on Search inclusion or ranking. Google’s AI Overviews draw on the normal Search index, which Googlebot builds, so only blocking Googlebot would remove you from them.
Why does the checker say my content is rendered by JavaScript?
The page’s initial HTML contains very little text while relying on scripts. ChatGPT’s and Claude’s crawlers don’t execute JavaScript, so they see that near-empty HTML. Server-side rendering, static generation or plain HTML fixes it.
What is WebMCP?
WebMCP is an experimental proposal, currently behind a Chrome flag, that lets a page expose actions to AI agents as typed tools. Forms can declare a tool with toolname and tooldescription attributes, and scripts can call document.modelContext.registerTool(). See Chrome’s WebMCP documentation.
Does Lighthouse check AI agent readiness?
Yes. Since May 2026 Lighthouse has an Agentic Browsing category covering accessible names, layout stability, llms.txt and WebMCP. It shows a pass ratio rather than a score. It runs in a real browser but doesn’t check robots.txt rules for AI crawlers or firewall blocks, which this checker does.
The firewall check failed. What should I do?
Your CDN or security plugin answered AI crawler user agents differently from a browser. Look for “block AI bots” or bot-fight settings in Cloudflare, your host or your security plugin, and allow verified AI search crawlers. Genuine crawlers come from published IP ranges, so confirm in your server logs.
Is this AI agent readiness checker free and private?
Yes. It is free with no sign-up. Our server fetches the page you enter plus its robots.txt, llms.txt and sitemap, and sends test requests with AI crawler user agents. Results are cached for 10 minutes and nothing else is stored.
Last reviewed against Chrome’s Lighthouse Agentic Browsing docs and the crawler documentation of OpenAI, Anthropic, Perplexity and Google.