Specification v1.0
The Agent Experience specification
Five pillars, 53 checks, published in full. This page is generated from the scoring code itself, so it cannot drift from what the scanner actually does.
Why a specification and not just a tool
A score is only worth something if you can argue with it. Publishing the rubric means you can check our reasoning, disagree with a weight, and hold us to the same standard we apply to you. It also means the number survives us: anyone can implement it.
The rubric is versioned. Scores are comparable within a major version and we will not silently reweight underneath you. Reports record the spec version they were produced against.
Scoring model
- Each check earns points toward its pillar: full for a pass, half for a warning, zero for a failure.
- Checks marked not applicable are removed from the denominator entirely.
- Pillar scores are 0–100 within the pillar, then combined by the weights below.
- Gates cap the total when a failure makes the average misleading.
Reachability
weight 25 · 13 checksCan an agent fetch your content at all, without a browser and without being blocked?
| Check | Why an agent cares | Points |
|---|---|---|
| http-ok Page responds to a plain HTTP request |
The baseline: an agent with no browser, no cookies and no JavaScript received your page. | 10 |
| not-blocked No bot wall in front of the content |
Bot protection that cannot tell a shopping assistant from a scraper silently removes you from AI answers, with no error anyone on your team will ever see. | 12 |
| server-rendered Content exists without running JavaScript |
This is the single most expensive AX failure. To a crawling agent the page is blank, so it cannot be summarised, cited or ranked at all. | 14 |
| robots-exists robots.txt is present and parseable config only |
Without a robots.txt every crawler applies its own defaults and you have no way to express a preference. It is also the file most agents read first. | 4 |
| robots-answer-engines Answer engines are allowed to crawl |
These crawlers are the ones that put your link inside an AI answer. Blocking them removes the referral traffic, not just the training use. | 12 |
| robots-user-triggered User-triggered fetchers are allowed |
These fetches happen because a human just pasted your link into an assistant. A block here fails in front of a real person who wanted to read you. | 8 |
| robots-search Classic search crawlers are allowed |
AI Overviews and Copilot answers are grounded on the classic search index. Blocking Googlebot or Bingbot now also removes you from AI surfaces. | 6 |
| agent-live-probe Real agent user-agents are served normally |
Some sites answer 200 to a generic client and 403 to a named AI crawler. Only a live probe finds that. | 8 |
| meta-robots No directive suppressing AI use of this page |
noindex and nosnippet remove you from the surfaces that would otherwise quote and link you. | 6 |
| redirect-chain Short redirect chain |
Every hop costs latency and some agents cap redirects at two or three. | 4 |
| response-speed Responds fast enough for an agent budget |
Agents fetch many pages per answer and drop the slow ones. Speed is a ranking input you control. | 5 |
| payload-weight HTML payload stays inside an agent context budget |
Agents truncate long documents. Anything past the cut simply is not read. | 5 |
| content-type Declares a correct content type config only |
Without a declared charset, accented and non-Latin text can be mangled in AI answers that quote you. | 3 |
Comprehension
weight 25 · 12 checksOnce fetched, can the agent work out what this page is and what it says?
| Check | Why an agent cares | Points |
|---|---|---|
| title Has a descriptive <title> config only |
The title is the label an assistant uses when it lists you among its sources. Without one you are cited as a bare URL, which almost nobody clicks. | 10 |
| meta-description Has a meta description config only |
It is the one sentence you get to write yourself. Without it the assistant summarises you from whatever text it happened to read first, which is often your cookie banner. | 8 |
| h1 Exactly one <h1> naming the page config only |
Without an h1 the agent has no anchor for the document topic and falls back to the title tag or the URL slug. | 8 |
| heading-hierarchy Headings form a sane outline config only |
Agents chunk long pages by heading. A page with no outline is chunked arbitrarily and quoted out of context. | 6 |
| content-volume Enough substance to be worth citing |
There is not enough here for an assistant to build an answer from, so it will use someone else as the source. | 10 |
| text-ratio Signal-to-markup ratio |
The document is almost entirely machinery. An agent truncating this page will likely truncate before reaching any of your content. | 8 |
| lang Declares a document language config only |
Without it an assistant guesses the language, and guesses wrong on short or mixed-language pages, producing garbled quotes. | 5 |
| image-alt Images carry alt text |
Agents cannot see images. Alt text is the only description they get. | 8 |
| landmarks Main content is marked as such config only |
Without a main landmark an extractor has to guess your content boundary heuristically, and on marketing pages it usually guesses the header. | 6 |
| semantic-density Uses semantic elements, not div soup |
6 | |
| noscript-fallback Provides a no-JavaScript fallback |
4 | |
| readable-url URL describes the content |
Assistants show the URL next to the citation, and readable URLs get clicked more than opaque ids. | 3 |
Structure
weight 18 · 10 checksIs the important data machine-readable instead of implied by visual layout?
| Check | Why an agent cares | Points |
|---|---|---|
| jsonld-present Ships structured data config only |
Structured data is the only part of your page an agent can consume without interpretation. Everything else it has to infer, and inference is where it invents facts about you. | 12 |
| jsonld-valid Structured data parses cleanly |
8 | |
| jsonld-types Structured data describes the right thing |
8 | |
| open-graph Open Graph card is complete config only |
When a page is JavaScript-heavy or truncated, Open Graph tags are frequently the only clean summary an agent gets. | 6 |
| canonical Declares a canonical URL config only |
Agents dedupe sources by canonical URL. Without one, the same content behind different query strings competes with itself and dilutes every citation. | 6 |
| sitemap Publishes a sitemap config only |
Without a sitemap, discovery depends entirely on crawlable links. Any page not reachable by a plain <a href> is invisible. | 8 |
| price-structured Prices are machine-readable |
6 | |
| breadcrumbs Declares its place in the site |
4 | |
| twitter-card Declares a link preview card config only |
Several agents fall back to card metadata when they cannot extract the body reliably. | 3 |
| microdata Inline microdata annotations |
2 |
Actionability
weight 17 · 10 checksCan the agent take the next step - navigate, search, submit, buy - without guessing?
| Check | Why an agent cares | Points |
|---|---|---|
| links-crawlable Navigation uses real links |
An agent moves through a site by following hrefs. With none, this page is a dead end and everything behind it is undiscoverable. | 12 |
| anchor-text Link text describes the destination |
8 | |
| internal-nav Links onward into the site |
A page with no onward links is a cul-de-sac. Everything you publish that is not linked from a crawlable page effectively does not exist. | 6 |
| form-labels Form fields are labelled |
8 | |
| form-names Form fields have stable names |
6 | |
| form-autocomplete Fields declare autocomplete tokens |
4 | |
| button-semantics Controls are real buttons |
6 | |
| contact-affordance Reachable by a human, discoverably config only |
"How do I get in touch with them" is one of the most common agent-mediated questions about a company. Make the answer machine-readable. | 5 |
| search-affordance Site search is machine-usable config only |
Without a GET-addressable search, an agent looking for one specific page on your site has to fetch dozens of pages to find it, and usually gives up first. | 4 |
| api-discovery Programmatic surface is discoverable config only |
This is the frontier of AX: sites that publish a machine-callable surface get used by agents directly instead of being scraped approximately. | 6 |
Agent Contract
weight 15 · 8 checksHave you told agents the rules: what they may use, where the docs are, who to contact?
| Check | Why an agent cares | Points |
|---|---|---|
| llms-txt Publishes an llms.txt config only |
Without it, an assistant deciding what your site is about crawls whatever it happens to find. llms.txt is the cheapest way to control that first impression, and today most of your competitors do not have one either. | 14 |
| llms-txt-quality llms.txt is actually useful |
6 | |
| ai-policy States an explicit AI policy config only |
With no explicit policy, every operator applies its own default to you and you have no documented position if you later object. | 6 |
| mcp-manifest Exposes an agent-callable manifest |
This is the highest-leverage unclaimed slot on the web right now. The sites that publish a callable surface get used directly instead of guessed at. | 6 |
| identity Identity is machine-verifiable config only |
Entity resolution is how assistants decide two mentions are the same company. Unresolved entities get merged with, or mistaken for, someone else. | 5 |
| freshness Says when the content changed config only |
Agents deprioritise content they cannot date, and re-fetch undated pages more often, which costs you bandwidth for nothing. | 5 |
| licence-signal Declares content licensing config only |
If you care how your content is reused, say so where machines can read it. A footer copyright line is not machine-readable. | 5 |
| security-txt Publishes security.txt config only |
It is the standard machine-readable place to say who to contact. Agents and researchers both look there first. | 4 |
Gates
These caps override the weighted average, because some failures make a good average untrue.
| Condition | Score capped at |
|---|---|
| The page does not return a successful response | 20 |
| A bot wall or challenge answers instead of the page | 30 |
| robots.txt blocks answer engines, user fetchers and search | 30 |
| A robots directive suppresses indexing or snippets | 50 |
| No content ships without JavaScript | 62 |
Grade bands
A+ 95+ · A 85–94 · B 72–84 · C 58–71 · D 44–57 · E 28–43 · F below 28
Agent roster
robots.txt is resolved for each of these crawlers individually, grouped by what blocking them costs you.
Answer engine
Indexes you so an assistant can cite and link you. Blocking it removes you from AI answers.
OAI-SearchBot— OpenAI. You will not appear as a source in ChatGPT search results.Claude-SearchBot— Anthropic. You will not be cited in Claude web search answers.PerplexityBot— Perplexity. You lose Perplexity citations, one of the highest-converting AI referral sources.Amazonbot— Amazon. Excluded from Alexa and Rufus style shopping answers.DuckAssistBot— DuckDuckGo. Excluded from DuckDuckGo AI assist answers.
User-triggered
Fetches your page because a human asked an assistant to open it right now. Blocking it breaks a live request.
ChatGPT-User— OpenAI. When a user asks ChatGPT to open your link, it fails in front of them.Claude-User— Anthropic. Claude cannot open your page when a user pastes the link.Perplexity-User— Perplexity. Live user-requested fetches from Perplexity fail.
Classic search
Traditional search indexing, increasingly the substrate AI answers are built on.
Googlebot— Google. You are invisible to Google Search and to AI Overviews built on it.Bingbot— Microsoft. You are invisible to Bing and to Copilot answers grounded on it.
Training crawler
Collects content for model training. Blocking it is a legitimate business choice with no traffic cost.
GPTBot— OpenAI. Your content is excluded from OpenAI model training.ClaudeBot— Anthropic. Your content is excluded from Anthropic model training.Google-Extended— Google. Excluded from Gemini training and grounding.Applebot-Extended— Apple. Excluded from Apple Intelligence training.meta-externalagent— Meta. Excluded from Meta AI training.CCBot— Common Crawl. Excluded from Common Crawl, the base corpus behind most open models.