# The AI Readiness Index **Version 2.0 — specification** **License:** This document and the checks it defines are published under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Copy it, adapt it, build your own scanner from it — for any purpose, free, forever — as long as you credit TEN7 as the creator of the AI Readiness Index and license what you build the same way. The scanner that computes a score from this document is separate, proprietary software. Full terms at [air.ten7.com/license](https://air.ten7.com/license). --- ## What this is This document defines what a website has to do to score 100 on the AI Readiness Index. It is written for two readers at once: the engineer building the scanner in this repository, and the consultant explaining a score to a client. Every check has an ID, a weight, a detection method, and a scoring rubric. If a check is not described here, the scanner should not score it. We built this because "is your site ready for AI" is currently an unanswerable question. Vendors assert things. Nobody measures. Our answer is a number you can reproduce, defend line by line, and re-run in six months to show what changed. The Index measures one thing: **can machines find, read, trust, and use this website?** It does not measure design quality, marketing effectiveness, or whether the content is any good. A beautiful site can score 12. A plain one can score 94. --- ## How to read a score A result has three parts, and all three matter. **The score (0–100).** A single weighted number across eight dimensions. **The gate status.** Two checks are gates. If a site blocks AI crawlers in `robots.txt` or rejects them at the CDN, the total is capped at 40 no matter how good the rest is. A perfectly marked-up site that returns 403 to `ClaudeBot` is not ready — it is invisible. The cap makes that impossible to hide behind a good average. **The automated coverage.** The percentage of applicable weight the scanner verified on its own. Some checks — log retention, a monthly prompt panel, whether schema is generated from fields or hardcoded in a template — cannot be seen from outside. Those are scored from attestation and flagged. A score of 82 with 91% automated coverage is a much stronger claim than 82 with 64%. Report all three together. A bare number invites arguments we don't need to have. ### Grade bands | Score | Grade | What it means | |---|---|---| | 90–100 | Exemplary | Machine-legible by design. Remaining work is optional or emerging. | | 75–89 | Strong | Solid fundamentals with specific, addressable gaps. | | 60–74 | Adequate | Findable and readable, but leaving significant value unclaimed. | | 40–59 | Weak | Partially legible. Real content is not reaching answer engines. | | 0–39 | Not ready | Structural problems. Start with retrievability before anything else. | | Any | **Gated** | A gate failed. Fix access first; the rest of the score is theoretical. | When a gate fails, `grade` is `Gated`. The result also carries `ungated_grade` — the band the ungated score would have earned — so the report can say "Gated, and Strong at 81.2 once access is fixed" rather than leaving the consultant to supply the adjective. --- ## The scoring model ### Bands Every check scores on a 0–4 band. The check's contribution is `weight × (band ÷ 4)`. Unless a check says otherwise, use the **default coverage rubric** against the sampled page set: | Band | Condition | |---|---| | 4 | Present and correct on ≥95% of applicable sampled pages | | 3 | Present and correct on 75–94% | | 2 | Present on 40–74%, or present everywhere but with correctness errors | | 1 | Present on 1–39% | | 0 | Absent, or present but broken everywhere | Coverage percentages are weighted, not raw page counts. Each money page (see Sampling, step 4) counts as **2** in both the numerator and the denominator; every other sampled page counts as 1. Ten money pages in a 250-URL sample therefore score as a 260-page sample. A failure on `/apply` costs twice what the same failure on a 2019 press release costs, which is the reason the client names those URLs in the first place. Checks that are inherently site-wide and binary (a file exists, a header is set) define their own bands in the check entry. ### Weights | # | Dimension | Weight | Checks | |---|---|---|---| | 1 | Retrievability and rights | 20.0 | 10 | | 2 | Crawl and index hygiene | 12.0 | 8 | | 3 | Rendering and extraction | 20.0 | 8 | | 4 | Alternate representations | 8.0 | 6 | | 5 | Structured data and the entity graph | 22.0 | 12 | | 6 | Authorship, provenance, freshness | 10.0 | 8 | | 7 | Agent interfaces | 5.0 | 9 | | 8 | Measurement and instrumentation | 3.0 | 7 | | | **Total** | **100.0** | **68** | Dimensions 1, 3 and 5 carry 62 of the 100 points. That is deliberate. Access, extraction, and entity identity determine whether anything else matters. Dimension 8 carries only 3 points despite being the most commercially valuable work we do. Weight here reflects what a scanner can observe from outside a site, not what the work is worth. Track measurement maturity in the engagement, not in this number. ### Applicability and N/A Each check declares an applicability rule: - **`always`** — scored on every site. - **`conditional`** — scored only if a precondition holds. State the precondition in the check. - **`vertical`** — scored only for a declared client vertical (higher ed, healthcare, nonprofit, government, research). A conditional check states its precondition in one of two forms, and never in both: - **A named site fact** — a single named predicate over collected evidence, such as `sample_contains_tables` or `site_has_search`. Predicates are code, not expressions in a config file, so each one is unit-tested and each one names itself in the N/A reason an auditor reads. - **A dependency on another check** — written as `requires: AR-1.6 ≥ 2`. A dependency is satisfied when the named check scores **band 2 or higher**. Below band 2 the foundation is broken, and measuring what sits on top of it produces a finding nobody can act on: telling a client their malformed `llms.txt` is well cached is noise. A dependency that is itself N/A cascades N/A to everything that depends on it. Five checks carry a dependency: AR-1.7 → AR-1.6, AR-4.4 → AR-4.3, AR-4.6 → AR-4.1 **or** AR-4.3, AR-7.2 → AR-7.1, AR-8.4 → AR-8.3. #### Facts the scanner cannot observe Six checks are conditional on a business fact no crawler can determine: AR-1.6 and AR-1.8 (licensing and access posture), AR-6.3 (YMYL content), AR-6.5 (research and policy publishing), AR-6.6 (original asset production), and AR-7.8 with AR-7.9 (named critical flows). AR-5.5 needs a declared vertical. These come from an engagement profile supplied per client — never from inference. A fact not declared is N/A with the reason `not_declared_in_engagement_profile`. The scanner does not sniff a site for medical keywords and decide it is YMYL. A guess inside a number we ask a client to defend line by line is worse than a blank row, and a blank row is an easy conversation: *we did not assess this because you have not told us yet.* A pre-sales scan run before any client contact will therefore leave those seven checks unassessed. The score renormalizes correctly around them. A check ruled N/A is removed from the denominator. Recompute: ``` score = (Σ earned_points ÷ Σ applicable_weight) × 100 ``` Never score an N/A check as zero. Penalizing a museum for having no `Physician` markup makes the Index useless. Record every N/A decision with its reason in the result file. An auditor must be able to see what we chose not to measure. ### Gates Two checks are gates: **AR-1.1** and **AR-1.2**. If either scores 0 or 1, cap the final score at 40 and set `gated: true`. Report the uncapped score alongside it as `ungated_score` and `ungated_grade`, so the client can see what the site would be worth once access is fixed. That contrast is usually the most persuasive number in the whole report. The cap is a floor-preserving minimum: `total = min(ungated_score, 40)`. A site scoring 22 that also fails a gate stays at 22. Failing a gate never raises a score. ### Verification mode Every check declares how its band was determined: - **`auto`** — the scanner determined the band from evidence it collected. - **`auto-partial`** — the scanner produced a strong signal, but a human must confirm. Heuristics that can be fooled live here. - **`manual`** — not observable from outside. Scored from a client attestation, or left `null` and excluded. Automated coverage counts `auto-partial` at half weight, because a strong signal awaiting human confirmation is genuinely half-verified: ``` automated_coverage = (Σ auto_weight + 0.5 × Σ auto_partial_weight) ÷ Σ applicable_weight ``` The half is not arbitrary. Counting `auto-partial` at zero puts a ceiling of 0.83 on every possible run, because the always-applicable `auto-partial` checks are 15.75 points that never leave the denominator. Counting it in full reports 0.96 for a site nobody has confirmed anything about, and erases the distinction the mode exists to draw. Never let a `manual` check silently default to 4. An unanswered attestation scores `null` and drops out of the denominator, and the result records it as unverified. #### Attestation against an `auto-partial` check Four `auto-partial` checks have a top band that asks something no scanner can see. This document says so in each case, and each has a different observable ceiling because their band tables differ: | Check | Unattested ceiling | What the attestation confirms | |---|---|---| | AR-2.8 | 3 | That IndexNow submissions fire on publish, not on a schedule | | AR-4.1 | 3 | That `llms.txt` is generated from the content model rather than hand-maintained | | AR-5.1 | 3 | That JSON-LD comes from mapped fields, confirmed with the build team | | AR-8.7 | 3 | That sitemaps are submitted and alerts reach a named person | Each of these four now has a band 3 that describes exactly what a scanner can see, so the attestation lifts 3 to 4 rather than jumping a gap. A check whose band table skips a value cannot express *mostly right*, and *mostly right* is the state most real sites are in. An attestation may **raise the ceiling** on these four. It never sets a band and never lowers one: the scanner scores what it observed, and a supplied attestation permits that observed band to rise as far as the evidence supports. The result records which points came from a claim rather than a probe, and the report prints them as attested. Without an attestation the check scores at its observable ceiling and the report says what would be needed to go higher. A client who did the work can prove it; a client who says nothing earns nothing. Every other `auto-partial` check is fully observable. It carries that mode because the heuristic can be fooled — AR-1.8 compares a signed request against an unsigned one, AR-7.3 calls a live tool — not because it needs a client to vouch for it. #### Checks that need a previous run Four checks are defined against change over time: AR-2.5 (is `lastmod` a real signal or deploy noise), AR-3.6 (do anchor IDs survive a deploy), AR-6.4 (do dates move only when content moves), and AR-7.9 (do critical-flow selectors churn). On a first run there is nothing to compare against. Score the observable half, **cap the band at 3**, set `confidence: medium`, and record `stability_unconfirmed` in the evidence. AR-6.4 already words its band 3 this way; the other three follow the same rule. Supplying a previous run directory as a baseline lifts the cap and permits band 4. Every ceiling — this one, the attestation ceiling above, and a check's own declaration that it could not see further — records the reason it applied. A client told a number was held below the evidence is owed the sentence explaining what would lift it. This makes the re-audit a first-class input to the score rather than a report-writing exercise, and it gives the client a visible, earnable reason the number rises next quarter. --- ## How the audit runs ### Sampling Most checks score against a sampled page set, not the whole site. Build the sample like this: 1. **Always include:** the homepage, `/robots.txt`, `/sitemap.xml` (and every child sitemap), `/llms.txt`, `/.well-known/` probes, and every URL in the site's primary navigation. 2. **Stratify by content type.** Use the sitemap index, URL patterns, or a declared content-type map. Take up to 10 URLs per type. 3. **Add depth.** Include at least 5 URLs three or more levels deep, to catch templates the homepage never exercises. 4. **Add the money pages.** The client names up to 10 URLs that matter most — programs, services, donate, apply, contact. These are weighted double in coverage math. 5. **Cap the crawl.** Default 250 URLs. Configurable. Record the actual count. Minimum viable sample is 25 URLs. Mark the run `low_confidence` only when the sample is under 25 **and** more URLs were discovered than were sampled. A twelve-page nonprofit site that was crawled in full is a complete measurement, not a weak one. #### The representation sub-sample Four checks want an extra request against every sampled URL: AR-4.3 (`{path}.md`), AR-4.5 (`Accept: text/markdown`), AR-4.6 (cache behavior), and AR-1.9 (`X-Robots-Tag` on non-HTML). At the default 250-URL cap and one request per second, that is over 800 requests and roughly fifteen minutes of politeness delay stacked on top of the base crawl, against infrastructure that often belongs to a hospital. Those four checks run against a **representation sub-sample**: 25 URLs by default, drawn from the main sample, always including every money page. The size is configurable and the selected URLs are recorded in the run metadata, so the sub-sample is reproducible across audits. Twenty-five pages resolves a rubric whose finest distinction is 75% against 95%. ### Fetching Fetch every sampled URL twice: - **Raw** — plain HTTP GET, no JavaScript, with a declared TEN7 audit user-agent. - **Rendered** — a headless browser with JavaScript enabled. The difference between those two responses is the entire evidence base for AR-3.1 and AR-3.2, which together carry 9 points. Do not skip the second fetch. Separately, probe a small set of URLs with each AI crawler user-agent to feed AR-1.2. Identify yourself honestly in a `From` header. We are auditing on the client's behalf, with permission, and we do not pretend to be somebody else's bot. #### The agent classes The Index groups the ten agents three ways. AR-1.1 evaluates all ten against `robots.txt`. AR-1.2 probes only the eight that actually issue requests. | Class | Agents | AR-1.1 | AR-1.2 probe | |---|---|---|---| | Answer-serving | `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Claude-User` | yes | yes | | Training and corpus | `GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider` | yes | yes | | Directive-only | `Google-Extended`, `Applebot-Extended` | yes | **no** | `Google-Extended` and `Applebot-Extended` are not crawlers. They are opt-out tokens that Google and Apple honor in `robots.txt`; no request ever arrives carrying either string. Blocking them is a rights decision with no fetch behavior attached, which is why they count in AR-1.1 and cannot be probed in AR-1.2. A user-agent matrix that lists them is describing a crawler that does not exist. Where a band anchor says **major agents**, it means the four answer-serving agents plus `GPTBot` and `ClaudeBot`. ### Politeness Respect `Crawl-delay`. Default to 1 request per second with 4 concurrent workers. Support an allowlist of client-approved hosts and refuse to run against anything else. Cache every response to disk so re-scoring never re-crawls. ### Reproducibility Store the raw evidence — headers, HTML, extracted JSON-LD, screenshots — in a run directory keyed by timestamp. A score you cannot reproduce next quarter is an opinion, not a measurement. --- ## Result schema The scanner writes one `result.json` per run. Everything else (Markdown report, HTML dashboard, remediation backlog) renders from this file. ```json { "schema_version": "1.0", "index_version": "1.0", "site": { "name": "Example University", "base_url": "https://example.edu", "vertical": "higher-ed", "cms": "drupal-10" }, "run": { "id": "2026-09-01T20-14-33Z", "started_at": "2026-09-01T20:14:33Z", "finished_at": "2026-09-01T20:41:02Z", "scanner_version": "0.1.0", "urls_sampled": 187, "representation_sample": 25, "low_confidence": false, "baseline_run_id": null, "extraction_method": "readability-density", "extraction_version": "1" }, "score": { "total": 61.4, "ungated_score": 61.4, "gated": false, "grade": "Adequate", "ungated_grade": "Adequate", "applicable_weight": 94.5, "earned_points": 58.0, "automated_coverage": 0.87 }, "dimensions": [ { "id": 3, "name": "Rendering and extraction", "weight": 20.0, "applicable_weight": 20.0, "earned_points": 9.5, "score_pct": 47.5 } ], "checks": [ { "id": "AR-3.1", "title": "Primary content is present without JavaScript", "dimension": 3, "weight": 6.0, "applicability": "always", "applicable": true, "na_reason": null, "verification": "auto", "band": 1, "earned_points": 1.5, "confidence": "high", "summary": "Raw HTML contains 22% of rendered text on average across 187 URLs.", "evidence": [ { "url": "https://example.edu/admissions", "raw_text_chars": 812, "rendered_text_chars": 9440, "coverage": 0.086 } ], "remediation": { "severity": "critical", "effort": "large", "summary": "Server-render the decoupled front end.", "detail": "Program and admissions templates ship an empty shell..." } } ], "remediation_backlog": [ { "rank": 1, "check": "AR-3.1", "points_available": 4.5, "severity": "critical", "effort": "large" } ] } ``` #### Schema notes - `site.cms` is best-effort, from the `generator` meta tag and response headers. It is nullable and nothing scores against it. - A dimension with no applicable checks reports `score_pct: null`, not `0`. Zero means measured and failed. - **Rounding happens once, at the end.** All arithmetic runs unrounded; the values written to this file are rounded to one decimal for display. The per-check `earned_points` therefore will not sum exactly to `score.earned_points`, and neither is wrong. Do not "fix" this by rounding intermediates — that reintroduces the drift the single rounding step exists to prevent. - `extraction_method` and `extraction_version` record how main-content text was isolated. When extraction improves, old scores stay interpretable because the method that produced them is on the record. ### Remediation ranking Sort the backlog by **points available** (`weight − earned_points`), then by effort ascending, then by check ID ascending. The last key is not cosmetic: two runs of the same evidence must produce the same backlog in the same order, or the re-audit comparison is unreadable. A check appears in the backlog only if it is applicable, has a band, and has points available above zero. N/A checks, unattested `manual` checks, and checks already at band 4 are not remediation items. Use three severities — `critical`, `major`, `minor` — and three effort sizes — `small`, `medium`, `large`. Effort is a property of the check, defined in this document. Severity is derived: gate failures and any check scoring **0 or 1** with weight ≥3.0 are `critical`; band ≤2 with weight ≥1.5 is `major`; everything else is `minor`. Band 1 on a heavy check is a critical finding. A site whose program pages ship 22% of their content scores AR-3.1 at band 1, not 0, and calling that `major` alongside a missing caption on a table understates it to the person deciding what to fund. The client-facing output is not the score. It is the ordered list of what to fix and what each fix is worth. --- # The checks Each entry gives: weight, applicability, verification mode, effort, what the check measures, how to detect it, what evidence to keep, and the band rule when it differs from the default coverage rubric. --- ## Dimension 1 — Retrievability and rights **Weight 20.0** Can crawlers and agents reach the site, and what has the organization said they may do with what they take? This dimension comes first because nothing downstream matters if the answer to the first half is no. ### AR-1.1 — No blanket robots.txt blocks on AI crawler user-agents - **Weight:** 5.0 · **Gate** · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Whether `robots.txt` disallows the crawlers that feed answer engines. - **Detect:** Parse `robots.txt`. Evaluate each of `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `PerplexityBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Bytespider` against the rule set. Resolve `User-agent: *` fallbacks correctly, including the longest-match group rule. - **Evidence:** Raw `robots.txt`, the parsed rule tree, and a per-agent allow/deny verdict for `/` and for three sampled content URLs. - **Bands:** anchored on *which* agents are blocked, not how many. - **4** — every listed agent may fetch content paths. - **3** — one training or directive-only agent blocked; every answer-serving agent allowed. - **2** — two or more training or directive-only agents blocked, up to and including all of them; every answer-serving agent allowed. - **1** — exactly one answer-serving agent blocked. - **0** — `Disallow: /` for `*`, or two or more answer-serving agents blocked. - **Why the gate turns on the answer-serving class:** the Index measures whether machines can find, read and *use* a site. An answer engine blocked at `robots.txt` cannot answer a question about the client, which is the failure the gate exists to catch. A training block is a rights posture, and this document declines to judge rights postures — AR-1.4 scores any combination of `yes` and `no` at 4 for exactly that reason. A university that blocks `GPTBot` and welcomes `ChatGPT-User` is reachable, and capping its score at 40 would be measuring our approval rather than machine reach. - **Consequence:** blocking every training crawler costs 2 of the 5 points and does not gate. Blocking a single answer-serving agent gates. - **Absent robots.txt:** a 404 scores 4. Nothing is disallowed, so nothing is blocked. This is correct and frequently queried; a missing file is not a failure of this check. - **Note:** A deliberate, documented block is a business decision, not a defect. Record it, score it honestly, and say in the report that the client chose it. The number should reflect machine reach, not our approval. - **Fix:** Remove inherited blocks. Keep only directives the client can explain. ### AR-1.2 — CDN and WAF do not silently reject AI crawlers - **Weight:** 5.0 · **Gate** · **Applicability:** always · **Verification:** auto · **Effort:** medium - **Measures:** Whether the edge returns real content to AI user-agents, regardless of what `robots.txt` permits. - **Detect:** Request 10 sampled URLs with each of the eight fetching AI user-agents — the answer-serving and training classes defined under Fetching. `Google-Extended` and `Applebot-Extended` are excluded; they issue no requests. Record status code, response size, and whether the body matches the baseline fetch. Flag 403, 429, 503, interstitial challenge markup, and bodies more than 40% smaller than baseline. - **Evidence:** A user-agent × URL matrix of status codes, sizes, and challenge detection. - **Bands:** 4 — all agents get 200 with full content. 3 — one agent rate-limited but eventually served. 2 — one agent blocked or challenged. 1 — several blocked. 0 — most blocked. - **Fix:** Audit Cloudflare bot settings, Pantheon AGCDN rules, Fastly VCL, and origin WAF. Verify by re-running this probe, not by reading configuration. ### AR-1.3 — No interactive challenges on read-only paths - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Whether anonymous GETs on browse, search, listing, and detail pages return content without a challenge. - **Detect:** Look for Turnstile, reCAPTCHA, hCaptcha, and JS-challenge signatures in raw responses on non-form URLs. Detect meta-refresh interstitials and challenge cookies. - **Evidence:** Per-URL challenge detection with the matching signature. - **Fix:** Scope challenges to POST endpoints and authenticated routes. ### AR-1.4 — Content Signals declared in robots.txt - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Whether the site states a machine-readable position on `search`, `ai-input`, and `ai-train` using Cloudflare's Content Signals Policy. - **Detect:** Parse `Content-Signal:` lines. Validate syntax (`key=yes|no`, comma-delimited) and check for contradiction with `Disallow` rules. - **Bands:** 4 — all three signals declared, syntactically valid, consistent with `robots.txt`. 3 — declared but one signal omitted. 2 — declared with a syntax error or an internal contradiction. 1 — a partial or malformed attempt. 0 — absent. - **Note:** Any combination of `yes`/`no` scores 4. We are measuring that a deliberate position exists, not which position it is. - **Fix:** Run the policy conversation with the client, then add the line. ### AR-1.5 — Human-readable AI policy comment in robots.txt - **Weight:** 1.0 · **Applicability:** always · **Verification:** auto-partial · **Effort:** small - **Measures:** Whether a person reading `robots.txt` finds a plain-language statement of the organization's position. - **Detect:** Extract `#` comment blocks. Require ≥120 characters of prose and at least two policy terms (`AI`, `training`, `crawl`, `license`, `permission`). - **Bands:** 4 — substantive block present and consistent with the machine signals. 2 — a comment exists but is thin or boilerplate. 0 — none. - **Fix:** Write four sentences above the signals. Journalists and counsel read this file. ### AR-1.6 — RSL licensing document published and referenced - **Weight:** 1.0 · **Applicability:** conditional — the client holds licensable archives (journals, museums, associations, research bodies) or has opted into a licensing posture · **Verification:** auto · **Effort:** medium - **Measures:** Whether machine-readable licensing terms exist under RSL 1.0 and are discoverable. - **Detect:** Fetch `/license.xml` and any path referenced from `robots.txt`. Validate against the RSL 1.0 schema. Check for an HTTP `Link:` header on content responses. - **Bands:** 4 — valid document, referenced from both `robots.txt` and headers. 3 — valid, referenced from one. 2 — present but invalid. 1 — referenced but missing. 0 — absent. - **Fix:** Decide terms with the client first. The XML is the easy half. ### AR-1.7 — RSL terms propagated to feeds and schema - **Weight:** 1.0 · **Applicability:** conditional — requires AR-1.6 to pass · **Verification:** auto · **Effort:** small - **Measures:** Whether licensing travels with the content through every channel, not just the well-known file. - **Detect:** Check RSS and Atom feeds for licensing elements. Check sampled JSON-LD for a `license` property on licensable types. - **Fix:** Add the license reference to feed generation and to the schema field mapping. ### AR-1.8 — Web Bot Auth verification configured - **Weight:** 1.0 · **Applicability:** conditional — the client wants selective agent access rather than open or closed · **Verification:** auto-partial · **Effort:** large - **Measures:** Whether the edge distinguishes cryptographically verified agents from unsigned scrapers. - **Detect:** Probe `/.well-known/http-message-signatures-directory` if the site operates its own agents. Otherwise send a signed and an unsigned request and compare treatment. - **Bands:** 4 — verified agents pass, unsigned agents claiming an AI UA are challenged. 2 — signature headers accepted but not acted on. 0 — no differentiation. - **Fix:** Configure edge bot rules on `Signature-Agent` rather than user-agent strings. ### AR-1.9 — Correct X-Robots-Tag on non-HTML resources - **Weight:** 1.0 · **Applicability:** conditional — the site serves PDFs, documents, media, or JSON endpoints · **Verification:** auto · **Effort:** small - **Measures:** Whether indexing directives reach resources that cannot carry a meta robots tag. - **Detect:** HEAD every sampled PDF, document, media file, and API endpoint. Record `X-Robots-Tag`. Flag `noindex` on substantive content and absent directives on endpoints that should be excluded. - **Bands:** 4 — every non-HTML type carries a deliberate, correct value. 2 — mixed or partial. 0 — absent everywhere, or substantive PDFs marked `noindex`. - **Fix:** Set headers by path pattern at the CDN or web server. ### AR-1.10 — No stray noarchive or nosnippet directives - **Weight:** 1.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Whether the site suppresses the excerpting that answer engines depend on. - **Detect:** Parse meta robots tags and `X-Robots-Tag` headers across the sample for `noarchive`, `nosnippet`, and `max-snippet:0`. - **Bands:** 4 — none present. 2 — present on a minority of pages. 0 — present site-wide. - **Fix:** Remove them. They are almost always left over from a legacy SEO module. --- ## Dimension 2 — Crawl and index hygiene **Weight 12.0** AI crawlers visit less often than Googlebot and tolerate less noise. Budget wasted on faceted search never reaches the pages that answer questions. ### AR-2.1 — Missing pages return 404 or 410 - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Soft 404s — "not found" content served with a 200 status. - **Detect:** Request 10 deliberately invalid URLs under real path prefixes. Flag 200 responses. Separately, flag sampled pages whose body matches a not-found template or falls below a minimum content threshold while returning 200. - **Evidence:** Probe URLs with status codes and body fingerprints. - **Fix:** Return real status codes. Use 410 for content that is permanently gone. ### AR-2.2 — Redirects resolve in a single hop - **Weight:** 1.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Redirect chain depth and loops. - **Detect:** Follow redirects without auto-resolution and count hops per sampled URL and per internal link target. - **Bands:** 4 — no chain exceeds one hop. 3 — a few two-hop chains. 2 — chains of three or more. 1 — widespread chains, meaning both more than 10% of the sample and at least three URLs. 0 — loops present. - **Fix:** Flatten the redirect table so every source points at its final destination. ### AR-2.3 — Every page emits a valid, self-consistent canonical - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Presence and correctness of `rel=canonical`. - **Detect:** For each sampled page, extract the canonical and resolve it. Flag: missing, multiple, cross-host, pointing at a redirect, pointing at a non-200, or pointing at the homepage from a deep page. - **Fix:** Emit a self-referencing canonical by default and correct the exceptions. ### AR-2.4 — Faceted, calendar, and parameterized URLs are controlled - **Weight:** 2.0 · **Applicability:** conditional — the site has search, filtering, or a calendar · **Verification:** auto-partial · **Effort:** medium - **Measures:** Whether the crawlable URL space is finite and predictable. - **Detect:** Crawl listing pages two levels deep with parameters followed. Measure URL growth rate and count distinct parameter combinations. Check whether those URLs are disallowed, `noindex`, or canonicalized to the unfiltered listing. - **Bands:** 4 — parameter space bounded by robots rules, `noindex`, or canonicals. 3 — mostly controlled with gaps. 2 — partially controlled. 1 — minimal control. 0 — unbounded expansion observed. - **Fix:** Combine disallow patterns, `noindex` on parameterized views, and canonicals back to the base listing. ### AR-2.5 — Sitemap lastmod reflects real content changes - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto-partial · **Effort:** medium - **Measures:** Whether `lastmod` is a freshness signal or deploy noise. - **Detect:** Parse all `lastmod` values. Flag clustering — more than 60% of URLs sharing a single date, or timestamps identical to the minute across content types. Cross-check against on-page `dateModified` where present. Re-run across two audits to confirm. - **Bands:** 4 — values are distributed and match on-page dates. 3 — mostly accurate with some clustering. 2 — heavy clustering. 1 — all identical. 0 — absent. - **First run:** cap at 3 without a baseline run. See *Checks that need a previous run*. - **Fix:** Wire `lastmod` to the node's changed timestamp, not to cron. ### AR-2.6 — Sitemap index split by type, with media sitemaps - **Weight:** 1.0 · **Applicability:** always · **Verification:** auto · **Effort:** small - **Measures:** Whether sitemap structure makes crawl coverage diagnosable. - **Detect:** Fetch `/sitemap.xml`. Confirm it is an index with child sitemaps. Check for image and video sitemaps where the site has substantial media. Validate against the sitemap schema and the 50k URL / 50MB limits. - **Fix:** Generate per-content-type children and submit the index to both search consoles. ### AR-2.7 — Paginated series are coherently signaled - **Weight:** 1.0 · **Applicability:** conditional — the site has paginated listings or multi-page articles · **Verification:** auto · **Effort:** small - **Measures:** Whether page 2 and beyond are reachable and correctly related to page 1. - **Detect:** Identify paginated series by URL pattern. Check for `rel=next`/`rel=prev`, a view-all page carrying the canonical, and whether deep pages are reachable by crawl. - **Fix:** Prefer a view-all canonical where page weight allows. ### AR-2.8 — IndexNow fires on publish and update - **Weight:** 1.0 · **Applicability:** always · **Verification:** auto-partial · **Effort:** small - **Measures:** Whether the site pushes change notifications instead of waiting to be crawled. - **Detect:** Look for an IndexNow key file at the site root (a 32–128 character hex filename returning its own key). Confirm the key resolves. Whether submissions actually fire on publish requires attestation. - **Bands:** 4 — key file present, valid, and submission confirmed. 3 — key file present and valid, submission unconfirmed. 2 — a key file is referenced but does not resolve or does not contain its own key. 0 — absent or not discoverable. - **Limitation:** the key file's name *is* the key, and the key is not published. A scanner can only find it where `robots.txt` or a conventional path advertises it, so a zero here means *not discoverable* rather than *absent*. Score it honestly and say which. - **Fix:** Hook submission into the publish workflow rather than a schedule. --- ## Dimension 3 — Rendering and extraction **Weight 20.0** Most AI crawlers do not execute JavaScript. This dimension asks whether the facts exist in the server response, in a structure a parser can follow. ### AR-3.1 — Primary content is present without JavaScript - **Weight:** 6.0 · **Applicability:** always · **Verification:** auto · **Effort:** large - **Measures:** The gap between the raw HTTP response and the hydrated DOM. This is the single heaviest check in the Index. - **Detect:** For every sampled URL, extract main-content text from the raw fetch and from the rendered fetch. Compute `coverage = raw_chars ÷ rendered_chars` after normalizing whitespace and stripping nav, header, and footer regions. Average across the sample and report the per-template distribution. - **Evidence:** Per-URL character counts, coverage ratio, and the first 500 characters of each version for spot-checking. - **Bands:** 4 — mean coverage ≥0.95. 3 — 0.80–0.94. 2 — 0.50–0.79. 1 — 0.20–0.49. 0 — below 0.20. - **Edge cases:** Coverage above 1.0 happens when JavaScript *removes* content that the raw response contained. Clamp the per-URL value to 1.0 for banding and keep the unclamped ratio in the evidence. A URL whose rendered text is empty is excluded from the mean and the exclusion is recorded — it is a fetch problem, not a coverage measurement. - **Note:** Report the worst-performing template by name. "Your program pages ship 8% of their content" lands harder than a mean. - **Fix:** Server-render or statically render. For decoupled Drupal this is the largest line item in most remediations. ### AR-3.2 — Progressive disclosure content ships in the initial HTML - **Weight:** 3.0 · **Applicability:** conditional — the site uses tabs, accordions, load-more, or infinite scroll · **Verification:** auto · **Effort:** medium - **Measures:** Whether tabbed, collapsed, or lazily paged content exists before interaction. - **Detect:** Identify disclosure widgets by ARIA roles (`tab`, `tabpanel`, `region` with `aria-expanded`), `
`, and common class patterns. For each, check whether the panel body is populated in the raw HTML. Detect infinite scroll by watching for XHR content growth on scroll in the rendered fetch, and check for a paginated fallback. - **Bands:** 4 — every panel populated in raw HTML; infinite scroll has crawlable pagination. 3 — one pattern incomplete. 2 — panels populated on some templates only. 1 — most content loads on interaction. 0 — all disclosure content is fetched on demand. - **Fix:** Render all panels and hide with CSS. Give infinite scroll a paginated fallback. ### AR-3.3 — Semantic HTML structure - **Weight:** 2.0 · **Applicability:** always · **Verification:** auto · **Effort:** medium - **Measures:** Whether document structure carries meaning or only styling. - **Detect:** Per page, require exactly one `
`, at least one `
` or `
` wrapping the primary content, and a `