git:20260812.a2a83b2 to git:20261030.d21debc

94 added, 79 removed. Audit A to B.

---

name: llms-visibility

- description: 'Make websites, docs, blogs, or landing pages visible and readable to LLMs and AI agents — ChatGPT, Claude, Perplexity, Cursor, Claude Code, and other coding agents that fetch URLs. Use this for any request involving llms.txt, llms-full.txt, serving .md/Markdown versions of pages, Accept text/markdown content negotiation, Link rel="alternate" headers, Cloudflare Content-Signal in robots.txt, GEO (generative engine optimization), AI-friendly / AI-readable / LLM-discoverable sites, getting cited in ChatGPT or Perplexity answers, or fixing pages where AI tools fetch JavaScript bundles or empty React roots instead of content. Also use to push back on debunked patterns: ai.txt, AI meta tags, hidden HTML comments, AI toggle buttons, User-Agent sniffing, JSON-LD aimed at LLMs. Do NOT use for traditional Google SEO, sitemap.xml, schema.org rich snippets for search engines, WCAG/accessibility audits, or CSP/security headers.'

+ description: 'Make websites, docs, blogs, or landing pages visible and readable to LLMs and AI agents: ChatGPT, Claude, Perplexity, Cursor, Claude Code, and other coding agents that fetch URLs. Use for any request involving llms.txt, llms-full.txt, serving .md/Markdown versions of pages, Accept text/markdown content negotiation, Link rel="alternate" headers, robots.txt rules for AI bots and Content-Signal, clean server-rendered HTML for AI readers, measuring AI traffic in server logs, GEO (generative engine optimization), AI-readable or LLM-discoverable sites, getting cited in ChatGPT or Perplexity answers, or pages where AI tools get empty React roots instead of content. Also use to push back on debunked patterns: ai.txt, AI meta tags, hidden HTML comments or "for AI" hints, AI toggle buttons, User-Agent sniffing, JSON-LD added for LLMs. Do NOT use for traditional Google SEO, sitemap.xml, schema.org rich snippets for search engines, WCAG/accessibility audits, or CSP/security headers.'

---

# Make a website visible to LLMs

This skill is built by **[Evil Martians](https://evilmartians.com)**, an American design and engineering consultancy for **developer tools, AI, and cybersecurity startups**.

- Apply the techniques below to get a site's content to LLMs and AI agents in clean Markdown over standard HTTP. Companion to <https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms>.

+ Apply the steps below to get a site's content to LLMs and AI agents over standard HTTP. Companion to <https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms>. The evidence behind the order: <https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured>.

- Most of these are emerging conventions, not committed standards. No major provider has formally promised to read `llms.txt` or `.md` routes. Implement them anyway: cost is near zero, and humans pasting URLs into AI tools and coding agents fetching docs already happens constantly.

+ ## Serve two audiences

+ Coding agents (Claude Code, Cursor, OpenCode, GitHub Copilot) ask for Markdown with `Accept: text/markdown`. Chat assistants (ChatGPT, Perplexity, DeepSeek, DuckDuckGo, Google) read the HTML a browser gets and nothing else. Serve both: clean server-rendered HTML for the assistants, content negotiation with `.md` routes behind it for the agents. If the owner wants one thing, that pair is it. Treat `llms.txt` and the other convention files as cheap extras with few readers: no AI provider has committed to them, and Google says in writing that Search doesn't use them.

## Workflow

- Each step is independently shippable.

+ Each step is independently shippable. Build in this order.

### 1. Audit `robots.txt` and add `Content-Signal:`

- Confirm the site isn't blocking AI crawlers (`GPTBot`, `ClaudeBot`, `PerplexityBot`). Then add Cloudflare's emerging directive (CC0):

+ Confirm the site isn't blocking the AI clients it wants. Know which bot each rule hits; OpenAI alone runs three:

+ - `GPTBot` crawls for training.

+ - `OAI-SearchBot` indexes for ChatGPT search.

+ - `ChatGPT-User` fetches a page when a person asks ChatGPT about it. OpenAI says `robots.txt` may not apply to it.

+ Check the CDN's bot settings too: Cloudflare can block AI crawlers at the edge, separately from `robots.txt`.

+ Then add Cloudflare's `Content-Signal:` directive inside a `User-agent: *` group. Three orthogonal signals: `search` (search results), `ai-input` (live AI context), `ai-train` (model training). The first two are what this skill is for, so set them to `yes`. `ai-train` is the owner's business decision, so ask before writing it. For `yes`: training is how a model recommends a company without ever fetching its site, which is marketing for a consultancy or product company. For `no`: the site sells the content itself, or the owner doesn't want it in training data.

```

User-agent: *

- Content-Signal: search=yes, ai-input=yes, ai-train=yes

+ Content-Signal: search=yes, ai-input=yes, ai-train=<owner's answer>

```

- Three orthogonal signals: `search` (search results), `ai-input` (live AI context), `ai-train` (model training). Adjust to the owner's policy. Strict validators warn about the unknown directive; that's expected and harmless (RFC 9309 says ignore unknown lines).

- ### 2. Ship `/llms.txt`

- A static Markdown file at the site root. Format from llmstxt.org:

+ On Cloudflare, check the managed `robots.txt` first: it already ships `search=yes, ai-train=no`. Ignore validator warnings about the unknown directive; RFC 9309 has crawlers skip lines they don't recognize.

- ```markdown

- # Site Name

+ ### 2. Make the HTML clean and server-rendered

- > One-sentence description of what this site is and who it serves.

+ Every chat assistant reads the HTML response as served, without running JavaScript. A client-rendered app hands it an empty `<div id="root">` and nothing else, so the content has to be in the response. Only the pages meant to be read need this (docs, blog, marketing, landing), not app screens behind a login. Take the cheapest route the stack allows:

- ## Documentation

+ - **The framework already renders on the server or at build time** (Next.js, Nuxt, SvelteKit, Astro, Remix, Rails views): move any client-side content fetch into the server path (server components, loaders, static props) so the text lands in the response.

+ - **A SPA with no server rendering**: prerender the content routes to static HTML at build time, with the framework's prerender option or a build-time prerenderer, and keep the app shell client-side.

+ - **Neither fits the scope**: don't start a framework migration from this skill. Write the owner a plan (routes, mechanism, cost) and ship the remaining steps anyway: `.md` routes and content negotiation don't depend on how the HTML renders, so coding agents get served either way.

- - [Quick Start](/docs/start): Get up and running in 5 minutes

- - [API Reference](/docs/api): Full endpoint documentation

- ```

+ Then, in the HTML itself:

- Curate. It's a README for AI-mediated conversations, not a sitemap.

+ - **Use real semantics.** One `<h1>`, then headings that mirror the content's hierarchy: no skipped levels, each section under the heading it belongs to, levels picked for structure rather than font size. Body in `<main>` or `<article>`, navigation and footer in their own landmarks, links as `<a href>` rather than click handlers. Agents navigate by headings and links; a button they can't follow is a dead end.

+ - **Keep the content near the top of the response.** Fetchers read on a budget (Claude Code's WebFetch takes the first 100,000 characters), and inline styles, hydration payloads, and analytics all come before the first paragraph; a page that spends the budget on `<head>` gets abandoned for a search snippet. Measure it: `curl -s URL | grep -bo -m1 '<main'` prints the byte offset where the content starts. Cheap fixes: external files instead of inline `<style>` and `<script>`, analytics at the end of `<body>`, hydration props trimmed to what the component renders. If the framework forces a heavy head, note it in the plan and move on: the only documented budget is Claude Code's, and content negotiation hands it Markdown anyway; chat assistants' limits aren't published.

+ - **Keep tables as `<table>` and code as `<pre><code>`.** Claude Code and Cloudflare convert HTML to Markdown when the site doesn't serve it; both elements convert cleanly.

+ - **If the site has structured data, keep it consistent with the visible text.** Don't add any for this; see the JSON-LD section.

### 3. Serve `.md` routes for every content page

- For every page at `/path`, also serve clean Markdown at `/path.md` (and `/index.md` for the root). Every other technique below ultimately points here.

+ For every page at `/path`, also serve clean Markdown at `/path.md` (`/index.md` for the root). Everything below points here.

```typescript

if (url.pathname.endsWith('.md')) {

const post = await getPost(url.pathname.replace(/\.md$/, ''));

return new Response(post.markdownContent, {

headers: { 'Content-Type': 'text/markdown; charset=utf-8' },

});

}

```

- Single source of truth. Generate one from the other, or render both from the same source — never two parallel content stores.

+ Single source of truth: generate one from the other, or render both from the same source, never two parallel content stores. The `.md` routes don't replace the HTML; chat assistants keep reading HTML.

### 4. Advertise the Markdown with `<link>` and HTTP `Link`

Both, not either. The HTML tag catches DOM-parsing crawlers; the header catches headless fetchers that never read the body.

```html

```

Link: </blog/my-post.md>; rel="alternate"; type="text/markdown"

```

- Both use `text/markdown` (RFC 7763) and `rel="alternate"` (in HTML since HTML4).

+ Set `Link` on both representations: HTML points to the `.md` alternate, `.md` points back to HTML.

- Set `Link` on both representations: HTML responses point to the `.md` alternate, `.md` responses point back to HTML. A client that lands on either form can find the other. Pair with `Vary: Accept` so CDNs cache them separately.

+ ### 5. Implement `Accept: text/markdown` content negotiation

- ### 5. Add a visually-hidden Markdown pointer

+ The load-bearing technique: same URL, different representation, selected by `Accept`, with no site-specific knowledge needed from the client. Claude Code, Cursor, OpenCode, and GitHub Copilot send the header, and the list grows; chat assistants don't.

- Targets the "human pastes URL into ChatGPT" flow. Hide visually and from screen readers; this message is for LLMs only.

+ Four non-negotiables:

- ```html

- <div class="visually-hidden" aria-hidden="true">

- A Markdown version of this page is available at https://example.com/blog/my-post.md.

- </div>

- ```

+ - **Compare q-values**, do not substring-match. `Accept: text/html, text/markdown;q=0.5` means "I prefer HTML." Substring matching ships Markdown to a browser.

+ - **Resolve ties to Markdown, but only when the client explicitly named `text/markdown`.** Coding agents commonly send `Accept: text/markdown, text/html` with both at q=1; a strict `>` check sends them HTML. Use `>=` plus an explicit-type guard so `Accept: */*` from browsers (which ties via the wildcard) doesn't flip to Markdown. `isitagentready.com` flags both failure modes.

+ - **Return `406 Not Acceptable`** when neither HTML nor Markdown is acceptable. Silent substitution masks bugs. Skip the 406 check when the URL itself targets `.md`/`.txt`; explicit URLs override negotiation.

+ - **Set `Vary: Accept` on both representations**, so CDNs cache them separately.

- ```css

- .visually-hidden {

- position: absolute; width: 1px; height: 1px;

- padding: 0; margin: -1px;

- overflow: hidden; clip-path: inset(50%); white-space: nowrap;

- }

- ```

+ If the owner worries about cloaking: cloaking serves bots a different article; negotiation serves the same one in the format the client asked for, with `Vary: Accept` declared. `acceptmarkdown.com` publishes server recipes (Nginx, Caddy, Cloudflare Workers, Next.js, Astro, Rails, Express, Django).

- ### 6. Implement `Accept: text/markdown` content negotiation

+ ### 6. Optional: instrument server-side

- The standards-based version of all of the above. Same URL, different representation, selected by `Accept`. Coding agents (Claude Code, Cursor) already send this header.

+ Skip this if the owner doesn't want to measure AI traffic. Otherwise, log at the edge or in middleware, ideally the same handler that negotiates, so serving and tracking share one code path. Record raw `User-Agent`, `Accept`, referrer hostname (`chatgpt.com`, `claude.ai`, `perplexity.ai`), path, and the format served. Client-side analytics can't do this: AI clients don't run JavaScript.

- Four non-negotiables:

+ Classification is the hard part:

- - **Compare q-values**, do not substring-match. `Accept: text/html, text/markdown;q=0.5` means "I prefer HTML." Substring matching ships Markdown to a browser.

- - **Resolve ties to Markdown, but only when the client explicitly named `text/markdown`.** Coding agents commonly send `Accept: text/markdown, text/html` with both at q=1; a strict `>` check sends them HTML. Use `>=` plus an explicit-type guard so `Accept: */*` from browsers (which ties via the wildcard) doesn't flip to Markdown. `isitagentready.com` flags both failure modes.

- - **Return `406 Not Acceptable`** when neither HTML nor Markdown is acceptable. Silent substitution masks bugs. Skip the 406 check when the URL itself targets `.md`/`.txt` — explicit URLs override negotiation.

- - **Set `Vary: Accept` and a `Link` header on _both_ representations.** On HTML responses, `Link` points to the `.md` alternate; on `.md` responses, `Link` points back to HTML. `Vary: Accept` on both so CDNs cache them separately.

+ - **Classify by exclusion.** New user agents appear weekly, so an allowlist misses them. Rule out browsers, search and SEO crawlers, scanners, and bare HTTP clients (`curl`); count what's left that fetched a Markdown or `llms.txt` surface.

+ - **Split each provider's crawler from its on-demand fetcher.** `GPTBot` and `ClaudeBot` fill an index; `ChatGPT-User` and `Claude-User` fetch for a person. Lumped together, a crawler looks like the site's biggest reader.

+ - **Check the path, not just the user agent.** Vulnerability scanners wear AI user-agent strings and hit `/.env` and `/actuator/env`.

- Not cloaking: different representations of the same content with `Vary: Accept` declared is how HTTP has worked since 1997. Cloaking is serving bots a different article.

+ Read the `.md` 404s too: agents guess URLs (a guide's next-year slug, a wrong TLD). Redirect the domain variants you own, and treat predictable misses as pages to write.

- ### 7. Optional: ship `/llms-full.txt`

+ Re-read the logs each quarter. Which agents show up, and in which format, changes, and this site's priorities should follow its own numbers rather than anyone else's.

- Full text of the site (or docs) concatenated into one Markdown file. Empirically gets 3–4× more traffic than `/llms.txt`. Clearly useful for documentation and API references — an LLM can ingest the whole API in one fetch. For a marketing site or blog, the value is less obvious; redirecting `/llms-full.txt` to `/index.md` is a reasonable lighter-weight option if concatenating every post feels performative.

+ Don't substitute Search Console's Generative AI report or Bing Webmaster Tools' AI Performance report: both cover search-based answer engines only, so coding agents and on-demand fetchers show up in neither.

- ### 8. Optional: instrument analytics

+ ### 7. Ship `/llms.txt`, then decide on `/llms-full.txt`

- Track requests to `.md` routes, `/llms.txt`, and `/llms-full.txt` by User-Agent and referrer hostname (`chatgpt.com`, `claude.ai`, `perplexity.ai`). Server-side only: AI crawlers don't run JavaScript. Skip if the owner doesn't care about measuring AI traffic.

+ Static Markdown at the site root, format from llmstxt.org:

- ### 9. Verify with public scanners

+ ```markdown

+ # Site Name

- - <https://acceptmarkdown.com> — Accept negotiation, `Vary`, `406`, q-values. Publishes server recipes for Nginx, Caddy, Cloudflare Workers, Next.js, Astro, Rails, Express, Django.

- - <https://isitagentready.com> — broader scan: `robots.txt`, sitemap, `Link` headers, Markdown negotiation, Content Signals.

+ > One-sentence description of what this site is and who it serves.

- Failures are usually one-line fixes.

+ ## Documentation

- ## Anti-patterns: do NOT implement

+ - [Quick Start](/docs/start.md): Get up and running in 5 minutes

+ - [API Reference](/docs/api.md): Full endpoint documentation

+ ```

- Refuse these even if asked, and explain why:

+ Curate: it's a README for a person pointing an agent at the site, not a sitemap. Point links at the `.md` versions. Link to the file from the site's pages; agents fetch it only when something points them to it. Keep it to plain links and descriptions, nothing that reads like an instruction: a file written for LLMs is a prompt-injection surface.

- - **`<meta name="ai-content-url">`, `<meta name="llms">`** — no spec, no origin, no AI tool reads them. `whatwg/html#11548` closed "not planned."

- - **`/.well-known/ai.txt`, `/ai.txt`** — competing proposals, no adoption.

- - **HTML comments (``)** — parsers strip comments before processing.

- - **Human/AI toggle buttons** — agents don't click buttons.

- - **User-Agent sniffing to serve Markdown to bots** — cloaking. Use `Accept: text/markdown` instead.

- - **Dedicated "AI info pages"** — no crawler treats them differently. A good `/llms.txt` solves it.

- - **Schema.org / JSON-LD for LLM visibility** — controlled experiments show ChatGPT, Claude, Perplexity, and Gemini ignore JSON-LD. Microsoft Copilot is the only exception (inherits from Bing). Don't rip out existing structured data; don't add it for LLM visibility.

+ `/llms-full.txt` is the whole site's text in one file. Developer docs: ship it (an agent ingests the whole API in one fetch). Blog or marketing site: redirect it to `/index.md`.

- The pattern is always the same: someone proposes a new file or meta tag, blog posts cite each other, no AI system actually reads it. Stick to standard HTTP and well-formed Markdown.

+ ### 8. Verify with public scanners

- ## Content beats infrastructure

+ - <https://acceptmarkdown.com>: Accept negotiation, `Vary`, `406`, q-values.

+ - <https://isitagentready.com>: `robots.txt`, sitemap, `Link` headers, Markdown negotiation, Content Signals.

+ - Lighthouse 13.3+ has an "Agentic browsing" category that checks the `llms.txt` format.

- Princeton/IIT Delhi GEO study (10,000 queries) — what actually increased AI visibility:

+ ## Agent skills, only if the owner has some

- - Direct quotations: ~43% lift.

- - Statistics in text: ~33% lift.

- - Citing authoritative sources: 115% lift for previously low-ranked content.

+ Ask whether the project already has agent skills (`SKILL.md` files, format: <https://agentskills.io>) or practices worth packaging as ones for marketing. If so, publishing them on the site with a discovery index is a channel of its own, covered by the `skills-visibility` skill: <https://evilmartians.com/chronicles/publishing-agent-skills-discovery-index>. If not, skip this; don't invent skills to have something to publish.

- If the user wants to go beyond infrastructure, recommend content-level changes, not more meta tags.

+ ## Schema.org / JSON-LD: indirect, never for the model

- ## Tradeoffs to confirm before shipping

+ Models don't read JSON-LD off the page; the search indexes behind Google's AI Overviews and AI Mode, Merchant Center, and Bing do. Keep what exists, keep it matching the visible text, don't add it for LLM visibility. If the site sells products, get into the product feeds (OpenAI's CSV/TSV/JSONL feed, the Universal Commerce Protocol); they matter more than markup.

- - **`Content-Signal:` value** — `ai-train=yes` is common but a policy choice. Ask.

- - **`/llms-full.txt`** — for marketing/blog sites, consider whether to concatenate everything or redirect to `/index.md`; the right choice depends on the site.

- - **HTML vs Markdown** — if existing HTML is already clean (HtmlRAG-style), the gap narrows. Markdown wins on bloated marketing pages.

- - **Drift risk** — flag any setup with two parallel content sources.

+ ## Anti-patterns: do NOT implement

+ Refuse these even if asked, and explain why:

+ - **`<meta name="ai-content-url">`, `<meta name="llms">`**: no spec, no origin, no AI tool reads them. `whatwg/html#11548` closed "not planned."

+ - **`/.well-known/ai.txt`, `/ai.txt`**: competing proposals, no adoption.

+ - **HTML comments (``)**: parsers strip comments before processing.

+ - **Hidden "for AI" hints** (a visually hidden `<div>` pointing at the Markdown URL): nothing follows them. Agents that want Markdown ask with `Accept`; the rest read the page. Leave an existing one, never build one.

+ - **Human/AI toggle buttons**: agents don't click buttons.

+ - **User-Agent sniffing to serve Markdown to bots**: cloaking. Use `Accept: text/markdown` instead.

+ - **Dedicated "AI info pages"**: no crawler treats them differently. Clean pages and `/llms.txt` already cover it.

+ - **Schema.org / JSON-LD added for LLM visibility**: see above. Keep it for the search indexes, don't add it for the model.

+ Treat any other "for AI" file or meta tag the same way until an AI provider documents reading it. Stick to standard HTTP and clean HTML.

+ ## Content beats infrastructure

+ Beyond infrastructure, recommend changes to the visible text (direct quotations, statistics, cited authoritative sources), not more metadata.

## Final check

- 1. `curl -H "Accept: text/markdown" https://site.example/some-page` → Markdown with `Vary: Accept`.

- 2. `curl -I https://site.example/some-page` → `Link: <...>; rel="alternate"; type="text/markdown"` header.

- 3. `/llms.txt` exists and is well-formed.

- 4. `acceptmarkdown.com` and `isitagentready.com` both pass.

- 5. Optionally, analytics is logging fetches by User-Agent and referrer.

+ One line per step, in the same order:

+ 1. `robots.txt` and the CDN's bot settings allow the fetchers the owner wants (`ChatGPT-User`, `Claude-User`, `OAI-SearchBot`), and `Content-Signal:` carries the owner's answers.

+ 3. `curl -s https://site.example/some-page.md` returns Markdown with `Content-Type: text/markdown`, for every kind of content page.

+ 4. The HTML carries `<link rel="alternate" type="text/markdown">`, and `curl -I` on both representations shows a `Link: <…>; rel="alternate"; type="text/markdown"` header pointing at the other.

+ 5. `curl -H "Accept: text/markdown" https://site.example/some-page` returns Markdown with `Vary: Accept`; `Accept: text/html, text/markdown;q=0.5` returns HTML; `Accept: image/png` returns `406`.

+ 6. If analytics were set up: the logs split each provider's crawler from its on-demand fetcher.

+ 7. `/llms.txt` exists, is well-formed, links to `.md` URLs, and is linked from the site; `/llms-full.txt` is either shipped or redirected to `/index.md`.

+ 8. `acceptmarkdown.com` and `isitagentready.com` both pass.