sitemap-robots · git:20261105.f7f8170 · 2026-11-05 · sha256 572d2f85b5bef2e3
sitemap-robots git:20261105.f7f8170A
Immutable. This exact content never changes and is served at /api/v1/blob/572d2f85b5bef2e3.
No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_axddkd6cg3sxbfnu/572d2f85b5bef2e3a0ded5951b6e6d3380f84928e4d1c3cde92931b63a652c9a.json
Then /plugin install sitemap-robots@markdownregistry-axddkd6cg3sxbfnu in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.
--- name: sitemap-robots description: Audit robots.txt directives, AI crawler access rules, XML sitemap discovery, sitemap index structures, and crawl coverage using one-step-seo sitemap and audit commands. --- # sitemap-robots — Sitemap & Robots Discovery Policy ## 1. Description & Activation Criteria Activate `sitemap-robots` when the user wants to: - Inspect or troubleshoot `/robots.txt` and `/sitemap.xml` discovery (`T05-robots`, `T05-robots-disallow`, `T06-sitemap`, `G02-ai-blocked`). - Verify whether `Sitemap:` directives in `robots.txt` resolve properly or whether a `<sitemapindex>` is properly structured. - Seed a multi-page audit from the XML sitemap (`--crawl sitemap`) to uncover unlinked or orphaned pages. ## 2. Inputs & Prerequisites - **Target**: Public `http(s)` URL or local build directory containing `robots.txt` and `sitemap.xml`. - **Optional flags**: - `sitemap <url> --json`: Fast discovery inspection without running a full page crawl. - `audit <url> --crawl sitemap --pages <N>`: Crawl pages seeded directly from `/sitemap.xml` (or the first `Sitemap:` URL in `robots.txt`). - `--skills sitemap-robots`: Filter audit findings to robots, sitemap, and crawl-policy rules. ## 3. Step-by-Step Workflow 1. **Run discovery diagnostics**: - Execute `npx one-step-seo sitemap <url> --json`. - Inspect `robotsFound`, `robotsSitemaps`, `sitemapFound`, `sitemapUrl`, `sitemapUrls`, `sitemapTruncated`, and `llmsFound`. 2. **Audit `robots.txt` syntax and crawler permissions (`T05-robots`, `G02-ai-blocked`)**: - Verify `/robots.txt` returns `200 OK` with valid `User-agent:` / `Allow:` / `Disallow:` groups. - Check whether `G02-ai-blocked` fires, and inspect the exact list of blocked AI bot tokens (out of the 25 recognized AI crawlers such as `gptbot`, `claudebot`, `perplexitybot`, `google-extended`). 3. **Evaluate XML Sitemap health (`T06-sitemap`)**: - Confirm the sitemap contains valid `<loc>` entries (or plaintext URLs) and is not an HTML 404 page served with a `200` status. - Distinguish a standard `<urlset>` (`~N URLs`) from a `<sitemapindex>` (`~N child sitemaps`). 4. **Compare sitemap inventory against internal link discovery**: - Run `npx one-step-seo audit <url> --crawl sitemap --pages 20 --out ./seo-report`. - Inspect the terminal output and `index.md` / `index.html` for **Possibly orphaned pages** (URLs present in `sitemap.xml` that received zero inbound internal links from crawled pages). ## 4. Tools & CLI Commands ```bash # Quick JSON check of robots.txt, sitemap.xml, and llms.txt npx one-step-seo sitemap https://example.com --json # Audit 20 pages seeded from the XML sitemap npx one-step-seo audit https://example.com --crawl sitemap --pages 20 --out ./seo-report # Focused sitemap & robots rule check on a single URL npx one-step-seo page https://example.com --skills sitemap-robots --out ./seo-report ``` ## 5. Evidence Requirements - Quote `site.robots`, `site.sitemap`, `site.sitemapUrls`, and `site.aiCrawlersBlocked` from `report.json` or `sitemap --json`. - When `T06-sitemap` fails, specify whether the file was missing (HTTP 404) or returned 200 with zero valid `<loc>` entries. - When reporting orphan pages, cite the exact URLs listed under `Possibly orphaned pages` in `index.md`. ## 6. Expected Outputs 1. **Discovery Status Summary**: Status of `/robots.txt`, `Sitemap:` declarations, resolved `/sitemap.xml` URL, entry count, and `/llms.txt`. 2. **Crawler Access Policy**: Search engine vs AI crawler permissions (`Allow` vs `Disallow` by user-agent). 3. **Remediation Snippets**: Ready-to-use `robots.txt` or `sitemap.xml` structure fixes. ## 7. Limitations & External Data - **Same-origin enforcement**: Sitemap URLs and child `<sitemapindex>` files are filtered to the audited origin for security; cross-host sitemap seeds are ignored. - **Crawl cap**: `getSitemapUrls` follows up to 5 child sitemaps in a `<sitemapindex>` and caps crawled pages at `--pages` (max 200). - **External data (advisory)**: Live Google Search Console sitemap submission errors and index coverage ratios require GSC access. ## 8. Safety & Approval Rules - Never recommend exposing private admin, API, or user-account paths in `robots.txt` or `sitemap.xml`. - Remind users that `robots.txt` `Disallow` prevents crawling, **not** indexing of linked URLs — sensitive pages must use `noindex` or authentication. ## 9. Examples ```bash # Inspect sitemap and robots configuration in JSON mode npx one-step-seo sitemap https://example.com --json # Audit sitemap-seeded pages and check for orphans npx one-step-seo audit https://example.com --crawl sitemap --pages 15 --out ./seo-report ```