git:20260418.3c3158f to git:20260716.c5d769e

5 added, 0 removed. Audit C to A.

+ ---

+ name: agent-ready-robots-txt

+ description: >

+ Sub-skill de agent-ready-cloudflare: Implement robots.txt

+ ---

# Implement robots.txt

Publish a valid robots.txt at your site root per

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## Requirements

- Serve `/robots.txt` as `text/plain` with HTTP 200

- Include `User-agent` directives with `Allow`/`Disallow` rules

- Reference your sitemap if one exists: `Sitemap: https://example.com/sitemap.xml`

## Cloudflare

[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)

can manage your robots.txt from the dashboard, including AI-specific bot rules.

## Validate

```

POST https://isitagentready.com/api/scan

Content-Type: application/json

{"url": "https://YOUR-SITE.com"}

```

Check that `checks.discoverability.robotsTxt.status` is `"pass"`.