crawler-access-audit SKILL.md skillA
crawler-access-audit from getedgehq/skills is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.
mdr add getedgehq/skills/crawler-access-audit@git:20261005.43cc1a8Pins git:20261005.43cc1a8 in mdr.lock. Pin the fingerprint instead.
- Audit grade
- A, 17 of 17 checks passed
- Kind
- SKILL.md skill
- Repository
- getedgehq/skills
- Path
crawler-access-audit/SKILL.md- Versions recorded
- 1
- Latest version
- git:20261005.43cc1a8, committed Oct 5, 2026
- Size
- 6,019 bytes
- Repository stars
- 5
- License
- Apache-2.0
Audit web crawler access from server logs and robots.txt. Covers robots.txt matching and precedence (RFC 9309), user-agent groups, wildcards and end anchors, crawl-delay, meta robots and X-Robots-Tag, verifying crawler identity, real client IP behind proxies, and AI crawler user-agents. Use for log analysis of bots, compliance checks and crawl-efficiency reports.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Crawler access audit Goal: decide, for each crawler request in a log, who really made it, which rules applied at that time, and whether it was allowed. ## 1. robots.txt semantics (RFC 9309) - File is a list of groups. A group starts with one or more `User-agent` lines followed by rules (`Allow`, `Disallow`, `Crawl-delay`, `Sitemap`). Blank lines and comments do not end a group; a new `User-agent` line after rules starts a new group. - A crawler uses exactly ONE group: the one whose `User-agent` token best matches its product token (case-insensitive, longest match wins). If none matches, use the `*` group. Groups are NOT merged across different tokens, and `*` rules do not apply to a crawler that has its own group. If multiple groups name the same agent, merge those groups. - Match the product token only (e.g. the name before `/version`), not the whole user-agent string. …
Read the whole file at its exact version.
How to install
mdr add getedgehq/skills/crawler-access-audit@sha256:2e47660867df8398The line at the top of this page pins the label git:20261005.43cc1a8, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
[](https://markdownregistry.com/a/art_fo74nqlos2dng3n6)
3 badge image fetches in 30 days. What the badge shows.
Versions
The registry has recorded 1 version of this file, committed on Oct 5, 2026.
Audit of the latest version
- pass: Frontmatter block present
- pass: Frontmatter declares a name
- pass: Frontmatter declares a description
- pass: Size between 200 bytes and 200 KB (6019 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
getedgehq/skills · 5 stars · license Apache-2.0 · pushed 2026-11-06 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_fo74nqlos2dng3n6 GET https://markdownregistry.com/api/v1/resolve?ref=getedgehq/skills/crawler-access-audit GET https://markdownregistry.com/api/v1/blob/2e47660867df8398e61f680e747733a161fb7a7bcd9f37987b0cd9e9a90a0c75
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.