git:20260526.29eee45 to git:20260531.ff57928

2 added, 0 removed. Audit A to A.

# AGENTS.md

Instructions for AI coding agents working on servo-fetch. This file complements, not replaces, [`CONTRIBUTING.md`](CONTRIBUTING.md) (human-oriented) and [`skills/servo-fetch/SKILL.md`](skills/servo-fetch/SKILL.md) (consumer-oriented). Treat this file as living documentation; update it when conventions change.

## Project overview

servo-fetch is a Rust workspace with two crates:

- **`servo-fetch`** (library): wraps the [Servo](https://servo.org) browser engine to fetch, render, and extract web content. Main entry points: `fetch()`, `crawl()` / `crawl_each()` (streaming), `map()` (sitemap discovery).

- **`servo-fetch-cli`** (binary): CLI + HTTP API + MCP server. Depends on `servo-fetch`.

Toolchain pinned in `rust-toolchain.toml`; MSRV declared as `rust-version` in `Cargo.toml` — do not use features stabilized after the MSRV. Workspace-wide version bumping via `[workspace.package].version`.

The Servo dependency dominates build time (several minutes on first build, ~30-40 minutes for release). Plan long-running commands accordingly.

## Setup and commands

Run these from the workspace root unless noted. Copy-paste ready:

```sh

cargo build # Debug build

cargo build --release # Release build (slow)

cargo +nightly fmt --all # Format

cargo +nightly fmt --all -- --check # CI fmt check

cargo clippy --workspace --all-targets -- -D warnings # Lint (pedantic, warnings are errors)

cargo test --workspace # Unit + integration tests (fast)

cargo test -- --ignored # Include Servo-backed e2e tests (slow)

cargo deny check # License + advisory audit

taplo check # TOML lint (install: cargo install taplo-cli --locked)

taplo fmt # TOML format

typos # Spell check

```

Nextest profiles (preferred over `cargo test` in CI):

```sh

cargo nextest run --workspace --profile ci # Unit + integration, CI output

cargo nextest run -p servo-fetch-cli --run-ignored only --profile ci-e2e # E2E, retries on flake

```

Cargo aliases for exact CI parity (defined in `.cargo/config.toml`):

```sh

cargo lint # cargo-clippy job

cargo test-ci # cargo-test job (unit/integration via nextest)

cargo test-doc # cargo-test job (doc tests)

cargo test-e2e # e2e job (Servo-backed, retries on flake)

```

Ignored tests require a working Servo environment. Servo uses `SoftwareRenderingContext` internally, so no X server is needed in CI (the apt install list in `release.yml` includes X11 libs for linking, not for runtime). If an `ignored` test fails locally, first retry; if it still fails, check whether the test hits the network or a site that requires `--settle`.

## Code conventions

### Rust style

- **`cargo clippy` is a gate.** Use `-D warnings` in CI; keep it passing locally before pushing.

- **Inline `format!` args** — `format!("x={x}")`, not `format!("x={}", x)`. See the clippy rule `uninlined_format_args`.

- **`#[non_exhaustive]` on public structs/enums** that we expect to grow (e.g., `Page`, `CrawlResult`, `ConsoleMessage`). Do not add new public enum variants or struct fields without it unless the type is intentionally sealed.

- **Fields `pub(crate)` + builder methods** for option structs like `CrawlOptions`, `FetchOptions`, `MapOptions`. Callers construct via `Options::new(url)` and chain `.method(value)`. Adding a new `pub(crate)` field + builder method is always non-breaking.

- **`Result<T, Error>` end-to-end** — crate-local `error::Result`. No `unwrap()` in non-test code except where a panic is documented as the only sane behaviour (e.g., poisoned mutex).

- **Single-concern modules over strict line counts.** Large files are acceptable when they remain one semantic concern (e.g., `crawl.rs` owns all crawl logic). Split when a second concern emerges, not when LoC crosses a threshold.

- **No `#[async_trait]`** — prefer native RPITIT (`fn foo(&self, ...) -> impl Future<Output = T> + Send`) when the return type is nameable.

### Commits

Conventional Commits, **subject-only** by default. Use the body only when critical context cannot fit in the subject. Scopes used in this repo:

- `chore(deps)` — external dependencies the **product ships** (cargo crates, docker base image).

- `chore(ci)` — CI/workflow configuration, github-actions entries, Dependabot config itself.

- `fix(<area>)` — bug fixes; `<area>` is typically a module (`crawl`, `map`, `build`, `tests`) or `ci`.

- `feat(<area>)` — user-facing additions.

- `refactor(<area>)`, `test(<area>)`, `docs`, `perf`, `build`.

Examples: `fix(build): drop +crt-static on Windows to match mozjs-sys CRT`, `chore(deps): bump rustls`.

+ **Breaking changes:** mark with `!` in the subject (`feat(api)!: split sync and async`, `refactor(error)!: rename Error::Timeout`). The marker drives release-plz's minor/major bump and the `### Breaking Changes` changelog section. Put migration guidance for users in the release PR's `CHANGELOG.md` under `### Migration Guide`, not in commit bodies (release-plz does not surface commit bodies).

### Branch names

Kebab-case under a type prefix: `fix/windows-crt-mismatch`, `refactor/wiremock-body-mime-consistency`, `ci/trivy-image-scan`.

### GitHub Actions

**All third-party actions must be SHA-pinned** with a version comment: `uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2`. This is enforced for supply-chain hygiene (see the March 2026 `trivy-action` compromise). Dependabot (`chore(ci)`) keeps SHAs current.

## Testing idioms

### wiremock mocks

**Always use `set_body_raw(body, mime)`**, never `set_body_string` + `insert_header("content-type", ...)`. `set_body_string` unconditionally overwrites the content-type to `text/plain`, regardless of chain order, and the resulting mock silently serves the wrong MIME — which breaks browser-level HTML parsing (Servo receives `text/plain` and wraps the body in `<pre>`).

```rust

// WRONG — insert_header is overridden by set_body_string's forced text/plain

ResponseTemplate::new(200)

.insert_header("content-type", "text/html")

.set_body_string(html)

// RIGHT

ResponseTemplate::new(200).set_body_raw(html.into_bytes(), "text/html; charset=utf-8")

```

Shared helper: `crates/servo-fetch-cli/tests/common.rs::mock_page`.

### E2E tests

Mark Servo-backed tests with `#[ignore = "e2e: requires Servo engine"]` and run them via the `ci-e2e` nextest profile, which enables retries with a 2-second backoff. Keep the mock HTML minimal but syntactically valid (`<!doctype html><html>...</html>`).

### Assertions

Compare whole values with `assert_eq!` over field-by-field checks. Failing diffs are easier to read when the entire structure is dumped.

## When contributing

- Read [`CONTRIBUTING.md`](CONTRIBUTING.md) for the human-oriented workflow and the AI usage policy.

- Read [`SECURITY.md`](SECURITY.md) before changing anything SSRF-, TLS-, or signing-related.

- User-facing behavior changes must update:

- `README.md` (CLI + library usage)

- `crates/servo-fetch-cli/README.md` (CLI flags reference)

- `skills/servo-fetch/SKILL.md` and `skills/servo-fetch/references/guide.md` if behavior or flags change

- `SECURITY.md` if policy defaults change

- PR template: `.github/pull_request_template.md`. Keep the Summary concise; explain *why* where non-obvious, not *what* (the diff shows what).

- Do not submit autonomously-generated PRs. Human-in-the-loop is a hard requirement (see CONTRIBUTING.md § Use of AI).