verify · diff
git:20260609.7949293 to git:20260902.d668714
17 added, 0 removed. Audit A to A.
---
name: verify
description: "Fact-check LLM-generated content against source-of-truth — confirm imports import, CLI flags are real, links resolve, counts match. Triggers on: verify docs, fact-check, did the model hallucinate, check generated content, are these claims real."
---
# Verify
**IMPORTANT: Start your response by telling the user:**
> **Verify** — Fact-checking the named entities in this content
> against the project as source-of-truth.
## What It Does
`/verify` is the output-side fact-checker for LLM-generated content
(docs, READMEs, tutorials). It catches the hallucination classes that
unit tests never see — an invented CLI flag, an import of a package
that doesn't exist, a dead cross-reference, a wrong count — by
confirming the *named entities* in the content actually exist.
It runs in two layers:
1. **Deterministic checkers** (`attune_verify`, authoritative): the
*top-level package* of each import resolves via `find_spec`, CLI
flags appear in `--help`, markdown links resolve under the project
root, numeric claims match a declared count source. These are ground
truth — no LLM, no guessing.
2. **Ambient semantic cross-check** (you, the agent): after the
deterministic pass, read the content against its source and flag
claims the checkers can't see (a security caveat dropped, a route
path that's plausible-but-wrong, a *private submodule* of an
installed package that doesn't actually exist — the import checker
only validates the top-level package, so `from pkg.fake_sub import X`
passes deterministically when `pkg` is installed). This layer is a
*cross-check only* — it over-flags on truncated context, so never let
it override a deterministic pass. If the deterministic layer says a
top-level import resolves, it resolves.
## How To Run It
+ ### Shared command workspace (preferred)
+
+ Open adapter `verify` with the validated generated-content path and optional
+ `hard_gate=true` for pipeline callers. The invocation authorizes this
+ read-only run, so there is no confirmation action. Execute
+ `attune_verify.verify` exactly as below and publish its authoritative outcome
+ as `deterministic_result`, retaining every finding's kind, severity, detail,
+ evidence, and location.
+
+ Perform the ambient semantic pass only after the deterministic result and
+ publish it as `cross_check_result`. Ambient findings remain explicitly labeled
+ warnings and cannot override deterministic entity-existence results. In hard
+ gate mode, any deterministic error keeps `hard_gate_passed=false`. Checker or
+ cross-check failure must render “did not complete,” never a clean report.
+ Present the terminal widget or Markdown and preserve the full evidence chain
+ in text fallback.
+
The deterministic layer is `attune_verify.verify(content, context)`.
The caller declares the truth boundaries; verify performs the lookups —
it never auto-discovers them. Run this via Bash from the project root,
passing the file under test as `F`:
```bash
F="<path/to/generated.md>" python -c "
import json, os
from pathlib import Path
from attune_verify import verify, VerifyContext
content = Path(os.environ['F']).read_text(encoding='utf-8')
ctx = VerifyContext(project_root=Path.cwd())
result = verify(content, ctx)
print(json.dumps({
'ok': result.ok,
'checked': result.checked,
'findings': [
{'kind': f.kind.value, 'severity': f.severity,
'detail': f.detail, 'evidence': f.evidence, 'location': f.location}
for f in result.findings
],
}, ensure_ascii=False, indent=2))
"
```
Then present the findings as a readable report (errors first, then
warnings), grouped by kind. Each finding carries `kind`
(`unresolved_import` / `unknown_flag` / `dead_link` / `count_mismatch`),
`severity` (`error` | `warning`), `detail`, `evidence`, and `location`.
## Declaring Tighter Boundaries
The default context only checks imports and links (project root). To
catch flag and count hallucinations, declare the sources the content
claims against — `VerifyContext` is the single place to do it:
- **CLI flags** — pre-capture `--help` and pass it so the checker
knows the real flags:
```python
ctx = VerifyContext(
project_root=Path.cwd(),
help_commands={"attune": <captured --help text>},
allowed_help_cmds=frozenset({"attune"}), # safe to shell out to
)
```
- **Counts** — declare what a number should be, by label. Values may be
plain ints or zero-arg callables:
```python
ctx = VerifyContext(
project_root=Path.cwd(),
count_sources={"templates": 259, "skills": lambda: len(skill_dirs)},
)
```
A flag whose command is neither pre-captured nor in `allowed_help_cmds`
yields a **warning, not a silent pass** — surface it so the user knows
it went unchecked.
## Reading The Report
- **`result.ok is False`** → at least one error-severity finding. These
are confirmed hallucinations (the import does not resolve, the flag is
not in `--help`). Quote the `evidence` line and point at where to fix.
- **Warnings** → something couldn't be checked (no count source
declared, a command not on the allow-list). Not a failure — a gap in
the declared boundaries. Tell the user what to declare to close it.
- **Empty findings** → the deterministic layer found nothing wrong.
Then do the ambient semantic cross-check and report any claims worth a
human's eye, clearly labeled as cross-check (not deterministic).
Do **not** invent findings, and do **not** soften a deterministic error
into a "maybe" — the checkers are authoritative for entity existence.
## When It Belongs In A Pipeline
For a hard gate (fail the run on any error finding), the library
exposes `raise_if_failed(result)`. The `/verify` skill itself is the
interactive, report-only surface — it never hard-gates. The hard gate
is for callers like attune-author's post-generation step.