AGENTS.md · git:20261029.5a90ce2 · 2026-10-29 · sha256 0450dba55712af91

AGENTS.md git:20261029.5a90ce2A

Immutable. This exact content never changes and is served at /api/v1/blob/0450dba55712af91.

# AGENTS.md

CodeNib is a Python 3.10+ code analysis agent with tree-sitter chunking,
semantic graph construction, hybrid retrieval, and MCP-facing search tools.
Use this file as the repo-level contract for Codex and other code agents.

## Layered Objectives

Work in this order unless the user gives a narrower task:

1. Protect `main`: do not merge draft PRs, PRs with unresolved code failures, or
   stacked PRs whose dependency chain is not clear.
2. Make changes reviewable: keep commits focused, preserve existing ownership
   boundaries, and avoid unrelated refactors.
3. Verify the changed surface: run the smallest relevant test first, then widen
   to the job or marker tier that matches the risk.
4. Keep stacked work coherent: merge or rebase dependencies before dependents,
   and re-check diffs after a base branch changes.
5. Codify repeated feedback here or in the nearest directory-specific rules
   file so future agents do not repeat the same mistake.

For broad multi-step work, keep an explicit layered goal active until the
requested surface is implemented, locally verified, documented, and reconciled
with relevant PRs/issues. Do not mark a broad goal complete just because a
single subgoal landed. Prefer local targeted verification and continued
implementation over waiting on slow remote CI; run the remote/slow tier after
the merge queue or feature batch is otherwise coherent.
Update each milestone's outcomes and open gates in the PR that changes them;
an update only on a parallel branch does not keep that PR's roadmap current.

For the current multi-language SCIP cold-start and acceleration program, use
`docs/scip_multilanguage_roadmap.md` as the durable objective record. Update it
whenever a backend is promoted, a C++ acceleration gate changes, or a related
issue/PR is closed.

For the storage boundary and the time-bounded hybrid-index experiment, use
`docs/storage_backend_roadmap.md` as the durable objective record. Update it
whenever the Wiki database boundary changes, an experiment gate changes, or a
related issue or PR is reconciled.

For the cognitive-debt reduction program, use
`docs/cognitive_debt_roadmap.md` as the durable objective record. Update it
whenever a package changes lifecycle class, an experimental surface is retired,
an architectural dependency direction changes, or a subtraction iteration is
completed.

For the adoption, OpenRouter, and Wiki preview program, use
`docs/adoption_roadmap.md` as the durable objective record. Keep shipping
CodeGraph capabilities separate from experimental grep/Jev results, and
update the relevant milestone when its acceptance gates or PR status change.
The immediate CodeGraph product priority is the real-workflow net-benefit
gate in A0 of that roadmap, with the pilot contract in
`docs/experiments/codegraph_net_benefit.md`. Keep workflow acceptance,
matched task outcomes, and observed repeat use separate from retrieval scores
and demonstrations. Record a promotion, narrowed scope, or no-go decision;
setting up the study does not complete the program. Preserve source-identity
checks and the Storage Scope Guard while improving working-tree usability.

The public repository entry must produce a navigable Repo Wiki through the
existing Wiki pipeline. A source browser or a short AI explanation does not
satisfy that outcome. Generation progress must reflect actual work; completed
pages and interrupted attempts must survive refresh. Saved visitor Wikis start as
unlisted, link-readable results. An explicit owner publication may add a
completed, source-checked Wiki to Browse; never retroactively list older results
or equate generation consent with catalog publication. Use the shared
architecture reader for prepared and visitor Wikis. Reduce initial chapter
breadth before cutting supported mechanisms, rationale or boundaries from prose;
source-check success alone does not establish reader quality.
Visitor Wiki search must plan chapter coverage, not reuse an issue-localization
prompt or silently exclude tests needed by a validation chapter. Keep the same
reader rubric available on prepared and newly generated visitor pages. Compare
the same source/outline/evidence when evaluating prose; a longer explanation or
a green citation check does not establish parity with indexed examples.
A ready chapter must not silently hide a rejected architecture or unavailable
code index: display explicit visual coverage. Hosted graph views must come from
the pinned compiler/source artifacts, never model-authored arrows. Keep source
previews self-contained so opening a saved graph cannot trigger provider calls.
Public first-visit examples must also be self-contained: retain the source
commit, indexed relationship anchors and exact excerpts. Label a selected call
path as such; do not present it as a complete architecture or runtime trace.
Keep the marketing and Wiki example in one shared asset, and keep source-only
previews separate from claims about arbitrary-repository graph coverage.
Switching first-visit nodes or calls must keep the surrounding layout and page
scroll position stable. Preserve syntax coloring and source-line alignment;
scroll long excerpts only inside the source pane.

## Storage Scope Guard

- The only product database boundary is the Wiki-only
  `codenib.storage.WikiStore` facade; its supported implementation is
  `SQLiteWikiStore`, owned by the Wiki domain. Keep pluggability to constructor
  injection plus the Wiki conformance harness. Do not add a backend registry,
  capability matrix, or dynamic discovery system.
- Keep `RepoManifest`, BM25, FAISS, igraph, and portable context payloads as
  manifest-bound file artifacts. Do not move their refs, snapshots, jobs,
  leases, or lifecycle into a generic database or object-store abstraction.
- At the v0.2.3 boundary, `codenib.storage` became a single-file Wiki-only
  facade and the retained-storage CLI commands were removed. Do not turn that
  module back into a package or reintroduce a generic catalog, CAS, schema
  state, storage protocol, backend registry, or product route.
- The v0.2.3 release was the complete migration window for legacy
  `agentwiki_*.json` caches. Do not restore JSON discovery, read-through,
  adoption, writes, file locking, provenance markers, or audit accounting;
  Wiki persistence requires an injected `WikiStore`.
- Keep built-in Wiki generation waiting bounded without stealing an owner's
  lock or starting duplicate work. Maintenance audits and dry runs must not
  mutate the source database or broaden the public storage facade.
- One exception is authorized through 2026-10-04: the source-checkout-only H1
  experiment under `scripts/experimental/hybrid_index/`, its developer script,
  and focused tests. H1 accepts exactly one verified BM25-only portable
  artifact, stores its immutable archive in one local SHA-256 CAS, and records
  generation, snapshot, and ref identity in one private four-table SQLite WAL
  catalog. It must not be installed under `codenib`, shipped in a wheel,
  imported by product code, exposed by the official CLI, or wired into Web,
  MCP, configuration, or a default route.
- The H1 owner is the index/artifact maintainers. By 2026-10-04 they must either
  promote it through an explicit scope decision backed by a current non-test
  consumer, a representative benchmark, portable-artifact compatibility, and
  the recorded safety gates, or delete its implementation, executable script,
  and dedicated tests while retaining the decision record. Default-off or
  source-only status is not permission to keep an unevaluated experiment.
- H2-H7 are demand-gated hypotheses, not implementation authorization. Do not
  add FAISS/igraph persistence, jobs, leases, hot switching, per-file CAS,
  xref de-materialization, GC, overlays, PostgreSQL, object storage, shared ANN,
  distributed coordination, or backend discovery before the preceding gate
  and a current product need are recorded in the roadmap. Tests, roadmap
  entries, and planned adapters do not count as consumers.
- Put any future persistence protocol at a real domain I/O boundary. Name the
  current production call site, use one domain protocol and one implementation
  first, and remove the old path in the same program. Sharing SQLite does not
  justify recreating a generic catalog or object-store protocol.
- For a non-trivial cross-thread/process lock, retained owner, or multi-phase
  lifecycle, state the invariant and linearization point in the PR or adjacent
  docstring. Also state lock order when multiple locks exist and recovery
  ownership when state outlives the call; add one deterministic test of the
  invariant or reproduced race. Otherwise reuse an existing transaction,
  lease, pin, or recovery path, or simplify the design.
- Prefer one domain conformance harness driven by a store fixture/factory, one
  implementation suite, and one product-vertical regression. Fault tests must
  target observable boundaries or a named prior bug, not arbitrary Python-line
  interruption.
- Replace a milestone's prior status with its current outcomes, open gates, and
  PR/ADR links; do not append implementation chronology. Put implementation
  traces in code, tests, ADRs, or the changelog.

## Dev Commands

```bash
make dev          # pip install -e ".[dev,test]"
make test         # run the local unit tier; never makes billed API calls
make test-all     # run every marker tier, including external/slow tests
make test-slow    # run credential- and model-dependent slow tests explicitly
make bootstrap-ubuntu # install Ubuntu system deps, Python dev deps, and all local toolchains
make bootstrap    # install Python dev deps and local toolchains without sudo/apt
make multilang-tools # install active SCIP/LSP plus cold-start SCIP/LSP smoke tools
make toolchain-doctor # verify locally installed toolchain commands resolve on PATH
make scip         # install active SCIP toolchain under CODENIB_SCIP_TOOLS_DIR
make zoekt-tool   # install Zoekt binaries under CODENIB_SCIP_TOOLS_DIR
make web-deps     # install Next.js frontend dependencies
make scip-cold-start-system-deps-ubuntu # install Ubuntu base deps for cold-start tools
make scip-cold-start-tools # install Java/C#/JVM/Ruby/PHP SCIP smoke toolchains
make lsp-smoke-tools # install Java/C#/Ruby/PHP/Kotlin LSP smoke tools
make ruby-project-bundle PROJECT_ROOT=/path/to/ruby/repo # prepare Ruby overlay Gemfile/bundle
make php-project-scip-tool PROJECT_ROOT=/path/to/php/repo # optionally prewarm project-local scip-php
make core-build   # build the C++ core/ pybind acceleration module
make ask-quality  # run the deterministic Ask retrieval gate for this checkout
make scip-cold-start-smoke # run generated SCIP cold-start smoke
make scip-project-smoke # run SCIP smoke on PROJECT_ROOT/PROJECT_LANGUAGE
make lsp-smoke    # run generated Java/C#/Ruby/PHP/Kotlin LSP graph smoke
make lsp-project-smoke # run LSP smoke on PROJECT_ROOT/PROJECT_LANGUAGE
make graph-route-alignment # compare graph routes for PROJECT_ROOT/PROJECT_LANGUAGE
make multilang-registry-check # check registry and capability matrix consistency
make install      # pip install -e .
```

For Ruby toolchain installs, prefer the Makefile targets over ad hoc `gem`
commands. If the default system gem lacks Ruby headers, pass
`RUBY_GEM=/path/to/gem` to the same target instead of installing tools outside
`CODENIB_SCIP_TOOLS_DIR`.

Pre-commit uses black with line length 88, isort, flake8+bugbear, and
clang-format for C/C++. Use the formatter versions pinned by
`.pre-commit-config.yaml` through their hooks; an unrelated environment's
Black or isort can produce changes that fail the release all-files check.

## Git, Commit, And PR Rules

- Do not add AI attribution footers or generated-by text to commits, PR bodies,
  or review comments.
- Use Conventional Commits: `type(scope): summary`, with `type` in `feat`,
  `fix`, `docs`, `refactor`, `perf`, `test`, `chore`, or `ci`.
- Keep commit subjects imperative and at most 72 characters. Use the body for
  why the change was made and how it was verified.
- When passing multiline commit, squash-merge, PR, or review bodies through a
  CLI, supply real newline bytes rather than literal `\n` escapes. After a
  squash or rebase merge, inspect the resulting commit message before treating
  reconciliation as complete.
- PR bodies must follow `.github/PULL_REQUEST_TEMPLATE.md` with the existing
  headings in order: `Summary`, `Changes`, `Type of Change`, `Testing`,
  `Checklist`.
- This repository allows squash and rebase merges, not merge commits. For
  stacked PRs, do not blindly squash each PR into `main`; first confirm the
  dependency chain and restack dependents so their diffs stay clean.
- Before merging, confirm the PR is not draft, is mergeable, has no failing
  required checks, and has all dependent changes merged or explicitly accounted
  for. Treat cancelled checks as incomplete, not green.
  Inspect unresolved review threads as well as check statuses. Resolve valid
  findings; verify provider-protocol findings against current official schemas
  and record the evidence before resolving a false positive.
- Pin every external GitHub Action in workflows and composite actions to a
  full 40-character commit SHA, with the exact release tag retained in a
  trailing comment. Local actions under `./` do not need a commit pin.

## Documentation Writing

Public pages (`README.md` and the published `docs/` site) are read by people
deciding whether CodeNib fits their problem and how to use it. Write for that
reader. A page is not a record of how the work was done.

- Lead with the point. Open each page and section with what it is for and
  what the reader should take away, in one or two plain sentences. Then
  explain the concept, then how to use it, and only then the reference detail:
  flags, paths, field names, and edge cases.
- Explain concepts before code names. Define each term in plain words where it
  first appears, and then use it consistently. A reader should be able to follow
  the explanation without knowing class names, module paths, run IDs, or
  experiment-arm labels.
- Write product documentation, not an experiment report. A page says what
  something is, when to use it, how to run it, and what it achieved, in that
  order. State a result in a sentence or a small table with its scope, then
  stop. Run provenance (dates, commits, seeds, digests, token and time
  figures), protocol mechanics (preflight gates, denominators, fallbacks), and
  lists of caveats go in a collapsed `??? note` block at the end of the
  section, on the harness reference page, or in the excluded
  `docs/experiments/` log, never in the main text.
- Keep only what the reader needs. Cut internal history, and cut caveats that
  do not change what the reader would do. Move implementation detail to the
  reference or harness page where it belongs. Do not paste lab-notebook prose
  into user-facing pages.
- Keep pages current. Replace an outdated statement in place; do not append
  "update" notes or a chronology. Experiment logs belong in the excluded
  `docs/experiments/` tree or in the PR.
- Scope every claim briefly. A result names the benchmark, the method and the
  setup difference that matters, and the sample size, in the same sentence,
  and it links to the evidence. Do not present a research-script result as
  product behavior.
- Use short paragraphs with one idea each. Use a table to compare items with the
  same fields, and prose to explain why. Avoid chains of footnotes and stacked
  qualifiers.
- Before committing, reread each changed page from the top as a newcomer would.
  After the first screen, the reader should know what the page is for and what
  to do next.

## Testing Guidance

- Dependency and optional-extra changes in `pyproject.toml` must update
  `uv.lock` in the same change. Run `uv lock --check`; the documentation CI
  installs with `uv sync --locked --only-group docs`.
- Public documentation changes require both `mkdocs build --strict` and
  `python scripts/check_public_docs.py`. Public pages must not link to excluded
  documents, including through GitHub URLs.
  When excluding a new internal document, add it to both `REQUIRED_EXCLUSIONS`
  and `FORBIDDEN_PUBLIC_PREFIXES` in `scripts/check_public_docs.py`.
  A change to the `mkdocs.yml` navigation also requires
  `pytest test/scripts/test_check_public_docs.py`, which pins the tab order.
  A change to benchmark or method docs, or to `codenib/eval/benchmarks/`,
  requires `python scripts/support_matrix.py --check`. Edit
  `scripts/support_matrix.yaml` and regenerate the tables with `--write`;
  do not hand-edit the generated blocks.
- Unit tier: `pytest -m "not slow and not integration and not integration_serial and not integration_serial_consumer" -x --tb=short`
- Integration tier: `pytest -m integration --tb=short`
- Serial integration tier: `pytest -m integration_serial -v --tb=short`
- Generic LSP graph smoke:
  `make lsp-smoke`
- SCIP cold-start smoke:
  `make scip-cold-start-smoke`
- Route-level backend alignment:
  `make graph-route-alignment PROJECT_LANGUAGE=java PROJECT_ROOT=/path/to/repo`
- Slow tier needs LLM credentials and GPU resources; inspect whether a red slow
  job is an infra/config failure before treating it as a code regression.
- New tests default to unit scope. Add heavier markers only when the test really
  needs external repos, SCIP indexing, LLM calls, GPU embeddings, or serial
  mutation.
- Native benchmark readiness must show that the empty control collects and
  executes the intended tests, as well as failing where the reference passes.
  A reference-only production dependency can otherwise conceal a collection
  failure. Record collection errors and model/native test-patch conflicts
  separately from behavioral failures; retain all original selected outcomes.
  Change grading methodology only in a separate, explicit protocol, and keep
  reference material and grading feedback outside model inputs.
- A committed Docker image can retain bind destinations in `Config.Volumes`.
  Bind the active workspace once and mask only other inherited destinations;
  masking the active bind again causes a duplicate-mount failure. Record each
  returned container identity before startup or inspection, verify its owner
  label before removal, and clean up only that invocation's identities.

## Critical Code Conventions

- `CodeChunk.start_line` and `CodeChunk.end_line` are 0-based tree-sitter line
  numbers.
- `CodeLocation.start_line` and `CodeLocation.end_line` are 1-based output
  locations. The conversion happens downstream in `dataset/gt_locate.py`.
- `.c` files use the `cpp` chunker; there is no separate C chunker.
- Graph node and edge types are centralized in `codenib/types.py`. Use the
  constants and helpers there instead of hard-coded string literals.
- Describe `dependency_subgraph` as bounded incoming/outgoing static references
  in public copy and tool reference pages. Legacy `callers`/`callees` labels do
  not make every edge an invocation: fields, types and function values also
  produce references. Keep documentation tables aligned with MCP semantics;
  source inspection must support call-path and runtime claims.
- Persisted graph schema changes require updating `_SCHEMA_VERSION` in
  `codenib/graph/code_graph.py` and checking C++ decoder parity under `core/`.
- New languages should flow through the language registry instead of adding
  isolated dispatch maps.
- ContextBench source-overlap diagnostics must use the pinned official
  `Gold.line_spans_init()` normalization (available as `gold_line_spans` in
  evaluator reports), not raw annotation path strings. Container-prefixed
  `/workspace/<repo>/` or `/testbed/` annotations otherwise silently lose credit
  in auxiliary counts even when official scores are correct. Preserve frozen
  results and label any retrospective diagnostic correction explicitly.
  If canonical annotations do not resolve in the pinned source, retain official
  paths and denominators and report evaluation-only `gold_source_issues`;
  do not silently remap paths or discard unavailable annotation lines.
- Solver source credit requires a version-bound execution receipt and an exact
  observation in a confirmed model input. Requested shell ranges, assistant
  summaries, elided output and undelivered submission diffs are not read evidence.
  Keep arbitrary shell output and modified-file coordinates explicitly
  unattributed until their original-source mapping is verified; partial coverage
  must not be presented as a complete ContextBench score.
  Multi-file search observations must bind each complete visible match to its
  file and captured version; a shared line number/text is not file identity.
  Report scan/output caps and skipped files rather than implying full coverage.
- ContextBench grep audits must validate the captured original reply and its
  actual model delivery against source-version receipts. Re-executing ripgrep
  is not a byte-for-byte oracle: traversal order and capped hit subsets may
  change on an unchanged checkout. Preserve recorded inputs and scores when a
  diagnostic rerun differs; do not replace observations with the rerun.
- Preserve prospective benchmark tasks by reading raw dataset READMEs and
  metadata-only columns instead of HTML dataset viewers, which may automatically
  render gold contexts and reference patches. Record any exposed preview task
  IDs and exclude them from future unseen validation. Keep completed cohorts and
  their timing intact rather than retroactively changing their outcomes.
- Before freezing a paid model experiment, verify the selected model's current
  thinking, tool-choice and output-limit constraints in official documentation.
  Requested CLI flags do not prove effective behavior. Distinguish main assistant
  models, auxiliary models in terminal usage, SDK turns and HTTP request counts.
  For source-only delivery checks, define accepted capped-response outcomes
  prospectively; do not change an earlier protocol's terminal-status or source
  credit rules after seeing its failures.

## Directory-Specific Rules

Codex does not automatically load `.claude/CLAUDE.md`, but those files remain
the authoritative local notes for several subtrees. Read the matching file
before editing the listed area:

| Path | Required extra context |
| --- | --- |
| `codenib/code_chunking/` | `codenib/code_chunking/CLAUDE.md` |
| `codenib/graph/` | `codenib/graph/CLAUDE.md` |
| `codenib/dataset/` | `codenib/dataset/CLAUDE.md` |
| `scripts/agent_compile/` | `scripts/agent_compile/CLAUDE.md` |
| `test/` | `test/CLAUDE.md` |

The project-wide Claude notes in `.claude/CLAUDE.md` are useful background, but
do not rely on Codex discovering them automatically.