AGENTS.md · git:20261030.2f39d30 · 2026-10-30 · sha256 1bd4b51c941ab412
AGENTS.md git:20261030.2f39d30A
Immutable. This exact content never changes and is served at /api/v1/blob/1bd4b51c941ab412.
# AGENTS.md CodeNib is a Python 3.10+ code analysis agent with tree-sitter chunking, semantic graph construction, hybrid retrieval, and MCP-facing search tools. Use this file as the repo-level contract for Codex and other code agents. ## Layered Objectives Work in this order unless the user gives a narrower task: 1. Protect `main`: do not merge draft PRs, PRs with unresolved code failures, or stacked PRs whose dependency chain is not clear. 2. Make changes reviewable: keep commits focused, preserve existing ownership boundaries, and avoid unrelated refactors. 3. Verify the changed surface: run the smallest relevant test first, then widen to the job or marker tier that matches the risk. 4. Keep stacked work coherent: merge or rebase dependencies before dependents, and re-check diffs after a base branch changes. 5. Codify repeated feedback here or in the nearest directory-specific rules file so future agents do not repeat the same mistake. For broad multi-step work, keep an explicit layered goal active until the requested surface is implemented, locally verified, documented, and reconciled with relevant PRs/issues. Do not mark a broad goal complete just because a single subgoal landed. Prefer local targeted verification and continued implementation over waiting on slow remote CI; run the remote/slow tier after the merge queue or feature batch is otherwise coherent. Update each milestone's outcomes and open gates in the PR that changes them; an update only on a parallel branch does not keep that PR's roadmap current. For the current multi-language SCIP cold-start and acceleration program, use `docs/scip_multilanguage_roadmap.md` as the durable objective record. Update it whenever a backend is promoted, a C++ acceleration gate changes, or a related issue/PR is closed. For the storage boundary and the gated persistence hypotheses, use `docs/storage_backend_roadmap.md` as the durable objective record. Update it whenever the Wiki database boundary changes, an experiment gate changes, or a related issue or PR is reconciled. For the cognitive-debt reduction program, use `docs/cognitive_debt_roadmap.md` as the durable objective record. Update it whenever a package changes lifecycle class, an experimental surface is retired, an architectural dependency direction changes, or a subtraction iteration is completed. For the adoption, OpenRouter, and Wiki preview program, use `docs/adoption_roadmap.md` as the durable objective record. Keep shipping CodeGraph capabilities separate from experimental grep/Jev results, and update the relevant milestone when its acceptance gates or PR status change. The immediate CodeGraph product priority is the real-workflow net-benefit gate in A0 of that roadmap, with the pilot contract in `docs/experiments/codegraph_net_benefit.md`. Keep workflow acceptance, matched task outcomes, and observed repeat use separate from retrieval scores and demonstrations. Record a promotion, narrowed scope, or no-go decision; setting up the study does not complete the program. Preserve source-identity checks and the Storage Scope Guard while improving working-tree usability. The public repository entry must produce a navigable Repo Wiki through the existing Wiki pipeline. A source browser or a short AI explanation does not satisfy that outcome. Generation progress must reflect actual work; completed pages and interrupted attempts must survive refresh. Saved visitor Wikis start as unlisted, link-readable results. An explicit owner publication may add a completed, source-checked Wiki to Browse; never retroactively list older results or equate generation consent with catalog publication. Use the shared architecture reader for prepared and visitor Wikis. Reduce initial chapter breadth before cutting supported mechanisms, rationale or boundaries from prose; source-check success alone does not establish reader quality. Visitor Wiki search must plan chapter coverage, not reuse an issue-localization prompt or silently exclude tests needed by a validation chapter. Keep the same reader rubric available on prepared and newly generated visitor pages. Compare the same source/outline/evidence when evaluating prose; a longer explanation or a green citation check does not establish parity with indexed examples. A ready chapter must not silently hide a rejected architecture or unavailable code index: display explicit visual coverage. Hosted graph views must come from the pinned compiler/source artifacts, never model-authored arrows. Keep source previews self-contained so opening a saved graph cannot trigger provider calls. Public first-visit examples must also be self-contained: retain the source commit, indexed relationship anchors and exact excerpts. Label a selected call path as such; do not present it as a complete architecture or runtime trace. Keep the marketing and Wiki example in one shared asset, and keep source-only previews separate from claims about arbitrary-repository graph coverage. Switching first-visit nodes or calls must keep the surrounding layout and page scroll position stable. Preserve syntax coloring and source-line alignment; scroll long excerpts only inside the source pane. ## Storage Scope Guard - The only product database boundary is the Wiki-only `codenib.storage.WikiStore` facade; its supported implementation is `SQLiteWikiStore`, owned by the Wiki domain. Keep pluggability to constructor injection plus the Wiki conformance harness. Do not add a backend registry, capability matrix, or dynamic discovery system. - Keep `RepoManifest`, BM25, FAISS, igraph, and portable context payloads as manifest-bound file artifacts. Do not move their refs, snapshots, jobs, leases, or lifecycle into a generic database or object-store abstraction. - At the v0.2.3 boundary, `codenib.storage` became a single-file Wiki-only facade and the retained-storage CLI commands were removed. Do not turn that module back into a package or reintroduce a generic catalog, CAS, schema state, storage protocol, backend registry, or product route. - The v0.2.3 release was the complete migration window for legacy `agentwiki_*.json` caches. Do not restore JSON discovery, read-through, adoption, writes, file locking, provenance markers, or audit accounting; Wiki persistence requires an injected `WikiStore`. - Keep built-in Wiki generation waiting bounded without stealing an owner's lock or starting duplicate work. Maintenance audits and dry runs must not mutate the source database or broaden the public storage facade. - The source-checkout-only H1 hybrid-index experiment was deleted on 2026-10-29: its 2026-10-04 deadline passed without a current non-test consumer or a promotion decision. Its evidence stays in `docs/experiments/hybrid_index_h1.md`; do not restore its implementation, developer script, or tests. Default-off or source-only status is not permission to keep an unevaluated experiment. - H2-H7 are demand-gated hypotheses, not implementation authorization. Do not add FAISS/igraph persistence, jobs, leases, hot switching, per-file CAS, xref de-materialization, GC, overlays, PostgreSQL, object storage, shared ANN, distributed coordination, or backend discovery before the preceding gate and a current product need are recorded in the roadmap. Tests, roadmap entries, and planned adapters do not count as consumers. - Put any future persistence protocol at a real domain I/O boundary. Name the current production call site, use one domain protocol and one implementation first, and remove the old path in the same program. Sharing SQLite does not justify recreating a generic catalog or object-store protocol. - For a non-trivial cross-thread/process lock, retained owner, or multi-phase lifecycle, state the invariant and linearization point in the PR or adjacent docstring. Also state lock order when multiple locks exist and recovery ownership when state outlives the call; add one deterministic test of the invariant or reproduced race. Otherwise reuse an existing transaction, lease, pin, or recovery path, or simplify the design. - Prefer one domain conformance harness driven by a store fixture/factory, one implementation suite, and one product-vertical regression. Fault tests must target observable boundaries or a named prior bug, not arbitrary Python-line interruption. - Replace a milestone's prior status with its current outcomes, open gates, and PR/ADR links; do not append implementation chronology. Put implementation traces in code, tests, ADRs, or the changelog. ## Dev Commands ```bash make dev # pip install -e ".[dev,test]" make test # run the local unit tier; never makes billed API calls make test-all # run every marker tier, including external/slow tests make test-slow # run credential- and model-dependent slow tests explicitly make bootstrap-ubuntu # install Ubuntu system deps, Python dev deps, and all local toolchains make bootstrap # install Python dev deps and local toolchains without sudo/apt make multilang-tools # install active SCIP/LSP plus cold-start SCIP/LSP smoke tools make toolchain-doctor # verify locally installed toolchain commands resolve on PATH make scip # install active SCIP toolchain under CODENIB_SCIP_TOOLS_DIR make zoekt-tool # install Zoekt binaries under CODENIB_SCIP_TOOLS_DIR make web-deps # install Next.js frontend dependencies make scip-cold-start-system-deps-ubuntu # install Ubuntu base deps for cold-start tools make scip-cold-start-tools # install Java/C#/JVM/Ruby/PHP SCIP smoke toolchains make lsp-smoke-tools # install Java/C#/Ruby/PHP/Kotlin LSP smoke tools make ruby-project-bundle PROJECT_ROOT=/path/to/ruby/repo # prepare Ruby overlay Gemfile/bundle make php-project-scip-tool PROJECT_ROOT=/path/to/php/repo # optionally prewarm project-local scip-php make core-build # build the C++ core/ pybind acceleration module make ask-quality # run the deterministic Ask retrieval gate for this checkout make scip-cold-start-smoke # run generated SCIP cold-start smoke make scip-project-smoke # run SCIP smoke on PROJECT_ROOT/PROJECT_LANGUAGE make lsp-smoke # run generated Java/C#/Ruby/PHP/Kotlin LSP graph smoke make lsp-project-smoke # run LSP smoke on PROJECT_ROOT/PROJECT_LANGUAGE make graph-route-alignment # compare graph routes for PROJECT_ROOT/PROJECT_LANGUAGE make multilang-registry-check # check registry and capability matrix consistency make install # pip install -e . ``` For Ruby toolchain installs, prefer the Makefile targets over ad hoc `gem` commands. If the default system gem lacks Ruby headers, pass `RUBY_GEM=/path/to/gem` to the same target instead of installing tools outside `CODENIB_SCIP_TOOLS_DIR`. Pre-commit uses black with line length 88, isort, flake8+bugbear, and clang-format for C/C++. Use the formatter versions pinned by `.pre-commit-config.yaml` through their hooks; an unrelated environment's Black or isort can produce changes that fail the release all-files check. ## Git, Commit, And PR Rules - Do not add AI attribution footers or generated-by text to commits, PR bodies, or review comments. - Use Conventional Commits: `type(scope): summary`, with `type` in `feat`, `fix`, `docs`, `refactor`, `perf`, `test`, `chore`, or `ci`. - Keep commit subjects imperative and at most 72 characters. Use the body for why the change was made and how it was verified. - When passing multiline commit, squash-merge, PR, or review bodies through a CLI, supply real newline bytes rather than literal `\n` escapes. After a squash or rebase merge, inspect the resulting commit message before treating reconciliation as complete. - PR bodies must follow `.github/PULL_REQUEST_TEMPLATE.md` with the existing headings in order: `Summary`, `Changes`, `Type of Change`, `Testing`, `Checklist`. - This repository allows squash and rebase merges, not merge commits. For stacked PRs, do not blindly squash each PR into `main`; first confirm the dependency chain and restack dependents so their diffs stay clean. - Before merging, confirm the PR is not draft, is mergeable, has no failing required checks, and has all dependent changes merged or explicitly accounted for. Treat cancelled checks as incomplete, not green. Inspect unresolved review threads as well as check statuses. Resolve valid findings; verify provider-protocol findings against current official schemas and record the evidence before resolving a false positive. - Pin every external GitHub Action in workflows and composite actions to a full 40-character commit SHA, with the exact release tag retained in a trailing comment. Local actions under `./` do not need a commit pin. ## Documentation Writing Public pages (`README.md` and the published `docs/` site) are read by people deciding whether CodeNib fits their problem and how to use it. Write for that reader. A page is not a record of how the work was done. - Lead with the point. Open each page and section with what it is for and what the reader should take away, in one or two plain sentences. Then explain the concept, then how to use it, and only then the reference detail: flags, paths, field names, and edge cases. - Explain concepts before code names. Define each term in plain words where it first appears, and then use it consistently. A reader should be able to follow the explanation without knowing class names, module paths, run IDs, or experiment-arm labels. - Write product documentation, not an experiment report. A page says what something is, when to use it, how to run it, and what it achieved, in that order. State a result in a sentence or a small table with its scope, then stop. Run provenance (dates, commits, seeds, digests, token and time figures), protocol mechanics (preflight gates, denominators, fallbacks), and lists of caveats go in a collapsed `??? note` block at the end of the section, on the harness reference page, or in the excluded `docs/experiments/` log, never in the main text. - Keep only what the reader needs. Cut internal history, and cut caveats that do not change what the reader would do. Move implementation detail to the reference or harness page where it belongs. Do not paste lab-notebook prose into user-facing pages. - Keep pages current. Replace an outdated statement in place; do not append "update" notes or a chronology. Experiment logs belong in the excluded `docs/experiments/` tree or in the PR. - Scope every claim briefly. A result names the benchmark, the method and the setup difference that matters, and the sample size, in the same sentence, and it links to the evidence. Do not present a research-script result as product behavior. - Use short paragraphs with one idea each. Use a table to compare items with the same fields, and prose to explain why. Avoid chains of footnotes and stacked qualifiers. - Before committing, reread each changed page from the top as a newcomer would. After the first screen, the reader should know what the page is for and what to do next. ## Testing Guidance - Dependency and optional-extra changes in `pyproject.toml` must update `uv.lock` in the same change. Run `uv lock --check`; the documentation CI installs with `uv sync --locked --only-group docs`. - Public documentation changes require both `mkdocs build --strict` and `python scripts/check_public_docs.py`. Public pages must not link to excluded documents, including through GitHub URLs. When excluding a new internal document, add it to both `REQUIRED_EXCLUSIONS` and `FORBIDDEN_PUBLIC_PREFIXES` in `scripts/check_public_docs.py`. A change to the `mkdocs.yml` navigation also requires `pytest test/scripts/test_check_public_docs.py`, which pins the tab order. A change to benchmark or method docs, or to `codenib/eval/benchmarks/`, requires `python scripts/support_matrix.py --check`. Edit `scripts/support_matrix.yaml` and regenerate the tables with `--write`; do not hand-edit the generated blocks. - Unit tier: `pytest -m "not slow and not integration and not integration_serial and not integration_serial_consumer" -x --tb=short` - Integration tier: `pytest -m integration --tb=short` - Serial integration tier: `pytest -m integration_serial -v --tb=short` - Generic LSP graph smoke: `make lsp-smoke` - SCIP cold-start smoke: `make scip-cold-start-smoke` - Route-level backend alignment: `make graph-route-alignment PROJECT_LANGUAGE=java PROJECT_ROOT=/path/to/repo` - Slow tier needs LLM credentials and GPU resources; inspect whether a red slow job is an infra/config failure before treating it as a code regression. - New tests default to unit scope. Add heavier markers only when the test really needs external repos, SCIP indexing, LLM calls, GPU embeddings, or serial mutation. - Native benchmark readiness must show that the empty control collects and executes the intended tests, as well as failing where the reference passes. A reference-only production dependency can otherwise conceal a collection failure. Record collection errors and model/native test-patch conflicts separately from behavioral failures; retain all original selected outcomes. Change grading methodology only in a separate, explicit protocol, and keep reference material and grading feedback outside model inputs. - A committed Docker image can retain bind destinations in `Config.Volumes`. Bind the active workspace once and mask only other inherited destinations; masking the active bind again causes a duplicate-mount failure. Record each returned container identity before startup or inspection, verify its owner label before removal, and clean up only that invocation's identities. ## Critical Code Conventions - `CodeChunk.start_line` and `CodeChunk.end_line` are 0-based tree-sitter line numbers. - `CodeLocation.start_line` and `CodeLocation.end_line` are 1-based output locations. The conversion happens downstream in `dataset/gt_locate.py`. - `.c` files use the `cpp` chunker; there is no separate C chunker. - Graph node and edge types are centralized in `codenib/types.py`. Use the constants and helpers there instead of hard-coded string literals. - Describe `dependency_subgraph` as bounded incoming/outgoing static references in public copy and tool reference pages. Legacy `callers`/`callees` labels do not make every edge an invocation: fields, types and function values also produce references. Keep documentation tables aligned with MCP semantics; source inspection must support call-path and runtime claims. - Persisted graph schema changes require updating `_SCHEMA_VERSION` in `codenib/graph/code_graph.py` and checking C++ decoder parity under `core/`. - New languages should flow through the language registry instead of adding isolated dispatch maps. - ContextBench source-overlap diagnostics must use the pinned official `Gold.line_spans_init()` normalization (available as `gold_line_spans` in evaluator reports), not raw annotation path strings. Container-prefixed `/workspace/<repo>/` or `/testbed/` annotations otherwise silently lose credit in auxiliary counts even when official scores are correct. Preserve frozen results and label any retrospective diagnostic correction explicitly. If canonical annotations do not resolve in the pinned source, retain official paths and denominators and report evaluation-only `gold_source_issues`; do not silently remap paths or discard unavailable annotation lines. - Solver source credit requires a version-bound execution receipt and an exact observation in a confirmed model input. Requested shell ranges, assistant summaries, elided output and undelivered submission diffs are not read evidence. Keep arbitrary shell output and modified-file coordinates explicitly unattributed until their original-source mapping is verified; partial coverage must not be presented as a complete ContextBench score. Multi-file search observations must bind each complete visible match to its file and captured version; a shared line number/text is not file identity. Report scan/output caps and skipped files rather than implying full coverage. - ContextBench grep audits must validate the captured original reply and its actual model delivery against source-version receipts. Re-executing ripgrep is not a byte-for-byte oracle: traversal order and capped hit subsets may change on an unchanged checkout. Preserve recorded inputs and scores when a diagnostic rerun differs; do not replace observations with the rerun. - Preserve prospective benchmark tasks by reading raw dataset READMEs and metadata-only columns instead of HTML dataset viewers, which may automatically render gold contexts and reference patches. Record any exposed preview task IDs and exclude them from future unseen validation. Keep completed cohorts and their timing intact rather than retroactively changing their outcomes. - Before freezing a paid model experiment, verify the selected model's current thinking, tool-choice and output-limit constraints in official documentation. Requested CLI flags do not prove effective behavior. Distinguish main assistant models, auxiliary models in terminal usage, SDK turns and HTTP request counts. For source-only delivery checks, define accepted capped-response outcomes prospectively; do not change an earlier protocol's terminal-status or source credit rules after seeing its failures. - Prove a separately installed tool's imported modules from outside the analyzed checkout. `python -c` otherwise imports checkout source through its working directory, which does not establish the runtime serving the tool. Check the imported paths and complete selected module hashes before timing. ## Directory-Specific Rules Codex does not automatically load `.claude/CLAUDE.md`, but those files remain the authoritative local notes for several subtrees. Read the matching file before editing the listed area: | Path | Required extra context | | --- | --- | | `codenib/code_chunking/` | `codenib/code_chunking/CLAUDE.md` | | `codenib/graph/` | `codenib/graph/CLAUDE.md` | | `codenib/dataset/` | `codenib/dataset/CLAUDE.md` | | `scripts/agent_compile/` | `scripts/agent_compile/CLAUDE.md` | | `test/` | `test/CLAUDE.md` | The project-wide Claude notes in `.claude/CLAUDE.md` are useful background, but do not rely on Codex discovering them automatically.