mkl-bisect-regression ยท diff
git:20261006.ca56570 to git:20261016.a313256
116 added, 115 removed. Audit A to A.
---
name: "mkl-bisect-regression"
description: "Pinpoint regressions using automated git bisect test scripts with zero prompt token waste."
---
- # Pinpoint regressions with automated git bisect
-
- When a bug or performance regression appears between two releases or commits, never attempt to
- manually read git commit diffs across days or weeks of history. Reading hundreds of commits burns
- tens of thousands of LLM prompt tokens, causes hallucinated attributions, and exhausts the context
- window.
-
- Instead, leverage Git's logarithmic binary search: `git bisect run <test_script>`. The local shell
- automatically tests commits in $\mathcal{O}(\log_2 N)$ steps with zero intermediate tokens sent to
- the model. Once Git outputs the breaking commit hash, inspect only that single commit diff.
-
- ## 1. Author a standalone bisect test script
-
- Author an executable script (e.g., `bisect_test.sh`) in the repository root or temporary directory.
- The script must be completely non-interactive and return standard Git exit codes:
-
- - **Exit code `0` (Good)**: The commit is good (the bug or regression is absent).
- - **Exit code `1` through `124`, `126` through `255` (Bad)**: The commit is bad (regression observed).
- - **Exit code `125` (Skip)**: The commit cannot be tested (e.g. broken build, syntax error unrelated
- to the bug, missing dependency). Git will choose an adjacent commit without marking it good or bad.
-
- ### Flakiness and build mitigation in the script
-
- To prevent flaky tests or broken intermediate commits from misleading the binary search:
-
- 1. **Trap compilation/build failures**: Return `125` if prerequisites or compilation fail so Git skips
- the commit instead of misclassifying it as a bug.
- 2. **Loop flaky checks**: If the regression is non-deterministic or timing-sensitive, run the test 5
- to 10 times in a tight shell loop. If any run fails, exit with failure.
- 3. **Clean artifacts**: Remove transient cache files, bytecode (`.pyc`), or object files between steps.
-
- ```bash
- #!/usr/bin/env bash
- set -e
+ <skill name="mkl-bisect-regression">
+ <description>Pinpoint regressions using automated git bisect test scripts with zero prompt token waste.</description>
+ <context>
+ When a bug or performance regression appears between two releases or commits, never attempt to
+ manually read git commit diffs across days or weeks of history. Reading hundreds of commits burns
+ tens of thousands of LLM prompt tokens, causes hallucinated attributions, and exhausts the context
+ window.
- # 1. Verify build or dependency health; skip untestable commits
- if ! python3 -m py_compile $(git ls-files '*.py') >/dev/null 2>&1; then
- echo "[bisect] Intermediate syntax or build error; skipping commit."
- exit 125
- fi
+ Instead, leverage Git's logarithmic binary search: `git bisect run <test_script>`. The local shell
+ automatically tests commits in $\mathcal{O}(\log_2 N)$ steps with zero intermediate tokens sent to
+ the model. Once Git outputs the breaking commit hash, inspect only that single commit diff.
+ </context>
+ <workflow>
+ <step name="author-a-standalone-bisect-test-script">
+ Author an executable script (e.g., `bisect_test.sh`) in the repository root or temporary directory.
+ The script must be completely non-interactive and return standard Git exit codes:
- # 2. Run targeted reproducer test (repeat 3 times to rule out test flakiness)
- for i in {1..3}; do
- if ! python3 -m pytest tests/test_regression.py -q >/dev/null 2>&1; then
- echo "[bisect] Regression reproduced on attempt $i!"
- exit 1
- fi
- done
+ - **Exit code `0` (Good)**: The commit is good (the bug or regression is absent).
+ - **Exit code `1` through `124`, `126` through `255` (Bad)**: The commit is bad (regression observed).
+ - **Exit code `125` (Skip)**: The commit cannot be tested (e.g. broken build, syntax error unrelated
+ to the bug, missing dependency). Git will choose an adjacent commit without marking it good or bad.
- echo "[bisect] Regression absent; commit is clean."
- exit 0
- ```
+ ### Flakiness and build mitigation in the script
- Mark the script executable before launching bisect:
+ To prevent flaky tests or broken intermediate commits from misleading the binary search:
- ```bash
- chmod +x bisect_test.sh
- ```
+ 1. **Trap compilation/build failures**: Return `125` if prerequisites or compilation fail so Git skips
+ the commit instead of misclassifying it as a bug.
+ 2. **Loop flaky checks**: If the regression is non-deterministic or timing-sensitive, run the test 5
+ to 10 times in a tight shell loop. If any run fails, exit with failure.
+ 3. **Clean artifacts**: Remove transient cache files, bytecode (`.pyc`), or object files between steps.
- ## 2. Execute automated git bisect
+ ```bash
+ #!/usr/bin/env bash
+ set -e
- Identify the known good commit (or release tag) and the known bad commit (typically `HEAD`):
+ # 1. Verify build or dependency health; skip untestable commits
+ if ! python3 -m py_compile $(git ls-files '*.py') >/dev/null 2>&1; then
+ echo "[bisect] Intermediate syntax or build error; skipping commit."
+ exit 125
+ fi
- ```bash
- # Start bisect session specifying bad and good endpoints
- git bisect start <bad_commit> <good_commit>
+ # 2. Run targeted reproducer test (repeat 3 times to rule out test flakiness)
+ for i in {1..3}; do
+ if ! python3 -m pytest tests/test_regression.py -q >/dev/null 2>&1; then
+ echo "[bisect] Regression reproduced on attempt $i!"
+ exit 1
+ fi
+ done
- # Launch the automated runner
- git bisect run ./bisect_test.sh
- ```
+ echo "[bisect] Regression absent; commit is clean."
+ exit 0
+ ```
- During execution, Git checks out each pivot commit, runs `./bisect_test.sh`, and evaluates the exit
- code. It repeats until it pinpoints the boundary.
+ Mark the script executable before launching bisect:
- Git will terminate with a clear identification:
+ ```bash
+ chmod +x bisect_test.sh
+ ```
+ </step>
+ <step name="execute-automated-git-bisect">
+ Identify the known good commit (or release tag) and the known bad commit (typically `HEAD`):
- ```text
- <commit_sha> is the first bad commit
- commit <commit_sha>
- Author: Contributor <contributor@example.com>
- Date: ...
+ ```bash
+ # Start bisect session specifying bad and good endpoints
+ git bisect start <bad_commit> <good_commit>
- refactor: optimize cache eviction pipeline
- ```
+ # Launch the automated runner
+ git bisect run ./bisect_test.sh
+ ```
- Always end the bisect session after capturing the culprit hash to return your working branch to
- its original state:
+ During execution, Git checks out each pivot commit, runs `./bisect_test.sh`, and evaluates the exit
+ code. It repeats until it pinpoints the boundary.
- ```bash
- git bisect reset
- ```
+ Git will terminate with a clear identification:
- ## 3. Targeted culprit analysis
+ ```text
+ <commit_sha> is the first bad commit
+ commit <commit_sha>
+ Author: Contributor <contributor@example.com>
+ Date: ...
- Once the single culprit commit hash is identified, inspect only that commit's changes:
+ refactor: optimize cache eviction pipeline
+ ```
- ```bash
- # View affected files and diffstat
- git show --stat <commit_sha>
+ Always end the bisect session after capturing the culprit hash to return your working branch to
+ its original state:
- # View targeted diff
- git show <commit_sha>
- ```
+ ```bash
+ git bisect reset
+ ```
+ </step>
+ </workflow>
+ <section name="targeted-culprit-analysis">
+ Once the single culprit commit hash is identified, inspect only that commit's changes:
- Analyze the exact causal mechanism:
- - Which specific logic, type conversion, or assumption changed?
- - Was an invariant broken, a default parameter modified, or an edge case dropped?
- - Author a minimal regression test capturing the exact scenario before proposing a fix.
+ ```bash
+ # View affected files and diffstat
+ git show --stat <commit_sha>
- ## 4. Worked example: Bisecting a memory leak
+ # View targeted diff
+ git show <commit_sha>
+ ```
- Assume a benchmark detects that `worker.process_batch()` leaks memory somewhere between `v2.4.0`
- and `HEAD`:
+ Analyze the exact causal mechanism:
+ - Which specific logic, type conversion, or assumption changed?
+ - Was an invariant broken, a default parameter modified, or an edge case dropped?
+ - Author a minimal regression test capturing the exact scenario before proposing a fix.
+ </section>
+ <example type="authored">
+ Assume a benchmark detects that `worker.process_batch()` leaks memory somewhere between `v2.4.0`
+ and `HEAD`:
- 1. Write `test_leak.sh`:
+ 1. Write `test_leak.sh`:
- ```bash
- #!/usr/bin/env bash
- # Exit 125 if package installation fails
- pip install -e . -q >/dev/null 2>&1 || exit 125
+ ```bash
+ #!/usr/bin/env bash
+ # Exit 125 if package installation fails
+ pip install -e . -q >/dev/null 2>&1 || exit 125
- # Execute memory benchmark; exits with 0 if within budget, 1 if leaked
- python3 -c "
- import worker, tracemalloc
- tracemalloc.start()
- worker.process_batch(size=1000)
- current, peak = tracemalloc.get_traced_memory()
- tracemalloc.stop()
- if peak > 50 * 1024 * 1024:
- raise SystemExit(1)
- "
- ```
+ # Execute memory benchmark; exits with 0 if within budget, 1 if leaked
+ python3 -c "
+ import worker, tracemalloc
+ tracemalloc.start()
+ worker.process_batch(size=1000)
+ current, peak = tracemalloc.get_traced_memory()
+ tracemalloc.stop()
+ if peak > 50 * 1024 * 1024:
+ raise SystemExit(1)
+ "
+ ```
- 2. Run bisect:
+ 2. Run bisect:
- ```bash
- chmod +x test_leak.sh
- git bisect start HEAD v2.4.0
- git bisect run ./test_leak.sh
- git bisect reset
- ```
+ ```bash
+ chmod +x test_leak.sh
+ git bisect start HEAD v2.4.0
+ git bisect run ./test_leak.sh
+ git bisect reset
+ ```
- 3. Read the single culprit commit diff and resolve the regression.
+ 3. Read the single culprit commit diff and resolve the regression.
+ </example>
+ </skill>