git:20260912.0ce18c0 to git:20260916.50d6810

- description: Measure command latency with hyperfine and HTTP throughput with oha, compare before and after, and report honest numbers. Use for performance comparisons, load tests, or a suspected slowdown.

- updated: "2026-09-11"

- Two tools, two questions. `hyperfine` answers "how long does this command take" with warmup, repeated runs, and a comparison; `oha` answers "how does this endpoint behave under load" with latency percentiles and a live TUI. Diagnosing why something is slow belongs to [systematic-debugging](../systematic-debugging/SKILL.md); this skill produces the numbers.

-

- ## Commands

-

- ```bash

- hyperfine --warmup 3 --runs 10 'old-cmd' 'new-cmd' # A/B with mean ± σ and a relative speed line

- hyperfine --warmup 3 --prepare 'uv sync --frozen' 'uv run --frozen pytest -q' # measure the test run, not uv's lock check

- hyperfine --parameter-list n 10,100,1000 'tool --items {n}' # scaling curve

- hyperfine --export-markdown bench.md --export-json bench.json 'cmd' # tables for the PR, raw data for later

- oha -z 30s -c 50 --latency-correction http://localhost:8080/health # 30 s, 50 connections, record the load model and correction setting

- oha -n 2000 -c 20 -m POST -H 'Content-Type: application/json' -d '{"q":1}' http://localhost:8080/api

- oha --no-tui -z 10s -c 10 --output-format json -o oha.json http://localhost:8080/ # scriptable output for CI or a report

- ```

- 1. **Fix the question**: one command or endpoint, one metric (mean latency, p99, requests per second), one hypothesis.

- 1. **Control the machine**: close heavy processes, run on AC power, and pin versions; record CPU, OS, and tool versions in the report.

- 1. **Warm up and repeat**: at least 3 warmup runs and 10 measured runs for commands; at least 30 seconds for endpoints. Compare against a baseline measured the same way in the same session.

- 1. **Quantify uncertainty**: compare repeated, equivalently controlled runs and report uncertainty in the difference; a run's standard deviation alone does not decide significance. Alternate baseline/candidate measurements when machine drift matters.

- 1. **Report**: the command lines, the exported table, the relative change, and the conditions. Keep `bench.json` if the number will be tracked over time.

- ## Gotchas

- - **Never load-test a remote service you do not own** or a production system without explicit approval; agree a load bound and stop condition appropriate to the target's capacity.

- - **Localhost numbers exclude the network**: `oha` against `localhost` measures the server, not the user experience.

- - **Shell startup pollutes short commands**: use `--shell=none` in hyperfine for sub-10 ms commands, or `-N`.

- - **Caches lie**: a second run of a build or query hits caches; use `--prepare` to clear them when the cold path is what matters.

- - **Cloud Run cold starts**: benchmark with `--min-instances` known, and separate first-request latency from steady state per the [cloud-run skill](../cloud-run/SKILL.md).

- ## Documentation

- - [hyperfine](https://github.com/sharkdp/hyperfine) · [oha](https://github.com/hatoo/oha)

- - Releases: [hyperfine](https://github.com/sharkdp/hyperfine/releases) · [oha](https://github.com/hatoo/oha/releases)

- - Companion skills: [quality-assurance](../quality-assurance/SKILL.md) (performance as part of a test campaign), [production-readiness](../production-readiness/SKILL.md) (capacity evidence before promotion).