benchmark git:20260531.0fd4b5fA

Immutable. This exact content is served forever at /api/v1/blob/6386e7a53a31e07f.

---
name: benchmark
description: Reference for the active benchmark harness — what it IS and how it behaves (the active benchmark is flashinfer-bench). Covers config.toml structure (`[solution]`/`[build]`/`[benchmark]` tables), the status enum, the workload / fresh-inputs model, reference-baseline + scoring mechanics, tolerance keys, TVM-FFI builder link limits, and the silent-skip cascade. Invoke whenever you decode a bench status string, hit an unfamiliar config.toml field, suspect "correctness passed but the headline is implausible", or need to know what's frozen before proposing a bench edit — do NOT guess field or status semantics. Bench commands and noise methodology (A/B compare, variance check, drift cancellation) live in the `bench` skill.
---

# Benchmark

The active benchmark harness — what this project's `bench` skill is a frontend for. Reference for **what the benchmark is and how it behaves** — distinct from the `bench` skill which covers **how to drive it from the harness shim**. The active benchmark is **flashinfer-bench**; this skill is the single home for its schema, status, scoring, and frozen-for-comparability behavior. Detailed body: `benchmark.md`.

When this skill applies:
- Looking up a `config.toml` field you don't recognize.
- Decoding a status string from bench output (`COMPILE_ERROR`, `INCORRECT_NUMERICAL`, etc.).
- Reasoning about why correctness "passed" but the headline looks too good (silent-skip cascade ends here).
- Checking what's frozen for bench comparability (scoring/baseline/tolerance) before proposing a harness edit.
- Understanding the TVM-FFI link limit for C++/CUDA (no compile-time linking) and which kernel libraries are disallowed (cuBLAS / cuDNN) vs. allowed building blocks (CUTLASS, the DSLs) — see "Valid solution: write your own kernel".