
inferencekit
Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.
- version
- 0.1.1
- GitHub stars
- 32
Install
/plugin install inferencekit@inferencekitRun it inside Claude Code after adding the marketplace with /plugin marketplace add KittenML/inferencekit.
Contents
Discovered from sourceSkills 9
Benchmarking inference speed
Choose the workload before the metric: complete-request latency, sustained throughput, peak memory or cost at the agreed quality. Use the image/audio route in affordability and video checks for those modalities. Token metrics, text load grids and GSM8K below apply to text; other tasks need a registered adapter, with unsupported checks NOT RUN.
bench rounds --mode audio|image|clip runs fresh-process media comparisons; CPU manifests use hardware.backend: cpu for a CPU lock and process-group telemetry. Compare the final evidence ID for each engine in the run plan; media contracts and round scheduling.
Speed numbers that survive review: a timer outside the engine, one precision for every engine, settings proven in effect, warm engines, fresh-process rounds in a balanced order, confidence intervals, a speed-of-light ceiling and anti-cheat gates. optimizing-inference calls this skill.
Use for latency and tok/s per input bucket, capacity (N* within an SLO, goodput, $/1M tokens), comparisons of engines, configs or builds, and any reported number. A change-loop step needs only a screen; the full protocol is for milestones (fixed rounds: references/statistics.md) and anything published (optimizing-kernels, "Checks"). Not for why it is slow (profiling), nor while builds' outputs differ unchecked (proving-correctness first).
GPU approval: benchmarks, ceiling bw, --gpu-bw measure and gates run only with budget.approved: true in intake.yaml or an explicit yes after you stated the cost (GPU, $/h x hours). Record who approved and when.
Which timer
bench rounds / bench serve (MCP bench_rounds / bench_serve) | bench decode (MCP bench_decode) | |
|---|---|---|
| Timer | the client, time.perf_counter_ns, outside the engine process | inside the engine's process (engine-timed) |
| Engines | any OpenAI-compatible server: vllm serve, SGLang, llama-server, trtllm-serve, a custom engine | in-process vLLM LLM, SGLang Engine, llama-bench, or --engine cmd |
| Modes | single-user (sequential) and open-loop load | batch 1 only |
| Published? | yes: published numbers come only from here | no: label every number "engine-timed" |
bench rounds launches each manifest engine fresh per round, warms it up, runs bench serve once, stops it, checks GPU memory came back and parses the resolved config, in a Williams order under one GPU lock. Why one client timer: an engine that times itself chooses its own boundaries.
Preflight checklist
- Approval recorded;
inferencekit doctorthis session -> ev_env. - Nothing else on the GPU or host CPUs while timing (they shift latency by tens of percent); the client on pinned cores or another host.
- Engines and model pinned in
runs/<id>/manifest.json;inferencekit manifest validatepasses. - Same precision (weights, activations, KV, lm_head) for every engine, or a numerics change the task allows, labelled "numerics changed"; the same sampling and seed for every engine (
--extra-body: the workload's, elsegeneration_config.jsondefaults); each baseline at its best documented config for the goal. - Prompt set verified (
inferencekit prompts verify --set <dev|path>), identical for every engine. - Claim written first: metric, bucket, threshold, decision rule, round count R.
Hard failure gates
Any of these makes a record unusable for a speed claim; stop and report it:
- A setting not proven in effect from the resolved config or log, or CUDA graphs off where the engine supports them.
- Generated tokens per request differ from the requested
max_tokens, or token counts disagree between methods. - Repeated prompts inside a load level, or a prefix/response cache serving repeats.
- Precision, model revision, GPU type or prompt set differ between the engines compared.
- Fewer completed rounds than pre-registered, rounds not interleaved, or warmups counted. R is intake
workload.rounds(default 5, never below 3; kernel-loop milestones excepted), sized withn ~= (2.5c/h)^2and passed as--rounds R, or an adaptive plan pre-registered as--ci-target,--min-roundsand a maximum--rounds(references/statistics.md). - A profiler or debugger attached, or the GPU shared during timing.
- Any
inferencekit gates runFAIL: each gate catches a known way to improve the number without improving the engine.
Workflow (copyable)
Benchmark:
- [ ] 1. Approval recorded; claim and R pre-registered; doctor -> ev_env
- [ ] 2. Ceiling: ceiling bw -> ev_bw; roofline at each context you report -> ev_roof
- [ ] 3. Run manifest: per engine its launch line, env, port, ready_check; best documented config for the goal
- [ ] 4. Prove each knob took effect (resolved config / log line); copy it into manifest resolved_config
- [ ] 5. Single-user: bench rounds --mode single --rounds R (fresh server per cell, Williams order, warmup excluded)
- [ ] 6. Capacity (if a goal): bench rounds --mode load (grid, SLO, price, one shared --ref-e2e-ms) -> N*
- [ ] 7. Gates, per engine, while its server is up: relaunch it from the manifest, gates run, stop it -> ev_gates_<label>
- [ ] 8. Verdicts per bucket and metric: bench rounds' <metric>_ratio_to_best with its paired CI95 (see Comparisons)
- [ ] 9. % of ceiling per engine (derived); fill the fixed tables; inferencekit report
Exact commands
inferencekit doctor
inferencekit ceiling bw # -> ev_bw
inferencekit roofline --model <hf-id or config.json> --dtype fp8 --gpu-bw <ev_bw> --context <prompt + gen/2>
inferencekit prompts verify --set dev # prints path + sha256; or --set <path.jsonl> --expected-sha256 <pre-registered>
inferencekit manifest validate runs/<id>/manifest.json
# single-user. Engines start in the calling directory (a relaunch: the run dir), so with relative launch paths
# cd runs/<id> first and pass --manifest manifest.json
inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <set.jsonl> --rounds <R> --mode single
# capacity: open loop, seeded; --concurrency repeats; the same --ref-e2e-ms gives every engine the same schedule
inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <dev-load set.jsonl> --rounds <R_load> --mode load \
--serve-args "--concurrency 1 --concurrency 2 --concurrency 4 --concurrency 8 --concurrency 16 --concurrency 32 \
--concurrency 64 --ref-e2e-ms <ms> --seed 0 --slo-ttft-p95-ms 500 --slo-tpot-p95-ms 50 --gpu-price-usd-h <price>"
# gates: per engine, its server up (launched from the manifest line), then stopped
inferencekit gates run --base-url http://127.0.0.1:8000 --engine <label> --run-dir runs/<id> \
--reference-engine <baseline label> # + --spec-off-url <same model, no speculation> if it speculates
inferencekit report <ids...> --format md
# engines without a server only (engine-timed; never published as such):
inferencekit bench decode --engine cmd --cmd '<argv>' --model <m> --prompt-len 1024 --gen-len 256 --rounds 1 --extend <ev_prev>
inferencekit compare <ev_baseline> <ev_candidate> --metric decode_tok_s # same-kind round records
MCP tools: doctor, measure_bandwidth, roofline, prompts_verify, validate_manifest, bench_rounds, bench_serve, run_gates, report, bench_decode, compare (OpenCode: long steps through the CLI, detached). If bench rounds cannot launch an engine: a fresh server per call, bench serve --rounds 1 --engine-kind <vllm|sglang|llamacpp|trtllm> --label <label> --extend <its previous ev>, in the Williams order yourself (generic sends no exact-length field; the default --rounds passes share one process). Serving method, buckets, SLOs, N*, goodput and cost: references/serving.md.
Prove every knob took effect
Record each setting's requested value, where it was set and the value the engine resolved (config or log line; bench rounds writes resolved_config/<label>.r<k>.json, RESOLVED / PARTIAL / UNPROVEN), copy it into the manifest, and read every warning: a silently dropped setting once turned an A/B into an A/A (references/engine-configs.md).
Ceilings (speed of light)
At batch 1 each decode step reads all weights plus the KV cache, so time per token >= bytes per token / measured DRAM bandwidth. inferencekit ceiling bw measures it (quote the large-buffer read plateau, never a figure above the datasheet or a copy test); inferencekit roofline gives bytes per token. Numbers quoted from the early engine are engine-timed (rule 18). Batched, prefill and non-NVIDIA ceilings: references/ceilings.md; worked example: references/worked-example-qwen3-8b.md; hand-off cost: references/launch-cost.md.
Rounds, order and comparisons
- Each round starts every engine as a fresh process, warms it up and measures, in a balanced (Williams) order (period k for k engines when k is even, 2k when odd; pick R as a multiple). CIs are bootstrap over rounds, not requests.
- Served engines: verdicts come from
bench rounds'<metric>_ratio_to_best= median(engine) / median(best), with a paired bootstrap CI95, read withcompare's rule (improved | regressed | within_noise | inconclusive, pre-registered effect e, default 0.02; references/statistics.md, "Reading the result"). A claim about two engines, neither of them the best one, needs its ownbench rounds --engines <baseline>,<candidate>run. compareaccepts matching decode, single-user serve, clip, image or audio records. Load mode is refused. Media needs separate quality and lifecycle proof.
Anti-cheat gates (inferencekit gates run, MCP run_gates)
Per engine, against its live server plus the run directory: caching, token counts, output length, unique prompts under load, speculative/MTP exactness (--spec-off-url), hidden logprob and temperature probes, rare tokens. A live gate run after the server is gone is NOT RUN, not a pass (references/reward-hacks.md). Video: references/video-checks.md.
Other hardware (AMD ROCm, x86 CPU, Apple Silicon)
The protocol is unchanged; the differences (hardware name via --serve-args "--gpu-name MI300X --gpu-count 1", GPU-only gates on a CPU host, the memory check, a vendor-measured ceiling under hw/ with ceiling bw NOT RUN even if it starts on ROCm) are in hardware.md.
Fixed result tables
| Engine (label) @ version | Precision W / A / KV / lm_head | Resolved key settings (evidence) | Bucket | TTFT p50 / p95 ms | TPOT p50 / p95 ms | Decode tok/s median [CI95] | % of ceiling | Timer | Evidence |
|---|---|---|---|---|---|---|---|---|---|
| client / engine-timed | ev_... |
| Engine (serving config) | SLO (TTFT p95, TPOT p95) | N* | Goodput req/s at N* | Output tok/s per GPU at N* | $/1M output tokens at N* | Knee | Gates | Evidence |
|---|---|---|---|---|---|---|---|---|
| PASS / FAIL | ev_... |
| Pre-registered claim | Ratio [CI95] (source: bench rounds / compare) | Comparison verdict | Claim verdict | Evidence |
|---|---|---|---|---|
| PASS / FAIL / UNPROVEN / NOT RUN |
Decision and reporting rules
- No comparison across precisions; if engines share none, compare at BF16 and report quantised runs against their own ceilings. The one exception is a format trade-off (evaluating-quantization, "The cross-precision rule"): speed alone across precisions is never reported.
- A speedup needs a ratio whose paired CI95 over interleaved rounds excludes 1. A CI wider than the effect means more rounds, never a conclusion (UNPROVEN, not a tie).
- Above 100% of the ceiling: stop and find the error (token count, dtype, L2-resident weights). After an exact byte cut, recompute bytes per token for the new storage format first.
N*comes from open-loop runs with each engine's best serving config; closed loops hide overload. - Latencies, tok/s and
N*are observed (cite the record),compareratios too;ratio_to_bestwith its CI, % of ceiling, goodput and cost are derived (show the formula); any why is inferred until a profile shows it. bench decodenumbers carry "engine-timed" and never sit in a published table next to client-timed ones. Publish losing engines, refuted configurations and NOT RUN buckets too (reporting-perf-claims).
Evaluating quantization
Quantization can reduce memory and traffic, but conversion/dequantization can erase the speed benefit and outputs may change. This skill answers one question with evidence: on this GPU and this workload, which format is fastest at a quality loss the user accepts? The answer is a pair, speed and quality, measured on the same configuration. A faster number on its own is not a result.
Choose the modality before the checks. For image/audio, use affordability; for video, use video checks. Record precision per component, peak weights/activations/scratch memory and complete output work. Text token metrics and GSM8K below apply only to text. Other tasks need a registered quality adapter.
Reuse starting choices: balanced permits numerical changes with preserved task quality; speed permits stronger approximations within agreed bounds. Neither authorizes a different architecture. Search current implementations for the exact device and shapes before assuming a smaller format has a fast kernel.
Use when a format or published quant may be worth it, when fewer bytes per token would help (in any ceiling band; exact byte cuts belong to optimizing-kernels), when the model does not fit or capacity needs KV room, or when optimizing-inference's model.precision_allowed lists several formats. Not for same-precision engine comparisons, training a quantization recipe, or ranking different models.
GPU approval: steps 1-4 are GPU-free. Launching, bench rounds, gates and quality runs occupy a GPU: only with budget.approved: true in intake.yaml or an explicit yes to a stated cost (GPU, $/h x hours, cap), including a full-precision reference on a second, larger GPU.
For consumer VRAM/unified memory limits or minimum-memory baselines under latency and quality gates, also read affordability.
The cross-precision rule (read first)
Evidence rule 8 (evidence-rules.md) forbids comparing speed across precisions as an engine result. This skill allows it under three conditions:
- Speed and quality as a pair. Every format row carries its speed with CI and its quality verdict, measured on the exact speed configuration. A row without a quality verdict is NOT RUN on quality and cannot be recommended.
- Labelled
cross-precision: format trade-off, not an engine comparison. The speed ratio comes frombench rounds(<metric>_ratio_to_best, paired CI95) and is quoted as a ratio with its CI, never withimproved/regressed. - Same engine isolates the format (e.g. vLLM BF16 vs FP8 vs AWQ). A row that changes engine and format (e.g. llama.cpp Q4_K_M) is a deployment option, labelled with both changes, never used to explain the format effect.
Inputs (existing intake fields)
model.* (the full-precision reference; precision_allowed bounds the formats, others NOT RUN; kv_cache_dtype_allowed: [fp8] tests an FP8 KV cache), hardware.*, goals.*, buckets and workload.* (contexts and concurrency the KV must hold), budget.wall_clock_h (drop rows to fit, never rounds or quality), fidelity (a format below the stored one needs fidelity: lossy and its precision_allowed / kv_cache_dtype_allowed entry, in its own block: fidelity.md), quality.budget (replaces the tier defaults), quality.reference (a vLLM endpoint for the full-precision model; required when it does not fit the target GPU). constraints: quality margins: <tier> kl=<nats> top1=<fraction> gsm8k=<points> (tiers fp8, int4, gguf5-8, sub4), offload: allowed. No intake: ask for the same facts in one message. Margins are agreed before measuring; non_interactive takes the tier defaults (references/measure-and-score.md, section 4), logged in QUESTIONS.md.
Preflight checklist
- The claim written before measuring: metric, bucket(s), the speed threshold that makes a format worth it (e.g. "decode tok/s >= 1.3x BF16, medium bucket") and the quality margin per tier. Why: a threshold chosen after the data is a forking-paths result.
- The reference is the full-precision checkpoint at a pinned 40-hex sha, served by vLLM (scoring uses its
prompt_logprobs). Why: two quants agreeing with each other proves nothing. - Every candidate pinned to its own repo and 40-hex sha. Why: community quants are re-uploaded under the same name.
- The KV cache dtype decided per row and identical across rows unless it is what is tested. Why: an FP8 KV cache halves KV bytes on its own.
- Same prompt set,
max_tokens, chat template and temperature 0 for every row and the reference (speed at the workload's sampling).
Hard failure gates
Stop that row (or the study) and report the gate when:
- the resolved config does not show the format you think you run (dequantized to BF16 at load, an unplanned weight-only fallback kernel);
inferencekit gates runFAILs for the row: its speed is void;- the quality reference is missing: quality is UNPROVEN and no format may be recommended over the reference;
- a row FAILs its pre-registered margin: it stays in the table as a loss, never the recommendation;
- offload would be needed without
offload: allowedinconstraints: the row is NOT RUN (does not fit), never quietly offloaded.
Workflow (copyable)
Quantization check:
- [ ] 1. Claim, speed threshold and quality margins per tier written down (state.decisions or the plan)
- [ ] 2. Candidates: quant-scout -> repo@sha, format, size, provenance, engines (inferred), one table
- [ ] 3. Fit + ceiling per candidate, GPU-free: roofline --method headers (safetensors) or
scripts/gguf_bytes_per_token.py (GGUF) -> bytes/token, predicted ceiling ratio vs the reference, VRAM need
- [ ] 4. Shortlist <= 6 (engine, format) rows that fit and are worth timing; every dropped row gets a reason
- [ ] 5. Approval; launch each row, smoke test, prove format + KV dtype from the log; one manifest entry per row
- [ ] 6. Speed: one bench rounds call over all rows (reference row included) -> ratio_to_best + CI; gates per row
- [ ] 7. Quality: reference A/A trajectories; trajectory per row; quality score per margin tier; gsm8k per row,
paired with the reference
- [ ] 8. Pareto table, recommendation with its trade-off, labels, NOT RUN rows; claims-skeptic on the draft
Steps
1-2. Candidates (GPU-free). Run the quant-scout subagent once per model (Claude Code: inferencekit:quant-scout; elsewhere follow that file yourself): official checkpoints first, then the most used community quants, pinned, with recipe, size and engines (inferred, sourced). It never picks a winner. Formats: references/formats.md.
3. Fit and predicted ceiling (GPU-free). inferencekit roofline --model <repo> --revision <sha> --method headers --context <ctx> --kv-dtype <dtype> --gpu-bw <ev_bw or GB/s> per safetensors candidate at each bucket context; for GGUF, python3 scripts/gguf_bytes_per_token.py --repo <repo> --file <name.gguf> --revision <sha> plus the reference's kv_bytes (and state_bytes) at the same context. Fields, the Qwen3-8B worked example and the batch and compute-bound cases: references/bytes-and-fit.md.
- Predicted ceiling speed-up = reference bytes per token / candidate bytes per token (derived): an upper bound. 4-bit is never 4x fewer bytes: scales cost bits, lm_head and embeddings often stay BF16, and the KV cache is untouched and grows with context and concurrency.
- Fit = stored bytes (every expert, the full embedding table) + sequences x (KV at the context + recurrent state, from a roofline record run at that context; never slope x context on sliding-window or hybrid models) + runtime overhead. It is inferred until the engine logs its KV capacity in tokens.
- Shortlist 3-6 rows (reference included) that fit and clear the threshold with room to spare, same engine across formats first; dropped rows go in the report with the reason.
4-6. Measure speed (same harness, every row). Launch each row with the engine's recipe for that format, prefix caching off, CUDA graphs on, and prove from the startup log the quantization method, GEMM kernel family and KV dtype it resolved (a weight-only fallback is its own row). One manifest holds every row, each downloading its own pinned checkpoint (references/measure-and-score.md, sections 2-3). One bench rounds call interleaves all rows for paired ratio CIs; for capacity measure N* per row (KV room often decides it, and W4A16 can lose once compute-bound). inferencekit gates run per row while its server is up.
7. Score quality against the full-precision reference (proving-correctness; commands in references/measure-and-score.md, section 5): reference trajectories twice (A/A floor), one per row on its exact speed configuration, quality score per margin tier (below 4 bits: the 4-bit margins unless a sub4 margin was agreed first), quality gsm8k per row paired with the reference (reasoning often degrades before KL looks alarming), quality niah for FP8 KV or long-context claims. Without logprobs the trajectory is text only: top-1 and NLL are scored by re-tokenizing it, KL and TV are not; GSM8K still counts.
8. Report. The Pareto table (all rows, losses, misfits and NOT RUN included) and the recommendation: the fastest row by the conservative end of its speed CI whose quality is PASS on every pre-registered check and which fits. If no quantized row passes, recommend the reference precision; if the fastest passing row's ratio CI includes 1, the format has no proven speed benefit (state any memory benefit). One sentence with both numbers, labelled. Table, frontier rule and wording: references/measure-and-score.md, sections 6-7.
Big model on a small GPU (references/small-gpu.md): fit every candidate at the claim's context and concurrency; start with the least aggressive format; quality needs a costed vLLM reference on a larger GPU (none: quality UNPROVEN in the headline); offload only with offload: allowed, labelled and measured, never projected.
MCP tools: roofline, inspect_evidence, validate_manifest, bench_rounds, run_gates, quality_trajectory, quality_score, quality_gsm8k, quality_niah, report. Check flag spellings with inferencekit <command> --help.
Decision and reporting rules
- No recommendation without a quality verdict from the full-precision reference (the fastest format is the one that computes least). Quote the conservative end of each speed CI.
- Predicted ratios never stand in for measured ones, and never appear in the measured column. Do not extrapolate a result to another GPU, engine version, context, bucket or concurrency. Model-card quality claims are inferred context, never evidence.
- Bytes per token, predicted ratios and fit are derived or inferred; tok/s, KL, agreement and GSM8K are observed with evidence ids;
bench roundsratios are derived with their CI. - Verdicts PASS / FAIL / UNPROVEN / NOT RUN; a trade-off claim passes only when both its speed and quality halves pass. Publish the losing formats, misfits and NOT RUN rows with the winner (reporting-perf-claims).
Inference triage
Entry point. Turn a vague request into one measurable question, collect the cheapest evidence that answers it, then hand off to exactly one skill. The rules here apply to every other skill in this plugin.
Which skill
| The request | Skill |
|---|---|
| Current techniques, implementation research, a stalled optimization | researching-optimizations |
| "Make this model fast", "which engine should I use", "fastest config for my model on my GPU", a target (tok/s, TTFT, users per GPU, $/1M tokens) | optimizing-inference: the end-to-end flow |
A speed number: tok/s, TTFT, per-token latency, capacity N*, engine vs engine, before vs after, % of ceiling | benchmarking |
| Where the time goes; why one kernel is slow | profiling |
| Make an engine or kernel faster (custom engine, engine patch, decode or prefill) | optimizing-kernels |
| Is it still the same model; did quality regress | proving-correctness |
| Fit consumer memory while meeting latency and quality | optimizing-inference, affordability |
| Should I quantize; is FP8 / 4-bit / a GGUF quant faster at acceptable quality; which quant; will it fit my GPU; a big model on a small GPU | evaluating-quantization |
| Write up, review or audit a claim | reporting-perf-claims |
| Not sure | stay here: run the workflow below |
When to use / not use
Use when the right skill or the question is unclear ("why is it slow", "is X tok/s good", "did my change help"), or a number is about to be quoted unchecked against a ceiling. Not when the request clearly matches a row above (go there), or to rank different models by quality (model evaluation, not equivalence).
GPU approval (every skill)
GPU work (benchmarks, captures, ceiling bw, --gpu-bw measure, quality runs, renting) runs only with budget.approved: true in intake.yaml or an explicit yes after you stated the cost (GPU, $/h x hours, cap): it occupies shared hardware and spends money. Analysing existing evidence, roofline arithmetic with a numeric bandwidth, and reporting need no approval.
Explicit session authorization persists. An approved finite-deadline run may specify budget.spend_cap: none, stage_timeboxes: false and report_reserve_h: 0; these run-scoped overrides replace the respective defaults without repeated approval questions.
Getting a GPU
- Your own GPU, or one rented by the hour from any cloud GPU marketplace; never name or recommend a provider in shared files.
- Rent only after an explicit yes to the GPU model, $/h x hours and a cost cap, or within a pre-approved budget (end to end: optimizing-inference's opening questions; existing authorization persists).
- Need ncu counters? Prefer a VM or bare-metal offer. Containers and many cloud instances (A100 and H100 included) block them;
inferencekit doctorreports it (data.profiling_counters.status). If blocked: nsys only with ncu NOT RUN, then the PyTorch profiler (no counters needed) and the person's own profiles (profiling, "Fallbacks"). - Record GPU model, count, driver, CUDA and price/h (
inferencekit doctor, and the price in the intake'shardware.price_usd_h). - Keep the provider's credentials out of commands, logs and profiles (capture only through inferencekit, under
env -i). - Copy
runs/back, destroy the instance after use (not just stop it), and confirm none is left running.
Not NVIDIA? (AMD ROCm, x86 CPU, Apple Silicon)
NVIDIA is fully supported, and the only hardware inferencekit has been run on so far. On other hardware, the skills guide the same flow, and the CLI does less: references/hardware.md has the table.
- Hardware-neutral, not yet run off NVIDIA:
bench rounds,bench serve,gates runand quality (they talk HTTP to any OpenAI-compatible server),rooflinewith a numeric--gpu-bw, and the intake, manifest, ledger and report tools. Treat a first run there as a test. - NOT RUN off NVIDIA:
ceiling bwand nsys/ncu captures.doctorrecords selected AMD DRM/rocminfo and Apple sysctl/system_profiler facts with a vendor profiling route; unknown details remain unknown. On AMD,rocprof captureandrocprof analyzecreate trace evidence with rocprofv3; device/version support must be checked. Use current official docs for unfamiliar tools and a vendor or STREAM-style bandwidth test. Captures made outside the CLI are vendor evidence files. - Platform guides: AMD ROCm, x86 CPU, Apple Silicon, Intel GPU and TPU.
Preflight checklist
- One sentence names the workload: model id + revision, precision (weights / activations / KV cache), GPU, input bucket(s), output length, and the goal (single-user latency or capacity under an SLO). Why: a number without its workload cannot be reproduced or compared.
-
inferencekit doctorhas run on the target machine this session (not NVIDIA: plus the hardware card from references/hardware.md). Why: driver, clocks, counter permissions and engine versions decide which captures are possible. - You know whether anything else uses the GPU or host CPUs. Why: shared GPUs and busy "spare" CPUs silently slow timed runs.
- The goal is a falsifiable claim (e.g. "candidate decode tok/s >= 1.10x baseline, medium bucket" or "N* >= 32 at p95 TTFT <= 300 ms"). Why: a claim chosen after the data is a forking-paths result.
Hard failure gates
Stop and report the gate instead of continuing when:
inferencekit doctorshows no usable GPU, a mismatched driver/CUDA pair, or an unknown engine version. On AMD, CPU or Apple hardware, empty GPU facts are expected: continue with references/hardware.md instead.- The two sides of a comparison differ in precision, model revision, GPU type or workload.
- A secret (API key, token) would end up in a capture, log or report.
- GPU-occupying work is about to run without recorded approval.
Workflow (copyable)
Triage:
- [ ] 1. Restate the request as one question + one pre-registered claim
- [ ] 2. End-to-end "make it fast" request? -> optimizing-inference, done
- [ ] 3. inferencekit doctor -> ev_env
- [ ] 4. inferencekit roofline -> bytes/token and ceiling tok/s (no GPU needed with a numeric --gpu-bw)
- [ ] 5. Existing evidence first: inferencekit evidence list; profiles the person already has
- [ ] 6. % of ceiling = measured tok/s / ceiling tok/s (derived), if a measurement exists
- [ ] 7. Pick ONE skill from the symptom table; say why in one line
- [ ] 8. Hand off with: workload sentence, evidence ids, the claim, what is still NOT RUN
Exact commands
| Step | CLI | MCP tool |
|---|---|---|
| Environment card | inferencekit doctor | doctor |
| Ceiling (arithmetic only) | inferencekit roofline --model Qwen/Qwen3-8B-FP8 --dtype fp8 --gpu-bw <measured GB/s or ev_bw> | roofline |
| Existing evidence | inferencekit evidence list / inferencekit evidence show <id> | list_evidence / inspect_evidence |
| Intake for the end-to-end flow | inferencekit intake template / inferencekit intake validate intake.yaml | intake_template / intake_validate |
| Report | inferencekit report <ids...> --format md | report |
| Analyse a profile the person supplied (no GPU) | inferencekit nsys analyze <file> / inferencekit ncu summarize <file> --peak-gbs <ev or GB/s> | analyze_nsys / summarize_ncu |
MCP clients prefix tool names (Claude Code: mcp__plugin_inferencekit_inferencekit__roofline); the arguments are the CLI's. MCP server does not start (no uvx, no network): the CLI at this plugin's version, uv tool install "inferencekit-cli==0.1.1". In OpenCode, long steps (bench rounds, gates run, quality runs, ceiling bw, captures) go through the CLI, detached with a log: it ends an MCP call after the server's timeout (120 s), and the server sends no progress. Bandwidth: your GPU's measured inferencekit ceiling bw (off NVIDIA, the platform's micro-benchmark: references/hardware.md), or say the ceiling is datasheet-based and therefore unreachable.
Symptom routing
| Symptom or question | Cheapest first evidence | Hand off to |
|---|---|---|
| "Make it fast", "which engine", a target to hit | intake | optimizing-inference |
| "How fast could this possibly go?" | roofline with a known bandwidth | benchmarking (ceilings) |
| "What tok/s / TTFT do we get?", engine vs engine | doctor + roofline | benchmarking |
| "How many users per GPU?", p95 latency rises under load, cost per token | the SLO and traffic shape | benchmarking (capacity, N*) |
| Below ~85% of the ceiling (bands), cause unknown | one nsys capture of the steady step | profiling |
"I already have a profile" (.nsys-rep, .sqlite, .ncu-rep) | inferencekit nsys analyze <file> / inferencekit ncu summarize <file> (no GPU, no approval) | profiling |
"I have a PyTorch profiler trace" (*.json.gz from vLLM, SGLang or torch.profiler) | read by hand or with profile-analyst: the CLI does not parse it (no GPU, no approval) | profiling |
| GPU idle between kernels, high kernel count | nsys per-step busy/idle | profiling, then optimizing-kernels |
| One kernel or family dominates the step | nsys critical-path shares | profiling (ncu) |
| "Make this engine/kernel faster" with evidence in hand | ceiling + profile | optimizing-kernels |
| TTFT slow (prefill) | TTFT per bucket, first-seen vs repeated length | benchmarking, then optimizing-kernels (prefill) |
| Outputs changed; before any speedup claim | trajectory score vs the noise floor | proving-correctness |
| "Should I quantize?", "is 4-bit faster?", "which quant?", "will it fit?", near the ceiling and only fewer bytes can help | roofline --method headers per checkpoint (no GPU): bytes/token and fit | evaluating-quantization |
| "Did my change help?" / regression hunt | interleaved A/B (bench rounds; compare for bench decode) | benchmarking, then reporting-perf-claims |
| Writing up a result, PR, blog post | the evidence ids | reporting-perf-claims |
Ceiling bands
One scheme for every skill in this plugin: % of a measured-bandwidth ceiling, for single-user decode. These are heuristics that pick the next measurement, not verdicts.
- Above 100%: impossible; the ceiling, the bytes per token or the token count is wrong.
- ~85% or more: near the floor. Only reading fewer bytes per token helps (an exact byte cut, or a lower precision: evaluating-quantization).
- ~50% to ~85%: look for launch gaps and under-filled kernels (profiling, then optimizing-kernels).
- Below ~50%: suspect eager mode, host syncs or a CPU-bound loop.
- In every band the ceiling can move (exact byte cuts, allowed speculation): estimate both at the start (optimizing-inference's headroom estimates) and work them alongside kernel fixes.
The table with the next evidence for each band: references/symptom-map.md.
Artifact/evidence contract
- Every inferencekit operation writes an evidence record (schema v2; v1 records still read) and returns its id. Why: the id is the citation; a number without one is hearsay.
- Read
identity,coverage.complete,coverage.missingandlimitationsbefore quoting any metric. Why: a record can be valid and still not cover your claim. - Never paste raw environment values; inferencekit stores none, and you must not add them.
- Global rules with their reasons: references/evidence-rules.md (the numbered rules are also in the repository's
AGENTS.md).
Fixed result table
| Fact | Value | Class | Evidence id | Note |
|---|---|---|---|---|
| Workload and goal | e.g. Qwen3-8B FP8 W / BF16 KV, medium bucket, single-user | observed | ev_... | from identity.workload |
| GPU, driver, clocks | observed | ev_... | doctor | |
| Ceiling tok/s | derived | ev_... | bytes/token and bandwidth source | |
| Measured value | ... or NOT RUN | observed | ev_... | statistic, n |
| % of ceiling | derived | (both ids) | formula shown | |
| Next skill | inferred | — | one-line reason |
Decision and no-go rules
- Measure before optimising. Why: most "obvious" bottlenecks are not where the time goes.
- One skill at a time. Why: profiling during benchmarking contaminates both.
- No speedup claim without a quality verdict. Why: the fastest kernel is the one that skips work.
- Measured above the ceiling: stop; the bytes/token model, the bandwidth or the token count is wrong.
- Datasheet bandwidth is a label, not a ceiling.
Reporting rules
- Label every number observed (a tool measured it; cite the evidence id), derived (arithmetic; show the formula) or inferred (reasoning; say what would confirm it).
- Claim verdicts are PASS / FAIL / UNPROVEN / NOT RUN; missing data is UNPROVEN or NOT RUN, never zero and never PASS. Comparison verdicts come from the tools (
bench roundsratio CIs for served engines,inferencekit compareforbench decode), never from eyeballed medians.
Optimize inference
Deliver reproducible code and evidence against the strongest tuned, supported baseline under the agreed quality policy. Record time, tokens/API cost and device cost. Unsupported checks are NOT RUN; missing proof is UNPROVEN.
Rules throughout the run
- Reuse the unmodified inferencekit harness and datasets. A missing modality adapter is a recorded limitation; do not disguise a custom timer as harness evidence.
- Fix input distribution, output work, sampling/seeds, component precisions and quality protocol before comparing. Treat baseline and candidate symmetrically. Numerical changes follow fidelity; cross-precision results disclose both formats and quality.
- Launch pinned implementations to establish support. Verify each knob in the resolved runtime config; an ignored flag is a finding.
- Keep every attempt, including failures, in the state and ledger. Label numbers observed, derived or inferred and claims PASS / FAIL / UNPROVEN / NOT RUN.
- Check
inferencekit intake budget runs/<run-id>before long steps. Preserve existing spending authorization; never expand it silently. Keep secrets, access details and provider names out of artifacts. - For text, keep prefix caching off in the cold-request track; never use
--no-cache-salt. Reuse needs a separate realistic workload. Speculation needs its own exactness proof when called lossless. - Reuse existing engine features, compiler output, harness and resource supervisor before adding code. Run the smallest experiment that can disprove a hypothesis. Avoid unused helpers, headers, configuration and abstractions. Quick screens guide exploration; complete proof belongs to finalists.
0. A few questions, then work
Use starting questions. Reuse answers in the conversation and intake; ask only missing decisions in short batches. Do not require technical metrics, YAML or a plan from the person.
Quality has three choices: Exactly the same, Same quality, small differences, Prioritize speed. Separately establish permission to change the model itself, the main objective, spending, deadline and desired check-ins. A speed preference does not authorize architecture changes or unchecked quality loss.
No plan or plan approval is required by default. Show or create one only when requested; plan.md covers that optional path and legacy runs. Start once required choices and resource authorization are established. Record their source in state; do not ask again within that scope. For unattended runs use non-interactive.md.
Start or resume
Existing run: first call inferencekit intake resume runs/<run-id>; inspect files for reported issues or required details (state.md).
Fresh run:
inferencekit intake template > intake.yaml # only when no intake was supplied
inferencekit intake validate intake.yaml --allow-placeholders
inferencekit intake validate intake.yaml --run-dir runs/<run-id>
export INFERENCEKIT_RUNS_DIR="$PWD/runs/<run-id>/evidence"
inferencekit intake budget runs/<run-id>
Use the derived run id; --run-dir pins the intake and starts the clock. SSH runtime and client share the device host.
1. Define the actual task
Validate intake; pin revision, component precisions, processing, dataset, shapes and output work. Derive workload details from examples/logs (workload.md); record assumptions.
Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit (affordability.md). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text.
Use researching-optimizations now, from the pinned model, shapes, hardware and runtime. It returns a short experiment queue, including existing engine paths to try before custom code.
2. Hardware, resource limits and headroom
Run inferencekit doctor. Check competing work, memory, swap, thermals and profiler permissions before occupying the device. Stop only the run's own jobs; do not kill unrelated services to obtain a clean measurement.
On NVIDIA, measured ceiling bw and roofline inform supported text models. Elsewhere use the hardware route and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN.
Estimate critical-path share, attainable efficiency, exact byte savings and permitted algorithmic savings (ceilings.md, stop-point.md). Recompute after changing precision or work. An inferred ceiling orders experiments; it cannot establish a speedup.
3. Establish the strongest baseline
Search the web at the start of each run for current engines and recipes for the exact model, hardware and goal; refresh when these or the runtime change. candidates.md gives examples, not a fixed list or count. Launch compatible engines in isolated environments; record sources, versions, commands, resolved configs and smoke results. NVIDIA graphs are not a generic launch flag for AMD or Apple.
Tune supported engines for the goal. A task-fixed baseline is labelled fixed, never automatically best in the field. Reconsider numerical-only baseline rejects under a newly authorized lossy policy.
For text, a fixed baseline with route build may skip the survey and use the engine skeleton. Other models use their native pipeline and custom-engine contract, not the text skeleton.
4. Screen and measure
Use harness-commands.md for text, affordability.md for image/audio and video.md for video. Screen representative cases before full rounds. Text screens include short and long lengths. Promote secondary wins the user values without hiding the lead metric.
Final comparisons use balanced fresh-process rounds, excluded warmup, registered stopping and confidence intervals. Include complete output work and transfers/synchronization/decoding. Capacity needs its own tuning and measured concurrency.
5. Quality on the exact speed configuration
Load proving-correctness and the selected policy. Exact-output claims need equality on the declared deterministic/seeded contract; a lossless label alone does not prove it. Lossy tracks need agreed task/perceptual bounds, not accidental bitwise rejection. Preserve structural validity, finite outputs and completion in every track.
For text, run applicable gates, reference trajectory/score, full task evaluation and sampling checks. Image/audio/video use registered semantic, structural and perceptual checks instead of token gates or GSM8K. Missing human listening/viewing remains UNPROVEN. Approval frequency changes when to ask, never whether quality has been established.
6. Choose the next experiment
Rank experiments by headroom, cost and confidence. Refresh research after two stalled attempts or changed bottleneck/toolchain. optimizing-kernels routes compiler inspection and conditional techniques.
At a met target follow goals.after_target. Otherwise continue within the approved scope and stop rule. With no target, honor the agreed check-in preference; do not repeatedly request approval for an already authorized search.
Default stage time boxes and reporting reserve come from intake. Explicit budget.stage_timeboxes: false or report_reserve_h: 0 override them. A finite deadline still binds. "Unreachable" and "no headroom" are verdicts, never stops under a stated deadline; pursue another useful lever while resources remain. Unavailable devices, evidence or permission are recorded honestly.
7. Implement and investigate
Use compiler inspection: source → graph → dispatch → executed kernel. Check fusion/fallbacks/conversions, form one hypothesis, change one cause, screen quality and measure the complete inference path. Inspect IR/assembly when useful.
Run independent implementation experiments in parallel under parallel.md. Integrate one candidate at a time; serialize device measurements and avoid host contention. Reuse previous numeric-only rejects when policy broadens, with new evaluation and preserved history (fidelity.md).
Keep a commit and screen evidence per retained change. Milestone/final checks follow state.md: text uses existing light/full jobs; other modalities use registered checks. Final proof applies to the commit you hand over. No full evaluation after every small edit; no published claim from a screen alone.
At freeze, one claims-skeptic reviews draft, diff, intake and evidence while final proof runs. Answer refutations with evidence. With zero reserve, keep validation and packaging current through the deadline.
8. Final report (always runs, from the report reserve, not before)
Use the shared report format and final reply: Result, Run it, Files and outputs, then Details. The plain-English summary requirements are in stop-point.md, "Summary". Separate end-to-end from kernel gains, exact from cross-precision tracks, and screened from verified results. Keep material limits visible in Result; retain unsuccessful attempts and time/cost accounting in Details.
Keep the manifest, reproduce commands, ledger and report consistent with the final commit. Run inferencekit review build runs/<run-id> and inferencekit manifest validate runs/<run-id>/manifest.json --strict. No freeze (no stage 7): one claims-skeptic on the draft before the host is released. Push only within explicit authorization. Record final spend and release resources as authorized.
After the final report, check the result for an optional engine contribution. Prepare a qualifying contribution locally, then open its PR when the user authorizes publication; reuse an explicit opt-in already recorded for this run. A declined contribution stays local. Missing proof or publication access does not restart optimization or hold rented hardware open.
Parallel work
Shipped read-only agents: engine-scout, quant-scout, profile-analyst, claims-skeptic, log-reader. Implementation workers may write kernels concurrently in isolated files/branches. One coordinator integrates; one owner per device serializes measurements. See parallel.md.
Optimising kernels: the profile-driven change loop
Start from the measured critical path: host overhead, synchronization, poor memory/compute utilization, redundant traffic or repeated work. These losses occur in text, vision, audio and other models. This skill closes them one change at a time, each checked fast and kept as a revertible commit with its numbers; the full proof runs at milestones (optimizing-inference stage 7).
When to use / not use
Use when:
- you have a ceiling, a benchmark and a profile, and want to close the gap (decode, prefill/TTFT, or the batched step);
- you are building a custom engine, patching an engine, or reviewing a kernel change.
Do not use when:
- you have not measured yet (inference-triage first: most guesses about the bottleneck are wrong);
- an engine flag might close the gap: tune engines first (benchmarking, engine-configs).
Authorization and parallel implementation
Reuse the run's starting choices, including quality and model-change permissions. Ask missing choices only when this skill is invoked alone. Device work stays inside existing authorization; inferencekit intake budget runs/<run-id> checks remaining resources, not a requirement to ask again.
Independent workers may write kernels in parallel in distinct files/branches. One coordinator integrates them sequentially, and one owner per device serializes warmup, captures, quality and timing. CPU-heavy builds on the benchmark host wait for measurements to finish (parallel.md).
Hardware and workload
Start with doctor and the hardware route. Use Nsight on NVIDIA, supported ROCm tracing/counters on AMD, Instruments/Metal/MLX capture on Apple, or a CPU profiler. For an unfamiliar architecture, runtime, flag or error, retrieve current official docs and verify installed support before selecting a tool. Keep unsupported checks NOT RUN; preserve available timeline or operator attribution with its tool version and source artifact, following profiling.
The CUDA skeleton, commands and kernel recipes below are NVIDIA examples. Use the target engine's native backend elsewhere; a CUDA API name in PyTorch does not establish CUDA-tool support on ROCm. For images and speech, use the registered media adapter, structural and semantic checks, and complete request timing in affordability. Token checks, GSM8K and the ikstep LLM jobs below do not prove media correctness. Keep the same fast-check, milestone and full-proof stages with the workload's own checks.
When the user authorizes numerical changes, judge image/speech kernels against the declared modality quality budget and hash-bound external protocol. Different output bytes alone do not reject that track; exact identity remains a pure-refactor requirement or a diagnostic. Each intermediate change gets one short fixed public correctness/quality screen; promoted milestones and the final candidate need full registered quality and balanced timing. Preserve own-output integrity and all cases. Do not infer permission to prune architecture or use another model from a lossy budget.
An explicit perceptual/task acceptance policy may make numerical similarity diagnostic as well. In that case, do not discard a kernel or baseline solely for LPIPS/SSIM, cross-engine tensor/PCM equality or rounding drift. Measure the prospectively chosen mathematical controls and task/perceptual checks; fresh blinded quality is still required for promotion. No missing perceptual evidence becomes PASS.
Research and inspect before implementing
Load researching-optimizations before selecting custom work and again when progress stalls or the bottleneck moves. Check existing engine features and emitted compiler code first. Follow compiler inspection from source through graph and dispatch to the executed kernel; inspect IR/assembly when it can test a specific hypothesis.
Use the smallest useful experiment, reuse the harness and supervisor, and keep the old path runnable. Avoid unused helpers, headers and abstractions. Independent experiments can be written concurrently; each integrated A/B isolates one change.
Preflight checklist
- Ceiling and % of ceiling known (ev_roof, ev_bench). Why: the size of the prize and when to stop.
- A profile of the steady step with critical-path shares (ev_tl, profiling). Why: tells you which of the four losses dominates.
- A correctness baseline: reference outputs and the A/A noise floor (proving-correctness). Why: every change below can silently break numerics.
- A harness that A/Bs two builds in interleaved fresh-process rounds. Why: the end-to-end effect is the only one that counts.
- Record build versus patch and both headroom bounds (stop-point.md). Any new engine meets the custom-engine contract; the bundled skeleton covers NVIDIA text only.
- A repo and branch for the ledger (
perf/<model>-<hardware>, from the intake'srepo), andruns/<run-id>/experiments.jsonl. Why: every kept change becomes a revertible checkpoint; nothing tried is lost.
Build route: NVIDIA text engine skeleton
The plugin ships a correctness-first engine (skeleton/README.md) that already meets the custom-engine contract: an OpenAI streaming server, seeded sampling, top-20 logprobs from the full output layer, a checkpoint loader, a CUDA-graph decode loop and torch modules per architecture.
- Copy
skeleton/ikengine/(next to this file;${CLAUDE_PLUGIN_ROOT}/skills/optimizing-kernels/when installed) into the run directory and commit it as the first ledger commit. - Run its reference server on the model (
python -m ikengine.serve --model <dir> --port <port>) and validate it as its README says: its model check, the gates, and quality against the reference. It is your correctness anchor and A/A partner. No module for the architecture: write one against theModelcontract inikengine/engine_api.py, correct first. - Replace internals behind the
Modelcontract one change at a time (the loop below): kernels, fusion, byte cuts. Keep the server, the sampler, the logprob path and the proofs. - Never ship or claim a build without its proofs (below).
Checks and implementation loop
- Profile the supported platform once per lever area; repeat when the bottleneck moves. Identify the critical-path limiter, not summed kernel duration.
- Read compiler/dispatch evidence and current research. Choose one hypothesis with a headroom bound and a falsifying test.
- Implement behind a switch. Keep the old path runnable; independent alternatives may be written concurrently.
- Run one fast screen on representative inputs, including awkward shapes. Exact changes preserve the declared outputs. Legacy lossless text changes use token/logprob checks with near-tie analysis; authorized lossy changes use their task/perceptual screen. Compare against a fixed reference as well as the previous build to catch cumulative drift.
- Screen the complete step/request using the existing timer. Keep a reversible commit and numbers, or log the rejection. A faster microkernel alone is insufficient.
- Promote promising milestones with the state.md schedule: text uses
ikstep job light; other modalities use registered checks. Baseline proof is reused until its config changes. - Run full proof once for the final candidate and any commit handed over (
ikstep job fullfor text). A claimed change needs interleaved A/B confidence intervals and a profile showing the mechanism. Missing reference: use the no-reference route. - Follow the approved stop rule and
goals.after_target. A stated deadline: work to it minus the report reserve. Explicit zero reserve means continuous validation/packaging. Log the next untried lever and why.
Do not run full gates, task evaluations, balanced rounds or a new profile after every edit. Exploration screens never become published proof. Final timing includes the complete client request; label step/microkernel measurements separately.
Commit trailers and retained/rejected experiment records: commit-ledger.md. Use the platform profiler and registered client timer; text CLI examples are in harness-commands.md.
Choose a technique from evidence
These apply across model families; select by the measured problem, not a list of fashionable names. For each, estimate headroom, confirm the mechanism, screen quality, then time the whole request.
| Observed problem | NVIDIA | AMD | Apple | Tradeoff / proof |
|---|---|---|---|---|
| CPU launches and tiny kernels | CUDA Graphs, fusion, device sampling | Supported HIP graph/runtime capture, Triton/HIP fusion | Reuse Metal command submission, MLX compiled/fused paths | Verify fewer gaps; include dispatch and synchronization. Sampling applies only where the workload samples. |
| Repeated intermediate traffic | Producer-consumer fusion, layout/producer quantization | HIP/Triton/CK layouts and fused epilogues | Metal/MLX fusion, layouts and quantized producers | Measure bytes and end-to-end cost; quantization needs permission. Fusion can increase registers. |
| Poor matrix utilization | Shape-specific GEMM/GEMV, tiling, split-K, supported matrix instructions | gfx-specific MFMA/WMMA, CK/Triton/other supported compiler | Metal SIMD-group matrices or native MLX kernels | Include packing/reduction overhead and shape tails; matrix paths are generation/dtype specific. |
| Synchronization dominates | Persistent kernels or megakernels | Supported persistent HIP kernels with explicit synchronization | Supported persistent/threadgroup design; do not assume grid-wide barriers | Check residency, occupancy, deadlock hazards and actual saved critical path. HazyResearch examples are hardware-specific research. |
| Model pages or barely fits | Packing, cache sizing, expert residency, staged loading/offload | Same memory strategies with supported allocator/transfer paths | Unified-memory working set, packing, cache limits and staged loading | Peak host/device memory, swap, complete latency and quality; fitting alone is not a latency win. |
| Repeated computation | Validated state/prefix reuse or incremental updates | Same algorithm with native runtime state | Same algorithm with native runtime state | Reset/invalidation across requests and changing inputs; approximate reuse needs quality permission and a separate realistic workload. |
See research sources for current implementations. CPU routes use the same hypotheses with vectorization, threading and cache-aware libraries.
NVIDIA text playbook: decode (memory-bound; batch 1 examples)
| Technique | Loss it closes | Mechanism evidence | Deep dive |
|---|---|---|---|
| CUDA graph for the whole step | host launch overhead | nsys: one graph launch per step, idle gaps gone | launch-overhead.md |
| GPU-side sampling, no per-token host sync | CPU on the critical path | nsys: no blocking D2H before the next launch | launch-overhead.md |
| Fusion: norm+RoPE+KV write, one QKV GEMV, gate-up with SiLU*up epilogue, residual+norm, act-quant into the producer | hand-offs (kernel count) | nsys: kernels per step down, step wall down | launch-overhead.md |
| FP8 block-scaled GEMV at M=1 (many blocks, 16-byte loads, per-block scales) | library GEMM tiles that cannot fill the GPU at M=1 | ncu: DRAM % of measured peak up, waves >= several | fp8-gemv.md |
| Split-K decode attention (flash-decoding) | too few blocks at batch 1, long context | ncu: waves up; nsys: attention's critical share down | attention-split-k.md |
| Programmatic dependent launch, trigger placed late | remaining hand-off between dependent kernels | step wall down in A/B; overlap between kernels shrinks | launch-overhead.md |
| Cheaper kernel prologue (hoist weight loads before the PDL wait, stage inputs asynchronously) | fixed per-kernel setup before streaming starts | prologue-only variant timing; in-step time above the bandwidth ideal | lessons.md |
| lm_head: precision, exact bound pruning (below), fusion with sampling | output-layer bytes measured for this checkpoint | bytes/token down; quality verdict at the logits | fp8-gemv.md, byte-cuts.md |
| Exact byte cuts: lossless re-encoding of 16-bit weights, bound-pruned output layer | bytes you did not need to read (moves the ceiling) | bytes/token down; proof tests pass; % of the recomputed ceiling | byte-cuts.md |
| MoE: gather-GEMV over the top-k experts, routing fused into the first expert kernel, top-k-order sum | library grouped-GEMM tiles and many helper kernels per MoE layer | nsys: kernels per layer down; ncu: DRAM % of measured peak up | moe-decode.md |
| Persistent kernel / megakernel | whatever hand-off is left | measured hand-off sum first | launch-overhead.md |
Why fusion helps at batch 1: activations are tiny (a 4,096-wide BF16 vector is 8 KB), so fusion saves almost no bytes. It saves kernels, and each dependent kernel costs a hand-off of the order of a microsecond or more.
Exact byte cuts keep the model identical (lossless track, no precision decision) and move the ceiling: estimate the exact byte floor at the start (ceilings.md, "Exact byte floor") and work them alongside kernels. Re-encoded weights store out-of-window elements as raw 16-bit values; a pruned head serves logprob requests from the full head. Each needs its proof recipe and a recomputed ceiling, and a rare_tokens PASS at the next milestone (byte-cuts.md); a lossy truncated vocabulary stays a hack.
NVIDIA text playbook: prefill / TTFT (compute-bound for long prompts, host-bound for short ones)
For short prompts investigate launch overhead; for long prompts inspect matrix utilization and neighboring elementwise work. NVIDIA text examples and traps: prefill.md.
Lessons that cost people weeks
- Tune and screen inside the real step. A configuration tuned in isolation (warm or flushed L2, no neighbours, no PDL overlap) can lose in context. Re-tune and rank variants by whole-step CUDA-graph replays (seconds); only the best get a
bench servescreen, and only claimed ones the A/B; a wait loop stops when the server dies. - Amdahl first, on the critical path. A 2x kernel at 4% of the step is ~1.02x; under PDL, use critical-path shares, not summed durations.
- Counters suggest; timing decides. ncu pointed at bytes in flight; a pipeline-depth sweep refuted it; the prologue was the cost.
- A faster kernel can slow its neighbours. A fusion that saved critical-path time in its own slot gained almost nothing end to end, because the kernels after it slowed by a similar amount. Only the whole-step A/B tells. More, with sources: references/lessons.md.
Artifact/evidence contract
Keep the candidate check, screen, commit or rejection, and experiment id. A claimed gain additionally needs interleaved timing, a comparison CI and mechanism evidence from the supported platform profiler. Preserve tool versions and source hashes; never invent Nsight records for other backends.
Decision and no-go rules
- Host launch overhead dominates? Try supported graph or command-buffer execution first and verify its effect. CUDA graphs are the NVIDIA example; support elsewhere must be established.
- Step at ~85% or more of the bandwidth ceiling (ceiling bands): only fewer bytes per token help: an exact byte cut (above; start these early anyway, and recompute the ceiling for the new format) or lower precision (a precision decision with its own quality bar). Why: the other three losses are nearly gone.
- Never change two things in one A/B. Why: you cannot attribute the result, and one change may hide a regression in the other.
- Losing changes stay out of the branch but go into
experiments.jsonland the report. Why: the next person will try the same idea. - A non-deterministic reduction (atomics in split-K) needs an explicit decision. Why: it breaks byte-identity testing and reproducibility.
- Push only within explicit authorization; do not ask again for an already authorized push. Otherwise commits stay local.
Reporting rules
When this skill finishes a standalone engine optimization, check for an optional engine contribution after the final report. Within an optimizing-inference run, leave that step to its coordinator so the same result produces only one PR. Publication requires the user's explicit authorization and the completed engine-level proof, not a faster microbenchmark alone.
- Speedups come only from an interleaved A/B (
bench roundsratio CI: derived from observed rounds;compare: observed); Amdahl bounds and bytes saved are derived; "this worked because X" is inferred until the mechanism profile is attached. - A claimed change without all three (correctness, interleaved A/B, mechanism profile) is UNPROVEN (or NOT RUN for the missing one), never "probably faster"; an unclaimed kept change is reported as screened. Engine-timed A/Bs (
bench decode) are labelled as such.
Profiling inference: timeline first, then kernels
nsys records a timeline (CPU threads, CUDA API calls, kernels, copies, NVTX ranges): where the time goes and whether CPU or GPU is on the critical path. ncu replays one kernel reading hardware counters: why it takes as long as it does. Neither proves a speedup; only an interleaved A/B does (benchmarking). In optimizing-kernels' change loop: one nsys capture per lever area (again when the bottleneck moves), a mechanism profile only for a claimed change.
Use when decode or prefill is below ~85% of its ceiling for unknown reasons (ceiling bands), for launch overhead, host syncs or idle gaps, for one dominant kernel family's limiter, or for the mechanism behind a measured speedup. Not for a speed number (never time with a profiler on), nor before you have a benchmark and a ceiling (a profile without them has no scale).
GPU approval: captures occupy the GPU; run them only with budget.approved: true in intake.yaml or an explicit yes to the stated cost. Analysing existing files needs none.
Route before capturing
Read doctor's data.hardware.platform/data.profiling_route and the relevant hardware reference. NVIDIA: nsys → ncu. AMD: rocprofv3 HIP/kernel/copy timeline → architecture-supported counters (Instinct support does not establish Radeon support). Apple: Instruments CPU/Metal → Xcode/MLX capture. CPU: perf when permitted, otherwise engine timers or CPU-only torch.profiler. nsys/ncu capture is NOT RUN elsewhere; profiling itself still applies.
Retrieve current official docs online for unknown architectures, runtime/tool support, flags or errors before choosing a path; record URL/date/version and check local help. Use AMD ROCm, Apple Developer/MLX and NVIDIA primary sources. Retain timeline/op attribution when counters fail, marking unsupported checks NOT RUN. All platforms follow critical path → Amdahl → change → correctness → unprofiled interleaved A/B, with the same environment hygiene. Include text encoding, image VAE, audio codec/vocoder and buffering where applicable.
Preflight checklist
- Steady-state work for a bounded number of steps (load and warmup rank the wrong things).
- Weights downloaded, no credentials needed (
HF_HUB_OFFLINE=1): the capture must hold no tokens. - You know whether the engine uses CUDA graphs and programmatic dependent launch (PDL): both change how the timeline reads.
- For ncu: counters readable ("Fallbacks"), measured bandwidth (
inferencekit ceiling bw), memory headroom for replay. - Asked whether the person already has profiles of this workload (no GPU, no approval needed).
Hard failure gates
- Never capture outside
env -iwith an allowlisted environment (inferencekit nsys capture/ncu capturedo it). Why: a.nsys-repstores the full environment, tokens included. - Never rank by summed kernel duration. Rank by critical-path time per step (
inferencekit nsys analyze->data.steps.families.*.critical_pct_of_step, GPU idle indata.steps.idle_pct_of_step).kernel_sum_us / span_us > 1means kernels overlap and duration shares are inflated. - Never take host-side costs (
cudaGraphLaunchtime) from a node-traced capture. Why: node tracing inflated it about 25x in one measured case. - Never use an unanchored ncu regex or a
--launch-skiplanding in warmup, autotuning, graph capture or prefill (the wrong launch). - Never quote timeline, trace or ncu durations as step time or speedup. Why: tracing adds overhead; ncu serialises kernels, flushes caches and removes PDL overlap.
NVIDIA workflow (copyable)
Profile:
- [ ] 1. Benchmark number and ceiling in hand (ms/token or TTFT, % of ceiling)
- [ ] 2. inferencekit nsys capture -- <steady command> -> ev_cap (.nsys-rep, .sqlite)
- [ ] 3. Check the record: trace mode (node/graph), env allowlist, capture window, limitations
- [ ] 4. inferencekit nsys analyze <rep|sqlite> -> ev_tl
- [ ] 5. Profile check: per-step wall p50 vs the benchmark's ms/token (within a few %?)
- [ ] 6. kernel_sum / span: > 1 means overlap; use critical-path shares only
- [ ] 7. Answer the six questions, each with a number
- [ ] 8. One family dominates and is far from speed of light -> ncu on it (counters blocked: NOT RUN, see "Fallbacks"):
inferencekit ncu capture --kernel-regex '^<name>$' --launch-skip <N_pre + 2k> --count 5 -- <cmd> -> ev_ncap
inferencekit ncu summarize <.ncu-rep> --peak-gbs <ev_bw> -> ev_k
- [ ] 9. Bound label per kernel; speed-of-light gap; Amdahl bound of the proposed change
- [ ] 10. Fill the fixed tables; every number observed/derived/inferred
Exact commands
inferencekit nsys capture -- python run_decode.py --model ./models/qwen3-8b-fp8 --gen-len 128
inferencekit nsys analyze <.nsys-rep or .sqlite path from the capture record's artifacts>
inferencekit ncu capture --kernel-regex '^fp8_block_gemv_kernel$' --launch-skip 400 --count 5 \
-- python run_decode.py --model ./models/qwen3-8b-fp8 --gen-len 64
inferencekit ncu summarize <.ncu-rep from the capture record> --peak-gbs <ev_bw>
inferencekit evidence show <ev_id>
MCP tools: capture_nsys, analyze_nsys, capture_ncu, summarize_ncu, capture_rocprof, analyze_rocprof, inspect_evidence (in OpenCode, run captures through the CLI, detached). Options: --help. Hygiene and manual env -i recipes: references/capture-hygiene.md.
Fallbacks: counters blocked, no nsys, supplied profiles, other hardware
Other skills link here for these rules.
- Check the counters.
doctor'sdata.profiling_counters.status, from the driver setting and the process's capabilities (not root: a container's root usually lacks CAP_SYS_ADMIN):allowed,admin_only(no ncu, unless its reason says this process has CAP_SYS_ADMIN: then probe) orunknown.inferencekit doctor --probe-profilerssettles it with a tiny nsys and ncu capture, or the first smallncu capturedoes (countersin itscoverage.missing: blocked). Containers and many cloud instances block them, A100 and H100 included: a host setting, not the GPU model. - Blocked: nsys only (it needs no counters): the whole timeline half (steps 1-7); every ncu row NOT RUN ("GPU counters blocked on this host", doctor evidence id), and the report says, in these words: "ncu was not possible on this host; profiling used nsys timelines only"; the bound label only from arithmetic (bytes a kernel must read ÷ its in-step time vs measured bandwidth), inferred, no counters.
- No nsys either, or you want PyTorch-op attribution: the PyTorch profiler (no counters, no Nsight): vLLM
--profiler-configwith/start_profileand/stop_profile, SGLang/start_profile,torch.profilerin a custom engine. The CLI does not parse its Chrome/Perfetto traces: read kernel time per step and GPU idle gaps by hand or withprofile-analyst. It slows the run: never time with it on (references/torch-profiler.md). - Ask for the person's own profiles captured where they have permission (
profiles: <path>, ...inconstraints; non_interactive: aQUESTIONS.mdrequest):inferencekit nsys analyze <file.nsys-rep or .sqlite>,inferencekit ncu summarize <file.ncu-rep> --peak-gbs <ev_bw or GB/s>(no GPU; without the ncu CLI, CSV exports: references/ncu-capture-recipes.md). A profile from another GPU or engine version explains that setup only: say so by every number. Never copy, commit or upload such files (they may hold secrets); cite the analysis. - Not NVIDIA: nsys and ncu rows are NOT RUN. On AMD use
inferencekit rocprof captureandrocprof analyzefor kernel/HIP traces, or a supported vendor profiler; rocprof-compute support depends on the exact GPU (amd-rocm.md). Use perf or likwid on CPU, Instruments or MLX on Apple, underenv -i. Vendor captures outside the CLI are cited by file; kernel totals may overlap and the bound label stays inferred.
Reading the timeline
- Node vs graph tracing.
--cuda-graph-trace=graphshows realistic host timings (host costs, CPU vs GPU critical path);nodeshows every kernel inside graphs (ranking, gaps) but inflatescudaGraphLaunch.inferencekit nsys capturetraces nodes; for host costs add one graph-level capture (references/capture-hygiene.md). - Critical path, not summed kernel time. Under PDL (and multiple streams) the next kernel starts early and waits on the GPU, so durations include waiting. When the per-step overlap
kernel_sum_us - busy_usis about one family's summed duration, that family is mostly hidden: size changes withcritical_pct_of_step(lessons, "Summed time lies under PDL"). - Profile check. The profiled token period (step wall plus the gap to the next step) vs the benchmark's ms/token: a few percent of inflation is normal; more means the profile measures something else (often the tracer): fix the window or trace mode first.
The six questions (top-down, each with a number)
- Does this stage matter? Its share of per-token (or per-prefill) wall time. 2. CPU or GPU on the critical path? GPU idle gaps and blocking calls (
cudaStreamSynchronize, D2H of the sampled token) mean the CPU is. 3. Which stage? Attention, MLP, lm_head or sampling, by critical-path family. 4. A few big kernels or a long tail? A long tail of microsecond kernels means hand-off bound. 5. Speed-of-light gap? Bytes a big kernel must read / measured bandwidth vs its in-step time. 6. Amdahl bound?1 / ((1 - f) + f / s)for critical-path fractionfmadestimes faster. Traps and SQL on the.sqlite: references/six-questions.md.
ncu: labelling the bound
| Label | Evidence pattern (heuristic thresholds) | Lever |
|---|---|---|
| memory-bound | DRAM throughput >= ~70% of the measured peak; SM pipes well below theirs | read fewer bytes (precision, fusion that avoids re-reads) |
| compute-bound | a pipe (tensor, FMA, ALU) >= ~70% of its peak; DRAM well below | fewer instructions (better maths path, tensor cores, cheaper dequant) |
| latency-bound | DRAM and SM both below ~50%; few eligible warps; long-scoreboard / barrier stalls; grid below one wave or low occupancy | more bytes in flight: more blocks (split-K), wider loads, fewer dependent steps |
The label is inferred: a hypothesis, not a verdict. Counters suggest; in-place timing decides (lessons). Capture recipes: references/ncu-capture-recipes.md; metrics and the L2-resident and nominal-peak traps: references/ncu-metrics.md.
Evidence kinds: nsys_capture -> nsys_timeline; ncu_capture -> ncu_kernels; rocprof_capture -> rocprof_summary. Profile files hold host names, paths and command lines: never upload one unless the user checked it.
Fixed result tables
| Question | Answer | Number | Class | Evidence |
|---|---|---|---|---|
| Profile vs benchmark (step ms vs ms/token) | % difference | derived | ev_tl, ev_bench | |
| kernel_sum / span | ratio | derived | ev_tl | |
| 1-6 (as above) | observed / derived | ev_tl, ev_bw |
| Kernel (anchored) | Launches profiled | Grid, waves | ncu duration us (not e2e) | DRAM GB/s | % of measured peak | SM % | Occupancy | Top 2 stalls | Bound (inferred) | Evidence |
|---|
Decision and reporting rules
- Idle above ~10% of the step: fix the host side (CUDA graphs, async scheduling, GPU-side sampling) before kernels, which would only lengthen the gaps. Memory-bound at >= ~85% of measured peak: stop tuning that kernel. Latency-bound with waves below 1: add parallelism (split-K, smaller tiles) first.
- Amdahl bound below the benchmark's CI half-width: do not start (you cannot prove the win). Stalls from kernels under ~10 us or one launch are qualitative only.
- Timeline and counter values are observed under tracing or replay; say so. Shares, Amdahl bounds, % of measured peak and speed-of-light gaps are derived (cite ev_bw). "Launch-bound", "memory-bound" are inferred until an A/B supports the change they motivate.
- A question the capture cannot answer (host costs from a node-only trace; % of peak without measured bandwidth) is UNPROVEN, naming the capture that would answer it. Every ncu duration says "ncu replay time, not end-to-end".
Proving correctness and quality
A speedup only counts if the engine still computes the same model. Engines almost never agree bit for bit, so "same" is defined per kind of change, against a noise floor of correct-but-different arithmetic. optimizing-inference runs the full workflow per candidate engine and at milestones; inside the change loop each change gets only the fast check below.
When to use / not use
Use when:
- before claiming any speedup (benchmarking, optimizing-kernels);
- a kernel, fusion, precision, engine, attention backend or serving config changes (inside a change loop: the fast check only);
N*is being claimed (quality must hold at that load level too).
Do not use when:
- you want to rank different models by quality (evaluation, with far larger sets);
- the change intentionally makes a new model (new weights, a 4-bit recipe you have not agreed to treat as the same model): that is a quality study with the user's sign-off.
GPU approval: reference and candidate runs occupy GPUs; run only with budget.approved: true in intake.yaml or an explicit yes to a stated cost.
Choose the contract and modality first
Read the run's quality choice. Exact requires equality on its declared deterministic/seeded outputs; legacy lossless distribution agreement alone is insufficient. Balanced/speed tracks use their registered task/perceptual budget, so rounding differences are not an automatic rejection. Model changes need separate permission and separate reporting.
Use modality-appropriate structural integrity and task quality: text accuracy/distributions, image semantics/readability, speech content/completion/speaker, video temporal consistency. Other tasks need a registered adapter; do not substitute GSM8K or token identity. Missing listening/viewing remains UNPROVEN. The text procedures below apply only where their inputs and claims match.
For TTS, register the acceptance policy before capture with quality audio-policy, then run quality audio on final configuration records and saved PCM (protocol). For changed outputs, quality audio-scorers pins local models and quality audio-score produces ASR/naturalness/voice observations (setup). Score after timing; WER/CER and MOS predictors do not replace required listening.
Which rule applies (decide before measuring)
| Kind of change | Examples | Fidelity | Pass rule |
|---|---|---|---|
| Pure refactor | moving code, launch restructuring, a switch that is off, prefill tiles that keep the K order | lossless | byte identity (ladder below) |
| Equivalence | CUDA graphs, fusion, the same precision in another engine or attention kernel | lossless | no regression beyond the A/A noise floor |
| Exact byte cut | lossless weight re-encoding, an exact bound-pruned lm_head | lossless | its proof recipe (byte-cuts.md: weights decode bit-equal; pruned token = full token every step; logprob requests take the full head), rare_tokens, then equivalence |
| Exact speculation | MTP or EAGLE heads, a same-tokenizer draft model, n-gram lookup, with standard verification | lossless | the spec_exact gate at temperature 0; under sampling, the accept/resample check and quality sampling (sampled exactness); then equivalence |
| Declared numerics change | 16-bit activations where the checkpoint quantises them; a KV dtype the task lists | lossless only if the task allows it; declared in the engine's numerics, labelled "numerics changed" | quality against the reference no worse than the baseline's, within the noise floor |
| Numeric | an FP8 KV cache the checkpoint does not declare, lower-precision attention maths | lossy | no regression beyond the noise floor plus a pre-registered margin |
| Approximation | 4-bit weights, lossy speculative decoding, weight or expert pruning, a truncated vocabulary | lossy | a pre-registered margin agreed with the user; otherwise it is a different model |
fidelity: lossless (the intake default) allows only the lossless rows. fidelity: lossy adds the others, each judged against the intake's quality.budget and reported on its own track, as speed at a stated quality cost. A task's own precision rule overrides this plugin's default. Full rules: fidelity.md.
For images, TTS and STT, an explicitly authorized numerical change uses the registered modality checks rather than LLM token/logprob gates. Relative degradation budgets belong in quality.budget.relative, with a matching modality and quality.protocol / quality.protocol_sha256 binding. A valid intake leaves external quality proof UNPROVEN. Different pixels, PCM, generated codes or rounding are diagnostic observations, not automatic quality failures for that numerical-change track. Pure refactors still use the declared exactness checks. Nonfinite values, invalid shapes, unsafe accesses, broken own-output/native integrity and omitted cases remain failures. A quality budget alone does not authorize architecture pruning or another model.
With explicit quality.acceptance: perceptual_and_task_noninferiority, image/TTS promotion requires the bound task checks and fresh blinded perceptual evidence. Numerical similarity (including image LPIPS/SSIM) is diagnostic, not an independent rejection gate; do not add dummy bounds. TTS ASR/VAD/speaker proxies do not replace listening. Missing perceptual evidence remains UNPROVEN. Retain old numerical failures under their original rules and reassess a baseline's eligibility prospectively under the new acceptance policy.
Inside the change loop: one fast check
For an image or speech numerical change, use one short fixed public modality correctness/quality screen under the agreed protocol. Full registered quality and balanced timing remain milestone/final promotion requirements; they are not required for every intermediate kernel. Retain failed screens and all cases, and label any historical output reuse exploratory. No numerical candidate is promoted from exact-output diagnostics alone.
For LLM candidate changes, against the previous build, greedy, on a handful of fixed prompts: a pure refactor or exact change keeps the greedy tokens identical; a numerics change (any change to the arithmetic) keeps next-token and logprob agreement, flips only at near-ties (inferencekit correctness --reference <prev.jsonl> --candidate <new.jsonl>); a fail drops the change. Inputs: one {"tokens": [...], "logprobs": [...]} line per prompt from each build's server (a bench decode id compares tokens only). For a numerics change, --min-prefix-agreement below 1 (written down first) and --logprob-tol <t>; at each first divergence, the top-2 gap from the previous build's top-20 logprobs must be below the A/A control's largest flip margin, or the change drops (margin test). The A/A floor, trajectory score, GSM8K, sampling and byte-identity ladder below are the full proof: milestones and published claims only (state.md).
Preflight checklist
- The prompt set fixed and verified (
inferencekit prompts verify --set <dev|path>), the same token ids for every engine. Why: changing prompts changes the result. - Greedy decoding, every sampler setting explicit, fixed length, same chat template. Why: sampling turns every comparison into noise. Speed is timed at the workload's sampling; a sampled claim (sampler, speculation, a pruned head above temperature 0) also needs sampled exactness.
- A higher-precision reference endpoint (intake
quality.reference): a vLLM server running the model in BF16 for an FP8 engine (quality scoreneeds vLLM'sprompt_logprobs). Why: distance to a better reference is the quality signal; two candidates agreeing with each other proves little. None: below. - Quality runs on the exact speed configuration (the same resolved config that was timed), never a debug or batch-invariant mode. Why: those modes run different, slower kernels.
- An A/A control planned, run through the identical pipeline: the reference's own greedy trajectory (
--baseline), ideally twice (--baseline-repeat, e.g. after a relaunch). Why: it sets the noise floor; no tolerance may be invented. - Margins pre-registered and passed explicitly (
--margin-agreement,--margin-kl,--margin-nll). Why: the tool's defaults (0.02 / 0.02 nats / 0.05 nats) are loose placeholders, ten times looser than a same-precision KL margin.
Hard failure gates
- NaN or Inf anywhere in logits or logprobs. Always a bug.
- A disagreement where the reference was confident (large top-1 vs candidate-token margin, above the control's largest). Why: near-ties flip under rounding; large-margin flips mean wrong maths.
- Teacher-forced agreement or trajectory score below the noise floor by more than the pre-registered margin, with the CI excluding the margin.
- Any rung of the byte-identity ladder failing for a change declared a pure refactor.
- An A/A control that is not identical when the ladder requires identity. Why: then the harness is non-deterministic and no A/B difference can be interpreted.
Workflow (copyable)
Quality:
- [ ] 1. Classify the change (table above); pre-register the rule and margin
- [ ] 2. Reference (vLLM, BF16) up: quality trajectory --label reference, then --label reference-r2
(restart your own reference server in between; a remote endpoint cannot be restarted:
run back to back and say so) -> the A/A floor's trajectories
- [ ] 3. Per engine, its exact speed config up: quality trajectory --label <engine> -> ev_traj_<engine>
- [ ] 4. quality score --trajectories <dir> (once per engine) --baseline <reference dir> [--baseline-repeat]
with the pre-registered margins -> ev_qs (verdict per engine vs the floor, paired CI over prompts)
- [ ] 5. Every disagreement: position + reference margin; any above the control's max -> investigate / FAIL
- [ ] 6. quality gsm8k --concurrency 0 on every engine: all 1,319 for a claim, the final candidate
or a scored commit; --limit 100 at the light milestone checks (only screens);
candidates with --paired-with <baseline run> (McNemar) -> ev_gsm
- [ ] 6b. Sampling in the claim? quality sampling on each engine, the baseline too (draws vs own logprobs) -> ev_samp;
plus the accept/resample check (speculation) or per-step support check (pruned head)
- [ ] 7. Long context in the claim? quality niah at the lengths claimed (optional otherwise)
- [ ] 8. Pure refactor? Byte-identity ladder instead, control first
- [ ] 9. Fill the fixed table; verdict per check
Exact commands
# A/A floor: the reference's own greedy run (vLLM serving the model in BF16)
inferencekit quality trajectory --base-url http://127.0.0.1:8001 --model <ref-name> --prompts <set.jsonl> \
--label reference --run-dir runs/<id> --top-logprobs 20
# each engine, at its exact speed configuration
inferencekit quality trajectory --base-url http://127.0.0.1:8000 --model <served-name> --prompts <set.jsonl> \
--label vllm-tuned --run-dir runs/<id> --top-logprobs 20 --engine-kind vllm # -> runs/<id>/quality/vllm-tuned
inferencekit quality score --trajectories runs/<id>/quality/vllm-tuned --reference-url http://127.0.0.1:8001 \
--reference-model <ref-name> --baseline runs/<id>/quality/reference [--baseline-repeat runs/<id>/quality/reference-r2] \
--margin-agreement <pre-registered> --margin-kl <pre-registered> --margin-nll <pre-registered> # -> ev_qs
inferencekit quality gsm8k --base-url http://127.0.0.1:8000 --model <served-name> --run-dir runs/<id> --label baseline \
--concurrency 0 # 0: sized from the server
inferencekit quality gsm8k --base-url http://127.0.0.1:8000 --model <served-name> --run-dir runs/<id> --label vllm-tuned \
--concurrency 0 --paired-with runs/<id>/quality/gsm8k/baseline.jsonl # McNemar vs the baseline
inferencekit quality sampling --help # sampled exactness (below)
inferencekit correctness --reference ref_hf.jsonl --candidate <ev_bench> # token agreement: the per-change fast check vs the previous build (or the older HF-reference method)
inferencekit report <ids...> --format md
MCP tools: quality_trajectory, quality_score, quality_gsm8k, quality_niah, check_correctness, report (in OpenCode, which ends an MCP call after 120 s, run quality through the CLI, detached with a log). Flag spellings: inferencekit quality <op> --help. Scoring, the noise floor and pass rules in detail: references/quality.md.
Trajectory scoring (summary)
Each engine generates its own greedy continuation through its serving (decode) path, the code users run, with the top-20 logprobs per step. The reference (vLLM) is teacher-forced on that sequence and scores every position: top-1 match, KL on the top-k supports (an approximation), the reference's log-probability of the engine's token, and the first divergence from the reference's own greedy run. The A/A floor is the reference's own trajectory scored the same way, so it already holds the gap between decode path and one-pass scoring. Prompt-logprob ("score mode") endpoints exercise only prefill kernels.
Sampled exactness
Lossless means the same sampling distribution, not only the same greedy token. The teacher-forced check holds the engine's log-probabilities to the reference's; inferencekit quality sampling shows the sampler draws from the distribution they give under the request's temperature, top-k and top-p (seeded first-token draws against the top-20 log-probabilities; mean TV beyond sampling noise, bootstrap CI over prompts; references/quality.md section 6). PASS when the CI's upper end is inside the pre-registered margin, FAIL when its lower end is above it, UNPROVEN otherwise or when not run.
It checks only the first generated token (max_tokens 1, produced by prefill), so a change that acts only in decode steps is not exercised. Speculation also needs a unit check of the accept/resample step against the target distribution (synthetic p and q, many draws, TV or chi-square inside a pre-registered bound). A pruned head needs the per-step support check in byte-cuts.md, or the first token must also go through the pruned head. Without one of these the sampled claim is UNPROVEN.
Why bitwise identity across engines is not expected
Floating-point addition is not associative: tiling, split-K, reduction order, FMA contraction, online softmax and shape-dependent heuristics each give a slightly different, equally valid result, and after one flip greedy outputs diverge. With the early engine almost no prompt matched the reference exactly for any engine (the reference's own second attention path included) while per-step agreement stayed inside the floor. Exact-match rates are a poor quality signal.
Byte-identity ladder (pure refactors only)
Climb in order; stop at the first failing rung (references/byte-identity-ladder.md):
- Kernel outputs byte-equal (memcmp) across a shape sweep, outputs pre-filled with NaN.
- Identical SASS for kernels the refactor should not touch (
cuobjdump -sass). - End-to-end logits (and KV cache) byte-equal on the fixed prompts, every path (eager, capture, replay).
- Under load: an off/off (A/A) control first, then old vs new.
Task eval
inferencekit quality gsm8k: the same problems, format and answer extraction for every engine, greedy, accuracy with a Wilson CI; --paired-with <the baseline's results JSONL or evidence id> adds McNemar's exact test. A claim uses all 1,319 items (the default); a --limit subset only screens (the first 100 detect only about 10-point effects). --concurrency 0 sizes concurrency from the server. The first run on a host downloads the pinned test split (419 KB) into ~/.cache/inferencekit/gsm8k/: unset HF_HUB_OFFLINE for it, or place the file with the curl command the error prints (references/gsm8k-parity.md; the older token-agreement method: references/token-agreement.md).
No higher-precision reference
With quality.reference: none, compare with the baseline engine run twice, restarted between (the A/A floor); label results "vs baseline, no reference".
- GSM8K, all 1,319: candidate
--paired-withbaseline; the A/A pair gives the noise-only discordant count. - Held-out NLL per token on a fixed text in no prompt set, from each engine's logprobs (outside the CLI; record the text's sha256 and code path). Pass: gap to baseline within the A/A gap plus a pre-registered margin.
- A vLLM baseline can also be the
quality scorereference: it bounds drift from the incumbent, not quality.
Artifact/evidence contract
quality trajectory->runs/<id>/quality/<label>/<prompt-id>.json+index.jsonand akind=quality_trajectoryrecord (observed);quality score->runs/<id>/quality/score.jsonand a derived record: per engine, per position bucket and input bucket, with a PASS / FAIL / UNPROVEN verdict against the floor;quality gsm8k->kind=quality_task(per-problem outcomes, not only accuracy);correctness->kind=correctness.- Keep the prompt-set sha256, template and decoding settings in each record. Why: a quality record without its inputs proves nothing.
Fixed result table
| Check | Engine (config) | Value [CI95] | Noise floor (A/A) [CI95] | Margin (pre-registered) | Verdict | Evidence |
|---|---|---|---|---|---|---|
| Top-1 agreement vs reference (decode path, tie-inclusive) | % | % | pp | |||
| Mean / p99 KL (top-20 approx.) | ||||||
| Disagreements above the control's max margin | count | must be 0 | PASS / FAIL | |||
| GSM8K (1,319), McNemar vs baseline | x / n, p | |||||
| Sampled distribution (mean TV vs baseline; speculation: + accept/resample check), if sampling is claimed | ||||||
| NIAH at lengths claimed (optional) | NOT RUN if skipped | |||||
| Byte ladder rung reached (refactors) | 1-4 | control identical? |
Decision and no-go rules
- FAIL blocks any speed claim for that engine or change. Why: performance of a wrong engine is meaningless.
- Tolerances come from the A/A control, never from "looks small". Why: invented tolerances drift toward whatever the candidate produced.
- Divergences explained by near-ties only: PASS with the explanation. Why: expected behaviour of correct, different arithmetic.
- No reference: the no-reference route; without it, UNPROVEN, not PASS. A test without enough items to detect the pre-registered effect: UNPROVEN (underpowered), not PASS.
- Speed claims stay within one precision even when quality is compared across precisions against the same reference. The one exception is a format trade-off (evaluating-quantization): a speed ratio with its CI95 shown only next to this skill's quality verdict for that row, labelled "cross-precision: format trade-off, not an engine comparison".
Reporting rules
- Agreement, KL and task scores are observed; noise floors from controls are observed; "divergence is due to rounding" is inferred, supported by the margin at divergence.
- State which path was measured: decode path (trajectory) or one prefill pass (score mode). The two are not comparable with each other.
- Verdicts PASS / FAIL / UNPROVEN / NOT RUN per check, combined with the precedence in reporting-perf-claims.
Reporting performance claims
A performance claim is a statement someone will act on: it says what was measured, under which conditions, with what uncertainty, and what was not shown. This skill turns evidence records into claims that survive review, and audits claims others wrote; it is the last stage of optimizing-inference.
Use to write up any benchmark, capacity, quality, profile or optimisation result, to review a PR description, README table, blog post or agent summary with performance numbers, or to decide what verdict evidence supports. Not while you still need to measure (benchmarking, profiling, proving-correctness).
Preflight checklist
- The claim was written before the measurement (statement, metric, workload, bucket, decision rule). Why: claims chosen after the data are forking-paths results.
- Every evidence id the report cites exists (
inferencekit evidence show). Why: dangling citations are fabricated evidence. -
coverageandlimitationsof every record have been read. Why: a valid record can still not cover the claim. - Applicable gates passed:
gates runfor LLMs; structural and semantic checks for media. A gate failure invalidates a win. - Losses, refuted ideas (
experiments.jsonl) and NOT RUN items are listed. Why: a report of only wins misleads about the base rate.
Hard failure gates
- A number without an evidence id or a formula.
- A speedup without an interleaved A/B whose ratio CI95 excludes 1.
- A speed comparison across precisions (weights, activations, KV cache, lm_head), GPUs or workloads presented as an engine result.
- An engine-timed (
bench decode) number in a published table, or next to client-timed (bench serve) numbers without its label. - A profile (nsys, ncu) cited as proof of a speedup.
- Missing data shown as 0, blank or PASS.
- A speed claim without a correctness/quality verdict.
Workflow (copyable)
Report:
- [ ] 1. Copy the pre-registered claim(s) verbatim, with their timestamp
- [ ] 2. inferencekit evidence list; select the records for each claim
- [ ] 3. For each claim, evaluate the precedence FAIL > UNDERPOWERED > INCOMPLETE > NO-DATA > PASS
- [ ] 4. Map to PASS / FAIL / UNPROVEN / NOT RUN
- [ ] 5. Label every number observed / derived / inferred
- [ ] 6. inferencekit report <ids...> --format md
- [ ] 7. inferencekit review build runs/<run-id> (outputs side by side, per bucket and engine)
- [ ] 8. Write REPORT.md using references/wording-guide.md; retain mechanism, losses, NOT RUN items and ledger in Details
- [ ] 9. Claims lint over the final text (below); fix or downgrade every failing line
Exact commands
inferencekit evidence list
inferencekit evidence show <id>
inferencekit compare <ev_baseline> <ev_candidate> --metric <metric> # same-kind round records; not load mode
inferencekit report <ids...> --format md # fixed layout; --format json for CI
inferencekit review build runs/<run-id> # -> runs/<run-id>/review/index.html
inferencekit history # kept commits with their Perf-* trailers
MCP tools: list_evidence, inspect_evidence, compare, report, build_review, history. The CLI/MCP report is a fixed technical export; link it or include its tables in Details of the user-facing report. Add quality/gates verdicts from their records; the export has no such columns. Served ratios come from bench rounds' <metric>_ratio_to_best. Follow references/wording-guide.md for REPORT.md and the final reply.
Verdict precedence
Evaluate in this order; the first that applies is the verdict.
| Step | Condition | Reported as |
|---|---|---|
| FAIL | a required check ran and contradicts the claim: regressed, CI95 entirely below the claimed ratio, a quality FAIL, a gate FAIL, or a hard gate in the candidate (except the reasons hardware.md, note 3, lists as expected on that platform, reported as "not applicable on ") | FAIL |
| UNDERPOWERED | checks ran but the CI includes 1 (or the claimed ratio and 1 both), or fewer rounds/items than pre-registered (a kernel-loop full proof pre-registers its fixed 3 rounds) | UNPROVEN (underpowered) |
| INCOMPLETE | coverage.complete is false, an engine/bucket/level/metric in the claim is missing, or rounds were not interleaved | UNPROVEN (incomplete) |
| NO-DATA | a required check produced no data | NOT RUN if never attempted; UNPROVEN if attempted |
| PASS | none of the above, and the decision rule is met | PASS |
Why this order: one contradicting check must never be outvoted by passing ones, and "not enough data" must never look like "fine". Worked examples: references/verdict-rules.md.
Claims lint (run over every final text)
Result links descriptively to Details or its evidence table for ids, numeric classifications and formulas; every claim stays traceable.
- Every number has an evidence id, or a formula whose inputs have ids.
- Every number is labelled observed / derived / inferred.
- Claim verdicts use only PASS / FAIL / UNPROVEN / NOT RUN; comparison verdicts only
improved | regressed | within_noise | inconclusive, frominferencekit compareor abench roundspaired ratio CI. - No speed claim crosses precisions; cross-precision rows are labelled "cross-precision: not an engine comparison", with no ratio verdict. Exception: a format trade-off (evaluating-quantization) may show its speed ratio with CI95 only next to its quality verdict against the full-precision reference, labelled "cross-precision: format trade-off, not an engine comparison", with no
improved/regressed. - Published timing is client-measured: served LLMs or complete media requests. Diagnostic engine timings are labelled.
- Every speedup has a ratio with its CI95 and the conservative bound quoted; "up to", "slightly faster", "near optimal" are absent.
- Every engine row states its version, resolved key settings and whether it was tuned for this goal.
- Capacity claims state the SLO,
N*, open-loop arrivals, and the price behind any cost. - Missing buckets, levels, GPUs and checks appear as NOT RUN, never as blanks or zeros.
- Losses and refuted ideas are listed with the same columns as wins.
- The review page exists for the run and is linked. Line-by-line examples and fixes: references/claims-lint.md.
Review page (inferencekit review build <run-dir>, MCP build_review)
One self-contained HTML file at runs/<run-id>/review/index.html that works offline: rows are inputs grouped by bucket, columns are engines plus the reference, with the first divergence from the reference marked; a summary table at the top; filters by bucket and engine. Link it from the report. Why: a reader can see what each engine actually produced for the inputs that were timed, which catches broken outputs no aggregate metric shows. The full page stays local; share it only after the user has checked it.
Fixed result table
Open with the plain-English summary from stop-point.md, "Summary". Put the technical claim table in Details, one row per pre-registered claim, with these columns:
| Claim | Workload / bucket | Baseline | Candidate | Ratio [CI95] | Quality | Mechanism | Gates | Verdict | Evidence |
|---|
Populate every applicable column from evidence; mark missing checks UNPROVEN or NOT RUN.
Decision and no-go rules
- Quote the conservative bound: "at least 1.09x (CI [1.09, 1.16])". Why: the point estimate is the luckiest honest number.
- Publish losses next to wins, with the same rigour. Why: readers calibrate trust from them.
- Never upgrade a verdict in prose ("UNPROVEN but clearly faster"). Why: the verdict exists to stop that sentence.
- Never extrapolate across GPUs, precisions, context lengths, buckets or concurrency levels you did not measure. Write NOT RUN.
- Refuse a cross-precision speed claim even when asked; offer the same-precision comparison and the cross-precision row labelled as such. The only exception is a format trade-off from evaluating-quantization: the speed ratio with its CI95 next to the row's quality verdict, labelled "cross-precision: format trade-off, not an engine comparison", with no
improved/regressedverdict. Why: a format buys its speed with quality, so only the pair is a result; speed alone across precisions is still refused.
Reporting rules
- observed: measured by a tool in this environment; cite the id and statistic. derived: arithmetic on observed values; show the formula. inferred: reasoning; name the measurement that would confirm it.
- Verdicts: PASS / FAIL / UNPROVEN / NOT RUN for claims;
improved | regressed | within_noise | inconclusivefor comparisons. - No environment values, secret tokens or raw argv in a report; inferencekit stores argv only as a digest.
Research to an experiment
Called by optimizing-inference and optimizing-kernels. Return a small ranked experiment queue, not a literature review.
- Fingerprint the problem. Model architecture/components and revision; modality and input/output shapes; batch/concurrency; device architecture and memory; runtime/compiler versions; quality and model-change permissions; current profile and target. Missing facts are unknown, not guessed.
- Search what can actually run. Check installed engine features first, then official releases, code, tests, issues/PRs, compiler/vendor docs and papers. Search architecture/operator and device identifiers, not just the model name. An issue is a hypothesis until code or a reproducer confirms it.
- Look beyond papers. Use the source map as a starting point, and actively discover sources beyond it: GPU MODE talks/slides/code, hardware and runtime engineering blogs, kernel libraries and compiler repositories often expose actionable details before papers. Search recent releases, merged PRs, conference/workshop material and new engineering groups for the latest relevant work and largest credible gains. Use original sources; follow secondary summaries to their implementation.
- Check freshness and applicability. Record URL, access date, release/commit, publication date when available, supported device, dtype, shape and assumptions. Check whether the installed compiler already implements the idea. Compare high reported gains against baseline strength, quality/precision, shapes and full-request scope; prioritize the biggest applicable opportunity with credible headroom. A newer article is not automatically better. Do not call old material “latest” without checking updates.
- Produce an experiment. Each useful finding needs the table below. Start implementing the cheapest informative test once a credible candidate exists; do not finish a catalog first. Retain at most three immediate candidates, with other links in a backlog.
- Refresh on evidence. Search again after a moved bottleneck, two uninformative attempts, a toolchain change or an unexplained fallback. Reuse prior findings and rejected hypotheses instead of repeating the same search.
| Finding | Required record |
|---|---|
| Source and evidence | Exact source/section/commit; what was read or inspected; reported result versus your inference. |
| Proposed change | File/operator/runtime path; implementation or flag to reuse; how to prove it executed. |
| Compatibility | Device generation, compiler/runtime, dtype, shapes, memory, license and model/quality permissions. |
| Expected headroom | Critical-path fraction and Amdahl bound, or bytes/FLOPs/launches saved; mark estimates inferred. Unknown profile means first measure attribution. |
| Cheapest useful test | Small representative shapes and quality screen, expected mechanism, falsifying outcome, complete-request confirmation. |
| Decision | Try, defer or reject with reason and linked experiment id. |
Read linked slides, code or transcripts when a video is unavailable. Record that limitation; never claim to have watched a talk from its title. A tutorial example or datacenter result is not proof of support or speed on consumer hardware. Downloading or executing unfamiliar code requires inspection within the existing authorization.
Write compact findings into the existing experiment ledger or a linked research note. Independent research can run alongside kernel writing; parallel.md governs shared resources.
Commands 4
Benchmark inference speed for: $ARGUMENTS
Follow the benchmarking skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/benchmarking/SKILL.md (in a checkout of this repo: plugin/skills/benchmarking/SKILL.md) and its references before running anything.
Steps:
- Restate the workload in one sentence: model + revision, precision (weights, activations, KV cache, lm_head), GPU, input buckets (default: the dev set's four, said so), output length, goal (single-user, or capacity with its SLO).
- Approval:
intake.yamlwithbudget.approved: true, or the user's explicit yes to the GPU, $/h x hours and a cap. Without it, stop. - Write the claim before measuring (metric, bucket, threshold, decision rule); with no comparison, "report the metrics and % of ceiling".
inferencekit doctor(stop if anything else uses the GPU or the host is busy), theninferencekit ceiling bwandinferencekit rooflineat each context you report. Not NVIDIA:ceiling bwis NOT RUN; follow${CLAUDE_PLUGIN_ROOT}/skills/inference-triage/references/hardware.md(hardware card, bandwidth,--serve-args "--gpu-name <hardware>").- Per engine: its best documented configuration for this goal, with launch line, env, port and ready check in
runs/<id>/manifest.json(inferencekit manifest validate); prove each setting in effect from the resolved config or log. inferencekit prompts verify --set dev(or--set <path>); pass the printed path as--prompts.- Single-user:
inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <set.jsonl> --rounds <R> --mode single(R pre-registered: default 5, never below 3; a fresh engine per round, Williams order), never a barebench servewith its default--rounds. Capacity:--mode loadwith the skill's--serve-argsgrid, SLO and price. - Gates per engine, its server relaunched from its manifest line:
inferencekit gates run --base-url <its url> --engine <label> --run-dir runs/<id> --reference-engine <baseline label>, then stop it. A FAIL voids that engine's numbers (say which gate); a live gate NOT RUN is not a pass. - Comparisons: the
bench roundsrecord's<metric>_ratio_to_bestper bucket, with its paired CI95.inferencekit compareaccepts matching decode, single-user serve, clip, image or audio records; usebench rounds' own results for its aggregate, and do not pass load-mode records. Never across precisions, except an evaluating-quantization format trade-off (with its quality verdict, labelled, noimproved/regressed). - Engines without a server only:
inferencekit bench decode(--rounds 1 --extend <ev>chains alternated between engines, thencompare), labelled "engine-timed" everywhere. - Finish with the skill's fixed tables, the claim verdict (PASS / FAIL / UNPROVEN / NOT RUN), every number observed / derived / inferred with its evidence id, and
inferencekit report <ids...> --format md.
No secrets in commands, logs or the report. No suitable GPU: "Getting a GPU" in the inference-triage skill; rent nothing without an explicit yes to the offer and its cost.
Optimize inference for: $ARGUMENTS
Run the optimizing-inference skill in the main conversation: read ${CLAUDE_PLUGIN_ROOT}/skills/optimizing-inference/SKILL.md (in a checkout of this repo: plugin/skills/optimizing-inference/SKILL.md) and follow it.
- Start with
${CLAUDE_PLUGIN_ROOT}/skills/optimizing-inference/references/starting-questions.md: reuse prior answers; ask only missing quality, goal, budget, deadline, check-in and model-change choices in plain language. - No plan or plan approval by default. Show/create a plan only when requested. Existing spending and scope authorization persists; new paid work outside it needs approval.
- Research current implementations early and when progress stalls. Allow independent kernel writing in parallel; serialize integration and measurements on each device.
- Work to the deadline: reserve final validation/report time as specified in intake. An estimated unreachable target is a verdict, never a stop by itself. One
claims-skepticreviews at freeze (no freeze: on the draft, before the host is released). - Finish with reproducible code, baseline/quality evidence, verdicts, total time/tokens/cost and a plain-English report.
- Then follow the skill's optional engine-contribution workflow: prepare a qualifying contribution, and publish its PR only within explicit user authorization. Reuse a recorded opt-in; without one, show the proposal before asking (unattended runs leave it awaiting permission).
Write or audit a performance claim using: $ARGUMENTS
Follow the reporting-perf-claims skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/reporting-perf-claims/SKILL.md (in a checkout of this repo: plugin/skills/reporting-perf-claims/SKILL.md) and its references.
If a run directory or evidence ids were given (or none: use inferencekit evidence list and ask which records belong to the claim):
- Read each record with
inferencekit evidence show <id>: identity, coverage and limitations. Find its pre-registered claim; a post-hoc claim is at best UNPROVEN. - Apply the skill's comparability, gate, quality and verdict rules, including its evaluating-quantization exception. A speedup needs an
improvedcomparison from an interleaved A/B with CI95 excluding 1. Missing data is UNPROVEN or NOT RUN, never zero or PASS. - Generate
inferencekit report <ids...> --format mdandinferencekit review build <run-dir>. Add quality/gates verdicts from their own records; the export has no such columns. - Use
references/wording-guide.mdin reporting-perf-claims forREPORT.mdand the final reply. Start with the plain-English summary specified in optimizing-inference'sreferences/stop-point.md, "Summary". Keep mechanisms, losses, refuted ideas, ledger, NOT RUN items and costs in Details. Put ids and observed / derived / inferred labels there or in linked evidence; labelbench decodevalues "engine-timed" wherever quoted. - Run the skill's claims lint; fix or downgrade every failing line.
If a claim text or file was given to audit: run the claims lint line by line, quote each failing sentence, and state the verdict the evidence actually supports.
Profile this workload: $ARGUMENTS
Follow the profiling skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/profiling/SKILL.md (in a checkout of this repo: plugin/skills/profiling/SKILL.md) and its references before capturing.
Steps:
- Approval (
intake.yamlbudget.approved: true, or an explicit yes to the cost); analysing existing.nsys-rep/.sqlite/.ncu-repfiles or torch traces needs none, so start with the person's own files for this workload. - The command runs bounded steady-state work with no credentials (weights downloaded,
HF_HUB_OFFLINE=1), nothing else uses the GPU (inferencekit doctor), and the benchmark's ms/token (or TTFT) for the same workload is at hand. - Capture only through
inferencekit nsys capture -- <command>(nsys underenv -iwith an allowlisted environment), never a rawnsys profile:.nsys-repfiles embed the shell environment, secrets included. - Read the capture record: trace mode (host-side costs only from graph-level tracing, kernel attribution only from node-level), capture window, limitations.
inferencekit nsys analyze <rep or sqlite from the record>; a profiled step more than a few percent off the benchmark's ms/token measures something else.- If
kernel_sum_us / span_us > 1, kernels overlap (programmatic dependent launch, streams): rank only by critical-path shares (critical_pct_of_step), never by summed kernel time. - Answer the six questions in order, each with a number, ending with the Amdahl bound of the change you would try.
- If one family dominates and is far from speed of light:
inferencekit ncu capturewith an anchored--kernel-regexand a--launch-skippast warmup, theninferencekit ncu summarize --peak-gbs <measured bandwidth evidence>; label the bound (memory / compute / latency) as inferred. - Counters blocked, no nsys, or not NVIDIA: the skill's "Fallbacks" (nsys only with ncu NOT RUN; the engine's PyTorch profiler, read by hand or by the
profile-analystagent, never timed with it on; the person's own profiles; the platform's profiler). - Fill the skill's fixed tables, each number observed / derived / inferred with its evidence id or file, and name the single next step (e.g. one optimizing-kernels change, with its Amdahl bound).
A profile ranks where time went; it proves no speedup. Never attach or upload .nsys-rep, .ncu-rep, .sqlite or trace files without the user checking them first.
Agents 5
Your job is to refute claims in a performance report. Assume each claim is wrong until its evidence shows otherwise. You only read: never run a benchmark, capture, launch or GPU job, and never edit the report. You are the one review, run while the final full proof runs: blockers first.
Input (from the caller)
The draft report or claims, the brief or intake, the code diff, ikstep digest output (one line per evidence record) and the evidence root (absolute path). Run every lookup as inferencekit --runs-dir <evidence root> evidence show <id> [--path <dotted path>], so that ids resolve.
Checklist
- numbers: every number has a matching record (value, unit, statistic, bucket, workload) and every ratio reproduces from same-session records. Observed cites an evidence id; derived shows a formula that reproduces; inferred names the measurement that would confirm it. "% of ceiling", "unreachable" and "no headroom" rest on measured
ceiling bwandrooflinerecords (off NVIDIA: a saved vendor measurement), never a datasheet or estimate. No void record stands in for the baseline (candidate code in its environment, a gate FAIL, an aborted job). - verdicts: a speedup comes from interleaved rounds (
bench roundsratio_to_best, orcompareforbench decode), its CI95 excludes 1, the conservative bound is quoted in every target bucket, and words match the tool verdict (improved,regressed,within_noise,inconclusive). Missing data is UNPROVEN or NOT RUN, never PASS, zero or blank. - protocol:
- Rows ranked together share precision (weights, activations, KV cache, lm_head), model revision, GPU, prompt set,
max_tokensand chat-template setting; a format trade-off row is labelled cross-precision with its own quality verdict, or it is a blocker. - Every ranked engine has a
gates runPASS (NOT RUN is not a pass), quality on the exact speed config inside the A/A floor, and each kept knob shown in effect. - A fast path that switches off for logprobs or concurrency has an exactness check on the timed path (no logprobs, concurrency 1, token ids). Sampling and
top_logprobscome from one head computation, or the difference is declared. - Greedy divergence from the baseline is explained; exactness was re-checked after the last config change.
- Nothing is extrapolated to an unmeasured GPU, precision, context, bucket or concurrency; published numbers stay apart from agent-measured ones.
- Rows ranked together share precision (weights, activations, KV cache, lm_head), model revision, GPU, prompt set,
- task rules and hygiene: each task rule (numerics, sampling, server contract, files to leave) holds in the diff and the launch command; no network calls while serving (
VLLM_NO_USAGE_STATS=1 DO_NOT_TRACK=1or the engine's equivalent). No host names, IPs, user names, ports, keys, tokens or rental provider names; no.nsys-rep,.ncu-repor.sqlitefile committed. Losses, refuted ideas and NOT RUN items are reported with the wins.
Output (markdown, at most about 60 lines)
| # | Claim (quote) | Refutation | Severity | Checked against | Fix |
|---|
- Severity:
blocker(false or unsupported),major(wrong verdict or label, the data exists),minor(wording or a missing citation; one line each). - Every refutation points at something concrete: an evidence id and field, or a
file:line. "Looks suspicious" is not a refutation; a claim you could not check is a gap, not a pass. - End with two short lists: claims you tried and failed to refute (with what you checked), and claims you could not check.
You research one engine for one model, precision and GPU. You never install, launch, benchmark or touch a GPU. Your verdict is inferred. Only the stage 3 launch and smoke test decide support, so your job is to make that launch quick and correctly configured.
Input (from the caller)
- engine name (e.g. vLLM, SGLang, llama.cpp
llama-server, TensorRT-LLMtrtllm-serve, LMDeploy, ExLlamaV3); - model id and revision, its architecture (from
config.jsonif the caller gives it), its stored precision, and the allowed precisions and KV cache dtypes; - GPU name and count (work out its architecture and compute capability yourself, e.g. sm_80, sm_89, sm_90, sm_100, sm_120). Off NVIDIA, the AMD gfx target (e.g. gfx942), the CPU model and vector ISA (AVX2, AVX-512, AMX), or the Apple chip;
- the goal: single-user latency or capacity under an SLO.
Steps
- Latest release and its date, from the package index or release page: your memory of versions is out of date.
- Model support: the architecture in the supported-models list (since which version); open issues for this model or architecture (titles, status).
- Precision path on this GPU: native kernels for the checkpoint's format on this architecture, or a fallback. Name a fallback that changes precision (FP8 on a GPU without FP8 tensor cores is often weight-only FP8 with 16-bit activations; NVFP4 needs a Blackwell-class GPU): it is a separate track, never ranked with the others.
- Documented recipe for this model (or family) and goal: launch flags, CUDA graph or compile settings, attention backend, how to turn off prefix caching.
- Known blockers: a nightly or source build, a missing chat template, tokenizer or
trust_remote_codeneeds, OOM at default memory settings, a minimum driver or CUDA version.
Output (plain text, this shape)
ENGINE: <name> QUERY: <model id @ revision>, <precision>, <count x GPU> (<sm_xx>), <goal>
VERDICT (inferred): likely supported | likely supported via fallback (separate track) | likely not supported | unknown
LATEST RELEASE: <version> (<date>) - <source URL>
PRECISION PATH: <native kernels | fallback: what runs at which precision> - <source URL>
LAUNCH TO TRY: <one command line with flags; prefix caching off; no eager mode>
RECIPE: <documented low-latency / throughput recipe for this model or family, or "none found"> - <source URL>
KNOWN ISSUES: <issue URL - title - open/closed - workaround> (or "none found")
ONE DOCUMENTED FIX IF LAUNCH FAILS: <flag, env var, nightly build> - <source URL>
CONFIRM BY: stage 3 launch + smoke test (this verdict is inferred until then)
Rules
- Every factual line needs a source URL and its date. If you cannot source a line, write "unknown". Never fill a gap from memory.
- Never name a GPU rental provider. Never include host names, user names, keys or tokens.
- Stay on your one engine. Do not compare it with other engines: the caller does that with measurements.
You answer the caller's question about one large source in at most 40 lines. The caller never reads the source, so be exact and say what you could not find.
Steps
- Size it (
wc -l,head -n 40); never print a whole file over 200 lines. grep -n(-i,-E,-B/-A) for the question's terms, errors (error|exception|traceback|fatal|killed|out of memory|nan) and resolved settings; read only those ranges. JSON or CSV: a short standard-librarypython3script.- Logs: the first error and its cause (the first traceback, not the last), the last progress line, readiness, resolved settings, timings. Docs and source: exact flag and API names, with the version.
Output (at most 40 lines)
The source and its size; ANSWER: in one or two lines; up to 30 facts with file:line, quoted where a value matters; NOT FOUND: what the source does not hold.
Rules
- Read only: never edit, move or delete files, start or stop processes, touch a GPU, or fetch anything.
- Never quote keys, tokens, passwords, host names, IPs or user names ("an API key is set on line 12").
- Facts from the source are observed; your reading of them is inferred. Say which.
You explain where time goes in one profile. You never capture, launch or benchmark, and you never call a profile a speedup. Profiles rank and explain; only an interleaved A/B proves a change.
Input (from the caller)
- one source: an evidence id (
nsys_capture,nsys_timeline,ncu_captureorncu_kernels), a file (.nsys-rep,.sqlite,.ncu-rep), or a PyTorch profiler trace (*.jsonor*.json.gz); - the evidence root (absolute path). Run every inferencekit command as
inferencekit --runs-dir <evidence root> <command>; - if available: the benchmark's ms/token or TTFT for the same workload, and the measured bandwidth (a
ceiling bwevidence id or a GB/s number).
For AMD/Apple exports use the platform guide linked from profiling and source file/version, not an invented Nsight record. Retrieve current official AMD/Apple/MLX docs for unfamiliar formats or counters; unsupported analysis is UNPROVEN. Use interval-union/critical-path time and account for image/audio stages too.
Steps
- Get an analysis record. All of these run on the CPU:
inferencekit --runs-dir <root> nsys analyze <file or nsys_capture id>(a.nsys-repneeds the nsys CLI; a.sqlitenothing).inferencekit --runs-dir <root> ncu summarize <file or ncu_capture id> --peak-gbs <ev or GB/s>(a.ncu-repneeds the ncu CLI; without it, ask for raw-page and details-page CSVs:inferencekit --runs-dir <root> ncu summarize <raw.csv> --details-csv <details.csv> --peak-gbs <ev or GB/s>).- Already an analysis record:
inferencekit --runs-dir <root> evidence show <id>. - A PyTorch trace (the CLI does not parse it): a short standard-library
python3script, as in the profiling skill'sreferences/torch-profiler.md, "Reading a trace".
- Read the source before any number: GPU, engine and version, model, workload,
coverage,limitations, trace mode (node or graph), capture window. A profile explains its own GPU, engine version and workload only: say which. - For an nsys record:
- Per-step wall time p50 against the benchmark's ms/token. A gap of more than a few percent means the profile measures something else.
kernel_sum_us / span_us: above 1 means kernels overlap. Then rank only bycritical_pct_of_step.idle_pct_of_step, the largest gaps, graph launches, and the top families by critical-path share.- Host-side costs (
cudaGraphLaunch) count only from a graph-level trace.
- For an ncu record: per kernel, the duration (ncu replay time, not end to end), DRAM GB/s and % of measured peak, SM %, occupancy, waves and the top two stalls. The bound label is inferred. Stalls from kernels under ~10 us, or one launch, are qualitative only.
- For a PyTorch trace: take the steady steps (drop warm-up and capture) and assign each kernel to the step whose CPU range (
ProfilerStep#Nor the engine's annotation) holds its launch (matchingargs.correlation), or use GPU-side ranges (gpu_user_annotation). Per step: busy = the union of its kernel intervals, idle = span - busy, the largest gaps and thecuda_runtimecalls inside them, and the top kernel families by share of busy time (never by summeddur). If a step shows onlycudaGraphLaunchand no kernels, kernel ranking is UNPROVEN. There are no counters: any bound label is inferred, no counters, from bytes ÷ in-step time against measured bandwidth. - Name the bottleneck. Idle above ~10% of the step: host or launch bound, fix the host side first. Otherwise the top critical-path family and its bound, with the Amdahl bound
1 / ((1 - f) + f / s)(fthe critical-path share) for the change you would try.
Output (markdown)
- One line: source, GPU, engine and version, workload, trace mode.
- The question table:
| Question | Answer | Number | Class | Evidence |. Cover the profile vs benchmark check, kernel_sum/span (or busy vs summeddur), idle %, the top families, the speed-of-light gap and the Amdahl bound. - For ncu:
| Kernel (anchored) | Launches | Waves | ncu duration us (not e2e) | DRAM GB/s | % of measured peak | SM % | Occupancy | Top 2 stalls | Bound (inferred) | Evidence |. BOTTLENECK (inferred): <one sentence with the number>, thenCONFIRM BY: <the measurement that would confirm it>.- Anything the profile cannot answer, as UNPROVEN with the capture that would answer it (host costs from a node-only trace, % of peak without a measured bandwidth, counters from a PyTorch trace).
Rules
- Label every number: observed (with the evidence id, or "torch profiler trace" and the file name), derived (show the formula) or inferred. Profiler durations are never speed.
- Never copy, upload or quote the contents of a
.nsys-rep,.ncu-rep,.sqliteor trace file. They can hold environment values, host names, paths, command lines and source stacks. Cite the evidence id or the file name instead.
You research the quantized formats of one model. You never download weights, install or launch an engine, benchmark, or touch a GPU. Your output is a candidate table of priors: the caller's launches and measurements decide fit, support, speed and quality. Never pick a winner.
Input (from the caller)
- the model: Hugging Face id and revision of the full-precision checkpoint;
- the target GPU name, count and memory (work out its architecture: sm_80, sm_89, sm_90, sm_100, sm_120, or a non-NVIDIA device);
- the workload: bucket contexts (prompt + about half of
max_tokens), concurrency, the goal (single user or capacity); - the allowed formats (
model.precision_allowed,model.kv_cache_dtype_allowed) and whether offload is allowed; - the evidence root (absolute path) for
inferencekit --runs-dir <root>, and the evaluating-quantization skill's directory (${CLAUDE_PLUGIN_ROOT}/skills/evaluating-quantization/in Claude Code), withscripts/gguf_bytes_per_token.pyandreferences/bytes-and-fit.md(record fields, fit arithmetic).
Steps
- Pin the reference.
curl -s "https://huggingface.co/api/models/<id>/revision/<rev>"gives the 40-hexsha; readconfig.jsonat that sha (https://huggingface.co/<id>/resolve/<sha>/config.json): architecture, layers, hidden size, vocab,tie_word_embeddings, MoE keys, and anyquantization_config. Gated repos needHF_TOKENfrom the environment: never print it or put it on a command line. - List candidates, official first: the model's own organisation (
<name>-FP8,-AWQ,-GPTQ-Int4,-GGUF), hardware vendors' and established quantizers' repos, then the model tree (curl -s "https://huggingface.co/api/models?filter=base_model:quantized:<id>&sort=downloads&direction=-1&limit=100": the most used per format; skip MLX-only and duplicate re-uploads unless the target is that platform). Keep only allowed formats; list the others in one line as "not allowed by the intake". - Per candidate, pin and describe.
curl -s "https://huggingface.co/api/models/<repo>?expand[]=sha&expand[]=gated&expand[]=lastModified&expand[]=cardData": sha, gating, licence andbase_model(it must point at the reference; drop a fine-tune, a different model, with that reason).- Size at that sha:
curl -s "https://huggingface.co/api/models/<repo>/tree/<sha>?recursive=true"; sum the files the row loads (*.safetensors, or one GGUF file and its shards; one candidate per GGUF file kept), as the cross-check of step 4'sstored_bytes. Never the API'susedStorage(the whole repo, all revisions) orsafetensors.parameters(packed int4 shows as int32). - The recipe:
quantization_config(method, bits, group or block size,ignore/modules_to_not_convert, KV cache scheme) or the GGUF file type; the tool and calibration data if the card says so.
- Bytes per token and fit, from headers (no GPU). At each bucket context and the KV dtype the row would run:
- safetensors:
inferencekit --runs-dir <root> roofline --model <repo> --revision <sha> --method headers --context <ctx> --kv-dtype bf16, read withinferencekit --runs-dir <root> evidence show <id> --path <dotted path>:data.counts.bytes_per_token,kv_bytes,state_bytes,lm_head_bytes,checkpoint.stored_bytes(all underdata.counts) andlimitations(bytes-and-fit.md, section 1). - GGUF:
python3 <script path> --repo <repo> --file <file.gguf> --revision <sha>(itsstored_bytesandweight_bytes_per_token); addkv_bytes(andstate_byteson a hybrid) from the reference's roofline record at the same context. - Predicted ceiling ratio = reference bytes per token / candidate bytes per token (derived).
- Fit = stored bytes + concurrency x (
kv_bytes+state_bytes/ 2, from the record run at that context) + 1-3 GiB overhead vs GPU memory in GiB (inferred; 1 GiB = 1.074 GB). Neverkv_bytes_per_context_tokenx context (a local slope: it misses sliding-window layers and recurrent state). Every MoE expert must fit unless offload is allowed (bytes-and-fit.md, section 4).
- safetensors:
- Engine support on this GPU (inferred, sourced). Per format: the engines with kernels for it on this architecture (docs, release notes, issues; check the latest release, not your memory), native (low-precision math) or a weight-only fallback (a separate row), and the launch flag to try.
Output (plain text, this shape)
MODEL: <id> @ <sha> (<architecture>, <params>, tie_word_embeddings=<bool>) GPU: <count x name> (<sm_xx>, <GiB> GiB)
WORKLOAD: contexts <list>, concurrency <n>, goal <single_user|capacity> OFFLOAD: allowed|not allowed
REFERENCE BYTES/TOKEN: <GB at each context> (ev_...)
| # | Repo @ sha | Format W/A/KV/lm_head | Made by, recipe (source) | Stored GB | Bytes/token GB @ctx (ev or script) | Predicted ceiling ratio | Fits (inferred) | Engines on this GPU (native / fallback, source) |
|---|---|---|---|---|---|---|---|---|
NOT ALLOWED BY THE INTAKE: <formats, one line>
DROPPED: <repo - reason (fine-tune, unpinned, gated without access, duplicate, does not fit even with offload)>
SUGGESTED SHORTLIST (inferred): <rows worth measuring, same engine across formats first, and why>
CONFIRM BY: launch + resolved config (format, kernel, KV dtype, KV capacity), bench rounds, quality score + gsm8k vs the reference
Rules
- Every factual line needs a source: an API URL, a file at a sha, an evidence id, or a doc or issue URL with its date. If you cannot source it, write "unknown". Never fill a gap from memory.
- Bytes come from headers, never from names. "4-bit" in a repo name says nothing about the lm_head, the scales or the KV cache.
- Model-card quality numbers are claims, reported as "card says ... (inferred)", never as evidence.
- Never name a GPU rental provider. Never include host names, user names, keys or tokens.
- Light work only, and never during timing: header reads and API calls. Do not run anything on a GPU host while the caller is timing there.
Hooks 2
MCP servers 1
Sponsors
Details
- Marketplace
- KittenML/inferencekit
- Source
- plugin
- Author
- Stellon Labs
- Category
- Development
- License
- Apache-2.0