Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.

version
0.1.1
GitHub stars
32

Install

/plugin install inferencekit@inferencekit

Run it inside Claude Code after adding the marketplace with /plugin marketplace add KittenML/inferencekit.

Contents

Discovered from source

Skills 9

Benchmarking inference speed

Choose the workload before the metric: complete-request latency, sustained throughput, peak memory or cost at the agreed quality. Use the image/audio route in affordability and video checks for those modalities. Token metrics, text load grids and GSM8K below apply to text; other tasks need a registered adapter, with unsupported checks NOT RUN.

bench rounds --mode audio|image|clip runs fresh-process media comparisons; CPU manifests use hardware.backend: cpu for a CPU lock and process-group telemetry. Compare the final evidence ID for each engine in the run plan; media contracts and round scheduling.

Speed numbers that survive review: a timer outside the engine, one precision for every engine, settings proven in effect, warm engines, fresh-process rounds in a balanced order, confidence intervals, a speed-of-light ceiling and anti-cheat gates. optimizing-inference calls this skill.

Use for latency and tok/s per input bucket, capacity (N* within an SLO, goodput, $/1M tokens), comparisons of engines, configs or builds, and any reported number. A change-loop step needs only a screen; the full protocol is for milestones (fixed rounds: references/statistics.md) and anything published (optimizing-kernels, "Checks"). Not for why it is slow (profiling), nor while builds' outputs differ unchecked (proving-correctness first).

GPU approval: benchmarks, ceiling bw, --gpu-bw measure and gates run only with budget.approved: true in intake.yaml or an explicit yes after you stated the cost (GPU, $/h x hours). Record who approved and when.

Which timer

bench rounds / bench serve (MCP bench_rounds / bench_serve)bench decode (MCP bench_decode)
Timerthe client, time.perf_counter_ns, outside the engine processinside the engine's process (engine-timed)
Enginesany OpenAI-compatible server: vllm serve, SGLang, llama-server, trtllm-serve, a custom enginein-process vLLM LLM, SGLang Engine, llama-bench, or --engine cmd
Modessingle-user (sequential) and open-loop loadbatch 1 only
Published?yes: published numbers come only from hereno: label every number "engine-timed"

bench rounds launches each manifest engine fresh per round, warms it up, runs bench serve once, stops it, checks GPU memory came back and parses the resolved config, in a Williams order under one GPU lock. Why one client timer: an engine that times itself chooses its own boundaries.

Preflight checklist

  • Approval recorded; inferencekit doctor this session -> ev_env.
  • Nothing else on the GPU or host CPUs while timing (they shift latency by tens of percent); the client on pinned cores or another host.
  • Engines and model pinned in runs/<id>/manifest.json; inferencekit manifest validate passes.
  • Same precision (weights, activations, KV, lm_head) for every engine, or a numerics change the task allows, labelled "numerics changed"; the same sampling and seed for every engine (--extra-body: the workload's, else generation_config.json defaults); each baseline at its best documented config for the goal.
  • Prompt set verified (inferencekit prompts verify --set <dev|path>), identical for every engine.
  • Claim written first: metric, bucket, threshold, decision rule, round count R.

Hard failure gates

Any of these makes a record unusable for a speed claim; stop and report it:

  • A setting not proven in effect from the resolved config or log, or CUDA graphs off where the engine supports them.
  • Generated tokens per request differ from the requested max_tokens, or token counts disagree between methods.
  • Repeated prompts inside a load level, or a prefix/response cache serving repeats.
  • Precision, model revision, GPU type or prompt set differ between the engines compared.
  • Fewer completed rounds than pre-registered, rounds not interleaved, or warmups counted. R is intake workload.rounds (default 5, never below 3; kernel-loop milestones excepted), sized with n ~= (2.5c/h)^2 and passed as --rounds R, or an adaptive plan pre-registered as --ci-target, --min-rounds and a maximum --rounds (references/statistics.md).
  • A profiler or debugger attached, or the GPU shared during timing.
  • Any inferencekit gates run FAIL: each gate catches a known way to improve the number without improving the engine.

Workflow (copyable)

Benchmark:
- [ ] 1. Approval recorded; claim and R pre-registered; doctor -> ev_env
- [ ] 2. Ceiling: ceiling bw -> ev_bw; roofline at each context you report -> ev_roof
- [ ] 3. Run manifest: per engine its launch line, env, port, ready_check; best documented config for the goal
- [ ] 4. Prove each knob took effect (resolved config / log line); copy it into manifest resolved_config
- [ ] 5. Single-user: bench rounds --mode single --rounds R  (fresh server per cell, Williams order, warmup excluded)
- [ ] 6. Capacity (if a goal): bench rounds --mode load (grid, SLO, price, one shared --ref-e2e-ms) -> N*
- [ ] 7. Gates, per engine, while its server is up: relaunch it from the manifest, gates run, stop it -> ev_gates_<label>
- [ ] 8. Verdicts per bucket and metric: bench rounds' <metric>_ratio_to_best with its paired CI95 (see Comparisons)
- [ ] 9. % of ceiling per engine (derived); fill the fixed tables; inferencekit report

Exact commands

inferencekit doctor
inferencekit ceiling bw                                                    # -> ev_bw
inferencekit roofline --model <hf-id or config.json> --dtype fp8 --gpu-bw <ev_bw> --context <prompt + gen/2>
inferencekit prompts verify --set dev            # prints path + sha256; or --set <path.jsonl> --expected-sha256 <pre-registered>
inferencekit manifest validate runs/<id>/manifest.json
# single-user. Engines start in the calling directory (a relaunch: the run dir), so with relative launch paths
# cd runs/<id> first and pass --manifest manifest.json
inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <set.jsonl> --rounds <R> --mode single
# capacity: open loop, seeded; --concurrency repeats; the same --ref-e2e-ms gives every engine the same schedule
inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <dev-load set.jsonl> --rounds <R_load> --mode load \
  --serve-args "--concurrency 1 --concurrency 2 --concurrency 4 --concurrency 8 --concurrency 16 --concurrency 32 \
  --concurrency 64 --ref-e2e-ms <ms> --seed 0 --slo-ttft-p95-ms 500 --slo-tpot-p95-ms 50 --gpu-price-usd-h <price>"
# gates: per engine, its server up (launched from the manifest line), then stopped
inferencekit gates run --base-url http://127.0.0.1:8000 --engine <label> --run-dir runs/<id> \
  --reference-engine <baseline label>    # + --spec-off-url <same model, no speculation> if it speculates
inferencekit report <ids...> --format md
# engines without a server only (engine-timed; never published as such):
inferencekit bench decode --engine cmd --cmd '<argv>' --model <m> --prompt-len 1024 --gen-len 256 --rounds 1 --extend <ev_prev>
inferencekit compare <ev_baseline> <ev_candidate> --metric decode_tok_s   # same-kind round records

MCP tools: doctor, measure_bandwidth, roofline, prompts_verify, validate_manifest, bench_rounds, bench_serve, run_gates, report, bench_decode, compare (OpenCode: long steps through the CLI, detached). If bench rounds cannot launch an engine: a fresh server per call, bench serve --rounds 1 --engine-kind <vllm|sglang|llamacpp|trtllm> --label <label> --extend <its previous ev>, in the Williams order yourself (generic sends no exact-length field; the default --rounds passes share one process). Serving method, buckets, SLOs, N*, goodput and cost: references/serving.md.

Prove every knob took effect

Record each setting's requested value, where it was set and the value the engine resolved (config or log line; bench rounds writes resolved_config/<label>.r<k>.json, RESOLVED / PARTIAL / UNPROVEN), copy it into the manifest, and read every warning: a silently dropped setting once turned an A/B into an A/A (references/engine-configs.md).

Ceilings (speed of light)

At batch 1 each decode step reads all weights plus the KV cache, so time per token >= bytes per token / measured DRAM bandwidth. inferencekit ceiling bw measures it (quote the large-buffer read plateau, never a figure above the datasheet or a copy test); inferencekit roofline gives bytes per token. Numbers quoted from the early engine are engine-timed (rule 18). Batched, prefill and non-NVIDIA ceilings: references/ceilings.md; worked example: references/worked-example-qwen3-8b.md; hand-off cost: references/launch-cost.md.

Rounds, order and comparisons

  • Each round starts every engine as a fresh process, warms it up and measures, in a balanced (Williams) order (period k for k engines when k is even, 2k when odd; pick R as a multiple). CIs are bootstrap over rounds, not requests.
  • Served engines: verdicts come from bench rounds' <metric>_ratio_to_best = median(engine) / median(best), with a paired bootstrap CI95, read with compare's rule (improved | regressed | within_noise | inconclusive, pre-registered effect e, default 0.02; references/statistics.md, "Reading the result"). A claim about two engines, neither of them the best one, needs its own bench rounds --engines <baseline>,<candidate> run.
  • compare accepts matching decode, single-user serve, clip, image or audio records. Load mode is refused. Media needs separate quality and lifecycle proof.

Anti-cheat gates (inferencekit gates run, MCP run_gates)

Per engine, against its live server plus the run directory: caching, token counts, output length, unique prompts under load, speculative/MTP exactness (--spec-off-url), hidden logprob and temperature probes, rare tokens. A live gate run after the server is gone is NOT RUN, not a pass (references/reward-hacks.md). Video: references/video-checks.md.

Other hardware (AMD ROCm, x86 CPU, Apple Silicon)

The protocol is unchanged; the differences (hardware name via --serve-args "--gpu-name MI300X --gpu-count 1", GPU-only gates on a CPU host, the memory check, a vendor-measured ceiling under hw/ with ceiling bw NOT RUN even if it starts on ROCm) are in hardware.md.

Fixed result tables

Engine (label) @ versionPrecision W / A / KV / lm_headResolved key settings (evidence)BucketTTFT p50 / p95 msTPOT p50 / p95 msDecode tok/s median [CI95]% of ceilingTimerEvidence
client / engine-timedev_...
Engine (serving config)SLO (TTFT p95, TPOT p95)N*Goodput req/s at N*Output tok/s per GPU at N*$/1M output tokens at N*KneeGatesEvidence
PASS / FAILev_...
Pre-registered claimRatio [CI95] (source: bench rounds / compare)Comparison verdictClaim verdictEvidence
PASS / FAIL / UNPROVEN / NOT RUN

Decision and reporting rules

  • No comparison across precisions; if engines share none, compare at BF16 and report quantised runs against their own ceilings. The one exception is a format trade-off (evaluating-quantization, "The cross-precision rule"): speed alone across precisions is never reported.
  • A speedup needs a ratio whose paired CI95 over interleaved rounds excludes 1. A CI wider than the effect means more rounds, never a conclusion (UNPROVEN, not a tie).
  • Above 100% of the ceiling: stop and find the error (token count, dtype, L2-resident weights). After an exact byte cut, recompute bytes per token for the new storage format first. N* comes from open-loop runs with each engine's best serving config; closed loops hide overload.
  • Latencies, tok/s and N* are observed (cite the record), compare ratios too; ratio_to_best with its CI, % of ceiling, goodput and cost are derived (show the formula); any why is inferred until a profile shows it.
  • bench decode numbers carry "engine-timed" and never sit in a published table next to client-timed ones. Publish losing engines, refuted configurations and NOT RUN buckets too (reporting-perf-claims).

Evaluating quantization

Quantization can reduce memory and traffic, but conversion/dequantization can erase the speed benefit and outputs may change. This skill answers one question with evidence: on this GPU and this workload, which format is fastest at a quality loss the user accepts? The answer is a pair, speed and quality, measured on the same configuration. A faster number on its own is not a result.

Choose the modality before the checks. For image/audio, use affordability; for video, use video checks. Record precision per component, peak weights/activations/scratch memory and complete output work. Text token metrics and GSM8K below apply only to text. Other tasks need a registered quality adapter.

Reuse starting choices: balanced permits numerical changes with preserved task quality; speed permits stronger approximations within agreed bounds. Neither authorizes a different architecture. Search current implementations for the exact device and shapes before assuming a smaller format has a fast kernel.

Use when a format or published quant may be worth it, when fewer bytes per token would help (in any ceiling band; exact byte cuts belong to optimizing-kernels), when the model does not fit or capacity needs KV room, or when optimizing-inference's model.precision_allowed lists several formats. Not for same-precision engine comparisons, training a quantization recipe, or ranking different models.

GPU approval: steps 1-4 are GPU-free. Launching, bench rounds, gates and quality runs occupy a GPU: only with budget.approved: true in intake.yaml or an explicit yes to a stated cost (GPU, $/h x hours, cap), including a full-precision reference on a second, larger GPU.

For consumer VRAM/unified memory limits or minimum-memory baselines under latency and quality gates, also read affordability.

The cross-precision rule (read first)

Evidence rule 8 (evidence-rules.md) forbids comparing speed across precisions as an engine result. This skill allows it under three conditions:

  1. Speed and quality as a pair. Every format row carries its speed with CI and its quality verdict, measured on the exact speed configuration. A row without a quality verdict is NOT RUN on quality and cannot be recommended.
  2. Labelled cross-precision: format trade-off, not an engine comparison. The speed ratio comes from bench rounds (<metric>_ratio_to_best, paired CI95) and is quoted as a ratio with its CI, never with improved / regressed.
  3. Same engine isolates the format (e.g. vLLM BF16 vs FP8 vs AWQ). A row that changes engine and format (e.g. llama.cpp Q4_K_M) is a deployment option, labelled with both changes, never used to explain the format effect.

Inputs (existing intake fields)

model.* (the full-precision reference; precision_allowed bounds the formats, others NOT RUN; kv_cache_dtype_allowed: [fp8] tests an FP8 KV cache), hardware.*, goals.*, buckets and workload.* (contexts and concurrency the KV must hold), budget.wall_clock_h (drop rows to fit, never rounds or quality), fidelity (a format below the stored one needs fidelity: lossy and its precision_allowed / kv_cache_dtype_allowed entry, in its own block: fidelity.md), quality.budget (replaces the tier defaults), quality.reference (a vLLM endpoint for the full-precision model; required when it does not fit the target GPU). constraints: quality margins: <tier> kl=<nats> top1=<fraction> gsm8k=<points> (tiers fp8, int4, gguf5-8, sub4), offload: allowed. No intake: ask for the same facts in one message. Margins are agreed before measuring; non_interactive takes the tier defaults (references/measure-and-score.md, section 4), logged in QUESTIONS.md.

Preflight checklist

  • The claim written before measuring: metric, bucket(s), the speed threshold that makes a format worth it (e.g. "decode tok/s >= 1.3x BF16, medium bucket") and the quality margin per tier. Why: a threshold chosen after the data is a forking-paths result.
  • The reference is the full-precision checkpoint at a pinned 40-hex sha, served by vLLM (scoring uses its prompt_logprobs). Why: two quants agreeing with each other proves nothing.
  • Every candidate pinned to its own repo and 40-hex sha. Why: community quants are re-uploaded under the same name.
  • The KV cache dtype decided per row and identical across rows unless it is what is tested. Why: an FP8 KV cache halves KV bytes on its own.
  • Same prompt set, max_tokens, chat template and temperature 0 for every row and the reference (speed at the workload's sampling).

Hard failure gates

Stop that row (or the study) and report the gate when:

  • the resolved config does not show the format you think you run (dequantized to BF16 at load, an unplanned weight-only fallback kernel);
  • inferencekit gates run FAILs for the row: its speed is void;
  • the quality reference is missing: quality is UNPROVEN and no format may be recommended over the reference;
  • a row FAILs its pre-registered margin: it stays in the table as a loss, never the recommendation;
  • offload would be needed without offload: allowed in constraints: the row is NOT RUN (does not fit), never quietly offloaded.

Workflow (copyable)

Quantization check:
- [ ] 1. Claim, speed threshold and quality margins per tier written down (state.decisions or the plan)
- [ ] 2. Candidates: quant-scout -> repo@sha, format, size, provenance, engines (inferred), one table
- [ ] 3. Fit + ceiling per candidate, GPU-free: roofline --method headers (safetensors) or
         scripts/gguf_bytes_per_token.py (GGUF) -> bytes/token, predicted ceiling ratio vs the reference, VRAM need
- [ ] 4. Shortlist <= 6 (engine, format) rows that fit and are worth timing; every dropped row gets a reason
- [ ] 5. Approval; launch each row, smoke test, prove format + KV dtype from the log; one manifest entry per row
- [ ] 6. Speed: one bench rounds call over all rows (reference row included) -> ratio_to_best + CI; gates per row
- [ ] 7. Quality: reference A/A trajectories; trajectory per row; quality score per margin tier; gsm8k per row,
         paired with the reference
- [ ] 8. Pareto table, recommendation with its trade-off, labels, NOT RUN rows; claims-skeptic on the draft

Steps

1-2. Candidates (GPU-free). Run the quant-scout subagent once per model (Claude Code: inferencekit:quant-scout; elsewhere follow that file yourself): official checkpoints first, then the most used community quants, pinned, with recipe, size and engines (inferred, sourced). It never picks a winner. Formats: references/formats.md.

3. Fit and predicted ceiling (GPU-free). inferencekit roofline --model <repo> --revision <sha> --method headers --context <ctx> --kv-dtype <dtype> --gpu-bw <ev_bw or GB/s> per safetensors candidate at each bucket context; for GGUF, python3 scripts/gguf_bytes_per_token.py --repo <repo> --file <name.gguf> --revision <sha> plus the reference's kv_bytes (and state_bytes) at the same context. Fields, the Qwen3-8B worked example and the batch and compute-bound cases: references/bytes-and-fit.md.

  • Predicted ceiling speed-up = reference bytes per token / candidate bytes per token (derived): an upper bound. 4-bit is never 4x fewer bytes: scales cost bits, lm_head and embeddings often stay BF16, and the KV cache is untouched and grows with context and concurrency.
  • Fit = stored bytes (every expert, the full embedding table) + sequences x (KV at the context + recurrent state, from a roofline record run at that context; never slope x context on sliding-window or hybrid models) + runtime overhead. It is inferred until the engine logs its KV capacity in tokens.
  • Shortlist 3-6 rows (reference included) that fit and clear the threshold with room to spare, same engine across formats first; dropped rows go in the report with the reason.

4-6. Measure speed (same harness, every row). Launch each row with the engine's recipe for that format, prefix caching off, CUDA graphs on, and prove from the startup log the quantization method, GEMM kernel family and KV dtype it resolved (a weight-only fallback is its own row). One manifest holds every row, each downloading its own pinned checkpoint (references/measure-and-score.md, sections 2-3). One bench rounds call interleaves all rows for paired ratio CIs; for capacity measure N* per row (KV room often decides it, and W4A16 can lose once compute-bound). inferencekit gates run per row while its server is up.

7. Score quality against the full-precision reference (proving-correctness; commands in references/measure-and-score.md, section 5): reference trajectories twice (A/A floor), one per row on its exact speed configuration, quality score per margin tier (below 4 bits: the 4-bit margins unless a sub4 margin was agreed first), quality gsm8k per row paired with the reference (reasoning often degrades before KL looks alarming), quality niah for FP8 KV or long-context claims. Without logprobs the trajectory is text only: top-1 and NLL are scored by re-tokenizing it, KL and TV are not; GSM8K still counts.

8. Report. The Pareto table (all rows, losses, misfits and NOT RUN included) and the recommendation: the fastest row by the conservative end of its speed CI whose quality is PASS on every pre-registered check and which fits. If no quantized row passes, recommend the reference precision; if the fastest passing row's ratio CI includes 1, the format has no proven speed benefit (state any memory benefit). One sentence with both numbers, labelled. Table, frontier rule and wording: references/measure-and-score.md, sections 6-7.

Big model on a small GPU (references/small-gpu.md): fit every candidate at the claim's context and concurrency; start with the least aggressive format; quality needs a costed vLLM reference on a larger GPU (none: quality UNPROVEN in the headline); offload only with offload: allowed, labelled and measured, never projected.

MCP tools: roofline, inspect_evidence, validate_manifest, bench_rounds, run_gates, quality_trajectory, quality_score, quality_gsm8k, quality_niah, report. Check flag spellings with inferencekit <command> --help.

Decision and reporting rules

  • No recommendation without a quality verdict from the full-precision reference (the fastest format is the one that computes least). Quote the conservative end of each speed CI.
  • Predicted ratios never stand in for measured ones, and never appear in the measured column. Do not extrapolate a result to another GPU, engine version, context, bucket or concurrency. Model-card quality claims are inferred context, never evidence.
  • Bytes per token, predicted ratios and fit are derived or inferred; tok/s, KL, agreement and GSM8K are observed with evidence ids; bench rounds ratios are derived with their CI.
  • Verdicts PASS / FAIL / UNPROVEN / NOT RUN; a trade-off claim passes only when both its speed and quality halves pass. Publish the losing formats, misfits and NOT RUN rows with the winner (reporting-perf-claims).

Inference triage

Entry point. Turn a vague request into one measurable question, collect the cheapest evidence that answers it, then hand off to exactly one skill. The rules here apply to every other skill in this plugin.

Which skill

The requestSkill
Current techniques, implementation research, a stalled optimizationresearching-optimizations
"Make this model fast", "which engine should I use", "fastest config for my model on my GPU", a target (tok/s, TTFT, users per GPU, $/1M tokens)optimizing-inference: the end-to-end flow
A speed number: tok/s, TTFT, per-token latency, capacity N*, engine vs engine, before vs after, % of ceilingbenchmarking
Where the time goes; why one kernel is slowprofiling
Make an engine or kernel faster (custom engine, engine patch, decode or prefill)optimizing-kernels
Is it still the same model; did quality regressproving-correctness
Fit consumer memory while meeting latency and qualityoptimizing-inference, affordability
Should I quantize; is FP8 / 4-bit / a GGUF quant faster at acceptable quality; which quant; will it fit my GPU; a big model on a small GPUevaluating-quantization
Write up, review or audit a claimreporting-perf-claims
Not surestay here: run the workflow below

When to use / not use

Use when the right skill or the question is unclear ("why is it slow", "is X tok/s good", "did my change help"), or a number is about to be quoted unchecked against a ceiling. Not when the request clearly matches a row above (go there), or to rank different models by quality (model evaluation, not equivalence).

GPU approval (every skill)

GPU work (benchmarks, captures, ceiling bw, --gpu-bw measure, quality runs, renting) runs only with budget.approved: true in intake.yaml or an explicit yes after you stated the cost (GPU, $/h x hours, cap): it occupies shared hardware and spends money. Analysing existing evidence, roofline arithmetic with a numeric bandwidth, and reporting need no approval.

Explicit session authorization persists. An approved finite-deadline run may specify budget.spend_cap: none, stage_timeboxes: false and report_reserve_h: 0; these run-scoped overrides replace the respective defaults without repeated approval questions.

Getting a GPU

  • Your own GPU, or one rented by the hour from any cloud GPU marketplace; never name or recommend a provider in shared files.
  • Rent only after an explicit yes to the GPU model, $/h x hours and a cost cap, or within a pre-approved budget (end to end: optimizing-inference's opening questions; existing authorization persists).
  • Need ncu counters? Prefer a VM or bare-metal offer. Containers and many cloud instances (A100 and H100 included) block them; inferencekit doctor reports it (data.profiling_counters.status). If blocked: nsys only with ncu NOT RUN, then the PyTorch profiler (no counters needed) and the person's own profiles (profiling, "Fallbacks").
  • Record GPU model, count, driver, CUDA and price/h (inferencekit doctor, and the price in the intake's hardware.price_usd_h).
  • Keep the provider's credentials out of commands, logs and profiles (capture only through inferencekit, under env -i).
  • Copy runs/ back, destroy the instance after use (not just stop it), and confirm none is left running.

Not NVIDIA? (AMD ROCm, x86 CPU, Apple Silicon)

NVIDIA is fully supported, and the only hardware inferencekit has been run on so far. On other hardware, the skills guide the same flow, and the CLI does less: references/hardware.md has the table.

  • Hardware-neutral, not yet run off NVIDIA: bench rounds, bench serve, gates run and quality (they talk HTTP to any OpenAI-compatible server), roofline with a numeric --gpu-bw, and the intake, manifest, ledger and report tools. Treat a first run there as a test.
  • NOT RUN off NVIDIA: ceiling bw and nsys/ncu captures. doctor records selected AMD DRM/rocminfo and Apple sysctl/system_profiler facts with a vendor profiling route; unknown details remain unknown. On AMD, rocprof capture and rocprof analyze create trace evidence with rocprofv3; device/version support must be checked. Use current official docs for unfamiliar tools and a vendor or STREAM-style bandwidth test. Captures made outside the CLI are vendor evidence files.
  • Platform guides: AMD ROCm, x86 CPU, Apple Silicon, Intel GPU and TPU.

Preflight checklist

  • One sentence names the workload: model id + revision, precision (weights / activations / KV cache), GPU, input bucket(s), output length, and the goal (single-user latency or capacity under an SLO). Why: a number without its workload cannot be reproduced or compared.
  • inferencekit doctor has run on the target machine this session (not NVIDIA: plus the hardware card from references/hardware.md). Why: driver, clocks, counter permissions and engine versions decide which captures are possible.
  • You know whether anything else uses the GPU or host CPUs. Why: shared GPUs and busy "spare" CPUs silently slow timed runs.
  • The goal is a falsifiable claim (e.g. "candidate decode tok/s >= 1.10x baseline, medium bucket" or "N* >= 32 at p95 TTFT <= 300 ms"). Why: a claim chosen after the data is a forking-paths result.

Hard failure gates

Stop and report the gate instead of continuing when:

  • inferencekit doctor shows no usable GPU, a mismatched driver/CUDA pair, or an unknown engine version. On AMD, CPU or Apple hardware, empty GPU facts are expected: continue with references/hardware.md instead.
  • The two sides of a comparison differ in precision, model revision, GPU type or workload.
  • A secret (API key, token) would end up in a capture, log or report.
  • GPU-occupying work is about to run without recorded approval.

Workflow (copyable)

Triage:
- [ ] 1. Restate the request as one question + one pre-registered claim
- [ ] 2. End-to-end "make it fast" request? -> optimizing-inference, done
- [ ] 3. inferencekit doctor -> ev_env
- [ ] 4. inferencekit roofline -> bytes/token and ceiling tok/s (no GPU needed with a numeric --gpu-bw)
- [ ] 5. Existing evidence first: inferencekit evidence list; profiles the person already has
- [ ] 6. % of ceiling = measured tok/s / ceiling tok/s (derived), if a measurement exists
- [ ] 7. Pick ONE skill from the symptom table; say why in one line
- [ ] 8. Hand off with: workload sentence, evidence ids, the claim, what is still NOT RUN

Exact commands

StepCLIMCP tool
Environment cardinferencekit doctordoctor
Ceiling (arithmetic only)inferencekit roofline --model Qwen/Qwen3-8B-FP8 --dtype fp8 --gpu-bw <measured GB/s or ev_bw>roofline
Existing evidenceinferencekit evidence list / inferencekit evidence show <id>list_evidence / inspect_evidence
Intake for the end-to-end flowinferencekit intake template / inferencekit intake validate intake.yamlintake_template / intake_validate
Reportinferencekit report <ids...> --format mdreport
Analyse a profile the person supplied (no GPU)inferencekit nsys analyze <file> / inferencekit ncu summarize <file> --peak-gbs <ev or GB/s>analyze_nsys / summarize_ncu

MCP clients prefix tool names (Claude Code: mcp__plugin_inferencekit_inferencekit__roofline); the arguments are the CLI's. MCP server does not start (no uvx, no network): the CLI at this plugin's version, uv tool install "inferencekit-cli==0.1.1". In OpenCode, long steps (bench rounds, gates run, quality runs, ceiling bw, captures) go through the CLI, detached with a log: it ends an MCP call after the server's timeout (120 s), and the server sends no progress. Bandwidth: your GPU's measured inferencekit ceiling bw (off NVIDIA, the platform's micro-benchmark: references/hardware.md), or say the ceiling is datasheet-based and therefore unreachable.

Symptom routing

Symptom or questionCheapest first evidenceHand off to
"Make it fast", "which engine", a target to hitintakeoptimizing-inference
"How fast could this possibly go?"roofline with a known bandwidthbenchmarking (ceilings)
"What tok/s / TTFT do we get?", engine vs enginedoctor + rooflinebenchmarking
"How many users per GPU?", p95 latency rises under load, cost per tokenthe SLO and traffic shapebenchmarking (capacity, N*)
Below ~85% of the ceiling (bands), cause unknownone nsys capture of the steady stepprofiling
"I already have a profile" (.nsys-rep, .sqlite, .ncu-rep)inferencekit nsys analyze <file> / inferencekit ncu summarize <file> (no GPU, no approval)profiling
"I have a PyTorch profiler trace" (*.json.gz from vLLM, SGLang or torch.profiler)read by hand or with profile-analyst: the CLI does not parse it (no GPU, no approval)profiling
GPU idle between kernels, high kernel countnsys per-step busy/idleprofiling, then optimizing-kernels
One kernel or family dominates the stepnsys critical-path sharesprofiling (ncu)
"Make this engine/kernel faster" with evidence in handceiling + profileoptimizing-kernels
TTFT slow (prefill)TTFT per bucket, first-seen vs repeated lengthbenchmarking, then optimizing-kernels (prefill)
Outputs changed; before any speedup claimtrajectory score vs the noise floorproving-correctness
"Should I quantize?", "is 4-bit faster?", "which quant?", "will it fit?", near the ceiling and only fewer bytes can helproofline --method headers per checkpoint (no GPU): bytes/token and fitevaluating-quantization
"Did my change help?" / regression huntinterleaved A/B (bench rounds; compare for bench decode)benchmarking, then reporting-perf-claims
Writing up a result, PR, blog postthe evidence idsreporting-perf-claims

Ceiling bands

One scheme for every skill in this plugin: % of a measured-bandwidth ceiling, for single-user decode. These are heuristics that pick the next measurement, not verdicts.

  • Above 100%: impossible; the ceiling, the bytes per token or the token count is wrong.
  • ~85% or more: near the floor. Only reading fewer bytes per token helps (an exact byte cut, or a lower precision: evaluating-quantization).
  • ~50% to ~85%: look for launch gaps and under-filled kernels (profiling, then optimizing-kernels).
  • Below ~50%: suspect eager mode, host syncs or a CPU-bound loop.
  • In every band the ceiling can move (exact byte cuts, allowed speculation): estimate both at the start (optimizing-inference's headroom estimates) and work them alongside kernel fixes.

The table with the next evidence for each band: references/symptom-map.md.

Artifact/evidence contract

  • Every inferencekit operation writes an evidence record (schema v2; v1 records still read) and returns its id. Why: the id is the citation; a number without one is hearsay.
  • Read identity, coverage.complete, coverage.missing and limitations before quoting any metric. Why: a record can be valid and still not cover your claim.
  • Never paste raw environment values; inferencekit stores none, and you must not add them.
  • Global rules with their reasons: references/evidence-rules.md (the numbered rules are also in the repository's AGENTS.md).

Fixed result table

FactValueClassEvidence idNote
Workload and goale.g. Qwen3-8B FP8 W / BF16 KV, medium bucket, single-userobservedev_...from identity.workload
GPU, driver, clocksobservedev_...doctor
Ceiling tok/sderivedev_...bytes/token and bandwidth source
Measured value... or NOT RUNobservedev_...statistic, n
% of ceilingderived(both ids)formula shown
Next skillinferred—one-line reason

Decision and no-go rules

  • Measure before optimising. Why: most "obvious" bottlenecks are not where the time goes.
  • One skill at a time. Why: profiling during benchmarking contaminates both.
  • No speedup claim without a quality verdict. Why: the fastest kernel is the one that skips work.
  • Measured above the ceiling: stop; the bytes/token model, the bandwidth or the token count is wrong.
  • Datasheet bandwidth is a label, not a ceiling.

Reporting rules

  • Label every number observed (a tool measured it; cite the evidence id), derived (arithmetic; show the formula) or inferred (reasoning; say what would confirm it).
  • Claim verdicts are PASS / FAIL / UNPROVEN / NOT RUN; missing data is UNPROVEN or NOT RUN, never zero and never PASS. Comparison verdicts come from the tools (bench rounds ratio CIs for served engines, inferencekit compare for bench decode), never from eyeballed medians.

Optimize inference

Deliver reproducible code and evidence against the strongest tuned, supported baseline under the agreed quality policy. Record time, tokens/API cost and device cost. Unsupported checks are NOT RUN; missing proof is UNPROVEN.

Rules throughout the run

  • Reuse the unmodified inferencekit harness and datasets. A missing modality adapter is a recorded limitation; do not disguise a custom timer as harness evidence.
  • Fix input distribution, output work, sampling/seeds, component precisions and quality protocol before comparing. Treat baseline and candidate symmetrically. Numerical changes follow fidelity; cross-precision results disclose both formats and quality.
  • Launch pinned implementations to establish support. Verify each knob in the resolved runtime config; an ignored flag is a finding.
  • Keep every attempt, including failures, in the state and ledger. Label numbers observed, derived or inferred and claims PASS / FAIL / UNPROVEN / NOT RUN.
  • Check inferencekit intake budget runs/<run-id> before long steps. Preserve existing spending authorization; never expand it silently. Keep secrets, access details and provider names out of artifacts.
  • For text, keep prefix caching off in the cold-request track; never use --no-cache-salt. Reuse needs a separate realistic workload. Speculation needs its own exactness proof when called lossless.
  • Reuse existing engine features, compiler output, harness and resource supervisor before adding code. Run the smallest experiment that can disprove a hypothesis. Avoid unused helpers, headers, configuration and abstractions. Quick screens guide exploration; complete proof belongs to finalists.

0. A few questions, then work

Use starting questions. Reuse answers in the conversation and intake; ask only missing decisions in short batches. Do not require technical metrics, YAML or a plan from the person.

Quality has three choices: Exactly the same, Same quality, small differences, Prioritize speed. Separately establish permission to change the model itself, the main objective, spending, deadline and desired check-ins. A speed preference does not authorize architecture changes or unchecked quality loss.

No plan or plan approval is required by default. Show or create one only when requested; plan.md covers that optional path and legacy runs. Start once required choices and resource authorization are established. Record their source in state; do not ask again within that scope. For unattended runs use non-interactive.md.

Start or resume

Existing run: first call inferencekit intake resume runs/<run-id>; inspect files for reported issues or required details (state.md).

Fresh run:

inferencekit intake template > intake.yaml   # only when no intake was supplied
inferencekit intake validate intake.yaml --allow-placeholders
inferencekit intake validate intake.yaml --run-dir runs/<run-id>
export INFERENCEKIT_RUNS_DIR="$PWD/runs/<run-id>/evidence"
inferencekit intake budget runs/<run-id>

Use the derived run id; --run-dir pins the intake and starts the clock. SSH runtime and client share the device host.

1. Define the actual task

Validate intake; pin revision, component precisions, processing, dataset, shapes and output work. Derive workload details from examples/logs (workload.md); record assumptions.

Choose response latency, sustained throughput, peak memory or affordable hardware under a latency/quality limit (affordability.md). Token metrics and GSM8K belong to text workloads. Speech needs content, completion and speaker checks; images need semantic/visual checks; video needs temporal checks. Other architectures need a registered task-specific adapter and protocol; unsupported CLI modalities remain UNPROVEN rather than being relabelled as text.

Use researching-optimizations now, from the pinned model, shapes, hardware and runtime. It returns a short experiment queue, including existing engine paths to try before custom code.

2. Hardware, resource limits and headroom

Run inferencekit doctor. Check competing work, memory, swap, thermals and profiler permissions before occupying the device. Stop only the run's own jobs; do not kill unrelated services to obtain a clean measurement.

On NVIDIA, measured ceiling bw and roofline inform supported text models. Elsewhere use the hardware route and vendor measurements; a PyTorch CUDA namespace does not establish NVIDIA-tool support. For non-token workloads derive bytes/FLOPs per request or step with assumptions. Unsupported ceilings stay NOT RUN.

Estimate critical-path share, attainable efficiency, exact byte savings and permitted algorithmic savings (ceilings.md, stop-point.md). Recompute after changing precision or work. An inferred ceiling orders experiments; it cannot establish a speedup.

3. Establish the strongest baseline

Search the web at the start of each run for current engines and recipes for the exact model, hardware and goal; refresh when these or the runtime change. candidates.md gives examples, not a fixed list or count. Launch compatible engines in isolated environments; record sources, versions, commands, resolved configs and smoke results. NVIDIA graphs are not a generic launch flag for AMD or Apple.

Tune supported engines for the goal. A task-fixed baseline is labelled fixed, never automatically best in the field. Reconsider numerical-only baseline rejects under a newly authorized lossy policy.

For text, a fixed baseline with route build may skip the survey and use the engine skeleton. Other models use their native pipeline and custom-engine contract, not the text skeleton.

4. Screen and measure

Use harness-commands.md for text, affordability.md for image/audio and video.md for video. Screen representative cases before full rounds. Text screens include short and long lengths. Promote secondary wins the user values without hiding the lead metric.

Final comparisons use balanced fresh-process rounds, excluded warmup, registered stopping and confidence intervals. Include complete output work and transfers/synchronization/decoding. Capacity needs its own tuning and measured concurrency.

5. Quality on the exact speed configuration

Load proving-correctness and the selected policy. Exact-output claims need equality on the declared deterministic/seeded contract; a lossless label alone does not prove it. Lossy tracks need agreed task/perceptual bounds, not accidental bitwise rejection. Preserve structural validity, finite outputs and completion in every track.

For text, run applicable gates, reference trajectory/score, full task evaluation and sampling checks. Image/audio/video use registered semantic, structural and perceptual checks instead of token gates or GSM8K. Missing human listening/viewing remains UNPROVEN. Approval frequency changes when to ask, never whether quality has been established.

6. Choose the next experiment

Rank experiments by headroom, cost and confidence. Refresh research after two stalled attempts or changed bottleneck/toolchain. optimizing-kernels routes compiler inspection and conditional techniques.

At a met target follow goals.after_target. Otherwise continue within the approved scope and stop rule. With no target, honor the agreed check-in preference; do not repeatedly request approval for an already authorized search.

Default stage time boxes and reporting reserve come from intake. Explicit budget.stage_timeboxes: false or report_reserve_h: 0 override them. A finite deadline still binds. "Unreachable" and "no headroom" are verdicts, never stops under a stated deadline; pursue another useful lever while resources remain. Unavailable devices, evidence or permission are recorded honestly.

7. Implement and investigate

Use compiler inspection: source → graph → dispatch → executed kernel. Check fusion/fallbacks/conversions, form one hypothesis, change one cause, screen quality and measure the complete inference path. Inspect IR/assembly when useful.

Run independent implementation experiments in parallel under parallel.md. Integrate one candidate at a time; serialize device measurements and avoid host contention. Reuse previous numeric-only rejects when policy broadens, with new evaluation and preserved history (fidelity.md).

Keep a commit and screen evidence per retained change. Milestone/final checks follow state.md: text uses existing light/full jobs; other modalities use registered checks. Final proof applies to the commit you hand over. No full evaluation after every small edit; no published claim from a screen alone.

At freeze, one claims-skeptic reviews draft, diff, intake and evidence while final proof runs. Answer refutations with evidence. With zero reserve, keep validation and packaging current through the deadline.

8. Final report (always runs, from the report reserve, not before)

Use the shared report format and final reply: Result, Run it, Files and outputs, then Details. The plain-English summary requirements are in stop-point.md, "Summary". Separate end-to-end from kernel gains, exact from cross-precision tracks, and screened from verified results. Keep material limits visible in Result; retain unsuccessful attempts and time/cost accounting in Details.

Keep the manifest, reproduce commands, ledger and report consistent with the final commit. Run inferencekit review build runs/<run-id> and inferencekit manifest validate runs/<run-id>/manifest.json --strict. No freeze (no stage 7): one claims-skeptic on the draft before the host is released. Push only within explicit authorization. Record final spend and release resources as authorized.

After the final report, check the result for an optional engine contribution. Prepare a qualifying contribution locally, then open its PR when the user authorizes publication; reuse an explicit opt-in already recorded for this run. A declined contribution stays local. Missing proof or publication access does not restart optimization or hold rented hardware open.

Parallel work

Shipped read-only agents: engine-scout, quant-scout, profile-analyst, claims-skeptic, log-reader. Implementation workers may write kernels concurrently in isolated files/branches. One coordinator integrates; one owner per device serializes measurements. See parallel.md.

Optimising kernels: the profile-driven change loop

Start from the measured critical path: host overhead, synchronization, poor memory/compute utilization, redundant traffic or repeated work. These losses occur in text, vision, audio and other models. This skill closes them one change at a time, each checked fast and kept as a revertible commit with its numbers; the full proof runs at milestones (optimizing-inference stage 7).

When to use / not use

Use when:

  • you have a ceiling, a benchmark and a profile, and want to close the gap (decode, prefill/TTFT, or the batched step);
  • you are building a custom engine, patching an engine, or reviewing a kernel change.

Do not use when:

  • you have not measured yet (inference-triage first: most guesses about the bottleneck are wrong);
  • an engine flag might close the gap: tune engines first (benchmarking, engine-configs).

Authorization and parallel implementation

Reuse the run's starting choices, including quality and model-change permissions. Ask missing choices only when this skill is invoked alone. Device work stays inside existing authorization; inferencekit intake budget runs/<run-id> checks remaining resources, not a requirement to ask again.

Independent workers may write kernels in parallel in distinct files/branches. One coordinator integrates them sequentially, and one owner per device serializes warmup, captures, quality and timing. CPU-heavy builds on the benchmark host wait for measurements to finish (parallel.md).

Hardware and workload

Start with doctor and the hardware route. Use Nsight on NVIDIA, supported ROCm tracing/counters on AMD, Instruments/Metal/MLX capture on Apple, or a CPU profiler. For an unfamiliar architecture, runtime, flag or error, retrieve current official docs and verify installed support before selecting a tool. Keep unsupported checks NOT RUN; preserve available timeline or operator attribution with its tool version and source artifact, following profiling.

The CUDA skeleton, commands and kernel recipes below are NVIDIA examples. Use the target engine's native backend elsewhere; a CUDA API name in PyTorch does not establish CUDA-tool support on ROCm. For images and speech, use the registered media adapter, structural and semantic checks, and complete request timing in affordability. Token checks, GSM8K and the ikstep LLM jobs below do not prove media correctness. Keep the same fast-check, milestone and full-proof stages with the workload's own checks.

When the user authorizes numerical changes, judge image/speech kernels against the declared modality quality budget and hash-bound external protocol. Different output bytes alone do not reject that track; exact identity remains a pure-refactor requirement or a diagnostic. Each intermediate change gets one short fixed public correctness/quality screen; promoted milestones and the final candidate need full registered quality and balanced timing. Preserve own-output integrity and all cases. Do not infer permission to prune architecture or use another model from a lossy budget.

An explicit perceptual/task acceptance policy may make numerical similarity diagnostic as well. In that case, do not discard a kernel or baseline solely for LPIPS/SSIM, cross-engine tensor/PCM equality or rounding drift. Measure the prospectively chosen mathematical controls and task/perceptual checks; fresh blinded quality is still required for promotion. No missing perceptual evidence becomes PASS.

Research and inspect before implementing

Load researching-optimizations before selecting custom work and again when progress stalls or the bottleneck moves. Check existing engine features and emitted compiler code first. Follow compiler inspection from source through graph and dispatch to the executed kernel; inspect IR/assembly when it can test a specific hypothesis.

Use the smallest useful experiment, reuse the harness and supervisor, and keep the old path runnable. Avoid unused helpers, headers and abstractions. Independent experiments can be written concurrently; each integrated A/B isolates one change.

Preflight checklist

  • Ceiling and % of ceiling known (ev_roof, ev_bench). Why: the size of the prize and when to stop.
  • A profile of the steady step with critical-path shares (ev_tl, profiling). Why: tells you which of the four losses dominates.
  • A correctness baseline: reference outputs and the A/A noise floor (proving-correctness). Why: every change below can silently break numerics.
  • A harness that A/Bs two builds in interleaved fresh-process rounds. Why: the end-to-end effect is the only one that counts.
  • Record build versus patch and both headroom bounds (stop-point.md). Any new engine meets the custom-engine contract; the bundled skeleton covers NVIDIA text only.
  • A repo and branch for the ledger (perf/<model>-<hardware>, from the intake's repo), and runs/<run-id>/experiments.jsonl. Why: every kept change becomes a revertible checkpoint; nothing tried is lost.

Build route: NVIDIA text engine skeleton

The plugin ships a correctness-first engine (skeleton/README.md) that already meets the custom-engine contract: an OpenAI streaming server, seeded sampling, top-20 logprobs from the full output layer, a checkpoint loader, a CUDA-graph decode loop and torch modules per architecture.

  1. Copy skeleton/ikengine/ (next to this file; ${CLAUDE_PLUGIN_ROOT}/skills/optimizing-kernels/ when installed) into the run directory and commit it as the first ledger commit.
  2. Run its reference server on the model (python -m ikengine.serve --model <dir> --port <port>) and validate it as its README says: its model check, the gates, and quality against the reference. It is your correctness anchor and A/A partner. No module for the architecture: write one against the Model contract in ikengine/engine_api.py, correct first.
  3. Replace internals behind the Model contract one change at a time (the loop below): kernels, fusion, byte cuts. Keep the server, the sampler, the logprob path and the proofs.
  4. Never ship or claim a build without its proofs (below).

Checks and implementation loop

  1. Profile the supported platform once per lever area; repeat when the bottleneck moves. Identify the critical-path limiter, not summed kernel duration.
  2. Read compiler/dispatch evidence and current research. Choose one hypothesis with a headroom bound and a falsifying test.
  3. Implement behind a switch. Keep the old path runnable; independent alternatives may be written concurrently.
  4. Run one fast screen on representative inputs, including awkward shapes. Exact changes preserve the declared outputs. Legacy lossless text changes use token/logprob checks with near-tie analysis; authorized lossy changes use their task/perceptual screen. Compare against a fixed reference as well as the previous build to catch cumulative drift.
  5. Screen the complete step/request using the existing timer. Keep a reversible commit and numbers, or log the rejection. A faster microkernel alone is insufficient.
  6. Promote promising milestones with the state.md schedule: text uses ikstep job light; other modalities use registered checks. Baseline proof is reused until its config changes.
  7. Run full proof once for the final candidate and any commit handed over (ikstep job full for text). A claimed change needs interleaved A/B confidence intervals and a profile showing the mechanism. Missing reference: use the no-reference route.
  8. Follow the approved stop rule and goals.after_target. A stated deadline: work to it minus the report reserve. Explicit zero reserve means continuous validation/packaging. Log the next untried lever and why.

Do not run full gates, task evaluations, balanced rounds or a new profile after every edit. Exploration screens never become published proof. Final timing includes the complete client request; label step/microkernel measurements separately.

Commit trailers and retained/rejected experiment records: commit-ledger.md. Use the platform profiler and registered client timer; text CLI examples are in harness-commands.md.

Choose a technique from evidence

These apply across model families; select by the measured problem, not a list of fashionable names. For each, estimate headroom, confirm the mechanism, screen quality, then time the whole request.

Observed problemNVIDIAAMDAppleTradeoff / proof
CPU launches and tiny kernelsCUDA Graphs, fusion, device samplingSupported HIP graph/runtime capture, Triton/HIP fusionReuse Metal command submission, MLX compiled/fused pathsVerify fewer gaps; include dispatch and synchronization. Sampling applies only where the workload samples.
Repeated intermediate trafficProducer-consumer fusion, layout/producer quantizationHIP/Triton/CK layouts and fused epiloguesMetal/MLX fusion, layouts and quantized producersMeasure bytes and end-to-end cost; quantization needs permission. Fusion can increase registers.
Poor matrix utilizationShape-specific GEMM/GEMV, tiling, split-K, supported matrix instructionsgfx-specific MFMA/WMMA, CK/Triton/other supported compilerMetal SIMD-group matrices or native MLX kernelsInclude packing/reduction overhead and shape tails; matrix paths are generation/dtype specific.
Synchronization dominatesPersistent kernels or megakernelsSupported persistent HIP kernels with explicit synchronizationSupported persistent/threadgroup design; do not assume grid-wide barriersCheck residency, occupancy, deadlock hazards and actual saved critical path. HazyResearch examples are hardware-specific research.
Model pages or barely fitsPacking, cache sizing, expert residency, staged loading/offloadSame memory strategies with supported allocator/transfer pathsUnified-memory working set, packing, cache limits and staged loadingPeak host/device memory, swap, complete latency and quality; fitting alone is not a latency win.
Repeated computationValidated state/prefix reuse or incremental updatesSame algorithm with native runtime stateSame algorithm with native runtime stateReset/invalidation across requests and changing inputs; approximate reuse needs quality permission and a separate realistic workload.

See research sources for current implementations. CPU routes use the same hypotheses with vectorization, threading and cache-aware libraries.

NVIDIA text playbook: decode (memory-bound; batch 1 examples)

TechniqueLoss it closesMechanism evidenceDeep dive
CUDA graph for the whole stephost launch overheadnsys: one graph launch per step, idle gaps gonelaunch-overhead.md
GPU-side sampling, no per-token host syncCPU on the critical pathnsys: no blocking D2H before the next launchlaunch-overhead.md
Fusion: norm+RoPE+KV write, one QKV GEMV, gate-up with SiLU*up epilogue, residual+norm, act-quant into the producerhand-offs (kernel count)nsys: kernels per step down, step wall downlaunch-overhead.md
FP8 block-scaled GEMV at M=1 (many blocks, 16-byte loads, per-block scales)library GEMM tiles that cannot fill the GPU at M=1ncu: DRAM % of measured peak up, waves >= severalfp8-gemv.md
Split-K decode attention (flash-decoding)too few blocks at batch 1, long contextncu: waves up; nsys: attention's critical share downattention-split-k.md
Programmatic dependent launch, trigger placed lateremaining hand-off between dependent kernelsstep wall down in A/B; overlap between kernels shrinkslaunch-overhead.md
Cheaper kernel prologue (hoist weight loads before the PDL wait, stage inputs asynchronously)fixed per-kernel setup before streaming startsprologue-only variant timing; in-step time above the bandwidth ideallessons.md
lm_head: precision, exact bound pruning (below), fusion with samplingoutput-layer bytes measured for this checkpointbytes/token down; quality verdict at the logitsfp8-gemv.md, byte-cuts.md
Exact byte cuts: lossless re-encoding of 16-bit weights, bound-pruned output layerbytes you did not need to read (moves the ceiling)bytes/token down; proof tests pass; % of the recomputed ceilingbyte-cuts.md
MoE: gather-GEMV over the top-k experts, routing fused into the first expert kernel, top-k-order sumlibrary grouped-GEMM tiles and many helper kernels per MoE layernsys: kernels per layer down; ncu: DRAM % of measured peak upmoe-decode.md
Persistent kernel / megakernelwhatever hand-off is leftmeasured hand-off sum firstlaunch-overhead.md

Why fusion helps at batch 1: activations are tiny (a 4,096-wide BF16 vector is 8 KB), so fusion saves almost no bytes. It saves kernels, and each dependent kernel costs a hand-off of the order of a microsecond or more.

Exact byte cuts keep the model identical (lossless track, no precision decision) and move the ceiling: estimate the exact byte floor at the start (ceilings.md, "Exact byte floor") and work them alongside kernels. Re-encoded weights store out-of-window elements as raw 16-bit values; a pruned head serves logprob requests from the full head. Each needs its proof recipe and a recomputed ceiling, and a rare_tokens PASS at the next milestone (byte-cuts.md); a lossy truncated vocabulary stays a hack.

NVIDIA text playbook: prefill / TTFT (compute-bound for long prompts, host-bound for short ones)

For short prompts investigate launch overhead; for long prompts inspect matrix utilization and neighboring elementwise work. NVIDIA text examples and traps: prefill.md.

Lessons that cost people weeks

  • Tune and screen inside the real step. A configuration tuned in isolation (warm or flushed L2, no neighbours, no PDL overlap) can lose in context. Re-tune and rank variants by whole-step CUDA-graph replays (seconds); only the best get a bench serve screen, and only claimed ones the A/B; a wait loop stops when the server dies.
  • Amdahl first, on the critical path. A 2x kernel at 4% of the step is ~1.02x; under PDL, use critical-path shares, not summed durations.
  • Counters suggest; timing decides. ncu pointed at bytes in flight; a pipeline-depth sweep refuted it; the prologue was the cost.
  • A faster kernel can slow its neighbours. A fusion that saved critical-path time in its own slot gained almost nothing end to end, because the kernels after it slowed by a similar amount. Only the whole-step A/B tells. More, with sources: references/lessons.md.

Artifact/evidence contract

Keep the candidate check, screen, commit or rejection, and experiment id. A claimed gain additionally needs interleaved timing, a comparison CI and mechanism evidence from the supported platform profiler. Preserve tool versions and source hashes; never invent Nsight records for other backends.

Decision and no-go rules

  • Host launch overhead dominates? Try supported graph or command-buffer execution first and verify its effect. CUDA graphs are the NVIDIA example; support elsewhere must be established.
  • Step at ~85% or more of the bandwidth ceiling (ceiling bands): only fewer bytes per token help: an exact byte cut (above; start these early anyway, and recompute the ceiling for the new format) or lower precision (a precision decision with its own quality bar). Why: the other three losses are nearly gone.
  • Never change two things in one A/B. Why: you cannot attribute the result, and one change may hide a regression in the other.
  • Losing changes stay out of the branch but go into experiments.jsonl and the report. Why: the next person will try the same idea.
  • A non-deterministic reduction (atomics in split-K) needs an explicit decision. Why: it breaks byte-identity testing and reproducibility.
  • Push only within explicit authorization; do not ask again for an already authorized push. Otherwise commits stay local.

Reporting rules

When this skill finishes a standalone engine optimization, check for an optional engine contribution after the final report. Within an optimizing-inference run, leave that step to its coordinator so the same result produces only one PR. Publication requires the user's explicit authorization and the completed engine-level proof, not a faster microbenchmark alone.

  • Speedups come only from an interleaved A/B (bench rounds ratio CI: derived from observed rounds; compare: observed); Amdahl bounds and bytes saved are derived; "this worked because X" is inferred until the mechanism profile is attached.
  • A claimed change without all three (correctness, interleaved A/B, mechanism profile) is UNPROVEN (or NOT RUN for the missing one), never "probably faster"; an unclaimed kept change is reported as screened. Engine-timed A/Bs (bench decode) are labelled as such.

Profiling inference: timeline first, then kernels

nsys records a timeline (CPU threads, CUDA API calls, kernels, copies, NVTX ranges): where the time goes and whether CPU or GPU is on the critical path. ncu replays one kernel reading hardware counters: why it takes as long as it does. Neither proves a speedup; only an interleaved A/B does (benchmarking). In optimizing-kernels' change loop: one nsys capture per lever area (again when the bottleneck moves), a mechanism profile only for a claimed change.

Use when decode or prefill is below ~85% of its ceiling for unknown reasons (ceiling bands), for launch overhead, host syncs or idle gaps, for one dominant kernel family's limiter, or for the mechanism behind a measured speedup. Not for a speed number (never time with a profiler on), nor before you have a benchmark and a ceiling (a profile without them has no scale).

GPU approval: captures occupy the GPU; run them only with budget.approved: true in intake.yaml or an explicit yes to the stated cost. Analysing existing files needs none.

Route before capturing

Read doctor's data.hardware.platform/data.profiling_route and the relevant hardware reference. NVIDIA: nsys → ncu. AMD: rocprofv3 HIP/kernel/copy timeline → architecture-supported counters (Instinct support does not establish Radeon support). Apple: Instruments CPU/Metal → Xcode/MLX capture. CPU: perf when permitted, otherwise engine timers or CPU-only torch.profiler. nsys/ncu capture is NOT RUN elsewhere; profiling itself still applies.

Retrieve current official docs online for unknown architectures, runtime/tool support, flags or errors before choosing a path; record URL/date/version and check local help. Use AMD ROCm, Apple Developer/MLX and NVIDIA primary sources. Retain timeline/op attribution when counters fail, marking unsupported checks NOT RUN. All platforms follow critical path → Amdahl → change → correctness → unprofiled interleaved A/B, with the same environment hygiene. Include text encoding, image VAE, audio codec/vocoder and buffering where applicable.

Preflight checklist

  • Steady-state work for a bounded number of steps (load and warmup rank the wrong things).
  • Weights downloaded, no credentials needed (HF_HUB_OFFLINE=1): the capture must hold no tokens.
  • You know whether the engine uses CUDA graphs and programmatic dependent launch (PDL): both change how the timeline reads.
  • For ncu: counters readable ("Fallbacks"), measured bandwidth (inferencekit ceiling bw), memory headroom for replay.
  • Asked whether the person already has profiles of this workload (no GPU, no approval needed).

Hard failure gates

  • Never capture outside env -i with an allowlisted environment (inferencekit nsys capture / ncu capture do it). Why: a .nsys-rep stores the full environment, tokens included.
  • Never rank by summed kernel duration. Rank by critical-path time per step (inferencekit nsys analyze -> data.steps.families.*.critical_pct_of_step, GPU idle in data.steps.idle_pct_of_step). kernel_sum_us / span_us > 1 means kernels overlap and duration shares are inflated.
  • Never take host-side costs (cudaGraphLaunch time) from a node-traced capture. Why: node tracing inflated it about 25x in one measured case.
  • Never use an unanchored ncu regex or a --launch-skip landing in warmup, autotuning, graph capture or prefill (the wrong launch).
  • Never quote timeline, trace or ncu durations as step time or speedup. Why: tracing adds overhead; ncu serialises kernels, flushes caches and removes PDL overlap.

NVIDIA workflow (copyable)

Profile:
- [ ] 1. Benchmark number and ceiling in hand (ms/token or TTFT, % of ceiling)
- [ ] 2. inferencekit nsys capture -- <steady command>          -> ev_cap (.nsys-rep, .sqlite)
- [ ] 3. Check the record: trace mode (node/graph), env allowlist, capture window, limitations
- [ ] 4. inferencekit nsys analyze <rep|sqlite>                 -> ev_tl
- [ ] 5. Profile check: per-step wall p50 vs the benchmark's ms/token (within a few %?)
- [ ] 6. kernel_sum / span: > 1 means overlap; use critical-path shares only
- [ ] 7. Answer the six questions, each with a number
- [ ] 8. One family dominates and is far from speed of light -> ncu on it (counters blocked: NOT RUN, see "Fallbacks"):
         inferencekit ncu capture --kernel-regex '^<name>$' --launch-skip <N_pre + 2k> --count 5 -- <cmd>  -> ev_ncap
         inferencekit ncu summarize <.ncu-rep> --peak-gbs <ev_bw>                                      -> ev_k
- [ ] 9. Bound label per kernel; speed-of-light gap; Amdahl bound of the proposed change
- [ ] 10. Fill the fixed tables; every number observed/derived/inferred

Exact commands

inferencekit nsys capture -- python run_decode.py --model ./models/qwen3-8b-fp8 --gen-len 128
inferencekit nsys analyze <.nsys-rep or .sqlite path from the capture record's artifacts>
inferencekit ncu capture --kernel-regex '^fp8_block_gemv_kernel$' --launch-skip 400 --count 5 \
  -- python run_decode.py --model ./models/qwen3-8b-fp8 --gen-len 64
inferencekit ncu summarize <.ncu-rep from the capture record> --peak-gbs <ev_bw>
inferencekit evidence show <ev_id>

MCP tools: capture_nsys, analyze_nsys, capture_ncu, summarize_ncu, capture_rocprof, analyze_rocprof, inspect_evidence (in OpenCode, run captures through the CLI, detached). Options: --help. Hygiene and manual env -i recipes: references/capture-hygiene.md.

Fallbacks: counters blocked, no nsys, supplied profiles, other hardware

Other skills link here for these rules.

  1. Check the counters. doctor's data.profiling_counters.status, from the driver setting and the process's capabilities (not root: a container's root usually lacks CAP_SYS_ADMIN): allowed, admin_only (no ncu, unless its reason says this process has CAP_SYS_ADMIN: then probe) or unknown. inferencekit doctor --probe-profilers settles it with a tiny nsys and ncu capture, or the first small ncu capture does (counters in its coverage.missing: blocked). Containers and many cloud instances block them, A100 and H100 included: a host setting, not the GPU model.
  2. Blocked: nsys only (it needs no counters): the whole timeline half (steps 1-7); every ncu row NOT RUN ("GPU counters blocked on this host", doctor evidence id), and the report says, in these words: "ncu was not possible on this host; profiling used nsys timelines only"; the bound label only from arithmetic (bytes a kernel must read ÷ its in-step time vs measured bandwidth), inferred, no counters.
  3. No nsys either, or you want PyTorch-op attribution: the PyTorch profiler (no counters, no Nsight): vLLM --profiler-config with /start_profile and /stop_profile, SGLang /start_profile, torch.profiler in a custom engine. The CLI does not parse its Chrome/Perfetto traces: read kernel time per step and GPU idle gaps by hand or with profile-analyst. It slows the run: never time with it on (references/torch-profiler.md).
  4. Ask for the person's own profiles captured where they have permission (profiles: <path>, ... in constraints; non_interactive: a QUESTIONS.md request): inferencekit nsys analyze <file.nsys-rep or .sqlite>, inferencekit ncu summarize <file.ncu-rep> --peak-gbs <ev_bw or GB/s> (no GPU; without the ncu CLI, CSV exports: references/ncu-capture-recipes.md). A profile from another GPU or engine version explains that setup only: say so by every number. Never copy, commit or upload such files (they may hold secrets); cite the analysis.
  5. Not NVIDIA: nsys and ncu rows are NOT RUN. On AMD use inferencekit rocprof capture and rocprof analyze for kernel/HIP traces, or a supported vendor profiler; rocprof-compute support depends on the exact GPU (amd-rocm.md). Use perf or likwid on CPU, Instruments or MLX on Apple, under env -i. Vendor captures outside the CLI are cited by file; kernel totals may overlap and the bound label stays inferred.

Reading the timeline

  • Node vs graph tracing. --cuda-graph-trace=graph shows realistic host timings (host costs, CPU vs GPU critical path); node shows every kernel inside graphs (ranking, gaps) but inflates cudaGraphLaunch. inferencekit nsys capture traces nodes; for host costs add one graph-level capture (references/capture-hygiene.md).
  • Critical path, not summed kernel time. Under PDL (and multiple streams) the next kernel starts early and waits on the GPU, so durations include waiting. When the per-step overlap kernel_sum_us - busy_us is about one family's summed duration, that family is mostly hidden: size changes with critical_pct_of_step (lessons, "Summed time lies under PDL").
  • Profile check. The profiled token period (step wall plus the gap to the next step) vs the benchmark's ms/token: a few percent of inflation is normal; more means the profile measures something else (often the tracer): fix the window or trace mode first.

The six questions (top-down, each with a number)

  1. Does this stage matter? Its share of per-token (or per-prefill) wall time. 2. CPU or GPU on the critical path? GPU idle gaps and blocking calls (cudaStreamSynchronize, D2H of the sampled token) mean the CPU is. 3. Which stage? Attention, MLP, lm_head or sampling, by critical-path family. 4. A few big kernels or a long tail? A long tail of microsecond kernels means hand-off bound. 5. Speed-of-light gap? Bytes a big kernel must read / measured bandwidth vs its in-step time. 6. Amdahl bound? 1 / ((1 - f) + f / s) for critical-path fraction f made s times faster. Traps and SQL on the .sqlite: references/six-questions.md.

ncu: labelling the bound

LabelEvidence pattern (heuristic thresholds)Lever
memory-boundDRAM throughput >= ~70% of the measured peak; SM pipes well below theirsread fewer bytes (precision, fusion that avoids re-reads)
compute-bounda pipe (tensor, FMA, ALU) >= ~70% of its peak; DRAM well belowfewer instructions (better maths path, tensor cores, cheaper dequant)
latency-boundDRAM and SM both below ~50%; few eligible warps; long-scoreboard / barrier stalls; grid below one wave or low occupancymore bytes in flight: more blocks (split-K), wider loads, fewer dependent steps

The label is inferred: a hypothesis, not a verdict. Counters suggest; in-place timing decides (lessons). Capture recipes: references/ncu-capture-recipes.md; metrics and the L2-resident and nominal-peak traps: references/ncu-metrics.md.

Evidence kinds: nsys_capture -> nsys_timeline; ncu_capture -> ncu_kernels; rocprof_capture -> rocprof_summary. Profile files hold host names, paths and command lines: never upload one unless the user checked it.

Fixed result tables

QuestionAnswerNumberClassEvidence
Profile vs benchmark (step ms vs ms/token)% differencederivedev_tl, ev_bench
kernel_sum / spanratioderivedev_tl
1-6 (as above)observed / derivedev_tl, ev_bw
Kernel (anchored)Launches profiledGrid, wavesncu duration us (not e2e)DRAM GB/s% of measured peakSM %OccupancyTop 2 stallsBound (inferred)Evidence

Decision and reporting rules

  • Idle above ~10% of the step: fix the host side (CUDA graphs, async scheduling, GPU-side sampling) before kernels, which would only lengthen the gaps. Memory-bound at >= ~85% of measured peak: stop tuning that kernel. Latency-bound with waves below 1: add parallelism (split-K, smaller tiles) first.
  • Amdahl bound below the benchmark's CI half-width: do not start (you cannot prove the win). Stalls from kernels under ~10 us or one launch are qualitative only.
  • Timeline and counter values are observed under tracing or replay; say so. Shares, Amdahl bounds, % of measured peak and speed-of-light gaps are derived (cite ev_bw). "Launch-bound", "memory-bound" are inferred until an A/B supports the change they motivate.
  • A question the capture cannot answer (host costs from a node-only trace; % of peak without measured bandwidth) is UNPROVEN, naming the capture that would answer it. Every ncu duration says "ncu replay time, not end-to-end".

Proving correctness and quality

A speedup only counts if the engine still computes the same model. Engines almost never agree bit for bit, so "same" is defined per kind of change, against a noise floor of correct-but-different arithmetic. optimizing-inference runs the full workflow per candidate engine and at milestones; inside the change loop each change gets only the fast check below.

When to use / not use

Use when:

  • before claiming any speedup (benchmarking, optimizing-kernels);
  • a kernel, fusion, precision, engine, attention backend or serving config changes (inside a change loop: the fast check only);
  • N* is being claimed (quality must hold at that load level too).

Do not use when:

  • you want to rank different models by quality (evaluation, with far larger sets);
  • the change intentionally makes a new model (new weights, a 4-bit recipe you have not agreed to treat as the same model): that is a quality study with the user's sign-off.

GPU approval: reference and candidate runs occupy GPUs; run only with budget.approved: true in intake.yaml or an explicit yes to a stated cost.

Choose the contract and modality first

Read the run's quality choice. Exact requires equality on its declared deterministic/seeded outputs; legacy lossless distribution agreement alone is insufficient. Balanced/speed tracks use their registered task/perceptual budget, so rounding differences are not an automatic rejection. Model changes need separate permission and separate reporting.

Use modality-appropriate structural integrity and task quality: text accuracy/distributions, image semantics/readability, speech content/completion/speaker, video temporal consistency. Other tasks need a registered adapter; do not substitute GSM8K or token identity. Missing listening/viewing remains UNPROVEN. The text procedures below apply only where their inputs and claims match.

For TTS, register the acceptance policy before capture with quality audio-policy, then run quality audio on final configuration records and saved PCM (protocol). For changed outputs, quality audio-scorers pins local models and quality audio-score produces ASR/naturalness/voice observations (setup). Score after timing; WER/CER and MOS predictors do not replace required listening.

Which rule applies (decide before measuring)

Kind of changeExamplesFidelityPass rule
Pure refactormoving code, launch restructuring, a switch that is off, prefill tiles that keep the K orderlosslessbyte identity (ladder below)
EquivalenceCUDA graphs, fusion, the same precision in another engine or attention kernellosslessno regression beyond the A/A noise floor
Exact byte cutlossless weight re-encoding, an exact bound-pruned lm_headlosslessits proof recipe (byte-cuts.md: weights decode bit-equal; pruned token = full token every step; logprob requests take the full head), rare_tokens, then equivalence
Exact speculationMTP or EAGLE heads, a same-tokenizer draft model, n-gram lookup, with standard verificationlosslessthe spec_exact gate at temperature 0; under sampling, the accept/resample check and quality sampling (sampled exactness); then equivalence
Declared numerics change16-bit activations where the checkpoint quantises them; a KV dtype the task listslossless only if the task allows it; declared in the engine's numerics, labelled "numerics changed"quality against the reference no worse than the baseline's, within the noise floor
Numerican FP8 KV cache the checkpoint does not declare, lower-precision attention mathslossyno regression beyond the noise floor plus a pre-registered margin
Approximation4-bit weights, lossy speculative decoding, weight or expert pruning, a truncated vocabularylossya pre-registered margin agreed with the user; otherwise it is a different model

fidelity: lossless (the intake default) allows only the lossless rows. fidelity: lossy adds the others, each judged against the intake's quality.budget and reported on its own track, as speed at a stated quality cost. A task's own precision rule overrides this plugin's default. Full rules: fidelity.md.

For images, TTS and STT, an explicitly authorized numerical change uses the registered modality checks rather than LLM token/logprob gates. Relative degradation budgets belong in quality.budget.relative, with a matching modality and quality.protocol / quality.protocol_sha256 binding. A valid intake leaves external quality proof UNPROVEN. Different pixels, PCM, generated codes or rounding are diagnostic observations, not automatic quality failures for that numerical-change track. Pure refactors still use the declared exactness checks. Nonfinite values, invalid shapes, unsafe accesses, broken own-output/native integrity and omitted cases remain failures. A quality budget alone does not authorize architecture pruning or another model.

With explicit quality.acceptance: perceptual_and_task_noninferiority, image/TTS promotion requires the bound task checks and fresh blinded perceptual evidence. Numerical similarity (including image LPIPS/SSIM) is diagnostic, not an independent rejection gate; do not add dummy bounds. TTS ASR/VAD/speaker proxies do not replace listening. Missing perceptual evidence remains UNPROVEN. Retain old numerical failures under their original rules and reassess a baseline's eligibility prospectively under the new acceptance policy.

Inside the change loop: one fast check

For an image or speech numerical change, use one short fixed public modality correctness/quality screen under the agreed protocol. Full registered quality and balanced timing remain milestone/final promotion requirements; they are not required for every intermediate kernel. Retain failed screens and all cases, and label any historical output reuse exploratory. No numerical candidate is promoted from exact-output diagnostics alone.

For LLM candidate changes, against the previous build, greedy, on a handful of fixed prompts: a pure refactor or exact change keeps the greedy tokens identical; a numerics change (any change to the arithmetic) keeps next-token and logprob agreement, flips only at near-ties (inferencekit correctness --reference <prev.jsonl> --candidate <new.jsonl>); a fail drops the change. Inputs: one {"tokens": [...], "logprobs": [...]} line per prompt from each build's server (a bench decode id compares tokens only). For a numerics change, --min-prefix-agreement below 1 (written down first) and --logprob-tol <t>; at each first divergence, the top-2 gap from the previous build's top-20 logprobs must be below the A/A control's largest flip margin, or the change drops (margin test). The A/A floor, trajectory score, GSM8K, sampling and byte-identity ladder below are the full proof: milestones and published claims only (state.md).

Preflight checklist

  • The prompt set fixed and verified (inferencekit prompts verify --set <dev|path>), the same token ids for every engine. Why: changing prompts changes the result.
  • Greedy decoding, every sampler setting explicit, fixed length, same chat template. Why: sampling turns every comparison into noise. Speed is timed at the workload's sampling; a sampled claim (sampler, speculation, a pruned head above temperature 0) also needs sampled exactness.
  • A higher-precision reference endpoint (intake quality.reference): a vLLM server running the model in BF16 for an FP8 engine (quality score needs vLLM's prompt_logprobs). Why: distance to a better reference is the quality signal; two candidates agreeing with each other proves little. None: below.
  • Quality runs on the exact speed configuration (the same resolved config that was timed), never a debug or batch-invariant mode. Why: those modes run different, slower kernels.
  • An A/A control planned, run through the identical pipeline: the reference's own greedy trajectory (--baseline), ideally twice (--baseline-repeat, e.g. after a relaunch). Why: it sets the noise floor; no tolerance may be invented.
  • Margins pre-registered and passed explicitly (--margin-agreement, --margin-kl, --margin-nll). Why: the tool's defaults (0.02 / 0.02 nats / 0.05 nats) are loose placeholders, ten times looser than a same-precision KL margin.

Hard failure gates

  • NaN or Inf anywhere in logits or logprobs. Always a bug.
  • A disagreement where the reference was confident (large top-1 vs candidate-token margin, above the control's largest). Why: near-ties flip under rounding; large-margin flips mean wrong maths.
  • Teacher-forced agreement or trajectory score below the noise floor by more than the pre-registered margin, with the CI excluding the margin.
  • Any rung of the byte-identity ladder failing for a change declared a pure refactor.
  • An A/A control that is not identical when the ladder requires identity. Why: then the harness is non-deterministic and no A/B difference can be interpreted.

Workflow (copyable)

Quality:
- [ ] 1. Classify the change (table above); pre-register the rule and margin
- [ ] 2. Reference (vLLM, BF16) up: quality trajectory --label reference, then --label reference-r2
         (restart your own reference server in between; a remote endpoint cannot be restarted:
         run back to back and say so) -> the A/A floor's trajectories
- [ ] 3. Per engine, its exact speed config up: quality trajectory --label <engine> -> ev_traj_<engine>
- [ ] 4. quality score --trajectories <dir> (once per engine) --baseline <reference dir> [--baseline-repeat]
         with the pre-registered margins -> ev_qs (verdict per engine vs the floor, paired CI over prompts)
- [ ] 5. Every disagreement: position + reference margin; any above the control's max -> investigate / FAIL
- [ ] 6. quality gsm8k --concurrency 0 on every engine: all 1,319 for a claim, the final candidate
         or a scored commit; --limit 100 at the light milestone checks (only screens);
         candidates with --paired-with <baseline run> (McNemar) -> ev_gsm
- [ ] 6b. Sampling in the claim? quality sampling on each engine, the baseline too (draws vs own logprobs) -> ev_samp;
         plus the accept/resample check (speculation) or per-step support check (pruned head)
- [ ] 7. Long context in the claim? quality niah at the lengths claimed (optional otherwise)
- [ ] 8. Pure refactor? Byte-identity ladder instead, control first
- [ ] 9. Fill the fixed table; verdict per check

Exact commands

# A/A floor: the reference's own greedy run (vLLM serving the model in BF16)
inferencekit quality trajectory --base-url http://127.0.0.1:8001 --model <ref-name> --prompts <set.jsonl> \
  --label reference --run-dir runs/<id> --top-logprobs 20
# each engine, at its exact speed configuration
inferencekit quality trajectory --base-url http://127.0.0.1:8000 --model <served-name> --prompts <set.jsonl> \
  --label vllm-tuned --run-dir runs/<id> --top-logprobs 20 --engine-kind vllm      # -> runs/<id>/quality/vllm-tuned
inferencekit quality score --trajectories runs/<id>/quality/vllm-tuned --reference-url http://127.0.0.1:8001 \
  --reference-model <ref-name> --baseline runs/<id>/quality/reference [--baseline-repeat runs/<id>/quality/reference-r2] \
  --margin-agreement <pre-registered> --margin-kl <pre-registered> --margin-nll <pre-registered>   # -> ev_qs
inferencekit quality gsm8k --base-url http://127.0.0.1:8000 --model <served-name> --run-dir runs/<id> --label baseline \
  --concurrency 0                                                                  # 0: sized from the server
inferencekit quality gsm8k --base-url http://127.0.0.1:8000 --model <served-name> --run-dir runs/<id> --label vllm-tuned \
  --concurrency 0 --paired-with runs/<id>/quality/gsm8k/baseline.jsonl             # McNemar vs the baseline
inferencekit quality sampling --help                                               # sampled exactness (below)
inferencekit correctness --reference ref_hf.jsonl --candidate <ev_bench>          # token agreement: the per-change fast check vs the previous build (or the older HF-reference method)
inferencekit report <ids...> --format md

MCP tools: quality_trajectory, quality_score, quality_gsm8k, quality_niah, check_correctness, report (in OpenCode, which ends an MCP call after 120 s, run quality through the CLI, detached with a log). Flag spellings: inferencekit quality <op> --help. Scoring, the noise floor and pass rules in detail: references/quality.md.

Trajectory scoring (summary)

Each engine generates its own greedy continuation through its serving (decode) path, the code users run, with the top-20 logprobs per step. The reference (vLLM) is teacher-forced on that sequence and scores every position: top-1 match, KL on the top-k supports (an approximation), the reference's log-probability of the engine's token, and the first divergence from the reference's own greedy run. The A/A floor is the reference's own trajectory scored the same way, so it already holds the gap between decode path and one-pass scoring. Prompt-logprob ("score mode") endpoints exercise only prefill kernels.

Sampled exactness

Lossless means the same sampling distribution, not only the same greedy token. The teacher-forced check holds the engine's log-probabilities to the reference's; inferencekit quality sampling shows the sampler draws from the distribution they give under the request's temperature, top-k and top-p (seeded first-token draws against the top-20 log-probabilities; mean TV beyond sampling noise, bootstrap CI over prompts; references/quality.md section 6). PASS when the CI's upper end is inside the pre-registered margin, FAIL when its lower end is above it, UNPROVEN otherwise or when not run.

It checks only the first generated token (max_tokens 1, produced by prefill), so a change that acts only in decode steps is not exercised. Speculation also needs a unit check of the accept/resample step against the target distribution (synthetic p and q, many draws, TV or chi-square inside a pre-registered bound). A pruned head needs the per-step support check in byte-cuts.md, or the first token must also go through the pruned head. Without one of these the sampled claim is UNPROVEN.

Why bitwise identity across engines is not expected

Floating-point addition is not associative: tiling, split-K, reduction order, FMA contraction, online softmax and shape-dependent heuristics each give a slightly different, equally valid result, and after one flip greedy outputs diverge. With the early engine almost no prompt matched the reference exactly for any engine (the reference's own second attention path included) while per-step agreement stayed inside the floor. Exact-match rates are a poor quality signal.

Byte-identity ladder (pure refactors only)

Climb in order; stop at the first failing rung (references/byte-identity-ladder.md):

  1. Kernel outputs byte-equal (memcmp) across a shape sweep, outputs pre-filled with NaN.
  2. Identical SASS for kernels the refactor should not touch (cuobjdump -sass).
  3. End-to-end logits (and KV cache) byte-equal on the fixed prompts, every path (eager, capture, replay).
  4. Under load: an off/off (A/A) control first, then old vs new.

Task eval

inferencekit quality gsm8k: the same problems, format and answer extraction for every engine, greedy, accuracy with a Wilson CI; --paired-with <the baseline's results JSONL or evidence id> adds McNemar's exact test. A claim uses all 1,319 items (the default); a --limit subset only screens (the first 100 detect only about 10-point effects). --concurrency 0 sizes concurrency from the server. The first run on a host downloads the pinned test split (419 KB) into ~/.cache/inferencekit/gsm8k/: unset HF_HUB_OFFLINE for it, or place the file with the curl command the error prints (references/gsm8k-parity.md; the older token-agreement method: references/token-agreement.md).

No higher-precision reference

With quality.reference: none, compare with the baseline engine run twice, restarted between (the A/A floor); label results "vs baseline, no reference".

  • GSM8K, all 1,319: candidate --paired-with baseline; the A/A pair gives the noise-only discordant count.
  • Held-out NLL per token on a fixed text in no prompt set, from each engine's logprobs (outside the CLI; record the text's sha256 and code path). Pass: gap to baseline within the A/A gap plus a pre-registered margin.
  • A vLLM baseline can also be the quality score reference: it bounds drift from the incumbent, not quality.

Artifact/evidence contract

  • quality trajectory -> runs/<id>/quality/<label>/<prompt-id>.json + index.json and a kind=quality_trajectory record (observed); quality score -> runs/<id>/quality/score.json and a derived record: per engine, per position bucket and input bucket, with a PASS / FAIL / UNPROVEN verdict against the floor; quality gsm8k -> kind=quality_task (per-problem outcomes, not only accuracy); correctness -> kind=correctness.
  • Keep the prompt-set sha256, template and decoding settings in each record. Why: a quality record without its inputs proves nothing.

Fixed result table

CheckEngine (config)Value [CI95]Noise floor (A/A) [CI95]Margin (pre-registered)VerdictEvidence
Top-1 agreement vs reference (decode path, tie-inclusive)%%pp
Mean / p99 KL (top-20 approx.)
Disagreements above the control's max margincountmust be 0PASS / FAIL
GSM8K (1,319), McNemar vs baselinex / n, p
Sampled distribution (mean TV vs baseline; speculation: + accept/resample check), if sampling is claimed
NIAH at lengths claimed (optional)NOT RUN if skipped
Byte ladder rung reached (refactors)1-4control identical?

Decision and no-go rules

  • FAIL blocks any speed claim for that engine or change. Why: performance of a wrong engine is meaningless.
  • Tolerances come from the A/A control, never from "looks small". Why: invented tolerances drift toward whatever the candidate produced.
  • Divergences explained by near-ties only: PASS with the explanation. Why: expected behaviour of correct, different arithmetic.
  • No reference: the no-reference route; without it, UNPROVEN, not PASS. A test without enough items to detect the pre-registered effect: UNPROVEN (underpowered), not PASS.
  • Speed claims stay within one precision even when quality is compared across precisions against the same reference. The one exception is a format trade-off (evaluating-quantization): a speed ratio with its CI95 shown only next to this skill's quality verdict for that row, labelled "cross-precision: format trade-off, not an engine comparison".

Reporting rules

  • Agreement, KL and task scores are observed; noise floors from controls are observed; "divergence is due to rounding" is inferred, supported by the margin at divergence.
  • State which path was measured: decode path (trajectory) or one prefill pass (score mode). The two are not comparable with each other.
  • Verdicts PASS / FAIL / UNPROVEN / NOT RUN per check, combined with the precedence in reporting-perf-claims.

Reporting performance claims

A performance claim is a statement someone will act on: it says what was measured, under which conditions, with what uncertainty, and what was not shown. This skill turns evidence records into claims that survive review, and audits claims others wrote; it is the last stage of optimizing-inference.

Use to write up any benchmark, capacity, quality, profile or optimisation result, to review a PR description, README table, blog post or agent summary with performance numbers, or to decide what verdict evidence supports. Not while you still need to measure (benchmarking, profiling, proving-correctness).

Preflight checklist

  • The claim was written before the measurement (statement, metric, workload, bucket, decision rule). Why: claims chosen after the data are forking-paths results.
  • Every evidence id the report cites exists (inferencekit evidence show). Why: dangling citations are fabricated evidence.
  • coverage and limitations of every record have been read. Why: a valid record can still not cover the claim.
  • Applicable gates passed: gates run for LLMs; structural and semantic checks for media. A gate failure invalidates a win.
  • Losses, refuted ideas (experiments.jsonl) and NOT RUN items are listed. Why: a report of only wins misleads about the base rate.

Hard failure gates

  • A number without an evidence id or a formula.
  • A speedup without an interleaved A/B whose ratio CI95 excludes 1.
  • A speed comparison across precisions (weights, activations, KV cache, lm_head), GPUs or workloads presented as an engine result.
  • An engine-timed (bench decode) number in a published table, or next to client-timed (bench serve) numbers without its label.
  • A profile (nsys, ncu) cited as proof of a speedup.
  • Missing data shown as 0, blank or PASS.
  • A speed claim without a correctness/quality verdict.

Workflow (copyable)

Report:
- [ ] 1. Copy the pre-registered claim(s) verbatim, with their timestamp
- [ ] 2. inferencekit evidence list; select the records for each claim
- [ ] 3. For each claim, evaluate the precedence FAIL > UNDERPOWERED > INCOMPLETE > NO-DATA > PASS
- [ ] 4. Map to PASS / FAIL / UNPROVEN / NOT RUN
- [ ] 5. Label every number observed / derived / inferred
- [ ] 6. inferencekit report <ids...> --format md
- [ ] 7. inferencekit review build runs/<run-id>   (outputs side by side, per bucket and engine)
- [ ] 8. Write REPORT.md using references/wording-guide.md; retain mechanism, losses, NOT RUN items and ledger in Details
- [ ] 9. Claims lint over the final text (below); fix or downgrade every failing line

Exact commands

inferencekit evidence list
inferencekit evidence show <id>
inferencekit compare <ev_baseline> <ev_candidate> --metric <metric>  # same-kind round records; not load mode
inferencekit report <ids...> --format md                   # fixed layout; --format json for CI
inferencekit review build runs/<run-id>                    # -> runs/<run-id>/review/index.html
inferencekit history                                       # kept commits with their Perf-* trailers

MCP tools: list_evidence, inspect_evidence, compare, report, build_review, history. The CLI/MCP report is a fixed technical export; link it or include its tables in Details of the user-facing report. Add quality/gates verdicts from their records; the export has no such columns. Served ratios come from bench rounds' <metric>_ratio_to_best. Follow references/wording-guide.md for REPORT.md and the final reply.

Verdict precedence

Evaluate in this order; the first that applies is the verdict.

StepConditionReported as
FAILa required check ran and contradicts the claim: regressed, CI95 entirely below the claimed ratio, a quality FAIL, a gate FAIL, or a hard gate in the candidate (except the reasons hardware.md, note 3, lists as expected on that platform, reported as "not applicable on ")FAIL
UNDERPOWEREDchecks ran but the CI includes 1 (or the claimed ratio and 1 both), or fewer rounds/items than pre-registered (a kernel-loop full proof pre-registers its fixed 3 rounds)UNPROVEN (underpowered)
INCOMPLETEcoverage.complete is false, an engine/bucket/level/metric in the claim is missing, or rounds were not interleavedUNPROVEN (incomplete)
NO-DATAa required check produced no dataNOT RUN if never attempted; UNPROVEN if attempted
PASSnone of the above, and the decision rule is metPASS

Why this order: one contradicting check must never be outvoted by passing ones, and "not enough data" must never look like "fine". Worked examples: references/verdict-rules.md.

Claims lint (run over every final text)

Result links descriptively to Details or its evidence table for ids, numeric classifications and formulas; every claim stays traceable.

  • Every number has an evidence id, or a formula whose inputs have ids.
  • Every number is labelled observed / derived / inferred.
  • Claim verdicts use only PASS / FAIL / UNPROVEN / NOT RUN; comparison verdicts only improved | regressed | within_noise | inconclusive, from inferencekit compare or a bench rounds paired ratio CI.
  • No speed claim crosses precisions; cross-precision rows are labelled "cross-precision: not an engine comparison", with no ratio verdict. Exception: a format trade-off (evaluating-quantization) may show its speed ratio with CI95 only next to its quality verdict against the full-precision reference, labelled "cross-precision: format trade-off, not an engine comparison", with no improved / regressed.
  • Published timing is client-measured: served LLMs or complete media requests. Diagnostic engine timings are labelled.
  • Every speedup has a ratio with its CI95 and the conservative bound quoted; "up to", "slightly faster", "near optimal" are absent.
  • Every engine row states its version, resolved key settings and whether it was tuned for this goal.
  • Capacity claims state the SLO, N*, open-loop arrivals, and the price behind any cost.
  • Missing buckets, levels, GPUs and checks appear as NOT RUN, never as blanks or zeros.
  • Losses and refuted ideas are listed with the same columns as wins.
  • The review page exists for the run and is linked. Line-by-line examples and fixes: references/claims-lint.md.

Review page (inferencekit review build <run-dir>, MCP build_review)

One self-contained HTML file at runs/<run-id>/review/index.html that works offline: rows are inputs grouped by bucket, columns are engines plus the reference, with the first divergence from the reference marked; a summary table at the top; filters by bucket and engine. Link it from the report. Why: a reader can see what each engine actually produced for the inputs that were timed, which catches broken outputs no aggregate metric shows. The full page stays local; share it only after the user has checked it.

Fixed result table

Open with the plain-English summary from stop-point.md, "Summary". Put the technical claim table in Details, one row per pre-registered claim, with these columns:

ClaimWorkload / bucketBaselineCandidateRatio [CI95]QualityMechanismGatesVerdictEvidence

Populate every applicable column from evidence; mark missing checks UNPROVEN or NOT RUN.

Decision and no-go rules

  • Quote the conservative bound: "at least 1.09x (CI [1.09, 1.16])". Why: the point estimate is the luckiest honest number.
  • Publish losses next to wins, with the same rigour. Why: readers calibrate trust from them.
  • Never upgrade a verdict in prose ("UNPROVEN but clearly faster"). Why: the verdict exists to stop that sentence.
  • Never extrapolate across GPUs, precisions, context lengths, buckets or concurrency levels you did not measure. Write NOT RUN.
  • Refuse a cross-precision speed claim even when asked; offer the same-precision comparison and the cross-precision row labelled as such. The only exception is a format trade-off from evaluating-quantization: the speed ratio with its CI95 next to the row's quality verdict, labelled "cross-precision: format trade-off, not an engine comparison", with no improved / regressed verdict. Why: a format buys its speed with quality, so only the pair is a result; speed alone across precisions is still refused.

Reporting rules

  • observed: measured by a tool in this environment; cite the id and statistic. derived: arithmetic on observed values; show the formula. inferred: reasoning; name the measurement that would confirm it.
  • Verdicts: PASS / FAIL / UNPROVEN / NOT RUN for claims; improved | regressed | within_noise | inconclusive for comparisons.
  • No environment values, secret tokens or raw argv in a report; inferencekit stores argv only as a digest.

Research to an experiment

Called by optimizing-inference and optimizing-kernels. Return a small ranked experiment queue, not a literature review.

  1. Fingerprint the problem. Model architecture/components and revision; modality and input/output shapes; batch/concurrency; device architecture and memory; runtime/compiler versions; quality and model-change permissions; current profile and target. Missing facts are unknown, not guessed.
  2. Search what can actually run. Check installed engine features first, then official releases, code, tests, issues/PRs, compiler/vendor docs and papers. Search architecture/operator and device identifiers, not just the model name. An issue is a hypothesis until code or a reproducer confirms it.
  3. Look beyond papers. Use the source map as a starting point, and actively discover sources beyond it: GPU MODE talks/slides/code, hardware and runtime engineering blogs, kernel libraries and compiler repositories often expose actionable details before papers. Search recent releases, merged PRs, conference/workshop material and new engineering groups for the latest relevant work and largest credible gains. Use original sources; follow secondary summaries to their implementation.
  4. Check freshness and applicability. Record URL, access date, release/commit, publication date when available, supported device, dtype, shape and assumptions. Check whether the installed compiler already implements the idea. Compare high reported gains against baseline strength, quality/precision, shapes and full-request scope; prioritize the biggest applicable opportunity with credible headroom. A newer article is not automatically better. Do not call old material “latest” without checking updates.
  5. Produce an experiment. Each useful finding needs the table below. Start implementing the cheapest informative test once a credible candidate exists; do not finish a catalog first. Retain at most three immediate candidates, with other links in a backlog.
  6. Refresh on evidence. Search again after a moved bottleneck, two uninformative attempts, a toolchain change or an unexplained fallback. Reuse prior findings and rejected hypotheses instead of repeating the same search.
FindingRequired record
Source and evidenceExact source/section/commit; what was read or inspected; reported result versus your inference.
Proposed changeFile/operator/runtime path; implementation or flag to reuse; how to prove it executed.
CompatibilityDevice generation, compiler/runtime, dtype, shapes, memory, license and model/quality permissions.
Expected headroomCritical-path fraction and Amdahl bound, or bytes/FLOPs/launches saved; mark estimates inferred. Unknown profile means first measure attribution.
Cheapest useful testSmall representative shapes and quality screen, expected mechanism, falsifying outcome, complete-request confirmation.
DecisionTry, defer or reject with reason and linked experiment id.

Read linked slides, code or transcripts when a video is unavailable. Record that limitation; never claim to have watched a talk from its title. A tutorial example or datacenter result is not proof of support or speed on consumer hardware. Downloading or executing unfamiliar code requires inspection within the existing authorization.

Write compact findings into the existing experiment ledger or a linked research note. Independent research can run alongside kernel writing; parallel.md governs shared resources.

Commands 4

Benchmark inference speed for: $ARGUMENTS

Follow the benchmarking skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/benchmarking/SKILL.md (in a checkout of this repo: plugin/skills/benchmarking/SKILL.md) and its references before running anything.

Steps:

  1. Restate the workload in one sentence: model + revision, precision (weights, activations, KV cache, lm_head), GPU, input buckets (default: the dev set's four, said so), output length, goal (single-user, or capacity with its SLO).
  2. Approval: intake.yaml with budget.approved: true, or the user's explicit yes to the GPU, $/h x hours and a cap. Without it, stop.
  3. Write the claim before measuring (metric, bucket, threshold, decision rule); with no comparison, "report the metrics and % of ceiling".
  4. inferencekit doctor (stop if anything else uses the GPU or the host is busy), then inferencekit ceiling bw and inferencekit roofline at each context you report. Not NVIDIA: ceiling bw is NOT RUN; follow ${CLAUDE_PLUGIN_ROOT}/skills/inference-triage/references/hardware.md (hardware card, bandwidth, --serve-args "--gpu-name <hardware>").
  5. Per engine: its best documented configuration for this goal, with launch line, env, port and ready check in runs/<id>/manifest.json (inferencekit manifest validate); prove each setting in effect from the resolved config or log.
  6. inferencekit prompts verify --set dev (or --set <path>); pass the printed path as --prompts.
  7. Single-user: inferencekit bench rounds --manifest runs/<id>/manifest.json --prompts <set.jsonl> --rounds <R> --mode single (R pre-registered: default 5, never below 3; a fresh engine per round, Williams order), never a bare bench serve with its default --rounds. Capacity: --mode load with the skill's --serve-args grid, SLO and price.
  8. Gates per engine, its server relaunched from its manifest line: inferencekit gates run --base-url <its url> --engine <label> --run-dir runs/<id> --reference-engine <baseline label>, then stop it. A FAIL voids that engine's numbers (say which gate); a live gate NOT RUN is not a pass.
  9. Comparisons: the bench rounds record's <metric>_ratio_to_best per bucket, with its paired CI95. inferencekit compare accepts matching decode, single-user serve, clip, image or audio records; use bench rounds' own results for its aggregate, and do not pass load-mode records. Never across precisions, except an evaluating-quantization format trade-off (with its quality verdict, labelled, no improved / regressed).
  10. Engines without a server only: inferencekit bench decode (--rounds 1 --extend <ev> chains alternated between engines, then compare), labelled "engine-timed" everywhere.
  11. Finish with the skill's fixed tables, the claim verdict (PASS / FAIL / UNPROVEN / NOT RUN), every number observed / derived / inferred with its evidence id, and inferencekit report <ids...> --format md.

No secrets in commands, logs or the report. No suitable GPU: "Getting a GPU" in the inference-triage skill; rent nothing without an explicit yes to the offer and its cost.

Optimize inference for: $ARGUMENTS

Run the optimizing-inference skill in the main conversation: read ${CLAUDE_PLUGIN_ROOT}/skills/optimizing-inference/SKILL.md (in a checkout of this repo: plugin/skills/optimizing-inference/SKILL.md) and follow it.

  • Start with ${CLAUDE_PLUGIN_ROOT}/skills/optimizing-inference/references/starting-questions.md: reuse prior answers; ask only missing quality, goal, budget, deadline, check-in and model-change choices in plain language.
  • No plan or plan approval by default. Show/create a plan only when requested. Existing spending and scope authorization persists; new paid work outside it needs approval.
  • Research current implementations early and when progress stalls. Allow independent kernel writing in parallel; serialize integration and measurements on each device.
  • Work to the deadline: reserve final validation/report time as specified in intake. An estimated unreachable target is a verdict, never a stop by itself. One claims-skeptic reviews at freeze (no freeze: on the draft, before the host is released).
  • Finish with reproducible code, baseline/quality evidence, verdicts, total time/tokens/cost and a plain-English report.
  • Then follow the skill's optional engine-contribution workflow: prepare a qualifying contribution, and publish its PR only within explicit user authorization. Reuse a recorded opt-in; without one, show the proposal before asking (unattended runs leave it awaiting permission).

Write or audit a performance claim using: $ARGUMENTS

Follow the reporting-perf-claims skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/reporting-perf-claims/SKILL.md (in a checkout of this repo: plugin/skills/reporting-perf-claims/SKILL.md) and its references.

If a run directory or evidence ids were given (or none: use inferencekit evidence list and ask which records belong to the claim):

  1. Read each record with inferencekit evidence show <id>: identity, coverage and limitations. Find its pre-registered claim; a post-hoc claim is at best UNPROVEN.
  2. Apply the skill's comparability, gate, quality and verdict rules, including its evaluating-quantization exception. A speedup needs an improved comparison from an interleaved A/B with CI95 excluding 1. Missing data is UNPROVEN or NOT RUN, never zero or PASS.
  3. Generate inferencekit report <ids...> --format md and inferencekit review build <run-dir>. Add quality/gates verdicts from their own records; the export has no such columns.
  4. Use references/wording-guide.md in reporting-perf-claims for REPORT.md and the final reply. Start with the plain-English summary specified in optimizing-inference's references/stop-point.md, "Summary". Keep mechanisms, losses, refuted ideas, ledger, NOT RUN items and costs in Details. Put ids and observed / derived / inferred labels there or in linked evidence; label bench decode values "engine-timed" wherever quoted.
  5. Run the skill's claims lint; fix or downgrade every failing line.

If a claim text or file was given to audit: run the claims lint line by line, quote each failing sentence, and state the verdict the evidence actually supports.

Profile this workload: $ARGUMENTS

Follow the profiling skill. Read ${CLAUDE_PLUGIN_ROOT}/skills/profiling/SKILL.md (in a checkout of this repo: plugin/skills/profiling/SKILL.md) and its references before capturing.

Steps:

  1. Approval (intake.yaml budget.approved: true, or an explicit yes to the cost); analysing existing .nsys-rep / .sqlite / .ncu-rep files or torch traces needs none, so start with the person's own files for this workload.
  2. The command runs bounded steady-state work with no credentials (weights downloaded, HF_HUB_OFFLINE=1), nothing else uses the GPU (inferencekit doctor), and the benchmark's ms/token (or TTFT) for the same workload is at hand.
  3. Capture only through inferencekit nsys capture -- <command> (nsys under env -i with an allowlisted environment), never a raw nsys profile: .nsys-rep files embed the shell environment, secrets included.
  4. Read the capture record: trace mode (host-side costs only from graph-level tracing, kernel attribution only from node-level), capture window, limitations.
  5. inferencekit nsys analyze <rep or sqlite from the record>; a profiled step more than a few percent off the benchmark's ms/token measures something else.
  6. If kernel_sum_us / span_us > 1, kernels overlap (programmatic dependent launch, streams): rank only by critical-path shares (critical_pct_of_step), never by summed kernel time.
  7. Answer the six questions in order, each with a number, ending with the Amdahl bound of the change you would try.
  8. If one family dominates and is far from speed of light: inferencekit ncu capture with an anchored --kernel-regex and a --launch-skip past warmup, then inferencekit ncu summarize --peak-gbs <measured bandwidth evidence>; label the bound (memory / compute / latency) as inferred.
  9. Counters blocked, no nsys, or not NVIDIA: the skill's "Fallbacks" (nsys only with ncu NOT RUN; the engine's PyTorch profiler, read by hand or by the profile-analyst agent, never timed with it on; the person's own profiles; the platform's profiler).
  10. Fill the skill's fixed tables, each number observed / derived / inferred with its evidence id or file, and name the single next step (e.g. one optimizing-kernels change, with its Amdahl bound).

A profile ranks where time went; it proves no speedup. Never attach or upload .nsys-rep, .ncu-rep, .sqlite or trace files without the user checking them first.

Agents 5

Your job is to refute claims in a performance report. Assume each claim is wrong until its evidence shows otherwise. You only read: never run a benchmark, capture, launch or GPU job, and never edit the report. You are the one review, run while the final full proof runs: blockers first.

Input (from the caller)

The draft report or claims, the brief or intake, the code diff, ikstep digest output (one line per evidence record) and the evidence root (absolute path). Run every lookup as inferencekit --runs-dir <evidence root> evidence show <id> [--path <dotted path>], so that ids resolve.

Checklist

  • numbers: every number has a matching record (value, unit, statistic, bucket, workload) and every ratio reproduces from same-session records. Observed cites an evidence id; derived shows a formula that reproduces; inferred names the measurement that would confirm it. "% of ceiling", "unreachable" and "no headroom" rest on measured ceiling bw and roofline records (off NVIDIA: a saved vendor measurement), never a datasheet or estimate. No void record stands in for the baseline (candidate code in its environment, a gate FAIL, an aborted job).
  • verdicts: a speedup comes from interleaved rounds (bench rounds ratio_to_best, or compare for bench decode), its CI95 excludes 1, the conservative bound is quoted in every target bucket, and words match the tool verdict (improved, regressed, within_noise, inconclusive). Missing data is UNPROVEN or NOT RUN, never PASS, zero or blank.
  • protocol:
    • Rows ranked together share precision (weights, activations, KV cache, lm_head), model revision, GPU, prompt set, max_tokens and chat-template setting; a format trade-off row is labelled cross-precision with its own quality verdict, or it is a blocker.
    • Every ranked engine has a gates run PASS (NOT RUN is not a pass), quality on the exact speed config inside the A/A floor, and each kept knob shown in effect.
    • A fast path that switches off for logprobs or concurrency has an exactness check on the timed path (no logprobs, concurrency 1, token ids). Sampling and top_logprobs come from one head computation, or the difference is declared.
    • Greedy divergence from the baseline is explained; exactness was re-checked after the last config change.
    • Nothing is extrapolated to an unmeasured GPU, precision, context, bucket or concurrency; published numbers stay apart from agent-measured ones.
  • task rules and hygiene: each task rule (numerics, sampling, server contract, files to leave) holds in the diff and the launch command; no network calls while serving (VLLM_NO_USAGE_STATS=1 DO_NOT_TRACK=1 or the engine's equivalent). No host names, IPs, user names, ports, keys, tokens or rental provider names; no .nsys-rep, .ncu-rep or .sqlite file committed. Losses, refuted ideas and NOT RUN items are reported with the wins.

Output (markdown, at most about 60 lines)

#Claim (quote)RefutationSeverityChecked againstFix
  • Severity: blocker (false or unsupported), major (wrong verdict or label, the data exists), minor (wording or a missing citation; one line each).
  • Every refutation points at something concrete: an evidence id and field, or a file:line. "Looks suspicious" is not a refutation; a claim you could not check is a gap, not a pass.
  • End with two short lists: claims you tried and failed to refute (with what you checked), and claims you could not check.

You research one engine for one model, precision and GPU. You never install, launch, benchmark or touch a GPU. Your verdict is inferred. Only the stage 3 launch and smoke test decide support, so your job is to make that launch quick and correctly configured.

Input (from the caller)

  • engine name (e.g. vLLM, SGLang, llama.cpp llama-server, TensorRT-LLM trtllm-serve, LMDeploy, ExLlamaV3);
  • model id and revision, its architecture (from config.json if the caller gives it), its stored precision, and the allowed precisions and KV cache dtypes;
  • GPU name and count (work out its architecture and compute capability yourself, e.g. sm_80, sm_89, sm_90, sm_100, sm_120). Off NVIDIA, the AMD gfx target (e.g. gfx942), the CPU model and vector ISA (AVX2, AVX-512, AMX), or the Apple chip;
  • the goal: single-user latency or capacity under an SLO.

Steps

  1. Latest release and its date, from the package index or release page: your memory of versions is out of date.
  2. Model support: the architecture in the supported-models list (since which version); open issues for this model or architecture (titles, status).
  3. Precision path on this GPU: native kernels for the checkpoint's format on this architecture, or a fallback. Name a fallback that changes precision (FP8 on a GPU without FP8 tensor cores is often weight-only FP8 with 16-bit activations; NVFP4 needs a Blackwell-class GPU): it is a separate track, never ranked with the others.
  4. Documented recipe for this model (or family) and goal: launch flags, CUDA graph or compile settings, attention backend, how to turn off prefix caching.
  5. Known blockers: a nightly or source build, a missing chat template, tokenizer or trust_remote_code needs, OOM at default memory settings, a minimum driver or CUDA version.

Output (plain text, this shape)

ENGINE: <name>    QUERY: <model id @ revision>, <precision>, <count x GPU> (<sm_xx>), <goal>
VERDICT (inferred): likely supported | likely supported via fallback (separate track) | likely not supported | unknown
LATEST RELEASE: <version> (<date>) - <source URL>
PRECISION PATH: <native kernels | fallback: what runs at which precision> - <source URL>
LAUNCH TO TRY: <one command line with flags; prefix caching off; no eager mode>
RECIPE: <documented low-latency / throughput recipe for this model or family, or "none found"> - <source URL>
KNOWN ISSUES: <issue URL - title - open/closed - workaround> (or "none found")
ONE DOCUMENTED FIX IF LAUNCH FAILS: <flag, env var, nightly build> - <source URL>
CONFIRM BY: stage 3 launch + smoke test (this verdict is inferred until then)

Rules

  • Every factual line needs a source URL and its date. If you cannot source a line, write "unknown". Never fill a gap from memory.
  • Never name a GPU rental provider. Never include host names, user names, keys or tokens.
  • Stay on your one engine. Do not compare it with other engines: the caller does that with measurements.

You answer the caller's question about one large source in at most 40 lines. The caller never reads the source, so be exact and say what you could not find.

Steps

  1. Size it (wc -l, head -n 40); never print a whole file over 200 lines.
  2. grep -n (-i, -E, -B/-A) for the question's terms, errors (error|exception|traceback|fatal|killed|out of memory|nan) and resolved settings; read only those ranges. JSON or CSV: a short standard-library python3 script.
  3. Logs: the first error and its cause (the first traceback, not the last), the last progress line, readiness, resolved settings, timings. Docs and source: exact flag and API names, with the version.

Output (at most 40 lines)

The source and its size; ANSWER: in one or two lines; up to 30 facts with file:line, quoted where a value matters; NOT FOUND: what the source does not hold.

Rules

  • Read only: never edit, move or delete files, start or stop processes, touch a GPU, or fetch anything.
  • Never quote keys, tokens, passwords, host names, IPs or user names ("an API key is set on line 12").
  • Facts from the source are observed; your reading of them is inferred. Say which.

You explain where time goes in one profile. You never capture, launch or benchmark, and you never call a profile a speedup. Profiles rank and explain; only an interleaved A/B proves a change.

Input (from the caller)

  • one source: an evidence id (nsys_capture, nsys_timeline, ncu_capture or ncu_kernels), a file (.nsys-rep, .sqlite, .ncu-rep), or a PyTorch profiler trace (*.json or *.json.gz);
  • the evidence root (absolute path). Run every inferencekit command as inferencekit --runs-dir <evidence root> <command>;
  • if available: the benchmark's ms/token or TTFT for the same workload, and the measured bandwidth (a ceiling bw evidence id or a GB/s number).

For AMD/Apple exports use the platform guide linked from profiling and source file/version, not an invented Nsight record. Retrieve current official AMD/Apple/MLX docs for unfamiliar formats or counters; unsupported analysis is UNPROVEN. Use interval-union/critical-path time and account for image/audio stages too.

Steps

  1. Get an analysis record. All of these run on the CPU:
    • inferencekit --runs-dir <root> nsys analyze <file or nsys_capture id> (a .nsys-rep needs the nsys CLI; a .sqlite nothing).
    • inferencekit --runs-dir <root> ncu summarize <file or ncu_capture id> --peak-gbs <ev or GB/s> (a .ncu-rep needs the ncu CLI; without it, ask for raw-page and details-page CSVs: inferencekit --runs-dir <root> ncu summarize <raw.csv> --details-csv <details.csv> --peak-gbs <ev or GB/s>).
    • Already an analysis record: inferencekit --runs-dir <root> evidence show <id>.
    • A PyTorch trace (the CLI does not parse it): a short standard-library python3 script, as in the profiling skill's references/torch-profiler.md, "Reading a trace".
  2. Read the source before any number: GPU, engine and version, model, workload, coverage, limitations, trace mode (node or graph), capture window. A profile explains its own GPU, engine version and workload only: say which.
  3. For an nsys record:
    • Per-step wall time p50 against the benchmark's ms/token. A gap of more than a few percent means the profile measures something else.
    • kernel_sum_us / span_us: above 1 means kernels overlap. Then rank only by critical_pct_of_step.
    • idle_pct_of_step, the largest gaps, graph launches, and the top families by critical-path share.
    • Host-side costs (cudaGraphLaunch) count only from a graph-level trace.
  4. For an ncu record: per kernel, the duration (ncu replay time, not end to end), DRAM GB/s and % of measured peak, SM %, occupancy, waves and the top two stalls. The bound label is inferred. Stalls from kernels under ~10 us, or one launch, are qualitative only.
  5. For a PyTorch trace: take the steady steps (drop warm-up and capture) and assign each kernel to the step whose CPU range (ProfilerStep#N or the engine's annotation) holds its launch (matching args.correlation), or use GPU-side ranges (gpu_user_annotation). Per step: busy = the union of its kernel intervals, idle = span - busy, the largest gaps and the cuda_runtime calls inside them, and the top kernel families by share of busy time (never by summed dur). If a step shows only cudaGraphLaunch and no kernels, kernel ranking is UNPROVEN. There are no counters: any bound label is inferred, no counters, from bytes ÷ in-step time against measured bandwidth.
  6. Name the bottleneck. Idle above ~10% of the step: host or launch bound, fix the host side first. Otherwise the top critical-path family and its bound, with the Amdahl bound 1 / ((1 - f) + f / s) (f the critical-path share) for the change you would try.

Output (markdown)

  1. One line: source, GPU, engine and version, workload, trace mode.
  2. The question table: | Question | Answer | Number | Class | Evidence |. Cover the profile vs benchmark check, kernel_sum/span (or busy vs summed dur), idle %, the top families, the speed-of-light gap and the Amdahl bound.
  3. For ncu: | Kernel (anchored) | Launches | Waves | ncu duration us (not e2e) | DRAM GB/s | % of measured peak | SM % | Occupancy | Top 2 stalls | Bound (inferred) | Evidence |.
  4. BOTTLENECK (inferred): <one sentence with the number>, then CONFIRM BY: <the measurement that would confirm it>.
  5. Anything the profile cannot answer, as UNPROVEN with the capture that would answer it (host costs from a node-only trace, % of peak without a measured bandwidth, counters from a PyTorch trace).

Rules

  • Label every number: observed (with the evidence id, or "torch profiler trace" and the file name), derived (show the formula) or inferred. Profiler durations are never speed.
  • Never copy, upload or quote the contents of a .nsys-rep, .ncu-rep, .sqlite or trace file. They can hold environment values, host names, paths, command lines and source stacks. Cite the evidence id or the file name instead.

You research the quantized formats of one model. You never download weights, install or launch an engine, benchmark, or touch a GPU. Your output is a candidate table of priors: the caller's launches and measurements decide fit, support, speed and quality. Never pick a winner.

Input (from the caller)

  • the model: Hugging Face id and revision of the full-precision checkpoint;
  • the target GPU name, count and memory (work out its architecture: sm_80, sm_89, sm_90, sm_100, sm_120, or a non-NVIDIA device);
  • the workload: bucket contexts (prompt + about half of max_tokens), concurrency, the goal (single user or capacity);
  • the allowed formats (model.precision_allowed, model.kv_cache_dtype_allowed) and whether offload is allowed;
  • the evidence root (absolute path) for inferencekit --runs-dir <root>, and the evaluating-quantization skill's directory (${CLAUDE_PLUGIN_ROOT}/skills/evaluating-quantization/ in Claude Code), with scripts/gguf_bytes_per_token.py and references/bytes-and-fit.md (record fields, fit arithmetic).

Steps

  1. Pin the reference. curl -s "https://huggingface.co/api/models/<id>/revision/<rev>" gives the 40-hex sha; read config.json at that sha (https://huggingface.co/<id>/resolve/<sha>/config.json): architecture, layers, hidden size, vocab, tie_word_embeddings, MoE keys, and any quantization_config. Gated repos need HF_TOKEN from the environment: never print it or put it on a command line.
  2. List candidates, official first: the model's own organisation (<name>-FP8, -AWQ, -GPTQ-Int4, -GGUF), hardware vendors' and established quantizers' repos, then the model tree (curl -s "https://huggingface.co/api/models?filter=base_model:quantized:<id>&sort=downloads&direction=-1&limit=100": the most used per format; skip MLX-only and duplicate re-uploads unless the target is that platform). Keep only allowed formats; list the others in one line as "not allowed by the intake".
  3. Per candidate, pin and describe.
    • curl -s "https://huggingface.co/api/models/<repo>?expand[]=sha&expand[]=gated&expand[]=lastModified&expand[]=cardData": sha, gating, licence and base_model (it must point at the reference; drop a fine-tune, a different model, with that reason).
    • Size at that sha: curl -s "https://huggingface.co/api/models/<repo>/tree/<sha>?recursive=true"; sum the files the row loads (*.safetensors, or one GGUF file and its shards; one candidate per GGUF file kept), as the cross-check of step 4's stored_bytes. Never the API's usedStorage (the whole repo, all revisions) or safetensors.parameters (packed int4 shows as int32).
    • The recipe: quantization_config (method, bits, group or block size, ignore / modules_to_not_convert, KV cache scheme) or the GGUF file type; the tool and calibration data if the card says so.
  4. Bytes per token and fit, from headers (no GPU). At each bucket context and the KV dtype the row would run:
    • safetensors: inferencekit --runs-dir <root> roofline --model <repo> --revision <sha> --method headers --context <ctx> --kv-dtype bf16, read with inferencekit --runs-dir <root> evidence show <id> --path <dotted path>: data.counts.bytes_per_token, kv_bytes, state_bytes, lm_head_bytes, checkpoint.stored_bytes (all under data.counts) and limitations (bytes-and-fit.md, section 1).
    • GGUF: python3 <script path> --repo <repo> --file <file.gguf> --revision <sha> (its stored_bytes and weight_bytes_per_token); add kv_bytes (and state_bytes on a hybrid) from the reference's roofline record at the same context.
    • Predicted ceiling ratio = reference bytes per token / candidate bytes per token (derived).
    • Fit = stored bytes + concurrency x (kv_bytes + state_bytes / 2, from the record run at that context) + 1-3 GiB overhead vs GPU memory in GiB (inferred; 1 GiB = 1.074 GB). Never kv_bytes_per_context_token x context (a local slope: it misses sliding-window layers and recurrent state). Every MoE expert must fit unless offload is allowed (bytes-and-fit.md, section 4).
  5. Engine support on this GPU (inferred, sourced). Per format: the engines with kernels for it on this architecture (docs, release notes, issues; check the latest release, not your memory), native (low-precision math) or a weight-only fallback (a separate row), and the launch flag to try.

Output (plain text, this shape)

MODEL: <id> @ <sha> (<architecture>, <params>, tie_word_embeddings=<bool>)   GPU: <count x name> (<sm_xx>, <GiB> GiB)
WORKLOAD: contexts <list>, concurrency <n>, goal <single_user|capacity>   OFFLOAD: allowed|not allowed
REFERENCE BYTES/TOKEN: <GB at each context> (ev_...)

| # | Repo @ sha | Format W/A/KV/lm_head | Made by, recipe (source) | Stored GB | Bytes/token GB @ctx (ev or script) | Predicted ceiling ratio | Fits (inferred) | Engines on this GPU (native / fallback, source) |
|---|---|---|---|---|---|---|---|---|

NOT ALLOWED BY THE INTAKE: <formats, one line>
DROPPED: <repo - reason (fine-tune, unpinned, gated without access, duplicate, does not fit even with offload)>
SUGGESTED SHORTLIST (inferred): <rows worth measuring, same engine across formats first, and why>
CONFIRM BY: launch + resolved config (format, kernel, KV dtype, KV capacity), bench rounds, quality score + gsm8k vs the reference

Rules

  • Every factual line needs a source: an API URL, a file at a sha, an evidence id, or a doc or issue URL with its date. If you cannot source it, write "unknown". Never fill a gap from memory.
  • Bytes come from headers, never from names. "4-bit" in a repo name says nothing about the lm_head, the scales or the KV cache.
  • Model-card quality numbers are claims, reported as "card says ... (inferred)", never as evidence.
  • Never name a GPU rental provider. Never include host names, user names, keys or tokens.
  • Light work only, and never during timing: header reads and API calls. Do not run anything on a GPU host while the caller is timing there.

MCP servers 1

Sponsors

Details

Marketplace
KittenML/inferencekit
Source
plugin
Author
Stellon Labs
Category
Development
License
Apache-2.0