
ai-infra-auto-driven-skills
LLM serving benchmarks, SGLang model Day-0 support, profiling, code review, prod incident triage, PR-history dossiers, and SGLang explainer videos.
- version
- 0.10.0
- GitHub stars
- 1.2K
Install
/plugin install ai-infra-auto-driven-skills@ai-infra-auto-driven-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add BBuf/AI-Infra-Auto-Driven-SKILLS.
Contents
Discovered from sourceSkills 9
LLM Pipeline Analysis
Overview
Use this when a whole-trace profiler summary is too coarse. The scripts read a Chrome-trace JSON file, find layer-boundary anchor kernels, group kernels into forward passes and layers, and print timing tables you can use for Perfetto navigation or detailed timing analysis.
When To Use It
- when you need to know which layers contribute most
- when the model has alternating layer types (e.g. models with
compress_ratioslike DeepSeek-V4 NSA, or hybrid GDN/GQA stacks such as Qwen3.8-27B) - when you need to compare cold-start vs steady-state forward passes
- when you need to navigate to a specific layer in Perfetto UI
- when you need to select representative layers for deep-dive analysis
Verify inputs from artifacts
Resolve model config, phase, rank/device and TP/EP/DP from the run manifest,
server arguments and trace. Ask only for missing information that changes the
analysis. TP-0 identifies a rank, not TP world size; do not default an unknown
run to TP8/EP8 or decode BS1. Speculative request BS and target-verify M differ.
If no config is available, use an explicit profile and verified layer count.
Model Profiles
Scripts use ModelProfile to determine layer boundary detection and kernel
classification. Profiles are auto-inferred from config.json or selected
via --profile:
| Profile | Anchor kernel | Blocks/layer | Auto-infer condition |
|---|---|---|---|
dsv4_csa_hca | mhc_pre_big_fuse | 2 | DeepSeek-V4 or compression fallback |
dsv41 | explicit verified anchor | 1 (override for sublayers) | V4.1 text/wrapper identity or KV-source layers |
dsv3_mla | explicit backend anchor or residual boundary | backend-dependent | homogeneous MLA after identity dispatch |
mla_dsa | verified residual boundary | 2 | GLM-5/5.2/5.3, V3.2, DSA MLA |
kimi_k3 | explicit attn_res_fused (SGLang) / attn_res_fwd_online_v2_kernel (vLLM) | 2 + one output aggregation/pass | K3 / Kimi linear with attention residual |
glm5_next | mhc_pre_big_fuse | 2 | GLM-5.3-Flash |
hy_v4 | explicit ihc_pre_stage2 on Triton fallback | 2 | Hy4 (hpc symbol requires trace check) |
qwen4_exp | explicit grouped_gemma_rmsnorm | 2 | Qwen3.8-Flash-Next (vLLM may fuse norm) |
nemotron_h | explicit residual norm | 1 | one Mamba/attention/MoE block per config layer |
longcat_flash | explicit residual boundary | 4 | LongCat-Flash architecture |
gemma4 | explicit _gemma_qkv_rmsnorm_kernel (SGLang CUDA) | 1 | Gemma4 text/wrapper identity |
generic_addnorm | verified fused-add RMSNorm or allreduce fusion | 2 | other configured models |
generic | explicit anchor; residual candidates only | 1, or 2 for residual candidates | config-less fallback |
Nested text_config, language_config and llm_config are flattened before
inference. Model identity takes priority over kv_lora_rank: K3, GLM-5 Next,
Hy4, dots3 and LongCat must not be treated as homogeneous V3 MLA.
Read layer count from config: V4 Flash has 43 layers, Pro 61, V4.1 40.
These anchor families are source-verified for SGLang/vLLM, with backend and
phase caveats; TensorRT-LLM/TokenSpeed cadence requires separate trace evidence.
Prerequisites
- A
torch.profilertrace in Chrome-trace JSON format (.jsonor.json.gz) - The model's
config.json(for profile inference,compress_ratios, etc.) - The trace must contain a recognizable layer-boundary anchor kernel
(auto-detected from the profile, or specified via
--anchor-kernel)
Layer Boundary Detection
The scripts use an anchor kernel as a layer-boundary marker. The anchor and layer structure are determined by the active ModelProfile.
Find the model's residual kind at its stage boundary first. SGLang declares
attention/FFN boundaries through python/sglang/srt/layers/layer_boundary/factories.py
(add-norm, gated, iHC, mHC, post-norm and stream). Backend dispatch determines
the visible boundary kernel.
V4 currently starts each half at mhc_pre_big_fuse*: attention-pre then
FFN-pre, two anchors per layer. Fused post+pre removes mhc_post_tilelang
for small-token SGLang traces; retain that old anchor only for verified unfused
traces or the >32-token path. V4.1 uses previous-sublayer pre-mix, which cannot
reuse V4's fused-post-pre formula. Its current deepseek_v4_mhc.py dispatch
still carries fused post/combine/norm/quant across eligible boundaries; opt-in
Mega mHC prefill also produces the next coefficients. Engram seams and row
selection can invalidate cached inputs. It always needs explicit anchor
and cadence evidence; a vanished stats/norm launch is not a missing layer.
See the current boundary matrix.
FlashMLA's flash_fwd_mla_combine depends on backend and split count; SM100
SGLang V3 defaults to TRT-LLM MLA, while DSA models use DSA.
All scripts accept --blocks-per-layer K --anchor-offset I --pid PID --device D.
I is a verified first-layer anchor in the selected GPU, not inferred from
counts. Mixed ranks/devices and a count residue modulo N*K are rejected.
A divisible count is necessary but insufficient: separate target/draft phases,
MTP/DSPARK passes and TBO microbatches first. K3's extra output aggregation
requires a phase-filtered trace for these timing analyzers; the layer-track
helper supports the extra anchor explicitly. The final selected pass ends at
the selected GPU event boundary, which must itself be verified as a complete
pass. Anchors are navigation intervals; cross-stream kernels may overlap them.
For compiled vLLM traces, norms can lower to Inductor kernels. Capture a short mapping trace with explicit norm custom ops or eager execution and verify an attention/backend anchor on the actual measured trace.
Source audit 2026-10-05: SGLang V4 dispatch, mHC kernels, vLLM mHC dispatch.
Scripts
1. layer_timeline_analyzer.py — Per-layer timeline and cluster stats
# Show all forward passes summary (cold-start vs steady-state)
python3 scripts/layer_timeline_analyzer.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--show-all-passes
# Detailed per-layer breakdown for a specific forward pass
python3 scripts/layer_timeline_analyzer.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5
# Auto-select the first relatively stable pass window
python3 scripts/layer_timeline_analyzer.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json
The script prints:
- Per-layer wall-clock time, sum-duration, and category breakdown (MLA, MoE, GEMM, NCCL, MHC, Hadamard)
- Layer cluster statistics grouped by type (CSA_C4, HCA_C128, HASH, etc.)
- All-passes summary showing cold-start → steady-state growth
Automatic steady-state selection requires two consecutive layer-0 timing
changes within 5%. It does not use an absolute latency threshold, so the same
rule works across model sizes and accelerators. If no stable window exists,
choose --fwd-pass explicitly.
2. layer_kernel_breakdown.py — Per-layer kernel detail and compute flow
# Single layer kernel dump
python3 scripts/layer_kernel_breakdown.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5 --layer 3
# Compute flow format (with model architecture summary and category column)
python3 scripts/layer_kernel_breakdown.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5 --layer 3 --format compute-flow
# JSON export
python3 scripts/layer_kernel_breakdown.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5 --layer 3 --format json
# Compare two layers side-by-side
python3 scripts/layer_kernel_breakdown.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5 --layer 2 --compare-layer 3
Output formats:
--format text(default): grouped summary + top hot kernels ranked by duration, with simplified names and percentages--format compute-flow: model architecture summary + per-kernel hotness table withCategory,%, andts_rel(ms)columns--format json: one machine-readable JSON document ranked by duration; with--compare-layer, it containsprimary,comparison, andkernel_diff- Kernel diff when comparing two layers (unique kernels in each)
3. perfetto_time_mapper.py — Perfetto UI time navigation
# Show all forward pass time ranges in Perfetto
python3 scripts/perfetto_time_mapper.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json
# Layer-level time ranges for a specific forward pass
python3 scripts/perfetto_time_mapper.py \
--trace /path/to/TP-0.trace.json.gz \
--config /path/to/config.json \
--fwd-pass 5 --layers 2,3,38,42
The script prints:
- Forward pass time ranges in Perfetto-relative seconds
- Per-layer start/end times with compress_ratio labels
Workflow
Step 1: Identify steady-state forward pass
python3 scripts/layer_timeline_analyzer.py \
--trace $TRACE --config $CONFIG --show-all-passes
Read the "all-passes" table. The first pass is cold-start (few tokens). Use the first relative-stability window selected by the script, or choose a pass explicitly when timings continue to change.
Step 2: Per-layer breakdown on steady-state pass
python3 scripts/layer_timeline_analyzer.py \
--trace $TRACE --config $CONFIG --fwd-pass 5
Identify:
- Which layer type dominates (CSA_C4 vs HCA_C128 vs HASH)
- The MLA / MoE / GEMM / NCCL proportion per layer type
- Which layer type is the best next target
Step 3: Compute flow for representative layer(s)
Select 1-2 representative layers (one per bottleneck type), then:
# Human-readable compute flow table
python3 scripts/layer_kernel_breakdown.py \
--trace $TRACE --config $CONFIG \
--fwd-pass 5 --layer 3 --format compute-flow
# JSON export
python3 scripts/layer_kernel_breakdown.py \
--trace $TRACE --config $CONFIG \
--fwd-pass 5 --layer 3 --format json > /tmp/layer3_detail.json
The --format compute-flow output includes:
- Model architecture summary at the top
- Per-kernel hotness table with
# | Half | Category | Simplified Name | dur(us) | % | ts_rel(ms) | Input Dims - Rows are ranked by
dur(us)descending by default; usets_rel(ms)to jump back to the kernel's trace location.
Step 4: Compare layer types (optional)
python3 scripts/layer_kernel_breakdown.py \
--trace $TRACE --config $CONFIG \
--fwd-pass 5 --layer 2 --compare-layer 3
This shows the exact kernel difference between the two layer types.
Step 5: Navigate in Perfetto UI (optional)
python3 scripts/perfetto_time_mapper.py \
--trace $TRACE --config $CONFIG \
--fwd-pass 5 --layers 2,3,38,42
Use the printed time ranges to navigate directly in Perfetto.
Layer Type Classification
The scripts classify layers based on config.json fields:
| Config field | Value | Layer Type | Description |
|---|---|---|---|
compress_ratios[i] | 0 | SWA_ONLY | window-only attention, no compressor/indexer |
compress_ratios[i] | 4 | CSA_C4 | compressor + lightning indexer (Hadamard/paged-MQA/top-k) + sparse attention |
compress_ratios[i] | 128 | HCA_C128 | compressor + all compressed rows and window; no indexer |
compress_ratios[i] | 2 / 1 | V41_C2 / V41_C1 | source-layer compressor/indexer; other layers share KV/index |
i < num_hash_layers | — | +HASH | hash routing in the FIRST N layers, composable with FIRST/FINAL |
i == 0 / N-1 | — | +FIRST / +FINAL | positional flags, never replace attention/routing work |
V4 Flash/Pro hash-route L0–2. V4.1 has no hash layers; its KV source layers
are [2,8,14,20], index sources [2,8,14,20,24,28,32,36], candidate source
20, and Engram layers [1,14]. Use these role fields when choosing representative
layers. Pro starts with ratio-128 layers; do not assume its first two are SWA.
K3's KDA layer ids are 1-indexed, whereas GLM-5 Next uses 0-indexed KDA ids.
Kernel Categories
Kernels are classified by the active ModelProfile's rules. Categories marked
with (DSv4) are specific to the dsv4_csa_hca profile; all profiles include
the universal categories.
| Category | Match Pattern | Profile | Typical Share (DSv4) |
|---|---|---|---|
| ★ MLA Attention | flash_fwd_splitkv_mla | DSv4, DSv3 | 21-33% |
| ★ MoE Fused | fused_moe_kernel | DSv4, DSv3 | 11-17% |
| ● NCCL AllReduce | case-insensitive allreduce, all_reduce, cross_device_reduce, reduce_scatter, NCCL/Lamport | universal | 5-8% |
| GEMM fp8 | deep_gemm | universal | 12-25% |
| GEMM bf16 | nvjet | universal | 11-13% |
| Hadamard Xform | hadamard | DSv4 | 0-2.4% |
| Indexer Cache | indexer | DSv4 | 0-0.1% |
| Paged MQA | paged_mqa_logits | DSv4 | 0-1.8% |
| MHC | mhc_pre_gemm_sqrsum, mhc_pre_big_fuse, mhc_post_tilelang | DSv4 | 10-15% |
| C4/C128 Prefill | c4_prefill, c128_prefill | DSv4 | 0-0.3% |
| RMSNorm | case-insensitive rms_?norm, layernorm | universal | 1-2% |
| FP8 Quant | quant, Quant | universal | 1-2% |
| TopK | topk | universal | 0-0.7% |
| RoPE | deepseek_rope, fused_norm_rope | DSv4, DSv3 | 1-2% |
| Activation | silu_mul_clamp, act_and_mul | universal | 0-0.5% |
| Other | — | universal | 2-5% |
Reporting Checklist
Include:
- Trace metadata: trace path, model config path, GPU type, TP/EP
- Model Architecture Summary (from
config.json):- model name, num_layers, hidden_size, num_attention_heads, num_key_value_heads, head_dim
- Attention type (e.g. csa_hca), Q/O LoRA ranks
- MoE config: num_experts, topk, num_shared_experts, intermediate_size
- MHC config (if applicable)
- NSA config (if applicable): index_n_heads, index_head_dim, index_topk, qk_rope_head_dim, sliding_window
- compress_ratios distribution (how many CSA_C4 / HCA_C128 / SWA_ONLY / HASH layers)
- Per-batch forward passes summary table (from
layer_timeline_analyzer.py --show-all-passes):- Columns: Fwd#, Start(s), End(s), Duration(ms), Avg Layer(ms), First Layer(ms), Notes
- Identifies cold-start vs steady-state passes
- Chosen forward pass: index and rationale (cold-start vs steady-state)
- Per-layer wall-clock and sum-duration table (from
layer_timeline_analyzer.py --fwd-pass N):- Columns: L, c_r, Type, Wall(ms), SumDur(ms), MLA, MoE, GEMM, NCCL, MHC, Hadam, AR#, K#
- Each row is one layer, with layer type label
- Layer cluster statistics table grouped by type:
- Columns: Cluster, #, Avg Wall(ms), Avg Sum(ms), MLA%, MoE%, GEMM%, NCCL%, MHC%, Hadam%
- Identifies bottleneck layer type and likely next target
- Compute Flow Table for selected representative layer(s):
- Produced by
layer_kernel_breakdown.py --format compute-flow - Columns:
# | Half | Category | Simplified Name | dur(us) | % | ts_rel(ms) | Input Dims - Rows are sorted by top hot kernels (
dur(us)descending) by default - Optional JSON export (
--format json)
- Produced by
- Perfetto UI time ranges when requested
- One-line summary: bottleneck layer type and likely next target
LLM Serving Auto Benchmark
Overview
Use this skill to compare LLM serving frameworks such as SGLang, vLLM, TensorRT-LLM, and TokenSpeed for the same model and workload.
Use a config-driven workflow:
- keep launch-only capacity choices in each framework's
base_server_flags - put the search knobs in
search_space - run the same dataset scenarios for every framework
- generate a bounded candidate list from
search_space, with the baseline candidate included first - keep failed candidates in the result file
- pick the best SLA-passing candidate after normalizing the results
For model-specific starting points, prefer the shipped configs in
configs/cookbook-llm/. They define a framework-neutral LLM serving cookbook
model set and translate each entry into framework-native SGLang, vLLM, and
TensorRT-LLM, and TokenSpeed server flags. Validate those configs before a real
run:
python skills/llm-serving-auto-benchmark/scripts/validate_cookbook_configs.py \
skills/llm-serving-auto-benchmark/configs/cookbook-llm/*.yaml
If you have captured target-environment --help files, add
--help-dir <artifact-help-dir>. That check only loads configs, verifies the
server flag names, and renders candidate commands; it does not launch model
servers.
Prefer native tooling when it gives better coverage:
- SGLang: current public cookbooks use
sglang serve;python -m sglang.launch_serverremains accepted. Preferpython -m sglang.benchmark.serving - vLLM:
vllm bench sweep servefor server-parameter sweeps, otherwisevllm serveplusvllm bench serve - TensorRT-LLM:
trtllm-servefor the OpenAI-compatible server plus the TensorRT-LLM serving benchmark client or a common OpenAI-compatible benchmark client - TokenSpeed:
tokenspeed servefor the OpenAI-compatible server plusvllm bench serve --backend openai-chator the same OpenAI-compatible benchmark client used for the other frameworks
TensorRT-LLM has one hard scope rule in this skill: the current server is PyTorch-only; older images must select pytorch. Do not search TensorRT-LLM backend
choice. If a request, config, or candidate asks for trt, an engine backend, or
any other non-PyTorch TensorRT-LLM server backend, reject that candidate as
unsupported for this skill and record the reason. This does not change the
benchmark client backend; the TensorRT-LLM benchmark client still uses
OpenAI-compatible modes such as --backend openai or --backend openai-chat.
Only pick a winner after each requested framework has had its main serving knobs tuned.
Framework selection is caller-controlled. If the caller explicitly supplies a framework list, benchmark only those enabled frameworks. Do not silently add TensorRT-LLM or TokenSpeed just because cookbook configs or history docs exist; record omitted frameworks as user-excluded, not unsupported.
The parameter lists in this skill are not a compatibility contract. They are
version-sensitive candidate knob families. Before every real run, record the
exact framework version or git commit and verify the concrete CLI flag names
with --help in the target environment.
The default search style is framework-neutral: start from a mostly pure-TP baseline, sweep a small set of high-impact runtime knobs, and cap the first pass around 10 candidates per framework. Do not search memory fractions by default.
Source and experiment contracts
Use the source contracts for current profiler APIs, branch-specific defaults and KV sizing semantics. Before combining PRs or claiming kernel performance/accuracy equivalence, use paired-validation.md. It distinguishes code-only A/B from deployment tuning, real from simulated acceptance, and interactivity from aggregate output throughput. Profile after timing to prove that the candidate backend actually ran.
Validation Environment
This skill is target-agnostic. It assumes any one of the following is available, and nothing more:
- a local GPU host with Docker/Podman and the target framework images pulled;
- a remote GPU host reached via
ssh <host>with the framework images already running in a container there; - a CI runner that can exec into a pre-built image for each framework.
Do not assume a specific operator host name inside this skill's own workflow. The concrete SSH wiring, container names, workspace paths, and HF token plumbing for a given box live in operator-side per-host skills; this skill only requires that the caller can reach a shell inside a container with the requested framework installed.
Reference files are optional and version-sensitive. Treat historical flag notes as evidence from one image, not as a compatibility guarantee for the next run.
Historical smoke validation on 2026-05-01 (H100) and 2026-06-27 (B200) exercised small serving/search workloads. The B200 image provided only SGLang; missing vLLM/TRT-LLM/TokenSpeed CLIs were environment gaps. Those runs are not current cross-framework speed or accuracy results. The September source refresh ran offline tooling, not a new GPU matrix.
Skill Scope
This skill is a playbook plus a config+validator toolchain, not a turn-key orchestrator. The operator still launches servers, drives workloads, and writes one normalized JSONL row per candidate.
The scripts/ directory contains exactly two tools:
validate_cookbook_configs.py: load cookbook YAML, render bounded candidate server commands, and check flag names against captured--helpsnapshots without launching servers.compare_benchmark_results.py: turn normalized per-candidate JSONL into the markdown and optional CSV tables described in the Output Contract.
Cookbook configs under configs/cookbook-llm/ must pass the validator. The
shorter references/example-plan.yaml is a
one-off runtime-plan skeleton and is not expected to pass as-is. Use
references/result-schema.md as the single source
of truth for SLA key names.
Required Inputs
Collect these before a long run:
- model and tokenizer path, target frameworks, GPU model/count, multi-node allowance, precision, and quantization constraints
- endpoint shape, workload source, dataset scenarios, SLA target, search budget, and artifact output directory
- version manifest: framework package version or git commit, container/Python
environment,
--helpsnapshots, and whether each search parameter was accepted by that exact CLI
If real production traffic is the goal, use the real request distribution. A synthetic workload is fine for bring-up and first-pass comparison, but it is not enough for a production choice.
Record each scenario's input/output length distribution in the normalized
result rows. This is part of the profiler handoff contract: when a framework
is slower and the follow-up investigation invokes llm-torch-profiler-analysis,
the profiler workload must reuse the slow benchmark scenario lengths instead of
falling back to its generic prefill 4090->1 and decode 1->2048 defaults.
Version-sensitive deployment checks
- Consult the source contracts for
exact inspected revisions. Preserve installed-image
--helpand effective environment values; do not copy a historical release's defaults forward. - Current SGLang main uses
SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=False; the deleted historicaldsv4.1branch used True. Record that numerical difference when reproducing an old result. Main now pins FlashInfer 0.7.3; inspect the installed wheel and actual producer/consumer dispatch. The current kernel roots arepython/sglang/kernels/{aot,jit,ops}. - SGLang's native
/generateand OpenAI-compatible paths are different benchmark lanes. Use one common endpoint/client when comparing frameworks; separately label any native-client result. - Match attention/GEMM backend support to the GPU architecture and dtype. “Blackwell” includes different SM targets; B200/GB300/RTX 5090 are not one interchangeable backend contract. Verify support in the installed code.
- vLLM DBO depends on a supported all-to-all backend. Compile fusion depends on platform, graph partition and dtype guards. Confirm dispatch using the fusion reference.
- The current TensorRT-LLM lane uses
trtllm-serve serve <model>; older images may need--backend pytorch. Server backend is distinct from OpenAI benchmark-client backend. Verify the model format, KV-memory flag spelling and random-dataset sampling options from the selected CLI; do not silently fall back to another dataset. - TokenSpeed uses
tokenspeed serve <model>and a common OpenAI-compatible client. Its TP/attention-TP/MoE-TP/EP and memory flags need their own interpretation; similarly named SGLang/vLLM options are not an equivalence guarantee. - Every max-context candidate must cover
max(input_len + output_len)across all scenarios, not just the baseline. The cookbook validator checks this. - Profile after timing. For a fixed simulated-acceptance curve, label it as such; restart in real acceptance mode for model accuracy tests. Never promote an unused replacement kernel because end-to-end scores happen to match.
Secrets Hygiene
- Never print
HF_TOKEN,HUGGINGFACE_HUB_TOKEN, or any upstream API key into a saved artifact. Pass them through container-e VAR(unquoted on the right side so the host value is inherited) and keep them out ofserver_commandandbenchmark_commandfields written to the result JSONL. - When a framework echoes the full argv at startup, scrub the log or redact token-shaped substrings before uploading the artifact.
Fairness Rules
Use these rules throughout the benchmark:
- Run every framework on the same GPU type, GPU count, model weights, tokenizer, precision, quantization policy, prompt distribution, output length target, and sampling settings.
- Record framework version, git commit, container image, CUDA/NCCL versions, GPU driver, visible GPU ids, launch command, and benchmark command.
- Warm the server before measuring. Restart or clear state between candidate configurations when cache effects would bias the comparison.
- Compare steady-state fixed-QPS runs separately from burst throughput runs.
- Keep failed candidates in the final results with their failure reason.
- Report both raw throughput and SLA-passing throughput. The fastest failing candidate is not the best deployment command.
Workflow
1. Preflight
Verify SGLang plus all requested comparison frameworks before starting a search. Run only the commands for the requested framework set:
sglang serve --help
python -m sglang.launch_server --help
python -m sglang.benchmark.serving --help
vllm serve --help
vllm serve --help=all
vllm bench serve --help
vllm bench serve --help=all
vllm bench sweep serve --help=all
trtllm-serve serve --help
python -m tensorrt_llm.serve.scripts.benchmark_serving --help
tokenspeed serve --help
vllm bench serve --help
Use the framework-specific --help output in the target environment as the
source of truth. Do not keep a stale launch flag just because it appears in an
old note.
vLLM 0.19 and newer use grouped help. Plain vllm serve --help only shows the
groups, so capture --help=all before deciding whether a search knob exists.
Save these --help outputs into the run artifact directory. If a listed search
knob is missing from the current CLI, remove or translate that knob before
running the benchmark. Do not silently pass unknown flags.
Record trtllm-serve serve --help. Current main is PyTorch-only; omit its
deprecated backend option. On older images listing multiple backends, select
--backend pytorch.
For TokenSpeed, confirm both the server and benchmark entrypoints because some
installations alias the binary as ts. Record the exact binary used in
server_command.
For each framework, launch a minimal server, confirm /v1/models or the native
model-info endpoint, send one streaming request, run one tiny benchmark with at
least 5 requests, then save the launch command, benchmark command, server log,
and benchmark output.
Before any GPU-backed smoke run, check the requested GPU ids directly with
nvidia-smi. If a requested GPU is already in use, stop and record that fact.
Do not silently borrow a different GPU count for a performance comparison. It is
fine to run a smaller one-GPU smoke only when the result is clearly labeled as a
flow check rather than a fair throughput comparison.
If the target environment runs through containers, follow references/container-runbook.md and save image tags, pull commands, launch/benchmark logs, and cleanup commands.
2. Normalize The Workload
Use one canonical workload for all frameworks. Recommended JSONL row shape:
{"prompt": [{"role": "user", "content": "Summarize this text."}], "output_len": 256}
{"prompt": "Write a short explanation of CUDA graphs.", "output_len": 128}
Optional fields:
{
"prompt": [{"role": "user", "content": "Use low temperature."}],
"output_len": 256,
"extra_request_body": {"temperature": 0.0, "top_p": 0.95},
"metadata": {"source": "prod-sample"}
}
When converting user data:
- inspect at least 3 rows before conversion
- preserve request-level sampling options in
extra_request_body - do not include the final assistant answer in the prompt when that answer is the target completion
- keep multimodal or tool-call payloads only if all requested frameworks support the chosen endpoint shape
For synthetic bring-up, use the shipped two-scenario shape:
dataset:
kind: random
num_prompts: 80
scenario_names: [chat, summarization]
input_len: [1000, 8000]
output_len: [1000, 1000]
Each aligned input_len / output_len pair is one scenario. Do not take the
cartesian product unless the user asks for that.
Name each scenario and keep the aligned pair in the artifacts. For custom
datasets, compute or record representative input_len and output_len
buckets, at least p50 and p95 when possible, so later profiler runs can match
the slow bucket rather than profiling an unrelated synthetic shape.
Before searching any sequence-length limit, compute the largest
input_len + output_len in the dataset. SGLang context_length, vLLM
max_model_len, TensorRT-LLM max_seq_len, and TokenSpeed max_model_len
must be at least that value for every candidate that is expected to run all
scenarios.
3. Pick A Search Tier
Use the smallest tier that can answer the user's question:
- Tier 1: smoke and sanity. One baseline plus a few high-impact knobs.
- Tier 2: default. A bounded sweep over the most likely server settings.
- Tier 3: exhaustive. Only when the search space is already tight and the user accepts a long run.
Default budget:
num_prompts: 80for the default cross-framework comparison;num_prompts: 20per scenario is acceptable for a smoke/flow check and must be labeled as such in the artifact (not as a performance result).search.max_candidates_per_framework: 10for the first useful pass- candidate generation: baseline first, then a bounded product or ordered
candidate list from
search_space - at most 5 QPS search rounds unless the user asks for more
- stop early when every candidate in one framework is clearly OOM or fails the basic health check
Keep these in base_server_flags unless the user specifically wants a capacity
or memory study:
- SGLang
mem_fraction_static - SGLang
schedule_policy - vLLM
gpu_memory_utilization - TensorRT-LLM
kv_cache_free_gpu_memory_fraction - TokenSpeed
gpu_memory_utilization
These are real knobs, but they widen the search quickly and often turn a serving comparison into a memory-limit study.
4. Tune SGLang
Launch each SGLang candidate manually. The auto-benchmark module was removed in #31941.
Benchmark with the current serving client:
python -m sglang.benchmark.serving \
--backend sglang \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 256 \
--num-prompts 80 \
--request-rate 8 \
--output-file /path/to/sglang/results.json \
--output-details
Version-sensitive SGLang knob families to verify:
tp_size,pp_size,dp_size(replicas),attn_dp_size,ep_sizeattention_backend,prefill_attention_backend,decode_attention_backendsampling_backendmax_running_requests,max_queued_requestschunked_prefill_size,prefill_max_requests,max_prefill_tokensmax_total_tokens,page_size- phase-specific CUDA graph settings
- speculative or EAGLE settings only after the non-speculative baseline is tuned
Keep mem_fraction_static and schedule_policy pinned in the default pass,
matching the shared cookbook config style.
For quick smoke tests, use --cuda-graph-backend-decode disabled and
--cuda-graph-backend-prefill disabled if the goal is only to prove the framework flow. Record those
flags in the artifact. Do not carry that smoke setting into a performance winner
unless the user asked to tune eager-mode serving.
5. Tune vLLM
Use vLLM's sweep runner when available:
vllm bench sweep serve \
--serve-cmd 'vllm serve <model> --port 8000' \
--bench-cmd 'vllm bench serve --backend vllm --model <model> --port 8000 --dataset-name random --num-prompts 80' \
--serve-params /path/to/vllm_serve_params.json \
--bench-params /path/to/vllm_bench_params.json \
--output-dir /path/to/vllm_results
If sweep support is unavailable, run vllm serve for each candidate and measure
with vllm bench serve.
Version-sensitive vLLM knob families to verify:
- tensor, pipeline, data, decode-context, and expert parallelism
gpu_memory_utilizationmax_num_seqsmax_num_batched_tokensmax_model_lenenable_chunked_prefill,long_prefill_token_threshold, and DBO thresholds- KV cache dtype and block size
- dtype and quantization settings
- CUDA graph capture sizes or eager-mode toggles when relevant
- prefix cache and speculative decoding settings only when the workload needs those features
vLLM should get a normal sweep, not one baseline command. See
references/framework-reference.md for
native command templates and cross-framework knob families. Confirm each flag on
the target image's --help before a run.
Keep gpu_memory_utilization in the baseline for the default pass. Search it
only when the question is explicitly about fitting the model or trading capacity
against throughput.
Keep DBO and all2all backend settings out of the default pass unless the target
vLLM environment is already set up for them. They are real tuning knobs, but a
candidate can fail at startup if the required all2all backend is not available.
Verify long_prefill_token_threshold on the concrete vllm serve --help=all
surface before adding it to a search.
6. Tune TensorRT-LLM
Use trtllm-serve serve as the server entrypoint when the target environment
supports it:
trtllm-serve serve <model> \
--tp_size <tp> \
--pp_size <pp> \
--kv_cache_free_gpu_memory_fraction 0.75 \
--host 0.0.0.0 \
--port 8000
Then benchmark the OpenAI-compatible endpoint with the TensorRT-LLM serving benchmark client or with the same OpenAI-compatible client used for the other frameworks.
In the historical TensorRT-LLM 1.0.0 validation image,
benchmark_serving --dataset-name random sampled from ShareGPT unless either
--download-path or --random-ids was passed. For a fast synthetic smoke test,
pass --random-ids, then confirm the behavior on the target TensorRT-LLM image.
TensorRT-LLM flag names are especially version-sensitive. Historically, in the validated
TensorRT-LLM 1.0.0 image, the KV-cache memory flag accepted by
trtllm-serve serve was --kv_cache_free_gpu_memory_fraction, not
--free_gpu_memory_fraction. Current mainline was rechecked at
bb367fc8c1adf6e2c28c88cb1a8b46e1742a9d60 on 2026-10-05. Always verify flags
with trtllm-serve serve --help before running a search on any GPU target.
TensorRT-LLM backend policy for this skill:
- launch the PyTorch-only current server without its deprecated backend option
- use
backend: pytorchonly as an older-image compatibility setting - do not add
backendtosearch_space - reject
trt, engine-backed serving, or any other non-PyTorch TensorRT-LLM server backend as unsupported for this skill
Version-sensitive TensorRT-LLM knob families to verify:
tp_size,pp_size, andep_size- max batch size, max sequence length, max number of tokens, and KV-cache budget
- inflight batching and scheduler options
- extra LLM API options YAML used by
trtllm-servewith the PyTorch backend
The trtllm-serve serve CLI exposes fewer direct runtime knobs than SGLang or
vLLM. Use direct flags when they exist, then use --extra_llm_api_options for
PyTorch-backend settings that are not top-level CLI flags. Keep unsupported
backend or engine requests in the failure table instead of translating them.
Keep kv_cache_free_gpu_memory_fraction in the baseline for the default pass.
Search max_batch_size, max_num_tokens, max_seq_len, and validated
PyTorch-backend config options first. The server backend remains fixed to
pytorch.
7. Tune TokenSpeed
Use TokenSpeed as a first-class comparison framework, especially for agentic or multi-turn workloads where it may be the strongest non-SGLang baseline:
tokenspeed serve <model> \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size <tp> \
--gpu-memory-utilization 0.90 \
--max-model-len 12288 \
--max-num-seqs 64 \
--chunked-prefill-size 8192 \
--kv-cache-dtype auto \
--trust-remote-code
Benchmark either with the common OpenAI-compatible benchmark client:
vllm bench serve --backend openai-chat \
--base-url http://127.0.0.1:8000 \
--model <model> \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 256 \
--num-prompts 80
or with the same OpenAI-compatible client used for the other frameworks.
When TokenSpeed is a likely leader and profiler handoff will be needed, the
gateway can arm torch profiling for the same request shape when its installed
SMG wheel exposes the control route. Set CONTROL_URL to that verified URL;
current serve uses a headless engine plus SMG, not the retired Python
--control-port sidecar. Do not infer the URL by adding one to the serve port:
curl -X POST "${CONTROL_URL}/start_profile" \
-H 'Content-Type: application/json' \
-d '{"output_dir":"/artifacts/tokenspeed_profile","num_steps":5,"activities":["CPU","GPU"],"with_stack":true,"profile_id":"ts-bench"}'
# Drive the same workload with the common client, then POST /stop_profile if needed.
If output_dir is not supplied, TokenSpeed writes under
TOKENSPEED_PROFILER_DIR, defaulting to /tmp.
For large MoE or agentic checkpoints, validate these TokenSpeed knobs before adding them to the search:
tensor_parallel_size,attn_tp_size,moe_tp_size, andenable_expert_parallelattention_backend,drafter_attention_backend,moe_backend, anddraft_moe_backendmax_num_seqs,chunked_prefill_size,max_prefill_tokens, andmax_total_tokenskv_cache_dtype,quantization, and prefix-cache controls- speculative flags such as
speculative_algorithm,speculative_num_steps, andspeculative_num_draft_tokens comm_fusion_max_num_tokens,enable_allreduce_fusion, and related communication-fusion flags only after a target-image smoke run
If a TokenSpeed-native agentic config exists for the same model family, run it
as an additional workload lane rather than replacing the common cross-framework
scenario. Normalize its result rows with the same schema and mark
workload.kind accordingly.
Keep gpu_memory_utilization pinned in the default pass. Search it only when
the user explicitly wants a capacity study.
8. Normalize Results
Write one JSONL row per candidate using the schema in references/result-schema.md. Then run:
python skills/llm-serving-auto-benchmark/scripts/compare_benchmark_results.py \
--input /path/to/candidates.jsonl \
--output /path/to/summary.md
Rank candidates in this order:
- SLA passed
- highest request throughput or goodput
- highest output token throughput
- lower p50 TTFT
- lower p50 TPOT/ITL
- lower GPU count or simpler deployment if performance is close
Keep the SLA gate itself unchanged. In the cookbook configs and normalized
result schema, TTFT SLA still uses max_p50_ttft_ms and TPOT SLA still uses
max_p50_tpot_ms; only the default cross-candidate comparison order switches
to p50 TTFT and p50 TPOT.
Output Contract
Return a compact report with workload/SLA, hardware and framework versions, best deployment-command tables per framework/scenario, one cross-framework comparison table, exact launch and benchmark commands for winners, and artifact paths for workload, raw/normalized results, CSV or markdown summary, and server logs.
When SGLang is not the winner, include a profiler handoff note with the slow
SGLang scenario name and the exact input/output lengths or percentile bucket to
pass to llm-torch-profiler-analysis.
When a candidate uses speculative decoding, prefix cache, offload, or an agentic workload, record the optional normalized fields for accept length, pre-scheduler time, cache hit rate, and memory residency. The summary script will display those columns when present.
Include failed or excluded candidates with reasons. Explain that this table is a record of tried configs that were not selected: candidates that failed, were skipped by policy, or completed but missed the SLA. Add caveats for synthetic workloads, incomplete fair searches, or framework-specific parameter substitutions.
Use references/framework-reference.md when you need command templates, source links, or knob-family mappings. Use references/example-plan.yaml as the starting point for a full cross-framework run plan.
Interface audit — 2026-10-05
SGLang's canonical client is python -m sglang.benchmark.serving; bench_serving
is a deprecated shim and the auto-benchmark module was removed (#31941). Capture
sglang serve --help and the client help from the target image. Current images
require CUDA 13 and a compatible host driver (#38404); the final CUDA 12 tag was
v0.5.19-cu129. MiniMax-M3 requires lmsysorg/sglang:dev-minimax-m3;
Qwen3.8-Flash-Next H200/B200 requires lmsysorg/sglang:qwen38flashnext.
Current SGLang uses phase-specific graph flags (#38375):
--cuda-graph-backend-decode / --cuda-graph-backend-prefill accept
full, breakable, tc_piecewise, disabled; capture sizes use
--cuda-graph-bs-decode / --cuda-graph-bs-prefill and
--cuda-graph-max-bs-decode / --cuda-graph-max-bs-prefill. Smoke eager mode is
--cuda-graph-backend-decode disabled --cuda-graph-backend-prefill disabled.
The old size aliases were removed and --disable-cuda-graph is deprecated.
--attn-dp-size replaces the old attention-DP pair on main (#41818);
--dp-size remains replica DP. v0.5.21 predates this spelling: inspect image help.
Also inspect --attn-cp-size, --dcp-size, --moe-dp-size and the --ep alias.
TensorRT-LLM at bb367fc8 is PyTorch-only after #19028. --backend pytorch
is a deprecated compatibility option; omit it on current main and select it
only on older images whose help lists other backends. Both
--free_gpu_memory_fraction (primary) and --kv_cache_free_gpu_memory_fraction
(alias) work. The 1.0.0 image note is historical. --cluster_size is deprecated
and unsupported. --set PATH=YAML_VALUE can override config paths after --config.
TokenSpeed #1236 removed its benchmark subcommand. Use the common client
vllm bench serve --backend openai-chat for aligned workloads. ts remains
an alias for tokenspeed. The prefix-cache off switch is
--disable-prefix-caching; engine flags --api-key, --enable-cache-report,
--skip-server-warmup, --warmups were removed. --tool-call-parser and
--chat-template are gateway flags. Inspect --pipeline-parallel-size,
--prefill-context-parallel-size, --decode-context-parallel-size,
--lm-head-tp-size and --dense-gemm-backend when tuning parallelism.
Do not force vLLM --block-size 16: it excludes preferred MLA/DSA backends.
Let the backend select its block size, and record its startup dispatch line.
Use the same client version across compared rows: vLLM #55508 changed chat
TTFT/E2E chunk accounting, and SGLang #39889 reports server prompt-token usage
including template tokens. Record optional client queue latency separately.
The default SGLang scheduler is FCFS; historical cookbook YAMLs no longer
force LPM on random prompts. False for defaults-on booleans needs an explicit
off flag and must not silently duplicate the baseline. The YAML validator
requires Python 3.10 or newer.
Source evidence: SGLang graph flags, SGLang parallel flags, vLLM backend sizing, TensorRT-LLM serve, TokenSpeed CLI.
LLM Serving Capacity Planner
Overview
Use this when a serving log has enough memory lines to explain where GPU HBM went. The analyzer reads SGLang/vLLM startup logs, extracts weight load, KV pool, CUDA graph, framework overhead, and token-capacity lines, then estimates concurrent requests for common token lengths.
For DeepSeek-V4.1, read the KV layout and page-allocation contract before using 528 B/288 B in an estimate. These are per-stored-token payload and scale sizes, not allocation after page padding or total model KV bytes. Verify actual pool layout, compression ratios and sharing from the pinned source and logs.
Inputs
Before running analysis, collect or verify these inputs:
| Item | Why it matters | How to obtain | Default if user skips |
|---|---|---|---|
| Log file path | Primary input; all memory data comes from here | Ask user for the serving startup log | — (required) |
| GPU type | Determines total HBM for decomposition validation | Ask user or infer from log | Auto-detected from log if possible |
| nvidia-smi output | Provides per-rank actual memory for cross-validation | Capture with nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt | — (optional, but recommended) |
| Model config.json | Enables theoretical KV cache byte calculation and replication factor analysis | Ask user for the model's config.json path | — (optional, log data used instead) |
| Request token length | Determines concurrency estimate denominator | Ask user | 4096, 6144, 8192 |
Workflow
Step 1: Collect the serving log
The user should provide the startup log from an SGLang or vLLM serving instance. Key log lines that the analyzer needs:
Load weight begin. avail mem=XX GBserver_args={'model_path': '...', 'tp_size': 8, ...}(resolved parameters)Load weight end. ... avail mem=X GB, mem usage=X GB.(weight footprint)DSV4 memory calculation: ... bytes_per_full_token=X, available_bytes=X GB, ... full_token=N[label] KV Cache is allocated. dtype: ..., #tokens: N, K size: X GB, V size: Y GB(orKV size)Mamba Cache is allocated. ... conv_state size: X GB, ssm_state size: Y GBMemory pool end. avail mem=XX GBCapture target|draft decode|verify|prefill CUDA graph end. elapsed=X s, mem usage=X GB, avail mem=X GB.Post-capture KV sizing: KV cache allocated. ... KV size: X GB, avail mem=X GB(opt-in)max_total_num_tokens=XX, ... chunked_prefill_size=-1|N, ... available_gpu_mem=XX GB
Legacy ServerArgs(...), Memory profiling: and Capture cuda graph end.
remain accepted for older/private branch logs; they are not current main emitters.
Current vLLM V1 logs instead expose:
Initial free memory: ...; Requested memory: ...(DEBUG only)Model loading took ... GiBAvailable KV cache memory: ... GiBGraph capturing finished ... took ... GiB<device> KV cache size: ... tokens, Maximum concurrency for ... tokens per request: ...xFree memory on device (F/T GiB) ... Current kv cache memory in use is K GiB(INFO)CUDA graph pool memory: A GiB (actual), E GiB (estimated), ...Initial free memory X GiB, reserved Y GiB memory for KV Cache ...(explicit KV bytes)Maximum concurrency for ... tokens per request: ...x
If the log is from a running instance, capture it by redirecting stdout/stderr to a file at launch time.
Step 2: Optionally capture nvidia-smi data
For per-rank memory comparison:
docker exec <container> nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt
Step 3: Run the analyzer
python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
--log-file /path/to/sglang.log \
--nvidia-smi-file /path/to/smi.txt \
--gpu h200 \
--config-json /path/to/config.json
For JSON output (automation):
python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
--log-file /path/to/sglang.log \
--format json
Step 4: Review and interpret results
The analyzer prints:
- Memory breakdown table: each category (weights, KV pool, CUDA graph, framework, other) with GiB, MiB, percentage, and derivation
- Per-rank comparison: nvidia-smi data across all TP ranks
- KV pool detail: pool configuration, KV dtype, replication factor, per-token byte calculation
- Concurrency estimate: max concurrent requests for different token lengths
- Tuning notes: configuration changes that may increase capacity
For vLLM, the report also includes a vLLM Startup Evidence section. Values
that are not present in the log stay unknown; do not translate them into
SGLang-only fields.
When To Use It
- After launching an LLM serving instance, to understand how GPU memory is distributed
- When comparing different
--mem-fraction-staticvalues and their impact on KV pool capacity - When planning deployment capacity: how many concurrent requests can a given GPU configuration support
- When investigating OOM issues: identifying which memory category is consuming the most
- When evaluating whether fp8 KV cache or EP can improve concurrency
Key Concepts
mem-fraction-static
The inspected SGLang source reserves runtime headroom from pre-model-load
free memory: headroom = pre_model_load_memory * (1 - mem_fraction_static).
The KV budget starts from currently free memory minus this headroom and other
model-specific reservations. It is not post_weight_free * fraction.
Post-capture sizing may update target and eligible DFlash-family draft pools
in one round. Sum target plus distinct draft-owner summaries per rank; a new
target summary resets the round. VA upper bounds reserve address space rather
than physical bytes. Keep independent pools and recurrent state separate.
Use final logged capacity and
rank-local evidence; see source contracts.
The default is auto-resolved from the hardware and configuration. Read the
resolved value from server_args={...}. A lower value increases runtime
headroom at the cost of KV capacity. Do not infer 0.88 when it is absent.
For deployments actually using FlashInfer's Cake MoE all-reduce with an omitted all-reduce output, account separately for its live and retired scratch sinks. They grow geometrically and retain addresses used by CUDA graphs; they are not KV cache or necessarily reclaimable when a request ends. Startup KV logs alone cannot identify this allocation history. The address-retention behavior was introduced in FlashInfer #5894.
KV Head Replication
For standard GQA/MHA, each rank stores max(1, num_key_value_heads // tp_size) KV heads. When TP exceeds the head count, each head is replicated across a subgroup of ranks, not the whole head set on every rank. For kv_heads=4, tp=8, each rank stores one head; for kv_heads=1, tp=8, all eight ranks store that single head. MLA stores a replicated latent plus RoPE key instead. The theoretical calculation assumes all configured layers reside on the rank; use rank-local layer counts for pipeline parallelism.
SWA (Sliding Window Attention) Compression
Models like DeepSeek-V4 use CSA (Compressed Sliding Attention) and HCA
(Hierarchical Context Attention) with sliding windows. This reduces per-token
KV cache bytes compared to the theoretical full-attention calculation. The
bytes_per_full_token reported in the log already accounts for this
compression.
Reporting Checklist
Include:
- Serving configuration: model, GPU, TP/PP/EP, mem-fraction-static, kv-cache-dtype
- Memory breakdown table: category / GiB / MiB / percentage / derivation source
- Per-rank nvidia-smi comparison: used and free memory per TP rank
- KV pool detail: pool size, bytes_per_full_token, KV dtype, replication factor, theoretical per-token KV calculation (when config.json provided)
- Concurrency estimate table: request token length / token-limit / request-limit / max concurrent
- Tuning notes based on free memory and configuration
2026-08-23 Runtime Notes
SGLang v0.5.18 consolidates compiled-kernel caches under SGLANG_CACHE_DIR.
The first launch after an upgrade can look like extra "other" memory and a
longer startup because Triton / FlashInfer / DeepGEMM / Inductor recompile
once. Overlapped checkpoint staging (--startup-weight-load-mode overlap)
also changes the startup memory curve: weight pages and CUDA-graph capture
can overlap, so do not treat a shorter load-weight interval as a smaller
weight footprint.
Known Limitations
| Limitation | Detail | Workaround |
|---|---|---|
| Framework-specific checkpoints | SGLang and vLLM expose different memory checkpoints, so some fields are not comparable one-to-one | Report unknown fields and compare only shared, explicit evidence |
| SWA compression models | Per-token KV bytes cannot be independently calculated from model config for CSA/HCA attention — the framework's internal SWA window parameters are needed | Use bytes_per_full_token from the log directly |
| DeepGEMM JIT memory | The analyzer categorizes DeepGEMM JIT compilation memory as "other" because it is not explicitly reported in the log | Compare with nvidia-smi total for accurate accounting |
| PP (Pipeline Parallelism) | Memory decomposition is per-rank; PP configurations may have uneven memory across stages | Specify --target-rank for each PP stage |
| MoE expert buffer | Some frameworks allocate additional buffers for expert routing that are not separately reported | Included in "model weights" or "other" depending on when allocated |
References
references/log-patterns.md: log line patterns and their semantics for memory analysis.references/gpu-specs.json: GPU HBM specifications forh20,h100,h200, andb200aliases.scripts/capacity_analyzer.py: the core analysis script.
Memory capacity sources
The bundled hardware table records vendor GB maxima, not usable GiB. Prefer
nvidia-smi used+free or vLLM's logged total. B200 retains the DGX 180GB SKU;
other Blackwell SKUs may differ. DGX Spark's 128GB is unified system memory,
not dedicated HBM. L20 is the official 48GB SKU; do not infer a capacity for
l20n from the name. GB200 records the physical 192GB package maximum; its deployed SKU and
custom SKUs require measured totals. Official
source URLs are stored with each newly added hardware entry.
LLM Torch Profiler Analysis
Produce three tables: kernel/source attribution, overlap opportunities, and fusion patterns. Start from the user's trace or running server. Inspect the actual model, framework revision, phase, device/rank and parallelism before choosing a fast path. Existing traces need no GPU or framework installation. The Python analyzers use the standard library.
Read source contracts for dated source inspections and version-sensitive profiler APIs. Apply the general dispatch and numerical principles to every model. The fusion catalog contains concrete operator families, eligibility examples and storage contracts; the overlap catalog covers scheduling. Consult the model-history knowledge base when detailed PR evidence is needed. Source checks and historical captures are not fresh GPU validation.
Choose the evidence
| Mode | Evidence and limits |
|---|---|
| Existing trace | All four frameworks; SGLang Omni Python roots are also recognized. Use Chrome JSON or gzip with GPU events. A generic trace may need --framework explicitly. |
| Single live capture | Use a shared server/client output directory and a supported HTTP profiler. Save actual launch arguments and package revisions. |
| Mapping + formal | Eager/graph-off trace supplies call-site context; warmed graph trace supplies timing. Match signatures, shapes and stage before transferring attribution. Never transfer eager timing to the formal run. |
For a diffusion workload, record whether it uses the framework's native backend or a Diffusers fallback. A fallback trace can be analyzed, but does not establish native-kernel performance or dispatch.
Read a trace
Commands below are relative to this skill directory.
python3 scripts/analyze_llm_torch_profile.py \
--framework sglang --input /path/to/TP-0-DECODE.trace.json.gz
python3 scripts/analyze_llm_torch_profile.py \
--framework vllm \
--mapping-input /path/to/eager-mapping \
--formal-input /path/to/warmed-graph
Do not collapse a distributed run to rank0 when investigating rank skew or collective waits. Keep the original capture; a GPU-only or compact export is for sharing, not replacement evidence. Use torch-profiler-layer-track for verified L0/L1 guides and synthetic display lanes, or llm-pipeline-analysis for layer intervals.
Capture against a running server
The common helper defaults to separately driven prefill and decode workloads: nominal input/output lengths 4090/1 and 1/2048, with 10 warmup steps and 5 requested capture steps. Override these to match the slow serving scenario. Synthetic prompt lengths are not a tokenizer-independent guarantee; save actual usage and batch/graph dimensions. A long-context decode bottleneck needs a representative KV length, not automatically the one-token-input default.
python3 scripts/analyze_llm_torch_profile.py \
--framework sglang --url http://127.0.0.1:30000 \
--output-dir /shared/profiles/run-001 \
--profile-workload both --warmup-steps 10 --num-steps 5 \
--prefill-input-len 4090 --prefill-output-len 1 \
--decode-input-len 4090 --decode-output-len 2048
--profile-workload accepts both, prefill, decode, or legacy (explicit
probe prompt/token settings). Workload labels alone do not make a trace purely
decode: inspect stage annotations and exclude request prefill when needed.
The helper targets SGLang profiler v1 (default). Declare --sglang-profile-v2
if the remote server enables SGLANG_PROFILE_V2=1; the helper refuses this
incompatible flow before issuing control requests. /server_info does not
expose remote environment variables. Native v2 needs profile_by_stage:true
and num_steps, without start_step, merge_profiles or /stop_profile.
Current Rust ingress also supports start/stop with the same scheduler control.
Older images may lack these routes; check the actual response. Keyed servers
need the configured admin/API credential; the bundled helper has no auth flag,
so use authenticated native control and offline analysis for those captures.
The host launcher explicitly sets SGLANG_PROFILE_V2=0.
The helper puts its label in profile_id, so merging can discover rank files.
SGLang can emit stage-specific traces; its scheduler stop counter may require
a final forward to close the window. The helper's SGLang workload adds a guard
request/step. Check actual trace counts instead of assuming identical windows
across frameworks.
For other frameworks add --no-profile-by-stage. Their HTTP windows and
iteration controls differ:
- vLLM: launch with
--profiler-config '{"profiler":"torch","torch_profiler_dir":"/shared/profiles/run-001","max_iterations":5,"ignore_frontend":true}'. Current start accepts per-sessionprofile_prefix,max_iterations,delay_iterationsandprofiler_kwargs. The helper checks/openapi.jsonand sends a stage-qualified prefix and--num-stepsasmax_iterationsonly when advertised. Older/OpenAPI-disabled servers retain bodyless control and launch-configured limits. Output directory and optional profiler warmup remain launch settings for this helper; probe steps do not override a warmup/wait/active schedule or guarantee a pure decode window.torch_profiler_with_stackdefaults true in the inspected source. Frontend traces can contain no kernels; select GPU-worker traces for GPU analysis.active_iterationsonly takes effect with a warmup/wait schedule; usemax_iterationsto bound workers. The host option is--profiler-max-iterations(--profiler-active-iterationsremains an alias). Two capture rounds require vLLM #57460, merged 2026-09-21; older images need separate launches for prefill and decode. Keep CPU intorch_profiler_activitiesfor source attribution and step annotations; CUDA-only activities omit both Python stacks and annotations. - TensorRT-LLM: the inspected PyTorch server accepts runtime start with
output_dirandactivities; the helper sends both and stops after probes. Native traces are uniquely named per request and rank. There is no need for the retiredpy_executor.pyrank0-only patch or host-specific matrix runner. Current setup is in_torch/pyexecutor/profiling.py, still withoutwith_stack=True. Report unresolved Python attribution, or make a separate, recorded mapping build against that installed file. Older images without HTTP support must use their documented capture method plus--input. - TokenSpeed: the helper sends
output_dir, CPU/GPU activities,with_stack=true,record_shapes=false, and a stage-qualifiedprofile_id. Current servers expose start/stop; verify the deployed image. Proton's native Hatchet artifacts are not Torch Chrome traces. Currentserveuses SMG plus a headless Rust-hosted engine; the old Python control sidecar was removed. Resolve the actual gateway/control URL and validate forwarding on the installed pinned wheels. Do not assume serving port + 1.EXPERT_LOADwrites.expert-load.pt, not Chrome traces.TOKENSPEED_CUPTI_GRAPH_WARMUP=1can leave zero GPU events in later captures. Current Torch capture requires process-globalprofile_all_threadssupport for Rust-owned forward threads. AMD capture needsHSA_TOOLS_DISABLE_REGISTER=0; Proton cannot combine with GPU/CUDA_PROFILER, and expert-load recording refuses online EPLB. See the source contracts.
python3 scripts/analyze_llm_torch_profile.py \
--framework trtllm --url http://127.0.0.1:8000 \
--output-dir /shared/profiles/run-002 \
--no-profile-by-stage --profile-workload both
A server-side directory must be mounted here or copied back for analysis. The helper rejects a capture with no new artifact, recognizes overwritten filenames, and copies fresh remote traces into a capture-specific subdirectory without moving the server's originals. Preserve that returned directory rather than analyzing every historical run under a shared root. Concurrent profiling sessions still need separate output directories.
Native alternatives include SGLang graph-capture traces under
graph_capture_profile/, MEM snapshots, and CUDA_PROFILER with nsys;
vLLM CUDA/NVTX and Proton Chrome export (the old capture_torch_profiler
option is deprecated and has no effect);
TRT-LLM TLLM_PROFILE_START_STOP/TLLM_TORCH_PROFILE_TRACE; and TokenSpeed
VizTracer/Proton plus tokenspeed merge-traces. Graph-capture and VizTracer
artifacts are excluded from default runtime-trace discovery. Pass a file
explicitly when intentionally studying one of these artifacts. Native Hatchet,
MEM and expert-load outputs require their own readers.
Interpret before optimizing
- Confirm nonzero GPU events, a complete warmed forward, actual dispatch and
target/draft separation (
step[VERIFY ...]versusstep[DRAFT ...]in SGLang; vLLM usesexecute_context_*_generation_*). BS=1 need not mean kernel M=1 during speculation. - Read all relevant GPU streams. A CPU
cudaGraphLaunchspan is host time; missing lanes or dropped CUPTI records can look like GPU idle time. - Separate sum of kernel durations, union of GPU busy intervals, and end-to-end wall time. Overlapping kernels can sum to more than wall time. A table's duration share is not removable latency or an additive speedup estimate.
- Identify the producer/consumer join and exposed tail. PDL consumers may include dependency wait in kernel duration. Compare PDL on/off with the same warp configuration before attributing a bimodal duration to arithmetic.
- Consult source dispatch before proposing fusion. Existing routed-expert activation/quant does not establish shared-expert coverage. Match rounding, clamp, scale/layout, padding and graph reuse contracts. Check a quantization oracle against an independent formula: a kernel and its reference can share the same scale-floor error.
- Follow state ownership across fused boundaries: cached normalized rows, quantized values/scales and next coefficients can become invalid after a residual rewrite or row selection. V4.1 Engram/late-layer selection are concrete examples; fewer launches do not prove equivalent semantics.
- Time changes without profiling; run kernel correctness and actual-model accuracy on the candidate path. NCU replay/resource metrics help explain a kernel but can perturb or serialize the multi-stream behavior being studied.
For source mapping, prefer model/layer frames over dispatch/runtime wrappers. Retain unknown locations as unknown. Kernel names and eager similarities are hints; inspect the concrete producer, consumer and backend. The script's three tables are observations and candidates, not proof that an optimization is safe or already active in a different framework.
Capture labels must be filename components: the helper rejects path separators
and NUL before contacting the server. Live capture requires positive num_steps.
SGLang v1/v2 default to stacks on and shapes off; request shapes explicitly when
needed. Verify emitted phases and filenames against the deployed server.
References and output
Load only relevant references:
- source-map.md: immutable profiler and worker paths.
- heuristics.md: attribution and overlap labels, dispatch, numerical contracts, validation and their limitations.
- vllm-torch-compile-fusions.md: current pass registration, platform gates and source patterns.
- fuse-overlap-catalog.md and overlap-catalog.md: historical precedents; recheck a PR's current state and target revision before borrowing code. The overlap catalog also covers GPU idle gaps, scheduler overlap, phase graphs, metadata glue graphs, replay streams, PDL and collective dispatch.
Return the trace/report path, framework and available model/server arguments; the kernel, overlap-opportunity and fusion-pattern tables; the main exposed bottleneck; attribution confidence and whether mapping/formal traces were used. For a new capture also state which phases/ranks contain GPU events. Do not claim a speedup or accuracy equivalence without the corresponding experiment.
Model Architecture Diagram
Workflow
Return only public original diagrams indexed by this skill.
- Run the bundled resolver:
python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py "<model name>"
- If the resolver returns
kind: existing, return the raw image Markdown it prints and preserve the source attribution line. - If the resolver returns
kind: no_match, tell the user that no public original architecture diagram is indexed for that model.
Source Priority
Use references/diagram-index.json as the source of truth. It stores raw GitHub image URLs and official vendor/CDN or pinned Hugging Face
resolve/<sha> URLs. Official sources include Qwen and XiaomiMiMo; GitHub sources:
datawhalechina/self-llmCalvinXKY/InfraTechTongyi-MAI/Z-ImageWan-Video/Wan2.1Wan-Video/Wan2.2Tencent-Hunyuan/HunyuanVideoTencent-Hunyuan/Hunyuan3D-2brayevalerien/Flux.1-Architecture-Diagram
Prefer detailed implementation, cookbook, or architecture-card diagrams over paper figures. Good sources show module boundaries, dataflow, MoE / attention / cache paths, or model-specific runtime structure rather than only a high-level paper overview. Official repository diagrams and curated implementation diagrams are first choice; paper figures are fallback only when no more detailed public original diagram is indexed.
Do not copy remote image binaries into the skill. Return the indexed original URLs so the chat renderer can display the original image.
Existing Diagram Rule
For a direct match, show the original image. Good direct matches include:
- DeepSeek V3/V3.2/V4, GLM-5, Kimi K2/K2.5/K3, MiniMax M2.5, Qwen3.5, Qwen3-VL, and Step 3.5 Flash from InfraTech.
- Qwen3.8-Flash-Next has an official diagram (also named
qwen4_expin runtime code). Plain Qwen3.8 2.4T remainsno_match; these are different architectures. - MiMo V2.6, MiMo V2.5 Pro and GLM-5.2 have indexed originals. GLM-5.3 remains
no_matchbecause no direct diagram was verified. - Hunyuan-A13B, Kimi-VL, Qwen3, Qwen3-VL detail flows, MiniMax M2, and Llama 4 architecture/module diagrams from self-llm.
- Z-Image, Wan2.1, Wan2.2, HunyuanVideo, Hunyuan3D 2.0, and FLUX.1 diffusion architecture/module diagrams from public GitHub sources.
If multiple diagrams match, show all high-confidence matches up to the resolver's default limit. For example, DeepSeek V3 may return the full architecture plus MLA MHA/MQA diagrams.
Hosted Original Diagram Gallery
Do not commit the sgl-cookbook-model-architecture-images/ gallery into the repository. The public-original image set is hosted as a GitHub Release asset and indexed by a GitHub issue.
Current hosted artifact:
- Issue index: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/issues/31
- Release page: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/releases/tag/sgl-cookbook-architecture-images-2026-05-02
- Zip download: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/releases/download/sgl-cookbook-architecture-images-2026-05-02/sgl-cookbook-model-architecture-images-2026-05-02.zip
- Digest:
sha256:ea432081849a250429d3d1ecf246e267c5cc42f989aaf4b9ca695b581e7fa50f
The gallery is a 2026-05-02 snapshot containing 44 public original diagram image files; it does not mirror the current index, plus a lightweight index.html, index.md, manifest.json, HTML contact sheet, and architecture-audit.md.
To inspect the gallery locally:
curl -L -o /tmp/sgl-cookbook-model-architecture-images-2026-05-02.zip \
https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/releases/download/sgl-cookbook-architecture-images-2026-05-02/sgl-cookbook-model-architecture-images-2026-05-02.zip
unzip -q /tmp/sgl-cookbook-model-architecture-images-2026-05-02.zip -d /tmp
open /tmp/sgl-cookbook-model-architecture-images/index.html
Useful Commands
List known original diagram aliases:
python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py --list-known
Emit JSON for automation:
python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py "GLM-5" --format json
References
references/diagram-index.json: original diagram link index and aliases.references/source-notes.md: audited source repositories and local cache paths.
Model Compute Simulation
Overview
Use this when the question is about operator order, tensor dimensions, FLOPs, MFU, or parallelism checks. The simulator loads a model config, builds the representative operator sequence, prints tensor shapes and FLOPs, and can estimate MFU from measured latency.
Inputs
Before running a simulation, collect or verify these inputs:
| Item | Why it matters | How to obtain | Default if user skips |
|---|---|---|---|
| Model name | Resolves to config in model-config-index.json; determines entire architecture | Ask user or infer from trace context | — (required) |
| Config accuracy | Indexed values may differ from actual serving config (e.g. routed_expert_intermediate_size, compress_ratios) | Ask user to provide config.json or verify key params against HuggingFace | Use indexed values with a caveat |
| GPU type | Determines peak FLOPS for MFU denominator | Ask user | — (required for MFU) |
| dtype (bf16 / fp8) | Select the actual GPU SKU and arithmetic peak; mixed precision needs per-op treatment | Ask user | bf16 |
| Batch size & seq len | Directly affects FLOPs and tensor shapes | Ask user | B=1, S=1 (decode) |
| TP / DP / EP | TP splits GEMM FLOPs across GPUs; EP splits expert FLOPs | Ask user | Read the run manifest; otherwise report assumptions explicitly |
| Measured latency (ms) | Required for MFU numerator; must be per-GPU forward-pass wall-clock | Ask user or extract from a profiler trace | — (optional, no MFU without it) |
If the model is not in model-config-index.json, ask the user for a
config.json path or add an indexed config before running estimates.
The 2026-10-05 index pins public HF config revisions for V4 Flash/Pro/V4.1,
K3, GLM-5.2/5.3/5.3-Flash, Qwen3.8-Flash-Next, Hy3/Hy4, Inkling, dots3,
Step-3.7, Nemotron-3/3.5, LongCat and Gemma4, and corrects older model dimensions.
Qwen/Qwen3.8-A95B returned HTTP 401 at this audit and is not indexed.
Use --config /path/to/config.json for a raw nested HF config; the local
normalizer expands text wrappers and expert/layer aliases.
Shape and timing limits for speculative MoE
For DSV4.1, obtain actual target/draft row counts from the trace: request BS=1 can issue six target-verify rows with DSPARK block size 5. Do not substitute acceptance length for the number of rows computed. Account for TP-sharded shared-expert shapes separately from routed expert assignments and EP. Do not divide an unverified model template by TP/EP and call it measured FLOPs. The simulator applies uniform TP division outside routed experts and uniform EP division inside routed experts. This is a what-if approximation: replicated routers, norms, latent projections and backend-dependent shared-expert ownership require a run-specific adjustment before interpreting MFU. The pinned V4.1 entry is a static target-model template; DSPARK draft, vision, Engram/PLE memory traffic and backend dispatch require separate evidence.
Current SGLang detailed_annotations can emit c_ (context) and g_
(generation) aggregates: sq=ΣNq, sk=ΣNkv, sqsq=ΣNq²,
sqsk=ΣNq·Nkv. Verify/draft Nq includes speculative width. Mixed batches
emit both groups; replacing this distribution with average lengths loses
attention work. Use these values to audit shapes manually against
source contracts. The simulator
still accepts a homogeneous B/S shape and does not parse these suffixes or
model arbitrary sparse attention from their sums.
Kernel-duration sums include overlap and possibly PDL dependency waits. Use wall-clock time for a whole-pass MFU denominator, and measured GPU SKU/dtype peaks rather than assuming every FP8 operation doubles useful FLOPs. Follow the dispatch and timing principles when assigning kernel names to attention, shared experts, routing or mHC.
Workflow
Step 1: Load model config
Resolve the model name and load its configuration parameters:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "<model name>" --list-models
The script resolves the model name against references/model-config-index.json, which stores public HuggingFace config parameters (hidden_size, num_experts, MLA ranks, etc.).
If the model is not indexed, tell the user to provide a config.json path or request an index update.
Step 2: Generate execution flow and tensor dimensions
Run the simulator with batch size, sequence length, and parallelism configuration:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 1 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16
The simulator prints:
- Per-layer operator sequence with FLOPs and tensor shapes (shape_in → shape_out)
- Attention vs MoE/FFN FLOPs proportion per layer
- Total model FLOPs for a single forward pass
For decode: use --seq-len 1 --context-len TOTAL_KV_LENGTH.
For extend, --context-len is the prefix length, with causal work for the new
--seq-len tokens. Full/window attention includes KV context in QK and AV FLOPs.
The LM head computes last-token logits unless --all-token-logits is given.
Every run prints batch/shape/parallelism/GPU/dtype assumptions to stderr.
Sparse selection, KDA/Mamba scalar work, norm/Sinkhorn and zero-expert top-k
are architecture estimates, not measured instruction counts. LongCat routed
FLOPs assume all top-k are nonzero (an upper bound until routing is measured).
For prefill: use --seq-len <prompt_length>.
Step 3: Estimate MFU with measured latency
Provide the measured forward-pass latency to compute MFU:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 1 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 15.0
MFU = theoretical_min_time / measured_time × 100%
The simulator prints:
- Overall MFU
- Per-layer MFU (uniform layer-time assumption)
- Per-operator FLOPs proportion (for identifying which ops dominate)
GPU peak FLOPS are loaded from references/gpu-specs.json. The bundled
hardware table includes H100/H200/B200, B300, GB300, MI355X, RTX PRO 6000
and DGX Spark, each with official URLs. H20 is retained as an unverified
historical estimate. Unknown dense peaks are null and MFU for that dtype is
refused; RTX Server and Spark sparsity is unconfirmed. B300/GB300 memory
capacity is physical maximum, not usable allocation. --dtype accepts bf16,
fp8/mxfp8 and fp4/mxfp4/nvfp4; INT8 must use its separately published peak. Use aliases such as --gpu h100, --gpu b2000, or --gpu b200 when
running on those local boxes.
Step 4: Per-operator MFU with kernel-level latency
When you have per-kernel measured latency, compute per-operator MFU by mapping kernel durations to the compute flow.
Method A: --kernel-flow (kernel-level MFU, recommended)
Provide per-kernel detail as JSON, then feed it to the simulator for kernel-level MFU analysis. This preserves every kernel row from the compute flow and adds FLOPs/MFU columns.
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-flow @/tmp/layer3_detail.json
The --kernel-flow parameter accepts a JSON string or @file path. It produces
a kernel-level MFU table that preserves all kernel rows from the compute
flow and adds:
Mapped Op: which operator this kernel maps toFLOPs: operator's total FLOPsTheo(us): theoretical minimum timeMFU%: measured FLOPs utilizationshape_in→shape_out: operator tensor dimensions
When --kernel-flow is provided, the static per-operator template is omitted
because the kernel-level MFU table already carries per-kernel shape and FLOPs
information. The output keeps the model summary, serving configuration, total
FLOPs, overall MFU calculated from the measured layer duration, and the
kernel-level MFU table. In --format json, the raw evidence is stored under
kernel_flow and no text table is appended.
Mapping rules:
- Direct-match kernels (mla, moe, mhc, rmsnorm, hadamard, rope, quant, topk, etc.): time is assigned directly to the corresponding operators
- Generic GEMM kernels (gemm_fp8, gemm_bf16): time is distributed to remaining unassigned projection GEMM operators by FLOPs share
- Overhead kernels (allreduce, moe_align, moe_sort, other): rows preserved, FLOPs/MFU marked as N/A
FP8 kernel MFU correction: Kernels in categories moe (fused_moe_kernel)
and gemm_fp8 use fp8 math internally even when --dtype bf16 is specified.
For these kernels, the MFU denominator uses the GPU's fp8 peak FLOPS
(2x bf16 peak) instead of bf16 peak. The resulting MFU is marked with a
superscript ⁸ (for example, 63.7%⁸) to show that the fp8 denominator was
used. gemm_bf16 kernels still use the bf16 peak FLOPS denominator.
Method B: --kernel-detail (operator-level MFU, legacy)
Same input as --kernel-flow but outputs an operator-level summary table
(aggregated by operator, not per-kernel). Use when you want a compact view.
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-detail @/tmp/layer3_detail.json
Method C: --kernel-ms (category-level approximation)
Use a hand-aggregated category map only when per-kernel detail is unavailable:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
--batch-size 1 --seq-len 8192 \
--tp 8 --dp 1 --ep 8 \
--gpu b200 --dtype bf16 \
--measured-ms 100 --kernel-ms '{
"mla": 4.922, "moe": 1.644, "allreduce": 0.769,
"hadamard": 0.348, "mhc": 1.388, "gemm_fp8": 1.692,
"gemm_bf16": 0.125, "rmsnorm": 0.227, "quant": 0.311,
"rope": 0.209, "topk": 0.122, "activation": 0.071,
"other": 0.437
}'
The --kernel-ms parameter accepts a JSON object mapping kernel category names
to their measured durations in milliseconds. It uses FLOPs-proportional
distribution across entire categories, which is less precise than --kernel-detail
because generic GEMM categories (gemm_fp8, gemm_bf16) span multiple operator categories.
Choose only one of --per-layer-ms, --kernel-ms, --kernel-detail, or
--kernel-flow. All measured durations must be positive. The three kernel
inputs also require --measured-ms from whole-pass wall-clock timing; the
100 ms in these examples is a placeholder to replace with the measurement.
Kernel-duration sums cannot establish elapsed time when streams overlap.
Output includes:
- Model architecture summary (layers, hidden_size, attention_type, MoE config)
- Per-layer compute flow: operator sequence with tensor dimensions, FLOPs, shape_in→shape_out
- Per-operator MFU table: each operator's FLOPs, theoretical time, measured time (from trace), MFU%
- Kernel → operator mapping explanation (direct-match vs FLOPs-proportional vs overhead)
- Overall and per-layer MFU
When To Use It
- when you need compute-level detail for a known model or config
- when the user asks about execution flow, tensor dimensions, or FLOPs for a specific serving shape
- when the user asks about MFU and can provide measured forward-pass latency
- when comparing compute profiles across different parallelism configurations
Useful Commands
List known model IDs:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-models
List known GPU types:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-gpus
Emit JSON for automation:
python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "GLM-5" --format json
Reporting Checklist
Include:
- Model architecture summary: model name, config source, num_layers, hidden_size, attention_type, MoE config (num_experts, topk, shared_experts), MHC, head_dim
- Serving configuration: batch_size, seq_len, TP, DP, EP, GPU, dtype
- Per-layer compute flow (showing first representative layer in detail):
- Operator sequence table: name, category, FLOPs, shape_in → shape_out
- Attention vs MoE/FFN FLOPs proportion
- Total model FLOPs for a single forward pass
- Kernel-level MFU table (when
--kernel-flowprovided):- Preserves ALL kernel rows from the compute flow (never deleted)
- Per-kernel columns:
# | Half | Category | Simplified Name | dur(us) | % | Mapped Op | FLOPs | Theo(us) | MFU% | shape_in→shape_out - Direct-match kernels: show mapped operator FLOPs/MFU
- Overhead kernels: show N/A for FLOPs/MFU, row preserved
- Operator-level MFU table (when
--kernel-detailor--measured-msprovided):- Each operator: name, category, total FLOPs, per-GPU FLOPs, theoretical time, measured time (from trace), MFU%
- Kernel category → operator mapping explained
- Overall MFU and per-layer MFU
- One-line summary: dominant compute category, MFU status, key bottleneck
Trace-Based Validation (extract_compute_flow_from_trace.py)
Use scripts/extract_compute_flow_from_trace.py to extract the real operator sequence and tensor dimensions from a torch profiler trace, then compare against the static template as ground truth validation.
# Extract compute flow from a trace
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
--input /path/to/trace.json.gz --format text
# Compare trace against static template
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
--input /path/to/trace.json.gz \
--compare qwen3-235b-a22b \
--batch-size 1 --seq-len 1 --tp 8 --ep 8
Trace/template operation presence is compared through semantic families such
as matmul, attention, norm, and router; raw aten::* names are not
compared directly with semantic template names. A positive --min-flops
threshold removes operations whose FLOPs could not be determined.
Compute Flow Confirmation Hierarchy
When the static template or trace extraction cannot fully confirm the compute process (e.g. ambiguous scope, missing shapes, new model architecture), follow this escalation hierarchy:
-
Static template (
model_compute_simulator.py+model-config-index.json) — fast, covers known models -
Trace extraction (
extract_compute_flow_from_trace.py) — validates template against real execution -
Inference framework source code — when trace is insufficient (missing
Input Dims, CUDA Graph replay, compiled kernels without scope), read the model's forward flow directly from the serving framework source:- SGLang:
python/sglang/srt/models/<model_name>.py— contains theforward()method anddeclare_attn/declare_ffnstage boundaries inpython/sglang/srt/layers/layer_boundary/factories.py - vLLM:
vllm/models/<family>/{common,nvidia,amd}/for migrated 2026 families; older models remain undermodel_executor/models - TensorRT-LLM:
tensorrt_llm/_torch/models/modeling_<family>.py+ model configs
When consulting framework source, focus on:
- The
forward()method: operator call order and residual connections - QKV / O projection: whether LoRA-style down/up projections are used (
q_lora_rank,o_lora_rank) - MoE routing: top-k selection, shared vs routed expert split
- TP/EP slicing: which dimensions are split and how FLOPs divide across GPUs
- Any model-specific ops not in the static template (e.g. MHC, Hadamard, indexer)
Action: If the framework source reveals discrepancies with the static template, update
model-config-index.jsonand/orbuild_layer_ops()accordingly. - SGLang:
Limitations of Trace Extraction
| Limitation | Detail | Workaround |
|---|---|---|
record_shapes=True required | Trace must be captured with shape recording enabled; without it, Input Dims fields are absent and FLOPs cannot be computed | Enable shape recording explicitly in the deployed profiler: stacks and shapes are distinct settings. SGLang defaults may be disabled by environment; TokenSpeed helper sends record_shapes=false. Inspect TensorRT-LLM _torch/pyexecutor/profiling.py for a recorded mapping build |
| CUDA Graph mode | During graph replay, cpu_op events may only appear once (at capture time); shape information for replayed iterations is not re-recorded | The script detects graph capture phases and annotates affected ops; use eager-mode traces for full coverage |
| TP-sliced dimensions | Trace shows post-TP-split dimensions (e.g. H/TP), not the full-model view | Use --tp in --compare mode to scale trace FLOPs back to full-model equivalents |
| Scope attribution quality | Python scope depends on with_stack=True; some frameworks or compiled paths may produce shallow or missing scope chains | Graceful degradation: ops with unresolved scope are categorized as "other" |
| Not a replacement for static templates | Trace extraction is a validation and discovery tool; static templates remain the primary fast-analysis path | Use trace extraction to verify templates for new models, then update model-config-index.json if discrepancies are found |
References
references/model-config-index.json: model configuration parameters (hidden_size, expert counts, MLA ranks, etc.).references/gpu-specs.json: GPU peak FLOPS specifications for MFU calculation.scripts/extract_compute_flow_from_trace.py: trace-based compute flow extraction and template validation tool.
Source audit: SGLang V4,
K3 latent experts,
NVIDIA HGX dense/sparse specs,
AMD MI355X specs.
HF field provenance is hf_config_source at hf_config_revision; construct
https://huggingface.co/<repo>/blob/<revision>/config.json for the exact source.
No GPU run is implied by a config refresh.
SGLang explainer video
Input: sglang PR <N> (or a github.com/sgl-project/sglang/pull/N URL), or a Markdown article path.
Output: ~/Desktop/SGLang_PR<N>_讲解_中英字幕.mp4 (article: SGLang_<slug>_技术解读_中英字幕.mp4) plus a still …_封面.png and a Bilibili cover …_封面_B站.png, rendered 1920×1080 with a 中文 narration and bilingual captions.
This skill drives HyperFrames' pr-to-video workflow (heygen-com/hyperframes, pinned) with a fixed house
style and several fixes that make it work for Chinese. Read references/house-style.md before writing any
script — it holds the rules the user already corrected once (terms, captions, cover, credits, PR status).
Core principle: understand it fully, explain all of it
The video exists so a viewer who has not read the PR / article understands every algorithm and every meaningful code change in it. That decides everything else:
- No target length. Length follows the material: a one-line docs PR may need ~1 min; a multi-file kernel or scheduler PR, or a long article, may need 5–10+ min. Never cut a mechanism, code path, edge case, number or caveat to make the film shorter — add frames instead. Keep each frame to one idea (≲ 60 汉字, ~11 s) so visuals keep up; the number of frames is unbounded.
- Understand before you script. Read all of the source (Step 2) and write
capture/understanding.mdfirst; every item in it must map to at least one frame. - Code is the spine, not a footnote. Open with the value (what the change fixes / speeds up), then walk
through the algorithm and every changed code path in a sensible order, with a worked example where it helps.
This overrides pr-to-video's
story-design.mdwhere it says otherwise (its "Whole-film target ≤ ~3 min", "feature 2–4 real diff hunks", "implementation is the footnote"): feature every hunk that carries logic (group pure renames / formatting / test boilerplate into one frame), split long hunks across frames (≤ 12 lines each), and explain why each change is needed, not just what it is.
0. Setup (start of every session)
bash <this-skill>/scripts/setup_env.sh # first run ~2 min; afterwards seconds (idempotent)
source ~/.cache/sglang-explainer-video/env.sh
Re-run it at the start of each session: env.sh records this skill's install path ($SEV_SKILL,
$HF_MEDIA_ENGINE), so it must point at the copy you are using now.
It needs Node ≥ 22 (brew install node@22), ffmpeg/ffprobe, git, curl and Chrome. It creates a venv
(edge-tts, fonttools, and mlx-whisper on Apple Silicon for pron_check.py), a pinned HyperFrames checkout with patches/hyperframes-captions-cjk-bilingual.patch
applied, the Noto Sans/Serif SC fonts, and env.sh ($SKILL_DIR = pr-to-video, hf = pinned HyperFrames CLI wrapper,
$HF_MEDIA_ENGINE = our edge-tts engine, $SEV_SKILL, $SEV_PY). Nothing is installed globally:
always run the CLI as hf …, never npx hyperframes init without the env (init would install skills
into ~/.claude/skills).
Every command below assumes source ~/.cache/sglang-explainer-video/env.sh and P=<project dir>.
1. Project + brief
P=$SEV_WS/projects/sglang-pr-<N> # article: $SEV_WS/projects/sglang-article-<slug>
hf init "$P" --non-interactive --example=blank --skill=pr-to-video
Write $P/BRIEF.md (template in references/house-style.md §Brief). Run autonomous (flow: automation,
storyboard: no): do not stop for plan/sketch approval; the only questions worth asking are real choices
(voice change) — otherwise follow the house style. Never ask about length: it follows the material.
2. Ingest the source
PR mode — always the full URL (owner/repo#N fails outside a git checkout):
(cd $P && node $SKILL_DIR/scripts/fetch-pr.mjs --pr https://github.com/sgl-project/sglang/pull/<N> --out-dir ./capture \
&& node $SKILL_DIR/scripts/ingest.mjs --pr-json ./capture/pr.json --diff ./capture/diff.patch --out-dir ./capture/extracted \
&& node $SKILL_DIR/scripts/fetch-people-avatars.mjs --people ./capture/extracted/people.json)
Article mode — copy the article to $P/capture/extracted/visible-text.txt, write a minimal
capture/extracted/tokens.json ({"title": "...", "colors": [], "fonts": []}), and download the article's own
figures into $P/assets/article/fig<i>.<ext> (they are the author's images; they become real assets on frames).
Read everything before writing anything. capture/extracted/visible-text.txt is budget-truncated, so it is
not enough on its own:
- PR mode: the PR body, the full
capture/diff.patch, review comments that changed the design (gh pr view <URL> --comments), and every touched function in full at the PR head (gh api "repos/sgl-project/sglang/contents/<path>?ref=<headRefOid>" --jq .content | base64 -d) so you can explain the code around each hunk, not just the hunk. Follow calls into helpers when the change depends on them. - Article mode: the whole article, every code block and figure, and any PR / source file it links to that the argument depends on.
Then write $P/capture/understanding.md: the problem; the algorithm step by step; every changed code path
(file:function → what changed and why); the numbers with their conditions (hardware, model, workload);
the model the PR involves (the one the bug shows up on, or the one it was measured on: the title and the description
will name it, references/publishing.md); the caveats the author states. This is the coverage checklist for Step 3.
Then put the SGLang brand on the project (frame.md, CJK/bilingual caption skin, ending block, logos). The theme is
sglang-orange unless the video is about a Kimi / Moonshot model or the user asks for a different look; then use
--theme kimi-dark (references/house-style.md §Themes):
$SEV_PY -I $SEV_SKILL/scripts/apply_brand.py --project $P [--theme kimi-dark]
3. Story + script
Follow $SKILL_DIR/SKILL.md Step 3 and its references/story-design.md (PR: fix-explainer / feature-reveal /
refactor-walkthrough; article: a value-first explainer — hook → landscape → method → every evidence beat the
article makes → lessons → summary), with the Core principle above overriding its length and hunk limits.
Before moving on, tick every item of capture/understanding.md against the storyboard; add frames for
anything uncovered. Then apply the house style:
-
Frame 1 is always the cover
compositions/frames/00-cover.html(built in Step 6, not by a worker),status: animated. Frame N's file isNN-<slug>.htmlwithNN = N-1. -
PR videos end on the author page (name/@login/avatar on screen, never spoken; the PR link; no status).
-
Every video's last frame carries the SGLang ending block (the project's
assets/brand/follow-strip.html), at y 652–872, so plan that frame's content to end above y≈630 (house style §Ending links):- a links card with this video's own links: the source it explains, and the cookbook or docs page to open next;
- the fixed follow strip (关注 SGLang · GitHub · 博客 · 文档).
- Pick the two links now and write them into that frame's Scene text. Take them from
mainand fromdocs.sglang.io/sitemap.xml, never from memory. The block is on screen only, not narrated.
-
Chinese narration, English technical terms, one idea per frame (≲ 60 汉字, ~11 s); as many frames as the material needs — no whole-film cap. Long films get section kickers (e.g. 「① 问题」「② 算法」「③ 代码」) so viewers keep their place.
-
SCRIPT.mdvoice line:zh-CN-XiaoxiaoNeural (edge-tts). Numbers only from the PR/article. -
Every English word must be read right and in one breath (
references/pronunciation.md): no_reaches the voice, CamelCase brand names are spaced, symbols (×, #, +%) are spoken as Chinese. Write the narration normally —pronunciation.jsonrewrites only what the voice hears, captions keep the written form — then run the check and fix every FAIL (lexicon entry or reword) before TTS:$SEV_PY -I $SEV_SKILL/scripts/pron_check.py --script $P/SCRIPT.md --asr -
Chinese polyphones must take the intended reading. 行 meaning "row / line" is respelled to 航 automatically (the voice read 一行 as yì xíng in 42 % of past lines). For any other polyphone the user flags, measure it with
scripts/poly_probe.py. Edge picks the reading per request, so it synthesizes several takes and compares each one acoustically with unambiguous homophones. Add apatternsrule when the spoken form still misreads. Seereferences/pronunciation.mdfor the details.
4. Narration (edge-tts, word timings)
(cd $P && node $SKILL_DIR/scripts/audio.mjs --script ./SCRIPT.md --storyboard ./STORYBOARD.md --hyperframes . \
--out ./audio_meta.json --voice zh-CN-XiaoxiaoNeural \
&& node $SKILL_DIR/scripts/audio.mjs sync-durations --audio-meta ./audio_meta.json --storyboard ./STORYBOARD.md \
&& cp audio_meta.json audio_meta.built.json)
HF_MEDIA_ENGINE swaps HyperFrames' TTS for scripts/edge_tts_engine.py: per-word timestamps from WordBoundary,
punctuation re-attached, split terms re-joined (FP32, FLUX.2, torch.equal), 2 s of silence after the last line;
the voice reads the spoken forms from pronounce.py and the timings map back to the written tokens.
Then check the recorded take — pauses inside a clause and every English term in context:
$SEV_PY -I $SEV_SKILL/scripts/pron_check.py --meta $P/audio_meta.json --hyperframes $P --asr
Then check where the voice cut the Chinese: seg_check.py compares Edge's word tokens with a Chinese
segmentation and flags a token that straddles a word (值相同时下标 → 「时下|标」, 下标取反 → 「标取|反放」), a pause
inside a word (1 除以 T → 「除 | 以」), or a number fused with 只 (乘 24 只要 → 「24只|要」, read as a measure word). Reword each hit (a comma, 的, or a different phrasing), re-synthesize the line,
and run it again until it prints ✓:
$SEV_PY -I $SEV_SKILL/scripts/seg_check.py --meta $P/audio_meta.json
Then listen to the Chinese: line_asr.py transcribes every line and diffs it with the narration in pinyin. It catches
a digit or a clause-final 时 that merges into another number (「L2 时」 heard as 二十, 「大时」 as 大十, 「batch 是 1」 as
11, 「batch 为 1」 as 唯一). Reword those (的话 / 的时候 / 只有 1), re-synthesize the line, and read the remaining hits;
homophone noise is expected, and a hit that repeats in a fresh take is not proof (pronunciation.md, §Listening to whole lines):
$SEV_PY -I $SEV_SKILL/scripts/line_asr.py --script $P/SCRIPT.md --meta $P/audio_meta.json
Then listen to every 行 in the take: the respelling lowers the misread rate, but Edge still picks the reading per
request, and Whisper writes 行 for either reading. hang_take.py compares each 行 in the take with fresh takes
that say 航 or 型 there and flags the ones that took the wrong reading (128 字节行 and 行距 were read xíng in the
PR 48380 video as delivered). Re-synthesize every WRONG line and run it again until it prints ✓:
$SEV_PY -I $SEV_SKILL/scripts/hang_take.py --script $P/SCRIPT.md --meta $P/audio_meta.json
Then listen for the polyphones that differ only in tone (切 qiē / qiè). hang_take and poly_probe compare spectra,
which carry almost no pitch, so they cannot hear these. tone_take.py force-aligns the take and same-line
controls (切 swapped for 接/些 vs 借/谢) with the Montreal Forced Aligner and compares the pitch slope of the
vowel with Praat. It needs the optional tone env once (bash $SEV_SKILL/scripts/setup_tone.sh, about 1.5 GB).
Edge reads a given text the same way every time, so reword every WRONG line (its reword: hint says how),
re-synthesize it, and run the check again until it prints ✓:
$SEV_PY -I $SEV_SKILL/scripts/tone_take.py --script $P/SCRIPT.md --meta $P/audio_meta.json
A FAIL there means re-synthesize that line after a lexicon fix or a reword (subset re-TTS, see troubleshooting).
Keep the pronunciation lexicon growing. Every term this video exposed as misread goes back into the skill
before delivery. That includes pron_check FAILs, OK~ terms that sound wrong, and anything the user flags
in review. Each fix lands in scripts/pronunciation.json (patterns / parts / lexicon / approved), with its
evidence row in references/pronunciation.md and a test case when the rule is new. Ship these as a PR to
the skill repo. The skill only stays reliable if each video leaves it a little better; see
references/pronunciation.md §The lexicon is a living file.
A frame running long means it holds two ideas — split it; never drop content to shorten the film.
Print the timings for Step 5:
python3 -c "import json;[print(v['frame'],v['duration_s'],' | '.join(f\"{w['text']}@{w['start']}\" for w in v['words'])) for v in json.load(open('$P/audio_meta.json'))['voices']]"
Keep audio_meta.built.json. edge-tts takes are not bit-identical; frames are cued to this take.
5. Visual design (Step 4 of pr-to-video)
Write ## Video direction + per-frame time-coded Scenes per $SKILL_DIR/references/visual-design.md, using the
real word start times above. Put the palette/type/font rules from references/house-style.md §Visuals into
Video direction. Code beats need a ### Source excerpt (≤ 12 real lines); article frames that show a real figure
carry - asset_candidates: assets/article/figN.png — ….
6. Cover
Every video gets its own cover design (house style §Cover). Do not reuse the previous video's.
- Look at what has been used:
references/cover-log.md, and the newest covers side by side.$SEV_PY $SEV_SKILL/scripts/cover_kit.py wall --out $P/cover_wall.png - Design this video's cover from its content: what the bug or the speed-up looks like. It differs from each of the last five covers in at least three of ground colour, accent colour, type, composition and illustration technique. It keeps the hook as the largest line and the small tag 「SGLang PR 讲解 · #N」 top-left.
- Write the design as
$P/work/make_cover.py:css(width, dy),body(prefix), the reveal steps and the list of text to fade.cover_kit.py's docstring shows the shape. Keep the content inside y 135–880.
Time the reveals to the cover narration's word cues in$SEV_PY $P/work/make_cover.py png # cover.png (1920×1080) + cover_bili.png (1440×1080) $SEV_PY $P/work/make_cover.py frame <frame-1 voice s> # compositions/frames/00-cover.htmlaudio_meta.json. - Compare, then log it:
If the new tile could be taken for a neighbour, redesign it. Then add its row to$SEV_PY $SEV_SKILL/scripts/cover_kit.py wall --out $P/cover_wall.png --add $P/cover_bili.pngreferences/cover-log.mdand ship that with the video's other skill updates.
cover_bili.png is the cover to upload to Bilibili, which crops one image to both 4:3 and 16:9.
The series template "A" (scripts/make_cover.py --project $P --series "SGLang PR 讲解" --badge "#<N>" --title-zh … --title-en … --meta … --duration <frame-1 voice s> --png $P/cover.png --png-bili $P/cover_bili.png) is only for a
video whose user asks for the series look.
7. Captions (bilingual)
(cd $P && node $SKILL_DIR/scripts/captions.mjs build --storyboard ./STORYBOARD.md --audio-meta ./audio_meta.json \
--hyperframes . --out ./caption_groups.json)
Translate every group's text into $P/caption-translations.json ({"<中文 group text>": "<English>"}),
then rebuild; the build warns no translation for N group(s) until all are covered.
8. Frames (≤ 5 parallel workers)
(cd $P && for b in <code-* blocks named in STORYBOARD>; do hf add "$b"; done)
node $SKILL_DIR/scripts/frame-packets.mjs --project "$P" --storyboard "$P/STORYBOARD.md"
Dispatch at most five background workers (Agent tool) at a time (five ran fine on an M4 Mac, 2026-10-23), each with the role file path, its packet paths and
the dispatch context from references/worker-dispatch.md verbatim (it carries the hard rules that broke
earlier runs). A packet over 48 KB means the blueprint + cited rules are too big: drop one cited rule or use
blueprint: compose. Mark frames animated as workers return. A long film (many frames) goes in waves: give each worker
≤ 5 packets per dispatch, keep frames that share a stage on the same worker, and start the next wave as workers
return. Then:
node $SEV_SKILL/scripts/check_frames.mjs $P # frame contract + no system CJK fonts
$SEV_PY -I $SEV_SKILL/scripts/subset_fonts.py --project $P # re-run after ANY text change
(cd $P && node $SKILL_DIR/scripts/captions.mjs build --storyboard ./STORYBOARD.md --audio-meta ./audio_meta.json --hyperframes . --out ./caption_groups.json \
&& node $SKILL_DIR/scripts/assemble-index.mjs --storyboard ./STORYBOARD.md --hyperframes . \
&& node $SKILL_DIR/scripts/transitions.mjs inject --storyboard ./STORYBOARD.md --hyperframes . \
&& node $SKILL_DIR/scripts/transitions.mjs verify --storyboard ./STORYBOARD.md --index ./index.html \
&& hf lint && hf check)
A failed frame goes back to its worker once with the exact finding (SendMessage). lint/check errors you
fix yourself with the smallest edit. id_requires_css_escape warnings and transition-time content_overlap
infos are expected.
Contrast warnings that all carry one timestamp inside a crossfade window come from the outgoing frame fading out
(PR 50374: 39 warnings at one instant, on text drawn in solid ink). Check the named elements' own colours and fix only
a warning at a time when one frame is alone on screen.
9. Render, QA, deliver
(cd $P && hf render --skill=pr-to-video --quality high --output renders/video.mp4)
bash $SEV_SKILL/scripts/qa_sheet.sh $P/renders/video.mp4 $P/qa 2.5
Read every sheet of the rendered MP4 (not Studio snapshots): 乱码, wrong characters, untranslated or mistranslated terms, captions out of sync, content under the caption band, stray status lines. Fix, re-render, re-check. Before delivering, make sure this video's pronunciation fixes are in the skill (§4 "Keep the pronunciation lexicon growing"). The same applies after the user's review: a word they say was misread becomes a lexicon entry plus a skill PR, not a one-off edit in the project.
Open the links the ending shows (and, once they are drafted, the publish texts):
$SEV_PY -I $SEV_SKILL/scripts/check_links.py --project $P [--text desc.txt body.txt]
Then copy to the Desktop:
cp $P/renders/video.mp4 ~/Desktop/SGLang_PR<N>_讲解_中英字幕.mp4
cp $P/cover.png ~/Desktop/SGLang_PR<N>_讲解_封面.png
cp $P/cover_bili.png ~/Desktop/SGLang_PR<N>_讲解_封面_B站.png
10. Publish (only when the user asks)
Follow references/publishing.md: the user picks each file in the native dialog, the agent fills the rest.
The title and the description are written for this video, from its own hook; they name the model the PR involves
(DeepSeek, GLM, Qwen, …), which draws more views; and they end with the same source and cookbook links as the last
frame (§Title and description).
Bilibili: 含AI生成内容 · 分区 人工智能 · ~8 tags · 1440×1080 B站 cover. 小红书: always 标记 SGLang via
添加组件 → 标记地点或标记朋友 → 用户 → SGLang (official, orange SGL avatar) — never an @ in the body — plus topics
#人工智能 #大模型 #SGLang and 笔记含AI合成内容.
Every publish runs the five checks in references/publishing.md §Checks:
publish_check.pyon the files and texts before anything is uploaded.- The B站 form before 立即投稿.
- The stored B站 archive (简介, tags, 分区, duration, cover, 合集) after submit.
- The 小红书 form before 发布 (topics, 标记, AI 声明, the cover actually used).
- The 小红书 note list after 发布.
The publish is done only when all five pass. Tell the user what each one showed.
Fixing a published video (a re-voiced line, a re-render) replaces the file in place on both sites; the post, its plays and its comments stay. Follow §Replacing the video of a published post: 小红书 drops the 标记 on every swap.
Changing narration after frames exist
Edit SCRIPT.md (and the storyboard voiceover), re-run Step 4, then — instead of rebuilding frames —
$SEV_PY -I $SEV_SKILL/scripts/retime_frames.py --project $P --ref-meta $P/audio_meta.built.json
re-maps each frame's timeline onto the new take (piecewise-linear, anchors = matched clause starts). Frames are
found through STORYBOARD.md - src: (so 19b-*.html works). --chain handles a project whose built take was
lost by an earlier retime: the new anchors are composed through each frame's existing WARP. Re-make the cover
frame (work/make_cover.py frame <new frame-1 voice s>, with its cues moved to the new take), or let retime warp
00-cover too.
Then retranslate changed caption groups, re-subset fonts, re-assemble, and re-render.
Re-voicing a finished video (new voice, same frames) is one command per project:
bash $SEV_SKILL/scripts/revoice.sh $P # VOICE=zh-CN-XiaoxiaoNeural by default
It runs these steps:
- Back up the old state once to
revoice_backup/. - Synthesize the new take, through the pronunciation layer.
- Restore the frames, then retime them.
- Run the pause check, captions, font subset, assemble, lint and check.
- Render
renders/video_revoiced.mp4. The oldrenders/video.mp4is never touched.
revoice.sh $P tts stops after the take. Use it to run pron_check --meta --asr across many projects, fix the
lexicon, and only then render. A batch of 13 published videos (132 min) took about 30 min of TTS plus about
2× real time to render.
References
references/house-style.md— narration, term glossary, captions, cover, author page, ending links, brief template.references/cover-log.md— what each cover so far looked like; read it before designing one, add a row after.references/pronunciation.md— English terms and Chinese polyphones read right: spoken-form rules, lexicon,pron_check.py,poly_probe.py,hang_take.py,tone_take.py.references/worker-dispatch.md— the dispatch context to paste into every frame worker.references/troubleshooting.md— every failure seen so far and its fix.references/publishing.md— publishing to Bilibili and 小红书 (always 标记 SGLang on 小红书, never @), and the checks every publish must pass (publish_check.pyplus what each site stored), and replacing the video of a published post.
SGLang Serving Debug
Use the maintained source contracts when applying historical evidence to a current branch. The review/history corpora retain their own capture dates; they do not certify today's dispatch or numerical defaults. For kernel replacements, verify that the real model executes the candidate before treating end-to-end tests as coverage.
Overview
Use this skill to turn a live serving problem into a debug path you can replay.
Use one loop:
- collect a baseline bundle
- save the failing request or crash dump
- replay on a clean target
- only then switch tools
Do not start with profiling.
Version-sensitive live-server signals:
--enable-session-radix-cacheplus/close_session: agent or RL rollouts that forget to close sessions can look like unexplained KV growth- current Rust ingress enforces API/admin keys and supports profiling; older Rust builds differ. Confirm the ingress binary and source revision before interpreting a 401 or 404 (see the endpoint reference).
This skill should work with more focused skills instead of re-implementing them:
debug-cuda-crashwhen replay plus coredump points to a CUDA crash pathdebug-distributed-hangwhen the problem is clearly a TP/PP/DP/EP hangllm-torch-profiler-analysiswhen the issue is already narrowed to a compute-side path
Three examples are included:
- TTFT spike with low queue time
- replay-first CUDA crash flow
- request-shaped distributed hang flow
Output Contract
Return:
- problem class
- what was checked
- strongest signal so far
- current best guess
- what was ruled out
- next step
- production risk
When To Use It
/healthor/health_generateis unhealthy- latency or throughput regressed under serving load
- queue size grows while health still looks green
- one request class times out or hangs
- the server crashes only after some requests
- outputs changed after a deploy, topology change, or weight switch
- one older commit is known-good and a newer commit is known-bad
Workflow
1. Collect a baseline bundle
Confirm the ingress and build first. Current Rust route assembly includes
load, metrics (when enabled), HiCache and profile controls but no /ready
route (it is only exempt from auth), so /ready returns 404 there; Python
route parity is not universal. A 404 can mean an older build, an unmounted
optional route or an ingress difference. Record status and response body,
then consult the actual route module. Health/readiness are different checks.
If a live server is reachable, collect a read-only bundle before anything more intrusive:
python3 scripts/incident_artifact_tool.py collect-bundle \
--base-url http://127.0.0.1:30000 \
--outdir /tmp/incident_bundle
python3 scripts/incident_artifact_tool.py summarize-bundle \
/tmp/incident_bundle
If the server is protected:
python3 scripts/incident_artifact_tool.py collect-bundle \
--base-url http://127.0.0.1:30000 \
--token "$SGLANG_BEARER_TOKEN" \
--outdir /tmp/incident_bundle
The bundle script collects:
/ready/health/health_generate/model_info/server_info/v1/loads?include=all/v1/loads?include=core,queues,disagg,spec/metrics/hicache/storage-backendon a best-effort basis
If stale or corrupt HiCache contents are the working diagnosis, capture the
bundle and relevant request dumps before calling
POST /hicache/storage-backend/clear. That endpoint clears the attached
backend's contents without detaching it; treat it as destructive and use it
only when losing the cached contents is acceptable. Attach, detach, and clear
details are in
references/endpoints-and-signals.md.
Use the summary for a quick read on:
- health vs. active health state
- topology and runtime flags
- point-in-time queue and token usage
- TTFT / E2E / queue-time heuristics from Prometheus metrics
If the summary says the bundle was captured while the server was idle, recollect it during traffic or move quickly to dump plus replay.
If no live server is reachable, start from the best dump or log already available:
- crash dump
- request dump
- logs
- CUDA coredump
- OTel trace
- torch profile
2. Save the failing request
Read references/decision-tree.md only if the problem class is still unclear:
- server down or unhealthy
- latency or throughput regression
- wrong output or behavior regression
- intermittent timeout or hang
Then preserve the request payload that actually triggers the problem:
- crash path: use
--crash-dump-folder - non-crash path: enable request dump or save the exact trigger request
Do not jump straight from a live symptom to low-level debugging without first saving something you can replay.
3. Replay on a clean target
Read references/endpoints-and-signals.md when you need help reading the baseline bundle or the replay target.
Read references/replay-trace-profile.md when you need the replay, trace, profile, or bisect paths.
Standard order:
- collect baseline bundle
- capture request dump or crash dump
- restart a clean debug target if needed
- replay the same issue
- collect replay-time logs and dumps
4. Only go deeper after replay
Replay
Use replay when:
- a crash dump exists
- a request dump exists
- the problem depends on request shape or workload mix
If a crash dump exists, summarize it first:
python3 scripts/incident_artifact_tool.py summarize-dump \
--input-file /path/to/crash_dump.pkl
Then replay:
python3 /path/to/sglang/scripts/playground/replay_request_dump.py \
--input-file /path/to/crash_dump.pkl \
--host 127.0.0.1 \
--port 30000 \
--parallel 128
If safe_pickle_load blocks a locally captured trusted dump, use:
python3 scripts/replay_trusted_request_dump.py \
--input-file /path/to/request_dump.pkl \
--host 127.0.0.1 \
--port 30000 \
--parallel 1
If replay indicates a CUDA crash path, restart the same build with coredumps enabled before reproducing again:
SGLANG_CUDA_COREDUMP=1 \
SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
python -m sglang.launch_server \
--model-path ... \
--crash-dump-folder /tmp/sglang_crash_dump \
...
Then inspect the generated coredump:
cuda-gdb "$(which python3)" \
-ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
For a replay-first crash example, read references/case-studies.md.
OTel trace
Use tracing when:
- request-stage timing is unclear
- router vs. worker attribution is unclear
- PD prefill/decode transfer may be implicated
If tracing was enabled at startup, you can change the level without restart:
curl "http://127.0.0.1:30000/set_trace_level?level=1"
curl "http://127.0.0.1:30000/set_trace_level?level=2"
Torch profile
Use profiling when:
- the issue is already narrowed to compute-side ownership
- replay already reproduces the problem
- metrics and loads do not explain the regression
At that point, switch to llm-torch-profiler-analysis. Do not duplicate
its profiling workflow here.
For a low-noise latency example, read references/case-studies.md.
Distributed hang
If this looks like a collective stall, save the failing request, replay it on a
clean target, collect the replay-time bundle and stacks, then switch to
debug-distributed-hang.
For an example of that flow, read references/case-studies.md.
Regression between two commits
If one commit is known-good and another is known-bad, build a deterministic harness before doing deeper manual debugging:
- choose a stable reproducer: request replay, benchmark command, or correctness check
- make the harness return
0on good behavior and non-zero on bad behavior - run
git bisect start <bad> <good> - run
git bisect run <harness> - return here only after a candidate commit is isolated
Prefer replay-backed bisect when the regression depends on request shape or long-running serving state.
5. Switch tools when the boundary is clear
Switch tools once the fault class is clear:
llm-torch-profiler-analysisfor kernel and overlap attributiondebug-distributed-hangfor collective or rank-divergence hangsdebug-cuda-crashfor CUDA crash reproduction and kernel API logging
Do not switch tools before collecting the first bundle unless the user already has decisive logs or dumps.
References
Load only what the current step needs:
- references/decision-tree.md
- problem classes, tool switch points, return shape
- references/endpoints-and-signals.md
- endpoint behavior, auth notes, field reading
- references/replay-trace-profile.md
- request dump, crash dump, replay, trace, profiler step, bisect
- references/case-studies.md
- compact examples for replay-first CUDA crash, latency, and distributed-hang triage
Scripts
- scripts/incident_artifact_tool.py
- collect a read-only live bundle
- summarize a collected bundle into a compact debug note
- summarize a trusted request dump or crash dump before replay
- scripts/replay_trusted_request_dump.py
- replay a trusted request dump when
safe_pickle_loadblocks stock replay
- replay a trusted request dump when
If a live bundle was collected, include its path.
If replay, trace, or profiling was chosen, say why bundle plus dump were not enough.
Current upstream debug controls — 2026-10-05
SGLang-repo companion skills now live under .agents/skills/ (#40515):
debug-cuda-crash, debug-distributed-hang, generate-profile, and
sglang-bisect-ci-regression. Verify the repository copy before invoking it.
--crash-dump-folder X configures CUDA coredumps when their environment keys
are unset, with CUDA_COREDUMP_FILE=X/%h/core.cuda.%t.%p and
CUDA_COREDUMP_PIPE=/tmp/corepipe.cuda.%h.%p. Before a crash, py-spy and
user-triggered CUDA dumps are enabled by default, with a 60 s coredump wait.
SGLANG_CUDA_COREDUMP=1 uses its separate cuda_coredump_%h.%p.%t naming;
by import-order code reading its already-set path wins. This precedence was
not GPU-tested. A configured pipe can also capture a wedged kernel.
Source: crash dump hook, Rust ingress.
Torch Profiler Layer Track
Create a normal Chrome JSON trace with L0, L1, ... guides beside at most ten
synthetic GPU activity lanes per device. Keep the source file. Compaction
changes event placement, not execution: original kernel names, timestamps,
durations and args.stream remain intact. CPU events stay unchanged. Label
the output as a compact view, not a runtime stream optimization.
First read the kernel attribution and timing principles. Fused mHC/AR can remove old anchors; graph-launch host gaps and synthetic lane counts do not establish GPU idle time or runtime stream counts.
For current DS V4.1, fused post/combine/norm and optional Mega mHC prefill can remove separate norm/statistics anchors. Engram and row selection can split that pattern again. Verify the measured phase's cadence against the current boundary dispatch before adding layer labels; never reuse a V4 or old V4.1 anchor count blindly.
Establish what the labels mean
Use the user's existing local trace, including a Desktop copy when requested. Obtain the layer count from matching model config/source. Identify the rank/GPU, execution phase, and one complete forward pass before counting. In speculative decoding, separate target verify from draft passes. BS=1 does not imply one token per target verify. A repeated block in the overview can be a whole iteration, not a transformer layer.
Choose a GPU kernel with a verified cadence per layer in the selected
phase. For two sublayer anchors use --anchor-stride 2; LongCat residual
boundaries can need stride 4. Use --extra-anchors-per-pass 1 for a verified
K3 output aggregation after the final layer. Verify its call site and cross-check the count with another landmark
or CPU module scopes. Record this evidence and code revision. Names, stream
numbers and layer counts depend on the model/build; do not count all Top-K
calls or blindly reuse a previous model's anchor. Inspect all GPU streams in
the rank: layers can change streams. CPU scopes mark host launch intervals,
not GPU execution boundaries.
The helper does not guess L0 from counts alone. --anchor-offset must identify
a verified first layer in timestamp-sorted matching events. Divisibility does
not rule out a partial first pass or mixed draft/target anchors. If evidence
is insufficient, explain the ambiguity instead of publishing confident labels.
Add the auxiliary track
Scripts require Python 3.10+ and the standard library only. Commands are relative to this skill directory.
python3 scripts/add_layer_track.py \
--trace /path/to/TP-0.trace.json.gz \
--output /path/to/TP-0.layers.trace.json \
--anchor-regex '^YOUR_ONCE_PER_LAYER_KERNEL$' \
--num-layers 40 --anchor-offset 0 --passes 1 \
--max-gpu-lanes 10 \
--phase target-verify \
--evidence 'Config/source revision ...; first L0 verified; 40 anchors per target pass; draft uses a different path.'
- Use
--pid GPU_PIDwhen anchors occur in multiple processes/GPUs. Use--device DEVICEif a single PID contains anchors from multiple devices. --anchor-stride Kkeeps every K-th anchor after the verified offset.--extra-anchors-per-pass Eskips E trailing output aggregations before selecting the next pass; stride/extra are recorded in the report.--passeslabels only that many complete passes.--first-layersupports a known pipeline partition. A truncated selection is rejected.- Optional
--end-anchor-regexidentifies a verified layer-ending kernel occurring once between anchors and closes the final label of each pass. Without it, the last label is a zero-duration marker; L39 is never extended across a draft pass or an inter-iteration gap. .jsonand.json.gzinput/output work. Output must be a new file. A mapping report is saved as<output>.layers.json.- Use
--max-gpu-lanes 10by default for a compact view; omit it when the user wants the original stream layout. The helper assigns non-overlapping activity to reusable display lanes and remaps matching GPU flow endpoints. It refuses selections needing more than ten lanes, rather than hiding concurrent work. Other unsupported or ambiguous events also fail with an explanation. - Original GPU
pid/tidand source event index are retained inargs._compact_gpu_track; retired stream-name metadata is in the report. GPU user scopes are preserved in a separateGPU scopesprocess so they do not recreate hundreds of rows beside the kernel lanes. CPU scopes and flows stay in their original process.
For a trace that already has verified guides beside the GPU, compact it without adding duplicate labels:
python3 scripts/compact_gpu_tracks.py \
--trace /path/to/TP-0.layers.trace.json \
--output /path/to/TP-0.compact-layers.trace.json \
--max-gpu-lanes 10
This saves <output>.compact.json. Existing guides are preserved. If a guide
is missing, under the wrong process, or has unverified boundaries, regenerate
it from the original trace with add_layer_track.py first. Never treat the
number of synthetic lanes as the number of real CUDA streams. The original
trace remains necessary for stream scheduling and synchronization analysis.
Every interval is a guide, from this layer's anchor to the next layer's anchor. Work before the anchor is outside its guide. The last interval, when supplied, ends at a terminal kernel's completion. These are not full layer boundaries, exclusive kernel ownership, or per-layer latency measurements. Overlapping streams can cross labels. Numbering is zero-based: L2 is the third transformer layer. Only auxiliary endpoints are rounded to the nearest microsecond to avoid floating-point overlaps; original timestamps and report anchor values stay intact.
Open and inspect
Open the annotated file with Perfetto's Open trace file. Expand the selected
GPU/process, locate Layer guide (...) — anchor intervals, and pin it beside
the GPU lane N (synthetic; ...) tracks. Use relative times in the report to locate L0, L1, a middle
layer and the final layer.
If the browser cannot fetch a local trace URL, use the loopback viewer. It transfers local bytes after Perfetto's PING/PONG handshake:
python3 scripts/serve_local_trace.py --trace /path/to/TP-0.layers.trace.json
# Open the printed http://127.0.0.1:PORT/ in the user's chosen browser.
# macOS example:
open -a Firefox http://127.0.0.1:PORT/
The server exposes only this trace and its viewer until Ctrl-C. It needs network access for the Perfetto UI; it does not upload the trace to storage. Manual file opening works when the UI is locally available. See navigation.md for SQL checks, screenshot guidance, time units, and the DeepSeek-V4.1 example.
Verify and deliver
- Check the report's PID, layer count, anchor offset, phase and source hash against the intended trace. Check two passes when claiming the mapping repeats across iterations.
- Verify that reversing the recorded placement changes and restoring retired metadata reconstructs the source event array exactly. Without compaction, the source array remains an unchanged prefix. Check the original file hash. In Perfetto, check both kernel count and actual imported track count: every GPU activity lane and layer guide should have depth zero. A raw JSON tid count alone can miss additional overflow tracks created by the importer.
- Inspect the guide beside the GPU kernels. If requested, capture the real viewer with readable labels and relevant streams. Preserve CPU tracks/flows when present; labels cannot reconstruct CPU events absent from the capture.
- Return the compact trace, report, and screenshot if requested. State the original stream count, synthetic lane count, numbering convention and anchor-boundary limitation. Keep large traces and private capture paths out of public skill/PR artifacts.
Sponsors
Details
- Marketplace
- BBuf/AI-Infra-Auto-Driven-SKILLS
- Category
- Ai Infra