BBuf avatar

ai-infra-auto-driven-skills

LLM serving benchmarks, SGLang model Day-0 support, profiling, code review, prod incident triage, PR-history dossiers, and SGLang explainer videos.

version
0.10.0
GitHub stars
1.2K

Install

/plugin install ai-infra-auto-driven-skills@ai-infra-auto-driven-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add BBuf/AI-Infra-Auto-Driven-SKILLS.

Contents

Discovered from source

Skills 9

LLM Pipeline Analysis

Overview

Use this when a whole-trace profiler summary is too coarse. The scripts read a Chrome-trace JSON file, find layer-boundary anchor kernels, group kernels into forward passes and layers, and print timing tables you can use for Perfetto navigation or detailed timing analysis.

When To Use It

  • when you need to know which layers contribute most
  • when the model has alternating layer types (e.g. models with compress_ratios like DeepSeek-V4 NSA, or hybrid GDN/GQA stacks such as Qwen3.8-27B)
  • when you need to compare cold-start vs steady-state forward passes
  • when you need to navigate to a specific layer in Perfetto UI
  • when you need to select representative layers for deep-dive analysis

Verify inputs from artifacts

Resolve model config, phase, rank/device and TP/EP/DP from the run manifest, server arguments and trace. Ask only for missing information that changes the analysis. TP-0 identifies a rank, not TP world size; do not default an unknown run to TP8/EP8 or decode BS1. Speculative request BS and target-verify M differ. If no config is available, use an explicit profile and verified layer count.

Model Profiles

Scripts use ModelProfile to determine layer boundary detection and kernel classification. Profiles are auto-inferred from config.json or selected via --profile:

ProfileAnchor kernelBlocks/layerAuto-infer condition
dsv4_csa_hcamhc_pre_big_fuse2DeepSeek-V4 or compression fallback
dsv41explicit verified anchor1 (override for sublayers)V4.1 text/wrapper identity or KV-source layers
dsv3_mlaexplicit backend anchor or residual boundarybackend-dependenthomogeneous MLA after identity dispatch
mla_dsaverified residual boundary2GLM-5/5.2/5.3, V3.2, DSA MLA
kimi_k3explicit attn_res_fused (SGLang) / attn_res_fwd_online_v2_kernel (vLLM)2 + one output aggregation/passK3 / Kimi linear with attention residual
glm5_nextmhc_pre_big_fuse2GLM-5.3-Flash
hy_v4explicit ihc_pre_stage2 on Triton fallback2Hy4 (hpc symbol requires trace check)
qwen4_expexplicit grouped_gemma_rmsnorm2Qwen3.8-Flash-Next (vLLM may fuse norm)
nemotron_hexplicit residual norm1one Mamba/attention/MoE block per config layer
longcat_flashexplicit residual boundary4LongCat-Flash architecture
gemma4explicit _gemma_qkv_rmsnorm_kernel (SGLang CUDA)1Gemma4 text/wrapper identity
generic_addnormverified fused-add RMSNorm or allreduce fusion2other configured models
genericexplicit anchor; residual candidates only1, or 2 for residual candidatesconfig-less fallback

Nested text_config, language_config and llm_config are flattened before inference. Model identity takes priority over kv_lora_rank: K3, GLM-5 Next, Hy4, dots3 and LongCat must not be treated as homogeneous V3 MLA. Read layer count from config: V4 Flash has 43 layers, Pro 61, V4.1 40. These anchor families are source-verified for SGLang/vLLM, with backend and phase caveats; TensorRT-LLM/TokenSpeed cadence requires separate trace evidence.

Prerequisites

  • A torch.profiler trace in Chrome-trace JSON format (.json or .json.gz)
  • The model's config.json (for profile inference, compress_ratios, etc.)
  • The trace must contain a recognizable layer-boundary anchor kernel (auto-detected from the profile, or specified via --anchor-kernel)

Layer Boundary Detection

The scripts use an anchor kernel as a layer-boundary marker. The anchor and layer structure are determined by the active ModelProfile.

Find the model's residual kind at its stage boundary first. SGLang declares attention/FFN boundaries through python/sglang/srt/layers/layer_boundary/factories.py (add-norm, gated, iHC, mHC, post-norm and stream). Backend dispatch determines the visible boundary kernel.

V4 currently starts each half at mhc_pre_big_fuse*: attention-pre then FFN-pre, two anchors per layer. Fused post+pre removes mhc_post_tilelang for small-token SGLang traces; retain that old anchor only for verified unfused traces or the >32-token path. V4.1 uses previous-sublayer pre-mix, which cannot reuse V4's fused-post-pre formula. Its current deepseek_v4_mhc.py dispatch still carries fused post/combine/norm/quant across eligible boundaries; opt-in Mega mHC prefill also produces the next coefficients. Engram seams and row selection can invalidate cached inputs. It always needs explicit anchor and cadence evidence; a vanished stats/norm launch is not a missing layer. See the current boundary matrix. FlashMLA's flash_fwd_mla_combine depends on backend and split count; SM100 SGLang V3 defaults to TRT-LLM MLA, while DSA models use DSA.

All scripts accept --blocks-per-layer K --anchor-offset I --pid PID --device D. I is a verified first-layer anchor in the selected GPU, not inferred from counts. Mixed ranks/devices and a count residue modulo N*K are rejected. A divisible count is necessary but insufficient: separate target/draft phases, MTP/DSPARK passes and TBO microbatches first. K3's extra output aggregation requires a phase-filtered trace for these timing analyzers; the layer-track helper supports the extra anchor explicitly. The final selected pass ends at the selected GPU event boundary, which must itself be verified as a complete pass. Anchors are navigation intervals; cross-stream kernels may overlap them.

For compiled vLLM traces, norms can lower to Inductor kernels. Capture a short mapping trace with explicit norm custom ops or eager execution and verify an attention/backend anchor on the actual measured trace.

Source audit 2026-10-05: SGLang V4 dispatch, mHC kernels, vLLM mHC dispatch.

Scripts

1. layer_timeline_analyzer.py — Per-layer timeline and cluster stats

# Show all forward passes summary (cold-start vs steady-state)
python3 scripts/layer_timeline_analyzer.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --show-all-passes

# Detailed per-layer breakdown for a specific forward pass
python3 scripts/layer_timeline_analyzer.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5

# Auto-select the first relatively stable pass window
python3 scripts/layer_timeline_analyzer.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json

The script prints:

  • Per-layer wall-clock time, sum-duration, and category breakdown (MLA, MoE, GEMM, NCCL, MHC, Hadamard)
  • Layer cluster statistics grouped by type (CSA_C4, HCA_C128, HASH, etc.)
  • All-passes summary showing cold-start → steady-state growth

Automatic steady-state selection requires two consecutive layer-0 timing changes within 5%. It does not use an absolute latency threshold, so the same rule works across model sizes and accelerators. If no stable window exists, choose --fwd-pass explicitly.

2. layer_kernel_breakdown.py — Per-layer kernel detail and compute flow

# Single layer kernel dump
python3 scripts/layer_kernel_breakdown.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5 --layer 3

# Compute flow format (with model architecture summary and category column)
python3 scripts/layer_kernel_breakdown.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5 --layer 3 --format compute-flow

# JSON export
python3 scripts/layer_kernel_breakdown.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5 --layer 3 --format json

# Compare two layers side-by-side
python3 scripts/layer_kernel_breakdown.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5 --layer 2 --compare-layer 3

Output formats:

  • --format text (default): grouped summary + top hot kernels ranked by duration, with simplified names and percentages
  • --format compute-flow: model architecture summary + per-kernel hotness table with Category, %, and ts_rel(ms) columns
  • --format json: one machine-readable JSON document ranked by duration; with --compare-layer, it contains primary, comparison, and kernel_diff
  • Kernel diff when comparing two layers (unique kernels in each)

3. perfetto_time_mapper.py — Perfetto UI time navigation

# Show all forward pass time ranges in Perfetto
python3 scripts/perfetto_time_mapper.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json

# Layer-level time ranges for a specific forward pass
python3 scripts/perfetto_time_mapper.py \
  --trace /path/to/TP-0.trace.json.gz \
  --config /path/to/config.json \
  --fwd-pass 5 --layers 2,3,38,42

The script prints:

  • Forward pass time ranges in Perfetto-relative seconds
  • Per-layer start/end times with compress_ratio labels

Workflow

Step 1: Identify steady-state forward pass

python3 scripts/layer_timeline_analyzer.py \
  --trace $TRACE --config $CONFIG --show-all-passes

Read the "all-passes" table. The first pass is cold-start (few tokens). Use the first relative-stability window selected by the script, or choose a pass explicitly when timings continue to change.

Step 2: Per-layer breakdown on steady-state pass

python3 scripts/layer_timeline_analyzer.py \
  --trace $TRACE --config $CONFIG --fwd-pass 5

Identify:

  • Which layer type dominates (CSA_C4 vs HCA_C128 vs HASH)
  • The MLA / MoE / GEMM / NCCL proportion per layer type
  • Which layer type is the best next target

Step 3: Compute flow for representative layer(s)

Select 1-2 representative layers (one per bottleneck type), then:

# Human-readable compute flow table
python3 scripts/layer_kernel_breakdown.py \
  --trace $TRACE --config $CONFIG \
  --fwd-pass 5 --layer 3 --format compute-flow

# JSON export
python3 scripts/layer_kernel_breakdown.py \
  --trace $TRACE --config $CONFIG \
  --fwd-pass 5 --layer 3 --format json > /tmp/layer3_detail.json

The --format compute-flow output includes:

  • Model architecture summary at the top
  • Per-kernel hotness table with # | Half | Category | Simplified Name | dur(us) | % | ts_rel(ms) | Input Dims
  • Rows are ranked by dur(us) descending by default; use ts_rel(ms) to jump back to the kernel's trace location.

Step 4: Compare layer types (optional)

python3 scripts/layer_kernel_breakdown.py \
  --trace $TRACE --config $CONFIG \
  --fwd-pass 5 --layer 2 --compare-layer 3

This shows the exact kernel difference between the two layer types.

Step 5: Navigate in Perfetto UI (optional)

python3 scripts/perfetto_time_mapper.py \
  --trace $TRACE --config $CONFIG \
  --fwd-pass 5 --layers 2,3,38,42

Use the printed time ranges to navigate directly in Perfetto.

Layer Type Classification

The scripts classify layers based on config.json fields:

Config fieldValueLayer TypeDescription
compress_ratios[i]0SWA_ONLYwindow-only attention, no compressor/indexer
compress_ratios[i]4CSA_C4compressor + lightning indexer (Hadamard/paged-MQA/top-k) + sparse attention
compress_ratios[i]128HCA_C128compressor + all compressed rows and window; no indexer
compress_ratios[i]2 / 1V41_C2 / V41_C1source-layer compressor/indexer; other layers share KV/index
i < num_hash_layers—+HASHhash routing in the FIRST N layers, composable with FIRST/FINAL
i == 0 / N-1—+FIRST / +FINALpositional flags, never replace attention/routing work

V4 Flash/Pro hash-route L0–2. V4.1 has no hash layers; its KV source layers are [2,8,14,20], index sources [2,8,14,20,24,28,32,36], candidate source 20, and Engram layers [1,14]. Use these role fields when choosing representative layers. Pro starts with ratio-128 layers; do not assume its first two are SWA. K3's KDA layer ids are 1-indexed, whereas GLM-5 Next uses 0-indexed KDA ids.

Kernel Categories

Kernels are classified by the active ModelProfile's rules. Categories marked with (DSv4) are specific to the dsv4_csa_hca profile; all profiles include the universal categories.

CategoryMatch PatternProfileTypical Share (DSv4)
★ MLA Attentionflash_fwd_splitkv_mlaDSv4, DSv321-33%
★ MoE Fusedfused_moe_kernelDSv4, DSv311-17%
● NCCL AllReducecase-insensitive allreduce, all_reduce, cross_device_reduce, reduce_scatter, NCCL/Lamportuniversal5-8%
GEMM fp8deep_gemmuniversal12-25%
GEMM bf16nvjetuniversal11-13%
Hadamard XformhadamardDSv40-2.4%
Indexer CacheindexerDSv40-0.1%
Paged MQApaged_mqa_logitsDSv40-1.8%
MHCmhc_pre_gemm_sqrsum, mhc_pre_big_fuse, mhc_post_tilelangDSv410-15%
C4/C128 Prefillc4_prefill, c128_prefillDSv40-0.3%
RMSNormcase-insensitive rms_?norm, layernormuniversal1-2%
FP8 Quantquant, Quantuniversal1-2%
TopKtopkuniversal0-0.7%
RoPEdeepseek_rope, fused_norm_ropeDSv4, DSv31-2%
Activationsilu_mul_clamp, act_and_muluniversal0-0.5%
Other—universal2-5%

Reporting Checklist

Include:

  1. Trace metadata: trace path, model config path, GPU type, TP/EP
  2. Model Architecture Summary (from config.json):
    • model name, num_layers, hidden_size, num_attention_heads, num_key_value_heads, head_dim
    • Attention type (e.g. csa_hca), Q/O LoRA ranks
    • MoE config: num_experts, topk, num_shared_experts, intermediate_size
    • MHC config (if applicable)
    • NSA config (if applicable): index_n_heads, index_head_dim, index_topk, qk_rope_head_dim, sliding_window
    • compress_ratios distribution (how many CSA_C4 / HCA_C128 / SWA_ONLY / HASH layers)
  3. Per-batch forward passes summary table (from layer_timeline_analyzer.py --show-all-passes):
    • Columns: Fwd#, Start(s), End(s), Duration(ms), Avg Layer(ms), First Layer(ms), Notes
    • Identifies cold-start vs steady-state passes
  4. Chosen forward pass: index and rationale (cold-start vs steady-state)
  5. Per-layer wall-clock and sum-duration table (from layer_timeline_analyzer.py --fwd-pass N):
    • Columns: L, c_r, Type, Wall(ms), SumDur(ms), MLA, MoE, GEMM, NCCL, MHC, Hadam, AR#, K#
    • Each row is one layer, with layer type label
  6. Layer cluster statistics table grouped by type:
    • Columns: Cluster, #, Avg Wall(ms), Avg Sum(ms), MLA%, MoE%, GEMM%, NCCL%, MHC%, Hadam%
    • Identifies bottleneck layer type and likely next target
  7. Compute Flow Table for selected representative layer(s):
    • Produced by layer_kernel_breakdown.py --format compute-flow
    • Columns: # | Half | Category | Simplified Name | dur(us) | % | ts_rel(ms) | Input Dims
    • Rows are sorted by top hot kernels (dur(us) descending) by default
    • Optional JSON export (--format json)
  8. Perfetto UI time ranges when requested
  9. One-line summary: bottleneck layer type and likely next target

LLM Serving Auto Benchmark

Overview

Use this skill to compare LLM serving frameworks such as SGLang, vLLM, TensorRT-LLM, and TokenSpeed for the same model and workload.

Use a config-driven workflow:

  • keep launch-only capacity choices in each framework's base_server_flags
  • put the search knobs in search_space
  • run the same dataset scenarios for every framework
  • generate a bounded candidate list from search_space, with the baseline candidate included first
  • keep failed candidates in the result file
  • pick the best SLA-passing candidate after normalizing the results

For model-specific starting points, prefer the shipped configs in configs/cookbook-llm/. They define a framework-neutral LLM serving cookbook model set and translate each entry into framework-native SGLang, vLLM, and TensorRT-LLM, and TokenSpeed server flags. Validate those configs before a real run:

python skills/llm-serving-auto-benchmark/scripts/validate_cookbook_configs.py \
  skills/llm-serving-auto-benchmark/configs/cookbook-llm/*.yaml

If you have captured target-environment --help files, add --help-dir <artifact-help-dir>. That check only loads configs, verifies the server flag names, and renders candidate commands; it does not launch model servers.

Prefer native tooling when it gives better coverage:

  • SGLang: current public cookbooks use sglang serve; python -m sglang.launch_server remains accepted. Prefer python -m sglang.benchmark.serving
  • vLLM: vllm bench sweep serve for server-parameter sweeps, otherwise vllm serve plus vllm bench serve
  • TensorRT-LLM: trtllm-serve for the OpenAI-compatible server plus the TensorRT-LLM serving benchmark client or a common OpenAI-compatible benchmark client
  • TokenSpeed: tokenspeed serve for the OpenAI-compatible server plus vllm bench serve --backend openai-chat or the same OpenAI-compatible benchmark client used for the other frameworks

TensorRT-LLM has one hard scope rule in this skill: the current server is PyTorch-only; older images must select pytorch. Do not search TensorRT-LLM backend choice. If a request, config, or candidate asks for trt, an engine backend, or any other non-PyTorch TensorRT-LLM server backend, reject that candidate as unsupported for this skill and record the reason. This does not change the benchmark client backend; the TensorRT-LLM benchmark client still uses OpenAI-compatible modes such as --backend openai or --backend openai-chat.

Only pick a winner after each requested framework has had its main serving knobs tuned.

Framework selection is caller-controlled. If the caller explicitly supplies a framework list, benchmark only those enabled frameworks. Do not silently add TensorRT-LLM or TokenSpeed just because cookbook configs or history docs exist; record omitted frameworks as user-excluded, not unsupported.

The parameter lists in this skill are not a compatibility contract. They are version-sensitive candidate knob families. Before every real run, record the exact framework version or git commit and verify the concrete CLI flag names with --help in the target environment.

The default search style is framework-neutral: start from a mostly pure-TP baseline, sweep a small set of high-impact runtime knobs, and cap the first pass around 10 candidates per framework. Do not search memory fractions by default.

Source and experiment contracts

Use the source contracts for current profiler APIs, branch-specific defaults and KV sizing semantics. Before combining PRs or claiming kernel performance/accuracy equivalence, use paired-validation.md. It distinguishes code-only A/B from deployment tuning, real from simulated acceptance, and interactivity from aggregate output throughput. Profile after timing to prove that the candidate backend actually ran.

Validation Environment

This skill is target-agnostic. It assumes any one of the following is available, and nothing more:

  • a local GPU host with Docker/Podman and the target framework images pulled;
  • a remote GPU host reached via ssh <host> with the framework images already running in a container there;
  • a CI runner that can exec into a pre-built image for each framework.

Do not assume a specific operator host name inside this skill's own workflow. The concrete SSH wiring, container names, workspace paths, and HF token plumbing for a given box live in operator-side per-host skills; this skill only requires that the caller can reach a shell inside a container with the requested framework installed.

Reference files are optional and version-sensitive. Treat historical flag notes as evidence from one image, not as a compatibility guarantee for the next run.

Historical smoke validation on 2026-05-01 (H100) and 2026-06-27 (B200) exercised small serving/search workloads. The B200 image provided only SGLang; missing vLLM/TRT-LLM/TokenSpeed CLIs were environment gaps. Those runs are not current cross-framework speed or accuracy results. The September source refresh ran offline tooling, not a new GPU matrix.

Skill Scope

This skill is a playbook plus a config+validator toolchain, not a turn-key orchestrator. The operator still launches servers, drives workloads, and writes one normalized JSONL row per candidate.

The scripts/ directory contains exactly two tools:

  • validate_cookbook_configs.py: load cookbook YAML, render bounded candidate server commands, and check flag names against captured --help snapshots without launching servers.
  • compare_benchmark_results.py: turn normalized per-candidate JSONL into the markdown and optional CSV tables described in the Output Contract.

Cookbook configs under configs/cookbook-llm/ must pass the validator. The shorter references/example-plan.yaml is a one-off runtime-plan skeleton and is not expected to pass as-is. Use references/result-schema.md as the single source of truth for SLA key names.

Required Inputs

Collect these before a long run:

  • model and tokenizer path, target frameworks, GPU model/count, multi-node allowance, precision, and quantization constraints
  • endpoint shape, workload source, dataset scenarios, SLA target, search budget, and artifact output directory
  • version manifest: framework package version or git commit, container/Python environment, --help snapshots, and whether each search parameter was accepted by that exact CLI

If real production traffic is the goal, use the real request distribution. A synthetic workload is fine for bring-up and first-pass comparison, but it is not enough for a production choice.

Record each scenario's input/output length distribution in the normalized result rows. This is part of the profiler handoff contract: when a framework is slower and the follow-up investigation invokes llm-torch-profiler-analysis, the profiler workload must reuse the slow benchmark scenario lengths instead of falling back to its generic prefill 4090->1 and decode 1->2048 defaults.

Version-sensitive deployment checks

  • Consult the source contracts for exact inspected revisions. Preserve installed-image --help and effective environment values; do not copy a historical release's defaults forward.
  • Current SGLang main uses SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=False; the deleted historical dsv4.1 branch used True. Record that numerical difference when reproducing an old result. Main now pins FlashInfer 0.7.3; inspect the installed wheel and actual producer/consumer dispatch. The current kernel roots are python/sglang/kernels/{aot,jit,ops}.
  • SGLang's native /generate and OpenAI-compatible paths are different benchmark lanes. Use one common endpoint/client when comparing frameworks; separately label any native-client result.
  • Match attention/GEMM backend support to the GPU architecture and dtype. “Blackwell” includes different SM targets; B200/GB300/RTX 5090 are not one interchangeable backend contract. Verify support in the installed code.
  • vLLM DBO depends on a supported all-to-all backend. Compile fusion depends on platform, graph partition and dtype guards. Confirm dispatch using the fusion reference.
  • The current TensorRT-LLM lane uses trtllm-serve serve <model>; older images may need --backend pytorch. Server backend is distinct from OpenAI benchmark-client backend. Verify the model format, KV-memory flag spelling and random-dataset sampling options from the selected CLI; do not silently fall back to another dataset.
  • TokenSpeed uses tokenspeed serve <model> and a common OpenAI-compatible client. Its TP/attention-TP/MoE-TP/EP and memory flags need their own interpretation; similarly named SGLang/vLLM options are not an equivalence guarantee.
  • Every max-context candidate must cover max(input_len + output_len) across all scenarios, not just the baseline. The cookbook validator checks this.
  • Profile after timing. For a fixed simulated-acceptance curve, label it as such; restart in real acceptance mode for model accuracy tests. Never promote an unused replacement kernel because end-to-end scores happen to match.

Secrets Hygiene

  • Never print HF_TOKEN, HUGGINGFACE_HUB_TOKEN, or any upstream API key into a saved artifact. Pass them through container -e VAR (unquoted on the right side so the host value is inherited) and keep them out of server_command and benchmark_command fields written to the result JSONL.
  • When a framework echoes the full argv at startup, scrub the log or redact token-shaped substrings before uploading the artifact.

Fairness Rules

Use these rules throughout the benchmark:

  • Run every framework on the same GPU type, GPU count, model weights, tokenizer, precision, quantization policy, prompt distribution, output length target, and sampling settings.
  • Record framework version, git commit, container image, CUDA/NCCL versions, GPU driver, visible GPU ids, launch command, and benchmark command.
  • Warm the server before measuring. Restart or clear state between candidate configurations when cache effects would bias the comparison.
  • Compare steady-state fixed-QPS runs separately from burst throughput runs.
  • Keep failed candidates in the final results with their failure reason.
  • Report both raw throughput and SLA-passing throughput. The fastest failing candidate is not the best deployment command.

Workflow

1. Preflight

Verify SGLang plus all requested comparison frameworks before starting a search. Run only the commands for the requested framework set:

sglang serve --help
python -m sglang.launch_server --help
python -m sglang.benchmark.serving --help
vllm serve --help
vllm serve --help=all
vllm bench serve --help
vllm bench serve --help=all
vllm bench sweep serve --help=all
trtllm-serve serve --help
python -m tensorrt_llm.serve.scripts.benchmark_serving --help
tokenspeed serve --help
vllm bench serve --help

Use the framework-specific --help output in the target environment as the source of truth. Do not keep a stale launch flag just because it appears in an old note.

vLLM 0.19 and newer use grouped help. Plain vllm serve --help only shows the groups, so capture --help=all before deciding whether a search knob exists.

Save these --help outputs into the run artifact directory. If a listed search knob is missing from the current CLI, remove or translate that knob before running the benchmark. Do not silently pass unknown flags.

Record trtllm-serve serve --help. Current main is PyTorch-only; omit its deprecated backend option. On older images listing multiple backends, select --backend pytorch.

For TokenSpeed, confirm both the server and benchmark entrypoints because some installations alias the binary as ts. Record the exact binary used in server_command.

For each framework, launch a minimal server, confirm /v1/models or the native model-info endpoint, send one streaming request, run one tiny benchmark with at least 5 requests, then save the launch command, benchmark command, server log, and benchmark output.

Before any GPU-backed smoke run, check the requested GPU ids directly with nvidia-smi. If a requested GPU is already in use, stop and record that fact. Do not silently borrow a different GPU count for a performance comparison. It is fine to run a smaller one-GPU smoke only when the result is clearly labeled as a flow check rather than a fair throughput comparison.

If the target environment runs through containers, follow references/container-runbook.md and save image tags, pull commands, launch/benchmark logs, and cleanup commands.

2. Normalize The Workload

Use one canonical workload for all frameworks. Recommended JSONL row shape:

{"prompt": [{"role": "user", "content": "Summarize this text."}], "output_len": 256}
{"prompt": "Write a short explanation of CUDA graphs.", "output_len": 128}

Optional fields:

{
  "prompt": [{"role": "user", "content": "Use low temperature."}],
  "output_len": 256,
  "extra_request_body": {"temperature": 0.0, "top_p": 0.95},
  "metadata": {"source": "prod-sample"}
}

When converting user data:

  • inspect at least 3 rows before conversion
  • preserve request-level sampling options in extra_request_body
  • do not include the final assistant answer in the prompt when that answer is the target completion
  • keep multimodal or tool-call payloads only if all requested frameworks support the chosen endpoint shape

For synthetic bring-up, use the shipped two-scenario shape:

dataset:
  kind: random
  num_prompts: 80
  scenario_names: [chat, summarization]
  input_len: [1000, 8000]
  output_len: [1000, 1000]

Each aligned input_len / output_len pair is one scenario. Do not take the cartesian product unless the user asks for that. Name each scenario and keep the aligned pair in the artifacts. For custom datasets, compute or record representative input_len and output_len buckets, at least p50 and p95 when possible, so later profiler runs can match the slow bucket rather than profiling an unrelated synthetic shape.

Before searching any sequence-length limit, compute the largest input_len + output_len in the dataset. SGLang context_length, vLLM max_model_len, TensorRT-LLM max_seq_len, and TokenSpeed max_model_len must be at least that value for every candidate that is expected to run all scenarios.

3. Pick A Search Tier

Use the smallest tier that can answer the user's question:

  • Tier 1: smoke and sanity. One baseline plus a few high-impact knobs.
  • Tier 2: default. A bounded sweep over the most likely server settings.
  • Tier 3: exhaustive. Only when the search space is already tight and the user accepts a long run.

Default budget:

  • num_prompts: 80 for the default cross-framework comparison; num_prompts: 20 per scenario is acceptable for a smoke/flow check and must be labeled as such in the artifact (not as a performance result).
  • search.max_candidates_per_framework: 10 for the first useful pass
  • candidate generation: baseline first, then a bounded product or ordered candidate list from search_space
  • at most 5 QPS search rounds unless the user asks for more
  • stop early when every candidate in one framework is clearly OOM or fails the basic health check

Keep these in base_server_flags unless the user specifically wants a capacity or memory study:

  • SGLang mem_fraction_static
  • SGLang schedule_policy
  • vLLM gpu_memory_utilization
  • TensorRT-LLM kv_cache_free_gpu_memory_fraction
  • TokenSpeed gpu_memory_utilization

These are real knobs, but they widen the search quickly and often turn a serving comparison into a memory-limit study.

4. Tune SGLang

Launch each SGLang candidate manually. The auto-benchmark module was removed in #31941.

Benchmark with the current serving client:

python -m sglang.benchmark.serving \
  --backend sglang \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 80 \
  --request-rate 8 \
  --output-file /path/to/sglang/results.json \
  --output-details

Version-sensitive SGLang knob families to verify:

  • tp_size, pp_size, dp_size (replicas), attn_dp_size, ep_size
  • attention_backend, prefill_attention_backend, decode_attention_backend
  • sampling_backend
  • max_running_requests, max_queued_requests
  • chunked_prefill_size, prefill_max_requests, max_prefill_tokens
  • max_total_tokens, page_size
  • phase-specific CUDA graph settings
  • speculative or EAGLE settings only after the non-speculative baseline is tuned

Keep mem_fraction_static and schedule_policy pinned in the default pass, matching the shared cookbook config style.

For quick smoke tests, use --cuda-graph-backend-decode disabled and --cuda-graph-backend-prefill disabled if the goal is only to prove the framework flow. Record those flags in the artifact. Do not carry that smoke setting into a performance winner unless the user asked to tune eager-mode serving.

5. Tune vLLM

Use vLLM's sweep runner when available:

vllm bench sweep serve \
  --serve-cmd 'vllm serve <model> --port 8000' \
  --bench-cmd 'vllm bench serve --backend vllm --model <model> --port 8000 --dataset-name random --num-prompts 80' \
  --serve-params /path/to/vllm_serve_params.json \
  --bench-params /path/to/vllm_bench_params.json \
  --output-dir /path/to/vllm_results

If sweep support is unavailable, run vllm serve for each candidate and measure with vllm bench serve.

Version-sensitive vLLM knob families to verify:

  • tensor, pipeline, data, decode-context, and expert parallelism
  • gpu_memory_utilization
  • max_num_seqs
  • max_num_batched_tokens
  • max_model_len
  • enable_chunked_prefill, long_prefill_token_threshold, and DBO thresholds
  • KV cache dtype and block size
  • dtype and quantization settings
  • CUDA graph capture sizes or eager-mode toggles when relevant
  • prefix cache and speculative decoding settings only when the workload needs those features

vLLM should get a normal sweep, not one baseline command. See references/framework-reference.md for native command templates and cross-framework knob families. Confirm each flag on the target image's --help before a run.

Keep gpu_memory_utilization in the baseline for the default pass. Search it only when the question is explicitly about fitting the model or trading capacity against throughput.

Keep DBO and all2all backend settings out of the default pass unless the target vLLM environment is already set up for them. They are real tuning knobs, but a candidate can fail at startup if the required all2all backend is not available. Verify long_prefill_token_threshold on the concrete vllm serve --help=all surface before adding it to a search.

6. Tune TensorRT-LLM

Use trtllm-serve serve as the server entrypoint when the target environment supports it:

trtllm-serve serve <model> \
  --tp_size <tp> \
  --pp_size <pp> \
  --kv_cache_free_gpu_memory_fraction 0.75 \
  --host 0.0.0.0 \
  --port 8000

Then benchmark the OpenAI-compatible endpoint with the TensorRT-LLM serving benchmark client or with the same OpenAI-compatible client used for the other frameworks.

In the historical TensorRT-LLM 1.0.0 validation image, benchmark_serving --dataset-name random sampled from ShareGPT unless either --download-path or --random-ids was passed. For a fast synthetic smoke test, pass --random-ids, then confirm the behavior on the target TensorRT-LLM image.

TensorRT-LLM flag names are especially version-sensitive. Historically, in the validated TensorRT-LLM 1.0.0 image, the KV-cache memory flag accepted by trtllm-serve serve was --kv_cache_free_gpu_memory_fraction, not --free_gpu_memory_fraction. Current mainline was rechecked at bb367fc8c1adf6e2c28c88cb1a8b46e1742a9d60 on 2026-10-05. Always verify flags with trtllm-serve serve --help before running a search on any GPU target.

TensorRT-LLM backend policy for this skill:

  • launch the PyTorch-only current server without its deprecated backend option
  • use backend: pytorch only as an older-image compatibility setting
  • do not add backend to search_space
  • reject trt, engine-backed serving, or any other non-PyTorch TensorRT-LLM server backend as unsupported for this skill

Version-sensitive TensorRT-LLM knob families to verify:

  • tp_size, pp_size, and ep_size
  • max batch size, max sequence length, max number of tokens, and KV-cache budget
  • inflight batching and scheduler options
  • extra LLM API options YAML used by trtllm-serve with the PyTorch backend

The trtllm-serve serve CLI exposes fewer direct runtime knobs than SGLang or vLLM. Use direct flags when they exist, then use --extra_llm_api_options for PyTorch-backend settings that are not top-level CLI flags. Keep unsupported backend or engine requests in the failure table instead of translating them.

Keep kv_cache_free_gpu_memory_fraction in the baseline for the default pass. Search max_batch_size, max_num_tokens, max_seq_len, and validated PyTorch-backend config options first. The server backend remains fixed to pytorch.

7. Tune TokenSpeed

Use TokenSpeed as a first-class comparison framework, especially for agentic or multi-turn workloads where it may be the strongest non-SGLang baseline:

tokenspeed serve <model> \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size <tp> \
  --gpu-memory-utilization 0.90 \
  --max-model-len 12288 \
  --max-num-seqs 64 \
  --chunked-prefill-size 8192 \
  --kv-cache-dtype auto \
  --trust-remote-code

Benchmark either with the common OpenAI-compatible benchmark client:

vllm bench serve --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --model <model> \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 80

or with the same OpenAI-compatible client used for the other frameworks. When TokenSpeed is a likely leader and profiler handoff will be needed, the gateway can arm torch profiling for the same request shape when its installed SMG wheel exposes the control route. Set CONTROL_URL to that verified URL; current serve uses a headless engine plus SMG, not the retired Python --control-port sidecar. Do not infer the URL by adding one to the serve port:

curl -X POST "${CONTROL_URL}/start_profile" \
  -H 'Content-Type: application/json' \
  -d '{"output_dir":"/artifacts/tokenspeed_profile","num_steps":5,"activities":["CPU","GPU"],"with_stack":true,"profile_id":"ts-bench"}'
# Drive the same workload with the common client, then POST /stop_profile if needed.

If output_dir is not supplied, TokenSpeed writes under TOKENSPEED_PROFILER_DIR, defaulting to /tmp.

For large MoE or agentic checkpoints, validate these TokenSpeed knobs before adding them to the search:

  • tensor_parallel_size, attn_tp_size, moe_tp_size, and enable_expert_parallel
  • attention_backend, drafter_attention_backend, moe_backend, and draft_moe_backend
  • max_num_seqs, chunked_prefill_size, max_prefill_tokens, and max_total_tokens
  • kv_cache_dtype, quantization, and prefix-cache controls
  • speculative flags such as speculative_algorithm, speculative_num_steps, and speculative_num_draft_tokens
  • comm_fusion_max_num_tokens, enable_allreduce_fusion, and related communication-fusion flags only after a target-image smoke run

If a TokenSpeed-native agentic config exists for the same model family, run it as an additional workload lane rather than replacing the common cross-framework scenario. Normalize its result rows with the same schema and mark workload.kind accordingly.

Keep gpu_memory_utilization pinned in the default pass. Search it only when the user explicitly wants a capacity study.

8. Normalize Results

Write one JSONL row per candidate using the schema in references/result-schema.md. Then run:

python skills/llm-serving-auto-benchmark/scripts/compare_benchmark_results.py \
  --input /path/to/candidates.jsonl \
  --output /path/to/summary.md

Rank candidates in this order:

  1. SLA passed
  2. highest request throughput or goodput
  3. highest output token throughput
  4. lower p50 TTFT
  5. lower p50 TPOT/ITL
  6. lower GPU count or simpler deployment if performance is close

Keep the SLA gate itself unchanged. In the cookbook configs and normalized result schema, TTFT SLA still uses max_p50_ttft_ms and TPOT SLA still uses max_p50_tpot_ms; only the default cross-candidate comparison order switches to p50 TTFT and p50 TPOT.

Output Contract

Return a compact report with workload/SLA, hardware and framework versions, best deployment-command tables per framework/scenario, one cross-framework comparison table, exact launch and benchmark commands for winners, and artifact paths for workload, raw/normalized results, CSV or markdown summary, and server logs.

When SGLang is not the winner, include a profiler handoff note with the slow SGLang scenario name and the exact input/output lengths or percentile bucket to pass to llm-torch-profiler-analysis.

When a candidate uses speculative decoding, prefix cache, offload, or an agentic workload, record the optional normalized fields for accept length, pre-scheduler time, cache hit rate, and memory residency. The summary script will display those columns when present.

Include failed or excluded candidates with reasons. Explain that this table is a record of tried configs that were not selected: candidates that failed, were skipped by policy, or completed but missed the SLA. Add caveats for synthetic workloads, incomplete fair searches, or framework-specific parameter substitutions.

Use references/framework-reference.md when you need command templates, source links, or knob-family mappings. Use references/example-plan.yaml as the starting point for a full cross-framework run plan.

Interface audit — 2026-10-05

SGLang's canonical client is python -m sglang.benchmark.serving; bench_serving is a deprecated shim and the auto-benchmark module was removed (#31941). Capture sglang serve --help and the client help from the target image. Current images require CUDA 13 and a compatible host driver (#38404); the final CUDA 12 tag was v0.5.19-cu129. MiniMax-M3 requires lmsysorg/sglang:dev-minimax-m3; Qwen3.8-Flash-Next H200/B200 requires lmsysorg/sglang:qwen38flashnext.

Current SGLang uses phase-specific graph flags (#38375): --cuda-graph-backend-decode / --cuda-graph-backend-prefill accept full, breakable, tc_piecewise, disabled; capture sizes use --cuda-graph-bs-decode / --cuda-graph-bs-prefill and --cuda-graph-max-bs-decode / --cuda-graph-max-bs-prefill. Smoke eager mode is --cuda-graph-backend-decode disabled --cuda-graph-backend-prefill disabled. The old size aliases were removed and --disable-cuda-graph is deprecated. --attn-dp-size replaces the old attention-DP pair on main (#41818); --dp-size remains replica DP. v0.5.21 predates this spelling: inspect image help. Also inspect --attn-cp-size, --dcp-size, --moe-dp-size and the --ep alias.

TensorRT-LLM at bb367fc8 is PyTorch-only after #19028. --backend pytorch is a deprecated compatibility option; omit it on current main and select it only on older images whose help lists other backends. Both --free_gpu_memory_fraction (primary) and --kv_cache_free_gpu_memory_fraction (alias) work. The 1.0.0 image note is historical. --cluster_size is deprecated and unsupported. --set PATH=YAML_VALUE can override config paths after --config.

TokenSpeed #1236 removed its benchmark subcommand. Use the common client vllm bench serve --backend openai-chat for aligned workloads. ts remains an alias for tokenspeed. The prefix-cache off switch is --disable-prefix-caching; engine flags --api-key, --enable-cache-report, --skip-server-warmup, --warmups were removed. --tool-call-parser and --chat-template are gateway flags. Inspect --pipeline-parallel-size, --prefill-context-parallel-size, --decode-context-parallel-size, --lm-head-tp-size and --dense-gemm-backend when tuning parallelism.

Do not force vLLM --block-size 16: it excludes preferred MLA/DSA backends. Let the backend select its block size, and record its startup dispatch line. Use the same client version across compared rows: vLLM #55508 changed chat TTFT/E2E chunk accounting, and SGLang #39889 reports server prompt-token usage including template tokens. Record optional client queue latency separately. The default SGLang scheduler is FCFS; historical cookbook YAMLs no longer force LPM on random prompts. False for defaults-on booleans needs an explicit off flag and must not silently duplicate the baseline. The YAML validator requires Python 3.10 or newer.

Source evidence: SGLang graph flags, SGLang parallel flags, vLLM backend sizing, TensorRT-LLM serve, TokenSpeed CLI.

LLM Serving Capacity Planner

Overview

Use this when a serving log has enough memory lines to explain where GPU HBM went. The analyzer reads SGLang/vLLM startup logs, extracts weight load, KV pool, CUDA graph, framework overhead, and token-capacity lines, then estimates concurrent requests for common token lengths.

For DeepSeek-V4.1, read the KV layout and page-allocation contract before using 528 B/288 B in an estimate. These are per-stored-token payload and scale sizes, not allocation after page padding or total model KV bytes. Verify actual pool layout, compression ratios and sharing from the pinned source and logs.

Inputs

Before running analysis, collect or verify these inputs:

ItemWhy it mattersHow to obtainDefault if user skips
Log file pathPrimary input; all memory data comes from hereAsk user for the serving startup log— (required)
GPU typeDetermines total HBM for decomposition validationAsk user or infer from logAuto-detected from log if possible
nvidia-smi outputProvides per-rank actual memory for cross-validationCapture with nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt— (optional, but recommended)
Model config.jsonEnables theoretical KV cache byte calculation and replication factor analysisAsk user for the model's config.json path— (optional, log data used instead)
Request token lengthDetermines concurrency estimate denominatorAsk user4096, 6144, 8192

Workflow

Step 1: Collect the serving log

The user should provide the startup log from an SGLang or vLLM serving instance. Key log lines that the analyzer needs:

  • Load weight begin. avail mem=XX GB
  • server_args={'model_path': '...', 'tp_size': 8, ...} (resolved parameters)
  • Load weight end. ... avail mem=X GB, mem usage=X GB. (weight footprint)
  • DSV4 memory calculation: ... bytes_per_full_token=X, available_bytes=X GB, ... full_token=N
  • [label] KV Cache is allocated. dtype: ..., #tokens: N, K size: X GB, V size: Y GB (or KV size)
  • Mamba Cache is allocated. ... conv_state size: X GB, ssm_state size: Y GB
  • Memory pool end. avail mem=XX GB
  • Capture target|draft decode|verify|prefill CUDA graph end. elapsed=X s, mem usage=X GB, avail mem=X GB.
  • Post-capture KV sizing: KV cache allocated. ... KV size: X GB, avail mem=X GB (opt-in)
  • max_total_num_tokens=XX, ... chunked_prefill_size=-1|N, ... available_gpu_mem=XX GB

Legacy ServerArgs(...), Memory profiling: and Capture cuda graph end. remain accepted for older/private branch logs; they are not current main emitters.

Current vLLM V1 logs instead expose:

  • Initial free memory: ...; Requested memory: ... (DEBUG only)
  • Model loading took ... GiB
  • Available KV cache memory: ... GiB
  • Graph capturing finished ... took ... GiB
  • <device> KV cache size: ... tokens, Maximum concurrency for ... tokens per request: ...x
  • Free memory on device (F/T GiB) ... Current kv cache memory in use is K GiB (INFO)
  • CUDA graph pool memory: A GiB (actual), E GiB (estimated), ...
  • Initial free memory X GiB, reserved Y GiB memory for KV Cache ... (explicit KV bytes)
  • Maximum concurrency for ... tokens per request: ...x

If the log is from a running instance, capture it by redirecting stdout/stderr to a file at launch time.

Step 2: Optionally capture nvidia-smi data

For per-rank memory comparison:

docker exec <container> nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt

Step 3: Run the analyzer

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --nvidia-smi-file /path/to/smi.txt \
  --gpu h200 \
  --config-json /path/to/config.json

For JSON output (automation):

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --format json

Step 4: Review and interpret results

The analyzer prints:

  1. Memory breakdown table: each category (weights, KV pool, CUDA graph, framework, other) with GiB, MiB, percentage, and derivation
  2. Per-rank comparison: nvidia-smi data across all TP ranks
  3. KV pool detail: pool configuration, KV dtype, replication factor, per-token byte calculation
  4. Concurrency estimate: max concurrent requests for different token lengths
  5. Tuning notes: configuration changes that may increase capacity

For vLLM, the report also includes a vLLM Startup Evidence section. Values that are not present in the log stay unknown; do not translate them into SGLang-only fields.

When To Use It

  • After launching an LLM serving instance, to understand how GPU memory is distributed
  • When comparing different --mem-fraction-static values and their impact on KV pool capacity
  • When planning deployment capacity: how many concurrent requests can a given GPU configuration support
  • When investigating OOM issues: identifying which memory category is consuming the most
  • When evaluating whether fp8 KV cache or EP can improve concurrency

Key Concepts

mem-fraction-static

The inspected SGLang source reserves runtime headroom from pre-model-load free memory: headroom = pre_model_load_memory * (1 - mem_fraction_static). The KV budget starts from currently free memory minus this headroom and other model-specific reservations. It is not post_weight_free * fraction. Post-capture sizing may update target and eligible DFlash-family draft pools in one round. Sum target plus distinct draft-owner summaries per rank; a new target summary resets the round. VA upper bounds reserve address space rather than physical bytes. Keep independent pools and recurrent state separate. Use final logged capacity and rank-local evidence; see source contracts.

The default is auto-resolved from the hardware and configuration. Read the resolved value from server_args={...}. A lower value increases runtime headroom at the cost of KV capacity. Do not infer 0.88 when it is absent.

For deployments actually using FlashInfer's Cake MoE all-reduce with an omitted all-reduce output, account separately for its live and retired scratch sinks. They grow geometrically and retain addresses used by CUDA graphs; they are not KV cache or necessarily reclaimable when a request ends. Startup KV logs alone cannot identify this allocation history. The address-retention behavior was introduced in FlashInfer #5894.

KV Head Replication

For standard GQA/MHA, each rank stores max(1, num_key_value_heads // tp_size) KV heads. When TP exceeds the head count, each head is replicated across a subgroup of ranks, not the whole head set on every rank. For kv_heads=4, tp=8, each rank stores one head; for kv_heads=1, tp=8, all eight ranks store that single head. MLA stores a replicated latent plus RoPE key instead. The theoretical calculation assumes all configured layers reside on the rank; use rank-local layer counts for pipeline parallelism.

SWA (Sliding Window Attention) Compression

Models like DeepSeek-V4 use CSA (Compressed Sliding Attention) and HCA (Hierarchical Context Attention) with sliding windows. This reduces per-token KV cache bytes compared to the theoretical full-attention calculation. The bytes_per_full_token reported in the log already accounts for this compression.

Reporting Checklist

Include:

  1. Serving configuration: model, GPU, TP/PP/EP, mem-fraction-static, kv-cache-dtype
  2. Memory breakdown table: category / GiB / MiB / percentage / derivation source
  3. Per-rank nvidia-smi comparison: used and free memory per TP rank
  4. KV pool detail: pool size, bytes_per_full_token, KV dtype, replication factor, theoretical per-token KV calculation (when config.json provided)
  5. Concurrency estimate table: request token length / token-limit / request-limit / max concurrent
  6. Tuning notes based on free memory and configuration

2026-08-23 Runtime Notes

SGLang v0.5.18 consolidates compiled-kernel caches under SGLANG_CACHE_DIR. The first launch after an upgrade can look like extra "other" memory and a longer startup because Triton / FlashInfer / DeepGEMM / Inductor recompile once. Overlapped checkpoint staging (--startup-weight-load-mode overlap) also changes the startup memory curve: weight pages and CUDA-graph capture can overlap, so do not treat a shorter load-weight interval as a smaller weight footprint.

Known Limitations

LimitationDetailWorkaround
Framework-specific checkpointsSGLang and vLLM expose different memory checkpoints, so some fields are not comparable one-to-oneReport unknown fields and compare only shared, explicit evidence
SWA compression modelsPer-token KV bytes cannot be independently calculated from model config for CSA/HCA attention — the framework's internal SWA window parameters are neededUse bytes_per_full_token from the log directly
DeepGEMM JIT memoryThe analyzer categorizes DeepGEMM JIT compilation memory as "other" because it is not explicitly reported in the logCompare with nvidia-smi total for accurate accounting
PP (Pipeline Parallelism)Memory decomposition is per-rank; PP configurations may have uneven memory across stagesSpecify --target-rank for each PP stage
MoE expert bufferSome frameworks allocate additional buffers for expert routing that are not separately reportedIncluded in "model weights" or "other" depending on when allocated

References

  • references/log-patterns.md: log line patterns and their semantics for memory analysis.
  • references/gpu-specs.json: GPU HBM specifications for h20, h100, h200, and b200 aliases.
  • scripts/capacity_analyzer.py: the core analysis script.

Memory capacity sources

The bundled hardware table records vendor GB maxima, not usable GiB. Prefer nvidia-smi used+free or vLLM's logged total. B200 retains the DGX 180GB SKU; other Blackwell SKUs may differ. DGX Spark's 128GB is unified system memory, not dedicated HBM. L20 is the official 48GB SKU; do not infer a capacity for l20n from the name. GB200 records the physical 192GB package maximum; its deployed SKU and custom SKUs require measured totals. Official source URLs are stored with each newly added hardware entry.

LLM Torch Profiler Analysis

Produce three tables: kernel/source attribution, overlap opportunities, and fusion patterns. Start from the user's trace or running server. Inspect the actual model, framework revision, phase, device/rank and parallelism before choosing a fast path. Existing traces need no GPU or framework installation. The Python analyzers use the standard library.

Read source contracts for dated source inspections and version-sensitive profiler APIs. Apply the general dispatch and numerical principles to every model. The fusion catalog contains concrete operator families, eligibility examples and storage contracts; the overlap catalog covers scheduling. Consult the model-history knowledge base when detailed PR evidence is needed. Source checks and historical captures are not fresh GPU validation.

Choose the evidence

ModeEvidence and limits
Existing traceAll four frameworks; SGLang Omni Python roots are also recognized. Use Chrome JSON or gzip with GPU events. A generic trace may need --framework explicitly.
Single live captureUse a shared server/client output directory and a supported HTTP profiler. Save actual launch arguments and package revisions.
Mapping + formalEager/graph-off trace supplies call-site context; warmed graph trace supplies timing. Match signatures, shapes and stage before transferring attribution. Never transfer eager timing to the formal run.

For a diffusion workload, record whether it uses the framework's native backend or a Diffusers fallback. A fallback trace can be analyzed, but does not establish native-kernel performance or dispatch.

Read a trace

Commands below are relative to this skill directory.

python3 scripts/analyze_llm_torch_profile.py \
  --framework sglang --input /path/to/TP-0-DECODE.trace.json.gz

python3 scripts/analyze_llm_torch_profile.py \
  --framework vllm \
  --mapping-input /path/to/eager-mapping \
  --formal-input /path/to/warmed-graph

Do not collapse a distributed run to rank0 when investigating rank skew or collective waits. Keep the original capture; a GPU-only or compact export is for sharing, not replacement evidence. Use torch-profiler-layer-track for verified L0/L1 guides and synthetic display lanes, or llm-pipeline-analysis for layer intervals.

Capture against a running server

The common helper defaults to separately driven prefill and decode workloads: nominal input/output lengths 4090/1 and 1/2048, with 10 warmup steps and 5 requested capture steps. Override these to match the slow serving scenario. Synthetic prompt lengths are not a tokenizer-independent guarantee; save actual usage and batch/graph dimensions. A long-context decode bottleneck needs a representative KV length, not automatically the one-token-input default.

python3 scripts/analyze_llm_torch_profile.py \
  --framework sglang --url http://127.0.0.1:30000 \
  --output-dir /shared/profiles/run-001 \
  --profile-workload both --warmup-steps 10 --num-steps 5 \
  --prefill-input-len 4090 --prefill-output-len 1 \
  --decode-input-len 4090 --decode-output-len 2048

--profile-workload accepts both, prefill, decode, or legacy (explicit probe prompt/token settings). Workload labels alone do not make a trace purely decode: inspect stage annotations and exclude request prefill when needed. The helper targets SGLang profiler v1 (default). Declare --sglang-profile-v2 if the remote server enables SGLANG_PROFILE_V2=1; the helper refuses this incompatible flow before issuing control requests. /server_info does not expose remote environment variables. Native v2 needs profile_by_stage:true and num_steps, without start_step, merge_profiles or /stop_profile. Current Rust ingress also supports start/stop with the same scheduler control. Older images may lack these routes; check the actual response. Keyed servers need the configured admin/API credential; the bundled helper has no auth flag, so use authenticated native control and offline analysis for those captures. The host launcher explicitly sets SGLANG_PROFILE_V2=0. The helper puts its label in profile_id, so merging can discover rank files. SGLang can emit stage-specific traces; its scheduler stop counter may require a final forward to close the window. The helper's SGLang workload adds a guard request/step. Check actual trace counts instead of assuming identical windows across frameworks.

For other frameworks add --no-profile-by-stage. Their HTTP windows and iteration controls differ:

  • vLLM: launch with --profiler-config '{"profiler":"torch","torch_profiler_dir":"/shared/profiles/run-001","max_iterations":5,"ignore_frontend":true}'. Current start accepts per-session profile_prefix, max_iterations, delay_iterations and profiler_kwargs. The helper checks /openapi.json and sends a stage-qualified prefix and --num-steps as max_iterations only when advertised. Older/OpenAPI-disabled servers retain bodyless control and launch-configured limits. Output directory and optional profiler warmup remain launch settings for this helper; probe steps do not override a warmup/wait/active schedule or guarantee a pure decode window. torch_profiler_with_stack defaults true in the inspected source. Frontend traces can contain no kernels; select GPU-worker traces for GPU analysis. active_iterations only takes effect with a warmup/wait schedule; use max_iterations to bound workers. The host option is --profiler-max-iterations (--profiler-active-iterations remains an alias). Two capture rounds require vLLM #57460, merged 2026-09-21; older images need separate launches for prefill and decode. Keep CPU in torch_profiler_activities for source attribution and step annotations; CUDA-only activities omit both Python stacks and annotations.
  • TensorRT-LLM: the inspected PyTorch server accepts runtime start with output_dir and activities; the helper sends both and stops after probes. Native traces are uniquely named per request and rank. There is no need for the retired py_executor.py rank0-only patch or host-specific matrix runner. Current setup is in _torch/pyexecutor/profiling.py, still without with_stack=True. Report unresolved Python attribution, or make a separate, recorded mapping build against that installed file. Older images without HTTP support must use their documented capture method plus --input.
  • TokenSpeed: the helper sends output_dir, CPU/GPU activities, with_stack=true, record_shapes=false, and a stage-qualified profile_id. Current servers expose start/stop; verify the deployed image. Proton's native Hatchet artifacts are not Torch Chrome traces. Current serve uses SMG plus a headless Rust-hosted engine; the old Python control sidecar was removed. Resolve the actual gateway/control URL and validate forwarding on the installed pinned wheels. Do not assume serving port + 1. EXPERT_LOAD writes .expert-load.pt, not Chrome traces. TOKENSPEED_CUPTI_GRAPH_WARMUP=1 can leave zero GPU events in later captures. Current Torch capture requires process-global profile_all_threads support for Rust-owned forward threads. AMD capture needs HSA_TOOLS_DISABLE_REGISTER=0; Proton cannot combine with GPU/CUDA_PROFILER, and expert-load recording refuses online EPLB. See the source contracts.
python3 scripts/analyze_llm_torch_profile.py \
  --framework trtllm --url http://127.0.0.1:8000 \
  --output-dir /shared/profiles/run-002 \
  --no-profile-by-stage --profile-workload both

A server-side directory must be mounted here or copied back for analysis. The helper rejects a capture with no new artifact, recognizes overwritten filenames, and copies fresh remote traces into a capture-specific subdirectory without moving the server's originals. Preserve that returned directory rather than analyzing every historical run under a shared root. Concurrent profiling sessions still need separate output directories.

Native alternatives include SGLang graph-capture traces under graph_capture_profile/, MEM snapshots, and CUDA_PROFILER with nsys; vLLM CUDA/NVTX and Proton Chrome export (the old capture_torch_profiler option is deprecated and has no effect); TRT-LLM TLLM_PROFILE_START_STOP/TLLM_TORCH_PROFILE_TRACE; and TokenSpeed VizTracer/Proton plus tokenspeed merge-traces. Graph-capture and VizTracer artifacts are excluded from default runtime-trace discovery. Pass a file explicitly when intentionally studying one of these artifacts. Native Hatchet, MEM and expert-load outputs require their own readers.

Interpret before optimizing

  1. Confirm nonzero GPU events, a complete warmed forward, actual dispatch and target/draft separation (step[VERIFY ...] versus step[DRAFT ...] in SGLang; vLLM uses execute_context_*_generation_*). BS=1 need not mean kernel M=1 during speculation.
  2. Read all relevant GPU streams. A CPU cudaGraphLaunch span is host time; missing lanes or dropped CUPTI records can look like GPU idle time.
  3. Separate sum of kernel durations, union of GPU busy intervals, and end-to-end wall time. Overlapping kernels can sum to more than wall time. A table's duration share is not removable latency or an additive speedup estimate.
  4. Identify the producer/consumer join and exposed tail. PDL consumers may include dependency wait in kernel duration. Compare PDL on/off with the same warp configuration before attributing a bimodal duration to arithmetic.
  5. Consult source dispatch before proposing fusion. Existing routed-expert activation/quant does not establish shared-expert coverage. Match rounding, clamp, scale/layout, padding and graph reuse contracts. Check a quantization oracle against an independent formula: a kernel and its reference can share the same scale-floor error.
  6. Follow state ownership across fused boundaries: cached normalized rows, quantized values/scales and next coefficients can become invalid after a residual rewrite or row selection. V4.1 Engram/late-layer selection are concrete examples; fewer launches do not prove equivalent semantics.
  7. Time changes without profiling; run kernel correctness and actual-model accuracy on the candidate path. NCU replay/resource metrics help explain a kernel but can perturb or serialize the multi-stream behavior being studied.

For source mapping, prefer model/layer frames over dispatch/runtime wrappers. Retain unknown locations as unknown. Kernel names and eager similarities are hints; inspect the concrete producer, consumer and backend. The script's three tables are observations and candidates, not proof that an optimization is safe or already active in a different framework.

Capture labels must be filename components: the helper rejects path separators and NUL before contacting the server. Live capture requires positive num_steps. SGLang v1/v2 default to stacks on and shapes off; request shapes explicitly when needed. Verify emitted phases and filenames against the deployed server.

References and output

Load only relevant references:

  • source-map.md: immutable profiler and worker paths.
  • heuristics.md: attribution and overlap labels, dispatch, numerical contracts, validation and their limitations.
  • vllm-torch-compile-fusions.md: current pass registration, platform gates and source patterns.
  • fuse-overlap-catalog.md and overlap-catalog.md: historical precedents; recheck a PR's current state and target revision before borrowing code. The overlap catalog also covers GPU idle gaps, scheduler overlap, phase graphs, metadata glue graphs, replay streams, PDL and collective dispatch.

Return the trace/report path, framework and available model/server arguments; the kernel, overlap-opportunity and fusion-pattern tables; the main exposed bottleneck; attribution confidence and whether mapping/formal traces were used. For a new capture also state which phases/ranks contain GPU events. Do not claim a speedup or accuracy equivalence without the corresponding experiment.

Model Architecture Diagram

Workflow

Return only public original diagrams indexed by this skill.

  1. Run the bundled resolver:
python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py "<model name>"
  1. If the resolver returns kind: existing, return the raw image Markdown it prints and preserve the source attribution line.
  2. If the resolver returns kind: no_match, tell the user that no public original architecture diagram is indexed for that model.

Source Priority

Use references/diagram-index.json as the source of truth. It stores raw GitHub image URLs and official vendor/CDN or pinned Hugging Face resolve/<sha> URLs. Official sources include Qwen and XiaomiMiMo; GitHub sources:

  • datawhalechina/self-llm
  • CalvinXKY/InfraTech
  • Tongyi-MAI/Z-Image
  • Wan-Video/Wan2.1
  • Wan-Video/Wan2.2
  • Tencent-Hunyuan/HunyuanVideo
  • Tencent-Hunyuan/Hunyuan3D-2
  • brayevalerien/Flux.1-Architecture-Diagram

Prefer detailed implementation, cookbook, or architecture-card diagrams over paper figures. Good sources show module boundaries, dataflow, MoE / attention / cache paths, or model-specific runtime structure rather than only a high-level paper overview. Official repository diagrams and curated implementation diagrams are first choice; paper figures are fallback only when no more detailed public original diagram is indexed.

Do not copy remote image binaries into the skill. Return the indexed original URLs so the chat renderer can display the original image.

Existing Diagram Rule

For a direct match, show the original image. Good direct matches include:

  • DeepSeek V3/V3.2/V4, GLM-5, Kimi K2/K2.5/K3, MiniMax M2.5, Qwen3.5, Qwen3-VL, and Step 3.5 Flash from InfraTech.
  • Qwen3.8-Flash-Next has an official diagram (also named qwen4_exp in runtime code). Plain Qwen3.8 2.4T remains no_match; these are different architectures.
  • MiMo V2.6, MiMo V2.5 Pro and GLM-5.2 have indexed originals. GLM-5.3 remains no_match because no direct diagram was verified.
  • Hunyuan-A13B, Kimi-VL, Qwen3, Qwen3-VL detail flows, MiniMax M2, and Llama 4 architecture/module diagrams from self-llm.
  • Z-Image, Wan2.1, Wan2.2, HunyuanVideo, Hunyuan3D 2.0, and FLUX.1 diffusion architecture/module diagrams from public GitHub sources.

If multiple diagrams match, show all high-confidence matches up to the resolver's default limit. For example, DeepSeek V3 may return the full architecture plus MLA MHA/MQA diagrams.

Hosted Original Diagram Gallery

Do not commit the sgl-cookbook-model-architecture-images/ gallery into the repository. The public-original image set is hosted as a GitHub Release asset and indexed by a GitHub issue.

Current hosted artifact:

The gallery is a 2026-05-02 snapshot containing 44 public original diagram image files; it does not mirror the current index, plus a lightweight index.html, index.md, manifest.json, HTML contact sheet, and architecture-audit.md.

To inspect the gallery locally:

curl -L -o /tmp/sgl-cookbook-model-architecture-images-2026-05-02.zip \
  https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/releases/download/sgl-cookbook-architecture-images-2026-05-02/sgl-cookbook-model-architecture-images-2026-05-02.zip
unzip -q /tmp/sgl-cookbook-model-architecture-images-2026-05-02.zip -d /tmp
open /tmp/sgl-cookbook-model-architecture-images/index.html

Useful Commands

List known original diagram aliases:

python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py --list-known

Emit JSON for automation:

python3 skills/model-architecture-diagram/scripts/model_architecture_diagram.py "GLM-5" --format json

References

  • references/diagram-index.json: original diagram link index and aliases.
  • references/source-notes.md: audited source repositories and local cache paths.

Model Compute Simulation

Overview

Use this when the question is about operator order, tensor dimensions, FLOPs, MFU, or parallelism checks. The simulator loads a model config, builds the representative operator sequence, prints tensor shapes and FLOPs, and can estimate MFU from measured latency.

Inputs

Before running a simulation, collect or verify these inputs:

ItemWhy it mattersHow to obtainDefault if user skips
Model nameResolves to config in model-config-index.json; determines entire architectureAsk user or infer from trace context— (required)
Config accuracyIndexed values may differ from actual serving config (e.g. routed_expert_intermediate_size, compress_ratios)Ask user to provide config.json or verify key params against HuggingFaceUse indexed values with a caveat
GPU typeDetermines peak FLOPS for MFU denominatorAsk user— (required for MFU)
dtype (bf16 / fp8)Select the actual GPU SKU and arithmetic peak; mixed precision needs per-op treatmentAsk userbf16
Batch size & seq lenDirectly affects FLOPs and tensor shapesAsk userB=1, S=1 (decode)
TP / DP / EPTP splits GEMM FLOPs across GPUs; EP splits expert FLOPsAsk userRead the run manifest; otherwise report assumptions explicitly
Measured latency (ms)Required for MFU numerator; must be per-GPU forward-pass wall-clockAsk user or extract from a profiler trace— (optional, no MFU without it)

If the model is not in model-config-index.json, ask the user for a config.json path or add an indexed config before running estimates. The 2026-10-05 index pins public HF config revisions for V4 Flash/Pro/V4.1, K3, GLM-5.2/5.3/5.3-Flash, Qwen3.8-Flash-Next, Hy3/Hy4, Inkling, dots3, Step-3.7, Nemotron-3/3.5, LongCat and Gemma4, and corrects older model dimensions. Qwen/Qwen3.8-A95B returned HTTP 401 at this audit and is not indexed. Use --config /path/to/config.json for a raw nested HF config; the local normalizer expands text wrappers and expert/layer aliases.

Shape and timing limits for speculative MoE

For DSV4.1, obtain actual target/draft row counts from the trace: request BS=1 can issue six target-verify rows with DSPARK block size 5. Do not substitute acceptance length for the number of rows computed. Account for TP-sharded shared-expert shapes separately from routed expert assignments and EP. Do not divide an unverified model template by TP/EP and call it measured FLOPs. The simulator applies uniform TP division outside routed experts and uniform EP division inside routed experts. This is a what-if approximation: replicated routers, norms, latent projections and backend-dependent shared-expert ownership require a run-specific adjustment before interpreting MFU. The pinned V4.1 entry is a static target-model template; DSPARK draft, vision, Engram/PLE memory traffic and backend dispatch require separate evidence.

Current SGLang detailed_annotations can emit c_ (context) and g_ (generation) aggregates: sq=ΣNq, sk=ΣNkv, sqsq=ΣNq², sqsk=ΣNq·Nkv. Verify/draft Nq includes speculative width. Mixed batches emit both groups; replacing this distribution with average lengths loses attention work. Use these values to audit shapes manually against source contracts. The simulator still accepts a homogeneous B/S shape and does not parse these suffixes or model arbitrary sparse attention from their sums.

Kernel-duration sums include overlap and possibly PDL dependency waits. Use wall-clock time for a whole-pass MFU denominator, and measured GPU SKU/dtype peaks rather than assuming every FP8 operation doubles useful FLOPs. Follow the dispatch and timing principles when assigning kernel names to attention, shared experts, routing or mHC.

Workflow

Step 1: Load model config

Resolve the model name and load its configuration parameters:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "<model name>" --list-models

The script resolves the model name against references/model-config-index.json, which stores public HuggingFace config parameters (hidden_size, num_experts, MLA ranks, etc.).

If the model is not indexed, tell the user to provide a config.json path or request an index update.

Step 2: Generate execution flow and tensor dimensions

Run the simulator with batch size, sequence length, and parallelism configuration:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 1 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16

The simulator prints:

  • Per-layer operator sequence with FLOPs and tensor shapes (shape_in → shape_out)
  • Attention vs MoE/FFN FLOPs proportion per layer
  • Total model FLOPs for a single forward pass

For decode: use --seq-len 1 --context-len TOTAL_KV_LENGTH. For extend, --context-len is the prefix length, with causal work for the new --seq-len tokens. Full/window attention includes KV context in QK and AV FLOPs. The LM head computes last-token logits unless --all-token-logits is given. Every run prints batch/shape/parallelism/GPU/dtype assumptions to stderr. Sparse selection, KDA/Mamba scalar work, norm/Sinkhorn and zero-expert top-k are architecture estimates, not measured instruction counts. LongCat routed FLOPs assume all top-k are nonzero (an upper bound until routing is measured). For prefill: use --seq-len <prompt_length>.

Step 3: Estimate MFU with measured latency

Provide the measured forward-pass latency to compute MFU:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 1 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 15.0

MFU = theoretical_min_time / measured_time × 100%

The simulator prints:

  • Overall MFU
  • Per-layer MFU (uniform layer-time assumption)
  • Per-operator FLOPs proportion (for identifying which ops dominate)

GPU peak FLOPS are loaded from references/gpu-specs.json. The bundled hardware table includes H100/H200/B200, B300, GB300, MI355X, RTX PRO 6000 and DGX Spark, each with official URLs. H20 is retained as an unverified historical estimate. Unknown dense peaks are null and MFU for that dtype is refused; RTX Server and Spark sparsity is unconfirmed. B300/GB300 memory capacity is physical maximum, not usable allocation. --dtype accepts bf16, fp8/mxfp8 and fp4/mxfp4/nvfp4; INT8 must use its separately published peak. Use aliases such as --gpu h100, --gpu b2000, or --gpu b200 when running on those local boxes.

Step 4: Per-operator MFU with kernel-level latency

When you have per-kernel measured latency, compute per-operator MFU by mapping kernel durations to the compute flow.

Method A: --kernel-flow (kernel-level MFU, recommended)

Provide per-kernel detail as JSON, then feed it to the simulator for kernel-level MFU analysis. This preserves every kernel row from the compute flow and adds FLOPs/MFU columns.

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-flow @/tmp/layer3_detail.json

The --kernel-flow parameter accepts a JSON string or @file path. It produces a kernel-level MFU table that preserves all kernel rows from the compute flow and adds:

  • Mapped Op: which operator this kernel maps to
  • FLOPs: operator's total FLOPs
  • Theo(us): theoretical minimum time
  • MFU%: measured FLOPs utilization
  • shape_in→shape_out: operator tensor dimensions

When --kernel-flow is provided, the static per-operator template is omitted because the kernel-level MFU table already carries per-kernel shape and FLOPs information. The output keeps the model summary, serving configuration, total FLOPs, overall MFU calculated from the measured layer duration, and the kernel-level MFU table. In --format json, the raw evidence is stored under kernel_flow and no text table is appended.

Mapping rules:

  • Direct-match kernels (mla, moe, mhc, rmsnorm, hadamard, rope, quant, topk, etc.): time is assigned directly to the corresponding operators
  • Generic GEMM kernels (gemm_fp8, gemm_bf16): time is distributed to remaining unassigned projection GEMM operators by FLOPs share
  • Overhead kernels (allreduce, moe_align, moe_sort, other): rows preserved, FLOPs/MFU marked as N/A

FP8 kernel MFU correction: Kernels in categories moe (fused_moe_kernel) and gemm_fp8 use fp8 math internally even when --dtype bf16 is specified. For these kernels, the MFU denominator uses the GPU's fp8 peak FLOPS (2x bf16 peak) instead of bf16 peak. The resulting MFU is marked with a superscript ⁸ (for example, 63.7%⁸) to show that the fp8 denominator was used. gemm_bf16 kernels still use the bf16 peak FLOPS denominator.

Method B: --kernel-detail (operator-level MFU, legacy)

Same input as --kernel-flow but outputs an operator-level summary table (aggregated by operator, not per-kernel). Use when you want a compact view.

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-detail @/tmp/layer3_detail.json
Method C: --kernel-ms (category-level approximation)

Use a hand-aggregated category map only when per-kernel detail is unavailable:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "Qwen3-235B-A22B" \
  --batch-size 1 --seq-len 8192 \
  --tp 8 --dp 1 --ep 8 \
  --gpu b200 --dtype bf16 \
  --measured-ms 100 --kernel-ms '{
    "mla": 4.922, "moe": 1.644, "allreduce": 0.769,
    "hadamard": 0.348, "mhc": 1.388, "gemm_fp8": 1.692,
    "gemm_bf16": 0.125, "rmsnorm": 0.227, "quant": 0.311,
    "rope": 0.209, "topk": 0.122, "activation": 0.071,
    "other": 0.437
  }'

The --kernel-ms parameter accepts a JSON object mapping kernel category names to their measured durations in milliseconds. It uses FLOPs-proportional distribution across entire categories, which is less precise than --kernel-detail because generic GEMM categories (gemm_fp8, gemm_bf16) span multiple operator categories.

Choose only one of --per-layer-ms, --kernel-ms, --kernel-detail, or --kernel-flow. All measured durations must be positive. The three kernel inputs also require --measured-ms from whole-pass wall-clock timing; the 100 ms in these examples is a placeholder to replace with the measurement. Kernel-duration sums cannot establish elapsed time when streams overlap.

Output includes:

  • Model architecture summary (layers, hidden_size, attention_type, MoE config)
  • Per-layer compute flow: operator sequence with tensor dimensions, FLOPs, shape_in→shape_out
  • Per-operator MFU table: each operator's FLOPs, theoretical time, measured time (from trace), MFU%
  • Kernel → operator mapping explanation (direct-match vs FLOPs-proportional vs overhead)
  • Overall and per-layer MFU

When To Use It

  • when you need compute-level detail for a known model or config
  • when the user asks about execution flow, tensor dimensions, or FLOPs for a specific serving shape
  • when the user asks about MFU and can provide measured forward-pass latency
  • when comparing compute profiles across different parallelism configurations

Useful Commands

List known model IDs:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-models

List known GPU types:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py --list-gpus

Emit JSON for automation:

python3 skills/model-compute-simulation/scripts/model_compute_simulator.py "GLM-5" --format json

Reporting Checklist

Include:

  1. Model architecture summary: model name, config source, num_layers, hidden_size, attention_type, MoE config (num_experts, topk, shared_experts), MHC, head_dim
  2. Serving configuration: batch_size, seq_len, TP, DP, EP, GPU, dtype
  3. Per-layer compute flow (showing first representative layer in detail):
    • Operator sequence table: name, category, FLOPs, shape_in → shape_out
    • Attention vs MoE/FFN FLOPs proportion
  4. Total model FLOPs for a single forward pass
  5. Kernel-level MFU table (when --kernel-flow provided):
    • Preserves ALL kernel rows from the compute flow (never deleted)
    • Per-kernel columns: # | Half | Category | Simplified Name | dur(us) | % | Mapped Op | FLOPs | Theo(us) | MFU% | shape_in→shape_out
    • Direct-match kernels: show mapped operator FLOPs/MFU
    • Overhead kernels: show N/A for FLOPs/MFU, row preserved
  6. Operator-level MFU table (when --kernel-detail or --measured-ms provided):
    • Each operator: name, category, total FLOPs, per-GPU FLOPs, theoretical time, measured time (from trace), MFU%
    • Kernel category → operator mapping explained
  7. Overall MFU and per-layer MFU
  8. One-line summary: dominant compute category, MFU status, key bottleneck

Trace-Based Validation (extract_compute_flow_from_trace.py)

Use scripts/extract_compute_flow_from_trace.py to extract the real operator sequence and tensor dimensions from a torch profiler trace, then compare against the static template as ground truth validation.

# Extract compute flow from a trace
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
  --input /path/to/trace.json.gz --format text

# Compare trace against static template
python3 skills/model-compute-simulation/scripts/extract_compute_flow_from_trace.py \
  --input /path/to/trace.json.gz \
  --compare qwen3-235b-a22b \
  --batch-size 1 --seq-len 1 --tp 8 --ep 8

Trace/template operation presence is compared through semantic families such as matmul, attention, norm, and router; raw aten::* names are not compared directly with semantic template names. A positive --min-flops threshold removes operations whose FLOPs could not be determined.

Compute Flow Confirmation Hierarchy

When the static template or trace extraction cannot fully confirm the compute process (e.g. ambiguous scope, missing shapes, new model architecture), follow this escalation hierarchy:

  1. Static template (model_compute_simulator.py + model-config-index.json) — fast, covers known models

  2. Trace extraction (extract_compute_flow_from_trace.py) — validates template against real execution

  3. Inference framework source code — when trace is insufficient (missing Input Dims, CUDA Graph replay, compiled kernels without scope), read the model's forward flow directly from the serving framework source:

    • SGLang: python/sglang/srt/models/<model_name>.py — contains the forward() method and declare_attn/declare_ffn stage boundaries in python/sglang/srt/layers/layer_boundary/factories.py
    • vLLM: vllm/models/<family>/{common,nvidia,amd}/ for migrated 2026 families; older models remain under model_executor/models
    • TensorRT-LLM: tensorrt_llm/_torch/models/modeling_<family>.py + model configs

    When consulting framework source, focus on:

    • The forward() method: operator call order and residual connections
    • QKV / O projection: whether LoRA-style down/up projections are used (q_lora_rank, o_lora_rank)
    • MoE routing: top-k selection, shared vs routed expert split
    • TP/EP slicing: which dimensions are split and how FLOPs divide across GPUs
    • Any model-specific ops not in the static template (e.g. MHC, Hadamard, indexer)

    Action: If the framework source reveals discrepancies with the static template, update model-config-index.json and/or build_layer_ops() accordingly.

Limitations of Trace Extraction

LimitationDetailWorkaround
record_shapes=True requiredTrace must be captured with shape recording enabled; without it, Input Dims fields are absent and FLOPs cannot be computedEnable shape recording explicitly in the deployed profiler: stacks and shapes are distinct settings. SGLang defaults may be disabled by environment; TokenSpeed helper sends record_shapes=false. Inspect TensorRT-LLM _torch/pyexecutor/profiling.py for a recorded mapping build
CUDA Graph modeDuring graph replay, cpu_op events may only appear once (at capture time); shape information for replayed iterations is not re-recordedThe script detects graph capture phases and annotates affected ops; use eager-mode traces for full coverage
TP-sliced dimensionsTrace shows post-TP-split dimensions (e.g. H/TP), not the full-model viewUse --tp in --compare mode to scale trace FLOPs back to full-model equivalents
Scope attribution qualityPython scope depends on with_stack=True; some frameworks or compiled paths may produce shallow or missing scope chainsGraceful degradation: ops with unresolved scope are categorized as "other"
Not a replacement for static templatesTrace extraction is a validation and discovery tool; static templates remain the primary fast-analysis pathUse trace extraction to verify templates for new models, then update model-config-index.json if discrepancies are found

References

  • references/model-config-index.json: model configuration parameters (hidden_size, expert counts, MLA ranks, etc.).
  • references/gpu-specs.json: GPU peak FLOPS specifications for MFU calculation.
  • scripts/extract_compute_flow_from_trace.py: trace-based compute flow extraction and template validation tool.

Source audit: SGLang V4, K3 latent experts, NVIDIA HGX dense/sparse specs, AMD MI355X specs. HF field provenance is hf_config_source at hf_config_revision; construct https://huggingface.co/<repo>/blob/<revision>/config.json for the exact source. No GPU run is implied by a config refresh.

SGLang explainer video

Input: sglang PR <N> (or a github.com/sgl-project/sglang/pull/N URL), or a Markdown article path. Output: ~/Desktop/SGLang_PR<N>_讲解_中英字幕.mp4 (article: SGLang_<slug>_技术解读_中英字幕.mp4) plus a still …_封面.png and a Bilibili cover …_封面_B站.png, rendered 1920×1080 with a 中文 narration and bilingual captions.

This skill drives HyperFrames' pr-to-video workflow (heygen-com/hyperframes, pinned) with a fixed house style and several fixes that make it work for Chinese. Read references/house-style.md before writing any script — it holds the rules the user already corrected once (terms, captions, cover, credits, PR status).

Core principle: understand it fully, explain all of it

The video exists so a viewer who has not read the PR / article understands every algorithm and every meaningful code change in it. That decides everything else:

  • No target length. Length follows the material: a one-line docs PR may need ~1 min; a multi-file kernel or scheduler PR, or a long article, may need 5–10+ min. Never cut a mechanism, code path, edge case, number or caveat to make the film shorter — add frames instead. Keep each frame to one idea (≲ 60 汉字, ~11 s) so visuals keep up; the number of frames is unbounded.
  • Understand before you script. Read all of the source (Step 2) and write capture/understanding.md first; every item in it must map to at least one frame.
  • Code is the spine, not a footnote. Open with the value (what the change fixes / speeds up), then walk through the algorithm and every changed code path in a sensible order, with a worked example where it helps. This overrides pr-to-video's story-design.md where it says otherwise (its "Whole-film target ≤ ~3 min", "feature 2–4 real diff hunks", "implementation is the footnote"): feature every hunk that carries logic (group pure renames / formatting / test boilerplate into one frame), split long hunks across frames (≤ 12 lines each), and explain why each change is needed, not just what it is.

0. Setup (start of every session)

bash <this-skill>/scripts/setup_env.sh      # first run ~2 min; afterwards seconds (idempotent)
source ~/.cache/sglang-explainer-video/env.sh

Re-run it at the start of each session: env.sh records this skill's install path ($SEV_SKILL, $HF_MEDIA_ENGINE), so it must point at the copy you are using now.

It needs Node ≥ 22 (brew install node@22), ffmpeg/ffprobe, git, curl and Chrome. It creates a venv (edge-tts, fonttools, and mlx-whisper on Apple Silicon for pron_check.py), a pinned HyperFrames checkout with patches/hyperframes-captions-cjk-bilingual.patch applied, the Noto Sans/Serif SC fonts, and env.sh ($SKILL_DIR = pr-to-video, hf = pinned HyperFrames CLI wrapper, $HF_MEDIA_ENGINE = our edge-tts engine, $SEV_SKILL, $SEV_PY). Nothing is installed globally: always run the CLI as hf …, never npx hyperframes init without the env (init would install skills into ~/.claude/skills).

Every command below assumes source ~/.cache/sglang-explainer-video/env.sh and P=<project dir>.

1. Project + brief

P=$SEV_WS/projects/sglang-pr-<N>            # article: $SEV_WS/projects/sglang-article-<slug>
hf init "$P" --non-interactive --example=blank --skill=pr-to-video

Write $P/BRIEF.md (template in references/house-style.md §Brief). Run autonomous (flow: automation, storyboard: no): do not stop for plan/sketch approval; the only questions worth asking are real choices (voice change) — otherwise follow the house style. Never ask about length: it follows the material.

2. Ingest the source

PR mode — always the full URL (owner/repo#N fails outside a git checkout):

(cd $P && node $SKILL_DIR/scripts/fetch-pr.mjs --pr https://github.com/sgl-project/sglang/pull/<N> --out-dir ./capture \
  && node $SKILL_DIR/scripts/ingest.mjs --pr-json ./capture/pr.json --diff ./capture/diff.patch --out-dir ./capture/extracted \
  && node $SKILL_DIR/scripts/fetch-people-avatars.mjs --people ./capture/extracted/people.json)

Article mode — copy the article to $P/capture/extracted/visible-text.txt, write a minimal capture/extracted/tokens.json ({"title": "...", "colors": [], "fonts": []}), and download the article's own figures into $P/assets/article/fig<i>.<ext> (they are the author's images; they become real assets on frames).

Read everything before writing anything. capture/extracted/visible-text.txt is budget-truncated, so it is not enough on its own:

  • PR mode: the PR body, the full capture/diff.patch, review comments that changed the design (gh pr view <URL> --comments), and every touched function in full at the PR head (gh api "repos/sgl-project/sglang/contents/<path>?ref=<headRefOid>" --jq .content | base64 -d) so you can explain the code around each hunk, not just the hunk. Follow calls into helpers when the change depends on them.
  • Article mode: the whole article, every code block and figure, and any PR / source file it links to that the argument depends on.

Then write $P/capture/understanding.md: the problem; the algorithm step by step; every changed code path (file:function → what changed and why); the numbers with their conditions (hardware, model, workload); the model the PR involves (the one the bug shows up on, or the one it was measured on: the title and the description will name it, references/publishing.md); the caveats the author states. This is the coverage checklist for Step 3.

Then put the SGLang brand on the project (frame.md, CJK/bilingual caption skin, ending block, logos). The theme is sglang-orange unless the video is about a Kimi / Moonshot model or the user asks for a different look; then use --theme kimi-dark (references/house-style.md §Themes):

$SEV_PY -I $SEV_SKILL/scripts/apply_brand.py --project $P [--theme kimi-dark]

3. Story + script

Follow $SKILL_DIR/SKILL.md Step 3 and its references/story-design.md (PR: fix-explainer / feature-reveal / refactor-walkthrough; article: a value-first explainer — hook → landscape → method → every evidence beat the article makes → lessons → summary), with the Core principle above overriding its length and hunk limits. Before moving on, tick every item of capture/understanding.md against the storyboard; add frames for anything uncovered. Then apply the house style:

  • Frame 1 is always the cover compositions/frames/00-cover.html (built in Step 6, not by a worker), status: animated. Frame N's file is NN-<slug>.html with NN = N-1.

  • PR videos end on the author page (name/@login/avatar on screen, never spoken; the PR link; no status).

  • Every video's last frame carries the SGLang ending block (the project's assets/brand/follow-strip.html), at y 652–872, so plan that frame's content to end above y≈630 (house style §Ending links):

    • a links card with this video's own links: the source it explains, and the cookbook or docs page to open next;
    • the fixed follow strip (关注 SGLang · GitHub · 博客 · 文档).
    • Pick the two links now and write them into that frame's Scene text. Take them from main and from docs.sglang.io/sitemap.xml, never from memory. The block is on screen only, not narrated.
  • Chinese narration, English technical terms, one idea per frame (≲ 60 汉字, ~11 s); as many frames as the material needs — no whole-film cap. Long films get section kickers (e.g. 「① 问题」「② 算法」「③ 代码」) so viewers keep their place.

  • SCRIPT.md voice line: zh-CN-XiaoxiaoNeural (edge-tts). Numbers only from the PR/article.

  • Every English word must be read right and in one breath (references/pronunciation.md): no _ reaches the voice, CamelCase brand names are spaced, symbols (×, #, +%) are spoken as Chinese. Write the narration normally — pronunciation.json rewrites only what the voice hears, captions keep the written form — then run the check and fix every FAIL (lexicon entry or reword) before TTS:

    $SEV_PY -I $SEV_SKILL/scripts/pron_check.py --script $P/SCRIPT.md --asr
    
  • Chinese polyphones must take the intended reading. 行 meaning "row / line" is respelled to 航 automatically (the voice read 一行 as yì xíng in 42 % of past lines). For any other polyphone the user flags, measure it with scripts/poly_probe.py. Edge picks the reading per request, so it synthesizes several takes and compares each one acoustically with unambiguous homophones. Add a patterns rule when the spoken form still misreads. See references/pronunciation.md for the details.

4. Narration (edge-tts, word timings)

(cd $P && node $SKILL_DIR/scripts/audio.mjs --script ./SCRIPT.md --storyboard ./STORYBOARD.md --hyperframes . \
   --out ./audio_meta.json --voice zh-CN-XiaoxiaoNeural \
 && node $SKILL_DIR/scripts/audio.mjs sync-durations --audio-meta ./audio_meta.json --storyboard ./STORYBOARD.md \
 && cp audio_meta.json audio_meta.built.json)

HF_MEDIA_ENGINE swaps HyperFrames' TTS for scripts/edge_tts_engine.py: per-word timestamps from WordBoundary, punctuation re-attached, split terms re-joined (FP32, FLUX.2, torch.equal), 2 s of silence after the last line; the voice reads the spoken forms from pronounce.py and the timings map back to the written tokens. Then check the recorded take — pauses inside a clause and every English term in context:

$SEV_PY -I $SEV_SKILL/scripts/pron_check.py --meta $P/audio_meta.json --hyperframes $P --asr

Then check where the voice cut the Chinese: seg_check.py compares Edge's word tokens with a Chinese segmentation and flags a token that straddles a word (值相同时下标 → 「时下|标」, 下标取反 → 「标取|反放」), a pause inside a word (1 除以 T → 「除 | 以」), or a number fused with 只 (乘 24 只要 → 「24只|要」, read as a measure word). Reword each hit (a comma, 的, or a different phrasing), re-synthesize the line, and run it again until it prints ✓:

$SEV_PY -I $SEV_SKILL/scripts/seg_check.py --meta $P/audio_meta.json

Then listen to the Chinese: line_asr.py transcribes every line and diffs it with the narration in pinyin. It catches a digit or a clause-final 时 that merges into another number (「L2 时」 heard as 二十, 「大时」 as 大十, 「batch 是 1」 as 11, 「batch 为 1」 as 唯一). Reword those (的话 / 的时候 / 只有 1), re-synthesize the line, and read the remaining hits; homophone noise is expected, and a hit that repeats in a fresh take is not proof (pronunciation.md, §Listening to whole lines):

$SEV_PY -I $SEV_SKILL/scripts/line_asr.py --script $P/SCRIPT.md --meta $P/audio_meta.json

Then listen to every 行 in the take: the respelling lowers the misread rate, but Edge still picks the reading per request, and Whisper writes 行 for either reading. hang_take.py compares each 行 in the take with fresh takes that say 航 or 型 there and flags the ones that took the wrong reading (128 字节行 and 行距 were read xíng in the PR 48380 video as delivered). Re-synthesize every WRONG line and run it again until it prints ✓:

$SEV_PY -I $SEV_SKILL/scripts/hang_take.py --script $P/SCRIPT.md --meta $P/audio_meta.json

Then listen for the polyphones that differ only in tone (切 qiē / qiè). hang_take and poly_probe compare spectra, which carry almost no pitch, so they cannot hear these. tone_take.py force-aligns the take and same-line controls (切 swapped for 接/些 vs 借/谢) with the Montreal Forced Aligner and compares the pitch slope of the vowel with Praat. It needs the optional tone env once (bash $SEV_SKILL/scripts/setup_tone.sh, about 1.5 GB). Edge reads a given text the same way every time, so reword every WRONG line (its reword: hint says how), re-synthesize it, and run the check again until it prints ✓:

$SEV_PY -I $SEV_SKILL/scripts/tone_take.py --script $P/SCRIPT.md --meta $P/audio_meta.json

A FAIL there means re-synthesize that line after a lexicon fix or a reword (subset re-TTS, see troubleshooting).

Keep the pronunciation lexicon growing. Every term this video exposed as misread goes back into the skill before delivery. That includes pron_check FAILs, OK~ terms that sound wrong, and anything the user flags in review. Each fix lands in scripts/pronunciation.json (patterns / parts / lexicon / approved), with its evidence row in references/pronunciation.md and a test case when the rule is new. Ship these as a PR to the skill repo. The skill only stays reliable if each video leaves it a little better; see references/pronunciation.md §The lexicon is a living file. A frame running long means it holds two ideas — split it; never drop content to shorten the film. Print the timings for Step 5:

python3 -c "import json;[print(v['frame'],v['duration_s'],' | '.join(f\"{w['text']}@{w['start']}\" for w in v['words'])) for v in json.load(open('$P/audio_meta.json'))['voices']]"

Keep audio_meta.built.json. edge-tts takes are not bit-identical; frames are cued to this take.

5. Visual design (Step 4 of pr-to-video)

Write ## Video direction + per-frame time-coded Scenes per $SKILL_DIR/references/visual-design.md, using the real word start times above. Put the palette/type/font rules from references/house-style.md §Visuals into Video direction. Code beats need a ### Source excerpt (≤ 12 real lines); article frames that show a real figure carry - asset_candidates: assets/article/figN.png — ….

6. Cover

Every video gets its own cover design (house style §Cover). Do not reuse the previous video's.

  1. Look at what has been used: references/cover-log.md, and the newest covers side by side.
    $SEV_PY $SEV_SKILL/scripts/cover_kit.py wall --out $P/cover_wall.png
    
  2. Design this video's cover from its content: what the bug or the speed-up looks like. It differs from each of the last five covers in at least three of ground colour, accent colour, type, composition and illustration technique. It keeps the hook as the largest line and the small tag 「SGLang PR 讲解 · #N」 top-left.
  3. Write the design as $P/work/make_cover.py: css(width, dy), body(prefix), the reveal steps and the list of text to fade. cover_kit.py's docstring shows the shape. Keep the content inside y 135–880.
    $SEV_PY $P/work/make_cover.py png                       # cover.png (1920×1080) + cover_bili.png (1440×1080)
    $SEV_PY $P/work/make_cover.py frame <frame-1 voice s>   # compositions/frames/00-cover.html
    
    Time the reveals to the cover narration's word cues in audio_meta.json.
  4. Compare, then log it:
    $SEV_PY $SEV_SKILL/scripts/cover_kit.py wall --out $P/cover_wall.png --add $P/cover_bili.png
    
    If the new tile could be taken for a neighbour, redesign it. Then add its row to references/cover-log.md and ship that with the video's other skill updates.

cover_bili.png is the cover to upload to Bilibili, which crops one image to both 4:3 and 16:9.

The series template "A" (scripts/make_cover.py --project $P --series "SGLang PR 讲解" --badge "#<N>" --title-zh … --title-en … --meta … --duration <frame-1 voice s> --png $P/cover.png --png-bili $P/cover_bili.png) is only for a video whose user asks for the series look.

7. Captions (bilingual)

(cd $P && node $SKILL_DIR/scripts/captions.mjs build --storyboard ./STORYBOARD.md --audio-meta ./audio_meta.json \
   --hyperframes . --out ./caption_groups.json)

Translate every group's text into $P/caption-translations.json ({"<中文 group text>": "<English>"}), then rebuild; the build warns no translation for N group(s) until all are covered.

8. Frames (≤ 5 parallel workers)

(cd $P && for b in <code-* blocks named in STORYBOARD>; do hf add "$b"; done)
node $SKILL_DIR/scripts/frame-packets.mjs --project "$P" --storyboard "$P/STORYBOARD.md"

Dispatch at most five background workers (Agent tool) at a time (five ran fine on an M4 Mac, 2026-10-23), each with the role file path, its packet paths and the dispatch context from references/worker-dispatch.md verbatim (it carries the hard rules that broke earlier runs). A packet over 48 KB means the blueprint + cited rules are too big: drop one cited rule or use blueprint: compose. Mark frames animated as workers return. A long film (many frames) goes in waves: give each worker ≤ 5 packets per dispatch, keep frames that share a stage on the same worker, and start the next wave as workers return. Then:

node $SEV_SKILL/scripts/check_frames.mjs $P          # frame contract + no system CJK fonts
$SEV_PY -I $SEV_SKILL/scripts/subset_fonts.py --project $P   # re-run after ANY text change
(cd $P && node $SKILL_DIR/scripts/captions.mjs build --storyboard ./STORYBOARD.md --audio-meta ./audio_meta.json --hyperframes . --out ./caption_groups.json \
 && node $SKILL_DIR/scripts/assemble-index.mjs --storyboard ./STORYBOARD.md --hyperframes . \
 && node $SKILL_DIR/scripts/transitions.mjs inject --storyboard ./STORYBOARD.md --hyperframes . \
 && node $SKILL_DIR/scripts/transitions.mjs verify --storyboard ./STORYBOARD.md --index ./index.html \
 && hf lint && hf check)

A failed frame goes back to its worker once with the exact finding (SendMessage). lint/check errors you fix yourself with the smallest edit. id_requires_css_escape warnings and transition-time content_overlap infos are expected. Contrast warnings that all carry one timestamp inside a crossfade window come from the outgoing frame fading out (PR 50374: 39 warnings at one instant, on text drawn in solid ink). Check the named elements' own colours and fix only a warning at a time when one frame is alone on screen.

9. Render, QA, deliver

(cd $P && hf render --skill=pr-to-video --quality high --output renders/video.mp4)
bash $SEV_SKILL/scripts/qa_sheet.sh $P/renders/video.mp4 $P/qa 2.5

Read every sheet of the rendered MP4 (not Studio snapshots): 乱码, wrong characters, untranslated or mistranslated terms, captions out of sync, content under the caption band, stray status lines. Fix, re-render, re-check. Before delivering, make sure this video's pronunciation fixes are in the skill (§4 "Keep the pronunciation lexicon growing"). The same applies after the user's review: a word they say was misread becomes a lexicon entry plus a skill PR, not a one-off edit in the project.

Open the links the ending shows (and, once they are drafted, the publish texts):

$SEV_PY -I $SEV_SKILL/scripts/check_links.py --project $P [--text desc.txt body.txt]

Then copy to the Desktop:

cp $P/renders/video.mp4 ~/Desktop/SGLang_PR<N>_讲解_中英字幕.mp4
cp $P/cover.png ~/Desktop/SGLang_PR<N>_讲解_封面.png
cp $P/cover_bili.png ~/Desktop/SGLang_PR<N>_讲解_封面_B站.png

10. Publish (only when the user asks)

Follow references/publishing.md: the user picks each file in the native dialog, the agent fills the rest. The title and the description are written for this video, from its own hook; they name the model the PR involves (DeepSeek, GLM, Qwen, …), which draws more views; and they end with the same source and cookbook links as the last frame (§Title and description). Bilibili: 含AI生成内容 · 分区 人工智能 · ~8 tags · 1440×1080 B站 cover. 小红书: always 标记 SGLang via 添加组件 → 标记地点或标记朋友 → 用户 → SGLang (official, orange SGL avatar) — never an @ in the body — plus topics #人工智能 #大模型 #SGLang and 笔记含AI合成内容.

Every publish runs the five checks in references/publishing.md §Checks:

  1. publish_check.py on the files and texts before anything is uploaded.
  2. The B站 form before 立即投稿.
  3. The stored B站 archive (简介, tags, 分区, duration, cover, 合集) after submit.
  4. The 小红书 form before 发布 (topics, 标记, AI 声明, the cover actually used).
  5. The 小红书 note list after 发布.

The publish is done only when all five pass. Tell the user what each one showed.

Fixing a published video (a re-voiced line, a re-render) replaces the file in place on both sites; the post, its plays and its comments stay. Follow §Replacing the video of a published post: 小红书 drops the 标记 on every swap.

Changing narration after frames exist

Edit SCRIPT.md (and the storyboard voiceover), re-run Step 4, then — instead of rebuilding frames —

$SEV_PY -I $SEV_SKILL/scripts/retime_frames.py --project $P --ref-meta $P/audio_meta.built.json

re-maps each frame's timeline onto the new take (piecewise-linear, anchors = matched clause starts). Frames are found through STORYBOARD.md - src: (so 19b-*.html works). --chain handles a project whose built take was lost by an earlier retime: the new anchors are composed through each frame's existing WARP. Re-make the cover frame (work/make_cover.py frame <new frame-1 voice s>, with its cues moved to the new take), or let retime warp 00-cover too. Then retranslate changed caption groups, re-subset fonts, re-assemble, and re-render.

Re-voicing a finished video (new voice, same frames) is one command per project:

bash $SEV_SKILL/scripts/revoice.sh $P            # VOICE=zh-CN-XiaoxiaoNeural by default

It runs these steps:

  1. Back up the old state once to revoice_backup/.
  2. Synthesize the new take, through the pronunciation layer.
  3. Restore the frames, then retime them.
  4. Run the pause check, captions, font subset, assemble, lint and check.
  5. Render renders/video_revoiced.mp4. The old renders/video.mp4 is never touched.

revoice.sh $P tts stops after the take. Use it to run pron_check --meta --asr across many projects, fix the lexicon, and only then render. A batch of 13 published videos (132 min) took about 30 min of TTS plus about 2× real time to render.

References

  • references/house-style.md — narration, term glossary, captions, cover, author page, ending links, brief template.
  • references/cover-log.md — what each cover so far looked like; read it before designing one, add a row after.
  • references/pronunciation.md — English terms and Chinese polyphones read right: spoken-form rules, lexicon, pron_check.py, poly_probe.py, hang_take.py, tone_take.py.
  • references/worker-dispatch.md — the dispatch context to paste into every frame worker.
  • references/troubleshooting.md — every failure seen so far and its fix.
  • references/publishing.md — publishing to Bilibili and 小红书 (always 标记 SGLang on 小红书, never @), and the checks every publish must pass (publish_check.py plus what each site stored), and replacing the video of a published post.

SGLang Serving Debug

Use the maintained source contracts when applying historical evidence to a current branch. The review/history corpora retain their own capture dates; they do not certify today's dispatch or numerical defaults. For kernel replacements, verify that the real model executes the candidate before treating end-to-end tests as coverage.

Overview

Use this skill to turn a live serving problem into a debug path you can replay.

Use one loop:

  • collect a baseline bundle
  • save the failing request or crash dump
  • replay on a clean target
  • only then switch tools

Do not start with profiling.

Version-sensitive live-server signals:

  • --enable-session-radix-cache plus /close_session: agent or RL rollouts that forget to close sessions can look like unexplained KV growth
  • current Rust ingress enforces API/admin keys and supports profiling; older Rust builds differ. Confirm the ingress binary and source revision before interpreting a 401 or 404 (see the endpoint reference).

This skill should work with more focused skills instead of re-implementing them:

  • debug-cuda-crash when replay plus coredump points to a CUDA crash path
  • debug-distributed-hang when the problem is clearly a TP/PP/DP/EP hang
  • llm-torch-profiler-analysis when the issue is already narrowed to a compute-side path

Three examples are included:

  • TTFT spike with low queue time
  • replay-first CUDA crash flow
  • request-shaped distributed hang flow

Output Contract

Return:

  • problem class
  • what was checked
  • strongest signal so far
  • current best guess
  • what was ruled out
  • next step
  • production risk

When To Use It

  • /health or /health_generate is unhealthy
  • latency or throughput regressed under serving load
  • queue size grows while health still looks green
  • one request class times out or hangs
  • the server crashes only after some requests
  • outputs changed after a deploy, topology change, or weight switch
  • one older commit is known-good and a newer commit is known-bad

Workflow

1. Collect a baseline bundle

Confirm the ingress and build first. Current Rust route assembly includes load, metrics (when enabled), HiCache and profile controls but no /ready route (it is only exempt from auth), so /ready returns 404 there; Python route parity is not universal. A 404 can mean an older build, an unmounted optional route or an ingress difference. Record status and response body, then consult the actual route module. Health/readiness are different checks.

If a live server is reachable, collect a read-only bundle before anything more intrusive:

python3 scripts/incident_artifact_tool.py collect-bundle \
  --base-url http://127.0.0.1:30000 \
  --outdir /tmp/incident_bundle

python3 scripts/incident_artifact_tool.py summarize-bundle \
  /tmp/incident_bundle

If the server is protected:

python3 scripts/incident_artifact_tool.py collect-bundle \
  --base-url http://127.0.0.1:30000 \
  --token "$SGLANG_BEARER_TOKEN" \
  --outdir /tmp/incident_bundle

The bundle script collects:

  • /ready
  • /health
  • /health_generate
  • /model_info
  • /server_info
  • /v1/loads?include=all
  • /v1/loads?include=core,queues,disagg,spec
  • /metrics
  • /hicache/storage-backend on a best-effort basis

If stale or corrupt HiCache contents are the working diagnosis, capture the bundle and relevant request dumps before calling POST /hicache/storage-backend/clear. That endpoint clears the attached backend's contents without detaching it; treat it as destructive and use it only when losing the cached contents is acceptable. Attach, detach, and clear details are in references/endpoints-and-signals.md.

Use the summary for a quick read on:

  • health vs. active health state
  • topology and runtime flags
  • point-in-time queue and token usage
  • TTFT / E2E / queue-time heuristics from Prometheus metrics

If the summary says the bundle was captured while the server was idle, recollect it during traffic or move quickly to dump plus replay.

If no live server is reachable, start from the best dump or log already available:

  • crash dump
  • request dump
  • logs
  • CUDA coredump
  • OTel trace
  • torch profile

2. Save the failing request

Read references/decision-tree.md only if the problem class is still unclear:

  • server down or unhealthy
  • latency or throughput regression
  • wrong output or behavior regression
  • intermittent timeout or hang

Then preserve the request payload that actually triggers the problem:

  • crash path: use --crash-dump-folder
  • non-crash path: enable request dump or save the exact trigger request

Do not jump straight from a live symptom to low-level debugging without first saving something you can replay.

3. Replay on a clean target

Read references/endpoints-and-signals.md when you need help reading the baseline bundle or the replay target.

Read references/replay-trace-profile.md when you need the replay, trace, profile, or bisect paths.

Standard order:

  1. collect baseline bundle
  2. capture request dump or crash dump
  3. restart a clean debug target if needed
  4. replay the same issue
  5. collect replay-time logs and dumps

4. Only go deeper after replay

Replay

Use replay when:

  • a crash dump exists
  • a request dump exists
  • the problem depends on request shape or workload mix

If a crash dump exists, summarize it first:

python3 scripts/incident_artifact_tool.py summarize-dump \
  --input-file /path/to/crash_dump.pkl

Then replay:

python3 /path/to/sglang/scripts/playground/replay_request_dump.py \
  --input-file /path/to/crash_dump.pkl \
  --host 127.0.0.1 \
  --port 30000 \
  --parallel 128

If safe_pickle_load blocks a locally captured trusted dump, use:

python3 scripts/replay_trusted_request_dump.py \
  --input-file /path/to/request_dump.pkl \
  --host 127.0.0.1 \
  --port 30000 \
  --parallel 1

If replay indicates a CUDA crash path, restart the same build with coredumps enabled before reproducing again:

SGLANG_CUDA_COREDUMP=1 \
SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
python -m sglang.launch_server \
  --model-path ... \
  --crash-dump-folder /tmp/sglang_crash_dump \
  ...

Then inspect the generated coredump:

cuda-gdb "$(which python3)" \
  -ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"

For a replay-first crash example, read references/case-studies.md.

OTel trace

Use tracing when:

  • request-stage timing is unclear
  • router vs. worker attribution is unclear
  • PD prefill/decode transfer may be implicated

If tracing was enabled at startup, you can change the level without restart:

curl "http://127.0.0.1:30000/set_trace_level?level=1"
curl "http://127.0.0.1:30000/set_trace_level?level=2"
Torch profile

Use profiling when:

  • the issue is already narrowed to compute-side ownership
  • replay already reproduces the problem
  • metrics and loads do not explain the regression

At that point, switch to llm-torch-profiler-analysis. Do not duplicate its profiling workflow here.

For a low-noise latency example, read references/case-studies.md.

Distributed hang

If this looks like a collective stall, save the failing request, replay it on a clean target, collect the replay-time bundle and stacks, then switch to debug-distributed-hang.

For an example of that flow, read references/case-studies.md.

Regression between two commits

If one commit is known-good and another is known-bad, build a deterministic harness before doing deeper manual debugging:

  1. choose a stable reproducer: request replay, benchmark command, or correctness check
  2. make the harness return 0 on good behavior and non-zero on bad behavior
  3. run git bisect start <bad> <good>
  4. run git bisect run <harness>
  5. return here only after a candidate commit is isolated

Prefer replay-backed bisect when the regression depends on request shape or long-running serving state.

5. Switch tools when the boundary is clear

Switch tools once the fault class is clear:

  • llm-torch-profiler-analysis for kernel and overlap attribution
  • debug-distributed-hang for collective or rank-divergence hangs
  • debug-cuda-crash for CUDA crash reproduction and kernel API logging

Do not switch tools before collecting the first bundle unless the user already has decisive logs or dumps.

References

Load only what the current step needs:

Scripts

If a live bundle was collected, include its path.

If replay, trace, or profiling was chosen, say why bundle plus dump were not enough.

Current upstream debug controls — 2026-10-05

SGLang-repo companion skills now live under .agents/skills/ (#40515): debug-cuda-crash, debug-distributed-hang, generate-profile, and sglang-bisect-ci-regression. Verify the repository copy before invoking it.

--crash-dump-folder X configures CUDA coredumps when their environment keys are unset, with CUDA_COREDUMP_FILE=X/%h/core.cuda.%t.%p and CUDA_COREDUMP_PIPE=/tmp/corepipe.cuda.%h.%p. Before a crash, py-spy and user-triggered CUDA dumps are enabled by default, with a 60 s coredump wait. SGLANG_CUDA_COREDUMP=1 uses its separate cuda_coredump_%h.%p.%t naming; by import-order code reading its already-set path wins. This precedence was not GPU-tested. A configured pipe can also capture a wedged kernel.

Source: crash dump hook, Rust ingress.

Torch Profiler Layer Track

Create a normal Chrome JSON trace with L0, L1, ... guides beside at most ten synthetic GPU activity lanes per device. Keep the source file. Compaction changes event placement, not execution: original kernel names, timestamps, durations and args.stream remain intact. CPU events stay unchanged. Label the output as a compact view, not a runtime stream optimization.

First read the kernel attribution and timing principles. Fused mHC/AR can remove old anchors; graph-launch host gaps and synthetic lane counts do not establish GPU idle time or runtime stream counts.

For current DS V4.1, fused post/combine/norm and optional Mega mHC prefill can remove separate norm/statistics anchors. Engram and row selection can split that pattern again. Verify the measured phase's cadence against the current boundary dispatch before adding layer labels; never reuse a V4 or old V4.1 anchor count blindly.

Establish what the labels mean

Use the user's existing local trace, including a Desktop copy when requested. Obtain the layer count from matching model config/source. Identify the rank/GPU, execution phase, and one complete forward pass before counting. In speculative decoding, separate target verify from draft passes. BS=1 does not imply one token per target verify. A repeated block in the overview can be a whole iteration, not a transformer layer.

Choose a GPU kernel with a verified cadence per layer in the selected phase. For two sublayer anchors use --anchor-stride 2; LongCat residual boundaries can need stride 4. Use --extra-anchors-per-pass 1 for a verified K3 output aggregation after the final layer. Verify its call site and cross-check the count with another landmark or CPU module scopes. Record this evidence and code revision. Names, stream numbers and layer counts depend on the model/build; do not count all Top-K calls or blindly reuse a previous model's anchor. Inspect all GPU streams in the rank: layers can change streams. CPU scopes mark host launch intervals, not GPU execution boundaries.

The helper does not guess L0 from counts alone. --anchor-offset must identify a verified first layer in timestamp-sorted matching events. Divisibility does not rule out a partial first pass or mixed draft/target anchors. If evidence is insufficient, explain the ambiguity instead of publishing confident labels.

Add the auxiliary track

Scripts require Python 3.10+ and the standard library only. Commands are relative to this skill directory.

python3 scripts/add_layer_track.py \
  --trace /path/to/TP-0.trace.json.gz \
  --output /path/to/TP-0.layers.trace.json \
  --anchor-regex '^YOUR_ONCE_PER_LAYER_KERNEL$' \
  --num-layers 40 --anchor-offset 0 --passes 1 \
  --max-gpu-lanes 10 \
  --phase target-verify \
  --evidence 'Config/source revision ...; first L0 verified; 40 anchors per target pass; draft uses a different path.'
  • Use --pid GPU_PID when anchors occur in multiple processes/GPUs. Use --device DEVICE if a single PID contains anchors from multiple devices.
  • --anchor-stride K keeps every K-th anchor after the verified offset. --extra-anchors-per-pass E skips E trailing output aggregations before selecting the next pass; stride/extra are recorded in the report.
  • --passes labels only that many complete passes. --first-layer supports a known pipeline partition. A truncated selection is rejected.
  • Optional --end-anchor-regex identifies a verified layer-ending kernel occurring once between anchors and closes the final label of each pass. Without it, the last label is a zero-duration marker; L39 is never extended across a draft pass or an inter-iteration gap.
  • .json and .json.gz input/output work. Output must be a new file. A mapping report is saved as <output>.layers.json.
  • Use --max-gpu-lanes 10 by default for a compact view; omit it when the user wants the original stream layout. The helper assigns non-overlapping activity to reusable display lanes and remaps matching GPU flow endpoints. It refuses selections needing more than ten lanes, rather than hiding concurrent work. Other unsupported or ambiguous events also fail with an explanation.
  • Original GPU pid/tid and source event index are retained in args._compact_gpu_track; retired stream-name metadata is in the report. GPU user scopes are preserved in a separate GPU scopes process so they do not recreate hundreds of rows beside the kernel lanes. CPU scopes and flows stay in their original process.

For a trace that already has verified guides beside the GPU, compact it without adding duplicate labels:

python3 scripts/compact_gpu_tracks.py \
  --trace /path/to/TP-0.layers.trace.json \
  --output /path/to/TP-0.compact-layers.trace.json \
  --max-gpu-lanes 10

This saves <output>.compact.json. Existing guides are preserved. If a guide is missing, under the wrong process, or has unverified boundaries, regenerate it from the original trace with add_layer_track.py first. Never treat the number of synthetic lanes as the number of real CUDA streams. The original trace remains necessary for stream scheduling and synchronization analysis.

Every interval is a guide, from this layer's anchor to the next layer's anchor. Work before the anchor is outside its guide. The last interval, when supplied, ends at a terminal kernel's completion. These are not full layer boundaries, exclusive kernel ownership, or per-layer latency measurements. Overlapping streams can cross labels. Numbering is zero-based: L2 is the third transformer layer. Only auxiliary endpoints are rounded to the nearest microsecond to avoid floating-point overlaps; original timestamps and report anchor values stay intact.

Open and inspect

Open the annotated file with Perfetto's Open trace file. Expand the selected GPU/process, locate Layer guide (...) — anchor intervals, and pin it beside the GPU lane N (synthetic; ...) tracks. Use relative times in the report to locate L0, L1, a middle layer and the final layer.

If the browser cannot fetch a local trace URL, use the loopback viewer. It transfers local bytes after Perfetto's PING/PONG handshake:

python3 scripts/serve_local_trace.py --trace /path/to/TP-0.layers.trace.json
# Open the printed http://127.0.0.1:PORT/ in the user's chosen browser.
# macOS example:
open -a Firefox http://127.0.0.1:PORT/

The server exposes only this trace and its viewer until Ctrl-C. It needs network access for the Perfetto UI; it does not upload the trace to storage. Manual file opening works when the UI is locally available. See navigation.md for SQL checks, screenshot guidance, time units, and the DeepSeek-V4.1 example.

Verify and deliver

  1. Check the report's PID, layer count, anchor offset, phase and source hash against the intended trace. Check two passes when claiming the mapping repeats across iterations.
  2. Verify that reversing the recorded placement changes and restoring retired metadata reconstructs the source event array exactly. Without compaction, the source array remains an unchanged prefix. Check the original file hash. In Perfetto, check both kernel count and actual imported track count: every GPU activity lane and layer guide should have depth zero. A raw JSON tid count alone can miss additional overflow tracks created by the importer.
  3. Inspect the guide beside the GPU kernels. If requested, capture the real viewer with readable labels and relevant streams. Preserve CPU tracks/flows when present; labels cannot reconstruct CPU events absent from the capture.
  4. Return the compact trace, report, and screenshot if requested. State the original stream count, synthetic lane count, numbering convention and anchor-boundary limitation. Keep large traces and private capture paths out of public skill/PR artifacts.

Sponsors

Details

Marketplace
BBuf/AI-Infra-Auto-Driven-SKILLS
Category
Ai Infra