marslanalvi avatar

ai-machine-learning

ML training, fine-tuning, MLOps, model serving, PyTorch and data science

version
1.0.0
GitHub stars
5

Install

/plugin install ai-machine-learning@skills-stack

Run it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.

Contents

Discovered from source

Skills 26

Checkpoint Promotion

The Phase 5 gate for the whole plugin: a checkpoint that trains cleanly and beats its task metric still doesn't ship without clearing all four stages below. eval-harness-first built the suite re-run here — this skill is where that suite's baseline decides something.

Input: a trained checkpoint, eval/baseline-<model>.json from eval-harness-first, and the frozen eval/drift-suite.yaml. Output format: promotion-report.md — the four-stage evidence plus a terminal PROMOTE or REJECT verdict that /finetune Phase 5 and /promote-checkpoint consume directly.

The Four-Stage Gate

Each stage gates the next — a failure at stage 2 means stage 3 doesn't run. Stages 2 and 3 share one expensive inference pass, so running them concurrently and applying gate order at verdict time is licensed on a deterministic arena (nothing saved by serializing); a judge-based arena should still wait for stage 2 first — that's where the real savings are.

  1. Data-quality gate. Before any eval touches the checkpoint: dedup the training set, check for eval-goldens leakage (the exact failure trace-to-training-data's Hygiene section exists to prevent), and scan for label noise. A checkpoint trained on leaked goldens invalidates every later stage.
  2. Held-out + frozen capability-drift suite. Re-run eval-harness-first's eval/drift-suite.yaml — MMLU/GSM8K/IFEval plus 200–500 domain-adjacent items — against the checkpoint and diff against baseline-<model>.json per benchmark against the Drift Budget table below.
  3. Paired arena vs. base. Position-randomized judge, checkpoint vs. base model, same prompts — or the deterministic paired-comparison variant in references/gate-templates.md when every grader in the harness is deterministic (no LLM-judge; position randomization N/A there). A holdout win that loses the live arena does not ship — stage-2 numbers and stage-3 judgments must agree; a win on frozen goldens and a loss in paired comparison is a real signal, not a discrepancy to explain away.
  4. Canary. 5–10% stratified rollout with auto-rollback for any checkpoint reaching production traffic. Local-only users stop at stage 3 — skipping stage 4 for a local deployment is the correct stopping point, not a shortcut.

Drift Budget

Drift (pts)Verdict
≤1Noise — proceed
2–5Rerun with seed variation before deciding
>5HARD FAIL — no exception for task gains

The >5pt row governs regardless of the others: a checkpoint that gained 8 points on the target task and lost 6 points of general capability still fails here — task improvement never buys back a drift-budget breach.

Item count derives from the budget, not convenience: the strict n for a half-width under half the 5pt hard-fail threshold is ~1,300 at typical accuracy (p≈0.7); n=200 is a pragmatic floor (±6pt half-width at that same p, n=50 ±13pt) — report the half-width with every verdict, and treat a margin smaller than it as REJECT (uncertain), not PASS/HARD FAIL. Full math and a 5-run cautionary example: references/gate-templates.md.

RERUN is not a verdict. A 2–5pt drift only ever produces a PROMOTE or REJECT after the seed-variation rerun completes — PROMOTE requires landing back at ≤1pt (noise); any rerun still

1pt — 2–5pt band or >5pt breach alike — resolves stage 2 to a hard REJECT. No report may reach the Verdict section with stage 2 still showing RERUN.

Catastrophic Forgetting

Unmanaged LoRA fine-tuning loses real general capability, and stage 2 is what catches it:

  • ~43% knowledge loss unmanaged — no replay, no regularization.
  • ~10% with basic management — some replay or a conservative LR.
  • ~3% with replay + EWC — the disciplined case.
  • 10–30% general-data replay mix is the standard mitigation — blend general- domain data into training rather than target-task data alone.

If a checkpoint hits the >5pt hard fail in stage 2, work this escalation ladder in order — the one canonical order this skill and references/gate-templates.md both point to:

  1. Adjust the replay-mix fraction — swap rows, don't add them (adding confounds fraction with total optimizer steps). Dose is not monotonic at small-run scale (<~100 steps) — re-check drift after any swap.
  2. Lower the learning rate.
  3. Fewer epochs.
  4. A smaller LoRA rank — the same rank/LR levers lora-qlora-recipes and preference-optimization tune for the training run, applied here in reverse.

This order is a default, not a law: remediation guidance from a single before/after run pair is a hypothesis — label it low-confidence once any lever produces a reversal, and prefer a seed-variation repeat over trusting the next rung blindly. A lever that clears the drift breach but drops a success-criterion metric below target is a two-sided tradeoff for a human, not a reason to keep descending the ladder. Full reasoning and the 5-run trajectory behind both caveats: references/gate-templates.md.

Disclose drift-suite instruction reuse. A replay row copying the drift harness's exact instruction phrasing (not just disjoint source items) makes that benchmark's post-replay score an upper bound — flag it instruction-familiar, or re-probe with a paraphrase, before treating a near-budget pass as clean.

The Verdict

promotion-report.md covers all four stages as sections and must end with a terminal verdict: PROMOTE or REJECT, the evidence that produced it, and exactly one top remediation when the verdict is REJECT. Template: references/gate-templates.md. The terminal contract other skills parse:

## Verdict

REJECT

Evidence: domain-adjacent drift
suite dropped 6.2pt (threshold:
>5pt hard fail) despite +8pt on
the target task.

Top remediation: swap the
replay-mix fraction from 10%
toward 20%, holding step count
constant.
  • REJECT is a result, not an error. A checkpoint that fails stage 2's drift budget or stage 3's arena comparison did its job. Don't treat a REJECT as a failed run needing a rerun of this skill; it's the correct output of a working gate.
  • One remediation, not a menu. Evidence sections may list everything observed; the verdict section names the single highest-leverage fix per the escalation ladder above. A report that hedges across three possible fixes hasn't done the prioritization this skill exists to do.
  • No auto-retraining. This skill produces a verdict and a report, not a re-triggered training run. A REJECT hands the remediation back to a human decision at finetuning-method-selection or the relevant training skill.

Related Skills

  • eval-harness-first — owns the drift suite and baseline this skill re-runs and diffs against; no baseline-<model>.json means nothing to gate against.
  • quantized-export — the only valid next step after a PROMOTE verdict.
  • preference-optimization and lora-qlora-recipes — own the LR and rank levers in the Catastrophic Forgetting escalation path; this skill diagnoses the breach, those skills own the config that caused it.
  • dataset-curation — owns the replay-mix construction recipe the escalation ladder's first rung applies.

Complete promotion-report.md template with all four stages, the drift-suite scoring table, the paired-arena protocol (item count, position randomization, win-rate threshold), and a replay-mix configuration example: references/gate-templates.md.

Use this skill when

  • Working on data scientist tasks or workflows
  • Needing guidance, best practices, or checklists for data scientist

Do not use this skill when

  • The task is unrelated to data scientist
  • You need a different domain or tool outside this scope

Instructions

  • Clarify goals, constraints, and required inputs.
  • Apply relevant best practices and validate outcomes.
  • Provide actionable steps and verification.

You are a data scientist specializing in advanced analytics, machine learning, statistical modeling, and data-driven business insights.

Purpose

Expert data scientist combining strong statistical foundations with modern machine learning techniques and business acumen. Masters the complete data science workflow from exploratory data analysis to production model deployment, with deep expertise in statistical methods, ML algorithms, and data visualization for actionable business insights.

Capabilities

Statistical Analysis & Methodology

  • Descriptive statistics, inferential statistics, and hypothesis testing
  • Experimental design: A/B testing, multivariate testing, randomized controlled trials
  • Causal inference: natural experiments, difference-in-differences, instrumental variables
  • Time series analysis: ARIMA, Prophet, seasonal decomposition, forecasting
  • Survival analysis and duration modeling for customer lifecycle analysis
  • Bayesian statistics and probabilistic modeling with PyMC3, Stan
  • Statistical significance testing, p-values, confidence intervals, effect sizes
  • Power analysis and sample size determination for experiments

Machine Learning & Predictive Modeling

  • Supervised learning: linear/logistic regression, decision trees, random forests, XGBoost, LightGBM
  • Unsupervised learning: clustering (K-means, hierarchical, DBSCAN), PCA, t-SNE, UMAP
  • Deep learning: neural networks, CNNs, RNNs, LSTMs, transformers with PyTorch/TensorFlow
  • Ensemble methods: bagging, boosting, stacking, voting classifiers
  • Model selection and hyperparameter tuning with cross-validation and Optuna
  • Feature engineering: selection, extraction, transformation, encoding categorical variables
  • Dimensionality reduction and feature importance analysis
  • Model interpretability: SHAP, LIME, feature attribution, partial dependence plots

Data Analysis & Exploration

  • Exploratory data analysis (EDA) with statistical summaries and visualizations
  • Data profiling: missing values, outliers, distributions, correlations
  • Univariate and multivariate analysis techniques
  • Cohort analysis and customer segmentation
  • Market basket analysis and association rule mining
  • Anomaly detection and fraud detection algorithms
  • Root cause analysis using statistical and ML approaches
  • Data storytelling and narrative building from analysis results

Programming & Data Manipulation

  • Python ecosystem: pandas, NumPy, scikit-learn, SciPy, statsmodels
  • R programming: dplyr, ggplot2, caret, tidymodels, shiny for statistical analysis
  • SQL for data extraction and analysis: window functions, CTEs, advanced joins
  • Big data processing: PySpark, Dask for distributed computing
  • Data wrangling: cleaning, transformation, merging, reshaping large datasets
  • Database interactions: PostgreSQL, MySQL, BigQuery, Snowflake, MongoDB
  • Version control and reproducible analysis with Git, Jupyter notebooks
  • Cloud platforms: AWS SageMaker, Azure ML, GCP Vertex AI

Data Visualization & Communication

  • Advanced plotting with matplotlib, seaborn, plotly, altair
  • Interactive dashboards with Streamlit, Dash, Shiny, Tableau, Power BI
  • Business intelligence visualization best practices
  • Statistical graphics: distribution plots, correlation matrices, regression diagnostics
  • Geographic data visualization and mapping with folium, geopandas
  • Real-time monitoring dashboards for model performance
  • Executive reporting and stakeholder communication
  • Data storytelling techniques for non-technical audiences

Business Analytics & Domain Applications

Marketing Analytics
  • Customer lifetime value (CLV) modeling and prediction
  • Attribution modeling: first-touch, last-touch, multi-touch attribution
  • Marketing mix modeling (MMM) for budget optimization
  • Campaign effectiveness measurement and incrementality testing
  • Customer segmentation and persona development
  • Recommendation systems for personalization
  • Churn prediction and retention modeling
  • Price elasticity and demand forecasting
Financial Analytics
  • Credit risk modeling and scoring algorithms
  • Portfolio optimization and risk management
  • Fraud detection and anomaly monitoring systems
  • Algorithmic trading strategy development
  • Financial time series analysis and volatility modeling
  • Stress testing and scenario analysis
  • Regulatory compliance analytics (Basel, GDPR, etc.)
  • Market research and competitive intelligence analysis
Operations Analytics
  • Supply chain optimization and demand planning
  • Inventory management and safety stock optimization
  • Quality control and process improvement using statistical methods
  • Predictive maintenance and equipment failure prediction
  • Resource allocation and capacity planning models
  • Network analysis and optimization problems
  • Simulation modeling for operational scenarios
  • Performance measurement and KPI development

Advanced Analytics & Specialized Techniques

  • Natural language processing: sentiment analysis, topic modeling, text classification
  • Computer vision: image classification, object detection, OCR applications
  • Graph analytics: network analysis, community detection, centrality measures
  • Reinforcement learning for optimization and decision making
  • Multi-armed bandits for online experimentation
  • Causal machine learning and uplift modeling
  • Synthetic data generation using GANs and VAEs
  • Federated learning for distributed model training

Model Deployment & Productionization

  • Model serialization and versioning with MLflow, DVC
  • REST API development for model serving with Flask, FastAPI
  • Batch prediction pipelines and real-time inference systems
  • Model monitoring: drift detection, performance degradation alerts
  • A/B testing frameworks for model comparison in production
  • Containerization with Docker for model deployment
  • Cloud deployment: AWS Lambda, Azure Functions, GCP Cloud Run
  • Model governance and compliance documentation

Data Engineering for Analytics

  • ETL/ELT pipeline development for analytics workflows
  • Data pipeline orchestration with Apache Airflow, Prefect
  • Feature stores for ML feature management and serving
  • Data quality monitoring and validation frameworks
  • Real-time data processing with Kafka, streaming analytics
  • Data warehouse design for analytics use cases
  • Data catalog and metadata management for discoverability
  • Performance optimization for analytical queries

Experimental Design & Measurement

  • Randomized controlled trials and quasi-experimental designs
  • Stratified randomization and block randomization techniques
  • Power analysis and minimum detectable effect calculations
  • Multiple hypothesis testing and false discovery rate control
  • Sequential testing and early stopping rules
  • Matched pairs analysis and propensity score matching
  • Difference-in-differences and synthetic control methods
  • Treatment effect heterogeneity and subgroup analysis

Behavioral Traits

  • Approaches problems with scientific rigor and statistical thinking
  • Balances statistical significance with practical business significance
  • Communicates complex analyses clearly to non-technical stakeholders
  • Validates assumptions and tests model robustness thoroughly
  • Focuses on actionable insights rather than just technical accuracy
  • Considers ethical implications and potential biases in analysis
  • Iterates quickly between hypotheses and data-driven validation
  • Documents methodology and ensures reproducible analysis
  • Stays current with statistical methods and ML advances
  • Collaborates effectively with business stakeholders and technical teams

Knowledge Base

  • Statistical theory and mathematical foundations of ML algorithms
  • Business domain knowledge across marketing, finance, and operations
  • Modern data science tools and their appropriate use cases
  • Experimental design principles and causal inference methods
  • Data visualization best practices for different audience types
  • Model evaluation metrics and their business interpretations
  • Cloud analytics platforms and their capabilities
  • Data ethics, bias detection, and fairness in ML
  • Storytelling techniques for data-driven presentations
  • Current trends in data science and analytics methodologies

Reproducible analysis procedure

For churn prediction, first define the prediction date, outcome window and unit of analysis. Obtain authorized data and record its origin, date range, exclusions, units and missingness. Split customers/time before fitting preprocessing so future outcomes and repeated entities cannot leak across train and evaluation sets. Compare a simple baseline with the candidate model on an untouched time-appropriate holdout; report prevalence, sample size, calibration and uncertainty alongside business-relevant error costs. For observational campaign data, state confounders and identification assumptions rather than calling correlation causal impact. Preserve code/data versions and seeds, redact personal records and distinguish completed calculations from proposed experiments. Capability lists here do not certify regulatory compliance or production readiness.

Response Approach

  1. Understand business context and define clear analytical objectives
  2. Explore data thoroughly with statistical summaries and visualizations
  3. Apply appropriate methods based on data characteristics and business goals
  4. Validate results rigorously through statistical testing and cross-validation
  5. Communicate findings clearly with visualizations and actionable recommendations
  6. Consider practical constraints like data quality, timeline, and resources
  7. Plan for implementation including monitoring and maintenance requirements
  8. Document methodology for reproducibility and knowledge sharing

Example Interactions

  • "Analyze customer churn patterns and build a predictive model to identify at-risk customers"
  • "Design and analyze A/B test results for a new website feature with proper statistical testing"
  • "Perform market basket analysis to identify cross-selling opportunities in retail data"
  • "Build a demand forecasting model using time series analysis for inventory planning"
  • "Analyze the causal impact of marketing campaigns on customer acquisition"
  • "Create customer segmentation using clustering techniques and business metrics"
  • "Develop a recommendation system for e-commerce product suggestions"
  • "Investigate anomalies in financial transactions and build fraud detection models"

Limitations

  • Use this skill only when the task clearly matches the scope described above.
  • Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
  • Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.

Dataset Curation

This skill assumes finetuning-method-selection already routed here — the next step is preparing data, not choosing a method. What follows: format selection by target method, the template/packing mechanics behind the most common silent training failures, rules for mixing in synthetic data without collapse, and the dataset card that closes out Phase 2 before a run starts.

Input: raw examples (demonstrations, preference judgments, or task prompts) plus a routing decision from finetuning-method-selection. Output format: a formatted, packed, validated JSONL dataset plus a completed dataset card — the Phase 2 artifact /finetune checks before launching training.

Format Selection

MethodShapeRows
SFT, single-turnInstruct (instruction/response or prompt/completion)~1,000+ floor
SFT, multi-turnConversation / ChatML messages list~1,000+ floor
DPO / ORPOPreference pair (prompt, chosen, rejected)Method-dependent, see preference-optimization
KTOUnpaired (prompt, completion, label)Method-dependent, see preference-optimization
GRPO / RLVRPrompt-only (prompt + verifier metadata)Method-dependent, see grpo-rlvr-training
  • ~1,000+ rows is the recommended floor for SFT, not a target. Below it, a handful of low-quality or duplicate examples can dominate the gradient; above it, quality over quantity — a smaller verified, deduplicated set beats a larger noisy one.

  • The ChatML shape, for orientation; the other four formats plus a ShareGPT conversion note live in references/formats-and-templates.md:

    {"messages": [
      {"role": "user", "content": "..."},
      {"role": "assistant", "content": "..."}
    ]}
    

Chat Templates and Loss Masking

Apply the target model's chat template before any concatenation or packing, never after — packing raw text and templating the packed blob afterward corrupts turn boundaries, landing role markers in the wrong place relative to each example.

  • Train on assistant responses only. Mask the loss (-100 in the labels tensor) over system/user turns and the template's own role markers — only assistant-turn content tokens contribute to loss.

  • Template/tokenizer mismatches are a top silent failure mode. A model trained against one chat template but served or evaluated with a different one degrades without erroring. Verify the same template string used in training is applied at inference and eval time.

  • Keep the dataset in messages shape and let the trainer template and mask it (assistant_only_loss=True in current TRL) — pre-rendering to a flat text field destroys the turn boundaries masking needs. Full code sketch: references/formats-and-templates.md. Sanity-check before training — decode only unmasked positions; expect only assistant text:

    keep = batch["labels"][0] != -100
    print(tokenizer.decode(batch["input_ids"][0][keep]))
    

Packing

Without packing, 40–70% of compute is spent on padding — variable-length examples batched at a fixed sequence length waste the gap between each example's length and the batch's max. Packing concatenates multiple examples into one sequence up to the max length, cutting most of that waste.

  • Packing changes batch semantics. A packed sequence can contain several original examples, so "steps per epoch" and any LR schedule keyed to example count shift once packing is on — recompute schedule milestones against packed-sequence count.

  • MANDATORY: decode and manually inspect 5–10 packed sequences before scaling to a full run. Confirm example boundaries land where expected, template markers are intact per sub-example, and the loss mask is still assistant-only within each packed sequence. Not optional — packing bugs are silent (the loss curve looks normal) and only surface in eval quality, hours later:

    for seq in packed_dataset.select(range(10)):
        print(tokenizer.decode(seq["input_ids"]))
    

Synthetic Data Rules

  • Keep ≥25% real data as a collapse guard. Training on a growing share of model-generated data without a real-data floor drives measurable quality collapse over successive generations — 25% real is the minimum that holds the line. General-domain replay rows count toward this floor — "real" means "not generated for this task from this student," not "human-authored." An all-synthetic-by-construction dataset can meet the ≥25% floor through replay alone (see references/synthetic-data.md's Replay-Mix Construction recipe); state which rows count as "real" in the dataset card rather than leaving the floor structurally unmeetable.
  • Magpie and rejection sampling are the workhorses. Magpie extracts prompts from the model's own template prior; rejection sampling generates several candidates per prompt and keeps only the ones a filter passes. Both beat naive single-shot generation.
  • Targeted, student-aware generation beats static generation by 1.3–2x sample efficiency — aiming at the student's actual failure modes hits a quality bar with fewer filtered examples.
  • Typical accept rates after filtering run 10–30%. Plan volume accordingly — a 10,000-row target at 15% accept needs ~65,000+ raw generations.
  • Generation-method ranking, filter funnel, replay- mix construction, and distillation pattern: references/synthetic-data.md.

The Dataset Card

Every dataset that reaches training gets a card — the required Phase 2 artifact /finetune checks before launching. The card is not free-form documentation; it MUST carry these fields:

  • Provenance — where every row came from (real source(s), synthetic method(s), or both), traceable to trace-to-training-data output.
  • Counts — total rows, and rows per split (train/eval/held-out) if split.
  • Synthetic/real ratio — the measured ratio, checked against the ≥25% real floor above.
  • Dedup method — exact-match, semantic (embedding threshold), or both; see the filter funnel in references/synthetic-data.md.
  • Template used — the exact chat template string/identifier, kept consistent through inference and eval — this is what ties an eval-harness-first run back to the checkpoint.
  • Packing config — whether packing was used, max sequence length, and confirmation the 5–10-sequence manual inspection above was done.

A dataset missing any of these six fields isn't ready for /finetune — the card is a gate, not a summary written after the fact.

Phase 2 Exit Checklist

Before handing off to /finetune, confirm:

  1. Format matches the method (table above).
  2. Template applied before concatenation.
  3. Loss masked to assistant turns only.
  4. 5–10 packed sequences decoded and read.
  5. ≥25% real data in the final mix.
  6. Dataset card complete — all six fields.

References

  • references/formats-and-templates.md — JSONL examples per format, current-TRL masking code, and the ShareGPT conversion note.
  • references/synthetic-data.md — generation-method ranking, filter funnel, replay-mix construction, and teacher→student distillation pattern.

Related skills: finetuning-method-selection routes here; lora-qlora-recipes, vision-sft, and preference-optimization consume the datasets this skill produces; trace-to-training-data is the provenance source for graded-trajectory datasets; eval-harness-first grades the resulting checkpoint.

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

  1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.
  2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
  3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
  4. Prioritize by frequency × severity × value.
  5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.
  6. Train.
  7. Re-run the same harness on the checkpoint — not a different, looser one.
  8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.

Building Goldens

  • From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.
  • Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.
  • Goldens are versioned like code — commit eval/goldens.jsonl, diff it in review, tag it per release. It doubles as the CI regression suite.

Graders

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.

  • Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.
  • LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.
  • Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.
  • Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: references/grader-templates.md.

Judge Calibration Is a Prerequisite

Any bucket routed to an LLM-judge needs calibration before its verdicts count for anything beyond exploration — a hard prerequisite, not a nice-to-have. N/A when no bucket routes to a judge — an all-deterministic harness has nothing to calibrate; state that rather than leaving this section unaddressed.

  • Label ≥100 items, split train/dev/sealed test (report once, no re-touching after).
  • Report TPR and TNR, not one blended accuracy number — a judge can hit 90% by always saying "pass" on a skewed set.
  • Pin the judge to a fixed model snapshot and recalibrate on judge-model change, quarterly regardless.
  • The judge must come from a different model family than the model under test.
  • A judge that misses the agreed TPR/TNR bar ships advisory-only — flags for human review, never gates a promotion. Full protocol, bias correction, and recalibration checklist: references/judge-calibration.md.

The Baseline

Before Phase 1 (method selection) starts, run the full harness — goldens plus the capability-drift suite — against the unmodified base model. This is the number every later checkpoint gets compared against.

eval/baseline-<model>.json is the gate token. No baseline file, no comparison basis for checkpoint-promotion — a checkpoint that "looks better" against nothing measured isn't a finding.

Directory Contract

eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint

eval/ persists across runs and lives outside runs/ — the fixed measuring stick, not a run artifact. runs/ is disposable; eval/ is not. Never let a run script write into eval/. Canonical location: every per-trace results.json — the Phase 0 baseline included — lives at runs/<run-id>/results.json, never under eval/runs/...; an instruction requesting the latter is wrong, not this contract.

Phase 0 Exit Checklist

Before finetuning-method-selection, confirm:

  1. ≥100 traces open-coded; 4–8 failure buckets (N/A floor for synthetic goldens on a single-failure- surface task — see the Building Goldens exception; bucket count then comes from post-baseline error analysis instead).
  2. eval/goldens.jsonl committed and versioned.
  3. One grader per bucket, deterministic first.
  4. Judges calibrated — TPR/TNR, snapshot pinned, different family (N/A when no bucket routes to an LLM-judge; state that explicitly).
  5. eval/drift-suite.yaml frozen.
  6. eval/baseline-<model>.json written.

Missing any of the six (or its stated N/A)? Not Phase 0 complete — /finetune checks the baseline file before a run.

Related Skills

General-purpose evaluation guidance (dashboards, A/B testing, non-fine-tuning harnesses) lives in the llm-application-dev plugin's llm-evaluation skill — this skill covers only the fine-tuning coupling: goldens that double as training data, and the baseline that gates a checkpoint.

  • finetuning-method-selection — routes here first.
  • dataset-curation — formats these traces into training rows.
  • trace-to-training-data — turns graded traces into training examples.
  • checkpoint-promotion — consumes baseline-<model>.json, re-runs this harness on each candidate checkpoint.

References

  • references/grader-templates.md — runnable grader examples per shape, plus a drift-suite.yaml example and MMLU logprob-scoring note.
  • references/judge-calibration.md — the calibration protocol, including the all- deterministic N/A path.

Fine-Tuning Method Selection

This is the router skill for the fine-tuning lifecycle: it decides whether fine-tuning is the right tool at all, and if so, which method and which base-model size class. Every other skill in this plugin assumes this routing already happened — start here before opening lora-qlora-recipes, preference-optimization, or grpo-rlvr-training.

When to Use This Skill

  • Starting any fine-tuning effort, before a framework or base model has been chosen.
  • Unsure whether RAG or prompt engineering would solve the problem more cheaply than training.
  • Choosing between preference optimization (DPO family) and a reinforcement method (GRPO/RLVR) for the same underlying task.
  • Sizing a candidate model/method combination before committing to a run.

Quick Reference

SituationRoute
Facts change often (prices, docs, news)RAG, not fine-tuning
Desired behavior still being figured outPrompt engineering
Stable domain knowledge, ≥500MB textCPT then SFT — see Off-Ramps First
Have input/output demonstrationsSFT — see lora-qlora-recipes
Have preference pairs or thumbs-up/downDPO/ORPO/KTO — see preference-optimization
Have a verifiable pass/fail signalGRPO+RLVR — see grpo-rlvr-training
No eval harness yetStop — see eval-harness-first

Off-Ramps First

Most requests that sound like "fine-tune this" are served better and cheaper elsewhere. Check these off-ramps before opening a training run:

  • Knowledge-bound and volatile (the gap is facts that change — prices, docs, current events): route to RAG, not fine-tuning. A fine-tuned model bakes in a snapshot; volatile facts go stale immediately.
  • Behavior-bound and shifting (the desired behavior is still being figured out, or changes per request): route to prompt engineering. Fine-tuning locks in a behavior; don't lock in one that hasn't stabilized yet.
  • Stable, dense domain knowledge: this is where continued pretraining (CPT) enters, sized by how much domain text exists:
Domain text volumeRoute
<10MBRAG only
10MB–500MBRAG + fine-tune
500MB–10GBCPT, then SFT
>10GBCPT required

CPT learning rate ≈ 10% of the pretraining LR. CPT is guidance-only in this plugin — sizing and LR guidance live here, but this plugin does not execute a CPT run.

Method Router

Once the off-ramps are ruled out, this is the full decision tree (verbatim from the research this plugin is built on):

New FACTS?  volatile → RAG | stable+dense → CPT (LR ~10% of pretrain) → SFT
New BEHAVIOR? shifting → prompt-engineering | stable:
  demos → SFT (LoRA/QLoRA, all-linear, α=2r)
  preference pairs → DPO (SimPO if length-bias, ORPO if memory-bound)
  unpaired 👍/👎 → KTO
  verifiable success → RLVR + GRPO (DAPO/GSPO/Dr.GRPO per failure mode)
Deploy: FP8 (Hopper+) | NVFP4 (Blackwell scale) | AWQ (older) | GGUF+imatrix (edge)
BEFORE ANY OF THIS: the eval harness must exist first.

Read the tree top-down: answer "new facts or new behavior," then follow the branch that matches the data shape in hand (demos, preference pairs, thumbs up/down, or verifiable success/failure). The data shape picks the method — not the other way around.

Worked Routing Examples

  • "Users want the assistant to follow our support macros exactly." Behavior is stable and demonstrable from transcripts → demos → SFT.
  • "We have pairs of good/bad responses from reviewer thumbs-up/down, unpaired." → unpaired signal → KTO, not DPO (DPO needs paired preferences).
  • "The model can already solve some of these math problems and we can grade correctness automatically." → verifiable success signal → GRPO+RLVR, and only after confirming the model succeeds at least sometimes (see Key Routing Facts below).
  • "We want the model to know this week's pricing page." → volatile facts → RAG, no training run at all.

Key Routing Facts

  • Loss-function choice is low-leverage. A 240-H100-run study found method choice worth ~1 percentage point versus ~50 points for model scale, and zero of 20 DPO variants beat vanilla DPO. Don't spend a routing decision agonizing over DPO-variant selection — spend it on getting the data shape and scale right.
  • DPO is for taste, GRPO+RLVR is for reasoning. Preference pairs that encode a subjective judgment (tone, style, "which answer is better") route to DPO. Tasks with a verifiable pass/fail signal (math, code, tool calls) route to GRPO+RLVR instead.
  • RL is not the fix for a model that never succeeds. GRPO and other RL methods sharpen an existing capability — they don't teach one from zero. If the model doesn't yet understand the task or output format, run SFT first; only bring in RL once the model succeeds at least sometimes.

Common Routing Mistakes

  • Reaching for fine-tuning to fix facts that change weekly — that's a RAG problem, and fine-tuning will just go stale faster than the source data does.
  • Picking a DPO variant before checking whether the actual bottleneck is data quality or model scale — variant choice is the ~1pp lever, not the ~50pp one.
  • Starting an RL run on a model that fails every rollout — route to SFT first so RL has something to sharpen.
  • Treating CPT as the default for "the model doesn't know our domain" — check the data volume thresholds first; under 500MB, RAG or RAG+fine-tune iterates faster than a CPT run.

Model Selection

Base-model choice is size-class first, family second, and it goes stale fast — so it lives in exactly one place: references/model-catalog.md. That file is the only place in this plugin (and in the DGX Spark ops plugin) that names a base model family. Neither this skill nor references/memory-math.md names one; both describe models by size class only (for example, "8B-class LoRA," not a model name).

The catalog is dated on purpose — model rankings turn over quarterly. It carries a "last verified" date and a refresh checklist. Before trusting a row, check that date; if stale, work the refresh checklist in the catalog before recommending a model from it.

Precedence when the catalog and a method skill disagree: the catalog's per-row Notes column states hardware/size-class feasibility, not a method recommendation — lora-qlora-recipes's LoRA vs QLoRA vs Full FT table (routed by task shape) governs the actual method choice.

Memory Feasibility

Before committing to a method, size it: total memory ≈ params × dtype bytes + optimizer state + gradients + activations. Work each term for the chosen dtype and method (full fine-tune, LoRA, or QLoRA) — worked worksheets and size-class examples live in references/memory-math.md.

On DGX Spark specifically, unified-memory behavior breaks the naive estimate (transient load peaks, nvidia-smi underreporting, thermal throttling on long runs). Once the dgx-spark-ops plugin is installed, defer Spark-specific feasibility calls to its spark-memory-thermal-ops skill rather than re-deriving them here.

Related Skills

Once this skill has picked a method, hand off to the skill that executes it:

  • lora-qlora-recipes — SFT via LoRA/QLoRA
  • preference-optimization — DPO, ORPO, KTO
  • grpo-rlvr-training — GRPO with verifiable rewards

No method is selected before the eval harness exists — see eval-harness-first.

GRPO & RLVR Training

This skill assumes finetuning-method-selection already routed here because the target behavior has a verifiable pass/fail signal — not demonstrations (lora-qlora-recipes) or preference pairs (preference-optimization). What follows is when RL is the right tool, the reference recipe, the mandatory reward-inspection gate, and how to pick a GRPO variant when the base recipe misbehaves.

Input: a routing decision (RLVR via GRPO) plus a verifier (code executor, test suite, schema checker, or grader) for the target task. Output format: a validated GRPO config — the kwarg values in references/grpo-memory.md and the reward functions in references/reward-functions.md, not free-form advice — that llm-finetuning-training-engineer consumes directly.

When RL Applies

GRPO+RLVR only pays off when task success is algorithmically checkable — a unit test passes, a parser accepts the output, a tool call matches an expected schema, a math answer matches a ground truth. If grading the output requires human judgment or a subjective rubric, that's an eval-harness and judge-calibration problem first — see eval-harness-first — not a reason to skip straight to RL.

Before opening a GRPO run, confirm the model can sometimes succeed on the target task already. RL sharpens an existing capability by reweighting toward the samples that already work; it does not install a capability from zero.

  • The model never succeeds, even at low temperature across many samples: the gap is format or task understanding, not policy refinement. Route back to SFT first (lora-qlora-recipes) and only return to this skill once the base success rate is nonzero.
  • The model succeeds sometimes, inconsistently: this is the GRPO sweet spot — proceed to The Recipe below.

The standing rule for the whole plugin: DPO for taste, GRPO for reasoning. If the signal is a preference between two acceptable outputs, that's preference-optimization, not this skill.

The Recipe

The reference recipe is TRL's GRPOTrainer with vLLM-backed generation:

from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # single GPU; "server" for multi-GPU
    num_generations=8,          # floor — fewer starves the group-relative baseline
    learning_rate=5e-7,         # settled range for GRPO
    beta=0.01,                  # KL coefficient vs the reference policy
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # references/reward-functions.md
    train_dataset=prompts,       # prompt-only — GRPO generates its own completions
    processing_class=tokenizer,
)

trainer.train()
  • vllm_mode="colocate" runs generation and training on the same GPU — the default for a single-GPU box.
  • vllm_mode="server" points at a separate vLLM server process and is the multi-GPU path — generation and training don't compete for the same device.
  • num_generations ≥ 8 is a floor, not a suggestion: GRPO's advantage estimate is relative to the group mean, and fewer than 8 samples per prompt produces a noisy baseline.
  • Reward is composite — a format reward (did the output parse / match the required structure) plus a correctness reward (did the answer verify). A well-formed-but-wrong answer and a malformed one should not score identically; correctness alone loses that signal.
  • learning_rate=5e-7 and beta=0.01 are the settled starting point; deviate only after the base run is stable and reward-inspected (below).

Memory sizing for this recipe by target size class: references/grpo-memory.md.

The Inspection Rule

Run the reward function against 50–100 sampled outputs and manually read the results before starting the actual training run. This is a gate, not a one-time sanity check.

If the reward function's judgment disagrees with a human reading of that sample, fix the reward function first. Training against an uninspected reward, or tuning hyperparameters to compensate for one silently scoring the wrong thing, is how a run reward-hacks: the model optimizes cleanly toward the wrong target, and that doesn't surface as a training-loop bug.

This inspection is a Phase 1 gate input for /finetune — the same 50–100-sample read that catches a broken reward function here is what that command checks for before it lets a GRPO brief proceed.

Complete reward function implementations to inspect against — exact-match, schema-validation, unit-test-execution, a length-penalty wrapper, and a rubric-as-reward judge pattern: references/reward-functions.md.

Variant Selection

The base recipe above is the default. Reach for a variant only when a specific failure mode shows up, not preemptively:

Failure modeVariantWhy
Entropy collapse / degenerate long chain-of-thoughtDAPODecouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces
Reward or output length trends up regardless of qualityDr.GRPORemoves GRPO's length-normalization bias so reward tracks correctness, not completion length
Training a mixture-of-experts modelGSPOMoves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional

Start with plain GRPO. Watch for the specific symptom — collapsing entropy on long CoT, a length-reward correlation, or MoE instability — and only then swap in the matching variant above. Don't pre-select a variant before the base recipe has actually shown the failure mode.

VLM RL Is Reference-Only

Vision-language RL is not executed by this plugin in v1 — it's documented here for context, not as a runnable path. Tooling is fragmented across ms-swift and EasyR1-derived forks with no one-line TRL command yet, and naive text-only GRPO applied to a VLM tends to reward-hack by optimizing the text-reasoning trace while ignoring the image — the model learns to sound right without looking at the input. A VLM RL run is a research spike outside this skill's supported recipe, not a variant of The Recipe above.

References

  • references/reward-functions.md — complete Python reward functions (exact-match correctness, schema validation, unit-test execution, a length-penalty wrapper, and a rubric-as-reward judge pattern) to inspect under The Inspection Rule before any training run.
  • references/grpo-memory.md — memory sizing by target size class, vLLM sleep-mode and optimizer-state tactics, Unsloth's long-context RL chunking, and the DGX Spark bandwidth caveat for decode-heavy rollouts.

Related skills: finetuning-method-selection routes here once a verifiable pass/fail signal exists; preference-optimization is the sibling skill for preference pairs rather than verifiable rewards; eval-harness-first covers judge calibration for any reward that isn't purely code-checkable. On DGX Spark, defer to the dgx-spark-ops plugin's skills, when installed, for the memory/thermal remediation ladder this skill's memory table doesn't cover.

When to Use

  • Use when this upstream workflow matches the user's stated goal.
  • Use when the task requires the procedures documented in this skill.

Hugging Face Dataset Viewer

Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction.

Core workflow

  1. Optionally validate dataset availability with /is-valid.
  2. Resolve config + split with /splits.
  3. Preview with /first-rows.
  4. Paginate content with /rows using offset and length (max 100).
  5. Use /search for text matching and /filter for row predicates.
  6. Retrieve parquet links via /parquet and totals/metadata via /size and /statistics.

Defaults

  • Base URL: https://datasets-server.huggingface.co
  • Default API method: GET
  • Query params should be URL-encoded.
  • offset is 0-based.
  • length max is usually 100 for row-like endpoints.
  • Gated/private datasets require Authorization: Bearer <HF_TOKEN>.

Dataset Viewer

  • Validate dataset: /is-valid?dataset=<namespace/repo>
  • List subsets and splits: /splits?dataset=<namespace/repo>
  • Preview first rows: /first-rows?dataset=<namespace/repo>&config=<config>&split=<split>
  • Paginate rows: /rows?dataset=<namespace/repo>&config=<config>&split=<split>&offset=<int>&length=<int>
  • Search text: /search?dataset=<namespace/repo>&config=<config>&split=<split>&query=<text>&offset=<int>&length=<int>
  • Filter with predicates: /filter?dataset=<namespace/repo>&config=<config>&split=<split>&where=<predicate>&orderby=<sort>&offset=<int>&length=<int>
  • List parquet shards: /parquet?dataset=<namespace/repo>
  • Get size totals: /size?dataset=<namespace/repo>
  • Get column statistics: /statistics?dataset=<namespace/repo>&config=<config>&split=<split>
  • Get Croissant metadata (if available): /croissant?dataset=<namespace/repo>

Pagination pattern:

curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100"
curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100"

When pagination is partial, use response fields such as num_rows_total, num_rows_per_page, and partial to drive continuation logic.

Search/filter notes:

  • /search matches string columns (full-text style behavior is internal to the API).
  • /filter requires predicate syntax in where and optional sort in orderby.
  • Keep filtering and searches read-only and side-effect free.

For CLI-based parquet URL discovery or SQL, use the hf-cli skill with hf datasets parquet and hf datasets sql.

Creating and Uploading Datasets

Use one of these flows depending on dependency constraints.

Zero local dependencies (Hub UI):

  • Create dataset repo in browser: https://huggingface.co/new-dataset
  • Upload parquet files in the repo "Files and versions" page.
  • Verify shards appear in Dataset Viewer:
curl -s "https://datasets-server.huggingface.co/parquet?dataset=<namespace>/<repo>"

Low dependency CLI flow (npx @huggingface/hub / hfjs):

  • Set auth token:
export HF_TOKEN=<your_hf_token>
  • Upload parquet folder to a dataset repo (auto-creates repo if missing):
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data
  • Upload as private repo on creation:
npx -y @huggingface/hub upload datasets/<namespace>/<repo> ./local/parquet-folder data --private

After upload, call /parquet to discover <config>/<split>/<shard> values for querying with @~parquet.

Agent Traces

The Hub supports raw agent session traces from Claude Code, Codex, and Pi Agent. Upload them to Hugging Face Datasets as original JSONL files and the Hub can auto-detect the trace format, tag the dataset as Traces, and enable the trace viewer for browsing sessions, turns, tool calls, and model responses. Common local session directories:

  • Claude Code: ~/.claude/projects
  • Codex: ~/.codex/sessions
  • Pi: ~/.pi/agent/sessions

Session traces contain private prompts/files/tool outputs and possibly secrets or third-party information. Read-only exploration does not authorize upload. For an explicit trace-export request, select only named sessions, prepare a redacted export and verify actual destination visibility/retention before transmission. Creating with --private/--exist-ok does not prove an already-existing repo is private. Preserve raw originals locally.

hf repos create <namespace>/<repo> --type dataset --private --exist-ok
hf upload <namespace>/<repo> /absolute/reviewed-redacted-export codex/<selected-project> --type dataset

Examples

User: Apply this skill to my current task.
Assistant: Follow the workflow in this skill, cite limitations, and ask before risky steps.

Limitations

  • Imported upstream skill; verify credentials, permissions, and safety boundaries before execution.
  • Does not replace environment-specific validation, testing, or maintainer review.

Hugging Face ZeroGPU

When to Use

Use this skill when you need aI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. Use when writing or reviewing code that uses @spaces.GPU, configuring python_version or requirements.txt for a ZeroGPU Space, or handling ZeroGPU-specific code constraints — pickle-based process...

Rules and patterns for ML demos on Hugging Face Spaces with ZeroGPU hardware. Covers @spaces.GPU, duration and quota tuning, process isolation, the CUDA availability model, concurrency safety, and CUDA build constraints.

Scope

This skill is for Gradio SDK Spaces using ZeroGPU hardware. Docker and Static Spaces cannot schedule onto ZeroGPU, and Streamlit apps now run as Docker Spaces — so this skill applies only to Gradio. For general Gradio coding (components, layouts, event listeners), see the huggingface-gradio skill in this repo. The authoritative ZeroGPU docs live at https://huggingface.co/docs/hub/spaces-zerogpu — refer to them for the current backing GPU, runtime version lists, and tier thresholds, all of which change over time.

Reference Files

ReferenceWhen to read
references/concurrency.mdAlways read alongside SKILL.md when writing ZeroGPU code — handlers run in parallel by default
references/how-zerogpu-works.mdWhen reasoning about cold-starts, worker reuse, why module-scope warmup does not carry to requests, or why returning CUDA tensors hangs
references/how-quota-works.mdWhen choosing duration values, debugging illegal duration vs quota exceeded errors, or explaining why default 60s blocks short tasks
references/cuda-and-deps.mdWhen installing CUDA-dependent packages (e.g. flash-attn), pinning torch side-cars, or reading wheel filename tags

Hardware

ZeroGPU exposes two GPU sizes that map to a fraction of the backing card:

sizeSlice of backing GPUQuota cost
large (default)Half1x
xlargeFull2x

Default large gives half a physical GPU, so memory bandwidth and compute are significantly lower than the full card's specs. Use xlarge only when the workload genuinely needs the extra memory or compute.

Backing GPU changes without notice. ZeroGPU has already migrated across GPU generations several times; older write-ups may name A100 or H200, but those are outdated. For the current backing GPU and exact per-size VRAM, always check the ZeroGPU docs before sizing workloads.

Deployment and data boundaries

Select an authorized Space, supported runtime and bounded compute/credit budget before deployment. Pin reviewed model/kernel revisions and dependency artifacts; remote model code and binary wheels execute code and need separate review. Validate and cap prompt/steps/upload sizes, estimate duration with positive bounded values, and keep private prompts/tokens/uploads out of public logs/caches. Decorator/process separation is not a sandbox for untrusted pickle or code. Internal process/packing/quota descriptions below are release-specific; inspect the actual installed version and errors, not assumed internals.

Basic Pattern

import spaces
import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="...", device="cuda")

@spaces.GPU
def generate(prompt: str) -> str:
    return pipe(prompt, max_new_tokens=100)[0]["generated_text"]

Key rules:

  1. Instantiate models at module scope and call .to("cuda") eagerly. ZeroGPU handles the actual device mapping transparently (see CUDA availability model below).
  2. Decorate GPU functions with @spaces.GPU. The decorator is a no-op outside ZeroGPU, so it is safe to keep in all environments.
  3. Set duration to match the realistic worst-case workload (default 60s). The platform pre-checks requested duration against the user's remaining quota — not against the actual run time — so a 10-second task left at the 60s default fails with quota exceeded as soon as the user's remaining quota drops below 60s. Smaller declared duration also ranks higher in the node-level queue. See "Duration and Quota" below.
  4. torch.compile is NOT supported. Use PyTorch ahead-of-time compilation (AoTI) (torch 2.8+) instead.
  5. Use size="xlarge" sparingly. It allocates the full backing GPU, but costs 2x quota and tends to queue longer.
@spaces.GPU(duration=120)
def generate_image(prompt: str):
    return pipe(prompt).images[0]

CUDA Availability Model

Real GPU access is only available inside @spaces.GPU-decorated functions. Outside those functions, the GPU is not attached to the process.

However, import spaces monkey-patches torch so that:

  • torch.cuda.is_available() returns True globally.
  • .to("cuda") / device="cuda" calls at module scope succeed without error.

This is intentional. Module-scope model.to("cuda") calls register tensors with the ZeroGPU backend, which writes them to a disk offload directory at a startup "pack" step and frees the corresponding RAM. When a @spaces.GPU call lands, a forked GPU worker process streams those weights from disk into VRAM via a pinned-memory pipeline. Warm workers (reused across requests on the same GPU slot) keep weights resident on the GPU and skip the disk → VRAM step. The user-facing rule: write device="cuda" at module scope and it works — see references/how-zerogpu-works.md for the full lifecycle.

ActionWhereWhy
model.to("cuda") / pipe(..., device="cuda")Module scopeZeroGPU registers the tensor and manages device migration
Actual CUDA computation (inference, etc.)Inside @spaces.GPUReal GPU is only attached during the decorated call
Branching on torch.cuda.is_available()Avoid relying on itAlways returns True due to the monkey-patch

Do not run inference or CUDA kernels at module scope — the real GPU is not attached, so operations either silently run on CPU or fail.

Device selection idiom still works

The standard idiom remains correct under ZeroGPU:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = AutoModel.from_pretrained("...").to(device)
  • ZeroGPU — is_available() is True (monkey-patched), so the model is registered for automatic device migration.
  • Dedicated GPU Spaces / local GPU — is_available() is genuinely True.
  • CPU Spaces / local CPU — resolves to "cpu".

For code intended to run on CPU-only hosts, use the device-selection idiom. ZeroGPU-only examples above deliberately request cuda at module scope; portability requires selecting an available device.

Eager loading is the right default

Load models at module scope, not lazily on first request. The Space process starts before any user arrives, so cold-start cost is paid once. Lazy loading (global model; if model is None: ..., @lru_cache wrappers, factory functions instantiating on first call) just pushes that cost onto the first user.

Local Development: Just Install spaces

Do not wrap import spaces in try/except and redefine spaces.GPU as a no-op fallback for local runs. Off-ZeroGPU, the spaces package is already a true no-op:

  • Heavyweight behavior (CUDA monkey-patching, client init, startup hooks) is gated on the SPACES_ZERO_GPU env var, set only on ZeroGPU.
  • @spaces.GPU returns the undecorated function unchanged off-ZeroGPU.
  • Top-level import spaces performs only lightweight imports.

The Gradio SDK base image installs spaces on every hardware tier. So even after duplicating a Space onto a dedicated GPU (T4, L4, A10G, etc.) or CPU basic, no code changes are needed — import spaces still succeeds and @spaces.GPU becomes a transparent passthrough.

Anti-pattern

try:
    import spaces
except ImportError:
    class spaces:  # type: ignore
        @staticmethod
        def GPU(func=None, **kwargs):
            return func if func else (lambda f: f)

Problems:

  1. The fallback must mimic every @spaces.GPU call shape — bare decorator, duration=..., size=..., generators, aoti_* helpers — and drifts as the spaces API grows.
  2. It hides spaces from requirements.txt, even though the Space needs it at deploy time.
  3. It solves a non-problem: the real package is already a no-op locally.

Do this instead

Add spaces to dependencies and import it unconditionally:

import spaces

@spaces.GPU
def generate(prompt: str) -> str:
    ...

Duration and Quota

Three things happen when you declare @spaces.GPU(duration=N):

  1. Tier-max check — each visitor tier has a per-call duration cap. Declaring duration larger than the cap fails immediately with ZeroGPU illegal duration, regardless of remaining quota. (Tier numbers change over time — see the ZeroGPU docs.)
  2. Quota pre-check — the platform compares requested duration against the user's remaining quota. If remaining < requested, the call fails with ZeroGPU quota exceeded — even if the actual work would have fit. The error message shows the explicit numbers, e.g. "60s requested vs. 30s left". A 10-second task left at the default 60s therefore blocks the user once their remaining quota drops below 60s.
  3. Queue priority — the queue is node-level (requests from all Spaces on the same node compete for GPU slots), and shorter declared duration ranks higher.

All three favor declaring the smallest realistic duration — including for short tasks. Explicit @spaces.GPU(duration=15) on a 10-second task avoids premature quota exceeded rejections and ranks higher in the queue.

xlarge doubles the request. requested = N * 2 when size="xlarge", both for the tier-max check and the quota pre-check. So @spaces.GPU(duration=60, size="xlarge") is internally a 120s request.

Dynamic duration for variable workloads

For workloads whose runtime depends on inputs, pass a callable that estimates per request. A static high duration locks out low-tier users (whose tier cap may be smaller than the static value) and unnecessarily reserves quota for light inputs.

def estimate_duration(prompt, steps):
    return int(steps * 3.5)

@spaces.GPU(duration=estimate_duration)
def generate(prompt, steps):
    return pipe(prompt, num_inference_steps=steps).images[0]

For the full distinction between illegal duration vs quota exceeded, runs-per-day limits, the 24h quota window, and pay-as-you-go billing, see references/how-quota-works.md.

Process Isolation and Pickle

@spaces.GPU-decorated functions run in a separate process managed by the ZeroGPU scheduler. Arguments and return values cross the process boundary via pickle serialization.

Consequences:

  • Only picklable objects can be passed in or returned. Open file handles, database connections, locks, lambdas, and closures over unpicklable state will raise PicklingError.
  • Do NOT return CUDA tensors directly. Unpickling a CUDA tensor in the main process triggers torch.cuda._lazy_init(), which ZeroGPU blocks. Convert to CPU first: return tensor.cpu() or tensor.cpu().numpy().
  • CPU tensors, numpy arrays, PIL Images, and plain Python objects work fine.
  • Large objects incur serialization overhead. Prefer lightweight returns (tensors, arrays, file paths, base64 strings) over complex object graphs.

gr.State semantics across the boundary

Because handlers run in a separate process, gr.State values are pickled on every yield — they are NOT shared by reference.

  • The generator receives a copy of the state (id() differs from the caller's).
  • In-place mutations inside the generator are invisible to other handlers until the mutated state is explicitly yielded back.
  • Yielding gr.update() for a gr.State slot skips the update — other handlers continue to see the pre-yield value.
  • Each yield that returns the state object creates a new copy via pickle.

Practical guidance:

  • Do NOT assume reference semantics for gr.State on ZeroGPU. Code that mutates state in a generator and expects another handler to see those mutations will silently use stale data.
  • Every yield including a gr.State value triggers a full pickle round-trip. For large state (model sessions, frame buffers), minimize how often you yield it — ideally once at the end. Use gr.update() for the state slot on intermediate yields.
  • CUDA tensors inside state must be moved to CPU before yielding — same torch.cuda._lazy_init() issue as above.

Concurrency

Handlers run concurrently by default on ZeroGPU. This is not opt-in. Code that worked in single-user testing can silently corrupt or leak data in production.

Three rules. Full treatment with examples in references/concurrency.md.

  1. No mutable global state. Concurrent requests overwrite each other.
  2. No fixed file paths for outputs. Concurrent requests clobber the same file. Use tempfile for unique paths.
  3. Verify model/library concurrency safety. Startup-loaded models are encouraged, but generation may mutate caches/RNG/adapters internally; isolate request/session state or serialize incompatible operations.

Call Granularity

Each entry into a @spaces.GPU function carries non-trivial cost — pickle round-trip across the process boundary, worker warm-up, CUDA re-attach, and a fresh pass through the node-level queue. Calling a decorated function from inside a hot loop multiplies these costs and adds a new failure mode: a later iteration may fail to acquire a GPU slot, stalling the whole job mid-way.

Decorate the outer function that owns the loop, not the per-iteration worker:

# Avoid — N GPU entries for N frames
def process_video(frames):
    return [process_frame(f) for f in frames]

@spaces.GPU(duration=...)
def process_frame(frame):
    ...

# Prefer — one GPU entry for the whole video
@spaces.GPU(duration=...)
def process_video(frames):
    return [process_frame(f) for f in frames]

def process_frame(frame):
    ...

If the loop mixes heavy CPU work with GPU work, wrapping the whole loop charges that CPU time against the user's quota. When that cost is material, batching the GPU work so CPU pre/post-processing stays outside the decorator is a situational optimization — not the default.

CUDA Build Constraints

HF Spaces builds Docker images in a CPU-only environment. On ZeroGPU, the build phase has no nvcc because the base image is python:3.13 (dedicated-GPU Spaces use nvidia/cuda:*-devel-* and have nvcc at build time). A CUDA-dependent package whose only distribution is sdist — e.g. bare flash-attn — therefore cannot be installed via requirements.txt on ZeroGPU. Only pre-built wheels work.

ZeroGPU runtime does have nvcc available, mounted from a CUDA devel image at /cuda-image since 2025-07 (originally added for AoTI support). This is what makes torch.export / AoTI workflows possible inside @spaces.GPU calls.

Bottom line: install every CUDA-dependent package from a pre-built wheel. If no wheel is available on PyPI, build one externally (e.g. host on HF Hub) and pin the URL. For flash-attn, the upstream releases page ships a fairly complete wheel matrix covering most Python × CUDA × torch combinations.

For wheel-tag reading (cxx11 ABI, cu12torch2.X, cp3XX), torch-family side-car drift, and the kernels-community fallback, see references/cuda-and-deps.md.

Example Caching

gr.Examples behavior is environment-dependent. On ZeroGPU specifically:

  • cache_examples defaults to True (Spaces sets GRADIO_CACHE_EXAMPLES=true).
  • cache_mode defaults to "lazy" (Spaces sets GRADIO_CACHE_MODE=lazy only on ZeroGPU).

ZeroGPU defaults to lazy because eager caching pre-runs every example at app startup, but ZeroGPU has no GPU attached at startup — only during request handling. Eager caching of GPU-bound examples would fail there.

When cache_examples=True, the run_on_click / run_examples_on_click parameter is silently ignored. If your app relies on click-populates-only behavior, set cache_examples=False explicitly to preserve it.

To reproduce ZeroGPU example-caching behavior locally:

GRADIO_CACHE_EXAMPLES=true GRADIO_CACHE_MODE=lazy python app.py

Dependency Management

python_version pin in README frontmatter

Pinning python_version is effectively required for ZeroGPU. The default runtime can differ from local Python; mismatches can break dependencies, but a newer local Python does not by itself imply installation failure. Pin to a ZeroGPU-supported version (3.12 is a reasonable default); the authoritative supported list lives in the ZeroGPU docs — do not hardcode the full list, refer to the docs.

# README.md frontmatter
python_version: "3.12"

Both "3.12" and "3.12.12" forms are accepted.

Do not pin spaces in requirements.txt

The Space platform pins its own spaces version. A conflicting pin in requirements.txt causes pip resolution to fail at build time.

Rule: Do not include spaces in requirements.txt.

How to achieve this depends on your tooling:

  • Hand-written requirements.txt: simply omit spaces.
  • uv (pyproject.toml-managed): declare spaces in pyproject.toml so uv co-resolves transitive constraints (notably psutil, which spaces pins), then exclude it from the export:
    uv export --no-hashes --no-dev --no-emit-package spaces -o requirements.txt
    
    Without spaces in pyproject.toml, uv cannot see its transitive constraints and may resolve incompatible versions at build time.
  • pip-tools (pip-compile) / Poetry: use the equivalent exclude mechanism.

Pin torch to match wheel tags

If you install a CUDA-dependent wheel via direct URL, the wheel filename encodes the torch major.minor it was built against (e.g. cu12torch2.8). Pin torch==X.Y.Z in requirements.txt to match — otherwise pip may resolve torch to a different version and the Space fails on first import. Details and the kernels-community alternative are in references/cuda-and-deps.md.

Limitations

  • Use this skill only when the task clearly matches its upstream product or API scope.
  • Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
  • Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.

Itô Inference

ito-inference is the sole canonical ECC skill for inference serving on Itô compute. Requests naming ito-serve route here; do not create or install a second ito-serve skill. ECC never SSHes to nodes, downloads weights, launches an engine, or exposes an endpoint; it never books, reserves, or spends.

Current production boundary

Managed serving is unavailable today. The ECC bridge exposes only login, auth, find, status, and explicitly gated evals. It has no serve verb. The canonical runtime documents inference only as an unsupported compatibility probe; ECC does not invoke or depend on it. The MCP surface exposes only auth, find, and status. The locally enforceable guarantee is that ECC rejects serve before resolving or spawning the credential-bearing canonical client.

Therefore stop before authentication or any command invocation. Report the missing capability and return to the originating agent. Never substitute a local runner, SSH helper, browser workflow, purchase endpoint, or any untracked local ito-serve draft.

Required entitlement

When serving is implemented, its first gate is a server-verified completed booking. Harness memory, an RFQ, a quote, node IPs, or SSH access are not proof of entitlement. The backend must return fresh serving eligibility bound to the authenticated account, booking, GPU topology, region, fabric, term, and model policy. Expired, revoked, mismatched, incomplete, or already-released bookings fail closed before confirmation.

Future CLI and API contract

The intended command name is serve; inference may remain only as an explicitly deprecated compatibility alias after the production contract lands. The future handoff must be equivalent to:

ecc ito serve \
  --booking <server-verified-booking-id> \
  --manifest <absolute-reviewed-json-file> \
  --confirmation-ref <opaque-non-authorizing-reference> \
  --idempotency-key <stable-retry-key> \
  --json

The reviewed manifest must identify the model revision, engine and version, quantization, tensor/pipeline topology, endpoint exposure policy, artifact checksums, storage ceiling, runtime limits, optional TTFT/TPOT objectives, and maximum incremental cost. No raw API key, SSH key, node password, or bearer token belongs in arguments, manifests, logs, MCP results, or chat.

The client must canonicalize the manifest path, reject symlinks, open a regular file without following links, require appropriate ownership and restrictive permissions, enforce a bounded size, and hash bytes from the opened descriptor. That digest must exactly equal the digest bound into confirmation before any workload mutation. A path swap, digest mismatch, oversized file, or mutable unsafe file fails closed.

The canonical API—not ECC—must own workload creation and return structured JSON with ok, live_api_contacted, notice, and either data or error. Serving data must include stable booking, workload, manifest, and idempotency IDs plus a state enum; it must not claim an endpoint is live until health and model checks pass. Errors must include a stable code and safe message without secrets.

Confirmation and execution gates

Before workload creation, require all of the following:

  1. Fresh entitlement and serving eligibility from the canonical backend.
  2. A reviewable immutable manifest and deterministic digest.
  3. A separate single-use confirmation bound to account, action, manifest, and cost, with a short expiry and replay protection. CLI arguments carry only an opaque, non-authorizing confirmation reference; the server resolves and consumes the bearer capability out of band.
  4. A caller-supplied idempotency key reserved atomically with the workload.
  5. Server-side fabric, capacity, model-policy, storage, and cost validation.

Authentication is identity, not workload authority. A login, API key, quote, or completed booking never substitutes for the serving confirmation. Inspection and plan generation must not create a workload. Cancel and cleanup are separate mutations with their own scoped confirmation and idempotency boundaries.

Lifecycle and recovery

The production surface is incomplete until the same canonical client exposes tenant-scoped status, logs, metrics, cancel, and cleanup operations. Every operation needs bounded connect and overall timeouts, revocation-aware errors, and structured output. After an ambiguous transport failure, query status by the idempotency key before retrying; never create a second workload merely because the first response was lost. A revoked credential stops polling and returns control to the originating agent without starting login automatically.

Only report ready after endpoint health, model identity, and canary inference all pass. Report intermediate and terminal failure states honestly. Cleanup must be observable and must not release or modify the underlying booking unless that separate economic action was explicitly authorized.

Proposed backend stages

These stages describe the future backend, not code that exists in ECC:

  1. Verify entitlement, topology, fabric, and cost gates.
  2. Fetch checksum-pinned weights into backend-managed storage.
  3. Emit and validate a reviewable topology/engine plan.
  4. Launch through the provider control plane, never direct root SSH from ECC.
  5. Warm up, test health and model identity, run an SLO canary, then register the endpoint and redacted configuration.

Until every gate and lifecycle operation above exists in the canonical runtime, this skill remains a fail-closed availability check and documentation handoff.

Itô Training

ito-training is the canonical ECC skill for training on Itô compute. ECC never runs a trainer, scheduler, or data pipeline of its own; it never books, reserves, or spends. This skill chains off a completed booking from ito-compute.

Current production boundary

Managed training is unavailable today. The ECC bridge exposes only login, logout, auth, find, status, and explicitly gated evals. It has no train verb, and the canonical CLI's run verb and desk training-run backend remain scaffolds. The locally enforceable guarantee is that ECC rejects train before resolving or spawning the credential-bearing canonical client.

Therefore stop before authentication or any command invocation. Report the missing capability and return to the originating agent. Never substitute a local trainer, SSH helper, browser workflow, or purchase endpoint.

Required entitlement

When training is implemented, its first gate is a server-verified completed booking. Harness memory, an RFQ, a quote, node IPs, or SSH access are not proof of entitlement. The backend must return fresh training eligibility bound to the authenticated account, booking, GPU topology, region, fabric, and term. Expired, revoked, mismatched, incomplete, or already-released bookings fail closed before confirmation.

Future CLI and API contract

The intended command name is train. The future handoff must be equivalent to:

ecc ito train \
  --booking <server-verified-booking-id> \
  --manifest <absolute-reviewed-json-file> \
  --confirmation-ref <opaque-non-authorizing-reference> \
  --idempotency-key <stable-retry-key> \
  --json

The reviewed manifest must identify the model size and revision, data references with decontamination provenance, training target, post-training recipe, budget ceiling in USD, checkpoint policy, and maximum incremental cost. No raw API key, SSH key, node password, bearer token, or dataset credential belongs in arguments, manifests, logs, MCP results, or chat.

The client must canonicalize the manifest path, reject symlinks, open a regular file without following links, require appropriate ownership and restrictive permissions, enforce a bounded size, and hash bytes from the opened descriptor. That digest must exactly equal the digest bound into confirmation before any workload mutation. A path swap, digest mismatch, oversized file, or mutable unsafe file fails closed.

The canonical API—not ECC—must own workload creation and return structured JSON with ok, live_api_contacted, notice, and either data or error. Training data must include stable booking, run, manifest, and idempotency IDs plus a state enum. Errors must include a stable code and safe message without secrets.

Confirmation and execution gates

Before workload creation, require all of the following:

  1. Fresh entitlement and training eligibility from the canonical backend.
  2. A reviewable immutable manifest and deterministic digest.
  3. A separate single-use confirmation bound to account, action, manifest, and cost, with a short expiry and replay protection. CLI arguments carry only an opaque, non-authorizing confirmation reference; the server resolves and consumes the bearer capability out of band.
  4. A caller-supplied idempotency key reserved atomically with the run.
  5. Server-side fabric, capacity, data-policy, checkpoint-storage, and cost validation, including the manifest's budget ceiling.

Authentication is identity, not workload authority. A login, API key, quote, or completed booking never substitutes for the training confirmation. Inspection and plan generation must not create a workload. Cancel and cleanup are separate mutations with their own scoped confirmation and idempotency boundaries.

Lifecycle and recovery

The production surface is incomplete until the same canonical client exposes tenant-scoped status, logs, metrics, checkpoint listing, cancel, and cleanup. Every operation needs bounded connect and overall timeouts, revocation-aware errors, and structured output. After an ambiguous transport failure, query status by the idempotency key before retrying; never create a second run merely because the first response was lost. A revoked credential stops polling and returns control to the originating agent without starting login automatically.

Report stage gates honestly; never override a failed eval gate. Cleanup must be observable and must not release or modify the underlying booking unless that separate economic action was explicitly authorized.

Proposed backend stages

These stages describe the future backend (Layer 0.3), not code that exists in ECC:

  1. Data prep — manifest, dedup, decontamination against the eval suite; 150M-ladder decision job as the cheap pre-check for custom data.
  2. Parallelism and precision — selected from model size, node count, fabric; wasteful combinations refused.
  3. Checkpointing and fault tolerance — async DCP, torchft; detect < 10 min, resume < 15 min. Loss-spike restart is a proposed, human-gated action.
  4. Curriculum and eval gates — staged pretrain / mid-train / long-context / post-training, each with a fixed eval battery; a failed gate stops the run.
  5. Post-training — SFT → DPO → RLVR (GRPO with DAPO stability fixes), trainer/rollout separation with bounded staleness.

The backend emits desk telemetry (goodput, interruption rate, checkpoint bandwidth) so the desk prices training blocks honestly.

Until every gate and lifecycle operation above exists in the canonical runtime, this skill remains a fail-closed availability check and documentation handoff.

LLM Inference Scaling

Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.

When to Use This Skill

Use this skill when:

  • LLM API traffic is unpredictable and you need to scale up/down automatically
  • Managing a fleet of vLLM or TGI inference pods on Kubernetes
  • Reducing inference costs with spot/preemptible GPU instances
  • Implementing queue-based autoscaling for batch inference jobs
  • Building a multi-model serving platform that shares GPU resources

Prerequisites

  • Kubernetes cluster with GPU nodes (NVIDIA operator installed)
  • KEDA (Kubernetes Event-Driven Autoscaler) installed
  • Prometheus with GPU metrics (dcgm-exporter or gpu-operator)
  • Helm 3+ for chart deployments

GPU Node Setup

# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true \
  --set dcgm.enabled=true \
  --set devicePlugin.enabled=true

# Verify GPU nodes are recognized
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe node <gpu-node> | grep nvidia

vLLM Deployment with GPU Resources

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
        - "1"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-num-seqs"
        - "128"
        resources:
          requests:
            nvidia.com/gpu: "1"
            memory: "20Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

KEDA Autoscaling on Prometheus Metrics

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300          # 5 min before scale-down
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: "10"           # scale up if >10 requests waiting
      query: |
        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_gpu_cache_usage
      threshold: "0.8"          # scale up if KV cache >80% full
      query: |
        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})

Queue-Based Scaling (Redis + KEDA)

# ScaledJob for async batch inference
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: llm-batch-inference
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: inference-worker
          image: myapp/inference-worker:latest
          env:
          - name: REDIS_URL
            value: redis://redis:6379
          - name: QUEUE_NAME
            value: inference-jobs
        restartPolicy: OnFailure
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 5
  successfulJobsHistoryLimit: 3
  triggers:
  - type: redis
    metadata:
      address: redis:6379
      listName: inference-jobs
      listLength: "5"           # 1 worker per 5 queued jobs

Spot Instance Strategy

# Mixed node pool: on-demand + spot GPUs
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-config
data:
  priorities: |
    10:  # lower priority; higher numeric priority is preferred
    - .*spot.*
    50:
    - .*on-demand.*
---
# Node affinity for spot with on-demand fallback
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [spot]
      - weight: 20
        preference:
          matchExpressions:
          - key: node.kubernetes.io/lifecycle
            operator: In
            values: [on-demand]

Cluster Autoscaler for GPU Nodes

# AWS EKS — enable cluster autoscaler for GPU node group
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=my-cluster \
  --set awsRegion=us-east-1 \
  --set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
  --set extraArgs.skip-nodes-with-local-storage=false \
  --set extraArgs.expander=least-waste

# Annotate GPU node group for autoscaler
kubectl annotate node <node> \
  cluster-autoscaler.kubernetes.io/safe-to-evict="false"

Scaling Metrics to Monitor

# Prometheus queries for scaling decisions
# Requests waiting in vLLM queue
sum(vllm:num_requests_waiting) by (model)

# GPU KV cache utilization (>80% = bottleneck)
avg(vllm:gpu_cache_usage_perc) by (pod)

# Tokens per second throughput
sum(rate(vllm:generation_tokens_total[5m])) by (model)

# P99 time-to-first-token
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Common Issues

IssueCauseFix
Pods stuck in PendingNo GPU nodes availableCheck cluster autoscaler logs; verify node group limits
Scale-up too slowCluster autoscaler delay + model load timePre-warm replicas; increase minReplicaCount
GPU fragmentationMultiple small models on large GPUsUse MIG partitioning or consolidate model sizes
Spot eviction causes errorsSpot instance reclamationAdd PodDisruptionBudget; use graceful shutdown
KEDA not scalingPrometheus query returns no dataTest query in Prometheus UI first

Best Practices

  • Set minReplicaCount: 1 to avoid cold starts; scale to 0 only for batch jobs.
  • Use PodDisruptionBudget with minAvailable: 1 to survive spot evictions.
  • Pre-pull model weights into a shared PVC to potentially reduce model download time; measure startup on the actual cluster.
  • Separate model families across node pools (A10G for 7B, A100 for 70B).
  • Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.

Related Skills

  • vllm-server (vllm-server) - vLLM configuration and tuning
  • gpu-server-management (gpu-server-management) - GPU node setup
  • model-serving-kubernetes (model-serving-kubernetes) - KServe
  • kubernetes-ops (kubernetes-ops) - Core Kubernetes
  • llm-cost-optimization (llm-cost-optimization) - Cost strategies

Limitations

  • Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
  • Docs-only import: upstream scripts and templates not bundled.

PodDisruptionBudgets constrain voluntary disruptions, not spot preemption or node failure. GPU workers need explicit nvidia.com/gpu resource requests/limits and a compatible device plugin; a ScaledJob example without them is incomplete. Aggregate classic Prometheus histogram buckets with sum by (le) before histogram_quantile. Pin reviewed images, validate metric labels/scaler schemas and bound cost before autoscaling.

LoRA & QLoRA Recipes

This skill assumes the routing decision already happened — finetuning-method-selection should have already pointed here because the data shape is demonstrations (SFT), not preference pairs or a verifiable reward signal. What follows is the current best-practice recipe for configuring the adapter itself: which modules to target, how to size rank and alpha, what learning rate to use, and when QLoRA buys real headroom versus when it just adds risk. Dataset preparation and quality checks are a separate concern — see dataset-curation.

Input: a routing decision (SFT via LoRA/ QLoRA) plus a target size class. Output format: a validated adapter config — the kwarg values below, not free-form advice — that llm-finetuning-training-engineer consumes directly when it generates a runnable script.

The Reference Recipe

The reference recipe is "LoRA Without Regret" (Thinking Machines/Schulman, 2025-09), now the settled convention for LoRA/QLoRA SFT.

Target Modules

Target all-linear modules, not just attention:

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", "down_proj",      # MLP — matters most
]

The MLP layers (gate_proj, up_proj, down_proj) matter most — attention-only targeting was the older, weaker convention. Dropping modules to save memory is a Failure Mode below, not a valid optimization.

Alpha and Learning Rate

  • lora_alpha = 2 * r is the settled convention (NeurIPS 2025 "intruder dimensions" result). Don't hand-tune alpha independently of rank — derive it from rank every time.
  • LoRA learning rate ≈ 10x the equivalent full-fine-tune LR. For QLoRA specifically, 2e-4 is the standard starting point. Full hyperparameter tables and worked examples: references/hyperparameters.md.

Rank by Task

Rank is task-shaped, not a single global default:

TaskRank
RL (GRPO/RLVR adapters)1–32
General default16–32
SFT at scaleup to ~256

Higher rank isn't automatically better — it raises capacity to memorize as fast as it raises capacity to generalize. Start at the row matching the task, and only move up a row if the lower rank measurably underfits on held-out eval, not as a default hedge.

Effective Batch Size

Keep effective batch size under 32. This recipe was validated at that scale — pushing effective batch higher is an untested extrapolation, not a free throughput win.

Unsloth Defaults

Unsloth is the reference implementation this plugin assumes as the default fast path — except for messages-shaped conversational SFT with assistant_only_loss=True, where Unsloth 2026.7.x's compiled trainer has no messages-shaped path at all and the plain-TRL escape hatch (references/unsloth-trl-mapping.md) is the default for that combination, not a rare-regression fallback. Its out-of-the-box defaults, and why each one is set that way:

  • lora_dropout=0 — the optimized kernel path assumes zero dropout; setting a nonzero value forfeits the fused-kernel speedup.
  • bias="none" — bias terms add adapter parameters for negligible quality gain at this rank range.
  • use_gradient_checkpointing="unsloth" — Unsloth's checkpointing variant, not vanilla HF checkpointing; saves roughly 30% VRAM over no checkpointing.
  • optim="adamw_8bit" — 8-bit AdamW cuts optimizer-state memory with negligible quality impact at LoRA/QLoRA adapter scale.
  • random_state fixed — pins LoRA initialization for reproducibility across runs; treat it like any other seed, not a tunable.

These show up together on the get_peft_model call:

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,               # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Exact kwarg names and their plain-TRL/PEFT equivalents, plus a full worked config including SFTConfig: references/unsloth-trl-mapping.md and references/hyperparameters.md.

LoRA vs QLoRA vs Full FT

SituationDefault choice
Adapting behavior on demonstrationsLoRA
Base model doesn't fit in bf16 at target rankQLoRA
Injecting dense new domain knowledgeFull FT (see finetuning-method-selection)
Unsure which oneLoRA — upgrade to QLoRA only if memory forces it
  • QLoRA = NF4-quantized frozen base weights + BF16 adapters. This is what makes a 65B-class model trainable on 48GB — the quantized base is the memory win, not the adapter itself.
  • Full fine-tuning is not a default. Reserve it for dense knowledge injection where the goal is changing what the model knows at the weight level, not adapting a behavior. For everything else in this skill's scope, LoRA or QLoRA is the starting assumption.
  • On DGX Spark, QLoRA can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller — bitsandbytes dequantization buffers are transient CUDA-side allocations that spike during load. A QLoRA OOM is not proof the model doesn't fit; the dgx-spark-ops plugin's spark-memory-thermal-ops skill covers the full OOM remediation ladder (bf16 LoRA is the next thing to try, not a further QLoRA shrink).

Failure Modes

  • fp16 divergence on non-BF16 GPUs. Training in fp16 on hardware that doesn't have solid BF16 support is a known source of loss spikes and silent divergence. Force bf16=True wherever the hardware supports it; don't fall back to fp16 as if it were equivalent. Check hardware support before picking a dtype:

    python -c "import torch; print(torch.cuda.is_bf16_supported())"
    
  • Rank too high on a small dataset overfits. A rank picked for "SFT at scale" (up to ~256) on a dataset that doesn't have scale behind it memorizes rather than generalizes. Match rank to the Rank by Task table above, not to the largest number available.

  • Removing target modules to save memory costs quality for negligible savings. The adapter parameters on gate_proj/up_proj/down_proj are a small fraction of total model size — cutting them barely moves memory but measurably hurts quality. If memory is tight, move to QLoRA or reduce rank/batch/pack length before trimming target modules.

All three failure modes share a pattern: they look like a training-loop bug (loss spikes, plateaus, memorization) but are actually a config choice that contradicts the reference recipe above. Check configuration against this skill before debugging the training loop itself.

References

  • references/hyperparameters.md — full rank/ alpha/LR tables by task type, rsLoRA notes, batch/packing interactions, and a complete worked Unsloth config block.
  • references/unsloth-trl-mapping.md — every Unsloth kwarg mapped to its TRL/PEFT equivalent, current TRL API notes, and the escape-hatch rule for when to drop back to plain TRL.

Related skills: finetuning-method-selection routes here; dataset-curation covers the data side this skill doesn't; llm-finetuning-training-engineer is the downstream consumer of the config this skill produces.

Mac mini LLM Lab

Turn a Mac mini into a low-noise, always-on local AI appliance.

When to Use This Skill

Use this skill when:

  • Setting up a dedicated local LLM inference server
  • Building a private AI development environment
  • Need always-on model serving without cloud costs
  • Running models that fit the actual machine’s unified memory
  • Creating a home lab AI server for a small team

Prerequisites

  • An actual Apple Silicon Mac mini configuration with sufficient available memory for the selected model
  • macOS Sonoma 14+ or Sequoia 15+
  • Ethernet connection (recommended over Wi-Fi)
  • UPS for power protection (optional but recommended)

Host and exposure boundaries

Check actual model hardware/memory and installed macOS/tool support first. The throughput/RAM table is illustrative, not measured evidence or a fit guarantee; include KV cache/context/concurrency overhead. Keep Ollama loopback-only by default; TLS alone in the Caddy example does not add authentication. Configure an authenticated private proxy/tunnel and access policy before remote exposure. Loopback binding must be deliberately adapted for Docker access, without exposing an unauthenticated LAN listener. Approve persistent services, downloads, updates, reboots and FileVault/power changes separately as applicable; preserve existing settings and recovery access. Create script/log directories before writing them and do not run privileged diagnostics unattended from cron.

Initial System Setup

# Update macOS
softwareupdate --install --all

# Install Xcode command-line tools
xcode-select --install

# Install Homebrew
# Obtain the official Homebrew installer at a reviewed version, inspect it,
# and run the verified local file only after installation authorization.

# Core packages
brew install tmux htop btop wget jq git neovim

# Python environment (for MLX and custom scripts)
brew install python@3.12 uv

# Monitoring
brew install prometheus node_exporter

Ollama Setup

# Install Ollama
brew install ollama

# Pull models based on your RAM
# 16GB Mac mini:
ollama pull llama3.1:8b
ollama pull nomic-embed-text
ollama pull codellama:7b

# 32GB Mac mini:
ollama pull llama3.1:8b
ollama pull qwen2.5:14b
ollama pull deepseek-coder-v2:16b
ollama pull nomic-embed-text

# 64GB+ Mac mini:
ollama pull llama3.1:70b
ollama pull qwen2.5:32b
ollama pull codellama:34b

# Verify Metal acceleration
ollama run llama3.1:8b --verbose
# Look for: "metal" in output

MLX Framework (Apple Silicon Native)

MLX runs models natively on Apple Silicon with excellent performance:

# Install MLX
uv pip install mlx mlx-lm

# Run a model
python3 -c "
from mlx_lm import load, generate
model, tokenizer = load('mlx-community/Llama-3.1-8B-Instruct-4bit')
response = generate(model, tokenizer, prompt='Explain Docker in 3 sentences', max_tokens=200)
print(response)
"

# MLX server (OpenAI-compatible API)
uv pip install mlx-lm[server]
mlx_lm.server --model mlx-community/Llama-3.1-8B-Instruct-4bit --port 8080

Auto-Start with launchd

<!-- ~/Library/LaunchAgents/com.ollama.serve.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.ollama.serve</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/ollama</string>
        <string>serve</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
        <key>OLLAMA_HOST</key>
        <string>127.0.0.1:11434</string>
        <key>OLLAMA_NUM_PARALLEL</key>
        <string>4</string>
        <key>OLLAMA_MAX_LOADED_MODELS</key>
        <string>2</string>
        <key>OLLAMA_FLASH_ATTENTION</key>
        <string>1</string>
    </dict>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/ollama.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/ollama.err</string>
</dict>
</plist>
# Load the service
launchctl load ~/Library/LaunchAgents/com.ollama.serve.plist

# Check status
launchctl list | grep ollama

# Unload if needed
launchctl unload ~/Library/LaunchAgents/com.ollama.serve.plist

Power & Reliability

# Prevent idle sleep on this desktop (record existing values for rollback)
sudo pmset -a disablesleep 1
sudo pmset -a sleep 0

# Auto-restart after power failure
sudo pmset -a autorestart 1

# Schedule weekly reboot (Sunday 4 AM)
sudo pmset repeat shutdown U 03:55:00 poweron U 04:00:00

# Check power settings
pmset -g

Remote Access

Tailscale (Recommended)

# Install Tailscale for easy secure remote access
brew install --cask tailscale

# Enable from menu bar, authenticate
# Access your Mac mini from anywhere: http://mac-mini:11434

SSH Hardening

# Enable remote login
sudo systemsetup -setremotelogin on

# Edit SSH config
sudo nano /etc/ssh/sshd_config
# Add:
# PasswordAuthentication no
# PubkeyAuthentication yes
# PermitRootLogin no
# AllowUsers yourusername

# Restart SSH
sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
sudo launchctl load /System/Library/LaunchDaemons/ssh.plist

Reverse Proxy with Caddy

brew install caddy

# Caddyfile
cat > /opt/homebrew/etc/Caddyfile << 'EOF'
llm.local:443 {
    tls internal
    reverse_proxy localhost:11434

    @api path /v1/*
    handle @api {
        reverse_proxy localhost:11434
    }
}

webui.local:443 {
    tls internal
    reverse_proxy localhost:3000
}
EOF

brew services start caddy

Open WebUI Setup

# Run Open WebUI via Docker
docker run -d \
  --name open-webui \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -e WEBUI_AUTH=true \
  -v open-webui:/app/backend/data \
  --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main

# Or install Docker first if not available
brew install --cask docker

Monitoring

# Health check script
cat > ~/scripts/llm-health.sh << 'SCRIPT'
#!/bin/bash
# Check Ollama
if curl -sf http://localhost:11434/api/tags > /dev/null; then
    echo "$(date): Ollama OK"
    curl -s http://localhost:11434/api/ps | python3 -m json.tool
else
    echo "$(date): Ollama DOWN"
    # Restart
    launchctl kickstart -k gui/$(id -u)/com.ollama.serve
fi

# System stats
echo "CPU: $(top -l 1 -n 0 | grep 'CPU usage')"
echo "Memory: $(vm_stat | head -5)"
echo "Disk: $(df -h / | tail -1)"
echo "Thermal: $(sudo powermetrics --samplers smc -n 1 2>/dev/null | grep 'CPU die' || echo 'N/A')"
SCRIPT
chmod +x ~/scripts/llm-health.sh

# Schedule health check every 5 minutes
# Add to crontab: crontab -e
# */5 * * * * ~/scripts/llm-health.sh >> ~/logs/llm-health.log 2>&1

Memory Usage by Model

ModelRAM RequiredTokens/sec (M2)Tokens/sec (M4)
llama3.1:8b (Q4)~5 GB~25 t/s~45 t/s
qwen2.5:14b (Q4)~9 GB~15 t/s~30 t/s
llama3.1:70b (Q4)~40 GB~5 t/s~10 t/s
nomic-embed-text~300 MBN/AN/A
codellama:13b~8 GB~18 t/s~35 t/s

Security Checklist

# Enable FileVault disk encryption
sudo fdesetup enable

# Enable firewall
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setglobalstate on
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --setstealthmode on

# Disable unnecessary sharing services
sudo launchctl disable system/com.apple.screensharing
sudo launchctl disable system/com.apple.AirPlayXPCHelper

# Set strong admin password
# System Settings > Users & Groups

# Restrict Ollama to local network only (if not using Tailscale)
# Set OLLAMA_HOST=127.0.0.1 in launchd plist

Performance Tuning

# Increase file descriptor limits for concurrent requests
sudo launchctl limit maxfiles 65536 200000

# Check unified memory pressure
memory_pressure

# Monitor GPU usage (Metal)
sudo powermetrics --samplers gpu_power -n 1

# Optimize for inference (disable Spotlight indexing on model dirs)
mdutil -i off ~/.ollama

Troubleshooting

IssueSolution
Model loading slowFirst load caches to memory; subsequent loads are fast
Out of memoryUse smaller quantization (Q4_K_M), reduce OLLAMA_MAX_LOADED_MODELS
Mac sleepingRun sudo pmset -a disablesleep 1
Ollama not startingCheck `launchctl list
Slow over Wi-FiUse Ethernet; Wi-Fi adds latency to streaming responses
Thermal throttlingEnsure adequate ventilation, check powermetrics

Related Skills

  • ollama-stack (ollama-stack) — Software stack with Docker Compose and LiteLLM
  • ssh-configuration (ssh-configuration) — Secure remote access
  • vpn-setup (vpn-setup) — Remote access via WireGuard/Tailscale

Limitations

  • Infrastructure commands can disrupt services: confirm target host/scope and have backups/snapshots before mutating state.
  • Docs-only import: upstream scripts and templates not bundled.

ML Adoption Playbook

This skill provides an adaptive methodology for implementing machine learning models into existing software engineering projects. It bridges the gap between traditional SWE and MLOps by structuring how ML should be researched, decoupled, trained, and integrated.

When to Activate

  • A user asks to "add ML" or "add an algorithm" to their existing codebase.
  • Planning the integration of a new model (e.g., recommendation, classification, forecasting) into a non-ML application.
  • Structuring a workflow for an agent to build, train, and deploy an ML component adaptively.

Phase 1: Problem Framing & Feasibility

Before writing model code, establish the "why" and "how".

  • Heuristic Check: Ask the user if a simple heuristic (e.g., regex, rule-based sorting) could solve the problem faster. If yes, start there.
  • Metric Definition: Define what business metric the ML model is trying to improve (e.g., click-through rate, reduced latency).
  • Mistake Budget: Define what a "bad" prediction looks like and how the system should handle it.

Phase 2: Data Readiness

ML is useless without clean, accessible data.

  • Audit Data Sources: Identify where the training data lives. Is it a live database, a static CSV, or an API?
  • Data Contract: Establish a schema for the input data. What features are required? What happens if a feature is missing?
  • Leakage Prevention: Ensure the user's proposed data split does not accidentally leak future information into the training set (e.g., chronological splitting for time-series data).

Phase 3: Architectural Integration & Decoupling

Do not tightly couple model inference to core business logic.

  • API Boundary: Suggest placing the model behind an API endpoint (e.g., using fastapi-patterns or django-patterns) or a dedicated service class.
  • Fallback Mechanisms: Design a default state. If the model takes too long to respond or throws an error, the system must gracefully fall back to a hardcoded rule.
  • Feature Flags: Wrap the new ML inference call in a feature flag so it can be rolled out (or rolled back) safely.

Phase 4: Model Implementation & Training

Structure the code for reproducibility and iteration.

  • Start Simple: Build a baseline model first (e.g., a simple scikit-learn Logistic Regression or a barebones PyTorch linear layer).
  • Reproducibility: Apply pytorch-patterns or similar best practices: fix random seeds, make code device-agnostic, and explicitly document tensor/array shapes.
  • Automated Evidence: Require tests for the data transforms and inference schema. Do not accept a model without an evaluation script comparing it against the baseline.

Phase 5: Handoff to MLOps

Once the baseline model is integrated, shift focus to continuous operations.

  • Refer to mle-workflow: Guide the user toward setting up experiment tracking, model registries, and drift detection.
  • CI/CD: Add the model evaluation step to the existing CI pipeline to ensure future commits do not degrade model performance.

Iterative Agent Workflow

When assisting a user via this playbook, agents should:

  1. Ask clarifying questions to complete Phase 1 before proposing architectures.
  2. Draft a data contract in Phase 2 for user approval.
  3. Write the decoupling interface (API/Service) in Phase 3 before writing the training loop.
  4. Deliver a reproducible script in Phase 4 that trains the model and saves the artifact.

ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill

  • Building new ML pipelines from scratch
  • Designing workflow orchestration for ML systems
  • Implementing data → model → deployment automation
  • Setting up reproducible training workflows
  • Creating DAG-based ML orchestration
  • Integrating ML components into production systems

What This Skill Provides

Core Capabilities

  1. Pipeline Architecture

    • End-to-end workflow design
    • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
    • Component dependencies and data flow
    • Error handling and retry strategies
  2. Data Preparation

    • Data validation and quality checks
    • Feature engineering pipelines
    • Data versioning and lineage
    • Train/validation/test splitting strategies
  3. Model Training

    • Training job orchestration
    • Hyperparameter management
    • Experiment tracking integration
    • Distributed training patterns
  4. Model Validation

    • Validation frameworks and metrics
    • A/B testing infrastructure
    • Performance regression detection
    • Model comparison workflows
  5. Deployment Automation

    • Model serving patterns
    • Canary deployments
    • Blue-green deployment strategies
    • Rollback mechanisms

Reference Documentation

See the references/ directory for detailed guides:

  • data-preparation.md - Data cleaning, validation, and feature engineering
  • model-training.md - Training workflows and best practices
  • model-validation.md - Validation strategies and metrics
  • model-deployment.md - Deployment patterns and serving architectures

Assets and Templates

The assets/ directory contains:

  • pipeline-dag.yaml.template - DAG template for workflow orchestration
  • training-config.yaml - Training configuration template
  • validation-checklist.md - Pre-deployment validation checklist

Usage Patterns

Basic Pipeline Setup

# 1. Define pipeline stages
stages = [
    "data_ingestion",
    "data_validation",
    "feature_engineering",
    "model_training",
    "model_validation",
    "model_deployment"
]

# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example

Production Workflow

  1. Data Preparation Phase

    • Ingest raw data from sources
    • Run data quality checks
    • Apply feature transformations
    • Version processed datasets
  2. Training Phase

    • Load versioned training data
    • Execute training jobs
    • Track experiments and metrics
    • Save trained models
  3. Validation Phase

    • Run validation test suite
    • Compare against baseline
    • Generate performance reports
    • Approve for deployment
  4. Deployment Phase

    • Package model artifacts
    • Deploy to serving infrastructure
    • Configure monitoring
    • Validate production traffic

Best Practices

Pipeline Design

  • Modularity: Each stage should be independently testable
  • Idempotency: Re-running stages should be safe
  • Observability: Log metrics at every stage
  • Versioning: Track data, code, and model versions
  • Failure Handling: Implement retry logic and alerting

Data Management

  • Use data validation libraries (Great Expectations, TFX)
  • Version datasets with DVC or similar tools
  • Document feature engineering transformations
  • Maintain data lineage tracking

Model Operations

  • Separate training and serving infrastructure
  • Use model registries (MLflow, Weights & Biases)
  • Implement gradual rollouts for new models
  • Monitor model performance drift
  • Maintain rollback capabilities

Deployment Strategies

  • Start with shadow deployments
  • Use canary releases for validation
  • Implement A/B testing infrastructure
  • Set up automated rollback triggers
  • Monitor latency and throughput

Integration Points

Orchestration Tools

  • Apache Airflow: DAG-based workflow orchestration
  • Dagster: Asset-based pipeline orchestration
  • Kubeflow Pipelines: Kubernetes-native ML workflows
  • Prefect: Modern dataflow automation

Experiment Tracking

  • MLflow for experiment tracking and model registry
  • Weights & Biases for visualization and collaboration
  • TensorBoard for training metrics

Deployment Platforms

  • AWS SageMaker for managed ML infrastructure
  • Google Vertex AI for GCP deployments
  • Azure ML for Azure cloud
  • OCI Data Science for Oracle Cloud Infrastructure deployments
  • Kubernetes + KServe for cloud-agnostic serving

Progressive Disclosure

Start with the basics and gradually add complexity:

  1. Level 1: Simple linear pipeline (data → train → deploy)
  2. Level 2: Add validation and monitoring stages
  3. Level 3: Implement hyperparameter tuning
  4. Level 4: Add A/B testing and gradual rollouts
  5. Level 5: Multi-model pipelines with ensemble strategies

Common Patterns

Batch Training Pipeline

# See assets/pipeline-dag.yaml.template
stages:
  - name: data_preparation
    dependencies: []
  - name: model_training
    dependencies: [data_preparation]
  - name: model_evaluation
    dependencies: [model_training]
  - name: model_deployment
    dependencies: [model_evaluation]

Real-time Feature Pipeline

# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.md

Continuous Training

# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.md

Troubleshooting

Common Issues

  • Pipeline failures: Check dependencies and data availability
  • Training instability: Review hyperparameters and data quality
  • Deployment issues: Validate model artifacts and serving config
  • Performance degradation: Monitor data drift and model metrics

Debugging Steps

  1. Check pipeline logs for each stage
  2. Validate input/output data at boundaries
  3. Test components in isolation
  4. Review experiment tracking metrics
  5. Inspect model artifacts and metadata

Next Steps

After setting up your pipeline:

  1. Explore hyperparameter-tuning skill for optimization
  2. Learn experiment-tracking-setup for MLflow/W&B
  3. Review model-deployment-patterns for serving strategies
  4. Implement monitoring with observability tools

Related Skills

  • experiment-tracking-setup: MLflow and Weights & Biases integration
  • hyperparameter-tuning: Automated hyperparameter optimization
  • model-deployment-patterns: Advanced deployment strategies

Machine Learning Engineering Workflow

Use this skill to turn model work into a production ML system with clear data contracts, repeatable training, measurable quality gates, deployable artifacts, and operational monitoring.

When to Activate

  • Planning or reviewing a production ML feature, model refresh, ranking system, recommender, classifier, embedding workflow, or forecasting pipeline
  • Converting notebook code into a reusable training, evaluation, batch inference, or online inference pipeline
  • Designing model promotion criteria, offline/online evals, experiment tracking, or rollback paths
  • Debugging failures caused by data drift, label leakage, stale features, artifact mismatch, or inconsistent training and serving logic
  • Adding model monitoring, canary rollout, shadow traffic, or post-deploy quality checks

Scope Calibration

Use only the lanes that fit the system in front of you. This skill is useful for ranking, search, recommendations, classifiers, forecasting, embeddings, LLM workflows, anomaly detection, and batch analytics, but it should not force one architecture onto all of them.

  • Do not assume every model has supervised labels, online serving, a feature store, PyTorch, GPUs, human review, A/B tests, or real-time feedback.
  • Do not add heavyweight MLOps machinery when a data contract, baseline, eval script, and rollback note would make the change reviewable.
  • Do make assumptions explicit when the project lacks labels, delayed outcomes, slice definitions, production traffic, or monitoring ownership.
  • Treat examples as interchangeable scaffolds. Replace metrics, serving mode, data stores, and rollout mechanics with the project-native equivalents.

Related Skills

  • python-patterns and python-testing for Python implementation and pytest coverage
  • pytorch-patterns for deep learning models, data loaders, device handling, and training loops
  • eval-harness and ai-regression-testing for promotion gates and agent-assisted regression checks
  • database-migrations, postgres-patterns, and clickhouse-io for data storage and analytics surfaces
  • deployment-patterns, docker-patterns, and security-review for serving, secrets, containers, and production hardening

Reuse the SWE Surface

Do not treat MLE as separate from software engineering. Most ECC SWE workflows apply directly to ML systems, often with stricter failure modes:

The recommended minimal --with capability:machine-learning install keeps the core agent surface available alongside this skill. For skill-only or agent-limited harnesses, pair skill:mle-workflow with agent:mle-reviewer where the target supports agents.

SWE surfaceMLE use
product-capability / architecture-decision-recordsTurn model work into explicit product contracts and record irreversible data, model, and rollout choices
repo-scan / codebase-onboarding / code-tourFind existing training, feature, serving, eval, and monitoring paths before introducing a parallel ML stack
plan / feature-devScope model changes as product capabilities with data, eval, serving, and rollback phases
tdd-workflow / python-testingTest feature transforms, split logic, metric calculations, artifact loading, and inference schemas before implementation
code-reviewer / mle-reviewerReview code quality plus ML-specific leakage, reproducibility, promotion, and monitoring risks
build-fix / pr-test-analyzerDiagnose broken CI, flaky evals, missing fixtures, and environment-specific model or dependency failures
quality-gate / test-coverageRequire automated evidence for transforms, metrics, inference contracts, promotion gates, and rollback behavior
eval-harness / verification-loopTurn offline metrics, slice checks, latency budgets, and rollback drills into repeatable gates
ai-regression-testingPreserve every production bug as a regression: missing feature, stale label, bad artifact, schema drift, or serving mismatch
api-design / backend-patternsDesign prediction APIs, batch jobs, idempotent retraining endpoints, and response envelopes
database-migrations / postgres-patterns / clickhouse-ioVersion labels, feature snapshots, prediction logs, experiment metrics, and drift analytics
deployment-patterns / docker-patternsPackage reproducible training and serving images with health checks, resource limits, and rollback
canary-watch / dashboard-builderMake rollout health visible with model-version, slice, drift, latency, cost, and delayed-label dashboards
security-review / security-scanCheck model artifacts, notebooks, prompts, datasets, and logs for secrets, PII, unsafe deserialization, and supply-chain risk
e2e-testing / browser-qa / accessibilityTest critical product flows that consume predictions, including explainability and fallback UI states
benchmark / performance-optimizerMeasure throughput, p95 latency, memory, GPU utilization, and cost per prediction or retrain
cost-aware-llm-pipeline / token-budget-advisorRoute LLM/embedding workloads by quality, latency, and budget instead of defaulting to the largest model
documentation-lookup / search-firstVerify current library behavior for model serving, feature stores, vector DBs, and eval tooling before coding
git-workflow / github-ops / opensource-pipelinePackage MLE changes for review with crisp scope, generated artifacts excluded, and reproducible test evidence
strategic-compact / dmux-workflowsSplit long ML work into parallel tracks: data contract, eval harness, serving path, monitoring, and docs

Ten MLE Task Simulations

Use these simulations as coverage checks when planning or reviewing MLE work. A strong MLE workflow should reduce each task to explicit contracts, reusable SWE surfaces, automated evidence, and a reviewable artifact.

IDCommon MLE taskStreamlined ECC pathRequired outputPipeline lanes covered
MLE-01Frame an ambiguous prediction, ranking, recommender, classifier, embedding, or forecast capabilityproduct-capability, plan, architecture-decision-records, mle-workflowIteration Compact naming who cares, decision owner, success metric, unacceptable mistakes, assumptions, constraints, and first experimentproduct contract, stakeholder loss, risk, rollout
MLE-02Define metric goals, labels, data sources, and the mistake budgetrepo-scan, database-reviewer, database-migrations, postgres-patterns, clickhouse-ioData and metric contract with entity grain, label timing, label confidence, feature timing, point-in-time joins, split policy, and dataset snapshotdata contract, metric design, leakage, reproducibility
MLE-03Build a baseline model and scoring path before adding complexitytdd-workflow, python-testing, python-patterns, code-reviewerBaseline scorer with confusion matrix, calibration notes, latency/cost estimate, known weaknesses, and tests for score shape and determinismbaseline, scoring, testing, serving parity
MLE-04Generate features from hypotheses about what separates outcomespython-patterns, pytorch-patterns, docker-patterns, deployment-patternsFeature plan and transform module covering signal source, missing values, outliers, correlations, leakage checks, and train/serve equivalencefeature pipeline, leakage, training, artifacts
MLE-05Tune thresholds, configs, and model complexity under tradeoffseval-harness, ai-regression-testing, quality-gate, test-coverageThreshold/config report comparing precision, recall, F1, AUC, calibration, group slices, latency, cost, complexity, and acceptable error classesevaluation, threshold, promotion, regression
MLE-06Run error analysis and turn mistakes into the next experimenteval-harness, ai-regression-testing, mle-reviewer, silent-failure-hunterError cluster report for false positives, false negatives, ambiguous labels, stale features, missing signals, and bug traces with lessons capturederror analysis, bug trace, iteration, regression
MLE-07Package a model artifact for batch or online inferenceapi-design, backend-patterns, security-review, security-scanVersioned artifact bundle with preprocessing, config, dependency constraints, schema validation, safe loading, and PII-safe logsartifact, security, inference contract
MLE-08Ship online serving or batch scoring with feedback captureapi-design, backend-patterns, e2e-testing, browser-qa, accessibilityPrediction endpoint or batch job with response envelope, timeout, batching, fallback, model version, confidence, feedback logging, and product-flow testsserving, batch inference, fallback, user workflow
MLE-09Roll out a model with shadow traffic, canary, A/B test, or rollbackcanary-watch, dashboard-builder, verification-loop, performance-optimizerRollout plan naming traffic split, dashboards, p95 latency, cost, quality guardrails, rollback artifact, and rollback triggerdeployment, canary, rollback
MLE-10Operate, debug, and refresh a production model after launchsilent-failure-hunter, dashboard-builder, mle-reviewer, doc-updater, github-opsObservation ledger and refresh plan with drift checks, delayed-label health, alert owners, runbook updates, retrain criteria, and PR evidencemonitoring, incident response, retraining

Iteration Compact

Before touching model code, compress the work into one reviewable artifact. This should be short enough to fit in a PR description and precise enough that another engineer can challenge the tradeoffs.

Goal:
Who cares:
Decision owner:
User or system action changed by the model:
Success metric:
Guardrail metrics:
Mistake budget:
Unacceptable mistakes:
Acceptable mistakes:
Assumptions:
Constraints:
Labels and data snapshot:
Baseline:
Candidate signals:
Threshold or config plan:
Eval slices:
Known risks:
Next experiment:
Rollback or fallback:

This compact is the MLE equivalent of a strong SWE design note. It keeps the team from optimizing a metric no one trusts, adding features that do not address the real error mode, or shipping complexity without a rollback.

Decision Brain

Use this loop whenever the task is ambiguous, high-impact, or metric-heavy:

  1. Start from the decision, not the model. Name the action that changes downstream behavior.
  2. Name who cares and why. Different stakeholders pay different costs for false positives, false negatives, latency, compute spend, opacity, or missed opportunities.
  3. Convert ambiguity into hypotheses. Ask what signal would separate outcomes, what evidence would disprove it, and what simple baseline should be hard to beat.
  4. Research prior art or a nearby known problem before inventing a bespoke system.
  5. Score choices with (probability, confidence) x (cost, severity, importance, impact).
  6. Consider adversarial behavior, incentives, selective disclosure, distribution shift, and feedback loops.
  7. Prefer the simplest change that reduces the most important mistake. Simplicity is not laziness; it is a way to minimize blunders while preserving iteration speed.
  8. Capture the decision, evidence, counterargument, and next reversible step.

Metric and Mistake Economics

Choose metrics from failure costs, not habit:

  • Use a confusion matrix early so the team can discuss concrete false positives and false negatives instead of abstract accuracy.
  • Favor precision when the cost of an incorrect positive decision dominates.
  • Favor recall when the cost of a missed positive dominates.
  • Use F1 only when the precision/recall tradeoff is genuinely balanced and explainable.
  • Use AUC or ranking metrics when ordering quality matters more than a single threshold.
  • Track latency, throughput, memory, and cost as first-class metrics because they shape feasible model complexity.
  • Compare against a baseline and the current production model before celebrating an offline gain.
  • Treat real-world feedback signals as delayed labels with bias, lag, and coverage gaps; do not treat them as ground truth without analysis.

Every metric choice should state which mistake it makes cheaper, which mistake it makes more likely, and who absorbs that cost.

Data and Feature Hypotheses

Features should come from a theory of separation:

  • Text, categorical fields, numeric histories, graph relationships, recency, frequency, and aggregates are candidate signal families, not automatic features.
  • For every feature family, state why it should separate outcomes and how it could leak future information.
  • For noisy labels, consider adjudication, label confidence, soft targets, or confidence weighting.
  • For class imbalance, compare weighted loss, resampling, threshold movement, and calibrated decision rules.
  • For missing values, decide whether absence is informative, imputable, or a reason to abstain.
  • For outliers, decide whether to clip, bucket, investigate, or preserve them as rare but important signal.
  • For correlated features, check whether they are redundant, unstable, or proxies for unavailable future state.

Do not add model complexity until error analysis shows that the baseline is failing for a reason additional signal or capacity can plausibly fix.

Error Analysis Loop

After each baseline, training run, threshold change, or config change:

  1. Split mistakes into false positives, false negatives, abstentions, low-confidence cases, and system failures.
  2. Cluster errors by shared traits: language, entity type, source, time, geography, device, sparsity, recency, feature freshness, label source, or model version.
  3. Separate model mistakes from data bugs, label ambiguity, product ambiguity, instrumentation gaps, and serving mismatches.
  4. Trace each major cluster to one of four moves: better labels, better features, better threshold/config, or better product fallback.
  5. Preserve every important mistake as a regression test, eval slice, dashboard panel, or runbook entry.
  6. Write the next iteration as a falsifiable experiment, not a vague "improve model" task.

The strongest MLE loop is not train -> metric -> ship. It is mistake -> cluster -> hypothesis -> experiment -> evidence -> simpler system.

Observation Ledger

Keep a compact decision and evidence trail beside the code, PR, experiment report, or runbook:

Iteration:
Change:
Why this mattered:
Metric movement:
Slice movement:
False positives:
False negatives:
Unexpected errors:
Decision:
Tradeoff accepted:
Lesson captured:
Regression added:
Debt created:
Next iteration:

Use the ledger to make model work cumulative. The goal is for each iteration to make the next decision easier, not merely to produce another artifact.

Core Workflow

1. Define the Prediction Contract

Capture the product-level contract before writing model code:

  • Prediction target and decision owner
  • Input entity, output schema, confidence/calibration fields, and allowed latency
  • Batch, online, streaming, or hybrid serving mode
  • Fallback behavior when the model, feature store, or dependency is unavailable
  • Human review or override path for high-impact decisions
  • Privacy, retention, and audit requirements for inputs, predictions, and labels

Do not accept "improve the model" as a requirement. Tie the model to an observable product behavior and a measurable acceptance gate.

2. Lock the Data Contract

Every ML task needs an explicit data contract:

  • Entity grain and primary key
  • Label definition, label timestamp, and label availability delay
  • Feature timestamp, freshness SLA, and point-in-time join rules
  • Train, validation, test, and backtest split policy
  • Required columns, allowed nulls, ranges, categories, and units
  • PII or sensitive fields that must not enter training artifacts or logs
  • Dataset version or snapshot ID for reproducibility

Guard against leakage first. If a feature is not available at prediction time, or is joined using future information, remove it or move it to an analysis-only path.

3. Build a Reproducible Pipeline

Training code should be runnable by another engineer without hidden notebook state:

  • Use typed config files or dataclasses for all hyperparameters and paths
  • Pin package and model dependencies
  • Set random seeds and document any nondeterministic GPU behavior
  • Record dataset version, code SHA, config hash, metrics, and artifact URI
  • Save preprocessing logic with the model artifact, not separately in a notebook
  • Keep train, eval, and inference transformations shared or generated from one source
  • Make every step idempotent so retries do not corrupt artifacts or metrics

Prefer immutable values and pure transformation functions. Avoid mutating shared data frames or global config during feature generation.

import hashlib
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class TrainingConfig:
    dataset_uri: str
    model_dir: Path
    seed: int
    learning_rate: float
    batch_size: int


def artifact_name(config: TrainingConfig, code_sha: str) -> str:
    config_key = f"{config.dataset_uri}:{config.seed}:{config.learning_rate}:{config.batch_size}"
    config_hash = hashlib.sha256(config_key.encode("utf-8")).hexdigest()[:12]
    return f"{code_sha[:12]}-{config_hash}"

4. Evaluate Before Promotion

Promotion criteria should be declared before training finishes:

  • Baseline model and current production model comparison
  • Primary metric aligned to product behavior
  • Guardrail metrics for latency, calibration, fairness slices, cost, and error concentration
  • Slice metrics for important cohorts, geographies, devices, languages, or data sources
  • Confidence intervals or repeated-run variance when metrics are noisy
  • Failure examples reviewed by a human for high-impact models
  • Explicit "do not ship" thresholds
PROMOTION_GATES = {
    "auc": ("min", 0.82),
    "calibration_error": ("max", 0.04),
    "p95_latency_ms": ("max", 80),
}


def assert_promotion_ready(metrics: dict[str, float]) -> None:
    missing = sorted(name for name in PROMOTION_GATES if name not in metrics)
    if missing:
        raise ValueError(f"Model promotion metrics missing required gates: {missing}")

    failures = {
        name: value
        for name, (direction, threshold) in PROMOTION_GATES.items()
        for value in [metrics[name]]
        if (direction == "min" and value < threshold)
        or (direction == "max" and value > threshold)
    }
    if failures:
        raise ValueError(f"Model failed promotion gates: {failures}")

Use offline metrics as gates, not guarantees. When the model changes product behavior, plan shadow evaluation, canary rollout, or A/B testing before full rollout.

5. Package for Serving

An ML artifact is production-ready only when the serving contract is testable:

  • Model artifact includes version, training data reference, config, and preprocessing
  • Input schema rejects invalid, stale, or out-of-range features
  • Output schema includes model version and confidence or explanation fields when useful
  • Serving path has timeout, batching, resource limits, and fallback behavior
  • CPU/GPU requirements are explicit and tested
  • Prediction logs avoid PII and include enough identifiers for debugging and label joins
  • Integration tests cover missing features, stale features, bad types, empty batches, and fallback path

Never let training-only feature code diverge from serving feature code without a test that proves equivalence.

6. Operate the Model

Model monitoring needs both system and quality signals:

  • Availability, error rate, timeout rate, queue depth, and p50/p95/p99 latency
  • Feature null rate, range drift, categorical drift, and freshness drift
  • Prediction distribution drift and confidence distribution drift
  • Label arrival health and delayed quality metrics
  • Business KPI guardrails and rollback triggers
  • Per-version dashboards for canaries and rollbacks

Every deployment should have a rollback plan that names the previous artifact, config, data dependency, and traffic-switch mechanism.

Review Checklist

  • Prediction contract is explicit and testable
  • Data contract defines entity grain, label timing, feature timing, and snapshot/version
  • Leakage risks were checked against prediction-time availability
  • Training is reproducible from code, config, data version, and seed
  • Metrics compare against baseline and current production model
  • Slice metrics and guardrails are included for high-risk cohorts
  • Promotion gates are automated and fail closed
  • Training and serving transformations are shared or equivalence-tested
  • Model artifact carries version, config, dataset reference, and preprocessing
  • Serving path validates inputs and has timeout, fallback, and rollback behavior
  • Monitoring covers system health, feature drift, prediction drift, and delayed labels
  • Sensitive data is excluded from artifacts, logs, prompts, and examples

Anti-Patterns

  • Notebook state is required to reproduce the model
  • Random split leaks future data into validation or test sets
  • Feature joins ignore event time and label availability
  • Offline metric improves while important slices regress
  • Thresholds are tuned on the test set repeatedly
  • Training preprocessing is copied manually into serving code
  • Model version is missing from prediction logs
  • Monitoring only checks service uptime, not data or prediction quality
  • Rollback requires retraining instead of switching to a known-good artifact

Output Expectations

When using this skill, return concrete artifacts: data contract, promotion gates, pipeline steps, test plan, deployment plan, or review findings. Call out unknowns that block production readiness instead of filling them with assumptions.

Model Serving on Kubernetes

Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.

When to Use This Skill

Use this skill when:

  • Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
  • Implementing canary deployments and A/B testing for ML models
  • Autoscaling inference pods based on request rate or GPU metrics
  • Deploying LLMs with Triton or KServe on Kubernetes
  • Managing multiple model versions with traffic splitting

Prerequisites

These are version-dependent sketches, not a complete cluster installation. Pin chart/image versions and check installed CRD schemas and serving mode (Knative versus standard). Canary fields and KEDA targets depend on that mode; target an actual scalable workload supported by the chosen operator, not an assumed InferenceService scale subresource. Configure ingress authentication/TLS, least-privilege model-store access and resource/cost bounds. Remove or implement the placeholder transformer; arbitrary models and “-v2” IDs must exist and be licensed.

  • Kubernetes 1.28+ with GPU nodes
  • KServe installed (or Triton standalone)
  • kubectl and helm configured
  • NVIDIA GPU Operator installed on cluster

KServe Installation

# Install KServe with Helm
helm repo add kserve https://kserve.github.io/helm-charts
helm repo update

helm install kserve kserve/kserve \
  --namespace kserve \
  --create-namespace \
  --set kserve.controller.gateway.ingressGateway.className=nginx

# Verify
kubectl get pods -n kserve
kubectl get crd | grep kserve

Basic InferenceService (KServe)

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: models
spec:
  predictor:
    sklearn:
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi
kubectl apply -f inference-service.yaml

# Get inference service URL
kubectl get inferenceservice sklearn-iris -n models
# NAME           URL                                          READY   ...
# sklearn-iris   http://sklearn-iris.models.example.com       True

# Test prediction
curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'

GPU-Enabled LLM InferenceService

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-8b
  namespace: models
  annotations:
    serving.kserve.io/enable-prometheus-scraping: "true"
spec:
  predictor:
    containers:
    - name: vllm-container
      image: vllm/vllm-openai:latest
      args:
      - "--port"
      - "8080"
      - "--model"
      - "meta-llama/Llama-3.1-8B-Instruct"
      - "--tensor-parallel-size"
      - "1"
      - "--gpu-memory-utilization"
      - "0.90"
      ports:
      - containerPort: 8080
        protocol: TCP
      resources:
        requests:
          nvidia.com/gpu: "1"
          memory: "20Gi"
          cpu: "4"
        limits:
          nvidia.com/gpu: "1"
          memory: "24Gi"
          cpu: "8"
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        initialDelaySeconds: 60
        periodSeconds: 10
      env:
      - name: HUGGING_FACE_HUB_TOKEN
        valueFrom:
          secretKeyRef:
            name: hf-token
            key: token
    nodeSelector:
      nvidia.com/gpu.present: "true"
  transformer:
    containers:
    - name: kserve-container
      image: kserve/kserve-transformer:latest

Canary Deployment (Traffic Splitting)

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-8b
  namespace: models
spec:
  predictor:
    canaryTrafficPercent: 20    # 20% to new version, 80% to stable
    containers:
    - name: vllm-container
      image: vllm/vllm-openai:latest
      args:
      - "--model"
      - "meta-llama/Llama-3.1-8B-Instruct-v2"  # new model version
      resources:
        limits:
          nvidia.com/gpu: "1"
# Gradually increase canary traffic
kubectl patch inferenceservice llama-3-8b -n models \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]'

# Promote canary to stable
kubectl patch inferenceservice llama-3-8b -n models \
  --type='json' \
  -p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]'

Autoscaling with KEDA

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama-scaler
  namespace: models
spec:
  scaleTargetRef:
    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    name: llama-3-8b
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: kserve_request_count
      threshold: "10"
      query: |
        sum(rate(kserve_request_count_total{namespace="models",
            service="llama-3-8b"}[1m]))

NVIDIA Triton Inference Server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-server
  namespace: models
spec:
  replicas: 2
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:24.05-py3
        args:
        - "tritonserver"
        - "--model-store=s3://my-model-store/models"
        - "--model-control-mode=poll"        # auto-load new model versions
        - "--repository-poll-secs=30"
        - "--metrics-port=8002"
        ports:
        - containerPort: 8000   # HTTP
        - containerPort: 8001   # gRPC
        - containerPort: 8002   # Metrics
        resources:
          limits:
            nvidia.com/gpu: "1"
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
          initialDelaySeconds: 30

Triton Model Repository Structure

s3://my-model-store/models/
├── text-classifier/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx          # new version; auto-loaded
├── embedding-model/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
# config.pbtxt for ONNX model
name: "text-classifier"
backend: "onnxruntime"
max_batch_size: 64
dynamic_batching {
  preferred_batch_size: [16, 32]
  max_queue_delay_microseconds: 1000
}
input [
  { name: "input_ids" data_type: TYPE_INT64 dims: [-1] }
  { name: "attention_mask" data_type: TYPE_INT64 dims: [-1] }
]
output [
  { name: "logits" data_type: TYPE_FP32 dims: [-1] }
]
instance_group [
  { kind: KIND_GPU count: 2 }   # 2 model instances on GPU
]

Model Management Commands

# List loaded models (Triton)
curl -X POST http://triton:8000/v2/repository/index

# Load a model (requires explicit model-control mode, unlike poll mode above)
curl -X POST http://triton:8000/v2/repository/models/text-classifier/load

# Unload a model (requires explicit model-control mode)
curl -X POST http://triton:8000/v2/repository/models/text-classifier/unload

# KServe — watch rollout status
kubectl rollout status deployment/llama-3-8b-predictor -n models
kubectl get inferenceservice llama-3-8b -n models -w

Common Issues

IssueCauseFix
InferenceService not readyModel loading or OOMCheck predictor pod logs; increase memory limits
Canary stuck at 0%KNative routing issueCheck kubectl get ksvc -n models
Triton missing modelS3 permissions or pathVerify IAM role; check --model-store path
Low GPU utilizationDynamic batching offEnable dynamic_batching in Triton config
Autoscaler not triggeringPrometheus query wrongTest query in Prometheus UI

Best Practices

  • Use canary deployments for all model updates — roll back in seconds if metrics degrade.
  • Enable Triton dynamic batching — benchmark throughput and latency for your model and request distribution.
  • Store models in S3/GCS with versioned paths (s3://bucket/model/v1/, v2/).
  • Pin GPU node selectors to prevent model pods landing on CPU-only nodes.
  • Monitor p99 latency and error rates per model version during canary rollouts.

Related Skills

  • vllm-server (vllm-server) - vLLM for LLM serving
  • llm-inference-scaling (llm-inference-scaling) - KEDA autoscaling
  • kubernetes-ops (kubernetes-ops) - Core Kubernetes operations
  • gpu-server-management (gpu-server-management) - GPU nodes

Limitations

  • Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
  • Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.

Example

git status && git diff --stat
kubectl diff -f manifest.yaml

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.

Preference Optimization

This skill assumes finetuning-method-selection already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's lora-qlora-recipes) or a verifiable reward signal (that's grpo-rlvr-training). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place.

Input: a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. Output format: a validated method choice plus a config — the kwarg values in references/method-configs.md, not free-form advice — that llm-finetuning-training-engineer consumes directly.

Method Selection

Data shapeMethodKey parameters
Preference pairs, default caseDPOβ=0.1, LR 5e-7–1e-6, 1–2 epochs
Memory-bound or no SFT checkpointORPOreference-free, fused SFT+preference in one loss
Unpaired thumbs-up/downKTObinary label per example, no pairing needed
Length bias observed, sweep budget availableSimPOreference-free; see sweep grid below
  • DPO is the safe default. Use β=0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs. This LR is lower than the SFT LR that produced the checkpoint being aligned — porting an SFT- scale LR into a DPO run is the most common misconfiguration here, not an edge case.
  • ORPO routes in when memory is the constraint, or when there's no separate SFT checkpoint to start from — it's reference-free and fuses the SFT and preference objectives into one loss, skipping the separate SFT pass and the reference-model memory cost DPO carries.
  • KTO routes in when feedback is unpaired binary signal (thumbs-up/down) rather than matched preference pairs — don't force unpaired feedback into synthetic pairs to use DPO instead.
  • SimPO fixes DPO's length bias but only pays off with disciplined sweeping — its published gains are a ceiling reported under a tuned sweep, not a baseline any single config will reproduce. Route here only when there's sweep budget; use DPO instead if there isn't.
  • Classic RLHF (reward model + PPO) is retired outside frontier labs. Don't reach for it in a production pipeline — every method above is cheaper and better-supported for the same data shapes.

Worked Examples

  • "We have an SFT checkpoint and clean paired preference data, no length-bias complaints yet." → default case → DPO at β=0.1.
  • "Reviewers click thumbs-up/down per response; nothing is paired." → unpaired signal → KTO, not DPO — don't synthesize pairs to force DPO onto unpaired data.
  • "GPU budget doesn't cover a separate SFT pass plus a DPO reference model." → memory-bound, no separate checkpoint → ORPO.
  • "DPO output favors longer answers regardless of quality, and there's time to run a sweep." → length bias plus sweep budget → SimPO. Skip it if the sweep budget isn't actually there.

The Low-Leverage Truth

A 2026 240-H100-run study (arXiv 2603.19335) is the load-bearing evidence behind the table above: loss-function choice is worth roughly 1 percentage point of leverage, model scale is worth roughly 50. Zero of 20 DPO variants tested beat vanilla DPO. Rankings also invert with scale — a variant that wins in a small pilot can lose at deployment size.

Two practical consequences:

  • Don't spend a routing decision agonizing over DPO-variant bake-offs. The table above is sufficient; deeper variant selection is low-leverage compared to data quality and scale.
  • Validate at deployment scale before trusting a ranking. A method comparison run on a small pilot model doesn't transfer to the production size class — re-check the winner once scale changes.

This is also why the Method Selection table above is deliberately short: it encodes the ~1pp lever, not a ranking of DPO variants that the same study shows doesn't hold up across scale. Treat any variant-selection advice that isn't in that table — including advice that claims a specific variant "wins" — as unproven until it's been validated at the target deployment size.

Production Pattern: Iterative On-Policy DPO

A single offline DPO pass on a static preference dataset is a starting point, not the production pattern. The policy drifts away from the distribution the pairs were sampled from as training proceeds, and a static dataset goes stale against that drift. Production pipelines run DPO iteratively and on-policy instead:

  1. Sample completions from the current policy checkpoint.
  2. Score or rank the completions (reward model, judge, or task grader).
  3. Run a DPO pass using the current checkpoint as the reference model.
  4. The resulting checkpoint becomes both the new policy and the new reference for the next round.

Repeat. Each round's reference model is the prior round's output, not a fixed initial checkpoint — that's what keeps the preference signal on-policy instead of scoring against an increasingly stale distribution.

A single-pass DPO run is still a reasonable first iteration — it just isn't the whole pipeline. Plan for at least one more round once the first checkpoint exists, rather than treating pass one as the finished artifact.

Pair Construction

Build DPO/ORPO pairs from same-task passing-vs-failing trajectories — two attempts at the same underlying task, not unrelated best-and-worst examples pulled from different tasks. Within that trajectory set, select the rejected member at μ−2σ of the reward distribution, never the minimum. Naive best-vs-worst pair construction (max reward vs. absolute minimum) degrades as scale increases; the μ−2σ selection is more robust to the same scale sensitivity the low-leverage study surfaced above.

sorted_by_reward = sort(trajectories, key=reward)
chosen   = sorted_by_reward[-1]                # highest reward
mu, sigma = mean(rewards), stdev(rewards)
rejected = closest(sorted_by_reward, mu - 2 * sigma)
# NOT sorted_by_reward[0] — the absolute minimum
# is the naive best-vs-worst construction that
# degrades as scale increases.

For the mechanics of turning graded traces into these pairs — including rejection sampling and judge-scored delta selection — see trace-to-training-data.

References

Complete TRL config blocks per method — DPOConfig, ORPOConfig, KTOConfig, and the SimPO sweep grid — plus Unsloth wrappers and a catastrophic-forgetting note live in references/method-configs.md. Those configs use the same current-TRL API conventions established in lora-qlora-recipes's references/unsloth-trl-mapping.md (processing_class, not tokenizer=).

references/method-configs.md also carries the catastrophic-forgetting note: a too-high learning rate is the usual cause when a preference-tuned checkpoint loses general capability, and the fix is almost always to drop the LR toward the low end of the range in the Method Selection table above before reaching for any other remediation.

Related skills: finetuning-method-selection routes here once preference pairs or unpaired feedback exist; lora-qlora-recipes produces the SFT checkpoint DPO/KTO/SimPO align (ORPO's fused path can skip it); trace-to-training-data converts passing/failing trajectories into the pairs this skill's Pair Construction section consumes.

PyTorch Development Patterns

Idiomatic PyTorch patterns and best practices for building robust, efficient, and reproducible deep learning applications.

When to Activate

  • Writing new PyTorch models or training scripts
  • Reviewing deep learning code
  • Debugging training loops or data pipelines
  • Optimizing GPU memory usage or training speed
  • Setting up reproducible experiments

Core Principles

1. Device-Agnostic Code

Always write code that works on both CPU and GPU without hardcoding devices.

# Good: Device-agnostic
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = MyModel().to(device)
data = data.to(device)

# Bad: Hardcoded device
model = MyModel().cuda()  # Crashes if no GPU
data = data.cuda()

2. Reproducibility First

Set all random seeds for reproducible results.

# Good: Full reproducibility setup
def set_seed(seed: int = 42) -> None:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

# Bad: No seed control
model = MyModel()  # Different weights every run

3. Explicit Shape Management

Always document and verify tensor shapes.

# Good: Shape-annotated forward pass
def forward(self, x: torch.Tensor) -> torch.Tensor:
    # x: (batch_size, channels, height, width)
    x = self.conv1(x)    # -> (batch_size, 32, H, W)
    x = self.pool(x)     # -> (batch_size, 32, H//2, W//2)
    x = x.view(x.size(0), -1)  # -> (batch_size, 32*H//2*W//2)
    return self.fc(x)    # -> (batch_size, num_classes)

# Bad: No shape tracking
def forward(self, x):
    x = self.conv1(x)
    x = self.pool(x)
    x = x.view(x.size(0), -1)  # What size is this?
    return self.fc(x)           # Will this even work?

Model Architecture Patterns

Clean nn.Module Structure

# Good: Well-organized module
class ImageClassifier(nn.Module):
    def __init__(self, num_classes: int, dropout: float = 0.5) -> None:
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(dropout),
            nn.Linear(64 * 16 * 16, num_classes),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.features(x)
        x = x.view(x.size(0), -1)
        return self.classifier(x)

# Bad: Everything in forward
class ImageClassifier(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        x = F.conv2d(x, weight=self.make_weight())  # Creates weight each call!
        return x

Proper Weight Initialization

# Good: Explicit initialization
def _init_weights(self, module: nn.Module) -> None:
    if isinstance(module, nn.Linear):
        nn.init.kaiming_normal_(module.weight, mode="fan_out", nonlinearity="relu")
        if module.bias is not None:
            nn.init.zeros_(module.bias)
    elif isinstance(module, nn.Conv2d):
        nn.init.kaiming_normal_(module.weight, mode="fan_out", nonlinearity="relu")
    elif isinstance(module, nn.BatchNorm2d):
        nn.init.ones_(module.weight)
        nn.init.zeros_(module.bias)

model = MyModel()
model.apply(model._init_weights)

Training Loop Patterns

Standard Training Loop

# Good: Complete training loop with best practices
def train_one_epoch(
    model: nn.Module,
    dataloader: DataLoader,
    optimizer: torch.optim.Optimizer,
    criterion: nn.Module,
    device: torch.device,
    scaler: torch.amp.GradScaler | None = None,
) -> float:
    model.train()  # Always set train mode
    total_loss = 0.0

    for batch_idx, (data, target) in enumerate(dataloader):
        data, target = data.to(device), target.to(device)

        optimizer.zero_grad(set_to_none=True)  # More efficient than zero_grad()

        # Mixed precision training
        with torch.amp.autocast("cuda", enabled=scaler is not None):
            output = model(data)
            loss = criterion(output, target)

        if scaler is not None:
            scaler.scale(loss).backward()
            scaler.unscale_(optimizer)
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            scaler.step(optimizer)
            scaler.update()
        else:
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            optimizer.step()

        total_loss += loss.item()

    return total_loss / len(dataloader)

Validation Loop

# Good: Proper evaluation
@torch.no_grad()  # More efficient than wrapping in torch.no_grad() block
def evaluate(
    model: nn.Module,
    dataloader: DataLoader,
    criterion: nn.Module,
    device: torch.device,
) -> tuple[float, float]:
    model.eval()  # Always set eval mode — disables dropout, uses running BN stats
    total_loss = 0.0
    correct = 0
    total = 0

    for data, target in dataloader:
        data, target = data.to(device), target.to(device)
        output = model(data)
        total_loss += criterion(output, target).item()
        correct += (output.argmax(1) == target).sum().item()
        total += target.size(0)

    return total_loss / len(dataloader), correct / total

Data Pipeline Patterns

Custom Dataset

# Good: Clean Dataset with type hints
class ImageDataset(Dataset):
    def __init__(
        self,
        image_dir: str,
        labels: dict[str, int],
        transform: transforms.Compose | None = None,
    ) -> None:
        self.image_paths = list(Path(image_dir).glob("*.jpg"))
        self.labels = labels
        self.transform = transform

    def __len__(self) -> int:
        return len(self.image_paths)

    def __getitem__(self, idx: int) -> tuple[torch.Tensor, int]:
        img = Image.open(self.image_paths[idx]).convert("RGB")
        label = self.labels[self.image_paths[idx].stem]

        if self.transform:
            img = self.transform(img)

        return img, label

Efficient DataLoader Configuration

# Good: Optimized DataLoader
dataloader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,            # Shuffle for training
    num_workers=4,           # Parallel data loading
    pin_memory=True,         # Faster CPU->GPU transfer
    persistent_workers=True, # Keep workers alive between epochs
    drop_last=True,          # Consistent batch sizes for BatchNorm
)

# Bad: Slow defaults
dataloader = DataLoader(dataset, batch_size=32)  # num_workers=0, no pin_memory

Custom Collate for Variable-Length Data

# Good: Pad sequences in collate_fn
def collate_fn(batch: list[tuple[torch.Tensor, int]]) -> tuple[torch.Tensor, torch.Tensor]:
    sequences, labels = zip(*batch)
    # Pad to max length in batch
    padded = nn.utils.rnn.pad_sequence(sequences, batch_first=True, padding_value=0)
    return padded, torch.tensor(labels)

dataloader = DataLoader(dataset, batch_size=32, collate_fn=collate_fn)

Checkpointing Patterns

Save and Load Checkpoints

# Good: Complete checkpoint with all training state
def save_checkpoint(
    model: nn.Module,
    optimizer: torch.optim.Optimizer,
    epoch: int,
    loss: float,
    path: str,
) -> None:
    torch.save({
        "epoch": epoch,
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
        "loss": loss,
    }, path)

def load_checkpoint(
    path: str,
    model: nn.Module,
    optimizer: torch.optim.Optimizer | None = None,
) -> dict:
    checkpoint = torch.load(path, map_location="cpu", weights_only=True)
    model.load_state_dict(checkpoint["model_state_dict"])
    if optimizer:
        optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
    return checkpoint

# Bad: Only saving model weights (can't resume training)
torch.save(model.state_dict(), "model.pt")

Performance Optimization

Mixed Precision Training

# Good: AMP with GradScaler
scaler = torch.amp.GradScaler("cuda")
for data, target in dataloader:
    with torch.amp.autocast("cuda"):
        output = model(data)
        loss = criterion(output, target)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad(set_to_none=True)

Gradient Checkpointing for Large Models

# Good: Trade compute for memory
from torch.utils.checkpoint import checkpoint

class LargeModel(nn.Module):
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Recompute activations during backward to save memory
        x = checkpoint(self.block1, x, use_reentrant=False)
        x = checkpoint(self.block2, x, use_reentrant=False)
        return self.head(x)

torch.compile for Speed

# Good: Compile the model for faster execution (PyTorch 2.0+)
model = MyModel().to(device)
model = torch.compile(model, mode="reduce-overhead")

# Modes: "default" (safe), "reduce-overhead" (faster), "max-autotune" (fastest)

Quick Reference: PyTorch Idioms

IdiomDescription
model.train() / model.eval()Always set mode before train/eval
torch.no_grad()Disable gradients for inference
optimizer.zero_grad(set_to_none=True)More efficient gradient clearing
.to(device)Device-agnostic tensor/model placement
torch.amp.autocastMixed precision for 2x speed
pin_memory=TrueFaster CPU→GPU data transfer
torch.compileJIT compilation for speed (2.0+)
weights_only=TrueSecure model loading
torch.manual_seedReproducible experiments
gradient_checkpointingTrade compute for memory

Anti-Patterns to Avoid

# Bad: Forgetting model.eval() during validation
model.train()
with torch.no_grad():
    output = model(val_data)  # Dropout still active! BatchNorm uses batch stats!

# Good: Always set eval mode
model.eval()
with torch.no_grad():
    output = model(val_data)

# Bad: In-place operations breaking autograd
x = F.relu(x, inplace=True)  # Can break gradient computation
x += residual                  # In-place add breaks autograd graph

# Good: Out-of-place operations
x = F.relu(x)
x = x + residual

# Bad: Moving data to GPU inside the training loop repeatedly
for data, target in dataloader:
    model = model.cuda()  # Moves model EVERY iteration!

# Good: Move model once before the loop
model = model.to(device)
for data, target in dataloader:
    data, target = data.to(device), target.to(device)

# Bad: Using .item() before backward
loss = criterion(output, target).item()  # Detaches from graph!
loss.backward()  # Error: can't backprop through .item()

# Good: Call .item() only for logging
loss = criterion(output, target)
loss.backward()
print(f"Loss: {loss.item():.4f}")  # .item() after backward is fine

# Bad: Not using torch.save properly
torch.save(model, "model.pt")  # Saves entire model (fragile, not portable)

# Good: Save state_dict
torch.save(model.state_dict(), "model.pt")

Remember: PyTorch code should be device-agnostic, reproducible, and memory-conscious. When in doubt, profile with torch.profiler and check GPU memory with torch.cuda.memory_summary().

Quantized Export

The last stop after checkpoint-promotion hands off a PROMOTE verdict: a checkpoint that cleared the four-stage gate still isn't deployed until it's exported in the right format for its target runtime and proven to still work post-export. A REJECT verdict never reaches this skill — export starts only from a promoted checkpoint.

Input: a promoted checkpoint (or LoRA adapter) plus the target deployment surface — GPU class, serving stack, and whether long-context/code/math workloads are in scope. Output format: an exported artifact in the chosen format plus a smoke-test diff report comparing 3–5 golden outputs pre-export and post-export.

Format Map

Pick format by hardware and deployment shape, not by habit — the wrong pick either wastes throughput headroom or breaks silently on specific workloads (see Workload Overrides).

  • FP8 is the default on Hopper-class GPUs and newer. It preserves near-bf16 quality at roughly half the memory, and it's the safe first choice whenever the target GPU supports it and no edge-device constraint applies.
  • AWQ INT4 targets older GPUs that predate FP8 hardware support. GPTQ is superseded for new deployments — don't reach for it on a fresh export; AWQ has better accuracy retention at the same bit width and wider current tooling support.
  • GGUF with Q4_K_M quantization, built from an imatrix, is the edge/llama.cpp format. Use it for local or CPU-adjacent deployment, not for GPU-serving throughput — it optimizes for footprint, not tokens/sec on a datacenter GPU.
  • NVFP4 is for Blackwell-at-scale deployments only — and explicitly NOT on GB10. NVFP4 on SM121 (GB10) runs ~32% slower than FP8 because the hardware lacks a native cvt.e2m1x2 path unless the kernel is compiled sm_121a. Choosing NVFP4 on a GB10 target is a regression, not an upgrade — pick FP8 there instead.
  • Merged vs. LoRA-only is a separate axis from quant format. A merged export folds the adapter into the base weights: larger artifact, no base-model dependency at serve time. LoRA-only keeps the adapter separate: much smaller artifact, but the serving stack must load the exact same base model alongside it — a mismatched or wrong-revision base silently changes outputs. Pick merged when artifact portability matters more than storage; pick LoRA-only when disk footprint or multi-adapter serving matters more.

Worked Picks

The core format-selection tradeoff, read as a lookup table for common scenarios:

TargetWorkloadFormat
Datacenter GPUgeneric chatFP8
Datacenter GPUlong-context/code/mathFP8 or W8A8 — never INT4
Older GPU generationgenericAWQ INT4
Edge device / laptopllama.cpp servingGGUF Q4_K_M + imatrix
GB10any workloadFP8 via vLLM nightly, or GGUF via llama.cpp locally — skip NVFP4
# quick decision snippet — see the table above for the full map
hopper_or_newer: fp8
older_gpu: awq-int4
edge_llama_cpp: gguf-q4_k_m+imatrix
gb10_any_workload: fp8-vllm-nightly   # never nvfp4 on GB10

Workload Overrides

The Format Map above is a default, not a rule that survives every workload. Long-context, code, and math workloads break at INT4 — quantization error compounds across long sequences and precise token-level reasoning in ways that don't show up on short, generic prompts. For any of these three workload classes, stay on FP8 or W8A8 even if the target hardware would otherwise justify INT4 on cost grounds.

  • Don't validate this override with MMLU or similar broad-knowledge benchmarks — they don't stress the failure mode. Measure with the actual task evals — the goldens and graders from eval-harness-first, run through the exported artifact — because INT4 degradation on long-context, code, or math shows up as task-specific failures (dropped context, broken syntax, arithmetic errors) well before it moves a knowledge benchmark.
  • If a task eval regresses after an INT4 export on one of these three workload classes, the fix is switching format, not re-tuning the quantization recipe — AWQ and GPTQ variants at the same bit width share the same compounding-error failure mode on these workloads.

The Smoke Test

Export bugs are silent at the file level — a malformed export still produces a loadable artifact, so file-existence checks prove nothing. The smoke test is mandatory for every export, with no exception for a format that "should just work":

  1. Load the exported artifact in its actual target runtime — vLLM for FP8/AWQ, llama.cpp for GGUF, not a quick sanity load in a different framework than the one that will serve it in production.
  2. Run 3–5 golden prompts through it — pull these from the same eval/goldens.jsonl eval-harness-first maintains, not a fresh ad hoc set.
  3. Compare each output against the pre-export generation for the same prompt, same deterministic sampling settings — greedy decoding (temperature 0) and a fixed seed, persisted and reused between the pre- and post-export runs, not just nominally identical config. For a lossless export, byte match is the gate — any diff is a bug. For a lossy (quantized) export, byte match is expected to fail; the gate is task-grader verdict agreement instead — see references/export-commands.md's Smoke-Test Script Skeleton.

Run this as a gate, not a manual check:

python smoke_test.py "$EXPORT_PATH" \
    eval/goldens.jsonl pre-export-outputs.jsonl
# non-zero exit on any pre/post mismatch

Failure Signatures

What export bugs actually look like, not a clean pass/fail flag:

  • Template mismatch presents as garbled or run-on output — the chat template baked into the export doesn't match the one the checkpoint was trained and evaluated against, so turn boundaries or special tokens land in the wrong place.
  • Wrong quantization applied to lm_head presents as off-template or semantically nonsensical output that still looks fluent — the output head lost precision it needed even though the rest of the network quantized cleanly.

Never ship an export that skipped this step — a checkpoint's PROMOTE verdict says the un-exported checkpoint is good; it says nothing about the export pipeline. Re-run on any quant-method or runtime version bump, not only after the first export. Runnable command sequences for every format plus the smoke-test script skeleton: references/export-commands.md.

Related Skills

  • checkpoint-promotion — the only valid upstream source for this skill. A checkpoint without a PROMOTE verdict doesn't reach export.
  • eval-harness-first — owns the eval/goldens.jsonl this skill's smoke test draws its 3–5 prompts from, and the task evals the Workload Overrides section requires for long-context/code/math validation.
  • finetuning-method-selection — its references/model-catalog.md is the place to check hardware-class assumptions (which GPU generations a base model targets) before picking a format off the Format Map above.

Spark users: on GB10, GGUF via llama.cpp works well for local serving, and FP8 serving via vLLM nightly builds is the other proven path — NVFP4 is the one format to avoid there (see the Format Map exception above). Once the dgx-spark-ops plugin is installed, defer Spark-specific serving and thermal questions to its skills rather than re-deriving them here.

Recsys Pipeline Architect

A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. Encodes the six-stage pattern popularized by xAI's open-sourced For You algorithm (Apache 2.0) and applies it to any "top K for (user, context)" problem.

Overview

Most "recommendation systems" in production aren't exotic ML — they're pipelines: fetch candidates from one or more sources, enrich them with metadata, drop the ineligible, score the rest, sort and pick the top K, then fire async side effects. The pattern is universal. The scoring function and the items change; the pipeline shape doesn't.

This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.

When to Use This Skill

  • Building any system that returns "the top K items for a user/context"
  • Designing or refactoring a personalized feed (content, search results, notifications)
  • Wrapping an LLM/ML scorer in proper pipeline plumbing (sources, hydration, filters, side effects)
  • Adding multi-action prediction with tunable weights (instead of a single relevance score)
  • Building a RAG retrieval reranker (cheap retrieval → expensive rerank)
  • Designing a task prioritizer or alert triage system

The Six-Stage Framework

#StageJobParallel?
1SourceFetch candidates from one or more originsYes — multiple sources run in parallel
2HydratorEnrich candidates with metadata needed for filtering and scoringYes — independent hydrators run in parallel
3FilterDrop ineligible candidates (blocked, expired, duplicate, ineligible)Sequential — each filter sees fewer items
4ScorerAssign each surviving candidate one or more scoresSequential — later scorers see earlier scores
5SelectorSort by final score, return top KSingle op
6SideEffectCache, log, emit events, update served-historyAsync — must never block the response

Why this exact order

  • Sources before hydration: know what candidates exist before paying to enrich
  • Hydration before filtering: many filters need metadata the source didn't provide
  • Filtering before scoring: scoring is the expensive stage — drop the ineligible first
  • Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
  • Selector after scoring: keeps scoring deterministic and cacheable
  • SideEffects last and async: side effects must never block the user response

Workflow When Invoked

Walk the user through eight steps:

  1. Clarify the use case (one round, three questions only if missing): items being ranked, input context, language/runtime
  2. Identify the candidate sources (usually in-network + out-of-network, but single-source also valid)
  3. List required hydrations — for each filter and scorer, what data does it need that the source didn't provide?
  4. List the filters — cheap before expensive, universal before user-specific (duplicate, self, age, block/mute, previously-served, eligibility)
  5. Design the scorer chain — primary ML/heuristic → combiner (multi-action with weights) → diversity → business rules
  6. Selector — sort descending by final score, take top K (or stratified mix)
  7. SideEffects — cache served IDs, emit impression events, update counters, log analytics; all fire-and-forget
  8. Generate the scaffold in the user's stack

Key Trade-offs to Surface

Never default silently on these — they are product decisions disguised as technical ones.

1. Single score vs multi-action prediction

  • Single score: train one model to predict relevance. To change behavior → retrain.
  • Multi-action: predict P(action) for many actions (P(read), P(like), P(share), P(skip), P(report)), combine with weights at serving time. To change behavior → change weights. No retraining.

The X For You algorithm uses multi-action with both positive and negative weights. Recommend multi-action when the user expects to tune frequently.

2. Candidate isolation vs joint scoring

  • Isolated: each candidate scored independently. Deterministic, cacheable.
  • Joint: candidates attend to each other during scoring (e.g., transformer over the whole batch). More expressive but non-deterministic across batches.

Default to isolation. Joint only when there's a specific reason (e.g., explicit batch-aware diversity).

3. Online vs offline batch

  • Request-time (online): pipeline runs on each request. Latency budget: 100–300ms.
  • Pre-computed (offline batch): pipeline runs periodically, results cached. Lower latency, lower freshness.
  • Hybrid: candidate retrieval offline, ranking online.

Hard Rules

  1. Do not invent benchmark numbers. "How fast is this?" → "depends on workload, run it yourself."
  2. Attribution discipline. Attribute the pattern as "popularized by xAI's open-sourced For You algorithm" / github.com/xai-org/x-algorithm (Apache 2.0).
  3. No trademark use. Don't name the user's artifact "X-like" or use "For You" branding. Use neutral names: "candidate pipeline", "feed pipeline", "ranking pipeline".
  4. Surface trade-offs. Multi-action vs single, isolation vs joint, online vs offline — never default silently.
  5. The generated scaffold must run. No pseudocode passing as code.
  6. Filter order matters. Cheap before expensive. Universal before user-specific.
  7. Side effects never block. Wrap in fire-and-forget patterns (goroutines / promises without await / asyncio tasks).

Anti-Patterns

  • ❌ Scoring before filtering (wastes compute on candidates that will be dropped)
  • ❌ Synchronous side effects (cache writes / impression emits blocking the response)
  • ❌ A single "relevance" score when the product needs multi-objective tuning
  • ❌ Joint scoring as default (non-deterministic, uncacheable, doesn't compose with reranking)
  • ❌ Pseudocode "for illustration" — the scaffold must actually run

Common Use Cases

Content feed (Strapi v5 plugin, TypeScript)

User has a CMS with 50k articles, wants a personalized "for you" feed. Walk through 8 steps → generate a Strapi plugin scaffold with multi-action scoring, author diversity, standard filters, async side-effect lane.

RAG retrieval reranker (Python async)

User's RAG returns top-50 chunks from a vector DB, wants to rerank with a more expensive scorer and return top-5. Single-source pipeline with a scorer chain (cheap retrieval + expensive rerank).

Task prioritizer (FastAPI service)

User has a queue of incoming task suggestions, wants to rank by "what should this user work on next" considering their past patterns. Items reversed (tasks instead of content), same shape applies.

Notification triage (offline-batch job)

User wants a daily digest that picks the top 10 from the last 24h queue. Offline-batch pipeline. Source = queue, filters = age/dedup/eligibility, scorer = urgency × user-affinity, selector = top 10, side effect = email send (still async).

Upstream

This skill is a single-file adapter for the upstream repository, which ships 5 load-on-demand reference docs and 3 runnable example scaffolds (Strapi v5 / Go / Python — every one green on its test suite, 9/9 tests total).

  • Upstream: https://github.com/mturac/recsys-pipeline-architect
  • Release: v0.1.0 (MIT)
  • References: interfaces in 4 languages (TS/Go/Python/Rust), multi-action scoring, candidate isolation, filter cookbook (12 patterns), scorer cookbook
  • Cross-platform install: npx skills add mturac/recsys-pipeline-architect

Spark Environment Setup

DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121 GPU, 128GB unified memory, CUDA 13. This is a narrower and younger platform than a standard x86 CUDA 12 box, so package selection and ABI matching matter more than usual — the wheel ecosystem for aarch64 + CUDA 13 is still filling in.

When to Use This Skill

  • Setting up a fresh Spark box for training or inference.
  • Hitting an import error mentioning libcudart, a missing symbol, or a wheel that "installed fine but won't load."
  • A framework install (PyTorch, Unsloth, TRL, vLLM, xformers) fails, hangs, or silently falls back to CPU.
  • Deciding whether to use an NGC container or bare pip.
  • Restoring a working setup after an OS reinstall or a base-image update, needing to re-verify from scratch.

Each of these accepts the same general fix: match the container/wheel combination to CUDA 13 and SM121, don't fight the ABI.

Container-First Rule

Quick decision, before the detail below:

  • Standard training/inference work → NGC PyTorch container.
  • Unsloth-centric fine-tuning → Unsloth container (it ships the pinned Triton/xformers/transformers combination already validated for that path).
  • Neither fits (custom system package, local IDE interpreter) → bare pip, following the exact sequence further down.

Default to a container. Use nvcr.io/nvidia/pytorch:25.09-py3 as the base for general work — the newest tag confirmed working on this hardware; pull a newer blessed tag if locally available rather than hard-blocking on 25.11-py3. NGC's tag is dated, so running it directly is fine:

docker run --runtime=nvidia --gpus all -it --rm \
  nvcr.io/nvidia/pytorch:25.09-py3

unsloth/unsloth:dgxspark-latest is a moving tag by contrast — resolve and pin its digest before running it for anything reproducible; the bare tag is a discovery step only, not the default invocation. Full pull-inspect-pin sequence and flag rationale/volume mounts for finetuning/ run dirs: references/container-workflow.md. Treat bare pip as the exception.

The reason for the container-first stance is pinning, not convenience. Triton, xformers, and transformers versions interact narrowly with GB10's SM121 target and CUDA 13; a container locks all of them together against a combination already validated on this hardware. Bare pip leaves that resolution to you, one broken import at a time.

When bare pip is warranted, follow the NVIDIA playbook's install sequence verbatim and in order:

pip install "transformers==5.13.1" "peft==0.19.1" "hf_transfer==0.1.9" "datasets==4.3.0" "trl==1.8.0"
pip install --no-deps "unsloth==2026.7.2" "unsloth_zoo==2026.7.2" "bitsandbytes==0.49.2"
pip install -U "torchao==0.17.0"

The second command's --no-deps flag is not optional — letting pip re-resolve Unsloth's dependency tree on aarch64 is a common way to pull in an incompatible torch or triton build. The third line is not optional either: the NGC base image's bundled torchao is too old for current peft's LoRA-attach path (ImportError: ... torchao ... only versions above 0.16.0 are supported) — a hard blocker, not a warning. Every == pin above is load-bearing, taken from the dated known-good version matrix in references/stack-matrix.md (its Last verified date governs staleness) — an unpinned install resolves current PyPI versions well outside what this Unsloth release supports.

Pull a fresh tag when a new blessed release is announced. Rebuild locally from one of the two bases only when a project needs an extra system package layered in — not to "upgrade" a component the image already pins. Details on both paths: references/container-workflow.md.

One more preflight: official DGX Spark playbooks have shipped broken before. Check recent issues on github.com/NVIDIA/dgx-spark-playbooks (and the other resources in references/stack-matrix.md) before trusting a recipe verbatim for a long run.

The ABI Rule

The single most common failure on Spark is a CUDA 12/13 ABI mismatch: a wheel built against libcudart.so.12 loaded on a system that only has libcudart.so.13. The install usually succeeds; the failure surfaces later as a missing-symbol error or a segfault that doesn't obviously point at CUDA.

Fix: pull wheels from download.pytorch.org/whl/cu130 (the cu130-tagged aarch64 builds), or use one of the containers above, which already carry a matched build. Before chasing a stack trace that mentions a CUDA symbol, check which CUDA tag the installed wheel was built against:

python3 -c "import torch; print(torch.version.cuda)"

If that output doesn't start with 13, the ABI mismatch is the first thing to fix. NGC container builds (e.g. nvcr.io/nvidia/pytorch:25.09-py3) build torch internally against CUDA 13 with no +cu130 wheel tag — pip show torch won't say cu130 there, and that absence alone is not a failure.

Typical symptoms:

  • ImportError: undefined symbol referencing a CUDA runtime function.
  • A segfault on the first .cuda() call, no useful traceback.
  • A wheel that installs cleanly, then fails at import time — pip's resolver doesn't check CUDA ABI, only version constraints.
  • Two "identical" environments behaving differently — usually one has a cu130 wheel, the other a cu121/cu124 leftover.

The fix is the same regardless of symptom: match the wheel's CUDA tag to the system, or use a container that already does.

Component Quick Table

Condensed status for the components most likely to come up. Full table with wheel URLs, build flags, the sm_121 vs sm_121a distinction, and the dated known-good version matrix: references/stack-matrix.md.

ComponentStatus
PyTorch✅ official cu130 aarch64 wheels
bitsandbytes✅ works out of the box
Triton✅ needs the TRITON_PTXAS_PATH parameter set
flash-attn❌ skip pip build; NGC bundles a working one — see spark-training-gotchas G2
xformerssource build only (TORCH_CUDA_ARCH_LIST=12.1)
vLLMnightly wheels only
TransformerEngine / NVFP4 traincontainer-only

Everything else — Unsloth, Axolotl, TRL, PEFT — installs cleanly through the container-first path above. LLaMA-Factory and NeMo are fragile on Spark; check upstream issues first.

Verification Commands

Confirm the environment can actually see the GPU before running anything expensive:

import torch
print(torch.cuda.is_available(), torch.version.cuda)

This call returns two values; the exact output format is one line, <bool> <cuda-version>:

True 13.0

If it prints False instead, don't jump straight to a wheel reinstall — ABI mismatch is one cause among several:

HypothesisQuick check
Runtime/flagsnvidia-smi fails in-container too
Device visibilityecho $CUDA_VISIBLE_DEVICES
Permissionsls -l /dev/nvidia*
CUDA init statewedged process; retry fresh shell/container
ABI mismatch (usual culprit)torch.version.cuda not 13.x

Check nvidia-smi first — if it doesn't show the GPU, it's one of the first three, not ABI. Reinstall a wheel only once ABI is confirmed. Per-hypothesis detail: references/stack-matrix.md. Run right after the container starts, before installing project-specific packages.

One more check: if Triton kernel compilation fails once training starts, set TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas and retry — see references/stack-matrix.md for the full workaround list.

Next Steps

A verified environment is only the starting point. See also: spark-training-gotchas for failure preflights before a training run, and spark-memory-thermal-ops for unified-memory OOMs and thermal throttling during long ones.

Spark Memory & Thermal Ops

DGX Spark's GB10 chip has one 128GB unified memory (UMA) pool shared by CPU and GPU, and a sustained power ceiling well below its rated figure. Both break discrete-GPU assumptions: headroom isn't what nvidia-smi reports, and a run that starts fast will slow down mid-job with nothing misconfigured. This skill covers planning memory headroom, working an actual OOM, and watching thermals across a long job. For launch-time failure modes (ABI mismatches, flash-attn, playbook breakage), see spark-training-gotchas — this skill assumes the job starts.

Common Issues Quick Reference

SituationDo this
Planning headroom before launchBudget against free -g, not nvidia-smi — see UMA Memory Model
Job OOMs on unified memoryWork the OOM Ladder in order: flush, then batch/pack, then method downgrade
Throughput drops mid-runCheck the power/temp log before assuming a config bug — see Thermal Monitoring
Trainer + inference server both wantedRun one at a time — see Concurrent Workloads

When to Use This Skill

  • Sizing a training run against the 128GB pool before launch — will this model, method, and batch/pack combination fit.
  • A run OOMs mid-load or mid-step and the remediation order matters — what to try first, second, third.
  • Watching temperature and power during a multi-hour job, deciding whether a slowdown is thermal throttling or something else.
  • Planning to run a trainer alongside an inference server (vLLM, Ollama) on the same box.

UMA Memory Model

Spark has no separate GPU VRAM — the GPU and CPU share one 128GB pool. Two consequences:

  • nvidia-smi and cudaMemGetInfo underreport pressure — or report nothing at all. Both report CUDA-allocator-visible memory, not the pool's actual state — a box can show headroom in nvidia-smi and still OOM, because page-cache and mmap'd pages the allocator doesn't see consume the same pool. On some driver/setups, the memory query returns [N/A], [N/A] outright instead of a number — a script grepping for a numeric value there gets nothing, not a misleading undercount (see spark-training-gotchas gotcha G3).

  • Model load is a transient peak, not the steady state. Loading safetensors weights mmaps the file, then copies into CUDA tensors — for a window during load, both the mmap'd pages and the CUDA copy count against the pool at once. A model that fits while training can still OOM during load if headroom was sized for the post-load footprint instead of this doubled transient.

Plan and diagnose with free -g, not nvidia-smi:

free -g | awk 'NR==2 {print "free:", $4, "GB"}'

Rule of thumb: take that free figure, subtract a few GB for OS/driver overhead, and budget against the result — not the 128GB spec number. The worksheet in references/uma-accounting.md accepts parameter count, dtype, and method as input, and returns a memory estimate to compare against known anchors.

Planning Sequence

Before launch, work through these in order:

  1. Read free -g; subtract OS/driver overhead for the budget.
  2. Estimate weights + optimizer + gradients + activations from references/uma-accounting.md.
  3. Compare against the closest anchor (70B QLoRA, 27B LoRA, 9B full FT), not the estimate alone.
  4. If the estimate is close to the budget, start with shorter packing or a smaller batch — cheaper than hitting the OOM Ladder mid-run.

Example: Sizing a 70B QLoRA Run

A sanity check of the worksheet formula against the ≈40GB anchor:

params = 70e9
weights_gb = params * 0.5 / 1e9      # NF4, step 1
adapter_gb = 0.5                     # step 5, negligible
total_gb = weights_gb + adapter_gb   # + activations
print(f"{total_gb:.0f}GB before activations")

Weights alone land near the ≈40GB anchor — a plan estimating far above that for the same model class is a signal to recheck dtype and method.

The OOM Ladder

When a job OOMs on unified memory, work this ladder in order. Each step is more disruptive than the last — don't skip ahead: reducing batch size is never step 1.

  1. Flush the buffer cache. Page cache from a previous run or a large dataset read often accounts for GB of the "missing" headroom. This costs nothing but a rerun and doesn't touch the job's configuration:

    sync; echo 3 > /proc/sys/vm/drop_caches
    

    Needs root; a between-run reset, not a mid-training step. See spark-training-gotchas (gotcha G3) for the full diagnostic behind this step.

  2. Reduce batch size or packing length. Only after a flush fails to free enough headroom, cut batch size or packing length — the first step that changes what the run does. Prefer packing length first; it drives activation footprint more directly at long context.

  3. Downgrade the method: bf16 LoRA before QLoRA. If flushing and shrinking batch/pack still OOM, drop the method a tier — bf16 LoRA is next, not the reverse. QLoRA's bitsandbytes dequantization buffers are transient CUDA-side allocations that can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller. A QLoRA OOM is not proof the model doesn't fit.

Fall back further (smaller model, multi-Spark) only after all three steps and the job still won't fit.

Thermal Monitoring

Multi-hour runs push into Spark's sustained power ceiling, well under the rated figure — expected platform behavior, not a symptom to explain away:

  • Sample temperature and power alongside the training logs, not after a slowdown is noticed — every 30-60 seconds correlates a throughput drop with a thermal event. Keep the CSV output format assets/thermal-sample.sh writes, so timestamps line up against the log:

    bash assets/thermal-sample.sh 30 thermal.log
    
  • A sustained ~100W power draw is the platform cap, not a configuration bug. Don't re-tune batch size or precision to "fix" a plateau that's the box behaving normally under load. If temperature climbs while power stays flat under the rated 240W figure, that's the signature to recognize.

  • Log throttle events explicitly instead of letting a run silently slow down unrecorded. A run whose per-step time doubles two hours in should show that in the log, correlated against the thermal sample at that timestamp. Full throttling diagnostics: spark-training-gotchas (gotcha G4).

Concurrent Workloads

Because the 128GB pool is global, eviction happens without either process's logs showing an OOM:

  • The one-heavy-job rule applies to uncapped or near-capacity workloads — an uncapped trainer and inference server (vLLM, Ollama) compete for the same pool. A small, capped workload doesn't: a <4GB LoRA fine-tune coexists fine alongside vLLM capped at gpu-memory-utilization<=0.5 — check the other process's cap, not just its presence, before stopping it.

  • Inference servers evict trainer pages silently under uncapped/near-capacity contention, and vice versa — neither logs an error, so a slow run or lost KV cache is a contention symptom to check for. Stop unrelated uncapped servers before a long or full-pool run.

Check for GPU-resident processes first:

ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep

This procedure complements spark-training-gotchas (gotchas G3, G4, G6) — that skill covers launch-time failures; this one, the running job.

Memory math worksheets: references/uma-accounting.md.

Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

  • A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
  • A run OOMs while nvidia-smi still shows headroom.
  • Throughput degrades partway through a run that started fine.
  • Before any multi-hour or multi-epoch job on GB10.
  • Wiring two Sparks together, before picking a parallelism strategy.
  • Choosing between FP8 and NVFP4 for a Spark-hosted run.

Common Issues Quick Reference

#SymptomFix
G1undefined symbol / segfaultcu130 wheel or container
G2flash-attn wrong backend usedskip pip build; monkeypatch on NGC
G3OOM despite headroomdrop page cache
G4throughput drop / rebootexpect ~100W sustained cap
G5memory-bound step slowbudget 180–192 GB/s
G6cache evicted mid-runone GPU server at a time
G7NVFP4 slower than FP8stay FP8 unless sm_121a
G8playbook fails outrightcheck upstream issues
G9env breaks after installuse a container
G102-Spark TP hangsDDP/FSDP only, never TP

The Ten Gotchas

G1: CUDA 12/13 ABI Mismatch

  • SYMPTOM: ImportError: undefined symbol naming a CUDA function, or a segfault on the first .cuda() call.
  • CAUSE: most PyPI wheels link libcudart.so.12; Spark ships CUDA 13. pip never checks CUDA ABI, so it surfaces only at import or first kernel launch.
  • CHECK: references/gotcha-checks.md G1 — the wheel's CUDA build tag.
  • FIX: reinstall from download.pytorch.org/whl/cu130 or use a matched container.

G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect

  • SYMPTOM: pip install flash-attn still fails/hangs. Unsloth may also silently train flash-attn over an explicitly requested SDPA.
  • CAUSE: no aarch64/sm_121 wheel for bare pip — but NGC containers ship a working SM121 flash-attn, and Unsloth auto-prefers it, dropping attn_implementation="sdpa".
  • CHECK: references/gotcha-checks.md G2 — is flash-attn already present and working.
  • FIX: bare pip — skip flash-attn, use SDPA (unchanged). On NGC — the only reliable override is the monkeypatch in references/gotcha-checks.md G2.

G3: UMA OOM Below 128GB

  • SYMPTOM: OOM during model load/training while nvidia-smi still reports free memory under the 128GB cap — or, on some setups, [N/A] outright instead of a number.
  • CAUSE: mmap and the CUDA allocator double-count pages during safetensors load; QLoRA can OOM earlier than bf16 since dequantization adds transient allocs.
  • CHECK: references/gotcha-checks.md G3 — read free -g and /proc/meminfo, not nvidia-smi.
  • FIX: drop the page cache with sync; echo 3 > /proc/sys/vm/drop_caches — needs root, a between-run reset, not a mid-training step.

G4: Thermal Throttling

  • SYMPTOM: throughput drops partway through a multi-hour run, or the box spontaneously reboots under sustained load.
  • CAUSE: sustained power draw caps around 100W versus the 240W rated figure; long runs push into that ceiling and throttle or, sometimes, reboot.
  • CHECK: references/gotcha-checks.md G4 — sample nvidia-smi --query-gpu=temperature.gpu,power.draw.
  • FIX: if power plateaus under 240W while temperature climbs, treat throttling as the cause; improve cooling or cap run length.

G5: Bandwidth Ceiling

  • SYMPTOM: memory-bound workloads, decode-heavy RL loops especially, plateau well below expected throughput.
  • CAUSE: 273 GB/s is a spec ceiling, not sustained; measured bandwidth runs 180–192 GB/s.
  • CHECK: references/gotcha-checks.md G5 — observed step time vs. the measured range, not spec.
  • FIX: budget throughput from 180–192 GB/s; revise a plan built on the 273 GB/s figure.

G6: Global UMA Resource Contention

  • SYMPTOM: a process's KV cache/weights get evicted mid-run silently, no OOM in its own logs.
  • CAUSE: unified memory is one global pool; an uncapped or near-capacity process competes with anything else and can evict it. A small, bounded workload doesn't — a <4GB LoRA coexists fine alongside vLLM capped at gpu-memory-utilization<=0.5.
  • CHECK: references/gotcha-checks.md G6 — other GPU-resident processes and whether capped.
  • FIX: the one-heavy-job rule applies to uncapped or near-capacity workloads — cap or stop unrelated servers first. A small, capped workload need not stop.

G7: NVFP4 Slower Than FP8 on SM121

  • SYMPTOM: switching an inference workload from FP8 to NVFP4 on Spark makes it slower, not faster.
  • CAUSE: SM121 lacks cvt.e2m1x2 unless kernels target sm_121a; NVFP4 runs ~32% slower without it.
  • CHECK: references/gotcha-checks.md G7 — capability reports (12, 1); does the build target sm_121a?
  • FIX: stay on FP8 unless the build targets sm_121a.

G8: Stale Official Playbooks

  • SYMPTOM: following an official DGX Spark playbook still fails, with no local misconfiguration explaining it.
  • CAUSE: official playbooks have shipped broken before; the stack moves faster than the docs.
  • CHECK: references/gotcha-checks.md G8 — the playbook repo's recent issues.
  • FIX: check github.com/NVIDIA/dgx-spark-playbooks issues before trusting a recipe for an expensive run.

G9: Container-First, Not Bare Pip

  • SYMPTOM: a bare-pip environment that worked yesterday breaks after an unrelated pip install, or two "identical" environments behave differently.
  • CAUSE: bare pip lets Triton, xformers, and transformers drift independently; nothing pins them to GB10's SM121 target.
  • CHECK: references/gotcha-checks.md G9 — container or bare pip?
  • FIX: prefer an NGC container (see spark-environment-setup for tag guidance) or Unsloth's container. If bare pip is unavoidable, follow the NVIDIA install order, including --no-deps on Unsloth.

G10: Dual-Spark Is DDP/FSDP Only

  • SYMPTOM: a tensor-parallel launch across two Sparks hangs, runs far slower than single-Spark, or errors out.
  • CAUSE: ConnectX-7 is fast enough for gradient/parameter sync (DDP, FSDP) but too thin for TP's fine-grained traffic.
  • CHECK: references/gotcha-checks.md G10 — the configured parallelism strategy.
  • FIX: on a two-Spark setup, choose DDP or FSDP, never tensor parallelism — TP is single-node only here.

Fast Triage

The cheapest checks to run before anything else:

python3 -c "import torch; print(torch.version.cuda)"  # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure
import torch; print(torch.cuda.get_device_capability())  # expect (12, 1) (G7)
{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown  # G9

assets/preflight.sh runs G1, G3, G4, G7, G9 and produces one output line per gotcha in a fixed format: G-number first, then PASS/FAIL/WARN where automatable, SKIP when unavailable, or INFO: for a raw reading (G3, G4). Full commands: references/gotcha-checks.md. See also spark-environment-setup for the environment assumed working.

Trace To Training Data

This skill assumes eval-harness-first already graded the traces being converted here — goldens, graders, and runs/<run-id>/results.json all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.

Input: graded traces — eval/goldens.jsonl plus runs/<run-id>/results.json, each row carrying a task_id, a verdict from the grader, and a reward when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):

{"task_id": "t-042", "trace_id": "t-042-a3",
 "messages": [{"role": "user", "content": "..."}],
 "verdict": "pass", "reward": 0.91,
 "grader": "exact_match"}

Output format: rows shaped exactly like dataset-curation's Format Selection table — SFT messages rows or DPO prompt/chosen/rejected pairs — so this skill's output is that skill's input with no reshaping step in between.

The Principle

The eval harness already did the labeling work: every trace in results.json carries a verdict, and often a reward, before this skill ever touches it. Converting a graded trace into a training row is mechanical — pick a shape from dataset-curation's table, map fields, write JSONL. Curation is the work that remains — which traces clear a quality bar, which pairs are informative, and which rows must never enter the training set at all.

Treat any conversion step that requires re-judging a trace as a sign the harness is missing a grader, not a gap this skill should paper over. A trace with no verdict or reward isn't convertible yet — route it back to eval-harness-first first, don't hand-label it here to unblock conversion.

SFT From Traces

  • Keep the top-reward fraction of successful trajectories, not every passing one. Rank passing traces by reward and take a fraction (the Agent-lightning pattern) rather than every trace that merely cleared the pass bar — a trace that barely passed is a weaker SFT signal than one that scored well above threshold.
  • Expert-corrected failures become gold SFT examples directly (the Langfuse pattern) — when a human edits a failing trace's output into a correct one, that correction needs no reward threshold; a human already validated it. Route corrections straight into the SFT set.
  • Step-level masking beats whole-trajectory discard for multi-step traces. When only some steps in a multi-step trajectory are bad, mask the loss on the bad steps and keep the good ones, rather than discarding the whole trajectory. SRFT reports 32.2% vs. 30.9% on SWE-bench for step-level critic masking over trajectory discard — a real, if modest, gap from the finer-grained cut.

Preference Pairs From Traces

  • Build pairs from passing-vs-failing trajectories on the SAME task, never from unrelated best- and worst-scoring traces pulled across different tasks — cross-task pairs teach the model to prefer one task over another, not one response over another.
  • Select the rejected member at μ−2σ of the reward distribution for that task, never the absolute minimum. preference-optimization's Pair Construction section owns the full selection formula; this skill supplies the graded trajectories it consumes.
  • Judge-scored delta selection cuts pair volume without cutting signal. Score each candidate pair by chosen-minus-rejected judge delta and keep only the highest-delta subset — the top 5k of a 16.5k candidate pool matched the full pool's downstream result. Build the full candidate set first, then filter by delta; don't cap generation at 5k up front.

Hygiene

  • Scan for secrets and PII before any row ships, and redact what's found. Traces sourced from production logs can carry credentials, API keys, tokens, or customer data — run a secret/PII scan over every SFT and DPO row and redact matches; conversion fails closed (the row is dropped, not shipped with the raw content) if sensitive fields remain after redaction. Never commit secrets.
  • Eval goldens must never leak into training data. Hold every eval/goldens.jsonl ID out of every converted SFT and DPO set — a trace that also appears as a golden trains on the exact item the checkpoint gets graded against later, silently inflating every subsequent eval run.
  • Dedup against the training set, not just within the newly converted rows — exact-match or embedding-similarity, matching dataset-curation's dedup method field, run against whatever training data already exists before this batch merges in.
  • Provenance goes into the dataset card. Every converted row must trace back to its source run_id and trace_id — dataset-curation's Provenance field checks for exactly this link back to trace-to-training-data output; a row with no traceable source isn't ready to merge.

Related Skills

  • eval-harness-first — produces the graded traces this skill converts; a trace with no verdict or reward isn't convertible yet, route it back there before conversion.
  • dataset-curation — owns the target formats and the dataset card this skill's provenance data feeds; converted rows must match its Format Selection table field names exactly, not an approximation of them.
  • preference-optimization — consumes the DPO pairs this skill builds and owns the full μ−2σ rejection-selection formula referenced above.

Worked JSONL-to-JSONL conversions — graded trace to SFT row, trace pair to DPO pair, correction to SFT row, the rejection-sampling loop, and the goldens-holdout check — live in references/conversion-recipes.md.

Vision-Language SFT

This skill assumes finetuning-method-selection already routed here: the data shape is image+text demonstrations, not preference pairs or a verifiable reward signal, and the base is a vision-language model rather than a text-only one. lora-qlora-recipes covers the text-only LoRA/QLoRA recipe this skill specializes for the vision tower and projector; read that skill first if the LoRA fundamentals (rank, alpha, target modules) aren't already familiar.

Input: an image+text dataset and a VLM base model already picked from the model catalog. Output format: a validated adapter config — which components are frozen, LoRA target modules, and a min_pixels/max_pixels budget — that llm-finetuning-training-engineer consumes directly when it generates a runnable script.

Quick Reference

SituationDefault
Adapting behavior on familiar imagesFrozen tower+projector, LoRA r=8–16, α=16–32
Visual domain shiftUnfreeze last-6 ViT layers, vision LR 5–10x lower
Doesn't fit in bf16 at target rankQLoRA — frozen vision tower only
fast_inference=Truefinetune_vision_layers=False
Loss normal, eval not improvingCheck the Two Silent Killers below first

The Consensus Recipe

Freeze the vision tower and the projector. Put LoRA on the LLM only, all-linear (the same attention + MLP target list as text-only SFT — see lora-qlora-recipes), at r=8–16, α=16–32. This is the settled default for adapting a VLM's behavior without disturbing how it sees.

  • The vision tower and projector stay frozen by default. They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks.
  • LoRA rank runs lower than the text-only general default (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge.
  • QLoRA is permitted only with a frozen vision tower. Quantizing the base while also unfreezing and training vision layers is unsupported and unstable — treat this as a hard pairing rule, not a tunable. If the vision tower needs to unfreeze, drop QLoRA and use bf16 LoRA instead.
# freeze tower + projector; LoRA on LLM only
for name, param in model.named_parameters():
    if "vision_tower" in name or "projector" in name:
        param.requires_grad = False

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]  # LLM-only, all-linear — r=8-16, alpha=16-32

When to Unfreeze

Unfreezing vision layers is a deliberate escalation, not a default decision — reach for it only when the domain shift is visual, not textual.

  • Unfreeze only for visual domain shift. If the task is teaching new behavior on images the tower already understands (charts, everyday photos), the frozen-tower recipe above is sufficient. Unfreeze when the visual domain itself is unfamiliar to the tower — satellite imagery, medical scans, dense technical diagrams — and the frozen-tower recipe plateaus.
  • Last-6 ViT layers is the sweet spot. Unfreezing the final six vision-transformer layers (not the whole tower) measured +1.7pt DocVQA at ~1.75x training cost over the frozen baseline. Treat six layers as the ceiling worth paying for; going further spends compute without a matched result.
  • Vision LR must run 5–10x lower than the LLM LR when unfrozen. The vision tower's pretrained representation is more fragile than the LLM's adapter; the same LR for both risks overwriting the visual representation faster than the LLM adapter can compensate.
  • High LoRA rank on the patch- embedding layer risks NaN. If patch embedding is in the unfrozen set, keep its rank low and watch early-step loss closely — one of the most fragile places to apply LoRA in a VLM.

The Two Silent Killers

Both produce a run that trains without error and without learning: the loss curve looks normal, the model doesn't improve, and neither throws an exception — both need an explicit pre-training check, not just a clean training log.

  • Image-tag/count mismatch. Every image placeholder token in the templated text must map 1:1 to a media item actually passed to the collator. A mismatch (one placeholder, zero or two images attached; or an image with no placeholder) doesn't error in most collators — it silently misaligns image and text, and the model "trains but learns nothing." Validate the 1:1 placeholder-to-media mapping before training starts, on every example, not just a sample. Full validation-checklist detail: references/collators-and-pitfalls.md.
  • min_pixels/max_pixels resolution budget. This pair is the single most consequential hyperparameter for quality and memory in VLM SFT — more than rank, alpha, or LR. Too low silently downsamples images below what the task needs (small document text becomes unreadable even though training "succeeds"); too high blows the activation memory budget or forces too small a batch to train stably. Set it deliberately per dataset, don't leave it at a framework default.

Unsloth Specifics

  • UnslothVisionDataCollator is the collator Unsloth expects for VLM SFT — it handles the image-tag alignment and per-architecture processor contract described in references/collators-and-pitfalls.md. Don't substitute a text-only collator for VLM data.
  • finetune_vision_layers=False is required when fast_inference=True. vLLM cannot serve LoRA adapters on vision layers, so a fast- inference setup that also unfreezes vision layers fails at serve time even if training succeeds. If the recipe calls for unfreezing the last-6 ViT layers (see When to Unfreeze above), fast inference is off the table for that run — choose one or the other, not both.

Model Choice

Base VLM choice is out of scope for this skill — it lives in one place, the model catalog at finetuning-method-selection's references/model-catalog.md. This skill and its references describe recipes by architecture family only, never by recommending one model over another.

VLM reinforcement learning (VLM-GRPO) is reference-only in this plugin — the fragmented tooling and reward-hacking failure modes specific to VLM-RL are covered in grpo-rlvr-training, not here. This skill's scope stops at supervised fine-tuning.

Failure Modes

The recurring mistake across every section above is treating a clean loss curve as proof the run is healthy. A normal-looking curve is consistent with both a working run and either silent killer, since the model trains on something either way — just not the aligned image-text signal when a killer is present. A flat eval score next to a normal loss curve means re-run the checklist in references/collators-and-pitfalls.md before touching any hyperparameter.

References

  • references/collators-and-pitfalls.md — per- architecture collator table, dataset-format examples with image placeholders, a pre- training validation checklist, and the two- stage projector-alignment recipe as an advanced pattern.

Related skills: finetuning-method-selection routes here; lora-qlora-recipes covers the text-only LoRA fundamentals this skill specializes; grpo-rlvr-training covers VLM-RL (reference-only); dataset-curation covers image+text dataset preparation this skill doesn't.

Sponsors

Details

Marketplace
marslanalvi/SkillsStack
Source
plugins/ai/machine-learning
Category
Ai
Keywords
ai, machine-learning

More from marslanalvi/SkillsStack

All 45