
machine-learning-ops
ML pipeline design, experiment tracking, model evaluation, training debugging, and reproducibility.
- version
- 0.6.3
- GitHub stars
- 9
Install
/plugin install machine-learning-ops@agent-powerupsRun it inside Claude Code after adding the marketplace with /plugin marketplace add yeaight7/agent-powerups.
Contents
Discovered from sourceSkills 7
Purpose
Machine learning models add technical debt. A complex model earns its place only by clearly outperforming a "dumb" baseline; this review makes that comparison explicit before a new model is approved.
When to Use
- A new model architecture is proposed for approval
- A complex model would replace an existing heuristic or rule
- Reported gains have not been compared against any baseline
Inputs
- The candidate model's evaluation results (metric + data split)
- The evaluation code, or enough dataset/task detail to define fair baselines
Workflow
- Define the naive baseline:
- Classification: predict the majority class.
- Regression: predict the mean or median of the training target.
- Time series: predict the last known value (naive persistence).
- Define the heuristic baseline: what simple if/else rule would a domain expert write?
- Evaluate both baselines on the same split and metric as the candidate model.
- Evaluate the delta: if the complex model only beats the heuristic baseline marginally (e.g., ~1%), recommend keeping the heuristic — the complexity is not worth the maintenance cost.
- Demand a baseline evaluation script before approving the new architecture, so the comparison is rerunnable.
Output
- A baseline-vs-model comparison on identical data and metric, with an explicit keep/replace recommendation that weighs maintenance cost
Verification
- Naive baseline defined and evaluated
- Heuristic baseline defined and evaluated (or explicitly ruled out with a reason)
- Candidate compared on the same split and metric as the baselines
- Delta judged against maintenance cost, not just statistical improvement
- A rerunnable baseline evaluation script exists
Failure Modes
- No heuristic baseline — comparing only against the naive baseline makes weak models look strong. Ask what rule a domain expert would write.
- Unequal comparison — baseline evaluated on a different split or metric than the model. Re-run both on identical data.
- Complexity bias — approving a model for a marginal gain without stating the maintenance cost in the recommendation.
Purpose
A random split is often the wrong split. Incorrect splitting causes massive overestimation of model performance; this review verifies the split methodology matches the structure of the data.
When to Use
- Reviewing splitting code before training or sign-off
- Data has a time component, repeated entities, or class imbalance
- Validation metrics look too good for the problem
Inputs
- The splitting code
- The dataset's structure: time column? entity keys (user/patient/session)? target balance?
Workflow
- Time-series data: if the data has a time component, random
train_test_splitis strictly forbidden. Require a chronological split so the model cannot learn from the future. - Group leakage: if the dataset has multiple rows per user/patient/session, a standard split puts rows from the same entity in both train and test. Require GroupKFold or group-based splitting.
- Stratification: for imbalanced targets, verify stratification maintains the target distribution across all splits.
- Verify in code, not in description: read the actual splitting call and its parameters before issuing a verdict.
Output
- An explicit pass/fail verdict on Time, Group, and Stratification safety, citing the splitting code
Verification
- Time safety checked (chronological split whenever a time component exists)
- Group safety checked (no entity appears in both train and test)
- Stratification checked for imbalanced targets
- Verdict cites the actual splitting code, not the author's description of it
Failure Modes
- Trusting the description — authors often believe the split is grouped or chronological when the code says otherwise. Read the code.
- Random split on time data — silently inflates metrics; flag as blocking, not advisory.
- Partial grouping — grouped train/test but a random validation split still leaks. Check every split boundary.
Purpose
An ML experiment is useless if you cannot reconstruct exactly how it was run and what data it used. This review checks that everything needed for reconstruction is logged.
When to Use
- Reviewing a training script before serious experiment cycles begin
- Results exist but nobody can say which configuration produced them
- Metrics are printed to stdout only
Inputs
- The training script(s) and any tracking/config setup (MLflow, wandb, config files)
Workflow
- Hyperparameter logging: the script logs every hyperparameter (learning rate, batch size, architecture details). Hardcoded magic numbers must be extracted to a config and logged.
- Metric logging: training and validation metrics are logged at each epoch or step, not just at the end.
- Artifact saving: final model weights, preprocessing scalers/encoders, and the exact configuration file are saved together in a versioned directory or tracking system.
- Data reference: the run records which data it used (path, version, or hash).
- Enforce structured logging (JSON, MLflow, wandb) — stdout-only metric reporting is a blocking finding.
Output
- A finding list of what is not logged or saved, with the specific fix for each gap
Verification
- Every hyperparameter logged (no unlogged magic numbers remain)
- Per-epoch/step train and validation metrics logged
- Weights + preprocessors + config saved together, versioned
- Data reference (path/version/hash) recorded with the run
- No stdout-only metric reporting remains
Failure Modes
- End-only metrics — a final score without curves hides divergence and the onset of overfitting.
- Orphaned artifacts — weights saved without the matching scaler/encoder and config cannot be served correctly later.
- Config drift — the logged config differs from what the script actually used; log the resolved config at runtime.
Purpose
Target leakage is the most common and dangerous error in applied ML. It creates models that look perfect in validation but fail instantly in production. This check inspects the pipeline for the standard leakage vectors.
When to Use
- Reviewing preprocessing or feature-engineering code before training sign-off
- Validation metrics look too good to be true
- A model performed far worse in production than in validation
Inputs
- The preprocessing/feature pipeline code and the train/test split logic
Workflow
- Global scaling/imputation: was any statistic (mean, std, encoder vocabulary) computed on the entire dataset before splitting? That leaks the test distribution into training.
- Future features: is any training feature unavailable at the moment of prediction in real life? (e.g., using "surgery_outcome" to predict "hospital_admission_length")
- ID proxies: are database IDs or row numbers included as features? They often correlate with time or order of entry.
- Enforce the order: Split FIRST, then fit transformers on Train ONLY, then transform Train/Val/Test.
Output
- A leakage verdict per vector (global statistics, future features, ID proxies), citing the offending lines and the fix for each
Verification
- All fit/fit_transform calls occur after the split and only on training data
- Every feature audited for availability at prediction time
- No raw IDs or row numbers among the features
- Findings cite specific code lines
Failure Modes
- Pipeline blindness — leakage hidden inside helper functions or composed pipelines; trace where fitting actually happens.
- "It's just scaling" — dismissing global scaling as harmless; it still leaks distribution information.
- Fixing one vector only — re-running after one fix while the other vectors remain unchecked; audit all three every time.
Purpose
Raw accuracy metrics are not enough. Evaluation must reflect the actual business impact and failure modes of the model; this standard turns metrics into a report a stakeholder can act on.
When to Use
- Producing an evaluation report for a trained model
- Reviewing a report that quotes a single headline metric
- A go/no-go decision depends on understanding error costs
Inputs
- Model predictions on a held-out set
- Business context: cost of errors, key user/customer segments
Workflow
- Beyond accuracy: include the confusion matrix; report Precision, Recall, and F1; explain the cost of a False Positive vs a False Negative in the business context.
- Slice analysis: report performance on key segments. A model can be 95% accurate overall but only 40% accurate on new users.
- Calibration: if the model outputs probabilities, verify they are calibrated — a prediction of 0.8 should come true about 80% of the time.
- Format as a Markdown report that a non-technical stakeholder can read, highlighting trade-offs and worst-case scenarios.
Output
- A Markdown evaluation report: headline metrics, confusion matrix, per-segment table, calibration assessment, and explicit trade-offs/worst cases
Verification
- Confusion matrix plus Precision/Recall/F1 present (not accuracy alone)
- False Positive vs False Negative costs stated in business terms
- Key segments sliced and reported
- Calibration assessed whenever probabilities are output
- Report readable by a non-technical stakeholder
Failure Modes
- Headline-metric reporting — one aggregate number hides segment failures; always include slices.
- Cost-free framing — treating FP and FN as equivalent when their business impact differs by orders of magnitude.
- Uncalibrated probabilities read as confidence — stakeholders will read 0.8 as "80% sure"; verify calibration before they do.
Purpose
Enforce seed setting, deterministic operations, and environment tracking so a training run can be reproduced exactly.
When to Use
- Reviewing or modifying a training script that must be deterministic
- Two runs with the "same" configuration produced different results
- A past result needs to be reconstructed exactly
Inputs
- A target Python training script
Workflow
- Global seed initialization: ensure a single function sets seeds for all relevant libraries (
random,numpy,torch,tensorflow). - Deterministic algorithms: for PyTorch or TensorFlow, check that deterministic algorithms are enabled (e.g.,
torch.use_deterministic_algorithms(True)). - Data loading: verify data loaders use deterministic shuffling and that worker processes are seeded correctly to avoid identical augmentations.
- Environment & config tracking: ensure the script logs the exact configuration, dependency versions, and data hashes.
- Review first: point out missing reproducibility guards before rewriting the script. Provide the exact seed-initialization snippet — do not hide side effects.
Output
- A list of missing reproducibility guards with the exact code snippets to add, plus any performance trade-off warnings
Verification
- All library seeds set from one place
- Deterministic-algorithm flags enabled (or the gap explicitly accepted)
- Loader shuffling and worker seeding deterministic
- Configuration, dependency versions, and data hashes logged
- User warned if determinism flags significantly slow training
Failure Modes
- Partial seeding — seeding
randombut not the framework or loader workers still yields nondeterminism. - Silent slowdown — enabling deterministic algorithms can cost real training speed; surface the trade-off instead of hiding it.
- Rewriting before reviewing — changing the script without first listing the gaps loses the audit trail.
Purpose
ML training bugs are often silent mathematical errors rather than explicit code crashes. This protocol localizes the standard failure classes quickly.
When to Use
- Loss becomes NaN or diverges
- Training crashes with out-of-memory errors
- Shape/dimension errors at matrix multiplications or loss computation
- The pipeline runs but the model never learns
Inputs
- The failing training script and its error output or loss curve
Workflow
- Run the overfit test first: fit a single batch of ~10 examples. If the model cannot achieve near-zero loss on a single batch, the pipeline is fundamentally broken — do not debug full runs until the single-batch test passes.
- NaN losses — check:
- Learning rate too high?
- Missing data (NaNs in input)?
- Log/Exp/Divide-by-zero in custom loss functions?
- Exploding gradients (clip gradients)?
- OOM (out of memory):
- Reduce batch size.
- Check for memory leaks in the training loop (e.g., accumulating history across epochs without
.detach()).
- Shape mismatches:
- Add temporary print statements or assertions on
tensor.shapebefore matrix multiplications and loss calculations.
- Add temporary print statements or assertions on
Output
- The identified failure class and its specific fix, or a single-batch reproduction showing exactly where the pipeline breaks
Verification
- Single-batch overfit test run before any full-run debugging
- For NaN: learning rate, input NaNs, unsafe math, and gradient explosion all checked
- For OOM: batch size and training-loop accumulation both checked
- For shapes: assertions placed before matmul/loss sites
- Fix validated by re-running the previously failing case
Failure Modes
- Debugging full runs first — hours per iteration; the single-batch test gives answers in minutes.
- Treating NaN as random — NaN losses have a small set of causes; check all four systematically instead of restarting with a lower LR.
- Batch-size-only OOM fixes — shrinking the batch hides a loop leak that will OOM again later; check accumulation too.
- Leftover instrumentation — remove temporary shape prints/assertions once the fix is validated.
Commands 4
Evaluation Report Command
CRITICAL BEHAVIORAL RULES
- Compare to Baseline: Performance in a vacuum is useless. The report MUST contrast the model's metrics against a naive baseline (e.g., predicting the mean or majority class) or the previous model version.
- Highlight the Trade-offs: Emphasize what the model got worse at (e.g., "Recall improved by 5%, but Precision dropped by 2%").
Execution Steps
- Parse the
$ARGUMENTSto extract the metrics. - Format a clear Markdown report.
- Include sections for:
- Topline Metrics: The primary KPIs (Accuracy, F1, RMSE, etc.).
- Baseline Comparison: How much better is this than a dumb rule?
- Subgroup/Slice Analysis: Did performance degrade for specific subsets?
- Confusion Matrix/Error Analysis: Where is the model making its most costly mistakes?
- Output the Markdown report and optionally save it to
reports/YYYY-MM-DD-evaluation.md.
Experiment Preflight Command
CRITICAL BEHAVIORAL RULES
- Prevent Wasted Compute: Training runs take hours or days. The preflight check prevents starting a run that will immediately fail or whose results will be untrustworthy.
Execution Steps
- Read the target training script.
- Seed Check: Verify
random.seed,np.random.seed, andtorch.manual_seed(or equivalent) are set at the very beginning. - Tracking Check: Verify that hyperparameters, metrics, and the final model artifact are logged using a tracking system (MLflow, Weights & Biases, or simple JSON).
- Data Versioning: Verify that the dataset path or version is explicitly recorded in the configuration.
- Output: Provide a strict "Go/No-Go" decision. If No-Go, output the exact lines of code that need to be fixed to ensure the run is reproducible and trackable.
Leakage Check Command
CRITICAL BEHAVIORAL RULES
- Hunt the Future: Target leakage occurs when information from the future (the target variable) sneaks into the training features.
Execution Steps
- Read the preprocessing and splitting code.
- Split First: Verify that the dataset is split into train/val/test before any global transformations are applied.
- Imputation Leakage: Verify that scalers, encoders, and imputers are
fitONLY on the training set, andtransformis applied to train/val/test. Iffit_transformis used on the entire dataset, flag it as a critical leakage error. - Feature Leakage: Look for features that are direct proxies for the target (e.g., a "cancellation_date" feature when predicting churn).
- Output: Generate a Leakage Audit Report highlighting unsafe transformations and risky features.
ml-run-review
Review a machine learning training run or experiment configuration for soundness, reproducibility, and evaluation rigor.
Prerequisites
Provide the agent with the training script, experiment configuration, or metric logs you want reviewed.
Review Focus
- Evaluation Rigor: Is there a proper holdout set? Is the cross-validation strategy appropriate (e.g., time-based splitting for time series)?
- Data Leakage Check: Are preprocessing steps (scaling, imputing) fit only on the training set?
- Reproducibility: Are random seeds fixed? Are hyperparameters explicitly logged?
- Metrics: Do the chosen metrics align with the stated goal? Are baselines established?
Output
The agent will provide a brief markdown summary of identified risks and propose specific, targeted code fixes. The agent will not make changes without your explicit approval.
Agents 4
Leakage Auditor
You are an expert ML reviewer focused strictly on preventing data leakage. Your core purpose is to audit data preparation, feature engineering, and splitting logic to ensure models do not access information during training that they will not have at inference time.
Core Principles
- Review First: Always point out the exact line where leakage occurs before proposing a fix.
- Skepticism: Assume leakage exists if transformations (scaling, imputation, encoding) are applied before the train/test split.
- No Hidden Actions: Do not silently refactor code; provide explicit diffs for the user to approve.
Key Audit Areas
- Temporal Leakage: Check that future data is not used to predict past events. Ensure time-based splitting is used for time series data.
- Train-Test Contamination: Verify that
fit()for scalers/encoders is only called on the training data, andtransform()is called on both. - Feature-Target Correlation: Look for proxy features that inadvertently encode the target variable (e.g., "account_closed_date" predicting "churn").
- Group Leakage: Ensure overlapping groups (e.g., multiple records from the same patient) are not split across training and test sets.
Response Format
When auditing code, provide:
- Leakage Risks: A bulleted list of identified or potential leakage points.
- Code Evidence: The specific lines of code causing the issue.
- Safe Refactoring: The proposed changes to fix the pipeline, typically recommending the use of Pipeline objects (like
sklearn.pipeline.Pipeline).
ML Experiment Reviewer
You are a rigorous scientific reviewer for machine learning experiments. Your goal is to ensure that experiments are trustworthy, correctly evaluated, and comparable.
Core Principles
- Scientific Rigor: Focus on the validity of the comparison, not just the final metric.
- Review First: Point out flaws in the experimental design before suggesting code modifications.
- Explicit Baselines: Always ask for or look for a simple baseline (e.g., mean prediction, logistic regression) before accepting complex models.
Key Audit Areas
- Baseline Comparisons: Does the experiment compare against a naive or standard baseline?
- Metric Selection: Do the chosen metrics match the business problem (e.g., PR-AUC for highly imbalanced data instead of ROC-AUC)?
- Validation Strategy: Is the cross-validation strategy appropriate for the data (e.g., Stratified, Grouped, or TimeSeries split)?
- Experiment Tracking: Are hyperparameters, metrics, and data versions explicitly logged (e.g., using MLflow, Weights & Biases)?
Response Format
When reviewing an experiment, provide:
- Design Flaws: Any issues with how the experiment is structured.
- Metric Critique: Assessment of whether the metrics truly capture model performance.
- Recommended Action: Concrete code adjustments to improve tracking or evaluation rigor.
Model Evaluation Analyst
You are an expert in rigorous machine learning evaluation. Your goal is to prevent the team from deploying a model whose performance is a statistical illusion.
Operational Rules
- Metric Suitability: Do not accept accuracy on an imbalanced dataset. Ensure precision, recall, F1, PR-AUC, or ROC-AUC are used appropriately. For regression, look for MAE, RMSE, and MAPE.
- Confidence Intervals: A single point estimate is meaningless. Verify or add logic to compute confidence intervals (e.g., via bootstrapping) to prove the new model is statistically significantly better.
- Subgroup Analysis: Models that perform well on average might fail catastrophically on minority slices. Check if the evaluation script slices performance by key categorical variables.
- Output: Produce an "Evaluation Rigor Report" pointing out missing metrics, lack of statistical significance testing, or failure to evaluate edge cases.
Training Pipeline Debugger
You are an ML engineering specialist focused on debugging and optimizing deep learning and machine learning training loops.
Core Principles
- Isolate the Bottleneck: Determine if the issue is I/O, CPU, or GPU bound before changing model code.
- Explicit Fixes: Provide targeted fixes rather than rewriting the entire training loop.
- No Hidden Actions: Do not install profilers or execute training without user consent. Show the commands first.
Key Debugging Areas
- Hardware Utilization: Diagnose low GPU utilization (often caused by CPU-bound data loading or small batch sizes).
- Memory Leaks: Identify OOM (Out of Memory) errors, un-freed computational graphs, or accumulating metrics tracking tensors instead of floats.
- Numerical Instability: Look for vanishing/exploding gradients, missing normalization, or incorrect activation functions (e.g., missing softmax/sigmoid).
- I/O Bottlenecks: Check data loader configurations, prefetching, and pinning memory.
Response Format
When debugging a pipeline, provide:
- Root Cause Hypothesis: What is likely causing the bottleneck or crash.
- Diagnostic Commands: Shell commands or small code snippets to verify the hypothesis.
- Targeted Fix: The specific code change to resolve the issue.
Sponsors
Details
- Marketplace
- yeaight7/agent-powerups
- Source
- plugins/machine-learning-ops
- Category
- Developer Tools
- License
- Apache-2.0
- Keywords
- machine-learning, mlops, experiment-tracking, model-evaluation, pipelines
More from yeaight7/agent-powerups
All 21- mcp-developmentv0.6.3
- memory-optimizationv0.6.3
- quality-gatesv0.6.3
- security-guardrailsv0.6.3
- skill-authoringv0.6.3
- software-engineeringv0.6.3
- spec-driven-developmentv0.6.3
- spec-quality-gatesv0.6.3
- tool-integrationsv0.6.3
- agent-evaluation-labv0.6.3
- agentic-systemsv0.6.3
- codebase-intelligencev0.6.3
- codebase-maintenancev0.6.3
- connected-appsv0.6.3
- context-efficiencyv0.6.3
- data-engineeringv0.6.3
- debugging-diagnosticsv0.6.3
- dev-vitalsv0.6.3
- documentation-systemsv0.6.3
- github-opsv0.6.3