yeaight7 avatar

machine-learning-ops

ML pipeline design, experiment tracking, model evaluation, training debugging, and reproducibility.

version
0.6.3
GitHub stars
9

Install

/plugin install machine-learning-ops@agent-powerups

Run it inside Claude Code after adding the marketplace with /plugin marketplace add yeaight7/agent-powerups.

Contents

Discovered from source

Skills 7

Purpose

Machine learning models add technical debt. A complex model earns its place only by clearly outperforming a "dumb" baseline; this review makes that comparison explicit before a new model is approved.

When to Use

  • A new model architecture is proposed for approval
  • A complex model would replace an existing heuristic or rule
  • Reported gains have not been compared against any baseline

Inputs

  • The candidate model's evaluation results (metric + data split)
  • The evaluation code, or enough dataset/task detail to define fair baselines

Workflow

  1. Define the naive baseline:
    • Classification: predict the majority class.
    • Regression: predict the mean or median of the training target.
    • Time series: predict the last known value (naive persistence).
  2. Define the heuristic baseline: what simple if/else rule would a domain expert write?
  3. Evaluate both baselines on the same split and metric as the candidate model.
  4. Evaluate the delta: if the complex model only beats the heuristic baseline marginally (e.g., ~1%), recommend keeping the heuristic — the complexity is not worth the maintenance cost.
  5. Demand a baseline evaluation script before approving the new architecture, so the comparison is rerunnable.

Output

  • A baseline-vs-model comparison on identical data and metric, with an explicit keep/replace recommendation that weighs maintenance cost

Verification

  • Naive baseline defined and evaluated
  • Heuristic baseline defined and evaluated (or explicitly ruled out with a reason)
  • Candidate compared on the same split and metric as the baselines
  • Delta judged against maintenance cost, not just statistical improvement
  • A rerunnable baseline evaluation script exists

Failure Modes

  • No heuristic baseline — comparing only against the naive baseline makes weak models look strong. Ask what rule a domain expert would write.
  • Unequal comparison — baseline evaluated on a different split or metric than the model. Re-run both on identical data.
  • Complexity bias — approving a model for a marginal gain without stating the maintenance cost in the recommendation.

Purpose

A random split is often the wrong split. Incorrect splitting causes massive overestimation of model performance; this review verifies the split methodology matches the structure of the data.

When to Use

  • Reviewing splitting code before training or sign-off
  • Data has a time component, repeated entities, or class imbalance
  • Validation metrics look too good for the problem

Inputs

  • The splitting code
  • The dataset's structure: time column? entity keys (user/patient/session)? target balance?

Workflow

  1. Time-series data: if the data has a time component, random train_test_split is strictly forbidden. Require a chronological split so the model cannot learn from the future.
  2. Group leakage: if the dataset has multiple rows per user/patient/session, a standard split puts rows from the same entity in both train and test. Require GroupKFold or group-based splitting.
  3. Stratification: for imbalanced targets, verify stratification maintains the target distribution across all splits.
  4. Verify in code, not in description: read the actual splitting call and its parameters before issuing a verdict.

Output

  • An explicit pass/fail verdict on Time, Group, and Stratification safety, citing the splitting code

Verification

  • Time safety checked (chronological split whenever a time component exists)
  • Group safety checked (no entity appears in both train and test)
  • Stratification checked for imbalanced targets
  • Verdict cites the actual splitting code, not the author's description of it

Failure Modes

  • Trusting the description — authors often believe the split is grouped or chronological when the code says otherwise. Read the code.
  • Random split on time data — silently inflates metrics; flag as blocking, not advisory.
  • Partial grouping — grouped train/test but a random validation split still leaks. Check every split boundary.

Purpose

An ML experiment is useless if you cannot reconstruct exactly how it was run and what data it used. This review checks that everything needed for reconstruction is logged.

When to Use

  • Reviewing a training script before serious experiment cycles begin
  • Results exist but nobody can say which configuration produced them
  • Metrics are printed to stdout only

Inputs

  • The training script(s) and any tracking/config setup (MLflow, wandb, config files)

Workflow

  1. Hyperparameter logging: the script logs every hyperparameter (learning rate, batch size, architecture details). Hardcoded magic numbers must be extracted to a config and logged.
  2. Metric logging: training and validation metrics are logged at each epoch or step, not just at the end.
  3. Artifact saving: final model weights, preprocessing scalers/encoders, and the exact configuration file are saved together in a versioned directory or tracking system.
  4. Data reference: the run records which data it used (path, version, or hash).
  5. Enforce structured logging (JSON, MLflow, wandb) — stdout-only metric reporting is a blocking finding.

Output

  • A finding list of what is not logged or saved, with the specific fix for each gap

Verification

  • Every hyperparameter logged (no unlogged magic numbers remain)
  • Per-epoch/step train and validation metrics logged
  • Weights + preprocessors + config saved together, versioned
  • Data reference (path/version/hash) recorded with the run
  • No stdout-only metric reporting remains

Failure Modes

  • End-only metrics — a final score without curves hides divergence and the onset of overfitting.
  • Orphaned artifacts — weights saved without the matching scaler/encoder and config cannot be served correctly later.
  • Config drift — the logged config differs from what the script actually used; log the resolved config at runtime.

Purpose

Target leakage is the most common and dangerous error in applied ML. It creates models that look perfect in validation but fail instantly in production. This check inspects the pipeline for the standard leakage vectors.

When to Use

  • Reviewing preprocessing or feature-engineering code before training sign-off
  • Validation metrics look too good to be true
  • A model performed far worse in production than in validation

Inputs

  • The preprocessing/feature pipeline code and the train/test split logic

Workflow

  1. Global scaling/imputation: was any statistic (mean, std, encoder vocabulary) computed on the entire dataset before splitting? That leaks the test distribution into training.
  2. Future features: is any training feature unavailable at the moment of prediction in real life? (e.g., using "surgery_outcome" to predict "hospital_admission_length")
  3. ID proxies: are database IDs or row numbers included as features? They often correlate with time or order of entry.
  4. Enforce the order: Split FIRST, then fit transformers on Train ONLY, then transform Train/Val/Test.

Output

  • A leakage verdict per vector (global statistics, future features, ID proxies), citing the offending lines and the fix for each

Verification

  • All fit/fit_transform calls occur after the split and only on training data
  • Every feature audited for availability at prediction time
  • No raw IDs or row numbers among the features
  • Findings cite specific code lines

Failure Modes

  • Pipeline blindness — leakage hidden inside helper functions or composed pipelines; trace where fitting actually happens.
  • "It's just scaling" — dismissing global scaling as harmless; it still leaks distribution information.
  • Fixing one vector only — re-running after one fix while the other vectors remain unchecked; audit all three every time.

Purpose

Raw accuracy metrics are not enough. Evaluation must reflect the actual business impact and failure modes of the model; this standard turns metrics into a report a stakeholder can act on.

When to Use

  • Producing an evaluation report for a trained model
  • Reviewing a report that quotes a single headline metric
  • A go/no-go decision depends on understanding error costs

Inputs

  • Model predictions on a held-out set
  • Business context: cost of errors, key user/customer segments

Workflow

  1. Beyond accuracy: include the confusion matrix; report Precision, Recall, and F1; explain the cost of a False Positive vs a False Negative in the business context.
  2. Slice analysis: report performance on key segments. A model can be 95% accurate overall but only 40% accurate on new users.
  3. Calibration: if the model outputs probabilities, verify they are calibrated — a prediction of 0.8 should come true about 80% of the time.
  4. Format as a Markdown report that a non-technical stakeholder can read, highlighting trade-offs and worst-case scenarios.

Output

  • A Markdown evaluation report: headline metrics, confusion matrix, per-segment table, calibration assessment, and explicit trade-offs/worst cases

Verification

  • Confusion matrix plus Precision/Recall/F1 present (not accuracy alone)
  • False Positive vs False Negative costs stated in business terms
  • Key segments sliced and reported
  • Calibration assessed whenever probabilities are output
  • Report readable by a non-technical stakeholder

Failure Modes

  • Headline-metric reporting — one aggregate number hides segment failures; always include slices.
  • Cost-free framing — treating FP and FN as equivalent when their business impact differs by orders of magnitude.
  • Uncalibrated probabilities read as confidence — stakeholders will read 0.8 as "80% sure"; verify calibration before they do.

Purpose

Enforce seed setting, deterministic operations, and environment tracking so a training run can be reproduced exactly.

When to Use

  • Reviewing or modifying a training script that must be deterministic
  • Two runs with the "same" configuration produced different results
  • A past result needs to be reconstructed exactly

Inputs

  • A target Python training script

Workflow

  1. Global seed initialization: ensure a single function sets seeds for all relevant libraries (random, numpy, torch, tensorflow).
  2. Deterministic algorithms: for PyTorch or TensorFlow, check that deterministic algorithms are enabled (e.g., torch.use_deterministic_algorithms(True)).
  3. Data loading: verify data loaders use deterministic shuffling and that worker processes are seeded correctly to avoid identical augmentations.
  4. Environment & config tracking: ensure the script logs the exact configuration, dependency versions, and data hashes.
  5. Review first: point out missing reproducibility guards before rewriting the script. Provide the exact seed-initialization snippet — do not hide side effects.

Output

  • A list of missing reproducibility guards with the exact code snippets to add, plus any performance trade-off warnings

Verification

  • All library seeds set from one place
  • Deterministic-algorithm flags enabled (or the gap explicitly accepted)
  • Loader shuffling and worker seeding deterministic
  • Configuration, dependency versions, and data hashes logged
  • User warned if determinism flags significantly slow training

Failure Modes

  • Partial seeding — seeding random but not the framework or loader workers still yields nondeterminism.
  • Silent slowdown — enabling deterministic algorithms can cost real training speed; surface the trade-off instead of hiding it.
  • Rewriting before reviewing — changing the script without first listing the gaps loses the audit trail.

Purpose

ML training bugs are often silent mathematical errors rather than explicit code crashes. This protocol localizes the standard failure classes quickly.

When to Use

  • Loss becomes NaN or diverges
  • Training crashes with out-of-memory errors
  • Shape/dimension errors at matrix multiplications or loss computation
  • The pipeline runs but the model never learns

Inputs

  • The failing training script and its error output or loss curve

Workflow

  1. Run the overfit test first: fit a single batch of ~10 examples. If the model cannot achieve near-zero loss on a single batch, the pipeline is fundamentally broken — do not debug full runs until the single-batch test passes.
  2. NaN losses — check:
    • Learning rate too high?
    • Missing data (NaNs in input)?
    • Log/Exp/Divide-by-zero in custom loss functions?
    • Exploding gradients (clip gradients)?
  3. OOM (out of memory):
    • Reduce batch size.
    • Check for memory leaks in the training loop (e.g., accumulating history across epochs without .detach()).
  4. Shape mismatches:
    • Add temporary print statements or assertions on tensor.shape before matrix multiplications and loss calculations.

Output

  • The identified failure class and its specific fix, or a single-batch reproduction showing exactly where the pipeline breaks

Verification

  • Single-batch overfit test run before any full-run debugging
  • For NaN: learning rate, input NaNs, unsafe math, and gradient explosion all checked
  • For OOM: batch size and training-loop accumulation both checked
  • For shapes: assertions placed before matmul/loss sites
  • Fix validated by re-running the previously failing case

Failure Modes

  • Debugging full runs first — hours per iteration; the single-batch test gives answers in minutes.
  • Treating NaN as random — NaN losses have a small set of causes; check all four systematically instead of restarting with a lower LR.
  • Batch-size-only OOM fixes — shrinking the batch hides a loop leak that will OOM again later; check accumulation too.
  • Leftover instrumentation — remove temporary shape prints/assertions once the fix is validated.

Commands 4

Evaluation Report Command

CRITICAL BEHAVIORAL RULES

  1. Compare to Baseline: Performance in a vacuum is useless. The report MUST contrast the model's metrics against a naive baseline (e.g., predicting the mean or majority class) or the previous model version.
  2. Highlight the Trade-offs: Emphasize what the model got worse at (e.g., "Recall improved by 5%, but Precision dropped by 2%").

Execution Steps

  1. Parse the $ARGUMENTS to extract the metrics.
  2. Format a clear Markdown report.
  3. Include sections for:
    • Topline Metrics: The primary KPIs (Accuracy, F1, RMSE, etc.).
    • Baseline Comparison: How much better is this than a dumb rule?
    • Subgroup/Slice Analysis: Did performance degrade for specific subsets?
    • Confusion Matrix/Error Analysis: Where is the model making its most costly mistakes?
  4. Output the Markdown report and optionally save it to reports/YYYY-MM-DD-evaluation.md.

Experiment Preflight Command

CRITICAL BEHAVIORAL RULES

  1. Prevent Wasted Compute: Training runs take hours or days. The preflight check prevents starting a run that will immediately fail or whose results will be untrustworthy.

Execution Steps

  1. Read the target training script.
  2. Seed Check: Verify random.seed, np.random.seed, and torch.manual_seed (or equivalent) are set at the very beginning.
  3. Tracking Check: Verify that hyperparameters, metrics, and the final model artifact are logged using a tracking system (MLflow, Weights & Biases, or simple JSON).
  4. Data Versioning: Verify that the dataset path or version is explicitly recorded in the configuration.
  5. Output: Provide a strict "Go/No-Go" decision. If No-Go, output the exact lines of code that need to be fixed to ensure the run is reproducible and trackable.

Leakage Check Command

CRITICAL BEHAVIORAL RULES

  1. Hunt the Future: Target leakage occurs when information from the future (the target variable) sneaks into the training features.

Execution Steps

  1. Read the preprocessing and splitting code.
  2. Split First: Verify that the dataset is split into train/val/test before any global transformations are applied.
  3. Imputation Leakage: Verify that scalers, encoders, and imputers are fit ONLY on the training set, and transform is applied to train/val/test. If fit_transform is used on the entire dataset, flag it as a critical leakage error.
  4. Feature Leakage: Look for features that are direct proxies for the target (e.g., a "cancellation_date" feature when predicting churn).
  5. Output: Generate a Leakage Audit Report highlighting unsafe transformations and risky features.

ml-run-review

Review a machine learning training run or experiment configuration for soundness, reproducibility, and evaluation rigor.

Prerequisites

Provide the agent with the training script, experiment configuration, or metric logs you want reviewed.

Review Focus

  1. Evaluation Rigor: Is there a proper holdout set? Is the cross-validation strategy appropriate (e.g., time-based splitting for time series)?
  2. Data Leakage Check: Are preprocessing steps (scaling, imputing) fit only on the training set?
  3. Reproducibility: Are random seeds fixed? Are hyperparameters explicitly logged?
  4. Metrics: Do the chosen metrics align with the stated goal? Are baselines established?

Output

The agent will provide a brief markdown summary of identified risks and propose specific, targeted code fixes. The agent will not make changes without your explicit approval.

Agents 4

Leakage Auditor

You are an expert ML reviewer focused strictly on preventing data leakage. Your core purpose is to audit data preparation, feature engineering, and splitting logic to ensure models do not access information during training that they will not have at inference time.

Core Principles

  • Review First: Always point out the exact line where leakage occurs before proposing a fix.
  • Skepticism: Assume leakage exists if transformations (scaling, imputation, encoding) are applied before the train/test split.
  • No Hidden Actions: Do not silently refactor code; provide explicit diffs for the user to approve.

Key Audit Areas

  1. Temporal Leakage: Check that future data is not used to predict past events. Ensure time-based splitting is used for time series data.
  2. Train-Test Contamination: Verify that fit() for scalers/encoders is only called on the training data, and transform() is called on both.
  3. Feature-Target Correlation: Look for proxy features that inadvertently encode the target variable (e.g., "account_closed_date" predicting "churn").
  4. Group Leakage: Ensure overlapping groups (e.g., multiple records from the same patient) are not split across training and test sets.

Response Format

When auditing code, provide:

  1. Leakage Risks: A bulleted list of identified or potential leakage points.
  2. Code Evidence: The specific lines of code causing the issue.
  3. Safe Refactoring: The proposed changes to fix the pipeline, typically recommending the use of Pipeline objects (like sklearn.pipeline.Pipeline).

ML Experiment Reviewer

You are a rigorous scientific reviewer for machine learning experiments. Your goal is to ensure that experiments are trustworthy, correctly evaluated, and comparable.

Core Principles

  • Scientific Rigor: Focus on the validity of the comparison, not just the final metric.
  • Review First: Point out flaws in the experimental design before suggesting code modifications.
  • Explicit Baselines: Always ask for or look for a simple baseline (e.g., mean prediction, logistic regression) before accepting complex models.

Key Audit Areas

  1. Baseline Comparisons: Does the experiment compare against a naive or standard baseline?
  2. Metric Selection: Do the chosen metrics match the business problem (e.g., PR-AUC for highly imbalanced data instead of ROC-AUC)?
  3. Validation Strategy: Is the cross-validation strategy appropriate for the data (e.g., Stratified, Grouped, or TimeSeries split)?
  4. Experiment Tracking: Are hyperparameters, metrics, and data versions explicitly logged (e.g., using MLflow, Weights & Biases)?

Response Format

When reviewing an experiment, provide:

  1. Design Flaws: Any issues with how the experiment is structured.
  2. Metric Critique: Assessment of whether the metrics truly capture model performance.
  3. Recommended Action: Concrete code adjustments to improve tracking or evaluation rigor.

Model Evaluation Analyst

You are an expert in rigorous machine learning evaluation. Your goal is to prevent the team from deploying a model whose performance is a statistical illusion.

Operational Rules

  1. Metric Suitability: Do not accept accuracy on an imbalanced dataset. Ensure precision, recall, F1, PR-AUC, or ROC-AUC are used appropriately. For regression, look for MAE, RMSE, and MAPE.
  2. Confidence Intervals: A single point estimate is meaningless. Verify or add logic to compute confidence intervals (e.g., via bootstrapping) to prove the new model is statistically significantly better.
  3. Subgroup Analysis: Models that perform well on average might fail catastrophically on minority slices. Check if the evaluation script slices performance by key categorical variables.
  4. Output: Produce an "Evaluation Rigor Report" pointing out missing metrics, lack of statistical significance testing, or failure to evaluate edge cases.

Training Pipeline Debugger

You are an ML engineering specialist focused on debugging and optimizing deep learning and machine learning training loops.

Core Principles

  • Isolate the Bottleneck: Determine if the issue is I/O, CPU, or GPU bound before changing model code.
  • Explicit Fixes: Provide targeted fixes rather than rewriting the entire training loop.
  • No Hidden Actions: Do not install profilers or execute training without user consent. Show the commands first.

Key Debugging Areas

  1. Hardware Utilization: Diagnose low GPU utilization (often caused by CPU-bound data loading or small batch sizes).
  2. Memory Leaks: Identify OOM (Out of Memory) errors, un-freed computational graphs, or accumulating metrics tracking tensors instead of floats.
  3. Numerical Instability: Look for vanishing/exploding gradients, missing normalization, or incorrect activation functions (e.g., missing softmax/sigmoid).
  4. I/O Bottlenecks: Check data loader configurations, prefetching, and pinning memory.

Response Format

When debugging a pipeline, provide:

  1. Root Cause Hypothesis: What is likely causing the bottleneck or crash.
  2. Diagnostic Commands: Shell commands or small code snippets to verify the hypothesis.
  3. Targeted Fix: The specific code change to resolve the issue.

Sponsors

Details

Marketplace
yeaight7/agent-powerups
Source
plugins/machine-learning-ops
Category
Developer Tools
License
Apache-2.0
Keywords
machine-learning, mlops, experiment-tracking, model-evaluation, pipelines

More from yeaight7/agent-powerups

All 21