New advertisers: 50% off your first month with code FIRSTMONTHSee slots

Investigate OpenShift CI disruption regressions: when disruption changed, and why runs were disrupted

version
0.0.2
GitHub stars
123
View on GitHub

Add marketplace

/plugin marketplace add openshift-eng/ai-helpers

Run it inside Claude Code once, then install the plugin below.

Install

/plugin install disruption@ai-helpers

Run it inside Claude Code after adding the marketplace.

Files

Discovered from source

Disruption Analyze Run

This skill answers why these runs were disrupted: the mechanism in each run. It does not decide whether disruption got worse, compare cohorts, or search Jira; other disruption skills do that.

Prerequisites

  1. gcloud CLI (which gcloud). The test-platform-results-public bucket is public. Do not run gcloud auth login, and do not change the user's gcloud config. download_timelines.py sets CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true on each gcloud subprocess, and any direct gcloud storage command here uses the same variable. Anonymous access still needs network execution when the command runs in a sandbox.
  2. Python 3.7+.
  3. The disruption_common library: the disruption plugin's scripts/disruption_common, installed with the plugin. If it is missing, the scripts exit and say so. Report the message; do not work around it.

The scripts are in this skill's directory: ${CLAUDE_SKILL_DIR} in Claude Code, or the directory containing this SKILL.md elsewhere.

Rules shared by the disruption skills

  • Backend names are exact. Pass them as --backends unchanged. Do not shorten kube-api-new-connections to kube-api, and do not substitute derived probes (*-localhost-*, *-http1-*, *-http2-*, *-service-network-*, *-internal-lb-*) or cache-* backends unless the user named them.
  • Liveness. A run where ci-cluster-network-liveness is disrupted has unreliable disruption data. Keep it, say so, and do not rest a conclusion on its disruption counts alone.
  • Platform. The platform in the alert or job name is where disruption was measured, not evidence of the cause. Name a platform-specific cause only when the run evidence shows a platform mechanism (a cloud metric, a cloud API error, a platform-only component).
  • State. With a state file (investigation.json), {investigation_dir} is its directory; otherwise create .work/disruption-analysis/<YYYY-MM-DD>-<short-slug>/. Runs go under {runs_dir} = {investigation_dir}/runs.
  • One block. Do not run another disruption skill yourself. End with the checkpoint below.

Step 1: Inputs

  • Prow URLs (prow.ci.openshift.org or gcsweb-ci), one or more. Or job:build_id pairs, such as cohorts.handoff.download_runs from disruption-find-runs in the state file.
  • Backends (--backends): the exact names. Take them from the request, or from the state file's series.backends, or else its cohorts.backends (written by cohorts.py validate). Without any of these, analyze every backend that shows disruption and say so.
  • When the runs are cohorts (before/after), keep each run's cohort; the analysis of each run is the same, but the checkpoint says which cohort each run is in.

From each Prow URL take the job name and build ID: the last two path segments after /logs/. The job name is letters, digits, ., _, and -, and the build ID is all digits; skip and report anything else rather than putting it in a shell command. Presubmit runs (/pr-logs/pull/...) are not supported: the download reads only /logs/ artifacts, so say so and skip them. Prow URLs may say origin-ci-test; the bucket is always test-platform-results-public.

Step 2: Download timelines

Read references/artifacts.md and follow Timeline download: one download_timelines.py call for every run, with --output-dir {runs_dir}. When the runs are before/after cohorts, add --gather: disruption-compare-cohorts compares component versions from each run's releaseinfo.json, and the revision-trigger check needs the events and configmaps. On exit 2, continue with the runs that downloaded; on exit 1, stop and report each run's error.

Keep each run's payload pair (initial → target) from the output. For upgrade jobs, make no claim that depends on which payload a run started from or upgraded to without that run's pair; if it is unknown, say so.

Step 3: Analyze the timelines

For every run, save the parser's full JSON (the cohort comparison reads it) and print its one-line summary, in one loop:

for run in '{job_1}:{build_id_1}' '{job_2}:{build_id_2}'; do
  job="${run%%:*}"; build_id="${run##*:}"; dir="{runs_dir}/${build_id}"
  if python3 "${CLAUDE_SKILL_DIR}/parse_disruption.py" "${dir}"/logs/e2e-timelines_spyglass_*.json \
      --job-name "${job}" --build-id "${build_id}" --backends '{exact_backends}' --format json > "${dir}/parsed.json.tmp"; then
    mv "${dir}/parsed.json.tmp" "${dir}/parsed.json"
    python3 "${CLAUDE_SKILL_DIR}/parse_disruption.py" "${dir}"/logs/e2e-timelines_spyglass_*.json \
      --job-name "${job}" --build-id "${build_id}" --backends '{exact_backends}' --format summary
  else
    rm -f "${dir}/parsed.json.tmp" "${dir}/parsed.json"; echo "${build_id}: the parser failed; no parsed.json kept" >&2
  fi
done

A run without timelines gets no parsed.json, and a failed rerun removes the one from an earlier run, so the cohort comparison reports the run as missing instead of reading an empty or stale file.

Then read references/timeline-analysis.md and follow 3.2 through 3.4 (blast radius, deep-dive of the runs that need it, and how to read each signal).

Compare runs by the parser's episodes and waves (for example "3 episodes in 1 wave" against "6 episodes in 2 waves (3+3)"), not by the disruption count. Check which static-pod revision rollout (rollouts:) each wave falls inside. If some runs have an extra rollout that others do not, Step 4 is needed: follow 4.3 Revision Trigger.

Step 4: Deep dive (only when Step 3 is not enough)

Only if the parser output does not establish the mechanism, read the Deep-dive artifact download section of references/artifacts.md and follow it: audit logs, etcd logs, the revision trigger (revision_triggers.py), and PromQL for live clusters. Prefix any direct gcloud storage command with CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true.

When a wave sits inside a revision rollout that comparison runs do not have, run revision_triggers.py on a run with the rollout and one without it to find the write that started the revision. Use the script rather than reading events.json or configmaps.json by hand. The commits behind that write are the disruption-change-suspects skill's job: record the component and both runs for it (4.4).

Step 5: Additional checks

references/timeline-analysis.md section 5 has the node-shutdown and endpoint-slice checks. Do them when disruption coincides with node events.

Output

Write one short analysis per run to {runs_dir}/{build_id}/analysis.md:

  • the job, build ID (linked to Prow), cohort if any, payload pair, and liveness status;
  • the episodes and waves on the selected backends, and the phase;
  • the mechanism: what the evidence shows caused each wave, with the evidence linked inline (link shapes and payload rules: references/evidence-links.md). Say "not determined" when it is not, and what was checked;
  • the other disrupted backends (blast radius), briefly.

Then answer with a compact table (one row per run: build ID, cohort, episodes/waves, mechanism, liveness) and the mechanisms in a few sentences. When there are several runs, say which mechanisms repeat across runs and which appear once. Do not compare before and after cohorts here; that is disruption-compare-cohorts.

End the turn with:

```disruption-checkpoint
stage: run-analysis
state_file: <path to investigation.json>
summary: <one line: runs analyzed, the recurring mechanism, anything undetermined>
next:
  - compare the cohorts (disruption-compare-cohorts)
  - look for the change behind <component> between <run without> and <run with> (disruption-change-suspects)
  - search Jira for this mechanism (disruption-known-issues)
  - deep-dive run <build_id>
```

List only the options that fit: comparing cohorts only when the runs are cohorts, the change search only when a component is implicated. Then stop and wait for the user's choice, unless the request said to run without pausing (--auto, or "just run it"); then record the default (compare the cohorts when there are cohorts, otherwise search Jira) and continue.

Error handling

  1. No disruption found: report that the run is clean on these backends. This is a valid result.
  2. Interval files not found: a critical error for that run. Report it and continue with the others.
  3. Audit or etcd logs missing: note it and continue with the available data.
  4. gcloud errors: report the run's error text (credential-like values are removed). A sandbox that blocks the network is separate from an auth failure; request network execution and run the downloader again. Do not run gcloud auth login.

Performance

  • One downloader call for every run; it works in parallel.
  • Use --format summary to triage many runs before deep-diving, so large JSON stays in files.
  • Filter audit logs by time range rather than scanning whole files.

From the openshift-eng/ai-helpers README

AI Helpers

skillsaw grade

A collection of Claude Code plugins to automate and assist with various development tasks.

Discover available plugins

Installation

From the Claude Code Plugin Marketplace

  1. Add the marketplace:

    /plugin marketplace add openshift-eng/ai-helpers
    
  2. Install a plugin:

    /plugin install jira@ai-helpers
    
  3. Use the commands:

    /jira:solve OCPBUGS-12345 origin
    

Updating Plugins

To get the latest plugin versions:

  1. Update the marketplace (fetches latest plugin catalog):

    /plugin marketplace update ai-helpers
    
  2. Reinstall the plugin (downloads new version):

    /plugin install <plugin>@ai-helpers
    

Automatic Catalog Sync

Add a SessionStart hook to automatically sync the marketplace catalog on each session. In your project's .claude/settings.json:

{
  "hooks": {
    "SessionStart": [
      {
        "command": "claude plugin marketplace update ai-helpers",
        "timeout": 30000
      }
    ]
  }
}

Note: This only refreshes the catalog (what's available). To actually update an installed plugin to a newer version, you still need to reinstall it with /plugin install <plugin>@ai-helpers.

Other Tools

Coding agents like OpenCode, Gemini, Cursor and more can consume Claude Code plugins using the Agent Package Manager (APM).

Example apm.yml:

name: my-project
version: 1.0.0
description: My project is great. 
target: [claude, cursor, gemini, opencode]

dependencies:
  - openshift-eng/ai-helpers/plugins/bigquery-cost-analysis

Then run apm install. It can install to your project only, or with a --global scope.

Using the Container

A container is available with Claude Code and all plugins pre-installed. This is primarily for use in OpenShift CI.

The image includes two Claude Code binaries:

  • claude (default entrypoint) — installed from the stable RPM channel
  • claude-latest — installed from the latest RPM channel, for trying newer features or comparing behavior between versions

Building the Container

podman build -f images/Dockerfile -t ai-helpers .

Running with Vertex AI and gcloud Authentication

To use Claude Code with Google Cloud's Vertex AI, you need to pass through your gcloud credentials and set the required environment variables:

podman run -it \
  -e CLAUDE_CODE_USE_VERTEX=1 \
  -e CLOUD_ML_REGION=your-ml-region \
  -e ANTHROPIC_VERTEX_PROJECT_ID=your-project-id \
  -v ~/.config/gcloud:/home/claude/.config/gcloud:ro \
  -v $(pwd):/workspace \
  -w /workspace \
  ai-helpers

Environment Variables:

  • CLAUDE_CODE_USE_VERTEX=1 - Enable Vertex AI integration
  • CLOUD_ML_REGION - Your GCP region (e.g., us-east5)
  • ANTHROPIC_VERTEX_PROJECT_ID - Your GCP project ID

Volume Mounts:

  • -v ~/.config/gcloud:/home/claude/.config/gcloud:ro - Passes through your gcloud authentication (read-only)
  • -v $(pwd):/workspace - Mounts your current directory into the container

Running Commands Non-Interactively

You can execute Claude Code commands directly without entering an interactive session using the -p or --print flag:

podman run -it \
  -e CLAUDE_CODE_USE_VERTEX=1 \
  -e CLOUD_ML_REGION=your-ml-region \
  -e ANTHROPIC_VERTEX_PROJECT_ID=your-project-id \
  -v ~/.config/gcloud:/home/claude/.config/gcloud:ro \
  -v $(pwd):/workspace \
  -w /workspace \
  ai-helpers \
  --print "/hello-world:echo Hello from Claude Code!"

This will:

  1. Start the container with your gcloud credentials
  2. Execute the /hello-world:echo command with the provided message
  3. Print the response and exit when complete

Available Plugins

For a complete list of all available plugins and commands, see the AI Helpers Marketplace.

Plugin Development

Want to contribute or create your own plugins? Check out the plugins/ directory for examples. Make sure your commands and agents follow the conventions for the Sections structure presented in the hello-world reference implementation plugin (see hello-world:echo for an example).

Ethical Guidelines

Plugins, commands, skills, and hooks must NEVER reference real people by name, even as stylistic examples (e.g., "in the style of ").

Ethical rationale:

  1. Consent: Individuals have not consented to have their identity or persona used in AI-generated content
  2. Misrepresentation: AI cannot accurately replicate a person's unique voice, style, or intent
  3. Intellectual Property: A person's distinctive style may be protected
  4. Dignity: Using someone's identity without permission diminishes their autonomy

Instead, describe specific qualities explicitly

Good examples:

  • "Write commit messages that are direct, technically precise, and focused on the rationale behind changes"
  • "Explain using clear analogies, a sense of wonder, and accessible language for non-experts"
  • "Code review comments that are encouraging, constructive, and focus on collaborative improvement"

When you identify a desirable characteristic (clarity, brevity, formality, humor, etc.), describe it explicitly rather than using a person as proxy.

Adding New Commands

Check for overlaps first - Before coding, validate your idea:

/utils:review-ai-helpers-overlap --idea "brief description of your command"

Collaborating on existing work instead of duplicating parallel efforts is always encouraged when overlap is found. This helps maintain a clean, non-redundant plugin collection in such an actively developed project (see /utils:review-ai-helpers-overlap for detailed usage).

When contributing new commands:

  1. If your command fits an existing plugin: Add it to the appropriate plugin's commands/ directory
  2. If your command doesn't have a clear parent plugin: Add it to the utils plugin (plugins/utils/commands/)
    • The utils plugin serves as a catch-all for commands that don't fit existing categories
    • Once we accumulate several related commands in utils, they can be segregated into a new targeted plugin

Creating a New Plugin

If you're contributing several related commands that warrant their own plugin:

  1. Create a new directory under plugins/ with your plugin name
  2. Create the plugin structure:
    plugins/your-plugin/
    ├── .claude-plugin/
    │   └── plugin.json
    └── commands/
        └── your-command.md
    
  3. Register your plugin in .claude-plugin/marketplace.json

Validating Plugins

This repository uses skillsaw to validate plugin structure:

make lint

Previewing the Marketplace Site

Install the dependencies from site/requirements.txt, then run:

make site-serve

The MkDocs site is available at http://127.0.0.1:8000/ai-helpers/. GitHub Pages builds the same generated content after changes merge to main.

Additional Documentation

  • AI Helpers Marketplace - Complete list of all available plugins and commands
  • CONTRIBUTING.md - Guidelines for contributing plugins, including versioning policy
  • AGENTS.md - Complete guide for AI agents working with this repository
  • CLAUDE.md - Claude-specific configuration and notes

License

See LICENSE for details.

Details

Marketplace
openshift-eng/ai-helpers
Source
plugins/disruption
Category
Ci
Keywords
disruption, openshift-ci, prow, sippy, regression

More from openshift-eng/ai-helpers

All 57

Similar to disruption

More Development Tools