
disruption
Investigate OpenShift CI disruption regressions: when disruption changed, and why runs were disrupted
- version
- 0.0.2
- GitHub stars
- 123
Add marketplace
/plugin marketplace add openshift-eng/ai-helpersRun it inside Claude Code once, then install the plugin below.
Install
/plugin install disruption@ai-helpersRun it inside Claude Code after adding the marketplace.
Files
Discovered from sourceDisruption Analyze Run
This skill answers why these runs were disrupted: the mechanism in each run. It does not decide whether disruption got worse, compare cohorts, or search Jira; other disruption skills do that.
Prerequisites
- gcloud CLI (
which gcloud). Thetest-platform-results-publicbucket is public. Do not rungcloud auth login, and do not change the user's gcloud config.download_timelines.pysetsCLOUDSDK_AUTH_DISABLE_CREDENTIALS=trueon each gcloud subprocess, and any directgcloud storagecommand here uses the same variable. Anonymous access still needs network execution when the command runs in a sandbox. - Python 3.7+.
- The
disruption_commonlibrary: thedisruptionplugin'sscripts/disruption_common, installed with the plugin. If it is missing, the scripts exit and say so. Report the message; do not work around it.
The scripts are in this skill's directory: ${CLAUDE_SKILL_DIR} in Claude Code, or the directory containing this SKILL.md elsewhere.
Rules shared by the disruption skills
- Backend names are exact. Pass them as
--backendsunchanged. Do not shortenkube-api-new-connectionstokube-api, and do not substitute derived probes (*-localhost-*,*-http1-*,*-http2-*,*-service-network-*,*-internal-lb-*) orcache-*backends unless the user named them. - Liveness. A run where
ci-cluster-network-livenessis disrupted has unreliable disruption data. Keep it, say so, and do not rest a conclusion on its disruption counts alone. - Platform. The platform in the alert or job name is where disruption was measured, not evidence of the cause. Name a platform-specific cause only when the run evidence shows a platform mechanism (a cloud metric, a cloud API error, a platform-only component).
- State. With a state file (
investigation.json),{investigation_dir}is its directory; otherwise create.work/disruption-analysis/<YYYY-MM-DD>-<short-slug>/. Runs go under{runs_dir}={investigation_dir}/runs. - One block. Do not run another disruption skill yourself. End with the checkpoint below.
Step 1: Inputs
- Prow URLs (
prow.ci.openshift.orgorgcsweb-ci), one or more. Orjob:build_idpairs, such ascohorts.handoff.download_runsfromdisruption-find-runsin the state file. - Backends (
--backends): the exact names. Take them from the request, or from the state file'sseries.backends, or else itscohorts.backends(written bycohorts.py validate). Without any of these, analyze every backend that shows disruption and say so. - When the runs are cohorts (before/after), keep each run's cohort; the analysis of each run is the same, but the checkpoint says which cohort each run is in.
From each Prow URL take the job name and build ID: the last two path segments after /logs/. The job name is letters, digits, ., _, and -, and the build ID is all digits; skip and report anything else rather than putting it in a shell command. Presubmit runs (/pr-logs/pull/...) are not supported: the download reads only /logs/ artifacts, so say so and skip them. Prow URLs may say origin-ci-test; the bucket is always test-platform-results-public.
Step 2: Download timelines
Read references/artifacts.md and follow Timeline download: one download_timelines.py call for every run, with --output-dir {runs_dir}. When the runs are before/after cohorts, add --gather: disruption-compare-cohorts compares component versions from each run's releaseinfo.json, and the revision-trigger check needs the events and configmaps. On exit 2, continue with the runs that downloaded; on exit 1, stop and report each run's error.
Keep each run's payload pair (initial → target) from the output. For upgrade jobs, make no claim that depends on which payload a run started from or upgraded to without that run's pair; if it is unknown, say so.
Step 3: Analyze the timelines
For every run, save the parser's full JSON (the cohort comparison reads it) and print its one-line summary, in one loop:
for run in '{job_1}:{build_id_1}' '{job_2}:{build_id_2}'; do
job="${run%%:*}"; build_id="${run##*:}"; dir="{runs_dir}/${build_id}"
if python3 "${CLAUDE_SKILL_DIR}/parse_disruption.py" "${dir}"/logs/e2e-timelines_spyglass_*.json \
--job-name "${job}" --build-id "${build_id}" --backends '{exact_backends}' --format json > "${dir}/parsed.json.tmp"; then
mv "${dir}/parsed.json.tmp" "${dir}/parsed.json"
python3 "${CLAUDE_SKILL_DIR}/parse_disruption.py" "${dir}"/logs/e2e-timelines_spyglass_*.json \
--job-name "${job}" --build-id "${build_id}" --backends '{exact_backends}' --format summary
else
rm -f "${dir}/parsed.json.tmp" "${dir}/parsed.json"; echo "${build_id}: the parser failed; no parsed.json kept" >&2
fi
done
A run without timelines gets no parsed.json, and a failed rerun removes the one from an earlier run, so the cohort comparison reports the run as missing instead of reading an empty or stale file.
Then read references/timeline-analysis.md and follow 3.2 through 3.4 (blast radius, deep-dive of the runs that need it, and how to read each signal).
Compare runs by the parser's episodes and waves (for example "3 episodes in 1 wave" against "6 episodes in 2 waves (3+3)"), not by the disruption count. Check which static-pod revision rollout (rollouts:) each wave falls inside. If some runs have an extra rollout that others do not, Step 4 is needed: follow 4.3 Revision Trigger.
Step 4: Deep dive (only when Step 3 is not enough)
Only if the parser output does not establish the mechanism, read the Deep-dive artifact download section of references/artifacts.md and follow it: audit logs, etcd logs, the revision trigger (revision_triggers.py), and PromQL for live clusters. Prefix any direct gcloud storage command with CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true.
When a wave sits inside a revision rollout that comparison runs do not have, run revision_triggers.py on a run with the rollout and one without it to find the write that started the revision. Use the script rather than reading events.json or configmaps.json by hand. The commits behind that write are the disruption-change-suspects skill's job: record the component and both runs for it (4.4).
Step 5: Additional checks
references/timeline-analysis.md section 5 has the node-shutdown and endpoint-slice checks. Do them when disruption coincides with node events.
Output
Write one short analysis per run to {runs_dir}/{build_id}/analysis.md:
- the job, build ID (linked to Prow), cohort if any, payload pair, and liveness status;
- the episodes and waves on the selected backends, and the phase;
- the mechanism: what the evidence shows caused each wave, with the evidence linked inline (link shapes and payload rules:
references/evidence-links.md). Say "not determined" when it is not, and what was checked; - the other disrupted backends (blast radius), briefly.
Then answer with a compact table (one row per run: build ID, cohort, episodes/waves, mechanism, liveness) and the mechanisms in a few sentences. When there are several runs, say which mechanisms repeat across runs and which appear once. Do not compare before and after cohorts here; that is disruption-compare-cohorts.
End the turn with:
```disruption-checkpoint
stage: run-analysis
state_file: <path to investigation.json>
summary: <one line: runs analyzed, the recurring mechanism, anything undetermined>
next:
- compare the cohorts (disruption-compare-cohorts)
- look for the change behind <component> between <run without> and <run with> (disruption-change-suspects)
- search Jira for this mechanism (disruption-known-issues)
- deep-dive run <build_id>
```
List only the options that fit: comparing cohorts only when the runs are cohorts, the change search only when a component is implicated. Then stop and wait for the user's choice, unless the request said to run without pausing (--auto, or "just run it"); then record the default (compare the cohorts when there are cohorts, otherwise search Jira) and continue.
Error handling
- No disruption found: report that the run is clean on these backends. This is a valid result.
- Interval files not found: a critical error for that run. Report it and continue with the others.
- Audit or etcd logs missing: note it and continue with the available data.
- gcloud errors: report the run's
errortext (credential-like values are removed). A sandbox that blocks the network is separate from an auth failure; request network execution and run the downloader again. Do not rungcloud auth login.
Performance
- One downloader call for every run; it works in parallel.
- Use
--format summaryto triage many runs before deep-diving, so large JSON stays in files. - Filter audit logs by time range rather than scanning whole files.
From the openshift-eng/ai-helpers README
AI Helpers
A collection of Claude Code plugins to automate and assist with various development tasks.
Installation
From the Claude Code Plugin Marketplace
-
Add the marketplace:
/plugin marketplace add openshift-eng/ai-helpers -
Install a plugin:
/plugin install jira@ai-helpers -
Use the commands:
/jira:solve OCPBUGS-12345 origin
Updating Plugins
To get the latest plugin versions:
-
Update the marketplace (fetches latest plugin catalog):
/plugin marketplace update ai-helpers -
Reinstall the plugin (downloads new version):
/plugin install <plugin>@ai-helpers
Automatic Catalog Sync
Add a SessionStart hook to automatically sync the marketplace catalog on each session. In your project's .claude/settings.json:
{
"hooks": {
"SessionStart": [
{
"command": "claude plugin marketplace update ai-helpers",
"timeout": 30000
}
]
}
}
Note: This only refreshes the catalog (what's available). To actually update an installed plugin to a newer version, you still need to reinstall it with /plugin install <plugin>@ai-helpers.
Other Tools
Coding agents like OpenCode, Gemini, Cursor and more can consume Claude Code plugins using the Agent Package Manager (APM).
Example apm.yml:
name: my-project
version: 1.0.0
description: My project is great.
target: [claude, cursor, gemini, opencode]
dependencies:
- openshift-eng/ai-helpers/plugins/bigquery-cost-analysis
Then run apm install. It can install to your project only, or with a --global scope.
Using the Container
A container is available with Claude Code and all plugins pre-installed. This is primarily for use in OpenShift CI.
The image includes two Claude Code binaries:
claude(default entrypoint) — installed from the stable RPM channelclaude-latest— installed from the latest RPM channel, for trying newer features or comparing behavior between versions
Building the Container
podman build -f images/Dockerfile -t ai-helpers .
Running with Vertex AI and gcloud Authentication
To use Claude Code with Google Cloud's Vertex AI, you need to pass through your gcloud credentials and set the required environment variables:
podman run -it \
-e CLAUDE_CODE_USE_VERTEX=1 \
-e CLOUD_ML_REGION=your-ml-region \
-e ANTHROPIC_VERTEX_PROJECT_ID=your-project-id \
-v ~/.config/gcloud:/home/claude/.config/gcloud:ro \
-v $(pwd):/workspace \
-w /workspace \
ai-helpers
Environment Variables:
CLAUDE_CODE_USE_VERTEX=1- Enable Vertex AI integrationCLOUD_ML_REGION- Your GCP region (e.g.,us-east5)ANTHROPIC_VERTEX_PROJECT_ID- Your GCP project ID
Volume Mounts:
-v ~/.config/gcloud:/home/claude/.config/gcloud:ro- Passes through your gcloud authentication (read-only)-v $(pwd):/workspace- Mounts your current directory into the container
Running Commands Non-Interactively
You can execute Claude Code commands directly without entering an interactive session using the -p or --print flag:
podman run -it \
-e CLAUDE_CODE_USE_VERTEX=1 \
-e CLOUD_ML_REGION=your-ml-region \
-e ANTHROPIC_VERTEX_PROJECT_ID=your-project-id \
-v ~/.config/gcloud:/home/claude/.config/gcloud:ro \
-v $(pwd):/workspace \
-w /workspace \
ai-helpers \
--print "/hello-world:echo Hello from Claude Code!"
This will:
- Start the container with your gcloud credentials
- Execute the
/hello-world:echocommand with the provided message - Print the response and exit when complete
Available Plugins
For a complete list of all available plugins and commands, see the AI Helpers Marketplace.
Plugin Development
Want to contribute or create your own plugins? Check out the plugins/ directory for examples.
Make sure your commands and agents follow the conventions for the Sections structure presented in the hello-world reference implementation plugin (see hello-world:echo for an example).
Ethical Guidelines
Plugins, commands, skills, and hooks must NEVER reference real people by name, even as stylistic examples (e.g., "in the style of ").
Ethical rationale:
- Consent: Individuals have not consented to have their identity or persona used in AI-generated content
- Misrepresentation: AI cannot accurately replicate a person's unique voice, style, or intent
- Intellectual Property: A person's distinctive style may be protected
- Dignity: Using someone's identity without permission diminishes their autonomy
Instead, describe specific qualities explicitly
Good examples:
- "Write commit messages that are direct, technically precise, and focused on the rationale behind changes"
- "Explain using clear analogies, a sense of wonder, and accessible language for non-experts"
- "Code review comments that are encouraging, constructive, and focus on collaborative improvement"
When you identify a desirable characteristic (clarity, brevity, formality, humor, etc.), describe it explicitly rather than using a person as proxy.
Adding New Commands
Check for overlaps first - Before coding, validate your idea:
/utils:review-ai-helpers-overlap --idea "brief description of your command"
Collaborating on existing work instead of duplicating parallel efforts is always encouraged when overlap is found. This helps maintain a clean, non-redundant plugin collection in such an actively developed project (see /utils:review-ai-helpers-overlap for detailed usage).
When contributing new commands:
- If your command fits an existing plugin: Add it to the appropriate plugin's
commands/directory - If your command doesn't have a clear parent plugin: Add it to the utils plugin (
plugins/utils/commands/)- The utils plugin serves as a catch-all for commands that don't fit existing categories
- Once we accumulate several related commands in utils, they can be segregated into a new targeted plugin
Creating a New Plugin
If you're contributing several related commands that warrant their own plugin:
- Create a new directory under
plugins/with your plugin name - Create the plugin structure:
plugins/your-plugin/ ├── .claude-plugin/ │ └── plugin.json └── commands/ └── your-command.md - Register your plugin in
.claude-plugin/marketplace.json
Validating Plugins
This repository uses skillsaw to validate plugin structure:
make lint
Previewing the Marketplace Site
Install the dependencies from site/requirements.txt, then run:
make site-serve
The MkDocs site is available at http://127.0.0.1:8000/ai-helpers/. GitHub
Pages builds the same generated content after changes merge to main.
Additional Documentation
- AI Helpers Marketplace - Complete list of all available plugins and commands
- CONTRIBUTING.md - Guidelines for contributing plugins, including versioning policy
- AGENTS.md - Complete guide for AI agents working with this repository
- CLAUDE.md - Claude-specific configuration and notes
License
See LICENSE for details.
Details
- Marketplace
- openshift-eng/ai-helpers
- Source
- plugins/disruption
- Category
- Ci
- Keywords
- disruption, openshift-ci, prow, sippy, regression
More from openshift-eng/ai-helpers
All 57- docv0.0.2
- etcdv0.0.2
- gitv0.0.7
- githubv0.3.0
- golangv0.7.0
- gopls-lspv1.0.0
- gwapiv0.0.1
- hcpv0.0.3
- hello-worldv1.0.3
- jirav0.9.9
- lvmsv0.1.1
- marketplace-opsv0.1.2
- metricsv0.3.0
- must-gatherv0.0.3
- native-notificationsv0.0.2
- nid-teamv0.2.2
- nodev0.0.3
- node-bugv0.0.3
- node-cvev0.0.8
- node-onboardingv0.0.3
- node-rpmv0.0.3
- node-teamv1.0.1
- node-tuningv1.0.1
- olmv0.1.3
Similar to disruption
More Development Tools
reamorluto/rea466.8K starsReverse engineer anything with agents, from app behavior down to native binaries.
superpowersobra/superpowers319.5K starsCore skills library for Claude Code: TDD, debugging, collaboration patterns, and proven techniques
claude-apianthropics/skills196.9K starsClaude API and SDK documentation skill for building LLM-powered applications
prompts.chatf/prompts.chat178.2K starsAccess thousands of AI prompts and skills directly in your AI coding assistant. Search prompts, discover skills, save your own, and improve prompts with AI.
agent-sdk-devanthropics/claude-code156.1K starsDevelopment kit for working with the Claude Agent SDK
ui-ux-pro-maxnextlevelbuilder/ui-ux-pro-max-skill152.6K starsProfessional UI/UX design intelligence for AI coding assistants. Includes searchable databases of styles, colors, typography, charts, and UX guidelines for Reac