A plugin to work with OpenShift CI and analyze Prow job results
- version
- 0.0.111
- GitHub stars
- 123
Install
/plugin install ci@ai-helpersRun it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.
Contents
Discovered from sourceSkills 36
Add JIRA Triage Link
This skill adds a Component Readiness triage record link to a JIRA issue description so that anyone viewing the bug can monitor the on-going status of all regressions triaged to it.
When to Use This Skill
Use this skill after creating or updating a triage record via the triage-regression skill. The link gives bug viewers a way to see all regressions associated with the bug in Component Readiness.
Prerequisites
- Jira credentials in the environment, either:
- JIRA_USERNAME and JIRA_API_TOKEN (Atlassian account email and API token), or
- JIRA_CLIENT_ID, JIRA_CLIENT_SECRET and JIRA_CLOUD_ID (service-account OAuth credential; used whenever the client ID or secret is set, with no fallback to the API token)
- Python 3.6+
Implementation Steps
Step 1: Run the Script
script_path="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$script_path" OCPBUGS-12345 \
--triage-id 456 \
--format json
Arguments:
issue_key: JIRA issue key (e.g., OCPBUGS-12345)
Required Options:
--triage-id <id>: Triage record ID from Component Readiness
Options:
--format json|text: Output format (default: text)
Step 2: Parse the Output
output=$(python3 "$script_path" OCPBUGS-12345 --triage-id 456 --format json)
success=$(echo "$output" | jq -r '.success')
if [ "$success" = "true" ]; then
already=$(echo "$output" | jq -r '.already_present')
if [ "$already" = "true" ]; then
echo "Triage link already in description"
else
echo "Triage link added"
fi
fi
Script Output Format
JSON Format (--format json)
Success (link added):
{
"success": true,
"issue_key": "OCPBUGS-12345",
"triage_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/456",
"already_present": false
}
Success (link already present):
{
"success": true,
"issue_key": "OCPBUGS-12345",
"triage_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/456",
"already_present": true
}
Error:
{
"success": false,
"error": "Failed to update description: HTTP 403: ...",
"issue_key": "OCPBUGS-12345"
}
Notes
- The script is idempotent — if the triage link is already in the description, it skips the update
- Uses Atlassian Document Format (ADF) to append a horizontal rule and the triage link section
- Appends to the existing description without modifying existing content
- Requires JIRA API v3 write access to the issue
See Also
- Related Skill:
triage-regression(creates triage records that produce the triage ID) - Related Skill:
set-release-blocker(similar pattern for updating JIRA issue fields) - Related Command:
/ci:analyze-regression(calls this skill after triaging)
Bulk Triage Regressions
Input
bulk-triage-regressions <view> [--components comp1 comp2 ...] [--auto-triage] [--existing-only]
bulk-triage-regressions --views <v1> <v2> ... --policy post-ga [--write-mode report|auto]
Example: bulk-triage-regressions 5.0-main --components Installer Unknown
Description
This skill implements the Component Readiness triage duty workflow: it fetches all untriaged regressions for a set of components in a view (e.g., 5.0-main, components Installer and Unknown), analyzes them as a batch, clusters them into root-cause buckets, and then triages each bucket to a single JIRA bug (existing or new).
This differs from /ci:analyze-regression (which analyzes a single regression in depth). Triage duty requires a holistic view, because:
-
Many regressions, few root causes. One product bug commonly opens 5–30 regressions across variants (different platforms, arches, featuresets, upgrade modes) and across "wrapper" tests (
install should succeed: overall,: cluster bootstrap,: cluster creation,verify the cluster readiness and stability, mass-failure tests, etc.). Analyzing regressions one-by-one wastes effort and risks filing duplicate bugs. Cluster first, deep-dive once per cluster. -
Component attribution is often wrong. Regressions in
InstallerandUnknownare catch-all attributions. A failed installation or bootstrap is frequently caused by a specific component — e.g., a monitoring operator failing to go available blocks cluster creation, an etcd slowness issue breaks bootstrap, an MCO bug degrades nodes during install. The Sippy component label tells you which test failed, not whose bug it is. The real owner must be determined from artifacts (cluster operator status, log bundle, operator logs), and the JIRA bug must be filed against the actual owning component, not Installer.
Use this skill when doing triage duty for a view, or whenever a user asks to "look at all untriaged regressions from " rather than a single regression ID.
Implementation
Script invocation rules: Run Python skill scripts directly and analyze their JSON output with your own reasoning (pass --format json where the script offers the flag; scripts without it, such as list_regressions.py, emit JSON by default). Do not pipe script output through inline Python one-liners. Do not suppress stderr: if a script exits non-zero or returns invalid/empty JSON, stop and surface the error — an authentication or API failure must never be mistaken for an empty inventory ("nothing to triage").
Authentication: Read steps (listing, fetching details, test runs, GCS artifacts) require no auth — Phases 1–3 and a read-only report must work without any credentials. Two Sippy hosts exist and the split matters: https://sippy.dptools.openshift.org is public and read-only (regressions, test details, and the full symptom/label catalog — see the list-symptoms skill), while everything on https://sippy-auth.dptools.openshift.org requires a DPCR Bearer token, including the reevaluate probe, which has no public equivalent (the route returns 404 on the public host). The only Phase 3 step that needs a token is therefore the active symptom probe. When it is unavailable (no token, or a 401/403), treat the result as an unknown — never as "no symptoms matched", the same error as reading a failed listing script as an empty inventory, and work down this fallback: (1) passive label_summary / job_labels from fetch-regression-details; (2) list the catalog from the public API with the list-symptoms skill and grep each plausible symptom's match_string against its file_pattern in the GCS artifacts you are already fetching — this reproduces the probe locally for the handful of symptoms related to the bucket's stage and platform; (3) only then the full evidence ladder, recording that the active probe was unavailable so the next shift knows the catalog was checked by hand. Write credentials are validated only when writes are going to happen: with --auto-triage, validate both up front (so an expired token surfaces before hours of analysis); otherwise validate at the start of Phase 4, before the first write. Sippy writes (creating/updating triage records) require a Bearer token from the DPCR cluster (api.cr.j7t7.p1.openshiftapps.com:6443) — see the oc-auth skill and the token-extraction snippet in /ci:analyze-regression; check with an authenticated GET against https://sippy-auth.dptools.openshift.org/api/component_readiness/triages (200 vs 401/403).
JIRA writes (filing bugs, set-release-blocker, add-jira-triage-link) additionally require JIRA credentials in the environment: either JIRA_USERNAME and JIRA_API_TOKEN (API token from https://id.atlassian.com/manage-profile/security/api-tokens), or a service-account OAuth credential in JIRA_CLIENT_ID, JIRA_CLIENT_SECRET and JIRA_CLOUD_ID, which takes precedence when set and never falls back to the API token. Verify either kind with python3 plugins/ci/scripts/jira_auth.py check (exit 0 and the account vs an error). With a service-account credential, Basic auth against https://redhat.atlassian.net does not work: send every direct JIRA REST call (searches, comments, issue creation when the Jira MCP tools are unavailable) through python3 plugins/ci/scripts/jira_auth.py request <METHOD> '/rest/api/3/...' [--data-file body.json], which obtains the access token itself and prints only the response. If a required write credential is missing or invalid, pause before Phase 4 and ask the user to fix it — the analysis and report so far remain valid and must still be presented.
Phase 0: Run the pre-pass (post-ga only)
With --policy post-ga, first run scripts/prepass.py with the given views. Treat the prepass.json clusters as the complete batch inventory: do not re-derive clusters, and do not call the list/details/related APIs for inventory. Analyse clusters in file order (HIGH first) within the time budget. A symptom_probe.status other than ok means unknown, never "no symptoms matched". Read and follow references/post-ga-policy.md for everything else in this mode, and skip Phase 1's collection steps.
Phase 1: Collect the full batch
-
Load CI context: Read the files in
plugins/ci/references/(jobs.md,tests.md,sippy-apis.md) for conventions on tests, jobs, and Sippy APIs. -
Parse arguments:
view: required, e.g.5.0-main--components: component filter list, e.g.Installer Unknown. Matching is case-insensitive and hierarchy-aware: a filter matches the full component name or any/-separated segment of it, soInstalleralso coversInstaller / openshift-installer, andNetworkingcoversNetworking / ovn-kubernetes,Networking / router, and every otherNetworking / *component. If omitted, ask the user which components the duty covers.--auto-triage: if present, triage buckets without per-bucket confirmation when confidence is high (see Phase 4). Default is to present findings and confirm before writing.--existing-only: never file new JIRA issues; write only for matches to existing issues (see "Existing-only mode" in Phase 4).
-
List regressions with the
list-regressionsskill:python3 plugins/teams/skills/list-regressions/list_regressions.py \ --view <view> --components <components...>Keep only open, untriaged regressions (empty
triagesarray), but note recently-triaged ones — they are prime candidates for absorbing untriaged siblings.Closed regressions are out of scope — even when untriaged. A regression whose
closedfield is set has already resolved itself; do not inventory it, cluster it, deep-dive it, or recommend retroactive triage for it. The duty batch consists solely of open untriaged regressions. Closed regressions may be consulted as evidence (e.g., a closed sibling that shares a root cause with an open bucket, or a closed sibling whose existing triage/JIRA an open bucket should reuse — see Pitfalls), but they must never appear as bucket members, action items, or "leftovers" in the report. The only exception is the explicit closed-set audit mode (--audit-closed, see below), which the user must request by name — it is never part of a normal duty run. -
Build a batch inventory table: For every open untriaged regression record: regression ID, test name, component/capability, variants (Platform/Arch/Network/Topology/FeatureSet/Upgrade), opened date, failure/run counts. Present this table to the user up front so the scope of the duty run is visible. Do not include closed regressions in the inventory.
-
Stale-triage sweep (mandatory) — a 100% triaged board can still hide live defects. For every open, already-triaged regression, compare its
last_failureagainst the state of its triage's JIRA: a regression whose bug is Closed/Verified/resolved but which has failed after the resolution date is an alarm, not a statistic. Its fresh failure window is either a failed fix or — more often — a different cause hiding behind the old triage record. For each such regression, verify the recent runs' signature (armed Sippy symptoms are a cheap first oracle: dry-runreevaluatethe newest runs — label hits map windows to known causes instantly; unlabeled recent runs mean a new, uninvestigated cause; a hit label whosebugsdiffer from the triage's JIRA suggests a new cause, one listing the same closed bug a failed fix; a label linking several bugs is inconclusive — compare the runs' signature and timing with each) and add the correct triage(s) for the new window rather than trusting the stale one. (Case 4 in case-notes.md: three "triaged" regressions pointing at a Closed bug while failing daily from two new causes.) -
Long-lived wrapper regressions are cause timelines, not single buckets. A wrapper regression that stays open for weeks accumulates causes, each window separately verified and separately triaged. When a previously-analyzed regression shows new
last_failuredates, re-verify the new window from scratch — never assume the existing triage covers it. (Case 13: one record carried four causes.)
Phase 2: Cluster into candidate buckets (cheap signals first)
Before any deep log analysis, group regressions using signals already in hand:
- Same test, different variants — almost always one bucket.
- Same variant fingerprint, different tests — e.g.,
install should succeed: overall+: cluster creation+verify the cluster readiness and stabilityall failing onazure/amd64/techpreviewstarting the same day is one bucket. Wrapper tests fail together. - Same opened date — regressions opened the same day across components often share one payload-level cause.
- Shared job runs — fetch details for each regression (
fetch-regression-detailsskill) and comparejob_runsprowjob_run_ids. Regressions observed in the same failed runs are strong candidates for one bucket — but this is a clustering signal, not proof: the same run can contain independent defects or be a mass failure, so Phase 3 validation is still required. Also run thefetch-related-triagesskill per regression;same_last_failureandsimilarly_named_testmatches feed the clustering, andtriaged_matcheswith confidence ≥5 immediately suggest an existing triage/bug for the whole bucket. - Symptom labels —
fetch-regression-detailsreturnslabel_summary(per job) andjob_labels(per run). A label shared by most failed runs of several regressions is a strong bucket signal, and labels are precise (human-written matchers over artifacts).label_bugsmaps each label to its linked Jira keys: labels sharing a bug are a bucket signal, and the bug is a candidate triage target. An empty label list means nothing was detected — not that nothing is wrong, and often just that the runs were never swept: Phase 3 opens with an active dry-run reevaluation that closes this gap. - Mass-failure marker: high
test_failurescounts injob_runsmean the regression is likely collateral of a bigger event, not an independent issue.
Output of this phase: a draft bucket list, each bucket with member regression IDs, the shared fingerprint (test/variant/date/job-run overlap), and any candidate existing triage or JIRA bug.
Treat buckets as hypotheses — Phase 3 must confirm or split them. Do not merge buckets merely because both are "install failures"; installs fail at different stages for unrelated reasons.
Phase 3: Deep-dive each bucket (confirm root cause and real owner)
For each bucket, pick 2–5 representative failed job runs (spread across jobs/variants; include the newest) and analyze. 2–5 runs are sufficient only when they yield a consistent result (same error signature / failure stage across all of them). If the sample is mixed or unclear — different errors, different stages, or an inconclusive owner — extend the sample to 10–20 runs before concluding; a small ambiguous sample must never be the basis for splitting/merging a bucket or attributing an owner.
Cross-cutting analysis principles — most specific rules below (and most Pitfalls) are instances of these three; when a situation matches none of the specific rules, fall back to the principle:
- Messenger vs. owner. The component that reports a failure is presumed the messenger until artifacts prove it is the producer. This covers: the installer for install failures, wrapper tests for whatever blocked them, a test whose setup failed on another component's error (Case 9 in case-notes.md), a crash-looper whose error names a missing upstream dependency, and any component that names a bad value someone else computed (Case 12). Ask "who produced the condition this error describes?" and walk upstream until the answer is "this component itself".
- Population claims require population evidence. Any claim quantifying over runs — "resolved", "stopped", "all runs show X", "only variant Y" — must enumerate the population it quantifies over: all of the regression's
job_runsby date (never a 3–4 run sample; Case 1), only runs from the regression's ownjob_runslist (never a broad job-filter query; Case 10), and source-code gating for any "X-only" scope claim. - Consult known-cause catalogs before deriving from scratch. Cheap indices exist for almost every question and must be queried before artifact dives or new-bug filings: the armed symptom catalog (dry-run
reevaluate), existing triage records, the owning component's recent JIRA bugs regardless of keywords, the owning repo's merge history at onset/cessation boundaries, and closed/dropped sibling regressions. (Case 3: one dry-run call resolved a board that cost a parallel run 264 shell commands.)
Symptom-catalog oracle first (cheap, decisive — run before any artifact dive or subagent dispatch). The passive label_summary from Phase 2 only shows labels already applied by past sweeps; recent runs are typically unswept and show nothing. Actively probe the catalog instead: POST https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate with {"prow_job_build_ids": [...], "dry_run": true} (DPCR Bearer token) for 3–5 representative runs per bucket, spread across platforms. Reevaluation scans the runs' GCS artifacts server-side against every armed symptom matcher and returns symptoms_matched per run — one call can attribute an entire bucket to a known incident in seconds (Case 3 in case-notes.md). A matched label's bugs (see list-symptoms --labels) name the candidate triage target. Matched symptoms are a hypothesis with a named cause — still verify per the steps below (the label names the incident; confirm the causal chain to the bucket's specific test) — but they dictate where to look first and usually collapse the deep-dive to a single confirmatory artifact read. No matches means the cause is not yet cataloged — proceed with the full evidence ladder.
No token? The catalog is still public — degrade, never skip. A 401/403 from the probe is an unknown, never "no symptoms matched": use the credential-free fallback in the Authentication section above, and say in the report which rung was used.
Deep-dive at least one CI sample per bucket — always, even at high confidence. A Sippy triaged_matches confidence of 10 or a same-day triaged sibling is a hypothesis, not a verdict: Sippy matches on test names and shared job runs, which produces conf=10 for the same test name across unrelated platforms and root causes (see Pitfalls). Before accepting any disposition — including "extend existing triage" — read the actual failure evidence for at least one representative run of the bucket (failure output at minimum; installer logs / artifacts for install wrappers) and confirm it matches the target triage's root cause. You have deeper analysis capabilities than Sippy's heuristics — use your own judgement on the raw evidence, and if it contradicts the Sippy categorization, trust the evidence and re-bucket.
-
Failure outputs:
fetch-test-runsskill with the bucket's test IDs and job run IDs — check whether error messages are consistent within the bucket. >90% same error is strong evidence of a single cause, not confirmation: wrapper tests and mass failures print identical error text for unrelated defects (e.g., "nodes not ready" covers disk, memory, and network deaths alike). Confirm with items 2–3 below before treating the bucket as one cause; inconsistent errors ⇒ split the bucket. -
Job run context:
fetch-job-run-summaryskill per representative run — is the regressed test isolated, part of a consistent co-failure set, or one of hundreds of random failures? ForUnknown-component and mass-failure regressions this is where the real component reveals itself: read the names of the co-failing tests. -
Install/bootstrap failures — mandatory artifact dig: For any bucket whose tests include
install should succeed(any stage) or bootstrap/cluster-readiness wrappers, invoke theprow-job-analysisskill per representative run. Do not stop at Sippy's generic "install failed" wrapper. From the GCS artifacts determine:- Failure stage: infrastructure provisioning / bootstrap / cluster creation (operators rolling out) / stability window.
- The blocking condition: for cluster-creation failures, read
clusteroperators.json(or the installer log's "Cluster operator X is not available" lines) and the failing operator's pod logs from the log bundle. For bootstrap failures, read the bootstrap log bundle (etcd, bootkube, release-image pulls). - Bootstrap-era logs are full of normal transients — validate every causal theory against the gathered end-state. During any bootstrap, operator logs contain scary-looking messages that resolve on their own ("the server could not find the requested resource (post routes.route.openshift.io)" before openshift-apiserver is up, static pods "0 nodes at revision 0" early on). Quoting one of these as the root cause is only valid if the final cluster state agrees. The end-state oracle for bootstrap-stage failures is
clusteroperators.json, not pod existence: check etcd'sStaticPodsAvailable/revision status and openshift-apiserver'sAPIServicesAvailabledirectly — a bootstrap-timed-out run whose gather still showsetcd Available=False: 0 nodes are active; 3 nodes are at revision 0was blocked by etcd no matter what else looks broken. Do not infer "etcd/apiserver were fine" from workload pods running: during bootstrap the bootstrap-node etcd/apiserver serve the control plane, so pods run happily while cluster etcd never deploys. A crash-looping container's*_previous.logis primary evidence for that container's failure — but before assigning ownership, check whether its error names a missing upstream dependency: such a component is a victim, not an owner. - "Operators were not stable" — count condition transitions before classifying. "Healthy at gather time" does NOT imply a harmless transient. Grep the install log for the operator's condition lines and look at
LastTransitionTime/DurationSinceTransition: hundreds of transitions withDurationSinceTransition=1sacross the stability window means the operator is flapping continuously (a sync-loop product bug that will never pass the stability check — permafail), whereas a single longProgressing=Truewindow that eventually clears is a slow rollout (flaky race against the timeout). These two have opposite classifications and dispositions. (Case 1: a ~1 Hz flap, 1600+ transitions in 30 minutes, mislabeled "self-resolved slow rollout" from a 3-run sample.) - Race/ordering theories require timestamp proof — from artifacts, not from the error string. Any "X was not ready when Y ran" mechanism must be proven with timestamps the artifacts already contain (
creationTimestamps,Established/Availableconditions, phase logs, first/last occurrence of the error). If X was ready before the errors began, that refutes this ordering theory — but not every timing theory: readiness at the source does not mean visibility at the consumer, so the next candidates are cache/informer warmup, an in-progress rollout, or a stale client. Name which one the artifacts support rather than dropping the timing angle entirely. Also measure the failure window's duration: it is part of the mechanism claim and belongs in the bug. - Bad-value failures: trace the value to its producer — the error-emitter is usually just the messenger. When a component rejects or fails writing a value (a mapping, an ID range, a path, a quota number), walk the data flow upstream until you reach whoever computed it, however many hops away: the component that names the bad value in its error is the victim's messenger. The producer is then the prime suspect, not the automatic owner — before filing, confirm the value was already wrong as produced: check that no intermediate hop transformed a valid value into an invalid one, and that the value does not violate a documented input contract the consumer was itself obliged to enforce. Whichever hop first turned a valid value into an invalid one owns the bug; name the hops you cleared. (Case 12 in case-notes.md: a GID-mapping error filed against CRI-O when the producer was kubelet's ID allocator two hops upstream — pinns and CRI-O were verified pass-throughs.)
- Decode suspicious constants before writing "malformed" or "overflow". A weird-looking number is a mechanism clue, not evidence of corruption: check whether it is semantically meaningful —
2^32 − size, a type maximum, a page/alignment boundary, an epoch. Do the arithmetic; never pattern-match "big hex-ish number ⇒ overflow bug". (Case 12:4294901760 + 65536 = 2^32— an allocator boundary, handing you the whole mechanism.) - The claimed mechanism must predict the observed frequency. A deterministic-bug theory cannot explain n=1 across many runs exercising the same path — a low rate demands a stochastic or boundary mechanism, and the expected probability should be stated in the bug. A mechanism/frequency mismatch means the theory is wrong; resolve it before filing, and mark any remaining mechanism prose as an explicit unverified hypothesis ("unverified: possibly …"), never as "suggesting X bug in component Y". (Case 12: the filed theory predicted every-pod failure; the real mechanism predicted ~1/65535 per pod — matching n=1.)
- Differential recovery check — when the same error hits many objects but only some stay broken, explain the asymmetry before finalizing. Survivors show the failure was not uniformly permanent; before concluding the trigger was transient, rule out the alternatives (survivors had different exposure, a retry path, or different state). Either way the lasting failure lives in whatever failed to recover, and that component owns the bug. Pattern: transient trigger vs. persistent damage — ask which candidate defect explains the permanence of the failure, and lead with that one.
- Read the owning repo's source before claiming a missing safeguard. A claim that "component X lacks ordering/gating/protection Y" is only valid after locating (or proving absent) that safeguard in the repo — the same rule as reading a test's skip/gates before describing its scope, applied to product code.
- The real owner: the component whose operator/pods are actually failing. Examples from past duty: "cluster creation failed" ⇒ monitoring operator degraded ⇒ Monitoring bug; "bootstrap failed" ⇒
etcdserver: request timed outon Azure ⇒ etcd bug; nodes degraded during install ⇒ MCO bug; quota/DNS/cloud-API errors ⇒ ci-infra, not a product bug at all. - Stability-window failures — "operator X not available" is a symptom, not a root cause. For cluster-readiness/stability wrappers that fail on a ClusterVersion or ClusterOperator condition, two extra steps are mandatory before assigning an owner:
-
Recover the underlying condition message when the junit output is vague. Outputs like
clusterversion not available: False(empty reason) carry no cause. Run this literal check against the e2e step'sbuild-log.txt(and, if present, the monitor-intervals JSON under the step'sartifacts/junit/):curl -s <gcs-url-of-e2e-step>/build-log.txt | grep -oE '(Failing|Available|Degraded)=(True|False)[:,][^"\\]{0,160}' | sort | uniq -c | sort -rn | headThe condition messages recovered this way (e.g., a controller error string) name the real culprit. This step is complete only when the report quotes the recovered message for every vague-output run. Never attribute such a run from its co-failing tests: co-failures on techpreview jobs are usually unrelated background noise, and correlation with them has produced wrong owners in past duty runs.
-
Ask why the operator went unavailable, not just which operator. If everything is healthy at gather time, the wrapper caught a transient flap: read the operator's own pod log around the transition timestamps and find the trigger. Distinguish (a) the operator genuinely failing (⇒ product bug for that operator) from (b) a routine reconciliation/rollout triggered by a cluster mutation (config/secret change, node roll) — a rollout of a single-replica deployment flaps Available by design. A rollout-flap disposition must name the mutator, not stop at "the operator detected a configuration change": quote the operator-log line identifying which object changed (grep the operator pod log for
object changed/ secret and config names and read the surrounding lines), then identify who wrote it — audit logs (verbupdate/patchon that object: user + userAgent) or the job's test-harness step logs (search them for the object name or the value being set). If a test step caused the mutation mid-run, the bucket istestowned by that suite, not a product bug against the flapping operator; filing "spurious rollouts" against the operator without naming the mutator is an incomplete deep-dive. (Case 2 in case-notes.md: an image-registry flap drafted as a product bug until the operator log showed the harness had replaced the pull secret mid-run.)
-
-
Test-interference buckets — read the offending test's source before describing its scope. When the root cause is another test's or tool's behavior (a test creates pods/namespaces that break an invariant check, leaks resources, reboots nodes, etc.), do not infer why the failure is confined to certain variants from the regression's variant labels — correlation with a variant (techpreview, platform, upgrade mode) is not evidence of how the offending code is gated. Instead:
- Locate the offending code (
gh search codeinopenshift/originor the relevant repo for the test name, namespace prefix, or error string) and read its skip/gate conditions (skipIf...,e2eskipper.Skipf, feature-gate checks, platform checks, suite membership). - State the actual gating in the report (e.g., "gated on baremetal platform via
skipIfNotBaremetal"), and if that gating is broader than the regressed variants, say so — the regression's variant slice then reflects job-scheduling or sample-size effects, not the blast radius, and other variants are also at risk. - Check the file's merge history (
gh api repos/<org>/<repo>/commits?path=...) — a merge date matching the regression onset both confirms the attribution and identifies the owning team. - Never write "X-only" (techpreview-only, platform-only, arch-only) about a test or tool in the report or a bug unless the source code gating has been read and confirms it.
- Locate the offending code (
-
Onset and suspect PRs (when the bucket has a crisp start date): follow the "Determine Regression Start Date" and "Identify Suspect PRs in Payload" procedures from
/ci:analyze-regression(first failing run → payload tag viafetch-prowjob-json→fetch-new-prs-in-payload→ up to 5 candidate PRs vetted withgh). A LIKELY PR both strengthens the bucket and tells you the owning component/repo. -
Cross-check globally:
fetch-test-reportskill (with--no-collapse) for the bucket's main test — confirms whether the issue is variant-specific or global, and surfacesopen_bugsthat may already cover the bucket. -
Check Slack context (optional — only when Slack access is available): The TRT/release-oversight team discusses ongoing payload and CI issues in #forum-ocp-release-oversight (https://redhat.enterprise.slack.com/archives/C01CQA76KMX). Search/read the last 14 days of messages there for the bucket's signature (test name, error message, operator, platform, payload tag) — known payload-wide events, infra outages, and in-flight fixes are usually discussed there before triages/bugs exist, and a thread often names the owning team or an existing OCPBUGS ticket. If the agent has no Slack access (no Slack tooling/credentials), omit this step entirely — do not block or ask for access.
Mechanism self-review (mandatory before finalizing any bucket). Every rule above is an instance of one behavior: challenge your own conclusion before filing it. After drafting a bucket's root cause, run this adversarial pass on the mechanism claim itself:
- Enumerate the load-bearing assumptions in the claim — every "because", "not yet", "race", "missing", "computes wrong", "X-only". For each, cite a specific, checkable evidence locator: an artifact path plus the quoted line, a source-code location, a job-run ID, a structured API/JSON field, a condition's
LastTransitionTime, or the query that returned the result. A line number is one acceptable form, not a requirement — much of the evidence here (run metadata, API fields, object IDs, timestamps) has no stable line to cite, and inventing one is worse than quoting the field. What matters is that the next reader can go to the same place and see the same thing. An assumption with no such locator is downgraded in the filed text to an explicit unverified hypothesis ("unverified: possibly …") — never stated as fact. - Check the theory's predictions against everything observed, not just the failing line: does it predict the observed frequency (deterministic theory vs. n=1), the measured window duration, the recovery asymmetry (who healed vs. who stayed broken), and the variant/platform spread? Any mismatch means the theory is wrong or incomplete — resolve it before filing.
- Argue the strongest alternative. Spend one honest paragraph on the best competing mechanism (warmup instead of ordering, upstream producer instead of the error-emitter, boundary condition instead of corruption, unrecovered state instead of the transient trigger) and state the artifact evidence that discriminates between them. If nothing discriminates, the confidence is not HIGH.
A bucket whose mechanism claim fails any of these checks is not finalizable — keep digging or downgrade honestly.
After deep-dive, finalize buckets. Depth is mandatory, not optional: no bucket may be finalized at LOW or MEDIUM confidence, and the duty run must not end with open "action items" like "needs artifact deep-dive" or "spot-check installer logs first". When one regression's failed runs split into multiple distinct failure signatures, every signature must be root-caused independently — each may get its own triage record (a regression can carry several), and no run may be left labeled "ambiguous"/"unclear" in a bucket claimed at HIGH confidence: an unexplained run either gets dug into until it joins a signature, or the bucket's confidence is honestly downgraded and the digging continues. Never bundle an unexplained sub-pattern into another pattern's triage "pragmatically" or "since the regression is open anyway" — triage covers only the runs whose root cause it actually explains; attaching unexplained runs to it hides them from the next duty shift and mis-scopes the bug. If a sub-pattern remains unexplained, the whole bucket is not finalizable — keep digging. If confidence is not HIGH after the steps above, keep digging until it is — escalate through the evidence ladder yourself: raw failure outputs → job-run summaries → GCS artifacts (prow-job-artifact-search, install/test-failure analysis skills) → audit logs (grep for the failing object/namespace to identify the creating user and userAgent — this reliably resolves "who created this pod/namespace" questions) → junit timing correlation (what else ran in the same window) → suspect-PR vetting. Triage taking longer is acceptable; leaving an unexplained bucket is not. The only permitted low-confidence outcome is when the evidence is genuinely exhausted (artifacts expired, logs missing), and then the report must say exactly what was checked and what was missing.
Additional finalization rules (each has caused a wrong disposition in a real duty run):
- "Resolved"/"stopped"/"no recurrence" claims require the full run list, not a sample. Before classifying a signature as resolved or transient, enumerate all of the regression's
job_runsby date and confirm the newest runs' signature. A signature absent from a 3–4 run sample of a 20-run regression is not evidence it stopped. And when a sub-test starts "passing" recently, verify recent runs actually reach that stage: a bootstrap failure cannot "self-resolve" while newer runs of the same job die earlier at infrastructure provisioning — the earlier failure masks the later stage, it does not fix it. - "Leave untriaged" is not a permitted disposition for a bucket with an identified root cause and owner. If the deep-dive named the mechanism and the responsible component/suite, the bucket gets a triage (to an existing or new issue; under
--existing-only, a new-issue bucket instead gets a "Proposed new bugs (not filed)" entry) — "collateral of noisy runs" is only a valid leftover justification when the failure has no independent mechanism (pure co-occurrence). A failure that is deterministically produced by another test's behavior has an independent mechanism and must be triaged astest. - Extend, don't duplicate. When an existing triage record already covers the bucket's bug, extend that triage with the new regression IDs (
--triage-id); do not create a second triage record pointing at the same JIRA. - Subagent outputs must be verified, not trusted. If bucket deep-dives are delegated (subagents, parallel tasks), the orchestrator must check each returned bucket against this section's requirements before accepting it — in particular that the mandatory quotes are present (recovered condition messages for vague wrappers, mutator identification for rollout flaps, transition counts for stability failures,
*_previous.logreads for crash-looping containers). A missing mandatory quote means the sub-analysis is incomplete and must be redone, regardless of how confident its prose sounds. Batch size is never a reason to skip mandatory steps. - Never end your turn to "wait" for dispatched subagents — the session ends the moment you stop. There is no background execution across turns: an assistant message that ends with a status narration ("waiting for X analysis to complete") and no tool call terminates the run, and in CI the harness will tear the process down at that point. Collect every subagent's result within the turn that needs it, and treat the duty report as a hard checkpoint: if the run were killed right after your current message, the report file must already exist on disk — write intermediate versions early and update them, rather than deferring all writing to a final step that may never come. (Case 5 in case-notes.md: $12 of analysis, no report.)
Each bucket must have:
- Member regression IDs (re-check the untriaged list — new siblings may have opened during analysis)
- Root cause summary (one paragraph) and failure classification (permafail / flaky / resolved / recent)
- Owning component (may differ from the Sippy component — state both)
- Triage type:
product/test/ci-infra/product-infra - Disposition: existing triage to extend / existing JIRA to create a triage for / new JIRA needed / no action (resolved or pure infra noise — say so explicitly and leave untriaged only with justification)
Phase 4: Search for existing bugs, then triage each bucket
For each bucket, before filing anything new:
-
Check bugs linked to the bucket's fired labels, then
triaged_matchesfromfetch-related-triages(confidence ≥5 with an open JIRA is the default target) — but never act on a match, even conf=10, without the Phase 3 per-bucket CI-sample verification confirming the root cause actually matches. -
Check
open_bugsfrom the test report. -
Search Jira for the root-cause signature (error message, operator name,
component-regressionlabel) in OCPBUGS against the owning component — the right bug may exist under Monitoring/etcd/MCO even though the regression sits under Installer. -
Component-scoped JIRA listing (mandatory — keyword search is not enough). Owning teams describe defects in developer vocabulary that shares no keywords with the CI-side symptom. After determining the owning component, list its recent bugs regardless of keywords and read the summaries:
project = OCPBUGS AND component = "<owning component>" AND created >= -21d ORDER BY created DESCA bug whose creation date falls inside the bucket's failure window, on the owning component, is a duplicate candidate even with zero keyword overlap — open it and compare mechanisms.
-
Owning-repo merge-history check (mandatory when onset or cessation is dated). Query the owning repo for PRs merged around the bucket's onset and cessation dates:
gh pr list --repo <org>/<repo> --state merged --search "merged:<window>" --json number,title,mergedAtA merge at the cessation boundary is likely the fix — its
OCPBUGS-*title prefix names the existing bug: triage to that bug instead of filing a new one. A merge at the onset boundary is a suspect trigger. The phrase "resolved by unidentified payload change" is banned from reports and bugs unless this check was run and came back empty. For currently-live breakage, also scan the owning repo's newest merges for revert PRs: a freshRevert "..."title citing a TRT/OCPBUGS key hands you the trigger PR, the tracking ticket, and the expected recovery time in one query.
Then act (this is where --auto-triage applies; without it, confirm each bucket with the user):
- Extend existing triage:
triage-regressionskill with--triage-id(additive merge is automatic; pass only the new IDs). - New triage to existing bug:
triage-regressionskill with--url,--type, and a one-sentence--description(<120 chars). - New bug (under
--existing-only: do not file or triage; add a Phase 5 proposal instead): file with/jira:create bug(thecreateskill from the jira plugin) against the owning component, labelcomponent-regression, description per the bug-filing template in/ci:analyze-regression("Prepare Bug Filing Recommendations" section: full test names in{code}blocks, test IDs, regression IDs, variants, error signature, Sippy test-details UI links for every member regression, suspect PRs). Every JIRA issue or comment created by this workflow must end with an AI-attribution footer as a separate, visually marked block — not a sentence buried in the text: place it after a divider, as its own paragraph or note panel, e.g. arulefollowed by apanel(typenote) in ADF containing "AI-generated content: This bug was filed by AI as part of Component Readiness triage duty. Please verify before acting on it." Release blocker is conditional on the triage type and impact, not automatic: mark the bug a release blocker (set-release-blockerskill) only forproductbugs whose failures block or materially degrade blocking/informing payload jobs;testbugs (races, invariant-scan interference) andci-infraissues (cloud capacity, registry outages) are not release blockers — state the blocker decision and its one-line justification in the report. Then create the triage record. - Suggested fixes carry a higher evidentiary bar than attribution — and triage duty never opens fix PRs. A "suggested fix" in a filed bug is optional; when included it requires (a) the mechanism timeline-proven per Phase 3 (timestamps, window duration, alternatives refuted), and (b) a target that matches what the measured window actually proves. A seconds-long trigger window does not by itself justify changing steady-state behaviour for everyone — but it very often does justify a permanent resilience change (retry, re-discovery, gating, idempotent recovery) in whatever failed to survive that window. Aim the fix at the thing that stayed broken, not at the thing that blinked. Never propose weakening a safety mechanism (failurePolicy, validation, admission gating, health checks) as the primary fix; if failing open seems attractive, flag it as a question for the owning team instead. If the analysis surfaces a secondary resilience defect (e.g., a controller that cannot recover from one failed patch), do not demote it to a "defense-in-depth" footnote — re-run the differential-recovery check: it is often the actual primary bug. Opening code-change PRs is out of scope for this workflow entirely.
- Always finish a triage by running the
add-jira-triage-linkskill to put the triage URL into the JIRA description. The skill appends to the description — if the description ends with the AI-attribution footer, the appended link would land after it. After running the skill, verify the footer is still the final block; if not, move the footer back to the end of the description (or insert the triage link before the footer in the same update). - Comments on existing bugs for added regressions must use only jira-comment.md, never freeform: at most one comment per bug per run, with a linked triage, linked test-details UI page for each regression, and the AI-attribution footer last.
- If a label that fired on the bucket's runs does not list the triaged bug, propose
manage-labels update --bugs <existing>,<new>(replaces the list; needs user confirmation, not covered by--auto-triage). - While triaging each bucket, note whether its signature is symptom-worthy (crisp grep-able line in a durable artifact) — the Phase 5 report must carry a symptom proposal for every bucket where it is (see "Label the bucket's signature in Sippy").
With --auto-triage, only act autonomously when confidence is high: consistent error signature across the bucket, and either a confidence ≥5 triaged match or an unambiguous existing open bug. Buckets requiring a new bug, or with mixed signals, are always presented for confirmation.
Existing-only mode (--existing-only)
This mode records matches to existing bugs but never files new JIRA issues. Analysis is unchanged. --policy post-ga implies --existing-only. These rules apply to every write in any phase, including the Phase 1 stale-triage sweep:
-
Allowed writes: only for a verified match to an existing issue:
- extend a triage, or create one for the existing issue;
add-jira-triage-link;- comment on that issue, using the jira-comment.md template.
Everything else, including creating, reopening, or transitioning issues, goes in the report as a recommendation.
-
Otherwise, propose. New-bug buckets, failed fixes, and uncertain matches get a draft under "Proposed new bugs (not filed)" in the Phase 5 report. Their regressions stay untriaged. Never lower the match bar to avoid a proposal: a wrong triage hides the real defect.
-
Subagents do analysis only; the orchestrator does every write.
Phase 5: Duty report
Present a final report:
- Inventory: N untriaged regressions found → M buckets.
- Per bucket: member regression IDs, root cause, owning component (vs. Sippy component), classification, evidence highlights (error signature, stage, representative run links), action taken (triage ID + JIRA link) or recommendation awaiting confirmation.
- Leftovers: regressions deliberately left untriaged (resolved / one-off flake / inconclusive) with justification and what evidence would change the call.
- Proposed Sippy symptoms (mandatory section, even if empty): for every bucket whose root cause has a crisp, grep-able single-line signature in a durable artifact, include a concrete symptom proposal per the "Label the bucket's signature in Sippy" section below — label name,
bugs, matcher, file pattern, validation pair, retro-apply run set. Proposing costs nothing (creation still requires explicit user confirmation); a duty shift that root-caused a bucket and did not propose a symptom for an obviously grep-able signature has left cheap future-triage value on the table. If no bucket qualifies, say so and why (e.g., signature only visible via timing correlation, artifact expires, no unique string). - Cross-cutting observations: payload-wide events, infra instability windows, techpreview-only patterns — useful context for the next duty shift.
- Session usage (when available): if the run is orchestrated by a harness that captures usage telemetry (e.g. the CI job appends a "Session usage" section with model, turns, token counts, and cost after the session ends), do not fabricate these numbers yourself — the model cannot observe its own final token totals mid-session. In interactive runs simply omit the section.
- Proposed new bugs (not filed) (
--existing-onlyonly, mandatory even if empty): one draft per proposal, following the Phase 4 new-bug template, with member regression IDs and the release-blocker recommendation. A human must be able to file it as written.
Label the bucket's signature in Sippy (propose always, create on confirmation)
When a bucket has a crisp, grep-able artifact signature (an error string or log line unique to the root cause), define a Sippy symptom so future runs are labeled automatically and the signature can be applied retroactively. Symptoms have repeatedly paid for themselves in real duty runs — a validated matcher has adjudicated a disputed root-cause attribution and machine-verified a 19-vs-1 bucket split (Case 6 in case-notes.md). Good candidates: recurring infra events that spawn regressions across many job families, single-occurrence product signatures worth a tripwire (n=1 bugs), and any signature that a "known recurring family" keeps regenerating. These are writes: always present the proposed label/symptom definitions and the target run list to the user and get explicit confirmation first — --auto-triage does not authorize them.
-
Check for existing coverage before creating anything. List the current catalog — this read is public and needs no token (
GET https://sippy.dptools.openshift.org/api/jobs/labels,GET .../api/jobs/symptoms, or thelist-symptomsskill) — and search it for the bucket's signature: the label/symptom may already exist from the shift that first triaged the bug (Case 6: a proposed console-crash symptom fully duplicated an existing one; the duplicate-key 400 from the label POST was what revealed it). Also search by the bug key (list-symptoms --labels --search OCPBUGS-NNN); a label linked to the bug counts as coverage only if its matcher targets this bucket's signature. A dry-runreevaluateof one bucket run is an equivalent cheap probe when a DPCR token is available: if an existing symptom targeting this signature fires on the run, the bucket is already covered and nothing should be created. -
Create a label and symptom via the authenticated API (DPCR Bearer token, same as triage writes):
POST https://sippy-auth.dptools.openshift.org/api/jobs/labels({"label_title": ..., "explanation": ..., "bugs": ["<JIRA key>"]}— the title becomes the ID), thenPOST .../api/jobs/symptoms({"summary": "<descriptive name>", "matcher_type": "string|regex|file", "file_pattern": "<glob under the job-run GCS root>", "match_string": ..., "label_ids": [...]}). The symptom ID is derived from the summary; gzipped artifacts are decompressed transparently. Keep Jira keys in the label'sbugsarray rather than embedding them in label explanations or symptom summaries. To change a symptom, use its full-replacement PUT endpoint. -
Beware benign-transient false positives — presence is not pathology. Before finalizing a matcher, grep the target file in a healthy run: many error lines appear benignly during normal startup — sometimes once, sometimes in bulk (Case 6: a proposed match string appeared 120× in a healthy install's etcd-operator log as a normal warm-up transient; the failed run differed only in that the line persisted to log-end). String/regex matchers cannot count occurrences or check persistence, so if the line can appear benignly at all, re-anchor to an artifact that only exists — or only contains the string — in the failure mode: a container's
*_previous.log(only present after a crash/restart) instead of its current log, an events.json string that healthy runs never emit, or best of all the gather-time end state inclusteroperators.json(e.g.,StaticPodsAvailable: 0 nodes are activeappears there only when etcd truly never deployed — zero hits in healthy runs by construction, because gather runs after the failure window). -
Validate with a dry run against one known-positive and one known-negative run — the best negative is a near miss: a failed run of the same job family with a different root cause (it catches over-broad matchers that a green run would not).
POST .../api/jobs/runs/reevaluatewith{"prow_job_build_ids": [...], "dry_run": true}. Proceed only if both outcomes are correct; otherwise refine the matcher and re-validate. Note: reevaluation runs all symptoms, so unrelated pre-existing labels may legitimately appear on your negative — check only that your symptom's matching is correct. -
Then apply with
"dry_run": falseto the bucket'sprowjob_run_ids. The API caps requests at 50 IDs, but reevaluation scans GCS artifacts server-side and large batches time out at the gateway (504) — use batches of 5–10 with a retry, check each result'sstatus, and stop and report on the first failed batch or any non-successresult rather than continuing. -
Match the underlying error, not a transient side effect: a matcher keyed on a crash/panic goes silently false-negative the moment a partial fix removes the crash while the defect persists. If the primary evidence lives in pod logs that sometimes fail to gather (dying clusters), add a fallback symptom against an artifact that survives (e.g.,
pods.jsonstate/reason strings) mapped to the same label. When the same signature can surface under different step names (e.g., devscripts-driven vs plain IPI installs), create one symptom per file pattern, all mapped to the same label.
Closed-set audit mode (--audit-closed) — justify why every closed regression closed
bulk-triage-regressions <view> --components ... --audit-closed
An explicitly-requested, read-only mode that answers a different question than triage duty: not "who owns this failure?" but "why did this regression close, and can we prove it?" Every closed untriaged regression must end the audit in exactly one of these justification classes:
| Class | Meaning | Proof required |
|---|---|---|
fixed | A fix merged; closure follows it | Fix PR/bug with merge/resolution date at or before the close boundary |
event-ended | An infra/payload event ended | Event window bracketing the failures (bad payload, repo outage, cloud capacity, credentials) |
intermittent-cause-open | Pass rate recovered but the defect is still open | Signature matched to an open bug; state explicitly that closure ≠ resolution |
collateral | Closed with the window of a tracked sibling event | Signature or run overlap with the tracked event |
evidence-expired | Run history and artifacts are gone | List exactly what was checked (regression details, GCS) and the statistical argument (age, sibling consistency) — flagged, never silently absorbed |
No regression may be left as "unknown". evidence-expired is the only permitted terminal state without a mechanism, and it must be flagged in its own report section.
Scale technique (a closed set is 5–30x a duty batch — per-regression deep-dives do not scale)
- Inventory and cluster first: fetch details for all members (parallel, ~8 workers is safe for the details endpoint), then cluster by (test-family, platform, featureset, close-window). Expect heavy super-clusters — always look for same-day open/close waves across platforms before analyzing anything individually (Case 7 in case-notes.md: one bad nightly payload explained 21% of an entire closed set).
- Bypass the Sippy runs API for bulk signature extraction — it rate-limits (HTTP 429) far below audit volume, and backoff does not help at this scale. Go to GCS directly (unthrottled, parallel-safe):
- Install-family tests: classify from the installer log (
artifacts/*/ipi-install-install*/build-log.txt, devscripts equivalent on metal) with an ordered signature catalog (most-specific first). Seed the catalog from the view's known events and extend it with whatever the duty history has verified (ign-push 403s, cipherSuites/rev-0, toolchain download, quay pulls, registry.ci 5xx, quota, capacity, provisioning, per-operator stabilization). Guard against substring traps (Case 7:leasematching insidereleasemis-binned 31 regressions). - Everything else: parse the junit XML under the e2e step (
artifacts/*/<e2e-step>/artifacts/junit/junit_e2e*.xml) with a real XML parser (regex over XML mis-matched ~95% of testcases in practice) and take the<failure>of the exact testcase. - Sample the 2 newest failed runs per regression; extend only where signatures disagree.
- Install-family tests: classify from the installer log (
- Match against the full triage catalog: fetch all existing triage records once (
/api/component_readiness/triages) and index their regressions by test name — a closed untriaged record is very often the untriaged sibling of a triaged one, which supplies the bug and the closure mechanism for free. - Date-anchor every closure claim:
fixedrequires the fix date ≤ close boundary (JIRAresolutiondate,gh pr list --search "merged:<window>"on the owning repo);event-endedrequires the last failing run to fall inside the event window. The Phase 4 component-scoped JIRA listing and merge-history checks apply here unchanged. - Checkpoint long-running collection to disk and run it detached (nohup + periodic JSON dumps): bulk GCS extraction takes tens of minutes, harness timeouts will kill foreground loops, and two concurrent writers on one results file destroy each other — one writer, atomic-ish checkpoints, verify counts after every stage.
Audit report structure
- Executive summary: category table (name, member count, one-line verdict).
- Method: exactly which artifact paths / APIs were used and why.
- Per-category narrative: mechanism, proof, closure explanation, and for
intermittent-cause-openthe explicit warning that siblings will reopen. - Per-regression appendix — one row per member, no exceptions (ID, test, platform/featureset, opened→closed, category, evidence sample).
- Honest-limitations section listing every
evidence-expiredmember.
Audit-specific pitfalls
- Sippy retention creates a hard evidence horizon: regressions from the view's first tracking week may have no run history at all, and GCS artifacts expire (~3 months). Do not let the horizon silently shrink the audit — count and flag such members.
- The view-baseline start date masquerades as a mass event: many unrelated regressions "open" on the view's first tracking day. Check whether a suspicious same-day open wave is simply the earliest date in the dataset before hunting for a common cause.
- "Closed" is not a verdict: a closed regression whose signature maps to a still-open bug (
intermittent-cause-open) is a prediction of future regressions — surface these in the summary so the next duty shift expects them.
Pitfalls (learned from real duty runs)
Principles distilled from real mis-dispositions; full narratives in case-notes.md.
Detect Permafail
When to Use This Skill
Use this skill when you have 2-10 consecutive failures of the same job and need to determine if the failures represent a systematic/permanent failure (permafail) versus a flaky failure. This is critical for CI/CD pipeline analysis to distinguish between:
- Permafail: A systematic failure affecting the same test(s) or infrastructure issue, detected by analyzing comparable runs (same failure type):
- 2-3 comparable runs: All must have the same failure (100% match)
- 4-5 comparable runs: At least 4 must have the same failure (80% match)
- 6-10 comparable runs: At least ceil(count × 0.7) must have the same failure (70% match)
- Flaky: Non-deterministic failures with varying root causes, or failures that don't meet the permafail thresholds
Prerequisites
- Access to OpenShift CI Prow job artifacts via gcsweb URLs
- Access to the
Bashtool for runningplugins/ci/scripts/classify-job-failures.py - Python with the
requestspackage available for artifact fetching - Knowledge of Prow artifact structure (from
fetch-prowjob-jsonandprow-job-artifact-searchskills) - 2-10 URLs pointing to consecutive job failures (from Prow/OpenShift CI, ordered newest to oldest)
- Job name context to verify consistency across all failures
- PR information to provide context for analysis
Implementation Steps
Step 1: Validate Inputs
Standard Mode (URL-based):
Verify that all required inputs are present with expected types and constraints:
failure_urls: Array of 2-10 strings matching Prow job URL patternhttps://prow.ci.openshift.org/view/gs/<bucket>/<path>/<job-name>/<build-id>where path may belogs/,pr-logs/pull/, or other GCS paths (must be consecutive runs, ordered newest to oldest)job_name: Non-empty string identifier of the job being analyzedpr_info: Object containing PR number (integer) and repository context (string)- Each URL must match the Prow job URL pattern above
Reject requests if:
- URLs count is less than 2 or more than 10
- Job names don't match across all URLs (validate via prowjob.json metadata, not path position)
- PR context is missing
Notes:
- URL ordering (newest first) is assumed but not validated - the frontend provides them in this order
- Job name validation should check prowjob.json metadata or verify the name appears in the extracted GCS path, not assume a fixed path position (presubmit URLs include org/repo/pr segments)
Offline/Eval Mode (Pre-normalized Signatures):
For testing or offline analysis, the skill can accept pre-normalized failure signatures directly instead of fetching from URLs. This mode skips Steps 2-4 (artifact fetching and classification) and proceeds directly to Step 5 (threshold analysis).
Input format:
signatures: Array of 2-10 pre-normalized signature objects (see Step 4 output format)- Each signature must be either test_failure format
{type: "test_failure", url: "...", tests: [...], test_count: N}or infra_failure format{type: "infra_failure", url: "...", error: "...", error_hash: "..."}
When signatures are provided, skip to Step 5 and apply threshold logic directly. Use this mode only for evals or when signatures have been extracted by another tool.
Step 2: Classify Job Failures with Script
For standard URL-based analysis, call the deterministic classifier script to fetch artifacts and produce normalized signatures:
"$SKILL_DIR/../../scripts/classify-job-failures.py" --json-input '{
"failure_urls": ["https://prow.ci.openshift.org/view/gs/..."],
"job_name": "pull-ci-openshift-origin-master-e2e-aws",
"pr_info": {"pr_number": 12345, "repository": "openshift/origin"}
}'
The script performs the artifact-based work that does not require AI judgment:
- Parses the full GCS path from each Prow URL.
- Fetches
prowjob.jsonfrom gcsweb. - Browses the artifacts directory.
- Classifies each run as
test_failureorinfra_failurebased on artifacts, not error text alone. - Extracts failing test names from junit/build logs or normalized infrastructure errors.
- Returns the Step 4 normalized signature array.
If the script exits non-zero, report its JSON error and do not continue to threshold analysis unless at least two valid signatures are available from another trusted source.
Step 3: Artifact Classification Rules
The classifier follows these rules:
- TEST_FAILURE when artifacts contain
openshift-e2e-test/,e2e-*,junit/, junit XML files,openshift-tests-*, ormonitor-test-*. - INFRA_FAILURE only when artifacts show setup/build/gather output without test artifacts, or artifacts are empty/unavailable after a valid job fetch.
- MonitorTest failures count as test failures because MonitorTests run during e2e execution.
- Classification must be artifact-based. Do not classify from Prow descriptions or error messages alone.
Step 4: Use Normalized Failure Signatures
Use the script output directly as the normalized signature array.
For test failures:
{
"type": "test_failure",
"url": "job_url",
"tests": ["failing_test_name1", "failing_test_name2"],
"test_count": 2
}
For infrastructure failures:
{
"type": "infra_failure",
"url": "job_url",
"error": "normalized_error_message",
"error_hash": "md5_of_error_message"
}
Step 5: Compare Signatures for Permafail Pattern
Apply permafail detection logic using failure-type-based thresholds - the denominator is the count of comparable runs (same failure type), not the matching group size:
Detection Thresholds (based on comparable run count):
- 2-3 comparable runs: All must match (100% match required)
- 4-5 comparable runs: At least 4 must match (80% match required)
- 6-10 comparable runs: At least ceil(count × 0.7) must match (70% threshold)
For Test Failures:
- Count total test_failure signatures (this is the denominator)
- Extract all unique test names from test_failure signatures
- For each unique test name, count how many test_failure signatures contain it (this is the numerator)
- Apply threshold based on the denominator (total test_failure count):
- If denominator is 2-3: numerator must equal denominator (100%)
- If denominator is 4-5: numerator must be ≥4 (80%)
- If denominator is 6-10: numerator must be ≥ceil(denominator × 0.7) (70%)
- If ANY test name meets its threshold: PERMAFAIL = TRUE
- Report which test(s) met the threshold and their occurrence count
- If no test meets the threshold: PERMAFAIL = FALSE
Example: 8 test_failure signatures, "TestNetworkPolicy" appears in 6 → 6/8 = 75% → needs ceil(8×0.7)=6 → 6≥6 ✓ PERMAFAIL
For Infrastructure Failures:
- Count total infra_failure signatures (this is the denominator)
- Extract error messages and group similar errors (exact match or >70% character similarity using normalized Levenshtein:
similarity = 1 - (levenshtein_distance(a, b) / max(len(a), len(b))). Treat two normalized errors as similar whensimilarity > 0.70. If both strings are empty, do not group them as similar; require a non-empty normalized error.) - For each error group, count how many infra_failure signatures contain it (this is the numerator)
- Apply threshold based on the denominator (total infra_failure count):
- If denominator is 2-3: numerator must equal denominator (100%)
- If denominator is 4-5: numerator must be ≥4 (80%)
- If denominator is 6-10: numerator must be ≥ceil(denominator × 0.7) (70%)
- If ANY error group meets its threshold: PERMAFAIL = TRUE
- Report the error message and occurrence count
- If no error group meets the threshold: PERMAFAIL = FALSE
Example: 5 infra_failure signatures - 3 have "operator X timeout", 2 have random errors → 3/5 = 60% → needs 4/5 (80%) → 3<4 → NOT PERMAFAIL
For Mixed Failure Types:
- Separate signatures by type (test_failure vs infra_failure)
- For each type, count total signatures (the denominator for that type)
- Apply threshold logic to each type independently:
- Test failures: If ≥2 test_failure signatures exist, check if any test appears frequently enough to meet the threshold based on test_failure count
- Infra failures: If ≥2 infra_failure signatures exist, check if any error appears frequently enough to meet the threshold based on infra_failure count
- Minimum runs requirement: Need at least 2 runs of the same type to establish a permafail pattern. A single failure of either type is insufficient.
- If either type meets the threshold AND has ≥2 runs: PERMAFAIL = TRUE
- Report the dominant pattern (the one that triggered permafail)
- Explain which runs contributed and which were ignored (e.g., "All 4 runs that reached e2e tests failed on TestNetworkPolicy. 3 other runs failed during cluster setup and are not relevant to this test failure pattern.")
- If neither type meets criteria: PERMAFAIL = FALSE
Example Scenario 1: PERMAFAIL (4 test, 3 infra):
- 7 total runs provided
- 3 runs: infra_failure (cluster creation failed)
- 4 runs: test_failure (all failing on TestNetworkPolicy)
Analysis:
- Test failures: 4 runs → use 4/5 threshold (80%) → need 4 matching → have 4/4 (100%) ✓ PERMAFAIL
- Infra failures: 3 runs → use 3/3 threshold (100%) → different errors → NOT permafail
- Verdict: PERMAFAIL = TRUE (test failure group met criteria)
- Reason: "All 4 runs that reached e2e tests failed on TestNetworkPolicy (100% match). 3 additional runs failed during infrastructure setup and are not relevant to this test failure pattern."
Example Scenario 2: NOT PERMAFAIL (1 test, 6 diverse infra):
- 7 total runs provided
- 6 runs: infra_failure (each with different errors: cluster creation timeout, AWS quota, network issue, pod eviction, storage failure, DNS timeout)
- 1 run: test_failure (failed on TestNetworkPolicy)
Analysis:
- Test failures: 1 run → INSUFFICIENT (need minimum 2 runs to establish pattern)
- Infra failures: 6 runs with all different errors → no single error appears ≥5 times → NOT permafail
- Verdict: PERMAFAIL = FALSE
- Reason: "Only 1 out of 7 runs reached e2e tests and failed on '[sig-arch] daemonset cni-sysctl-allowlist-ds maxUnavailable requirement'. The other 6 runs failed during infrastructure setup, each with different unrelated issues (no consistent infra pattern). A single test failure is insufficient to establish a permafail pattern - we need multiple runs with consistent failures to confirm systematic breakage."
Step 6: Generate Verdict and Return JSON
Construct the final response object with:
- Boolean verdict:
permafail(true/false) - Reason string explaining the determination. Always include explicit slash-format ratios for the dominant pattern and strongest non-matching pattern, such as
7/10,4/4,5/6, or2/10. Do not write only7 of 10; include7/10. - Complete failure signatures array
- Common tests array (if applicable and permafail=true)
- Confidence score as a JSON number from 0.0 to 1.0, representing confidence in the final verdict, not the raw match ratio. Never use strings such as
"high","medium", or"0.95". For example, a clearly non-permafail2/10pattern can still have confidence around 0.70 because the verdict is well supported. - Stable
failure_typevalue:test_failurewhen the final verdict is driven by test-failure analysis, or only test failures are presentinfra_failurewhen the final verdict is driven by infrastructure-failure analysis, or only infrastructure failures are presentmixedwhen both test and infrastructure failures are present and no single type meets the permafail threshold
match_ratio: Slash-format string for the dominant pattern, such as"7/10","4/4","5/6", or"2/10".threshold_required: Integer count required for the comparable-run group to qualify as permafail.matching_runs: Integer numerator for the dominant or strongest pattern.comparable_runs: Integer denominator for the comparable-run group.
The final JSON must be machine-checkable. The ratio-bearing fields must be top-level fields, not nested under an analysis object. Use this exact style:
{
"permafail": true,
"confidence": 0.95,
"reason": "7/10 test_failure runs failed TestNetworkPolicy, meeting the required 7/10 threshold.",
"failure_type": "test_failure",
"match_ratio": "7/10",
"matching_runs": 7,
"comparable_runs": 10,
"threshold_required": 7,
"signatures": []
}
For a mixed non-permafail, use failure_type: "mixed" and include the word insufficient in the reason when one failure type has fewer than 2 comparable runs.
Do not set confidence equal to the match percentage. Confidence means confidence in the verdict:
- Clear non-permafail with complete data and a strongest pattern below threshold, such as
2/10, should useconfidence: 0.70or higher. - Exact threshold permafail, such as
7/10test failures or5/6infra failures, should useconfidence: 0.85or higher. - All comparable runs matching, such as
4/4, should useconfidence: 0.99. - Ambiguous or incomplete data should use lower confidence.
Output Format
The skill returns a JSON object with this schema:
{
"permafail": true,
"confidence": 0.95,
"reason": "3/3 test_failure runs show the same failing test 'test_node_scale' - consistent permanent failure",
"failure_type": "test_failure",
"match_ratio": "3/3",
"matching_runs": 3,
"comparable_runs": 3,
"threshold_required": 3,
"signatures": [
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"tests": ["test_node_scale"],
"test_count": 1
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"tests": ["test_node_scale"],
"test_count": 1
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"tests": ["test_node_scale"],
"test_count": 1
}
],
"common_tests": ["test_node_scale"]
}
For Infrastructure Failure (permafail=true)
{
"permafail": true,
"confidence": 0.92,
"reason": "All 3 runs fail at cluster creation with identical error: 'Insufficient quota for machine type n1-standard-4'",
"failure_type": "infra_failure",
"signatures": [
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"error": "Insufficient quota for machine type n1-standard-4",
"error_hash": "a1b2c3d4e5f6g7h8"
},
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"error": "Insufficient quota for machine type n1-standard-4",
"error_hash": "a1b2c3d4e5f6g7h8"
},
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
"error": "Insufficient quota for machine type n1-standard-4",
"error_hash": "a1b2c3d4e5f6g7h8"
}
]
}
For Non-Permafail (mixed or varying failures)
{
"permafail": false,
"confidence": 0.88,
"reason": "Mixed failure types detected: 2/3 runs are test_failure, 1/3 is infra_failure. Inconsistent pattern indicates flaky behavior, not a systematic permafail. Test failures: 2/3 (threshold: 2/3). Infra failures: 1/3 (threshold: 1/3).",
"failure_type": "mixed",
"match_ratio": "2/3",
"matching_runs": 2,
"comparable_runs": 3,
"threshold_required": 2,
"signatures": [
{
"type": "test_failure",
"url": "...",
"tests": ["test_networking"],
"test_count": 1
},
{
"type": "infra_failure",
"url": "...",
"error": "Pod evicted due to memory pressure",
"error_hash": "x1y2z3a4b5c6d7e8"
},
{
"type": "test_failure",
"url": "...",
"tests": ["test_storage", "test_deployment"],
"test_count": 2
}
]
}
Failure Signature Format
Test Failure Signature
{
"type": "test_failure",
"url": "string (job URL)",
"tests": ["array", "of", "failing_test_names"],
"test_count": "integer (length of tests array)"
}
Fields:
type: Always "test_failure"url: The Prow job URL for this runtests: Array of test names extracted from failure logs (deduplicated)test_count: Count of unique failing tests
Infrastructure Failure Signature
{
"type": "infra_failure",
"url": "string (job URL)",
"error": "string (normalized error message)",
"error_hash": "string (MD5 hash of normalized error)"
}
Fields:
type: Always "infra_failure"url: The Prow job URL for this runerror: Normalized error message with timestamps and build IDs removederror_hash: MD5 hash for fast similarity comparison
Permafail Detection Logic
Test Failure Logic
Apply the threshold rules from Step 5 based on the number of test_failure signatures (the comparable run count):
- Count total test_failure signatures (this is N, the denominator)
- Collect all unique test names from test_failure signatures
- For each unique test name, count how many test_failure signatures contain it (the numerator)
- Check if any test meets the threshold:
- N=2-3: Test must appear in ALL N runs (100% match required)
- N=4-5: Test must appear in ≥4 runs (80% match required)
- N=6-10: Test must appear in ≥ceil(N × 0.7) runs (70% match required)
- Permafail = TRUE if ANY test meets the threshold
- Set confidence based on match strength:
- 0.99 if all test_failure runs have identical test set
- 0.85 or higher if threshold is met exactly (e.g., 3/3, 4/5, 7/10)
- 0.92 if threshold is exceeded (e.g., 5/5, 8/10)
- 0.70 or higher if no test meets threshold but the non-permafail verdict is clear
- In the
reason, include the strongest test ratio in slash form, for example7/10 test_failure runs failed TestNetworkPolicyor2/10 test_failure runs failed TestNetworkPolicy. - Include top-level
match_ratio,matching_runs,comparable_runs, andthreshold_requiredfields for the strongest test pattern.
Infrastructure Failure Logic
Apply the threshold rules from Step 5 based on the number of infra_failure signatures (the comparable run count):
- Count total infra_failure signatures (this is N, the denominator)
- Extract error messages and group similar errors (exact hash match or >70% string similarity)
- For each error group, count how many infra_failure signatures contain it (the numerator)
- Apply threshold based on N (total infra_failure count, not the error group size):
- N=2-3: Error must appear in ALL N runs (100% required)
- N=4-5: Error must appear in ≥4 runs (80% required)
- N=6-10: Error must appear in ≥ceil(N × 0.7) runs (70% required)
- Permafail = TRUE if ANY error group meets its threshold
- Example: 5 total infra_failure signatures, 3 with "operator X timeout", 2 with random errors
- Denominator N=5 → needs 4/5 (80%) → 3/5 = 60% < 80% → NOT PERMAFAIL
- Set confidence based on match strength:
- 0.99 if all infra_failure runs have identical error hash
- 0.85 or higher if the threshold is met exactly
- 0.92 if threshold is met with >80% string similarity
- 0.88 if threshold is met with >70% string similarity
- 0.70 or higher if no error group meets threshold but the non-permafail verdict is clear
- In the
reason, include the strongest infra ratio in slash form, for example5/6 infra_failure runs share operator authentication timeoutor1/6 infra_failure runs share the strongest error. - Include top-level
match_ratio,matching_runs,comparable_runs, andthreshold_requiredfields for the strongest infra pattern.
Mixed Type Logic
When both test_failure and infra_failure types are present:
- Analyze each group independently using their respective thresholds
- Test failures: Check if test_failure signatures have common failing tests
- Infra failures: Check if infra_failure signatures have common errors
- If either group meets the permafail criteria: PERMAFAIL = TRUE
- Report the pattern that triggered permafail (tests or infra)
- Explain the breakdown (e.g., "4 of 4 test runs failed on the same test; 3 other runs failed during setup")
- Set
failure_typeto the type that triggered the permafail:test_failureorinfra_failure - Use the triggering type's top-level ratio fields. For example, if 4/4 test failures match and infra failures are noise, set
failure_type: "test_failure",match_ratio: "4/4",matching_runs: 4,comparable_runs: 4, and the test threshold.
- If neither group meets criteria: PERMAFAIL = FALSE
- Reason: "No consistent pattern found in test failures or infrastructure failures"
- If both test_failure and infra_failure signatures are present, set
failure_typetomixed, notflaky
- In mixed cases, include slash-format ratios for the evaluated groups in the
reason, such as1/1 test_failure runsand1/6 infra_failure runs.
Key Principle: Infrastructure failures (cluster setup, resource quota, network issues) are orthogonal to test failures. A PR can have a systematic test failure (permafail) even if some runs fail during infrastructure setup. Analyze each type separately and detect permafails in either category.
Error Handling
Scenario 1: Artifact Fetch Failure
If artifact fetching fails for a job URL:
- Return status: "error"
- Return error message with specific failure reason (network error, 404, timeout, etc.)
- Continue with remaining jobs when at least 2 jobs have not yet been attempted
- Do NOT return a permafail verdict when fewer than 2 jobs completed successfully
Response:
{
"status": "error",
"error": "Failed to fetch artifacts for job: 404 Not Found at gcsweb URL",
"action": "verify_job_url_validity"
}
Scenario 2: Timeout on Job Analysis
If the classifier script times out or returns an incomplete result:
- Report the script error.
- Continue only if at least 2 valid signatures are available from another trusted source.
- If only 1 or 0 signatures are available: Return error.
Response:
{
"status": "error",
"error": "Analysis timeout: Only 2 of 3 jobs analyzed successfully. Insufficient data for permafail determination.",
"completed_jobs": 2,
"action": "retry_with_single_job"
}
Scenario 3: Invalid Job URLs
If URL validation fails:
- Return status: "error"
- Return specific validation error message
- Do NOT attempt analysis
Response:
{
"status": "error",
"error": "Invalid job URL format: 'url3' is not a valid Prow job URL",
"invalid_url": "url3",
"action": "provide_valid_urls"
}
Scenario 4: Job Names Don't Match
If the job_name parameter doesn't match the actual job names extracted from URLs:
- Return status: "error"
- Return the expected vs actual job names
Response:
{
"status": "error",
"error": "Job name mismatch. Expected 'pull-ci-job-xyz' but found 'pull-ci-job-abc' in run 2",
"expected_job": "pull-ci-job-xyz",
"actual_job": "pull-ci-job-abc",
"action": "provide_matching_job_urls"
}
Scenario 5: Insufficient Comparable Runs (Different Types)
If analysis completes but there are insufficient comparable runs for permafail determination (e.g., 1 test_failure + 1 infra_failure):
- Return permafail: false
- Explain that neither failure type has ≥2 comparable runs
- This is NOT an error - the analysis succeeded but found no same-type pattern
Response:
{
"permafail": false,
"confidence": 0.70,
"reason": "Insufficient comparable runs: 1/2 test failures, 1/2 infra failures. Cannot establish a permafail pattern with only one run of each type. Need at least 2 runs of the same failure type to determine if failures are systematic. Test: 1/2 (threshold: 2/2). Infra: 1/2 (threshold: 2/2).",
"failure_type": "mixed",
"match_ratio": "1/2",
"matching_runs": 1,
"comparable_runs": 2,
"threshold_required": 2,
"signatures": [
{
"type": "test_failure",
"url": "...",
"tests": ["[sig-network] test"]
},
{
"type": "infra_failure",
"url": "...",
"error": "cluster creation failed"
}
]
}
Scenario 6: Failure to Extract Failure Details
If the classifier output doesn't contain expected failure information:
- Mark this run as "incomplete"
- Continue with other runs
- If ≥2 runs have valid failure data, proceed with analysis
- Otherwise, return error
Response:
{
"status": "incomplete",
"warning": "Run 1 analysis incomplete: could not extract failure details",
"completed_jobs": 2,
"incomplete_jobs": 1,
"permafail": "unknown",
"recommendation": "Review job logs manually or retry analysis"
}
Examples
Example 1: Permafail - Identical Failing Test
Input:
{
"failure_urls": [
"https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234567",
"https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234568",
"https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234569"
],
"job_name": "pull-ci-openshift-origin-master-e2e-aws",
"pr_info": {
"pr_number": 12345,
"repository": "openshift/origin"
}
}
Subagent analysis results for all 3 runs:
- Run 1: Failed tests = ["[sig-api] API discovery should provide capability information"]
- Run 2: Failed tests = ["[sig-api] API discovery should provide capability information"]
- Run 3: Failed tests = ["[sig-api] API discovery should provide capability information"]
Output:
{
"permafail": true,
"confidence": 0.99,
"reason": "3/3 consecutive runs fail with identical test: '[sig-api] API discovery should provide capability information'. This is a systematic permanent failure.",
"failure_type": "test_failure",
"match_ratio": "3/3",
"matching_runs": 3,
"comparable_runs": 3,
"threshold_required": 3,
"signatures": [
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234567",
"tests": ["[sig-api] API discovery should provide capability information"],
"test_count": 1
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234568",
"tests": ["[sig-api] API discovery should provide capability information"],
"test_count": 1
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234569",
"tests": ["[sig-api] API discovery should provide capability information"],
"test_count": 1
}
],
"common_tests": ["[sig-api] API discovery should provide capability information"]
}
Example 2: Permafail Despite Mixed Failure Types
Input:
{
"failure_urls": [
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1111",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1112",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1113",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1114",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1115",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1116",
"https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1117"
],
"job_name": "periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node",
"pr_info": {
"pr_number": 3186,
"repository": "openshift/ovn-kubernetes"
}
}
Subagent analysis results:
- Run 1: Infrastructure failure = "Cluster creation timeout" (infra_failure)
- Run 2: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
- Run 3: Infrastructure failure = "AWS quota exceeded" (infra_failure)
- Run 4: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
- Run 5: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
- Run 6: Infrastructure failure = "Cluster creation timeout" (infra_failure)
- Run 7: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
Analysis:
- 7 total runs: 3 infra_failures, 4 test_failures
- Test failures: 4/4 (100%) have identical failing test
- Infra failures: 3 runs, but different errors (not a permafail pattern in infra)
- Verdict: PERMAFAIL = TRUE based on test failure group
Output:
{
"permafail": true,
"confidence": 0.99,
"reason": "All 4 runs that reached e2e tests failed on '[sig-network] Networking should provide connectivity' (100% match). 3 additional runs failed during infrastructure setup (cluster creation, AWS quota) and are not relevant to this test failure pattern. This is a systematic test failure caused by the PR changes.",
"failure_type": "test_failure",
"signatures": [
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1111",
"error": "Cluster creation timeout",
"error_hash": "a1b2c3d4"
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1112",
"tests": ["[sig-network] Networking should provide connectivity"],
"test_count": 1
},
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1113",
"error": "AWS quota exceeded",
"error_hash": "e5f6g7h8"
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1114",
"tests": ["[sig-network] Networking should provide connectivity"],
"test_count": 1
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1115",
"tests": ["[sig-network] Networking should provide connectivity"],
"test_count": 1
},
{
"type": "infra_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1116",
"error": "Cluster creation timeout",
"error_hash": "a1b2c3d4"
},
{
"type": "test_failure",
"url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1117",
"tests": ["[sig-network] Networking should provide connectivity"],
"test_count": 1
}
],
"common_tests": ["[sig-network] Networking should provide connectivity"]
}
Example 3: NOT Permafail - Insufficient Matching Infra Errors
Input: 5 runs where 3 have identical operator installation failure, 2 have random infra issues
Subagent analysis results:
- Run 1: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
- Run 2: Infrastructure failure = "AWS quota exceeded for instance type m5.xlarge" (infra_failure)
- Run 3: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
- Run 4: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
- Run 5: Infrastructure failure = "Pod evicted due to memory pressure on node ip-10-0-1-2" (infra_failure)
Analysis:
- 5 total infra_failure signatures (denominator = 5)
- "operator authentication timeout" appears in 3 runs (numerator = 3)
- Threshold for N=5 (4-5 runs): need at least 4 matching (80% required)
- 3/5 = 60% < 80% → NOT PERMAFAIL
Output:
{
"permafail": false,
"confidence": 0.70,
"reason": "Infrastructure failures are not consistent enough. 3/5 runs failed with 'operator authentication timeout waiting for operator to reach Available=True' (60% match), but with 5 infra failures at least 4/5 must match (80% required). The presence of 2 different random errors indicates flaky infrastructure rather than systematic failure.",
"failure_type": "infra_failure",
"match_ratio": "3/5",
"matching_runs": 3,
"comparable_runs": 5,
"threshold_required": 4,
"signatures": [
{
"type": "infra_failure",
"url": "...",
"error": "operator authentication timeout waiting for operator to reach Available=True",
"error_hash": "a1b2c3d4"
},
{
"type": "infra_failure",
"url": "...",
"error": "AWS quota exceeded for instance type m5.xlarge",
"error_hash": "e5f6g7h8"
},
{
"type": "infra_failure",
"url": "...",
"error": "operator authentication timeout waiting for operator to reach Available=True",
"error_hash": "a1b2c3d4"
},
{
"type": "infra_failure",
"url": "...",
"error": "operator authentication timeout waiting for operator to reach Available=True",
"error_hash": "a1b2c3d4"
},
{
"type": "infra_failure",
"url": "...",
"error": "Pod evicted due to memory pressure on node ip-10-0-1-2",
"error_hash": "i9j0k1l2"
}
]
}
Example 4: Non-Permafail - No Consistent Pattern
Input: 3 runs with different test failures
Subagent analysis results:
- Run 1: Failed tests = ["[sig-network] networking should support networking"] (test_failure)
- Run 2: Failed tests = ["[sig-storage] storage should support volumes"] (test_failure)
- Run 3: Failed tests = ["[sig-api] API discovery should work"] (test_failure)
Output:
{
"permafail": false,
"confidence": 0.70,
"reason": "No consistent failure pattern detected. 0/3 runs show matching tests - each of the 3 runs failed with different tests: networking, storage, API discovery. This indicates flaky/non-deterministic behavior rather than a systematic permafail.",
"failure_type": "test_failure",
"match_ratio": "0/3",
"matching_runs": 0,
"comparable_runs": 3,
"threshold_required": 3,
"signatures": [
{
"type": "test_failure",
"url": "...",
"tests": ["[sig-network] networking should support networking"],
"test_count": 1
},
{
"type": "test_failure",
"url": "...",
"tests": ["[sig-storage] storage should support volumes"],
"test_count": 1
},
{
"type": "test_failure",
"url": "...",
"tests": ["[sig-api] API discovery should work"],
"test_count": 1
}
]
}
Threshold Logic Validation Examples
Table-driven test cases demonstrating correct threshold behavior:
| Total URLs | Test Failures | Infra Failures | Matching Test Count | Verdict | Reason |
|---|---|---|---|---|---|
| 10 | 10 | 0 | 2 same test | NOT permafail | 2/10 = 20% < 70% threshold (need ≥7) |
| 10 | 10 | 0 | 7 same test | PERMAFAIL | 7/10 = 70% ≥ 70% threshold ✓ |
| 7 | 4 | 3 | 4 same test (in test bucket) | PERMAFAIL | 4/4 = 100% ≥ 80% threshold (4-5 runs need ≥4) ✓ |
| 7 | 1 | 6 | 1 test, all 6 infra diverse | NOT permafail | Only 1 test_failure (need ≥2), and infra errors are all different |
| 7 | 1 | 6 | 1 test, 5 same infra error | PERMAFAIL | 5/6 = 83% ≥ ceil(6×0.7)=5 (70% threshold) → infra bucket meets threshold ✓ |
Technical Details
Artifact Analysis Approach
This skill analyzes Prow job artifacts directly using techniques from existing CI skills:
- URL parsing - Extract job name and build ID from Prow URLs
- Artifact fetching - Use
plugins/ci/scripts/classify-job-failures.pyto fetch prowjob.json and browse artifacts - Classification - Use artifact-based detection (junit files, test directories) to classify failures
- Failure extraction - Parse junit XML or build logs to extract test names or error messages
This approach follows patterns from fetch-prowjob-json, prow-job-artifact-search, and prow-job-analyze-test-failure skills.
Script Execution Strategy
The classifier script performs deterministic artifact analysis before AI threshold reasoning:
failure_urls ━━━ classify-job-failures.py ━━━ normalized signatures ━━━ Step 5 threshold analysis
Benefits:
- Keeps artifact fetching and parsing deterministic.
- Avoids spending agent reasoning on mechanical classification.
- Produces the same normalized signature schema used by offline eval cases.
Synchronization:
- Run the script once with all failure URLs.
- Require at least 2 valid normalized signatures to proceed.
- Stop and report the script's JSON error if URL validation or artifact fetching fails.
Error Message Normalization
Normalize infrastructure error messages for comparison:
- Remove timestamps:
2025-05-12T14:32:10Z→ "" - Remove build IDs:
build-12345-xyz→ "" - Remove resource names with IDs:
pod-abc123xyz→ "pod-*" - Remove request/limit values: Numbers in memory/CPU specs → ""
- Keep: Error classification, core message, error type
Example normalization:
Input: "Pod evicted at 2025-05-12T14:32:10Z (build-12345): insufficient memory (512M < 1Gi required)"
Output: "Pod evicted: insufficient memory (* < *Gi required)"
Confidence Scoring
Confidence reflects how certain the permafail verdict is:
- 0.99: All 3 runs have identical failure signature (test names or error hashes)
- 0.95: All 3 runs have ≥1 common failing test in test_failure
- 0.92: All 3 runs have >80% similar error messages in infra_failure
- 0.85: 2 of 3 runs share common failure or mixed types detected
- 0.70: Insufficient data or ambiguous failure patterns
Use confidence to determine remediation priority:
- Confidence ≥ 0.95: High priority permafail, block PR merge
- Confidence 0.85-0.94: Medium priority, warn but allow manual override
- Confidence < 0.85: Low confidence verdict, require manual review
Diagnose Job Run Symptoms
Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.
This skill takes a Prow job run URL and explains which symptoms matched the run and what each applied label means — including the matched file and text.
When to Use This Skill
Use this skill when:
- A CI job failed and you want to know if it is a known failure mode before debugging it from scratch
- You are about to create a new symptom and want to check what already matched the run (see
manage-symptoms) - You want a plain-language explanation of the labels shown on a run in the Sippy UI or Spyglass
Prerequisites
-
Default mode (already-applied labels): only network access to public GCS (
https://storage.googleapis.com) and the public Sippy API (https://sippy.dptools.openshift.org) — no authentication required. -
Deep mode (
--deep, server-side dry-run rescan): a Bearer token from the DPCR cluster.- Must be logged into the DPCR cluster via
oc login - Cluster API:
https://api.cr.j7t7.p1.openshiftapps.com:6443 - Use the
oc-authskill to obtain the token (see the token-acquisition snippet inreevaluate-job-runs/SKILL.md); preferexport SIPPY_TOKEN=...over--token— argv is visible in process listings
- Must be logged into the DPCR cluster via
-
Python 3: Python 3.6 or later, standard library only.
Implementation Steps
Step 1: Default mode — explain already-applied labels
python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id>"
For each applied label the summary output shows: the label title and explanation, links to associated Jira issues, the symptom that applied it (summary, matcher type, file pattern, match string), and the actual matched file and matched text from the run's artifacts.
Step 2: Deep mode — server-side rescan
Use when the run predates the current symptom set or shows no labels. This asks Sippy to re-scan the run with dry_run: true — it writes nothing and reports what would match now:
python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
"<prow_url>" --deep
Note: deep mode reports label IDs only — the matched file/text detail is only available in default mode (it comes from the GCS artifacts).
Deep mode submits an asynchronous batch, polls its same-origin status link
through pending, processing, and running, and reports results after the
batch reaches complete. A failed or cancelled batch is an error.
Step 3: Nothing matched?
If default mode finds no labels, first suggest --deep: the run may simply never have been scanned, and default mode cannot distinguish that from "scanned, nothing matched". If a deep rescan also finds nothing and the user has identified the failure cause, guide them to the manage-symptoms skill to create a new symptom (and reevaluate-job-runs to apply it retroactively to past runs).
Arguments:
prow_url: Prow job run URL (https://prow.ci.openshift.org/view/gs/..., positional, required)
Options:
--deep: Server-side dry-run rescan via the reevaluate API (requires a token)--token <token>: Bearer token from the oc-auth skill (only needed with--deep; optional if theSIPPY_TOKENenvironment variable is set, which is preferred — argv is visible in process listings;--tokentakes precedence)--format json|summary: Output format (default: summary)
API Details
Default mode uses the public GCS JSON API (no auth):
- List label artifacts:
GET https://storage.googleapis.com/storage/v1/b/{bucket}/o?prefix={path}/artifacts/job_labels/ - Download each object:
GET https://storage.googleapis.com/storage/v1/b/{bucket}/o/{object}?alt=media
GCS job_labels artifact schema (verified live 2026-07): each *.json object under artifacts/job_labels/ contains one wrapped entry (a label-summary.html file is also present and skipped):
{
"symptom_label_v1": {
"symptom": {"id": "...", "summary": "...", "matcher_type": "string",
"file_pattern": "...", "match_string": "...", "label_ids": ["..."]},
"label": {"id": "...", "label_title": "...", "explanation": "...",
"bugs": ["OCPBUGS-12345"]},
"file_match": "artifacts/.../nodes.json",
"text_match": "the exact text that matched"
}
}
The script also cross-references GET /api/jobs/labels and GET /api/jobs/symptoms on the public Sippy API to enrich entries with current explanations and Jira associations (embedded copies are snapshots from labeling time).
Deep mode calls POST https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate with {"prow_job_build_ids": ["<build_id>"], "dry_run": true}. Sippy returns HTTP 202 with batch_id, requested, and links.status. The client validates that status link, polls it with authenticated GET requests, and reads labels from each matching items[].result.labels_applied. Cross-origin status URLs and HTTP redirects are rejected rather than followed with or without credentials.
Error Handling
- Bad / non-Prow URL: Must contain
/view/gs/and end in a numeric build ID (exit 1 client-side). - GCS 404 or no
job_labelsartifacts: The run may be too old (artifacts pruned) or was never scanned — suggest--deepto rescan server-side. - No labels found: Not an error — the script prints guidance (try
--deepfirst, then create a new symptom viamanage-symptoms). - 401/403 or HTML login page in deep mode: Token missing/expired (the SSO proxy may return a login page instead of 401) — refresh via the
oc-authskill. - HTML gateway error page or non-JSON body in deep mode: Transient gateway error (likely 504) — retry later.
- Unknown or malformed batch status: Rejected as an API error instead of polling indefinitely.
- Failed or cancelled deep batch: Reported as an error (exit 1).
- Sippy API unreachable: exit 1 with a clear message.
Exit Codes:
0: Diagnosis completed (including "no labels found")1: Validation, network, or API error
See Also
- Related Skill:
list-symptoms(browse/search the symptom and label catalogs) - Related Skill:
manage-symptoms(create a new symptom when nothing matched) - Related Skill:
reevaluate-job-runs(apply symptoms retroactively to past runs) - Related Skill:
oc-auth(token for deep mode) - Related Skill:
prow-job-analyze-test-failure(deeper manual failure analysis)
Fetch JIRA Issue
This skill fetches detailed information about a JIRA issue from the Red Hat JIRA REST API. It retrieves status, assignee, priority, resolution, comments, linked PRs, and classifies the issue's progress as ACTIVE, STALLED, or NEEDS_ATTENTION.
When to Use This Skill
Use this skill when you need to:
- Check the status and progress of a JIRA bug linked to a regression triage
- Determine if a bug is being actively worked on or needs attention
- Extract PR links from JIRA comments
- Get assignee and component information for a JIRA issue
- Analyze recent comment activity on a bug
Prerequisites
-
JIRA credentials: Required for authentication, one of:
- API token:
- Set environment variable:
export JIRA_API_TOKEN="your-token" - Set environment variable:
export JIRA_USERNAME="your-atlassian-email" - Obtain from: https://id.atlassian.com/manage-profile/security/api-tokens
- Alternatively, pass
--token TOKENand--username USERNAMEon the command line
- Set environment variable:
- Service account (OAuth client credentials): set
JIRA_CLIENT_ID,JIRA_CLIENT_SECRETandJIRA_CLOUD_ID. Used whenever the client ID or secret is set and neither--tokennor--usernameis passed; there is no fallback to the API token. Requests then go toapi.atlassian.cominstead ofredhat.atlassian.net
- API token:
-
Network Access: Must be able to reach redhat.atlassian.net
- Check:
curl -s -o /dev/null -w '%{http_code}' https://redhat.atlassian.net
- Check:
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
Implementation Steps
Step 1: Run the Python Script
# Path to the Python script
script_path="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py"
# Fetch issue data in JSON format (default)
python3 "$script_path" OCPBUGS-74401 --format json
# Or fetch as human-readable summary
python3 "$script_path" OCPBUGS-74401 --format summary
# Pass token explicitly instead of using JIRA_API_TOKEN env var
python3 "$script_path" OCPBUGS-74401 --token "your-token" --format json
Step 2: Parse the Output
The script outputs structured JSON data that can be further processed:
# Store result in variable
jira_data=$(python3 "$script_path" OCPBUGS-74401 --format json)
# Extract key fields
status=$(echo "$jira_data" | jq -r '.status')
assignee=$(echo "$jira_data" | jq -r '.assignee.display_name // "Unassigned"')
progress_level=$(echo "$jira_data" | jq -r '.progress.level')
progress_reason=$(echo "$jira_data" | jq -r '.progress.reason')
# Check progress classification
echo "Status: $status"
echo "Assignee: $assignee"
echo "Progress: $progress_level - $progress_reason"
Step 3: Use Progress Classification
The skill automatically classifies issue progress:
- ACTIVE: Issue is being worked on (assigned with recent activity, PRs linked, or active status)
- STALLED: Issue is assigned but has no activity in 14+ days
- NEEDS_ATTENTION: Issue is NEW/unassigned or has no progress
- RESOLVED: Issue is closed/verified
# Example: Check if a triaged regression's bug needs attention
progress_level=$(echo "$jira_data" | jq -r '.progress.level')
case "$progress_level" in
"ACTIVE")
echo "Bug is being worked on - no action needed"
;;
"STALLED")
echo "Bug may need attention - consider commenting or reassigning"
;;
"NEEDS_ATTENTION")
echo "Bug needs someone to pick it up"
;;
"RESOLVED")
echo "Bug is closed"
;;
esac
Error Handling
The script exits with code 1 and prints errors to stderr for these cases:
- No credentials: neither the API token variables (or
--token) nor the service-account variables are set, or the service-account set is incomplete - Auth failure (401): Token is invalid or expired, or the service-account credential was rejected
- Access denied (403): Token lacks permissions for the project
- Not found (404): Issue key doesn't exist
- Network error: Cannot reach redhat.atlassian.net
# Example: Handle missing token gracefully
if [ -z "$JIRA_CLIENT_ID" ] && { [ -z "$JIRA_API_TOKEN" ] || [ -z "$JIRA_USERNAME" ]; }; then
echo "Warning: JIRA credentials not set. Skipping JIRA analysis."
else
jira_data=$(python3 "$script_path" "$jira_key" --format json 2>/dev/null)
if [ $? -ne 0 ]; then
echo "Warning: Failed to fetch JIRA issue. Continuing without JIRA data."
fi
fi
Output Format
JSON Format (--format json)
{
"key": "OCPBUGS-74401",
"url": "https://redhat.atlassian.net/browse/OCPBUGS-74401",
"summary": "ovn-ipsec-host creates duplicate openssl attribute",
"status": "Modified",
"resolution": null,
"priority": "Critical",
"assignee": {
"display_name": "Jane Developer",
"email": "jdeveloper@redhat.com"
},
"reporter": {
"display_name": "John Reporter",
"email": "jreporter@redhat.com"
},
"components": ["Networking / cluster-network-operator"],
"labels": ["Regression"],
"fix_versions": ["4.22"],
"created": "2026-01-20T14:30:00.000+0000",
"updated": "2026-02-10T09:15:00.000+0000",
"comment_count": 5,
"comments": [
{
"author": "Jane Developer",
"created": "2026-02-08T10:00:00.000+0000",
"body": "PR submitted: https://github.com/openshift/ovn-kubernetes/pull/1234"
}
],
"linked_prs": [
"https://github.com/openshift/ovn-kubernetes/pull/1234"
],
"progress": {
"level": "ACTIVE",
"label": "ACTIVE",
"reason": "PR in progress (1 linked)",
"days_since_update": 2,
"days_since_last_comment": 4
}
}
Summary Format (--format summary)
JIRA Issue: OCPBUGS-74401
============================================================
Summary: ovn-ipsec-host creates duplicate openssl attribute
URL: https://redhat.atlassian.net/browse/OCPBUGS-74401
Status: Modified
Priority: Critical
Assignee: Jane Developer
Reporter: John Reporter
Components: Networking / cluster-network-operator
Labels: Regression
Fix Versions: 4.22
Created: 2026-01-20
Updated: 2026-02-10
Progress: ACTIVE - PR in progress (1 linked)
Days since update: 2
Days since last comment: 4
Linked PRs (1):
- https://github.com/openshift/ovn-kubernetes/pull/1234
Recent Comments (1 of 5 total):
[2026-02-08] Jane Developer:
PR submitted: https://github.com/openshift/ovn-kubernetes/pull/1234
Notes
- The script uses only Python standard library modules (no pip dependencies)
- All comments are returned in the JSON output; the summary format shows only the last 3
- PR links are extracted from comment bodies using pattern matching for GitHub PR URLs
- Progress classification uses configurable thresholds: 7 days for "recent", 14 days for "stale"
- The
--tokenflag takes precedence over theJIRA_API_TOKENenvironment variable and over service-account credentials
See Also
- Related Command:
/ci:analyze-regression- Uses this skill to check JIRA progress on triaged regressions - Related Command:
/ci:check-if-jira-regression-is-ongoing- Uses this skill for JIRA bug analysis - Related Skill:
fetch-regression-details- Fetches regression data that may link to JIRA bugs - Related Skill:
triage-regression- Creates triage records linking regressions to JIRA bugs - JIRA REST API: https://developer.atlassian.com/server/jira/platform/rest-apis/
Fetch Job Run Summary
This skill fetches a summary of a Prow job run from the Sippy API, listing all tests that failed (excluding flakes) along with their error messages.
When to Use This Skill
Use this skill when you need to:
- List all tests that failed in a specific Prow job run
- Understand which components are affected by failures in a job
- Identify dominant error patterns across multiple test failures
- Investigate mass test failure regressions by examining the underlying test failures
- Get a quick overview of a job run's health (pass rate, infrastructure failure, etc.)
Prerequisites
- Python 3: Version 3.6 or later
- Network Access: Must be able to reach
https://sippy.dptools.openshift.org
Implementation
script_path="plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py"
# Text output (AI-readable)
python3 "$script_path" <prow_job_run_id>
# JSON output (structured)
python3 "$script_path" <prow_job_run_id> --format json
Parameters
| Parameter | Required | Description |
|---|---|---|
prow_job_run_id | Yes | The Prow job run ID (numeric, e.g., 2030845545290928128) |
--format | No | Output format: text (default) or json |
Extracting a Job Run ID
The job run ID can be extracted from a Prow job URL:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job-name>/<job_run_id>
The last path segment is the job run ID.
API Details
Endpoint: https://sippy.dptools.openshift.org/api/job/run/summary
Query Parameter: prow_job_run_id (required)
Response Fields:
| Field | Type | Description |
|---|---|---|
id | int | Prow job run ID |
name | string | Full job name |
release | string | OpenShift release (e.g., 4.22) |
cluster | string | Build cluster (e.g., build01) |
url | string | Prow job URL |
startTime | string | ISO 8601 start time |
durationSeconds | int | Job duration in seconds |
overallResult | string | S (success) or F (failure) |
reason | string | Human-readable result reason |
infrastructureFailure | bool | Whether this was an infrastructure failure |
testCount | int | Total number of tests run |
testFailureCount | int | Number of failed tests |
testFailures | dict | Map of test name to error message (excludes flakes) |
variants | list | Job variant tags |
Flake Handling
The Sippy API automatically handles flake detection. When a test has both a failure and a pass in the same job run's junit results (the standard flake pattern), it is not included in testFailures. Only true failures — tests that failed without a corresponding pass — appear in the results.
Output
Text Format
The text output is structured for AI consumption with:
- Job Run Summary: Metadata (job name, release, duration, result, variants)
- Test Statistics: Total tests, failures, pass rate
- Failed Tests: Failure count (sorted by count descending)
- Dominant Error Patterns: Error messages appearing in >5% of failures, with counts
- Detailed Failures: Each failed test with its error message, grouped by SIG
JSON Format
Structured JSON with the same data:
{
"success": true,
"job_name": "periodic-ci-openshift-hypershift-...",
"job_run_id": "2030845545290928128",
"release": "4.22",
"test_count": 3827,
"failure_count": 400,
"pass_rate": 89.5,
"failed_tests": [
{"test_name": "[sig-cli] example test name", "error": "error message..."}
],
"dominant_error_patterns": [
{"pattern": "stale GroupVersion discovery: ...", "count": 50, "percentage": 12.5}
]
}
Examples
Example 1: Investigate a Mass Test Failure
# Get the summary of a job run that triggered mass test failures
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845545290928128
Example 2: Compare Two Runs
# Fetch summaries for a failing and passing run to diff
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845545290928128 --format json > failing.json
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845542774345728 --format json > passing.json
Notes
- The
testFailureCountmay be slightly higher than the number of entries intestFailuresdue to internal deduplication - Error messages are truncated to 300 characters in text output to keep output manageable
- The dominant error pattern detection strips UUIDs and timestamps for better grouping
- For test-level output analysis (raw test failure output for a specific test across runs), use the
fetch-test-runsskill instead
See Also
- Related Skill:
fetch-test-runs- Fetch raw outputs for a specific test across multiple job runs - Related Skill:
fetch-regression-details- Fetch Component Readiness regression details - Related Skill:
ci:prow-job-analysis- Comprehensive analysis of test and install failures in a Prow job
Fetch New PRs in Payload
This skill fetches the list of pull requests that are new in a given OpenShift payload tag compared to the previous payload. It tries the Sippy payload diff API first, and falls back to the release controller API when Sippy has not yet ingested the payload (e.g., in-progress or very recent payloads).
When to Use This Skill
Use this skill when you need to:
- Determine what changed between two consecutive payloads
- Identify which PRs were included in a specific payload
- Investigate whether a specific PR or component change landed in a payload
- Correlate a regression or test failure with newly merged PRs
- Build a changelog of what shipped in a payload
The payload tag can be obtained from the fetch-prowjob-json skill (release.openshift.io/tag annotation) or from release controller pages.
Prerequisites
-
Network Access: Must be able to reach the Sippy API and/or the release controller
- Sippy:
curl -s https://sippy.dptools.openshift.org/api/health - Release controller:
curl -s https://amd64.ocp.releases.ci.openshift.org/api/v1/releasestream/4-stable/latest - No authentication required for either
- Sippy:
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
Implementation Steps
Step 1: Run the Python Script
The skill uses a Python script to fetch and format the payload diff data:
# Locate the Python script
FETCH_NEW_PRS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py"
if [ ! -f "$FETCH_NEW_PRS" ]; then
FETCH_NEW_PRS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py" 2>/dev/null | sort | head -1)
fi
if [ -z "$FETCH_NEW_PRS" ] || [ ! -f "$FETCH_NEW_PRS" ]; then echo "ERROR: fetch_new_prs_in_payload.py not found" >&2; exit 2; fi
# Fetch new PRs in JSON format
python3 "$FETCH_NEW_PRS" <payload_tag> --format json
# Or fetch as human-readable summary grouped by component
python3 "$FETCH_NEW_PRS" <payload_tag> --format summary
Step 2: Parse the Output
The script outputs structured JSON data that can be further processed:
# Store JSON output in a variable for processing
pr_data=$(python3 "$script_path" 4.22.0-0.ci-2026-02-06-195709 --format json)
# Extract specific fields using jq if needed
total=$(echo "$pr_data" | jq '.total_prs')
pr_urls=$(echo "$pr_data" | jq -r '.pull_requests[].url')
components=$(echo "$pr_data" | jq -r '[.pull_requests[].component] | unique[]')
# Find PRs for a specific component
echo "$pr_data" | jq '.pull_requests[] | select(.component == "hypershift")'
# Find PRs with associated bugs
echo "$pr_data" | jq '.pull_requests[] | select(.bug_url != "")'
Step 3: Use the Data
The structured data includes all PR details from the payload diff:
{
"payload_tag": "4.22.0-0.nightly-2026-01-15-114134",
"total_prs": 17,
"pull_requests": [
{
"url": "https://github.com/openshift/assisted-service/pull/8594",
"pull_request_id": "8594",
"component": "agent-installer-api-server",
"description": "Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer",
"bug_url": "https://redhat.atlassian.net/browse/MGMT-22584"
},
{
"url": "https://github.com/openshift/hypershift/pull/7470",
"pull_request_id": "7470",
"component": "hypershift",
"description": "use InfraStatus.APIPort for custom DNS kubeconfig",
"bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-72258"
}
]
}
Error Handling
The Python script handles common error cases automatically:
Case 1: Payload Not Found (404)
python3 fetch_new_prs_in_payload.py 4.22.0-0.ci-9999-99-99-000000
# Error: Payload '4.22.0-0.ci-9999-99-99-000000' not found.
# Verify the payload tag exists (e.g., 4.22.0-0.ci-2026-02-06-195709).
Case 2: Network Error
python3 fetch_new_prs_in_payload.py 4.22.0-0.ci-2026-02-06-195709
# Error: Failed to connect to Sippy API: [Errno -2] Name or service not known
# Check network connectivity.
Case 3: Missing Arguments
python3 fetch_new_prs_in_payload.py
# usage: fetch_new_prs_in_payload.py [-h] [--format {json,summary}] payload_tag
# fetch_new_prs_in_payload.py: error: the following arguments are required: payload_tag
Exit Codes:
0: Success1: Error (invalid input, API error, network error, etc.)
Data Sources
The script tries two APIs in order:
- Sippy (primary) — has longer history but only includes completed payloads
- Release controller (fallback) — available immediately for in-progress and recent payloads, but has shorter retention
The output format is identical regardless of which source is used. The fallback is automatic and transparent.
API Details
Sippy Endpoint
GET https://sippy.dptools.openshift.org/api/payloads/diff?toPayload={payload_tag}
Parameters
toPayload(required): The payload tag to diff against its predecessor (e.g.,4.22.0-0.ci-2026-02-06-195709)
Response Schema
The API returns a JSON array of PR objects:
[
{
"id": 0,
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"deleted_at": null,
"url": "https://github.com/openshift/hypershift/pull/7470",
"pull_request_id": "7470",
"name": "hypershift",
"description": "use InfraStatus.APIPort for custom DNS kubeconfig",
"bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-72258"
}
]
Raw API Fields:
id,created_at,updated_at,deleted_at: Database metadata (not useful for consumers)url: Full GitHub pull request URLpull_request_id: PR number as a stringname: Component name(s) affected by this PR (may contain comma-separated values for multi-component PRs, e.g.,"olm-catalogd, olm-operator-controller")description: PR title/descriptionbug_url: Associated Jira bug URL (empty string if none)
The Python script remaps name to component in its output for clarity.
Examples
Example 1: Fetch PRs as JSON
python3 "$FETCH_NEW_PRS" 4.22.0-0.nightly-2026-01-15-114134 --format json
Expected Output:
{
"payload_tag": "4.22.0-0.nightly-2026-01-15-114134",
"total_prs": 17,
"pull_requests": [
{
"url": "https://github.com/openshift/assisted-service/pull/8594",
"pull_request_id": "8594",
"component": "agent-installer-api-server",
"description": "Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer",
"bug_url": "https://redhat.atlassian.net/browse/MGMT-22584"
},
{
"url": "https://github.com/openshift/machine-config-operator/pull/5509",
"pull_request_id": "5509",
"component": "machine-config-operator",
"description": "Set `NodeDegraded` MCN condition when node state annotation is set to `Degraded`",
"bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-67229"
}
]
}
Example 2: Fetch PRs as Summary
python3 "$FETCH_NEW_PRS" 4.22.0-0.nightly-2026-01-15-114134 --format summary
Expected Output:
New PRs in payload 4.22.0-0.nightly-2026-01-15-114134
============================================================
Total: 17 new pull requests
agent-installer-api-server (1 PRs):
- Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer [https://redhat.atlassian.net/browse/MGMT-22584]
https://github.com/openshift/assisted-service/pull/8594
hypershift (4 PRs):
- [kubevirt] Make L3 migration labeling conditional [https://redhat.atlassian.net/browse/OCPBUGS-66205]
https://github.com/openshift/hypershift/pull/7308
- feat(api): add support for graceful service account signing key rotation [https://redhat.atlassian.net/browse/CNTRLPLANE-1768]
https://github.com/openshift/hypershift/pull/7324
- Scaffold OpenShiftManager controller [https://redhat.atlassian.net/browse/API-1835]
https://github.com/openshift/hypershift/pull/7445
- use InfraStatus.APIPort for custom DNS kubeconfig [https://redhat.atlassian.net/browse/OCPBUGS-72258]
https://github.com/openshift/hypershift/pull/7470
machine-config-operator (2 PRs):
- Set `NodeDegraded` MCN condition when node state annotation is set to `Degraded` [https://redhat.atlassian.net/browse/OCPBUGS-67229]
https://github.com/openshift/machine-config-operator/pull/5509
- Prevent unnecessary systemd unit disable [https://redhat.atlassian.net/browse/OCPBUGS-58023]
https://github.com/openshift/machine-config-operator/pull/5527
Example 3: Use with fetch-prowjob-json Skill
Combine with the fetch-prowjob-json skill to get the payload tag from a Prow job, then find what PRs were new in that payload:
# 1. Get payload tag from prowjob.json (via fetch-prowjob-json skill)
# payload_tag = metadata.annotations["release.openshift.io/tag"]
# e.g., "4.22.0-0.ci-2026-02-06-195709"
# 2. Fetch new PRs in that payload
python3 "$FETCH_NEW_PRS" "$payload_tag" --format json
Notes
- Neither API requires authentication for read-only access
- The Python script uses only standard library modules (no external dependencies)
- The previous payload is determined automatically; you only provide the target payload
- Payload tags follow the format:
{version}-0.{stream}-{date}-{time}(e.g.,4.22.0-0.ci-2026-02-06-195709or4.22.0-0.nightly-2026-01-15-114134) - The
componentfield (callednamein the raw API) may contain multiple comma-separated component names for PRs that affect multiple components - PRs without an associated bug will have an empty string for
bug_url - An empty response array means no new PRs were found (the payload may be identical to its predecessor)
See Also
- Related Skill:
fetch-prowjob-json(provides payload tag from Prow job metadata) - Related Skill:
fetch-regression-details(for correlating regressions with payload changes) - Related Command:
/ci:analyze-regression(analyzes regressions that may be caused by new PRs)
Fetch Payloads
This skill fetches recent release payloads from the OpenShift release controller, showing their tag name, acceptance phase, timestamp, blocking job results, and a link to the release details page.
When to Use This Skill
Use this skill when you need to:
- Find the latest accepted (or rejected) nightly or CI payloads for a given OCP version
- Check the current state of release payloads for any architecture
- See which blocking jobs failed for rejected payloads
- Get a link to the release controller page for a specific payload
Implementation Steps
Step 1: Determine defaults
If the user did not specify an architecture or stream, default to amd64 and nightly. If no version is specified, the script automatically fetches the latest from the Sippy API.
Step 2: Fetch payloads
FETCH_PAYLOADS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-payloads/fetch_payloads.py"
if [ ! -f "$FETCH_PAYLOADS" ]; then
FETCH_PAYLOADS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-payloads/fetch_payloads.py" 2>/dev/null | sort | head -1)
fi
if [ -z "$FETCH_PAYLOADS" ] || [ ! -f "$FETCH_PAYLOADS" ]; then echo "ERROR: fetch_payloads.py not found" >&2; exit 2; fi
python3 "$FETCH_PAYLOADS" [architecture] [version] [stream]
Examples:
# Latest amd64 nightly payloads (all defaults, last 5)
python3 "$FETCH_PAYLOADS"
# arm64 4.18 nightly
python3 "$FETCH_PAYLOADS" arm64 4.18 nightly
# Only accepted payloads
python3 "$FETCH_PAYLOADS" amd64 4.18 nightly --phase Accepted
# Show more results
python3 "$FETCH_PAYLOADS" amd64 4.18 nightly --limit 20
Step 3: Present results
The script outputs one block per payload to stdout with job details. Present to the user as-is or summarize.
Output Format
The script outputs a JSON object to stdout:
{
"hours_since_last_accepted": 23.5,
"last_accepted_tag": "4.22.0-0.nightly-2026-02-24-030944",
"payloads": [ ... ]
}
hours_since_last_accepted: Hours since the most recent Accepted payload in the stream (from the full unfiltered history), ornullif none found.last_accepted_tag: Tag name of the most recent Accepted payload, ornullif none found.payloads: Array of payload objects, each containing tag, phase, release controller URL, and job results (blocking and async jobs with Prow URLs and retry details).
Error Handling
- Unknown architecture: Exits with error listing valid architectures
- CI stream on non-amd64: Exits with error (CI stream is amd64-only)
- Network error: Prints connection failure to stderr, exits 1
- No payloads found: Prints message to stderr, exits 1
Fetch Prow Job Runs
Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.
This skill finds Prow job runs (and their prow_id values) via the Sippy /api/jobs/runs endpoint. It exists because this API does not accept ad-hoc query parameters like ?job_name= or ?period= — those do not exist and will be silently ignored or rejected. Filtering is done through a JSON filter parameter with a fixed set of column fields and operators (listed below), and this skill's script builds that JSON with verified fields and operators so you never have to guess.
When to Use This Skill
Use this skill when you need to:
- Find run IDs (
prow_id) to feed intoreevaluate-job-runsafter creating or updating a symptom - Check how widespread a failure is (e.g. all failed runs of a job family in the last day)
- List recent runs matching a platform/network/etc. variant
- Get the Prow URL of recent runs of a given job
Prerequisites
- Network Access: The Sippy API must be accessible at
https://sippy.dptools.openshift.org- No authentication required
- Python 3: Python 3.6 or later, standard library only
Implementation Steps
Invoke the script with flags matching the question:
script_path="plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py"
# Metal-platform runs in the last 24 hours
python3 "$script_path" --release 5.0 --variant Platform:metal --since-hours 24 --format summary
# Failed runs of jobs whose name contains e2e-metal
python3 "$script_path" --release 5.0 --job-contains e2e-metal --result F --format summary
# Pipe run IDs into symptom reevaluation
python3 "$script_path" --release 5.0 --job-contains e2e-metal --result F --since-hours 48 --ids-only \
| xargs python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py --dry-run
Flags:
--release <release>(required): OpenShift release, e.g.5.0--job-contains <substr>(repeatable): job name must contain each substring--variant <Key:Value>(repeatable): run must have each variant entry, e.g.Platform:metal--result <code>: overall result code (Ssuccess,Ffailure,ninfra failure — lowercase n)--since-hours <n>: only runs newer than N hours ago--filter-json <raw>: escape hatch — raw JSON array of extra filter items merged with the generated ones--limit <n>: max rows (default 100)--ids-only: print justprow_idper line (pipeline-friendly)--format {json,summary}: output format (defaultjson)
All filter items are combined with AND.
API Details
Endpoint
GET https://sippy.dptools.openshift.org/api/jobs/runs?release={release}&filter={urlencoded_json}&limit={n}&sortField=timestamp&sort=desc
No authentication required.
Filter JSON Shape
{"items": [{"columnField": "name", "operatorValue": "contains", "value": "e2e-metal"}], "linkOperator": "and"}
Verified Column Fields and Operators
Do NOT invent other query parameters or fields — only these are verified to work:
| columnField | operatorValue | value |
|---|---|---|
name | contains | job-name substring |
variants | has entry | Key:Value, e.g. Platform:metal |
timestamp | > / < | RFC 3339 UTC timestamp, e.g. 2026-10-14T12:00:00Z |
overall_result | equals | S (success), F (failure), n (infra failure — lowercase) |
An unknown columnField yields HTTP 400 with a "column does not exist" message; the script surfaces that message verbatim.
Response Row Fields
{"rows": [...]}; each row has prow_id (string), job, url, variants[], overall_result, failed, succeeded, infrastructure_failure, test_failures, timestamp (epoch millis), cluster.
Error Handling
Case 1: Unknown Column (HTTP 400)
Error: HTTP 400 from Sippy API: {"code":400,"message":"could not filter or sort data: error updating query for filter ... column does not exist"}
Use only the column fields in the table above. Exits 1.
Case 2: Network Error
Error: failed to connect to Sippy API: [Errno -2] Name or service not known
Exits 1. Check connectivity to sippy.dptools.openshift.org.
Case 3: Empty Results
No matching runs prints [] (JSON), Total: 0 (summary), or nothing (--ids-only) and exits 0 — not an error.
Exit Codes:
0: Success (including empty results)1: Error (HTTP error, network error, invalid--filter-json)
Examples
Example 1: Metal Runs in the Last 24 Hours
python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
--release 5.0 --variant Platform:metal --since-hours 24 --limit 5 --format summary
Expected Output (excerpt):
n periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-ha-cert-rotation-shutdown-5y-age-90d 2080684960133419008 1784908873000
Total: 5
Example 2: Failed Runs of a Job Family
python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
--release 5.0 --job-contains e2e-metal --result F --format summary
Example 3: Feed Run IDs to Reevaluation
python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
--release 5.0 --job-contains e2e-metal --since-hours 48 --ids-only \
| xargs python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py --dry-run
Notes
- Results are sorted newest-first (
sortField=timestamp&sort=desc) timestampfilter values must be RFC 3339 UTC strings (the script computes this for--since-hours)- Repeatable flags AND together; there is no OR support in this script (use
--filter-jsonif you need custom items)
See Also
- Related Skill:
reevaluate-job-runs(re-run symptom detection on the run IDs found here) - Related Skill:
diagnose-job-run-symptoms(explain which symptoms/labels apply to a run) - Related Skill:
list-symptoms(browse the symptom/label catalog) - Related Skill:
fetch-regression-details(another source of job run IDs, from regressions)
Fetch ProwJob JSON
This skill fetches the prowjob.json artifact for a given Prow job and returns key fields including job metadata, spec details, status, and release annotations.
When to Use This Skill
Use this skill when you need to:
- Get metadata about a specific Prow job run (status, timing, cluster, pod name)
- Determine what payload a job was tested against (
release.openshift.io/tag) - Determine the upgrade source version for upgrade jobs (
release.openshift.io/from-tag) - Get the release images (
RELEASE_IMAGE_LATEST,RELEASE_IMAGE_INITIAL) used in a job - Check the ci-operator target and variant for a job
- Get the GCS bucket path for a job's artifacts
Prerequisites
- Network Access: Must be able to reach the GCS web proxy
- Check:
curl -s -o /dev/null -w '%{http_code}' https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/ - No authentication required
- Check:
Implementation Steps
Step 1: Parse the Prow Job URL
Extract the GCS path from the Prow job URL. Prow URLs follow this pattern:
https://prow.ci.openshift.org/view/gs/<bucket>/<path>/<job-name>/<build-id>
Extract everything after /view/ to get the GCS reference:
gs/<bucket>/<path>/<job-name>/<build-id>
Step 2: Construct the GCS Web URL
Convert the Prow URL to a gcsweb URL that serves the raw prowjob.json:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/<bucket>/<path>/<job-name>/<build-id>/prowjob.json
Example:
- Input Prow URL:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672 - GCS Web URL:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672/prowjob.json
The conversion is: replace https://prow.ci.openshift.org/view/ with https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/ and append /prowjob.json.
Step 3: Fetch the prowjob.json
Use curl or WebFetch to retrieve the JSON content from the constructed URL.
curl -s "https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results-public/logs/<job-name>/<build-id>/prowjob.json"
Step 4: Extract and Return Key Fields
From the full prowjob.json, extract and return the following fields:
Required Fields
| Field | JSON Path | Description |
|---|---|---|
| Job Name | spec.job | Full periodic/presubmit/postsubmit job name |
| Job Type | spec.type | periodic, presubmit, or postsubmit |
| Build ID | status.build_id | Unique build identifier |
| State | status.state | success, failure, aborted, pending, triggered |
| Description | status.description | Human-readable status description |
| Start Time | status.startTime | ISO 8601 timestamp |
| Completion Time | status.completionTime | ISO 8601 timestamp (null if still running) |
| Cluster | spec.cluster | Build cluster (e.g., build02, build05) |
| Pod Name | status.pod_name | Kubernetes pod name |
| Prow URL | status.url | Link back to the Prow job view |
| Payload Tag | metadata.annotations["release.openshift.io/tag"] | The payload version tested or upgraded to (e.g., 4.22.0-0.ci-2026-02-06-195709). This is the payload the cluster was tested against or upgraded to. |
| Upgrade From Tag | metadata.annotations["release.openshift.io/from-tag"] | The original payload version before upgrade (e.g., 4.22.0-0.ci-2026-02-05-195709). Only present for upgrade jobs. This is the version the cluster starts at before upgrading to the payload tag. |
| Release Source | metadata.annotations["release.openshift.io/source"] | Release stream source (e.g., ocp/4.22) |
| Architecture | metadata.annotations["release.openshift.io/architecture"] | CPU architecture (e.g., amd64, arm64, multi) |
| Refs | spec.extra_refs | Array of repo references (org, repo, base_ref) |
| Target | From spec.pod_spec.containers[0].args | The --target= value from ci-operator args |
| Variant | From spec.pod_spec.containers[0].args | The --variant= value from ci-operator args |
| RELEASE_IMAGE_LATEST | From spec.pod_spec.containers[0].env | The release image the job upgrades to or installs |
| RELEASE_IMAGE_INITIAL | From spec.pod_spec.containers[0].env | The initial release image (for upgrade jobs) |
Notes on Release Annotations
release.openshift.io/tag: Present on release-controller-triggered jobs. Identifies the payload being validated. For upgrade jobs, this is the version the cluster upgrades to.release.openshift.io/from-tag: Only present on upgrade jobs. Identifies the version the cluster starts at before the upgrade. If absent, the job is not an upgrade job.- These annotations may not be present on manually triggered or presubmit jobs.
- The
RELEASE_IMAGE_LATESTandRELEASE_IMAGE_INITIALenvironment variables correspond to these tags and contain the full registry pull spec.
Error Handling
Case 1: Invalid Prow URL Format
If the URL doesn't match the expected pattern https://prow.ci.openshift.org/view/gs/...:
- Report the error to the user
- Show the expected URL format
Case 2: prowjob.json Not Found (404)
If the GCS web proxy returns a 404:
- The job may have been garbage collected or the URL may be incorrect
- Suggest verifying the Prow job URL is correct and the job exists
Case 3: Network Error
If the GCS web proxy is unreachable:
- Check connectivity to
gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com - The proxy does not require VPN or authentication
Examples
Example 1: Fetch a Periodic Upgrade Job
Input:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672
Key output fields:
Job Name: periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade
Job Type: periodic
Build ID: 2019864414127132672
State: failure
Start Time: 2026-02-06T20:02:58Z
Completion Time: 2026-02-07T00:36:09Z
Cluster: build02
Payload Tag: 4.22.0-0.ci-2026-02-06-195709
Upgrade From Tag: 4.22.0-0.ci-2026-02-05-195709
Architecture: amd64
Target: e2e-gcp-ovn-upgrade
Variant: ci-4.22
RELEASE_IMAGE_LATEST: registry.ci.openshift.org/ocp/release:4.22.0-0.ci-2026-02-06-195709
RELEASE_IMAGE_INITIAL:registry.ci.openshift.org/ocp/release:4.22.0-0.ci-2026-02-05-195709
Example 2: Non-Upgrade Job (No from-tag)
For a non-upgrade job, the Upgrade From Tag field will be absent:
Job Name: periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn
Job Type: periodic
Build ID: 2019864414127132999
State: success
Payload Tag: 4.22.0-0.ci-2026-02-06-195709
Upgrade From Tag: (not present - not an upgrade job)
Architecture: amd64
Target: e2e-gcp-ovn
Variant: ci-4.22
Notes
- The
prowjob.jsonis uploaded to GCS by Prow's sidecar after the job completes (or is aborted). It may not be available for jobs still in progress. - The
managedFieldssection of the metadata can be omitted from output as it contains only field ownership bookkeeping. - The
metadata.labelssection is truncated for some fields (e.g., job name) due to Kubernetes label length limits. Always usemetadata.annotationsorspec.jobfor the full job name. - The GCS web proxy at
gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.comserves raw file content without authentication. - Duration can be calculated from
completionTime - startTime. - The
spec.decoration_config.timeoutfield shows the maximum allowed runtime for the job.
See Also
- Related Skill:
ci:prow-job-analysis(comprehensive Prow CI job failure analysis — test, install, disruption, and more)
Fetch Regression Details
This skill fetches detailed regression information from the Component Readiness API. It retrieves comprehensive data about a specific regression including test name, affected variants, release information, triage status, and related metadata.
When to Use This Skill
Use this skill when you need to retrieve complete details about a Component Readiness regression, such as:
- Analyzing a specific regression from Component Readiness
- Getting test name, release, and variant information for a regression
- Checking triage status and existing bug assignments
- Building regression analysis reports
- Automating regression workflows
Prerequisites
-
Network Access: Must be able to reach the Sippy API
- Check:
curl -s https://sippy.dptools.openshift.org/api/health - No authentication required for public API endpoints
- Check:
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Should be available on most systems
- Uses only standard library (no external dependencies)
- Check:
Implementation Steps
Step 1: Run the Python Script
The skill uses a Python script to fetch and parse regression data (including sample failed jobs):
# Path to the Python script
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"
# Fetch regression data in JSON format (includes failed jobs)
python3 "$script_path" <regression_id> --format json
# Or fetch as human-readable summary
python3 "$script_path" <regression_id> --format summary
Step 2: Parse the Output
The script outputs structured JSON data that can be further processed:
# Store JSON output in a variable for processing
regression_data=$(python3 "$script_path" 34446 --format json)
# Extract specific fields using jq if needed
test_name=$(echo "$regression_data" | jq -r '.test_name')
component=$(echo "$regression_data" | jq -r '.component')
jira_keys=$(echo "$regression_data" | jq -r '.triages[].jira_key')
# Extract failed job URLs for analysis
failed_job_urls=$(echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_url')
Step 3: Use the Data
The structured data includes all necessary regression details:
{
"regression_id": 34446,
"test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
"test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
"release": "4.22",
"base_release": "4.21",
"component": "Build",
"capability": "Builds",
"view": "4.22-main",
"opened": "2026-01-28T08:02:45.127153Z",
"closed": null,
"status": "open",
"last_failure": "2026-01-30T07:03:46Z",
"variants": [
"Architecture:amd64",
"Platform:metal",
"Network:ovn",
"Upgrade:none"
],
"max_failures": 18,
"triages": [
{
"id": 344,
"url": "https://redhat.atlassian.net/browse/OCPBUGS-74651",
"jira_key": "OCPBUGS-74651",
"type": "product",
"description": "RHCOS 10 metal ipv4 job permafailing on ipv6 network access",
"bug_id": 17817794,
"resolved": false,
"created_at": "2026-01-29T17:58:20.858051Z",
"updated_at": "2026-01-30T13:28:33.429963Z"
}
],
"analysis_status": -2,
"analysis_explanations": [
"Test is failing consistently across multiple job runs",
"Regression detected compared to baseline release"
],
"sample_failed_jobs": {
"periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": {
"pass_sequence": "FFFFFFFFFFFFFFFFFF",
"label_summary": {
"ErrImagePullQuay502BadGateway": 3
},
"failed_runs": [
{
"job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
"job_run_id": "2017184460591599616",
"start_time": "2026-01-30T10:33:47",
"test_failures": 3,
"job_labels": ["ErrImagePullQuay502BadGateway"]
}
]
},
"periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-techpreview": {
"pass_sequence": "SSFSSSSSSSS",
"label_summary": {},
"failed_runs": [
{
"job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
"job_run_id": "2016460830022832128",
"start_time": "2026-01-28T10:37:27",
"test_failures": 1,
"job_labels": []
}
]
}
},
"job_runs": [
{
"id": 2600,
"regression_id": 34446,
"prowjob_run_id": "2044678206610477056",
"prowjob_name": "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview",
"prowjob_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2044678206610477056",
"start_time": "2026-04-16T07:23:39Z",
"test_failures": 3
}
],
"test_details_url": "https://sippy.dptools.openshift.org/api/component_readiness/test_details?...",
"api_url": "https://sippy.dptools.openshift.org/api/component_readiness/regressions/34446"
}
Note:
analysis_status: Integer status code from the regression analysis. Negative numbers indicate problems, with lower numbers representing more severe issues.analysis_explanations: List of human-readable explanations describing the status of the regression.sample_failed_jobs: Dictionary keyed by job name. Each job contains:pass_sequence: Chronological success/fail pattern for this specific job (newest to oldest). "S" = successful run, "F" = failing run. Example: "FFFFFSSS" shows 5 recent failures, then 3 older successes.label_summary: Dictionary of{label: count}aggregated across all failed runs for this job. A label appearing in many/all failed runs is a strong signal that the labelled symptom is the root cause. Labels are human-defined: a team member wrote a regex that matches specific artifact content and named the symptom (e.g.,"ErrImagePullQuay502BadGateway"). If many or all failed runs share a label, prioritize investigating that symptom.failed_runs: List of failed runs for this job, sorted by start_time (newest first). Each run includes:test_failures: Total number of tests failing in the entire job run (not just the regressed test). High values (>10) indicate a mass failure run where the regressed test may be collateral damage rather than the primary problem.job_labels: List of symptom labels applied to this specific run. Labels are precise — they fire only when a human-defined regex matches job artifacts. An empty list means no known symptom was detected for this run.
label_bugs:{label: [Jira keys]}for labels seen insample_failed_jobs, from the public label catalog. On fetch failure,label_bugs_erroris set instead — treat it as unknown, not "no bugs".job_runs: Complete list of all job runs where the failure was observed throughout the entire life of the regression (not just the last reporting period). Sorted by start_time (newest first). Each entry contains:id: Unique ID for this job run recordregression_id: The regression this run belongs toprowjob_run_id: The Prow job run IDprowjob_name: The Prow job nameprowjob_url: Direct URL to the Prow job runstart_time: ISO 8601 timestamp when the job startedtest_failures: Integer — total number of test failures in this job run. High values (e.g., >10) indicate a mass failure run where the regressed test may just be caught up in a larger issue rather than being the primary problem- This data is useful for: determining when the regression started (earliest entries), how frequently it occurs, whether failures cluster in certain jobs, identifying mass failure runs where the regression may be collateral damage, and linking related regressions that share the same job runs. Sippy uses this data automatically in the
fetch_related_triagesAPI to find regressions with shared job runs.
Error Handling
The Python script handles common error cases automatically:
Case 1: Regression Not Found (404)
python3 fetch_regression_details.py 99999999
# Error: Regression ID 99999999 not found.
# Verify the regression ID exists in Component Readiness.
Case 2: Invalid Regression ID
python3 fetch_regression_details.py abc
# Error: Regression ID must be a positive integer, got 'abc'
Case 3: Network Error
python3 fetch_regression_details.py 34446
# Error: Failed to connect to Sippy API: [Errno -2] Name or service not known
# Check network connectivity and VPN settings.
Case 4: Missing Arguments
python3 fetch_regression_details.py
# Usage: fetch_regression_details.py <regression_id> [--format json|summary]
Exit Codes:
0: Success1: Error (invalid input, API error, network error, etc.)
API Response Schema
The API returns a JSON object with the following structure:
{
"id": 34446,
"view": "4.22-main",
"release": "4.22",
"base_release": "4.21",
"component": "Build",
"capability": "Builds",
"test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
"test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
"variants": [
"Upgrade:none",
"Architecture:amd64",
"Platform:metal",
"Network:ovn"
],
"opened": "2026-01-28T08:02:45.127153Z",
"closed": {
"Time": "0001-01-01T00:00:00Z",
"Valid": false
},
"triages": [
{
"id": 344,
"created_at": "2026-01-29T17:58:20.858051Z",
"updated_at": "2026-01-30T13:28:33.429963Z",
"url": "https://redhat.atlassian.net/browse/OCPBUGS-74651",
"description": "Description of the triage",
"type": "product",
"bug_id": 17817794,
"resolved": {
"Time": "0001-01-01T00:00:00Z",
"Valid": false
}
}
],
"last_failure": {
"Time": "2026-01-30T07:03:46Z",
"Valid": true
},
"max_failures": 18,
"job_runs": [
{
"id": 2600,
"regression_id": 34446,
"prowjob_run_id": "2044678206610477056",
"prowjob_name": "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview",
"prowjob_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
"start_time": "2026-04-16T07:23:39Z",
"test_failures": 3
}
],
"links": {
"self": "https://sippy.dptools.openshift.org/api/component_readiness/regressions/34446",
"test_details": "https://sippy.dptools.openshift.org/api/component_readiness/test_details?..."
}
}
Key Fields:
id: Regression IDtest_name: Full test name with signature groupsrelease: Sample release (e.g., "4.22")base_release: Baseline release for comparison (e.g., "4.21")component: Component owning the testcapability: Higher-level capability areavariants: Array of variant key:value pairs (e.g., "Platform:metal", "Network:ovn")opened: ISO 8601 timestamp when regression was first detectedclosed: Object withValidboolean andTime(valid if regression is closed)triages: Array of triage objects with JIRA URL, type, and resolution statusjob_runs: Array of all job runs where the failure was observed throughout the regression's entire life. Each entry includesprowjob_run_id,prowjob_name,prowjob_url,start_time, andtest_failures(total failures in the run — high values indicate mass failure runs where the regression may be collateral damage)links.test_details: Direct URL to Sippy test details page with sample failures
Examples
Example 1: Fetch Regression as JSON
# Fetch regression 34446 in JSON format (includes failed jobs)
python3 plugins/ci/skills/fetch-regression-details/fetch_regression_details.py 34446 --format json
Expected Output:
{
"regression_id": 34446,
"test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
"test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
"release": "4.22",
"base_release": "4.21",
"component": "Build",
"capability": "Builds",
"status": "open",
"variants": ["Architecture:amd64", "Platform:metal", "Network:ovn"],
"triages": [
{
"jira_key": "OCPBUGS-74651",
"type": "product",
"resolved": false
}
],
"analysis_status": -2,
"analysis_explanations": [
"Test is failing consistently across multiple job runs",
"Regression detected compared to baseline release"
],
"sample_failed_jobs": {
"periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": {
"pass_sequence": "FFFFFFFFFFFFFFFFFF",
"failed_runs": [
{
"job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
"job_run_id": "2017184460591599616",
"start_time": "2026-01-30T10:33:47"
}
]
}
}
}
Example 2: Fetch Regression as Human-Readable Summary
# Fetch regression 34446 as formatted summary
python3 plugins/ci/skills/fetch-regression-details/fetch_regression_details.py 34446 --format summary
Expected Output:
Regression #34446 Details:
============================================================
Test Name: [sig-builds][Feature:Builds] custom build with buildah...
Release: 4.22 (baseline: 4.21)
Component: Build
Status: Open (opened: 2026-01-28)
Max Failures: 19
Triages:
- OCPBUGS-74651 (product, active): Description of the issue
Sample Failed Jobs (19 runs):
- periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview
Run ID: 2017184460591599616
Started: 2026-01-30
URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
Example 3: Extract Failed Job URLs
# Get all failed job URLs for analysis
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"
data=$(python3 "$script_path" 34446 --format json)
# Extract failed job URLs
echo "$data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_url'
Expected Output:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2017184460591599616
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2017131608699572224
...
Output Format
The script supports two output formats:
JSON Format (--format json)
Returns structured JSON data with all regression fields:
{
"regression_id": 34446,
"test_name": "...",
"test_id": "...",
"release": "4.22",
"base_release": "4.21",
"component": "Build",
"capability": "Builds",
"view": "4.22-main",
"opened": "2026-01-28T08:02:45.127153Z",
"closed": null,
"status": "open",
"last_failure": "2026-01-30T07:03:46Z",
"variants": [...],
"max_failures": 19,
"triages": [...],
"analysis_status": -2,
"analysis_explanations": ["...", "..."],
"sample_failed_jobs": {...},
"test_details_url": "...",
"api_url": "..."
}
Summary Format (--format summary)
Returns human-readable formatted summary:
Regression #34446 Details:
============================================================
Test Name: [sig-builds][Feature:Builds] custom build with buildah...
Release: 4.22 (baseline: 4.21)
Component: Build
Capability: Builds
Status: Open (opened: 2026-01-28)
Last Failure: 2026-01-30
Max Failures: 19
Analysis Status: -2
(Negative status indicates a problem - lower is more severe)
Analysis Explanations:
- Test is failing consistently across multiple job runs
- Regression detected compared to baseline release
Affected Variants:
- Architecture:amd64
- Platform:metal
- Network:ovn
- Upgrade:none
Triages:
- OCPBUGS-74651 (product, active): Description...
Test Details: https://sippy.dptools.openshift.org/...
API URL: https://sippy.dptools.openshift.org/api/...
Notes
- The Python script uses only standard library modules (no external dependencies)
- The API does not require authentication for read-only access
- Regression IDs are persistent and do not change
- The
test_details_urlprovides links to sample job failures for further analysis - Closed regressions will have
status: "closed"and a non-nullclosedtimestamp - The
variantsarray shows all platform/topology combinations where the test is failing - Default output format is JSON; use
--format summaryfor human-readable output analysis_statusis an integer where negative values indicate problems (lower = more severe)analysis_explanationsprovides human-readable context for the analysis statussample_failed_jobsis a dictionary keyed by job name, containing only jobs with at least one failed run- Each job includes a
pass_sequencestring (newest to oldest) with "S" for successful runs and "F" for failed runs - Each job includes a
label_summarydict aggregatingjob_labelsacross all failed runs for that job. This is the first place to look for a shared root cause: if most or all failed runs share a label, that symptom is almost certainly the root cause. Labels are human-defined and precise — they only fire when a regex written by a team member matches specific artifact content. - Each failed run includes
test_failures(total tests failing in the job run) andjob_labels(symptom labels for that specific run). Usetest_failures > 10as a signal for mass failure runs where the regressed test may be collateral damage. job_runscontains the complete history of all job runs where the failure was observed across the regression's entire lifetime, sorted by start_time (newest first). Use this to determine when failures started, how common they were, and which jobs are most affected. Thetest_failuresfield shows the total failure count in each run — high values (e.g., >10) indicate mass failure runs where the regression may just be caught up in a larger issue rather than being the primary problem
See Also
- Component Readiness API Documentation: https://sippy.dptools.openshift.org/api/docs
- Related Skill:
ci:prow-job-analysis(comprehensive Prow CI job failure analysis) - Related Command:
/ci:analyze-regression(uses this skill)
Fetch Related Triages
Queries the Sippy API to find existing triage records and untriaged regressions that may be related to a given Component Readiness regression. The API matches based on similarly named tests and shared job runs (using the complete job_runs history tracked across each regression's entire lifetime).
When to Use This Skill
- During regression analysis to find existing triages that could cover the current regression
- To find untriaged regressions that share the same root cause and should be triaged together
- To avoid filing duplicate bugs when a related triage already exists
API Endpoint
GET https://sippy.dptools.openshift.org/api/component_readiness/regressions/{id}/matches
No authentication required (uses production Sippy URL).
Usage
# Fetch all related triages (default: all confidence levels)
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479
# Filter to high confidence matches only
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479 --min-confidence 5
# Human-readable summary
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479 --format summary
Arguments
regression_id(required): The Component Readiness regression ID--min-confidence(optional): Minimum confidence level to include (1-10, default: 1)--format(optional): Output format —json(default) orsummary
Confidence Levels
The API assigns a confidence level (1-10) to each match:
- 10: High confidence — the triage's regressions include the same test (exact test_id match) or very closely related tests with shared failure patterns
- 5: Medium confidence — similarly named tests matched by edit distance, suggesting related tests in the same area
- 2: Low confidence — regressions share the same job runs (identified via the
job_runshistory tracked across each regression's lifetime), which may indicate a shared root cause but could be coincidental in mass failure scenarios
Output Format
JSON output
{
"success": true,
"regression_id": 35479,
"triaged_matches": [
{
"triage_id": 370,
"triage_ui_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/370",
"jira_key": "OCPBUGS-76612",
"jira_url": "https://redhat.atlassian.net/browse/OCPBUGS-76612",
"jira_status": "New",
"jira_summary": "Component Readiness: [Networking / ovn-kubernetes] ...",
"triage_type": "test",
"triage_description": "...",
"confidence_level": 10,
"regressions_on_triage": [
{
"id": 35217,
"test_name": "...",
"test_id": "...",
"component": "Networking / ovn-kubernetes",
"variants": ["..."],
"opened": "...",
"closed": null,
"triages": null,
"last_failure": "..."
}
]
}
],
"untriaged_regressions": [
{
"id": 35480,
"test_name": "...",
"test_id": "...",
"component": "...",
"variants": ["..."],
"opened": "...",
"closed": null,
"triages": null,
"last_failure": "...",
"match_reason": "similarly_named_test",
"edit_distance": 1
}
]
}
Key fields
- triaged_matches: Existing triage records that look related, sorted by confidence (highest first). Each includes the JIRA bug info and the regressions already on the triage.
- untriaged_regressions: Open regressions that are not yet triaged but appear related. These are candidates to be triaged together with the current regression. Each includes a
match_reasonfield:similarly_named_test: Matched by test name edit distance (lower edit_distance = more similar)same_last_failure: Matched by sharing the same job runs (identified via thejob_runshistory tracked across each regression's lifetime)
How the API Matches Work
The API uses two strategies to find related triages:
-
Similarly named tests: Compares the regression's test name against test names on existing triages using edit distance. An edit distance of 0 means the same test name (different variants); 1-2 means very similar test names (e.g., L2 vs L3 variant of the same test).
-
Shared job runs: Finds regressions that were observed in the same job runs. The regression data now includes a complete
job_runslist covering all runs where the failure was observed throughout the regression's entire life (not just the last reporting period). Sippy uses this data to find other regressions that failed in the same Prow job runs, which is a strong signal they share the same root cause (e.g., a mass CI infrastructure failure or a broad product regression). Regressions with hightest_failurescounts in their shared runs are more likely to be collateral damage from a larger issue.
Notes
- Uses the production Sippy URL (no port-forward needed)
- No authentication required
- The script deduplicates untriaged regressions by ID
- Only open, untriaged regressions are included in
untriaged_regressions - Triaged matches are sorted by confidence level (highest first)
- The 60-second timeout handles the potentially large response
Fetch Releases
This skill fetches the list of available OpenShift releases from the Sippy API. It can return all OCP releases or just the latest version.
When to Use This Skill
Use this skill when you need to:
- Determine the latest OCP release version
- Get a list of all available OCP releases
- Provide a default release version when the user hasn't specified one
Prerequisites
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
-
Network Access: Must be able to reach the public Sippy API
- Check:
curl -s https://sippy.dptools.openshift.org/api/releases | head -c 100
- Check:
Implementation Steps
Step 1: Get the Latest Release
To get just the latest OCP release version:
release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)
echo "$release"
# Output: 4.22
Step 2: List All Releases
To list all available OCP releases:
# JSON format (includes GA dates)
python3 plugins/ci/skills/fetch-releases/fetch_releases.py --format json
# Simple list, one per line
python3 plugins/ci/skills/fetch-releases/fetch_releases.py --format list
Step 3: Use with Other Skills
The primary use case is providing a default release to other skills:
# Determine release
release="${user_specified_release:-$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)}"
# Use with fetch-test-report
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test_name>" --release "$release"
Output Format
--latest
Prints a single release version string:
4.22
--format json (default)
[
{"release": "4.22"},
{"release": "4.21", "ga": "2025-06-17T00:00:00Z"},
{"release": "4.20", "ga": "2025-10-21T00:00:00Z"},
{"release": "4.19", "ga": "2025-06-17T00:00:00Z"}
]
Releases without a ga field are still in development.
--format list
4.22
4.21
4.20
4.19
4.18
Error Handling
- Network Error: If the Sippy API is unreachable, the script prints an error to stderr and exits with code 1.
- No Releases: If no standard OCP releases are found, exits with code 1.
Exit Codes:
0: Success1: Error
Notes
- Uses the public Sippy API at
https://sippy.dptools.openshift.org/api/releases(no port-forward needed) - Filters out non-standard releases (okd, aro, presubmits, etc.) — only returns releases matching
X.Yformat - The API returns releases in order with the latest first
- This skill is based on the same API as the
teamsplugin'sget-release-datesskill
See Also
- Related Skill:
fetch-test-report(uses release to query test data) - Related Skill (teams plugin):
get-release-dates(fetches detailed release dates and metadata)
Fetch Test Report
This skill fetches a report for an OpenShift CI test by its full name using the Sippy tests API. It returns test metadata including the BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous reporting periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name, which can help surface bugs that have been filed but not yet triaged in Component Readiness.
When to Use This Skill
Use this skill when you need to:
- Look up a test's Component Readiness / BigQuery test ID (
test_id) - Check current and previous pass rates for a test
- Find the Jira component associated with a test
- Determine if a test is flaking, failing, or stable
- Get run counts and failure/flake breakdowns for a test
- Check if there are open Jira bugs mentioning this test (may not yet be triaged in Component Readiness)
- See a per-variant breakdown of pass rates to identify if the test fails only in certain job types
Prerequisites
-
Network Access: The Sippy API must be accessible at
https://sippy.dptools.openshift.org- No authentication required
- Check:
curl -s https://sippy.dptools.openshift.org/api/health?release=4.22
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
Implementation Steps
Step 1: Determine the Release
If the user did not specify a release, use the fetch-releases skill to determine the latest OCP release:
release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)
If the user specified a release, use that directly.
Step 2: Run the Python Script
The skill uses a Python script to query the Sippy tests API:
# Path to the Python script
script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"
# Fetch test report in JSON format (collapsed — one row for the test across all variants)
python3 "$script_path" "<test_name>" --release "$release" --format json
# Or fetch as human-readable summary
python3 "$script_path" "<test_name>" --release "$release" --format summary
# Fetch per-variant breakdown (one row per variant combo — useful for identifying
# if the test is failing only in certain job types, e.g., aws vs metal, upgrade vs new-install)
python3 "$script_path" "<test_name>" --release "$release" --no-collapse --format json
Step 3: Parse the Output
The script outputs a JSON array (typically with one element for an exact name match):
# Store JSON output in a variable for processing
test_data=$(python3 "$script_path" "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" --format json)
# Extract specific fields using jq
test_id=$(echo "$test_data" | jq -r '.[0].test_id')
component=$(echo "$test_data" | jq -r '.[0].jira_component')
pass_rate=$(echo "$test_data" | jq -r '.[0].current_pass_percentage')
runs=$(echo "$test_data" | jq -r '.[0].current_runs')
API Details
Endpoint
GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}
GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}&collapse=false
When collapse=false is specified, the API returns one row per variant combination instead of a single collapsed row. Each row includes a variants array showing the specific variant combo (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]). This helps identify if the test is failing in certain types of jobs and not others.
Filter Format
The filter parameter is a JSON object using Sippy's standard filter syntax:
{
"items": [
{
"columnField": "name",
"operatorValue": "equals",
"value": "<test_name>"
}
]
}
Response Schema
The API returns a JSON array of test objects:
[
{
"id": 5987,
"test_id": "openshift-tests:e8f7491a505095b6356dfd8d4cf7218b",
"name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
"suite_name": "openshift-tests",
"variants": null,
"jira_component": "kube-apiserver",
"jira_component_id": "12367637",
"current_successes": 3425,
"current_failures": 22,
"current_flakes": 0,
"current_pass_percentage": 99.36,
"current_failure_percentage": 0.64,
"current_flake_percentage": 0,
"current_working_percentage": 99.36,
"current_runs": 3447,
"previous_successes": 2570,
"previous_failures": 43,
"previous_flakes": 0,
"previous_pass_percentage": 98.35,
"previous_failure_percentage": 1.65,
"previous_flake_percentage": 0,
"previous_working_percentage": 98.35,
"previous_runs": 2613,
"net_failure_improvement": 1.01,
"net_flake_improvement": 0,
"net_working_improvement": 1.01,
"net_improvement": 1.01,
"tags": null,
"open_bugs": 0
}
]
Key Fields:
id: Sippy PostgreSQL internal ID (less commonly used)test_id: BigQuery / Component Readiness test ID (format:suite:hash). This is the ID used with modern skills likefetch-regression-detailsandfetch-test-runs.name: Full test namesuite_name: Test suite (e.g.,openshift-tests)variants: Array of variant strings (only present when--no-collapse/collapse=falseis used). Shows the specific variant combo for this row (e.g.,["aws", "ovn", "amd64", "upgrade-micro"]).jira_component: Associated OCPBUGS Jira componentjira_component_id: Jira component numeric IDcurrent_*: Metrics for the current 7-day reporting periodprevious_*: Metrics for the 7 days before the current periodnet_working_improvement: Change in working percentage (positive = improving)open_bugs: Number of open Jira bugs that mention this test by name. This can surface bugs that have been filed but not yet triaged in Component Readiness. Useful for finding existing work before filing a duplicate.
Error Handling
Case 1: Sippy Not Reachable
python3 fetch_test_report.py "[sig-api-machinery] ..."
# Error: Failed to connect to Sippy API: [Errno 61] Connection refused
# Check network connectivity to sippy.dptools.openshift.org.
Case 2: No Test Found
If the test name doesn't match any test, the script returns an empty JSON array [] or "No tests found matching the given name." in summary mode.
Case 3: Missing Arguments
python3 fetch_test_report.py
# usage: fetch_test_report.py [-h] [--release RELEASE] [--format {json,summary}] test_name
# fetch_test_report.py: error: the following arguments are required: test_name
Exit Codes:
0: Success1: Error (API error, network error, etc.)
Examples
Example 1: Fetch Test Report in JSON Format
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
"[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
--release 4.22 --format json
Example 2: Get a Human-Readable Summary
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
"[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
--release 4.22 --format summary
Expected Output:
Test: [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]
Test ID: openshift-tests:abc123def456
Suite: openshift-tests
Jira Component: kube-apiserver
Open Bugs: 0
Current Period (last 7 days):
Runs: 3447
Pass Rate: 99.36%
Failures: 22 (0.64%)
Flakes: 0 (0.00%)
Previous Period (7 days before current):
Runs: 2613
Pass Rate: 98.35%
Failures: 43 (1.65%)
Flakes: 0 (0.00%)
Trend: improved (+1.01%)
Example 3: Fetch Per-Variant Breakdown
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
"[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
--release 4.22 --no-collapse --format summary
This returns one row per variant combination, showing which specific job types the test is passing or failing in. Useful for regression analysis to determine if the failure is platform-specific, upgrade-specific, etc.
Example 4: Fetch Test Report for a Different Release
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
"[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
--release 4.21 --format json
Notes
- The test name must be an exact match — use the full test name including sig prefix and suite tags
- The API is accessed via the production Sippy URL; no authentication required
- If
--releaseis not specified, use thefetch-releasesskill to determine the latest release current_*fields cover the last 7 days;previous_*fields cover the 7 days before thatcurrent_working_percentage= pass rate + flake rate (tests that ultimately passed, possibly after retries)- The
test_idfield is the one used in Component Readiness URLs and with thefetch-regression-detailsandfetch-test-runsskills - An empty result (
[]) means no test matched — double-check the exact test name spelling
See Also
- Related Skill:
fetch-releases(determines the latest OCP release) - Related Skill:
fetch-regression-details(usestest_idto fetch regression details) - Related Skill:
fetch-test-runs(uses test name to fetch individual test run results) - Related Command:
/ci:fetch-test-report(command that invokes this skill) - Related Command:
/ci:analyze-regression(full regression analysis workflow)
Fetch Test Runs
This skill fetches test runs from the Sippy API. It can return both failed and successful test runs, including JUnit output for AI-based analysis.
When to Use This Skill
Use this skill when you need to:
- Fetch test run data for a specific test across all jobs
- Get raw test failure outputs for AI-based similarity analysis
- Compare error messages across runs to determine if they share the same root cause
- Include successful runs in addition to failures (optional)
- Filter runs by job name substrings (e.g., only GCP techpreview jobs)
- Access JUnit test output for debugging and investigation
Prerequisites
-
Network Access: Must be able to reach the Sippy test runs API
- No authentication required
- Check:
curl -s https://sippy.dptools.openshift.org/api/tests/v2/runs?test_id=test
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
-
Input Data: Requires test_id (job_run_ids are optional)
- Get from
fetch-regression-detailsskill output test_id: Found in regression data (e.g., "openshift-tests:71c053c318c11cfc47717b9cf711c326")job_run_ids: Optional - extracted fromsample_failed_jobs[].failed_runs[].job_run_id
- Get from
Implementation Steps
Step 1: Run the Python Script
# Path to the Python script
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
# Fetch all test runs (failures only, by default)
python3 "$script_path" "$test_id" --format json
# Include successful runs as well
python3 "$script_path" "$test_id" --include-success --format json
# Filter to a specific Prow job (exact name works as substring of itself)
python3 "$script_path" "$test_id" --job-contains "periodic-ci-openshift-release-..." --format json
# Filter by multiple substrings (AND logic, case-insensitive, server-side)
python3 "$script_path" "$test_id" --job-contains gcp --job-contains techpreview --format json
# Filter to specific job run IDs (backward compatible with analyze-regression)
python3 "$script_path" "$test_id" "$job_run_ids" --format json
# Get human-readable summary
python3 "$script_path" "$test_id" --format summary
Arguments:
test_id: Required test identifier (e.g., "openshift-tests:abc123")job_run_ids: Optional comma-separated list of Prow job run IDs to filter by
Options:
--include-success: Include successful test runs (default: failures only)--job-contains <name>: Filter by job name substring (server-side, case-insensitive). Repeatable for AND logic — all substrings must appear in the job name. Full job names also work since they are substrings of themselves. E.g.,--job-contains gcp --job-contains techpreviewmatches jobs containing both "gcp" and "techpreview".--start-days-ago <days>: Number of days to look back (default API is 7 days)--exclude-output: Strip theoutputfield from each run to reduce response size. Use when you only need pass/fail status (e.g., regression start analysis). Significantly reduces JSON output size for large result sets.--output <path>: Write output to a file instead of stdout. Use when fetching large result sets (e.g.,--include-success --start-days-ago 28) that may exceed stdout buffer limits. The script prints a summary line to stderr confirming the write.--format json|summary: Output format (default: json)
Step 2: Prepare Input Data (for analyze-regression)
When used with regression analysis, extract required data from regression details:
# Assuming you have regression_data from fetch-regression-details skill
test_id=$(echo "$regression_data" | jq -r '.test_id')
# Collect all job_run_ids from sample_failed_jobs
# This creates a comma-separated list of all failed job run IDs
job_run_ids=$(echo "$regression_data" | jq -r '
.sample_failed_jobs
| to_entries[]
| .value.failed_runs[]
| .job_run_id
' | tr '\n' ',' | sed 's/,$//')
echo "Test ID: $test_id"
echo "Job Run IDs: $job_run_ids"
Step 3: Parse the Output
The script outputs structured JSON data:
# Store JSON output for processing
output_data=$(python3 "$script_path" "$test_id" --format json)
# Check if fetch was successful
success=$(echo "$output_data" | jq -r '.success')
if [ "$success" = "true" ]; then
# Extract runs array
runs=$(echo "$output_data" | jq -r '.runs')
# The runs array contains objects with: url, output, test_name, success
# The AI command will analyze these runs for similarity
echo "Fetched $(echo "$runs" | jq 'length') runs"
else
# Handle error case
error=$(echo "$output_data" | jq -r '.error')
echo "Error: $error"
echo "Test runs API may not be available"
fi
API Response Schema
The Sippy API returns a JSON array of test run objects:
[
{
"url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-aws-ovn-techpreview/2016123858595090432",
"output": "fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access: /apis/stable.e2e-validating-admission-policy-1181/: the server could not find the requested resource",
"test_name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
"success": false,
"failed_tests": 3
},
{
"url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
"output": "",
"test_name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
"success": true,
"failed_tests": 0
}
]
Script Output Format
The script supports two output formats:
JSON Format (--format json)
Returns structured JSON with raw runs:
{
"success": true,
"test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
"requested_job_runs": 0,
"include_success": false,
"job_name_filters": ["gcp", "techpreview"],
"runs": [
{
"url": "https://prow.ci.openshift.org/...",
"output": "fail [...]: error message",
"test_name": "[sig-api-machinery] test name",
"success": false,
"failed_tests": 3
}
],
"api_url": "https://sippy.dptools.openshift.org/api/tests/v2/runs?test_id=...&prowjob_name=gcp&prowjob_name=techpreview"
}
Field Descriptions:
- success: Boolean indicating if the API call succeeded
- test_id: The test identifier that was queried
- requested_job_runs: Number of job run IDs requested (0 if none specified)
- include_success: Whether successful runs were requested
- job_name_filters: List of job name substrings used for server-side filtering (null if not specified)
- runs: Raw array of test run objects from Sippy API
- url: Prow job URL for this specific run
- output: The actual JUnit test failure output text (empty for successes)
- test_name: Full test name
- success: Boolean indicating if this run passed
- failed_tests: Count of all tests that failed in this job run. If
failed_tests > 10, this is a mass failure job — the test may be caught up in a larger issue (e.g., infrastructure failure, installer failure) that needs further investigation to root cause. When many runs show mass failures, the regression may not be caused by a change specific to this test.
- api_url: The API URL that was called
Error Response (when success is false):
{
"success": false,
"error": "Failed to connect to test runs API: Connection refused",
"test_id": "openshift-tests:abc123",
"requested_job_runs": 0,
"include_success": false,
"job_name_filters": null
}
Summary Format (--format summary)
Returns human-readable formatted output with sample runs:
Test Runs
============================================================
Test ID: openshift-tests:71c053c318c11cfc47717b9cf711c326
Job Contains: ['gcp', 'techpreview']
Include Successes: False
Runs Fetched: 18
Successes: 0, Failures: 18
Mass Failure Runs (>10 test failures in job): 4 of 18
⚠ These runs had many other test failures — this test may be caught up in a larger issue.
Sample Runs:
1. [FAIL] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
Failed Tests in Job: 3
Output: fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access...
2. [FAIL] [MASS FAILURE] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
Failed Tests in Job: 47
Output: fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access...
3. [PASS] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
... and 15 more runs
Error Handling
Case 1: API Not Available
python3 fetch_test_runs.py "openshift-tests:abc"
Output (JSON format):
{
"success": false,
"error": "Failed to connect to test runs API: Connection refused.",
"test_id": "openshift-tests:abc",
"requested_job_runs": 0,
"include_success": false,
"job_name_filters": null
}
Output (summary format):
Test Runs - FETCH FAILED
============================================================
Error: Failed to connect to test runs API: Connection refused.
The test runs API may not be available.
Case 2: No Runs Returned
If the API returns an empty array:
{
"success": true,
"test_id": "openshift-tests:abc",
"requested_job_runs": 0,
"include_success": false,
"job_name_filters": null,
"runs": []
}
Case 3: Invalid Arguments
python3 fetch_test_runs.py
Output:
Usage: fetch_test_runs.py <test_id> [job_run_ids] [options]
Arguments:
test_id Test identifier (e.g., 'openshift-tests:abc123')
job_run_ids Optional comma-separated list of Prow job run IDs
Options:
--include-success Include successful test runs (default: failures only)
--job-contains Filter by job name substring (repeatable for AND logic)
--exclude-output Strip output text from runs to reduce response size
--output <path> Write output to file instead of stdout
--format json|summary Output format (default: json)
Exit Codes:
0: Success1: Error (invalid input, API error, network error, etc.)
Examples
Example 1: Fetch All Failures for a Test
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
python3 "$script_path" "openshift-tests:bb3a7d828630760296ef203c5cacf708" --format json
Example 2: Fetch All Runs Including Successes
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
python3 "$script_path" "openshift-tests:bb3a7d828630760296ef203c5cacf708" --include-success --format json
Example 3: Fetch Specific Job Runs (Backward Compatible)
Used by analyze-regression command:
# Assume regression_data is already fetched
test_id=$(echo "$regression_data" | jq -r '.test_id')
job_run_ids=$(echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_run_id' | tr '\n' ',' | sed 's/,$//')
# Fetch outputs for specific job runs
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
output_data=$(python3 "$script_path" "$test_id" "$job_run_ids" --format json)
# Check success
if [ "$(echo "$output_data" | jq -r '.success')" = "true" ]; then
echo "Successfully fetched runs"
fi
Example 4: Get Summary Report
python3 plugins/ci/skills/fetch-test-runs/fetch_test_runs.py \
"openshift-tests:71c053c318c11cfc47717b9cf711c326" \
--format summary
Example 5: Extract Output Messages for AI Analysis
# Fetch runs
output_data=$(python3 "$script_path" "$test_id" --format json)
# Extract all failure output messages
if [ "$(echo "$output_data" | jq -r '.success')" = "true" ]; then
# Get all output texts from failed runs
echo "$output_data" | jq -r '.runs[] | select(.success == false) | .output'
# AI command will analyze these for:
# - Similarity/consistency
# - Common error patterns
# - File references and API paths
# - Root cause determination
fi
Example 6: Filter Runs by Job Name Substrings
Filter to runs from jobs matching multiple criteria (e.g., GCP + techpreview):
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
# Get only GCP techpreview runs (both substrings must match, server-side)
python3 "$script_path" "openshift-tests:abc123" --include-success \
--job-contains gcp --job-contains techpreview \
--start-days-ago 28 --format json
# Get only metal upgrade runs
python3 "$script_path" "openshift-tests:abc123" \
--job-contains metal --job-contains upgrade \
--format summary
# Full job name also works (it's a substring of itself)
python3 "$script_path" "openshift-tests:abc123" \
--job-contains "periodic-ci-openshift-release-master-nightly-4.22-e2e-gcp-ovn-techpreview" \
--format json
Example 7: Determine Regression Start Date
Used by analyze-regression command to find when failures began:
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
# Get the job with the most failures
most_failed_job="periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn"
# Fetch all runs (including successes) for this specific job, going back 28 days
job_history=$(python3 "$script_path" "$test_id" \
--include-success \
--job-contains "$most_failed_job" \
--start-days-ago 28 \
--format json)
# Analyze the run history
if [ "$(echo "$job_history" | jq -r '.success')" = "true" ]; then
# Runs are returned newest to oldest
# Iterate to find where failures started
echo "$job_history" | jq -r '.runs[] | "\(.success) \(.url)"'
# Look for transition from passing to failing
# Find the first failure that's part of the current regression
fi
Notes
- The script uses only Python standard library - no external dependencies required
- Uses the production Sippy URL
- Handles API unavailability by returning a JSON error response with
success: falseand a descriptive error message - Returns raw outputs for AI-based interpretation and similarity analysis
- Job run IDs are optional - can fetch all runs for a test
--include-successallows analyzing both passing and failing runs--job-containsfilters results server-side using case-insensitive substring matching. Repeat for AND logic (all substrings must appear in the job name). Full job names work too since they are substrings of themselves.--start-days-agoallows looking back further than the default 7 days (e.g.,--start-days-ago 28)- Combine
--include-success,--job-contains, and--start-days-agoto get full test history for regression analysis - Backward compatible with analyze-regression command (accepts job_run_ids)
- Summary format shows first 5 runs only, to keep output manageable
- Runs are returned in order from most recent to least recent
- Each run includes a
failed_testscount — the total number of tests that failed in that job. Iffailed_tests > 10, it indicates a mass failure job where many tests failed together, suggesting the test may be caught up in a larger issue (infrastructure failure, installer failure, etc.) rather than a regression specific to this test
See Also
- Related Skill:
fetch-regression-details(provides test_id and job_run_ids) - Related Command:
/ci:analyze-regression(uses this skill for failure analysis)
Fetch Tide PR Status
CLI equivalent of the Prow PR Status page. Focuses on labels and required jobs — the most common merge blockers. Tide also checks other things (author, milestone) which may appear in the tide verdict but are not broken out here.
Only outputs open, non-draft PRs.
Prerequisites
ghCLI authenticatedpython3withpyyaml(pip install pyyaml)
Usage
# All open PRs by an author
python3 "${SKILL_DIR}/fetch_tide_pr_status.py" openshift/cluster-monitoring-operator machine424
# Specific PRs
python3 "${SKILL_DIR}/fetch_tide_pr_status.py" openshift/cluster-monitoring-operator 3057,3044
Output
JSON array, one object per PR:
[
{
"number": 3044,
"title": "OCPBUGS-105305: [release-4.21] wrap ...",
"author": "machine424",
"url": "https://github.com/openshift/cluster-monitoring-operator/pull/3044",
"branch": "release-4.21",
"state": "open",
"mergeable": true,
"tide": "Not mergeable. Needs jira/valid-bug label.",
"blockers": [
"missing required label: jira/valid-bug",
"has forbidden label: jira/invalid-bug",
"job not passing: ci/prow/e2e-aws-ovn (running)"
],
"labels": {
"met": false,
"required": {
"have": ["lgtm", "approved"],
"missing": ["jira/valid-bug"]
},
"forbidden": {
"have": ["jira/invalid-bug"],
"clear": ["needs-rebase", "do-not-merge/hold"]
}
},
"required_jobs": [
{"name": "ci/prow/unit", "state": "success"},
{"name": "ci/prow/e2e-aws-ovn", "state": "running", "description": "Job triggered."}
]
}
]
tide — Tide's own verdict. blockers — what's preventing merge. required_jobs — non-optional jobs from openshift/release config; if the config could not be fetched, contains a single {"error": "..."} entry instead of jobs (not reflected in blockers).
Data Sources
https://prow.ci.openshift.org/tide.js— required/forbidden labels (public)gh api repos/{o}/{r}/pulls/{n}— PR metadatagh api repos/{o}/{r}/commits/{sha}/status— job statuses + Tide contextraw.githubusercontent.com/openshift/release/master/ci-operator/jobs/...— presubmit config (required vs optional)
Generate OKD/SCOS ci-operator configuration YAML files for the openshift/release repository from ART image metadata using the bundled Python generator.
Arguments
Parse from: $ARGUMENTS
- First argument: path to the ocp-build-data directory (must contain
group.yml,streams.yml,images/). - Second argument: OKD version string (e.g.
4.18,5.0). --output-dir <path>: Directory to write ci-operator configs (default for this skill:./okd-ci-configs). In an openshift/release checkout, use itsci-operator/config/directory.--github-token <token>: Optional GitHub token for downloading upstream Dockerfiles. Requests are unauthenticated when omitted.--dry-run: Print generated YAML without writing files. Upstream Dockerfiles are still downloaded.
If no arguments are provided, look for ocp-build-data at ./tmp-dir/ocp-build-data/ or ../ocp-build-data/, and derive the OKD version from group.yml vars (MAJOR.MINOR). Use build data for the requested release; the OKD version argument controls the destination imagestream and does not change the metadata's branches.
Execution
1. Locate the bundled helper
Set SKILL_DIR to the absolute directory containing this SKILL.md. The entry point is generate_okd_ci_configs.py alongside this file.
- In this repository, the skill directory is
.claude/skills/generate-okd-ci-operator-configs-with-helper-script/. - In the ai-helpers CI plugin, it is
${CLAUDE_PLUGIN_ROOT}/skills/generate-okd-ci-operator-configs/.
The entry point imports the sibling okd_ci_configs package. Keep the whole skill directory together when installing or copying it. Running the entry point by its absolute path works from any working directory without setting PYTHONPATH or installing the helper package.
Python 3, PyYAML, and requests are required. Use the project's virtual environment if available. Check dependencies with:
python3 -c 'import yaml, requests'
2. Clone ocp-build-data (if not already present)
If the ocp-build-data directory is not provided or does not exist locally, clone it from GitHub:
git clone https://github.com/openshift-eng/ocp-build-data.git --branch openshift-$OKD_VERSION
The branch should match the target OKD version (e.g. openshift-5.0 for OKD 5.0, openshift-4.18 for OKD 4.18).
3. Validate inputs and run the generator
Verify the build-data directory contains group.yml, streams.yml, and images/. Resolve the arguments into OCP_BUILD_DATA_PATH, OKD_VERSION, and OUTPUT_DIR; pass ./okd-ci-configs explicitly when the skill's output default is used, since the Python CLI requires --output-dir.
python3 "$SKILL_DIR/generate_okd_ci_configs.py" \
--ocp-build-data "$OCP_BUILD_DATA_PATH" \
--okd-version "$OKD_VERSION" \
--output-dir "$OUTPUT_DIR"
Append --github-token with the provided token or --dry-run when requested.
The generator reads and substitutes ART metadata, resolves payload dependencies, downloads upstream Dockerfiles, and writes one config per (org, repo, branch) to:
{output-dir}/{org}/{repo}/{org}-{repo}-{branch}__okd-scos.yaml
It handles okd_alignment overrides for parent images, Dockerfiles, source paths, context directories, build arguments, build roots, and RPM repository injection.
4. Report results
Report the generated config count, output paths, skipped images with reasons, and warnings or errors. A successful invocation can still skip images or refuse incomplete repository configs; include those limitations in the report.
For configs generated into an openshift/release checkout, the follow-up generation commands are make ci-operator-configs and make jobs.
Helper layout
The entry point owns argument parsing and input validation. Read the relevant module when diagnosing or changing a specific part of generation:
| Module | Responsibility |
|---|---|
| helpers.py | Git URL normalization, public upstream mappings, variable substitution, and CI image coordinates |
| metadata.py | Load image metadata, resolve OKD tags and pullspecs, and find payload dependencies |
| dockerfiles.py | Download upstream Dockerfiles and parse FROM instructions |
| ci_operator.py | Build image/repository configurations and serialize ci-operator YAML |
| generator.py | Orchestrate loading, reconciliation, generation, and result reporting |
When copying this skill into ai-helpers, use plugins/ci/skills/generate-okd-ci-operator-configs/ and set the frontmatter name to generate-okd-ci-operator-configs to match that directory. The helper package needs no plugin manifest registration.
Key resolution rules
- Streams resolve in order:
okd.resolve_as.image,upstream_image, thenimage. - Images with
okd_alignment.resolve_as.streamor.imageuse the resolved pullspec and are not built by this generator. - Otherwise, images use
registry.ci.openshift.org/origin/scos-{version}:{tag}. The tag comes fromokd_alignment.tag_name, orpayload_name/namewith theose-prefix stripped. - Public upstream URLs use the longest matching
public_upstreamsmapping ingroup.yml. A mapping'spublic_branchoverrides the source branch; otherwise the source branch is used, withmainas the fallback when absent. - Images explicitly disabled for OKD, lacking usable source or parent metadata, or unnecessary for the payload are skipped. Dockerfile download failures and parent-count mismatches also skip images.
List Symptoms
Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags that categorize failures — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.
This skill lists, searches, and fetches symptoms and labels using the public (read-only, no auth) Sippy API.
When to Use This Skill
Use this skill when you need to:
- Browse the catalog of known CI failure signatures (symptoms)
- Check whether a failure pattern already has a symptom before creating a new one (avoid duplicates)
- Look up what a label means, including its title, explanation, and associated Jira issues
- Find which symptoms apply a given label
- Inspect a single symptom or label by its ID
Prerequisites
-
Network Access: The Sippy API must be accessible at
https://sippy.dptools.openshift.org- No authentication required
- Check:
curl -s https://sippy.dptools.openshift.org/api/jobs/labels | head -c 200
-
Python 3: Python 3.6 or later
- Check:
python3 --version - Uses only standard library (no external dependencies)
- Check:
Implementation Steps
Invoke the script with the flags that match the user's question:
script_path="plugins/ci/skills/list-symptoms/list_symptoms.py"
# List all symptoms (JSON by default)
python3 "$script_path"
# List all symptoms as a human-readable summary
python3 "$script_path" --format summary
# Search symptoms by text (matches id, summary, and match_string, case-insensitive)
python3 "$script_path" --search "credentials" --format summary
# Find all symptoms that apply a given label
python3 "$script_path" --label AWSAuthFailure --format summary
# Fetch a single symptom by ID
python3 "$script_path" --id AWSCouldNotValidateAccessCredentials
# List all labels instead of symptoms
python3 "$script_path" --labels --format summary
# Fetch a single label by ID
python3 "$script_path" --labels --id AWSAuthFailure
Flags:
--id <id>: fetch a single symptom (or label with--labels) by ID--search <text>: case-insensitive text search over id/summary/match_string (for labels: id/label_title/explanation/bugs)--label <label_id>: only symptoms that apply this label ID--matcher-type {string,regex,none,cel}: only symptoms of this matcher type--labels: operate on labels instead of symptoms--format {json,summary}: output format (defaultjson)
API Details
Endpoints
GET https://sippy.dptools.openshift.org/api/jobs/symptoms
GET https://sippy.dptools.openshift.org/api/jobs/symptoms/{id}
GET https://sippy.dptools.openshift.org/api/jobs/labels
GET https://sippy.dptools.openshift.org/api/jobs/labels/{id}
Example Symptom JSON
{
"id": "AWSCouldNotValidateAccessCredentials",
"summary": "AWS could not validate access credentials",
"matcher_type": "string",
"file_pattern": "build-log.txt",
"match_string": "api error AuthFailure: AWS was not able to validate the provided access credentials",
"label_ids": ["AWSAuthFailure"],
"created_by": "kenzhang",
"updated_by": "kenzhang",
"updated_at": "2026-04-27T16:09:12.660547Z"
}
Symptom Fields
| Field | Description |
|---|---|
id | Immutable identifier (generated from the summary on create) |
summary | Short, unique human-readable description (≤200 chars) |
matcher_type | One of string, regex, none (file exists), cel (expression over label names) |
file_pattern | Glob over the job run's artifact files, e.g. **/build-log.txt (not used for cel) |
match_string | Substring, regular expression, or CEL expression depending on matcher_type |
label_ids | Label IDs applied to a run when the symptom matches |
created_by, updated_by, updated_at | Metadata about who created/last modified the symptom and when |
Label Fields
| Field | Description |
|---|---|
id | Immutable identifier (≤80 chars) |
label_title | Unique human-readable title |
explanation | Markdown explanation of what the label means |
bugs | Jira issue keys associated with the label |
hide_display_contexts | UI contexts where the label is hidden (subset of spyglass, metrics, jaq-options) |
Error Handling
Case 1: Not Found (404)
python3 list_symptoms.py --id NoSuchSymptom
# Error: not found: https://sippy.dptools.openshift.org/api/jobs/symptoms/NoSuchSymptom (use list mode to see valid IDs)
Exits 1 and suggests listing symptoms/labels to find valid IDs.
Case 2: Network Error
# Error: failed to connect to Sippy API: [Errno -2] Name or service not known
Exits 1. Check network connectivity to sippy.dptools.openshift.org.
Case 3: Empty Results
Filters that match nothing print [] (JSON) or No results. (summary) and exit 0 — this is not an error.
Exit Codes:
0: Success (including empty results)1: Error (404, network error, etc.)
Examples
Example 1: List All Symptoms
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --format summary
Expected Output (excerpt):
Symptom: AWSCouldNotValidateAccessCredentials
Summary: AWS could not validate access credentials
Matcher: string
File pattern: build-log.txt
Match string: api error AuthFailure: AWS was not able to validate the provided access credentials
Labels: AWSAuthFailure
Updated by: kenzhang at 2026-04-27T16:09:12.660547Z
Total: 42
Example 2: Search for an Existing Symptom Before Creating One
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --search "credentials" --format summary
Example 3: Symptoms That Apply a Label
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --label AWSAuthFailure --format summary
Example 4: Fetch One Symptom by ID
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --id AWSCouldNotValidateAccessCredentials
Example 5: List All Labels
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --labels --format summary
Notes
--search,--label, and--matcher-typefiltering is performed client-side: the script fetches the full list and filters locally- This skill is read-only; to create, update, or delete symptoms or labels use the
manage-symptomsandmanage-labelsskills - No authentication is required — the public Sippy instance serves all read endpoints
See Also
- Related Skill:
manage-symptoms(create/update/delete symptoms) - Related Skill:
manage-labels(create/update/delete labels) - Related Skill:
diagnose-job-run-symptoms(explain which symptoms/labels apply to a job run)
Commands 20
Name
ci:add-debug-wait
Synopsis
/ci:add-debug-wait <workflow-or-job-name> [timeout]
Description
The ci:add-debug-wait command adds a wait step to a CI job/workflow for debugging test failures.
What it does:
- Takes job name, OCP version, and optional timeout as input
- Finds and edits the job config or workflow file
- Asks if any existing test steps should be removed (and cleans up orphaned env vars)
- Adds
- ref: waitbefore the last remaining test step (with optional timeout configuration) - Runs
make updateto validate and regenerate Prow jobs - Creates a git commit, pushes to your fork, and creates a PR via
gh
That's it! Simple, fast, and automated.
Implementation
The command performs the following steps:
Step 1: Gather Required Information
Prompt user for (in this order):
-
Workflow/Job Name: (from command argument $1 or prompt)
Workflow or job name: <user-input> Example: aws-c2s-ipi-disc-priv-fips-f7 Example: baremetalds-two-node-arbiter-e2e-openshift-test-private-tests -
Timeout (optional, from command argument $2):
Wait timeout in hours (optional, default: 3h): Examples: "1h", "2h", "8h", "24h", "72h" Valid range: 1h to 72h- If not provided, uses the wait step's default behavior (3 hours)
- Format: Integer followed by 'h' (e.g., "1h", "2h", "8h")
- Valid range: 1h to 72h (maximum enforced by wait step's timeout setting)
- For job config files (
ci-operator/config/): Set as theTIMEOUTenv var (e.g.,TIMEOUT: +8 hours) in the job'senv:section - For workflow files (
ci-operator/step-registry/): Set astimeout:(normalized to Go duration format, e.g., "8h" → "8h0m0s") andbest_effort: trueproperties on the ref step
-
OCP Version: (prompt - REQUIRED for searching job configs)
OCP version for debugging (e.g., 4.18, 4.19, 4.20, 4.21, 4.22):This is used to:
- Search the correct job config file (e.g., release-4.21)
- Document which version needs debugging
- Add context to the PR
-
OpenShift Release Repo Path: (prompt if not in current directory)
Path to openshift/release repository: Default: ~/repos/openshift-release
Step 2: Validate Environment and Prepare Branch
Silently validate and prepare (no user prompts):
cd <repo-path>
# Check 1: Repository exists and is correct
git remote -v | grep "openshift/release" || exit 1
# Check 2: Determine the default branch name (main or master)
default_branch=$(git remote show upstream 2>/dev/null | grep "HEAD branch" | awk '{print $NF}')
if [ -z "$default_branch" ]; then
# Fallback: check which branch exists locally
for branch in main master; do
if git rev-parse --verify "$branch" >/dev/null 2>&1; then
default_branch="$branch"
break
fi
done
fi
# Check 3: Checkout default branch and update from upstream
# Use --track -B to handle fresh clones where the local branch may not exist
git checkout --track -B "${default_branch}" "upstream/${default_branch}" 2>/dev/null \
|| git checkout "${default_branch}"
git pull --rebase upstream "${default_branch}"
Step 3: Search for Job/Test Configuration
Priority 1: Search job configs first (more specific and targeted):
cd <repo-path>
# Search for job config files matching the OCP version
# The job name could be in various config files, so search broadly
grep -r "as: ${job_name}" ci-operator/config/ --include="*release-${ocp_version}*.yaml" -l
Example searches:
- For
aws-c2s-ipi-disc-priv-fips-f7and OCP 4.21:grep -r "as: aws-c2s-ipi-disc-priv-fips-f7" ci-operator/config/ --include="*release-4.21*.yaml" -l
Handle job config search results:
-
1 file found:
✅ Found job configuration: ${file_path} Type: Job configuration file Proceeding with job config modification...→ Continue to Step 4a: Analyze Job Configuration
-
Multiple files found:
Found ${count} matching job config files: 1. ci-operator/config/.../release-4.21__amd64-nightly.yaml 2. ci-operator/config/.../release-4.21__arm64-nightly.yaml 3. ci-operator/config/.../release-4.21__ppc64le-nightly.yaml Select file (1-${count}) or 'q' to quit:Prompt user to select which file to modify, then continue to Step 4a: Analyze Job Configuration
-
0 files found:
ℹ️ No job config found for: ${job_name} (OCP ${ocp_version}) Searching for workflow files instead...→ Continue to Priority 2 below
Priority 2: Search workflow files (if job config not found):
cd <repo-path>
# Search for workflow files
find ci-operator/step-registry -type f -name "*${workflow_name}*workflow*.yaml"
Handle workflow search results:
-
0 files found:
❌ No job config or workflow file found for: ${job_name} Suggestions: 1. Check spelling of job/workflow name 2. Verify OCP version (${ocp_version}) 3. Try with partial name 4. Search manually: - Job configs: grep -r "as: ${job_name}" ci-operator/config/ - Workflows: find ci-operator/step-registry -name "*workflow*.yaml" | grep <partial-name> -
1 file found:
✅ Found workflow file: ${file_path} Type: Workflow file Proceeding with workflow modification...→ Continue to Step 4b: Analyze Workflow File
-
Multiple files found:
Found ${count} matching workflow files: 1. ci-operator/step-registry/.../workflow1.yaml 2. ci-operator/step-registry/.../workflow2.yaml 3. ci-operator/step-registry/.../workflow3.yaml Select file (1-${count}) or 'q' to quit:Prompt user to select which file to modify, then continue to Step 4b: Analyze Workflow File
Step 4a: Analyze Job Configuration
Read and parse the job config YAML:
# Find the specific test definition
grep -A 30 "as: ${job_name}" <job-config-file>
Check for:
- ✅ Has
steps:section - ✅ Has
test:section inside steps - ❌ Does NOT already have
- ref: wait
Example current structure:
- as: aws-c2s-ipi-disc-priv-fips-f7
cron: 36 16 3,12,19,26 * *
steps:
cluster_profile: aws-c2s-qe
env:
BASE_DOMAIN: qe.devcluster.openshift.com
FIPS_ENABLED: "true"
test:
- chain: openshift-e2e-test-qe
workflow: cucushift-installer-rehearse-aws-c2s-ipi-disconnected-private
If wait already exists:
ℹ️ Wait step already configured in job config
Current test section:
test:
- ref: wait
- chain: openshift-e2e-test-qe
No changes needed. The job is already set up for debugging.
If no test section found:
ℹ️ Job config found but no test: section
This job uses only the workflow's test steps.
Searching for the workflow: ${workflow_name}
→ Fall back to searching for workflow (Priority 2 in Step 3)
Ask about removing existing test steps: Display the test: steps and ask the user if any should be removed (common when only the cluster + wait step is needed). If steps are removed, also remove any env vars in the env: section that were only used by those steps (e.g., OPERATORS, SKIP_TESTS) to avoid make update validation errors about undeclared parameters.
→ Continue to Step 5a: Show Diff for Job Config
Step 4b: Analyze Workflow File
Read and parse the workflow YAML:
cat <workflow-file>
Check for:
- ✅ Has
workflow:section - ✅ Has
test:section - ❌ Does NOT already have
- ref: wait
Example current structure:
workflow:
as: baremetalds-two-node-arbiter-upgrade
steps:
pre:
- chain: baremetalds-ipi-pre
test:
- chain: baremetalds-ipi-test
post:
- chain: baremetalds-ipi-post
If wait already exists:
ℹ️ Wait step already configured in workflow
Current test section:
test:
- ref: wait
- chain: baremetalds-ipi-test
No changes needed. The workflow is already set up for debugging.
If no test section exists:
Add a test: section with the wait step before the post: section. Since these are debug PRs (never merged), it is safe to add a test section even if the workflow didn't have one.
test:
- ref: wait
post:
- chain: cucushift-installer-rehearse-aws-ipi-deprovision
Or with custom timeout:
test:
- ref: wait
timeout: 8h0m0s
best_effort: true
post:
- chain: cucushift-installer-rehearse-aws-ipi-deprovision
Ask about removing existing test steps: Same as Step 4a — display and ask. For workflow files, warn that removing steps affects ALL jobs using this workflow.
→ Continue to Step 5b: Modify Workflow File
Step 5a: Modify Job Config File
Edit the job config file directly - no confirmation needed:
# Add wait step before the last test step
# If timeout is provided, set it via the TIMEOUT env var in the job's env section
# See Step 6 for the YAML modification algorithm
After editing, validate the file by running the ref-step linter:
python3 <ai-helpers-repo>/plugins/ci/scripts/validate-ref-steps.py <job-config-file>
This catches invalid properties (e.g., timeout:, best_effort:) on ref: steps in job config files. If validation fails, fix the file before committing — use the TIMEOUT env var instead.
Two scenarios:
-
Without custom timeout (uses wait step's built-in default of 3h):
test: - ref: wait - chain: openshift-e2e-test-qeNote: No timeout or env change needed - the wait step will use its default TIMEOUT env var (3 hours)
-
With custom timeout (user-provided timeout parameter): Add
TIMEOUTto the existingenv:section within the job'ssteps:block, and add- ref: waitto thetest:section:env: BASE_DOMAIN: qe.devcluster.openshift.com TIMEOUT: +8 hours test: - ref: wait - chain: openshift-e2e-test-qeNote: The wait ref reads the
TIMEOUTenv var to determine how long to wait. Format:+N hourswhere N is the number of hours (e.g.,+1 hours,+8 hours,+24 hours,+72 hours). Do NOT quote the value — thedeterminize-ci-operatortool strips quotes. If the job already has anenv:section, add theTIMEOUTkey to it. If not, create theenv:section understeps:.
Show brief confirmation:
✅ Modified: ${job_name} (OCP ${ocp_version})
File: <job-config-file-path>
Added: - ref: wait${timeout:+ (TIMEOUT: +${hours} hours)}
Step 5b: Modify Workflow File
Edit the workflow file directly - no confirmation needed:
# Add wait step before the last test step
# If timeout is provided, add it as a step property
# See Step 6 for the YAML modification algorithm
Two scenarios:
-
Without custom timeout (uses wait step's built-in default of 3h):
test: - ref: wait - chain: baremetalds-ipi-testNote: No timeout or best_effort needed - the wait step will use its default TIMEOUT env var (3 hours)
-
With custom timeout (user-provided timeout parameter):
test: - ref: wait timeout: 8h0m0s best_effort: true - chain: baremetalds-ipi-testNote:
best_effort: trueis required when timeout is customized to prevent the wait step from failing the job if it times out
Show brief confirmation:
✅ Modified: ${workflow_name} workflow
File: <workflow-file-path>
Added: - ref: wait${timeout:+ (timeout: ${timeout})}
⚠️ Impact: Affects ALL jobs using this workflow
Step 6: Create Branch and Commit
Branch naming:
debug-${workflow_name}-${ocp_version}-$(date +%Y%m%d)
Example: debug-baremetalds-two-node-arbiter-4.21-20250131
Git operations:
# Create branch from the updated default branch (already checked out and updated in Step 2)
git checkout -b "${branch_name}"
# Modify the file (add wait step using the implementation below)
# Add '- ref: wait' before the last remaining step in the test: section
# Run make update to validate config and regenerate Prow jobs
make update
# Stage all changes (config file + any generated files from make update)
git add <config-file> <any-other-changed-files>
# Commit
git commit -m "[Debug] Add wait step to ${workflow_name} for OCP ${ocp_version}
This adds a wait step to enable debugging of test failures in OCP ${ocp_version}.
${removed_steps:+Removes ${removed_steps} steps and their associated orphaned env vars.}
The wait step pauses the workflow before tests run, allowing QE to:
- SSH into the test environment
- Inspect system state and logs
- Debug configuration issues
- Investigate test failures
OCP Version: ${ocp_version}
Workflow: ${workflow_name}"
YAML Modification Algorithm:
The modification process for both job configs and workflow files follows the same pattern:
-
Locate the target: Find the
test:section- For job configs: Within the specific job definition (
- as: ${job_name}) - For workflows: At the workflow level
- For job configs: Within the specific job definition (
-
Find test steps: Identify all steps (lines with
- ref:or- chain:) -
Check for duplicates: Ensure
- ref: waitdoesn't already exist -
Insert wait step: Add before the last test step with matching indentation
-
Handle timeout (differs between job configs and workflow files):
For job config files (
ci-operator/config/):- Without timeout: Add simple
- ref: wait - With timeout: Add simple
- ref: waitAND addTIMEOUT: +N hoursto the job'senv:section
For workflow files (
ci-operator/step-registry/):- Without timeout: Add simple
- ref: wait - With timeout: Add
- ref: waitwithtimeout:andbest_effort:properties (valid in workflow schema)
- Without timeout: Add simple
Example transformation (job config):
Before:
env:
BASE_DOMAIN: qe.devcluster.openshift.com
test:
- chain: openshift-e2e-test-qe
After (without timeout):
env:
BASE_DOMAIN: qe.devcluster.openshift.com
test:
- ref: wait
- chain: openshift-e2e-test-qe
After (with timeout=8h):
env:
BASE_DOMAIN: qe.devcluster.openshift.com
TIMEOUT: +8 hours
test:
- ref: wait
- chain: openshift-e2e-test-qe
Example transformation (workflow file):
Before:
test:
- chain: baremetalds-ipi-test
After (with timeout=8h):
test:
- ref: wait
timeout: 8h0m0s
best_effort: true
- chain: baremetalds-ipi-test
- Validate: Run
validate-ref-steps.pyon the modified file (see Step 5a). Fix any errors before committing.
Critical constraints:
- Preserve exact YAML indentation (typically 2 spaces per level)
- Insert BEFORE the last step, not after
- Normalize timeout format: for job configs use
+N hours, for workflow files use Go duration (e.g., "8h" → "8h0m0s")
Step 7: Push to Fork
Push the branch to the user's fork (origin remote):
git push origin "${branch_name}"
Step 8: Create PR
Create the PR using gh:
gh pr create --repo openshift/release --title "[Debug] Add wait step to ${job_name} for OCP ${ocp_version}" --body "$(cat <<'EOF'
## Summary
- Adds a wait step to enable debugging of test failures in OCP ${ocp_version}
- Job: ${job_name}
- Timeout: ${timeout:-default (3h)}
${removed_steps:+- Removed steps: ${removed_steps} and their associated orphaned env vars}
The wait step pauses the workflow before tests run, allowing QE to:
- SSH into the test environment
- Inspect system state and logs
- Debug configuration issues
- Investigate test failures
⚠️ **DO NOT MERGE** — close this PR after debugging is complete.
EOF
)"
Display the PR URL:
✅ PR created successfully!
PR: <pr_url>
Branch: ${branch_name}
Job: ${job_name}
OCP: ${ocp_version}
⚠️ Remember to close PR after debugging (DO NOT MERGE)
That's it! Simple and clean.
Error Handling
Error: Repository Not Found
❌ Error: Repository not found at ${repo_path}
Please provide the correct path to openshift/release repository.
To clone:
git clone https://github.com/openshift/release.git
Error: Not in openshift/release Repo
❌ Error: This doesn't appear to be the openshift/release repository
Remote URL: ${current_remote}
Expected: github.com/openshift/release
Please navigate to the correct repository.
Error: Workflow File Not Found
❌ Error: Workflow file not found
Searched for: *${workflow_name}*workflow*.yaml
Location: ci-operator/step-registry/
Suggestions:
1. Verify the workflow name
2. Try a partial match
3. Search manually: find ci-operator/step-registry -name "*workflow*.yaml"
Error: Wait Step Already Exists
ℹ️ Wait step already configured in this workflow
No action needed - you can proceed with debugging using the existing wait step.
Error: Invalid OCP Version
❌ Invalid OCP version: ${version}
Valid versions: 4.18, 4.19, 4.20, 4.21, 4.22, master
Please provide a valid version.
Error: Invalid Timeout Format
❌ Invalid timeout format: ${timeout}
Valid format: Integer followed by 'h' (e.g., "1h", "2h", "8h", "24h", "72h")
Valid range: 1h to 72h
Examples:
- "1h" (1 hour)
- "8h" (8 hours)
- "24h" (24 hours)
- "72h" (72 hours, maximum)
Please provide a valid timeout in hours.
Note: Timeout Normalization
When a user provides a timeout like "8h", the implementation should normalize it to the standard Go duration format "8h0m0s" for consistency with existing configurations in the codebase.
Return Value
- Success: Created PR URL and debugging instructions
- Error: Error message with suggestions for resolution
- Format: Text output with emoji indicators for status
Examples
Example 1: Without Timeout (Default 3h)
/ci:add-debug-wait aws-ipi-f7-longduration-workload
Prompts for: OCP version (4.21), repo path
Result:
test:
- ref: wait
- chain: openshift-e2e-test-qe
Returns: PR URL
Example 2: With Custom Timeout (Job Config)
/ci:add-debug-wait aws-ipi-f7-longduration-workload 8h
Prompts for: OCP version (4.21), repo path
Result (adds TIMEOUT env var + ref: wait):
env:
BASE_DOMAIN: qe.devcluster.openshift.com
TIMEOUT: +8 hours
test:
- ref: wait
- chain: openshift-e2e-test-qe
Returns: PR URL
Example 3: With Custom Timeout (Workflow File)
/ci:add-debug-wait baremetalds-two-node-arbiter-upgrade 24h
Behavior: Searches job config first, falls back to workflow if not found. Warns that workflow changes affect ALL jobs using it.
Result (timeout/best_effort on ref is valid in workflow files):
test:
- ref: wait
timeout: 24h0m0s
best_effort: true
- chain: baremetalds-ipi-test
Returns: PR URL
Example 4: With Step Removal (Job Config)
/ci:add-debug-wait tempo-upstream-tests 8h
User selects to remove - ref: install-operators and - ref: distributed-tracing-tests-tempo-upstream.
The - as: install step (operator bundle install) is kept since the user did not select it for removal.
Result (removes selected refs + their orphaned env vars OPERATORS and SKIP_TESTS):
env:
BASE_DOMAIN: devobscluster.devcluster.openshift.com
TIMEOUT: +8 hours
test:
- as: install
cli: latest
commands: |
oc create namespace openshift-tempo-operator
...
- ref: wait
Returns: PR URL
Arguments
- $1 (workflow-or-job-name): The name of the CI workflow or job to add the wait step to (required)
- $2 (timeout): Optional timeout in hours (1h-72h). Examples: "1h", "8h", "24h", "72h". If not provided, uses wait step's default (3h)
Notes
Best Practices for QE
Before Running Command:
- ✅ Confirm test is actually failing
- ✅ Check existing debug PRs
- ✅ Know which OCP version is affected
During Debugging:
- 📝 Take detailed notes
- 💾 Save logs and screenshots
- 🔍 Document root cause
- 📊 Record all findings
After Debugging:
- ✅ Document findings
- ✅ Close the debug PR
- ✅ Delete the branch
- ✅ Share learnings with team
- ✅ Create fix PR if needed
Future Enhancements
Add companion commands:
/ci:close-debug-pr- Lists open debug PRs, prompts for findings, closes PR/ci:list-debug-prs- Show all open debug PRs/ci:revert-debug-pr- Revert a debug PR that was merged by mistake
Name
ci:analyze-pr-reverts
Synopsis
/ci:analyze-pr-reverts [limit]
Description
The ci:analyze-pr-reverts command fetches recently reverted pull requests from the Sippy API and analyzes them to identify common failure patterns, the most affected repositories, and recommends concrete preventive measures such as new presubmit jobs, org-wide CodeRabbit review rules, or process changes.
This command is intended to be run periodically (e.g., monthly) to track revert trends over time.
Implementation
-
Fetch Revert PRs from Sippy API: Query the Sippy API for PRs with "revert" in the title.
curl -s 'https://sippy.dptools.openshift.org/api/pull_requests?release=Presubmits&filter=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22title%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22revert%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&sortField=merged_at&sort=desc&limit=<LIMIT>'- The default limit is 50 but can be overridden by the user's argument
- The API returns JSON with fields including:
merged_at,link,author,title - Parse the JSON response and extract all PR entries
-
Deduplicate: Remove duplicate entries (same PR URL appearing multiple times).
-
Scrape PR Details: For each unique PR URL, fetch details using
gh:gh pr view <url> --json title,body,files,labelsExtract from each PR:
- Repository: Parse org/repo from the URL
- Description: The PR body explaining why the revert was needed
- Files changed: What areas of code were affected
- Root cause category: Classify based on the description (see categories below)
If a PR cannot be fetched (permissions, deleted, etc.), note it and continue.
-
Categorize Each Revert: Classify each revert into one or more of these root cause categories based on the PR description and the original PR that was reverted:
- Test/platform coverage gap: New or migrated tests that fail on untested platforms (microshift, single-node, IPv6, FIPS, real-time, upgrade jobs). Look for mentions of specific failing job profiles.
- IPv6/network incompatibility: Changes that work on IPv4 but break IPv6/dual-stack clusters. Look for hardcoded IPv4 addresses, network bind issues.
- Operator/controller regression: Core operator logic changes causing CrashLoopBackOff, degraded states, install failures, or functional breakage.
- Build/CI tooling: Renovate configs, dependency automation, build pipeline changes that break CI.
- API/validation side effects: API changes, CRD validation changes, or policy enforcement that has unexpected downstream impact.
- Feature flag/config change: Premature feature gate removal, default value changes, config toggles.
- Cross-component dependency: Change that depends on another component's change that hasn't landed or built yet.
- Flaky/unstable new test: New test that passes sometimes but flakes at a high enough rate to block payloads.
-
Generate Statistics:
- Reverts by repository: Count and rank repos by number of reverts
- Reverts by category: Count and rank root cause categories
- Repeat offenders: PRs that were reverted, re-merged, and reverted again
- Most common failing job profiles: Which CI job profiles appear most in revert descriptions (e.g.,
e2e-metal-ipi-ovn-ipv6,e2e-aws-ovn-microshift) - Reverts by author: Identify if certain authors/teams have recurring patterns
- TRT-initiated vs. author-initiated: How many were quick-reverts by TRT (Revertomatic) vs. self-reverts
-
Generate Preventive Recommendations: Based on the patterns found, produce actionable recommendations in three areas:
a. New Presubmit Jobs / CI Coverage
For each pattern of reverts caused by missing platform coverage, recommend specific presubmit or payload jobs that should be added. Common recommendations include:
- If IPv6 reverts are frequent: Recommend repos add
/payload-job e2e-metal-ipi-ovn-ipv6to presubmit or require it before merge - If microshift reverts are frequent: Recommend repos add microshift conformance jobs
- If single-node reverts are frequent: Recommend single-node upgrade jobs
- If upgrade reverts are frequent: Recommend upgrade-from-previous-version jobs
- If FIPS reverts are frequent: Recommend FIPS-specific jobs
Be specific about which repos need which jobs based on the data.
b. CodeRabbit Review Rules
Recommend new review rules for the openshift/coderabbit repository's
.coderabbit.yamlconfiguration. The coderabbit repo contains org-wide review guidelines that apply to all OpenShift repositories.The existing
.coderabbit.yamlalready has checks for:- Stable/deterministic test names
- Test structure and quality
- MicroShift compatibility
Based on the revert patterns, recommend NEW
pre_merge_checks.custom_checksentries. For example:- IPv6 compatibility check: Flag hardcoded IPv4 addresses (
0.0.0.0,127.0.0.1) in network-related code, recommend dual-stack testing - Multi-node assumptions check: Flag tests that assume multiple nodes without checking node count
- Feature gate removal check: Flag removal of feature gates without verifying downstream consumers
- Cross-component dependency check: Flag changes that import or depend on unreleased changes in other repos
- Upgrade compatibility check: Flag changes to operator behavior that could break upgrade paths
For each recommendation, provide the complete YAML block that could be added to
.coderabbit.yamlin the same format as the existing checks (name, mode, instructions).c. Process / Documentation Recommendations
- Recommend documentation updates, team process changes, or other non-automated improvements
- Identify if specific teams need targeted guidance
- Suggest improvements to the revert process itself if patterns indicate issues
- If IPv6 reverts are frequent: Recommend repos add
-
Format the Report: Output a well-structured markdown report with the following sections:
## PR Revert Analysis Report ### Date Range: [earliest] to [latest merged_at] ### Summary ### Reverts by Repository ### Reverts by Root Cause Category ### Repeat Reverts (PRs reverted more than once) ### Most Common Failing Job Profiles ### Detailed Revert List (table with: repo, PR link, category, brief reason) ### Recommendations #### New Presubmit Jobs #### New CodeRabbit Review Rules #### Process ImprovementsAll PR links in the report MUST use full GitHub URLs (e.g.,
https://github.com/openshift/machine-config-operator/pull/5703), never shorthand references like#5703which would resolve relative to whatever repo the reader is viewing.
Return Value
- Markdown report: Full analysis report as described above
- Recommended CodeRabbit YAML: Ready-to-use YAML blocks for
.coderabbit.yaml - Summary statistics: Key numbers (total reverts, top repo, top category)
Examples
-
Basic usage (fetches last 50 revert PRs):
/ci:analyze-pr-reverts -
With custom limit:
/ci:analyze-pr-reverts 100
Arguments
$1(optional): Maximum number of revert PRs to fetch from the Sippy API. Defaults to 50.
Prerequisites
-
GitHub CLI (
gh): Must be installed and authenticated- Check:
gh auth status - Needed to fetch PR details (description, files, labels) for each revert
- Check:
-
Network access: Must be able to reach
sippy.dptools.openshift.org
Notes
- Some PRs may not be fetchable via
gh(private repos, deleted PRs, permission issues). These are noted in the report but don't block analysis. - The query captures PRs with "revert" or "Revert" in the title. Some legitimate non-revert PRs may be included (e.g., "unrevert" PRs that restore previously reverted changes). Filter these out during analysis.
- PRs from non-openshift orgs (e.g.,
Azure/ARO-HCP) may have limitedghaccess. Include them in stats but note if details couldn't be fetched. - The recommended CodeRabbit rules should be conservative (mode:
warningnoterror) to avoid blocking legitimate PRs while the rules are tuned.
See Also
- Related Command:
ci:revert-pr- Create a revert PR for a breaking change - CodeRabbit Config: https://github.com/openshift/coderabbit
- OpenShift Quick Revert Policy: https://github.com/openshift/enhancements/blob/master/enhancements/release/improving-ci-signal.md#quick-revert
- Revertomatic: https://github.com/stbenjam/revertomatic
Name
ci:analyze-prow-job-resource
Synopsis
Generate a report showing the lifecycle of a single resource:
/ci:analyze-prow-job-resource <prowjob-url> [namespace:][kind/][resource-name]
Generate a report showing multiple resource lifecycles:
/ci:analyze-prow-job-resource <prowjob-url> [namespace:][kind/][resource-name],[namespace:][kind/][resource-name],...
Description
Analyze the lifecycle of Kubernetes resource(s) in a Prow CI job by invoking the "prow-job-analyze-resource" skill.
Implementation
Pass the user's request to the skill, which will:
- Download Prow job artifacts from Google Cloud Storage
- Parse audit logs and pod logs
- Generate an interactive HTML report with timeline visualization
The skill handles all the implementation details including URL parsing, artifact downloading, log parsing, and HTML report generation.
Name
ci:analyze-regression
Synopsis
/ci:analyze-regression <regression id>
Description
The ci:analyze-regression command analyzes details for a specific Component Readiness regression and suggests next steps for investigation.
The command performs a full analysis regardless of whether the regression has been triaged. For triaged regressions, it also fetches the linked JIRA issue to analyze whether someone is actively working on the fix or if the issue needs attention.
This command is useful for:
- Understanding regression patterns and failure modes
- Checking if a triaged regression is being actively worked on or needs attention
- Identifying related regressions that might be caused by the same issue
- Getting pointers on where to investigate next
Implementation
Important: Avoiding user permission prompts when running scripts
When calling Python skill scripts via the Bash tool, always run the script directly without piping the output through inline Python (python3 -c "..."). Complex piped commands trigger user permission prompts, while simple python3 script.py args calls are auto-approved.
- Do:
python3 script.py args --format json 2>/dev/null— run the script directly and process the JSON output in your reasoning - Don't:
python3 script.py args | python3 -c "import json; ..."— piped inline Python triggers permission prompts
Parse and analyze the JSON output from scripts using your own reasoning capabilities rather than shell pipelines.
Obtaining the DPCR authentication token from mounted kubeconfig
The triage and bug filing steps (step 14) require a Bearer token from the DPCR cluster (api.cr.j7t7.p1.openshiftapps.com:6443). When running in a container with ~/.kube mounted (read-only), the token is extracted directly from the mounted kubeconfig using oc:
# Find the oc context for the DPCR cluster from the mounted kubeconfig
DPCR_CONTEXT=$(oc config get-contexts -o name 2>/dev/null | while read -r ctx; do
server=$(oc config view -o jsonpath="{.clusters[?(@.name=='$(oc config view -o jsonpath="{.contexts[?(@.name=='$ctx')].context.cluster}" 2>/dev/null)')].cluster.server}" 2>/dev/null || echo "")
server_clean=$(echo "$server" | sed -E 's|^https?://||')
if [ "$server_clean" = "api.cr.j7t7.p1.openshiftapps.com:6443" ]; then
echo "$ctx"
break
fi
done)
# Extract the token from the DPCR context
if [ -z "$DPCR_CONTEXT" ]; then
echo "ERROR: Could not find a DPCR cluster context in kubeconfig. Set DPCR_CONTEXT manually or run: oc login https://api.cr.j7t7.p1.openshiftapps.com:6443"
exit 1
fi
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT" 2>/dev/null)
This works because oc reads from ~/.kube/config which is bind-mounted from the host. The token stored in the kubeconfig was obtained when the user previously ran oc login to the DPCR cluster on the host. If the token is expired, instruct the user to re-authenticate on the host: oc login https://api.cr.j7t7.p1.openshiftapps.com:6443.
-
Load CI Context: Read all documentation files in
plugins/ci/references/for context on tests, jobs, and CI conventions. These contain important notes on specific test frameworks, job ownership, and debugging guidance that should inform the analysis.ls plugins/ci/references/Read each file found. Keep this context in mind throughout the analysis — it may affect how you interpret failure patterns, who to recommend contacting, or what the root cause is likely to be.
-
Parse Arguments: Extract regression ID
- Regression ID format: Integer ID from Component Readiness
- Example: 34446
-
Fetch Regression Details: Use the
fetch-regression-detailsskillRun the Python script to fetch comprehensive regression data:
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py" regression_data=$(python3 "$script_path" <regression_id> --format json)The skill automatically:
- Fetches regression metadata from: https://sippy.dptools.openshift.org/api/component_readiness/regressions/<regression_id>
- Fetches test details and parses job statistics
- Groups failed job runs by job name with pass sequences
Extract the following from the JSON response:
test_name: Full test namereleaseandbase_release: Version informationcomponentandcapability: Ownership mappingvariants: Platform/topology combinations where test is failingopenedandclosed: Regression timelinetriages: Existing JIRA ticketsanalysis_status: Integer status code (negative indicates problems, -1000 indicates failed fix)analysis_explanations: Human-readable explanations for the statustest_details_url: Link to Sippy test details (API URL - must be converted to UI URL before displaying, see note below)sample_failed_jobs: Dictionary keyed by job name, each containing:pass_sequence: Chronological S/F pattern (newest to oldest)failed_runs: List of failed runs with job_url, job_run_id, start_time
job_runs: Complete list of all job runs where the failure was observed throughout the regression's entire life (not just the last reporting period). Each entry contains:prowjob_run_id,prowjob_name,prowjob_url: Job identification and linkstart_time: When the job rantest_failures: Total number of test failures in the run. High values (e.g., >10) indicate mass failure runs where the regressed test may just be caught up in a larger issue rather than being the primary problem
Converting
test_details_urlto UI URL: Thetest_details_urlfrom the API is an API endpoint not suitable for display or bug reports. Convert it to the UI URL by replacing the base path. The query parameters are identical:# Convert API URL to UI URL test_details_ui_url=$(echo "$test_details_url" | sed 's|https://sippy.dptools.openshift.org/api/component_readiness/test_details|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details|')Always use the converted
test_details_ui_urlwhen displaying the link in the report or including it in bug descriptions.See
plugins/ci/skills/fetch-regression-details/SKILL.mdfor complete implementation details. -
Fetch Global Test Report: Use the
fetch-test-reportskill to check how this test is doing globallyUse the test name and release from the regression data to fetch the global test report with per-variant breakdown:
# Extract test name and release from regression data test_name=$(echo "$regression_data" | jq -r '.test_name') release=$(echo "$regression_data" | jq -r '.release') # Fetch per-variant breakdown to see which job types are affected script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py" test_report=$(python3 "$script_path" "$test_name" --release "$release" --no-collapse --format json)Analyze the test report:
- Global pass rate: Check the overall pass rate across all variants. A low pass rate confirms a widespread issue; a high pass rate with the regression suggests the problem is variant-specific.
- Per-variant breakdown: With
--no-collapse, each row shows a specific variant combo (e.g.,["aws", "ovn", "amd64", "upgrade-micro"]). Compare pass rates across variants to identify if the failure is platform-specific, network-specific, or upgrade-specific. - Open bugs: If
open_bugs > 0, someone has already filed a Jira bug mentioning this test. This bug may not yet be triaged in Component Readiness. Note these bugs in the report — they could be used to triage the regression without filing a duplicate. - Trend: Check
net_working_improvement— positive means improving, negative means getting worse.
Include the global test report findings in Section 1 of the final report.
-
Check Triage Status and Fetch JIRA Progress: Determine triage state and analyze bug progress
Check if the regression has already been triaged and fetch JIRA details:
# Check if regression has triages triage_count=$(echo "$regression_data" | jq '.triages | length')If
triage_count > 0(regression is triaged):This means a human has already attributed this regression to a specific bug. For each triage entry, fetch the JIRA issue to analyze progress. Also check if step 4's test report found open bugs that may be related.
# Check if JIRA auth environment variables are set if [ -z "$JIRA_USERNAME" ] || [ -z "$JIRA_API_TOKEN" ]; then echo "Warning: JIRA_USERNAME and/or JIRA_API_TOKEN not set. Skipping JIRA progress analysis." else # For each triage, fetch JIRA details using the fetch-jira-issue skill jira_script="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py" for jira_key in $(echo "$regression_data" | jq -r '.triages[].jira_key'); do jira_data=$(python3 "$jira_script" "$jira_key" --format json) # The fetch-jira-issue skill automatically classifies progress. Extract the classification: progress_level=$(echo "$jira_data" | jq -r '.progress.level') progress_reason=$(echo "$jira_data" | jq -r '.progress.reason') assignee=$(echo "$jira_data" | jq -r '.assignee.display_name // "Unassigned"') linked_prs=$(echo "$jira_data" | jq -r '.linked_prs | length') done fiSee
plugins/ci/skills/fetch-jira-issue/SKILL.mdfor complete implementation details.Progress levels returned by the skill:
- ACTIVE: Status is ASSIGNED/IN PROGRESS with recent activity, PRs linked, or recent comments
- STALLED: Status is ASSIGNED but no activity in 14+ days
- NEEDS_ATTENTION: Status is NEW/OPEN with no assignee or no recent progress
- RESOLVED: Status is Closed/Verified
Classification Output:
JIRA Progress Analysis: - OCPBUGS-12345: 🟢 ACTIVE - Assigned to user@redhat.com, PR in review (2 days ago) - OCPBUGS-12345: 🟡 STALLED - Assigned but no activity in 21 days - OCPBUGS-12345: 🔴 NEEDS ATTENTION - Status NEW, no assignee, no commentsNote for Failed Fixes (analysis_status == -1000):
- This indicates the test is still failing AFTER the triage was resolved
- Flag: "⚠️ FAILED FIX: Test continues to fail after triage was resolved"
- Recommend: Re-opening the bug or filing a new one
If
triage_count == 0(regression is NOT triaged):- No bug has been filed yet
- If step 4's test report found
open_bugs > 0, list these — someone may have filed a bug that hasn't been triaged yet - Continue with full investigation (steps 6-11)
- Bug filing recommendations will be provided in step 12
Always continue to step 6 regardless of triage status to provide full analysis.
-
Interpret Regression Data: Analyze failure patterns from
sample_failed_jobsParse the
sample_failed_jobsdictionary from the regression data:# Extract and analyze each job echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries | .[] | "\(.key)|\(.value.pass_sequence)|\(.value.failed_runs | length)"'-
Identify Jobs with Most Failures:
- Iterate through the
sample_failed_jobsdictionary keys (job names) - Count
failed_runsarray length for each job:echo "$regression_data" | jq '.sample_failed_jobs["job-name"].failed_runs | length' - Sort jobs by failure count (descending) to identify the most impacted jobs
- Example: Job A has 18 failures, Job B has 1 failure → Job A is the primary concern
- Iterate through the
-
Analyze Pass Sequence Patterns: For each job, examine the
pass_sequencestring.CRITICAL: Reading Direction
- The pass_sequence is ordered newest to oldest (left = most recent, right = oldest)
- First character = most recent job run
- Last character = oldest job run
- Example:
FFFSSSSSSSmeans: 3 most recent runs failed, 7 older runs passed
Pattern 1: Permafailing Test
pass_sequencestarts with many "F"s at the LEFT (e.g.,FFFFFFFFFForFFFFFFFSS)- The LEFT side (most recent runs) shows consistent failures
- Example:
FFFFFFFFFFFFFFFFFF- leftmost 18 characters are all F = 18 most recent runs all failed - Interpretation: Test is currently broken and consistently failing
- Action: High priority - test is completely broken for this job variant
Pattern 2: Resolved Issue
pass_sequencestarts with "S"s at the LEFT, with "F"s toward the RIGHT (e.g.,SSSSSFFFFForSSSSSFFFFFSS)- The LEFT side (most recent runs) shows successes, RIGHT side (older runs) shows failures
- Example:
SSSSFFFFFFFFFFFF- leftmost 4 chars are S (recent passes), rightward chars are F (older failures) - Example:
SSSSSSSSSSSSSSSSSSFFSFFF- many S's on left (recent passes), F's on right (older failures) - Interpretation: The problem existed in older runs but has been resolved in recent runs
- Action: Lower priority - verify if issue is truly resolved or if more monitoring is needed
Pattern 3: Flaky Test
pass_sequenceshows "F"s and "S"s interspersed throughout (e.g.,SFSFSFSFSForFSSFFSSFF)- No consistent block of failures - failures are scattered across the sequence
- Example:
SSFSSFSSFSSFshows intermittent failures mixed with successes - Interpretation: Test appears flaky rather than consistently failing
- Action: May require test stabilization or flake investigation rather than product bug
Pattern 4: Recent Regression
pass_sequencestarts with "F"s at the LEFT, followed by "S"s toward the RIGHT (e.g.,FFSSSSSSSSS)- The LEFT side (most recent runs) shows new failures, RIGHT side (older runs) shows the test was passing
- Example:
FFFFFSSSSSSSSSS- leftmost 5 chars are F (recent failures), rightward chars are S (older successes) - Interpretation: Test was passing historically but has recently started failing
- Action: High priority - recent regression, investigate recent code changes
-
Generate Pattern Summary: Create a summary for each job:
- Job name
- Total failed runs
- Pass sequence pattern classification (permafail / resolved / flaky / recent regression)
- Recommended priority level
- Suggested next steps
Example Analysis:
Given this
sample_failed_jobsstructure:{ "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": { "pass_sequence": "FFFFFFFFFFFFFFFFFF", "failed_runs": [/* 18 failed runs */] }, "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-techpreview": { "pass_sequence": "SSFSSSSSSSS", "failed_runs": [/* 1 failed run */] }, "periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-example": { "pass_sequence": "SSSSSSSSSSSSSSSSSSFFSFFF", "failed_runs": [/* 5 failed runs */] } }Remember: LEFT = newest, RIGHT = oldest
Pattern analysis output:
- Job 1 (18 failures):
FFFFFFFFFFFFFFFFFF- Reading: All 18 characters are F, starting from the LEFT (newest)
- Classification: Permafail - most recent runs are all failing
- Priority: High
- Action: "Test is completely broken for this metal+ovn+ipv4+rhcos10 variant. Investigate immediately."
- Job 2 (1 failure):
SSFSSSSSSSS- Reading: LEFT (newest) starts with SS, then one F in position 3, rest are S
- Classification: Flaky - isolated failure in mostly passing runs
- Priority: Low
- Action: "Single recent failure in mostly passing job. Monitor for pattern or investigate if recurring."
- Job 3 (5 failures):
SSSSSSSSSSSSSSSSSSFFSFFF- Reading: LEFT (newest) has 18 S's = most recent 18 runs passed; RIGHT (oldest) has FFSFFF = older runs had failures
- Classification: Resolved - failures were in OLDER runs, recent runs are passing
- Priority: Low
- Action: "Issue appears to have been resolved. The failures occurred in older runs, not recent ones."
-
-
Analyze Failure Output Consistency: Use the
fetch-test-runsskillUse the
fetch-test-runsskill to fetch and analyze actual test failure outputs from all failed job runs. This helps determine if all failures have the same root cause or if there are multiple issues.Implementation:
# Extract test_id from regression data (already fetched in step 3) test_id=$(echo "$regression_data" | jq -r '.test_id') # Collect all job_run_ids from sample_failed_jobs # This creates a comma-separated list of all failed job run IDs across all jobs job_run_ids=$(echo "$regression_data" | jq -r ' .sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_run_id ' | tr '\n' ',' | sed 's/,$//') # Use the fetch-test-runs skill script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py" output_analysis=$(python3 "$script_path" "$test_id" "$job_run_ids" --format json)The skill fetches raw test failure outputs from Sippy API.
See
plugins/ci/skills/fetch-test-runs/SKILL.mdfor complete implementation details.Parse Results and Analyze with AI:
# Check if fetch was successful success=$(echo "$output_analysis" | jq -r '.success') if [ "$success" = "true" ]; then # Extract outputs array outputs=$(echo "$output_analysis" | jq -r '.runs') num_outputs=$(echo "$outputs" | jq 'length') echo "Fetched $num_outputs test failure runs" # AI ANALYSIS: Compare the runs for similarity # Examine each output message to determine: # 1. How many failures show the same or very similar error messages # 2. What percentage of failures are consistent # 3. What the common error pattern is (if any) # 4. Extract file references, API paths, or resource names from error messages # # The runs are in format: {"url": "...", "output": "error text", "test_name": "..."} # # Classify consistency: # - Highly Consistent (>90%): All/nearly all show same error -> single root cause # - Moderately Consistent (50-90%): Most share patterns -> primary issue with variation # - Inconsistent (<50%): Different errors -> multiple causes or environmental issues # # Extract from error messages: # - File/line references (e.g., "discovery.go:145") # - API/resource paths (e.g., "/apis/stable.e2e-validating-admission-policy-1181/") # - Common error phrases (e.g., "server could not find the requested resource") else # API not available - this is acceptable, continue without output analysis error=$(echo "$output_analysis" | jq -r '.error') echo "Note: Test output analysis unavailable - $error" echo "Continuing with other analysis steps..." fiHow to Analyze Outputs with AI:
Read through the failure outputs and identify patterns:
-
Compare Error Messages: Count how many outputs have identical or very similar messages
- Example: If 17 out of 18 say "the server could not find the requested resource", that's 94% consistency
-
Extract Common Elements: Look for shared components in the error messages
- File references: "k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145"
- API paths: "/apis/stable.e2e-validating-admission-policy-1181/"
- Error phrases: "server could not find the requested resource"
-
Classify Consistency:
- Highly Consistent (>90%): Single root cause - all failures show same error
- Moderately Consistent (50-90%): Primary issue - most share patterns with some variation
- Inconsistent (<50%): Multiple causes - failures show different error types
-
Determine Root Cause: Based on the common error message and extracted information, infer what the underlying issue likely is
- Example: "API endpoint not available" if errors mention missing API resources
- Example: "Timeout issue" if errors mention timeouts or waiting conditions
-
7a. Analyze Job Run Context: Use the fetch-job-run-summary skill
For each failed job run, the regressed test is just one of potentially many test results. Understanding the broader job run context — how many other tests failed, which ones, and whether they share error patterns — is critical for diagnosing the root cause.
- If the regressed test is the only failure (or one of very few) in each run, it is likely a targeted issue specific to that test or its component.
- If the regressed test fails alongside a consistent set of other tests, those co-failures point to a shared root cause (e.g., an operator failing that breaks multiple dependent tests).
- If the regressed test is caught up in mass failures with different tests failing each time, it may be collateral damage from infrastructure instability or a foundational issue rather than a specific regression in the test's component.
This is especially important for mass test failure regressions ([Jira:"Test Framework"] there should not be mass test failures), where the fetch-test-runs output from step 7 only shows the mass failure count breakdown and the actual failing tests must be examined to find the real root cause.
Implementation:
Select a representative sample of failed job runs to analyze. Pick up to 3 runs from the sample_failed_jobs data — if failure counts vary, choose runs with different severities (e.g., the run with the most failures, one with a moderate count, and one with the fewest):
script_path="plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py"
# For each selected job run ID from the sample_failed_jobs data:
summary=$(python3 "$script_path" <job_run_id> --format json)
See plugins/ci/skills/fetch-job-run-summary/SKILL.md for complete implementation details.
Analyze the results across runs:
-
Assess isolation vs. co-failure: For each run, check
failure_count. If the regressed test is the only failure (or one of 2-3), the issue is isolated to this test. If it fails alongside 10+ other tests, it may be a symptom of a broader problem. -
Compare failed test lists across runs: Check whether the same set of other tests co-fail with the regressed test. Consistent co-failures suggest a shared root cause. Random co-failures suggest environmental issues.
-
Check dominant error patterns: The skill automatically detects error messages appearing in >5% of failures. A single dominant error across many tests (e.g., "stale GroupVersion discovery" at 94%) indicates one root cause cascading across many tests, and the regressed test is likely collateral damage. If the regressed test has a unique error distinct from other failures, it may be an independent issue.
-
Assess failure scale consistency: Compare
failure_countacross runs. If counts vary dramatically (e.g., 12 in one run, 397 in another), the issue may be intermittent or timing-dependent. Consistent failure counts suggest a deterministic problem. -
Identify the true failing component (especially for mass test failure regressions): The mass test failure regression is attributed to "Test Framework" in Component Readiness, but the actual product issue is in whatever component's tests are failing. Use the failed test names to identify the real affected component(s).
Classification:
- Isolated failure: The regressed test is the only failure or one of very few in each run. The issue is specific to this test's component. File the bug against that component.
- Consistent co-failures: The same set of tests fail together across runs, with a dominant error pattern. This points to a specific product bug affecting multiple tests. The bug should be filed against the component responsible for the shared failure mode.
- Inconsistent/random co-failures: The regressed test fails in runs with many other failures, but the set of co-failing tests varies. This suggests infrastructure instability, cluster health issues, or a foundational problem (e.g., API server instability) causing cascading and unpredictable failures.
- Scaling pattern: A consistent core set of failures plus a variable number of additional failures. The core failures are the root cause; the additional failures are secondary effects.
Output for Report: Include in the report:
- Number of runs analyzed and their failure counts
- Whether the regressed test is an isolated failure or part of broader co-failures
- Whether the same tests consistently co-fail across runs
- The dominant error pattern (if any) with percentage
- For mass test failure regressions: the real affected component(s) based on failing test names
- Assessment: isolated test issue, targeted product bug with co-failures, or collateral damage from widespread instability
7b. Deep Install Failure Analysis (install should succeed regressions only)
Only perform this step if the test name contains "install should succeed".
Install regressions require deeper per-run analysis because different runs may fail at entirely different installation stages (bootstrap, infrastructure, cluster creation, etc.) for different reasons. This step invokes the ci:prow-job-analysis skill on a representative sample of failed runs to determine whether all failures share a single root cause or have multiple independent causes.
Select runs to analyze:
Select up to 5 failed job runs to analyze. Choose from the jobs with the most failures (identified in step 6). If failure counts vary across jobs, spread the selection across jobs to get a representative sample. Extract job_url values from sample_failed_jobs:
# Example: extract up to 5 job URLs from sample_failed_jobs (one per job, preferring high-failure jobs)
echo "$regression_data" | jq -r '
.sample_failed_jobs
| to_entries
| sort_by(.value.failed_runs | length)
| reverse
| .[0:5]
| .[]
| .value.failed_runs[0].job_url
'
Invoke the install failure skill for each selected run:
For each selected job_url, use the Skill tool to invoke ci:prow-job-analysis with the job URL. The skill downloads and analyzes GCS artifacts (installer logs, log bundles, junit XML) and returns a structured report including:
- Failure stage (bootstrap, infrastructure, cluster creation, cluster operator stability, configuration, other)
- Root cause summary
- Key error messages
Synthesize findings across runs:
After analyzing all runs, compare the failure stages and root causes:
- Consistent — single root cause: All runs fail at the same stage with the same error pattern (e.g., all fail at
cluster bootstrapwith etcd not forming). High confidence this is one bug. - Consistent stage, multiple root causes: All runs fail at the same stage but with different error messages (e.g., all
infrastructurefailures but different cloud API errors). May be a single unstable area with varied surface failures. - Multiple causes: Runs fail at different stages (e.g., some
bootstrap, somecluster creation). Likely multiple independent issues or an intermittently unstable environment. Each failure mode may need separate investigation. - Inconclusive: Skill could not retrieve artifacts or determine root cause for most runs. Note this and continue.
Output for report: Include in Section 4b:
- Number of runs analyzed
- Failure stage breakdown (e.g., "2/3 runs: cluster bootstrap, 1/3 runs: infrastructure")
- Root cause consistency classification
- Per-run summary: job URL, failure stage, key error message
- Overall assessment and recommended next steps
-
Determine Regression Start Date: Analyze
job_runsdata from the regressionThe regression data from step 3 includes a
job_runsarray containing all job runs where the failure was observed throughout the entire life of the regression. Use this data to determine when the regression started without needing a separate API call.Implementation:
# Extract job_runs from regression data (already fetched in step 3) # job_runs are sorted newest-first; reverse to walk oldest-first for start date analysis job_runs=$(echo "$regression_data" | jq '.job_runs') num_job_runs=$(echo "$job_runs" | jq 'length')How to Determine Regression Start Date from
job_runs:-
Walk from oldest to newest: The
job_runsarray is sorted newest-first (index 0 = most recent). Reverse the order conceptually or iterate from the end to find the earliest entries. -
Find the first failure: The earliest
job_runentry (last in the array) represents the earliest observed occurrence of this failure. Itsstart_timegives the approximate regression start date andprowjob_urllinks to that first failing run. -
Analyze the pattern: Walk through the
job_runschronologically to understand:- How quickly failures ramped up after the first occurrence
- Whether failures are concentrated in specific jobs (
prowjob_name) - Whether
test_failurescounts are high in many runs — high values (e.g., >10) indicate mass failure runs where the regressed test may just be caught up in a larger issue (infrastructure instability, a foundational bug, etc.) rather than being the primary problem. If mostjob_runsshow hightest_failures, flag this as a potential mass failure regression.
-
Identify the most affected jobs: Group
job_runsbyprowjob_nameand count occurrences per job. This complements thesample_failed_jobsdata from step 6 with full historical coverage. -
Handle Edge Cases:
- If
job_runsspans the entire tracked period with no gap → the regression may predate the tracked data; note this - If runs are scattered sparsely → likely a flaky test, not a clear regression start
- If pattern is unclear → do not include this in the report
- If
-
Report the Findings (only if a clear start date can be determined):
- Report the
prowjob_urlof the first failure - Report the
start_timedate when that run occurred - Report the total number of runs tracked
- Flag if most runs have high
test_failurescounts (mass failure pattern) - Example: "Regression appears to have started on 2026-01-15 with job run: https://prow.ci.openshift.org/view/gs/..."
- Report the
Fallback: If
job_runsis empty or missing (e.g., regression predates this feature), fall back to using thefetch-test-runsskill with--include-successas before:most_failed_job=$(echo "$regression_data" | jq -r ' .sample_failed_jobs | to_entries | sort_by(.value.failed_runs | length) | reverse | .[0].key ') script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py" job_history=$(python3 "$script_path" "$test_id" --include-success --job-contains "$most_failed_job" --start-days-ago 28 --exclude-output --format json)Output Format (only include if start date can be determined):
Regression Start Analysis: - First Failing Run: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/... - Approximate Start Date: 2026-01-15 - Total Tracked Runs: 47 - Most Affected Job: periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn (18 occurrences) - Mass Failure Pattern: 35/47 runs had >10 test failures — regression may be collateral damage from a larger issue - Pattern: Failures began 2026-01-15, became consistent by 2026-01-17Note: If no clear start date can be determined, skip this section entirely. Do not include inconclusive results.
-
-
Identify Suspect PRs in Payload: Use
fetch-prowjob-jsonandfetch-new-prs-in-payloadskillsOnly perform this step if step 8 successfully identified a clear regression start point (i.e., a first failing run URL exists). If step 8 was skipped or inconclusive, skip this step entirely.
This step identifies pull requests that may have caused the regression by examining what was new in the payload where failures began.
Step 9a: Get the payload tag from the first failing run
Use the
fetch-prowjob-jsonskill to fetch the prowjob.json for the first failing run identified in step 8.# The first_failing_run_url comes from step 8 # Use the fetch-prowjob-json skill to convert to gcsweb URL and fetch # See plugins/ci/skills/fetch-prowjob-json/SKILL.md for URL conversion details # # Extract these annotations: payload_tag=$( ... ) # metadata.annotations["release.openshift.io/tag"] from_tag=$( ... ) # metadata.annotations["release.openshift.io/from-tag"] (may not exist)payload_tag: The payload the cluster was tested against (or upgraded to for upgrade jobs)from_tag: The original version before upgrade (only present on upgrade jobs). If present, note this in the report as it helps distinguish whether a regression is in the upgrade path itself vs the target version.
If the prowjob.json cannot be fetched or the
release.openshift.io/tagannotation is missing (e.g., manually triggered jobs), skip the rest of this step.Step 9b: Fetch new PRs in the payload
script_path="plugins/ci/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py" pr_data=$(python3 "$script_path" "$payload_tag" --format json)See
plugins/ci/skills/fetch-new-prs-in-payload/SKILL.mdfor complete implementation details.Step 9c: Identify potentially related PRs
From the list of new PRs, identify candidates that might be related to the regression. Filter based on:
- Component match: PR component name matches or overlaps with the regression's
componentorcapability - Repository match: PR is from a repo related to the failing test (e.g., a test in
[sig-network]and a PR fromopenshift/ovn-kubernetes) - Title keywords: PR description contains keywords related to the test name, error messages from step 7, or the affected subsystem
- Bug association: PR has a
bug_urlreferencing a fix for something in the same area
Select a maximum of 5 candidate PRs to investigate (to avoid excessive API calls). Prioritize PRs whose component or repo most closely matches the regression.
If no PRs look related based on filtering, state that in the report and skip step 9d.
Step 9d: Check PR details with GitHub CLI
For each candidate PR (up to 5), use the
ghCLI to fetch the PR description and diff summary:# Check if gh CLI is available if command -v gh &>/dev/null; then # Extract org/repo and PR number from the PR URL # e.g., https://github.com/openshift/machine-config-operator/pull/5509 # → owner=openshift, repo=machine-config-operator, pr_number=5509 # Fetch PR details (title, body, changed files) gh pr view "$pr_number" --repo "$owner/$repo" --json title,body,files,labels # Fetch the diff summary (file names and change counts) gh pr diff "$pr_number" --repo "$owner/$repo" --stat else echo "Note: gh CLI not available. Showing PR URLs only - install gh for deeper analysis." fiAnalyze each PR for relevance:
- Read the PR description and diff to determine if the changes could plausibly affect the failing test
- Look for changes to files, packages, or APIs referenced in the test error messages (from step 7)
- Note if the PR modifies test infrastructure, API schemas, or operator behavior relevant to the regression
- Classify each PR as:
- Likely related: Changes directly affect the area where the test is failing
- Possibly related: Changes are in a related subsystem but not directly in the failure path
- Unlikely related: Changes appear unrelated to the test failure
Output Format:
Suspect PRs in Payload (payload: 4.22.0-0.ci-2026-02-06-195709): Upgrade From: 4.22.0-0.ci-2026-02-05-195709 Investigated 3 of 17 new PRs in this payload: 1. [LIKELY] openshift/machine-config-operator#5509 "Set NodeDegraded MCN condition when node state annotation is set to Degraded" Bug: OCPBUGS-67229 Relevance: Changes MCN condition logic which is tested by the failing test Files changed: pkg/controller/node/status.go (+45/-12) 2. [POSSIBLY] openshift/hypershift#7470 "use InfraStatus.APIPort for custom DNS kubeconfig" Bug: OCPBUGS-72258 Relevance: Modifies API port handling, test involves API connectivity 3. [UNLIKELY] openshift/router#707 "Updating openshift-enterprise-haproxy-router-container image" Relevance: Router image update, unrelated to test failure area -
Identify Related Regressions: Use the
list-regressionsskill to find similar failing tests
Use the list-regressions skill with test name filtering to find related regressions that may share the same root cause. Run two queries: one for the exact same test (different variants) and one for similar tests (same namespace/sig).
Implementation:
# Extract test name and release from regression data
test_name=$(echo "$regression_data" | jq -r '.test_name')
release=$(echo "$regression_data" | jq -r '.release')
script_path="plugins/teams/skills/list-regressions/list_regressions.py"
# Query 1: Find regressions for the exact same test across all variants
same_test_regressions=$(python3 "$script_path" --release "$release" --test-name "$test_name")
# Query 2: Find regressions for similar tests (e.g., same namespace)
# Extract the namespace from the test name (e.g., "openshift-machine-config-operator")
# and search for other tests mentioning it
# Choose a distinctive substring from the test name that identifies related tests
similar_test_regressions=$(python3 "$script_path" --release "$release" --test-name-contains "<distinctive_substring>")
--test-name and --test-name-contains cannot be combined with --components or --team — they search across all components automatically.
Analyze Related Regressions:
From the filtered results, identify regressions related to the current one:
- Same test, different variants: From the
--test-namequery — other regressions for the same test but with different variant combinations (e.g., same test failing on bothawsandmetalplatforms) - Similar test names: From the
--test-name-containsquery — regressions with test names that share a common prefix or test suite (e.g., same[sig-api-machinery]tests)
For each related regression found, note:
- Regression ID
- Test name (if different from the current one)
- Variants where it is failing
- Whether it is triaged (has entries in
triagesarray) and to which JIRA bug - Whether it is open or closed
For each related regression that will be included in a bug report or triage, fetch its details to get the test_details_url:
# Fetch details for a related regression to get its test_details_url
related_data=$(python3 "plugins/ci/skills/fetch-regression-details/fetch_regression_details.py" <related_regression_id> --format json)
related_test_details_url=$(echo "$related_data" | jq -r '.test_details_url')
# Convert to UI URL
related_test_details_ui_url=$(echo "$related_test_details_url" | sed 's|https://sippy.dptools.openshift.org/api/component_readiness/test_details|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details|')
Summarize variant patterns:
- Identify if the test is failing across all jobs of one Platform variant (e.g., all
metaljobs) - Identify if failures cluster around a specific Upgrade, Network, or Topology variant
- Note any variant combinations that are NOT failing (helps narrow root cause)
- Find Related Triages and Untriaged Regressions: Use the
fetch-related-triagesskill
Query the Sippy API to find existing triage records and untriaged regressions related to this regression:
script_path="plugins/ci/skills/fetch-related-triages/fetch_related_triages.py"
related=$(python3 "$script_path" <regression_id> --format json)
See plugins/ci/skills/fetch-related-triages/SKILL.md for complete implementation details.
The API returns matches based on similarly named tests and shared job runs (using the complete job_runs history tracked across each regression's lifetime), each with a confidence level (1-10):
- 10: High confidence — same or very closely related tests
- 5: Medium confidence — similarly named tests matched by edit distance
- 2: Low confidence — regressions sharing the same job runs (may indicate shared root cause, but could be coincidental in mass failure scenarios)
Analyze the results:
-
triaged_matches: Existing triage records that look related. For each match:- Note the
triage_id,jira_key,jira_status, andconfidence_level - High confidence matches (>=5) with open JIRA bugs are strong candidates for adding this regression to
- Low confidence matches or closed JIRA bugs are informational
- Present these in the report as potential triage targets
- Note the
-
untriaged_regressions: Open regressions not yet triaged that appear related. These are candidates to be triaged together with the current regression under one bug. For each:- Note the
match_reason(similarly_named_testorsame_last_failure) similarly_named_testwith lowedit_distance(0-2) are likely the same or very similar tests in different variantssame_last_failureregressions may share the same root cause (same failing job runs) but could be from different components
- Note the
Combine with step 10 results: Merge these findings with the related regressions found via --test-name in step 10. The two sources are complementary:
- Step 10 finds regressions for the exact same test name (different variants)
- This step finds regressions with similar test names AND existing triages that may already cover the issue
- Prepare Bug Filing Recommendations or Existing Bug Status: Generate actionable information
Important: Component Readiness regressions are treated as release blockers. Any bug filed for a regression should be conveyed to the user as a release blocker. See the release blocker definition for details on the criteria and process.
If regression is NOT triaged (no existing JIRA):
- Component assignment (from test mappings)
- If step 4's test report found
open_bugs > 0, note these existing bugs — one may be suitable for triaging this regression without filing a duplicate - Bug summary suggestion based on failure pattern (informed by step 6 pattern analysis)
- Bug description template including:
- Test name(s) and release — use the full test name including all tags (e.g.,
[Suite:openshift/conformance/parallel],[Serial],[Conformance]). Format each test name on its own line wrapped in Jira{code}blocks so that tooling can find bugs by test name:{code} [Jira:"Test Framework"] there should not be mass test failures {code} - Test ID (
test_id— the BigQuery/Component Readiness ID, e.g.,openshift-tests:abc123) - Regression ID(s) — the Component Readiness regression ID(s) being triaged
- Regression opened date
- Affected variants
- Failure patterns identified (permafail/flaky/resolved/recent)
- Pass sequence analysis from step 6
- Global test report from step 4: overall pass rate, per-variant breakdown, open bugs
- Failure output consistency analysis from step 7 (if available):
- Common error message
- Consistency percentage
- Key debugging information (file references, resources, stack traces)
- Sippy Test Details report links - this is critical for debugging:
- Link for the current regression (converted
test_details_ui_url) - Links for each related regression found in step 10 (each regression has its own
test_details_urlfrom the list-regressions data - convert each to UI URL)
- Link for the current regression (converted
- Regression start date (if determined in step 8)
- Suspect PRs from payload analysis (if determined in step 9) — include PR URLs and relevance classification for LIKELY and POSSIBLY related PRs
- Related regressions (if any) with their regression IDs and test names
- Test name(s) and release — use the full test name including all tags (e.g.,
- Triage type recommendation:
product: actual product issues (default)test: clear issues in the test itself (especially for flaky patterns)ci-infra: CI outagesproduct-infra: customer-facing outages (e.g., quay)
If regression IS triaged (existing JIRA):
- Note: "A bug already exists for this regression - do not file a duplicate"
- Display JIRA key(s) with links
- Show JIRA progress analysis from step 5:
- 🟢 ACTIVE: Bug is being worked on, no action needed
- 🟡 STALLED: Bug may need attention, consider commenting or reassigning
- 🔴 NEEDS ATTENTION: Bug appears abandoned, consider taking ownership or escalating
- If analysis_status == -1000 (failed fix):
- Recommend: Re-open the existing bug OR file a new bug if the failure mode is different
- Provide new bug template if failure analysis suggests a different root cause
- Display Comprehensive Report: Present findings in clear format
Section 1: Regression Summary
- Test name
- Component
- Regression opened/closed dates
- Affected variants
- Analysis status and explanations
- Current triage status (none for untriaged regressions)
- Global test report (from step 4): overall pass rate, trend, open bugs count
- If open bugs exist, note them as potential triage targets
Section 2: Global Test Health (from fetch-test-report skill with --no-collapse)
- Overall pass rate across all variants for this release
- Per-variant breakdown highlighting variants with low pass rates
- Open bugs count — if > 0, list them as potential triage targets
- Trend direction (improving/regressing/unchanged)
Section 3: Failure Pattern Analysis
- Jobs ranked by number of failures (most to least impacted)
- For each job:
- Job name
- Total failed runs
- Pass sequence visualization
- Pattern classification:
- Permafail: All or most recent runs failing
- Resolved: Recent runs succeeding after a block of failures
- Flaky: Sporadic failures interspersed with successes
- Recent Regression: Recently started failing after consistent successes
- Priority level (High/Medium/Low)
- Recommended action
- Overall assessment of regression severity
Section 4: Failure Output Analysis (from fetch-test-runs skill)
- Number of test outputs analyzed
- Consistency classification (Highly Consistent / Moderately Consistent / Inconsistent)
- Most common error message with occurrence count
- Key debugging information:
- File/line references
- Resource or API paths
- Error messages
- Sample job URLs for manual inspection
- If the test outputs API is not available, this section states: "Test output analysis not available"
Section 4a: Job Run Context (from fetch-job-run-summary skill)
- Number of job runs analyzed and their individual failure counts
- Whether the regressed test is an isolated failure or part of broader co-failures
- Whether the same tests consistently co-fail across runs
- Dominant error pattern with percentage (if applicable)
- For mass test failure regressions: real affected component(s) based on which tests are failing
- Assessment: isolated test issue, targeted product bug with co-failures, or collateral damage from widespread instability
Section 5: Regression Start Analysis (only if determinable)
- Job analyzed (the job with most failures)
- Approximate start date of the regression
- URL of the first failing job run
- Pattern description (e.g., "18 consecutive failures followed by 12 successes")
- Note: This section is omitted if no clear start date can be determined
Section 6: Suspect PRs in Payload (only if regression start was determined)
- Payload tag and upgrade-from tag (if applicable)
- Total number of new PRs in the payload
- Number of PRs investigated (up to 5)
- For each investigated PR:
- Relevance classification: LIKELY, POSSIBLY, or UNLIKELY
- PR URL, title, and associated bug (if any)
- Brief explanation of why it may or may not be related
- Key files changed (from
gh pr diff --stat)
- This section is omitted if step 8 did not determine a clear regression start, if the prowjob.json lacks release annotations, or if no PRs looked potentially related. It is also omitted if
ghCLI is not available, though PR URLs are still listed.
Section 7: Related Regressions and Existing Triages (from fetch-related-triages skill and list-regressions --test-name)
- Existing triages that may cover this regression, ranked by confidence level (10 = high, 5 = medium, 2 = low)
- For each existing triage: JIRA key, status, summary, triage type, confidence level, triage UI link
- Untriaged regressions that appear related (same/similar tests, shared failure times)
- Regressions for the exact same test in different variants (from step 10)
- Recommendation: whether to add to an existing triage or file a new bug
- Offer to Triage: After presenting the report, offer to triage the regression
Based on the analysis, determine the appropriate triage action and ask the user if they want to proceed.
Generating the triage description (for new triage records only): Every new triage record must include a --description. Generate a single short sentence (under 120 characters) summarizing the failure — similar to a JIRA bug summary title. Be concise; do not use more than one sentence. Example: "InsightsDataGather CRD not found - InsightsRuntimeExtractor tests failing since Feb 6"
Scenario A: Related triage record found on another regression (from step 11)
If step 11 found that a related regression already has a triage record (i.e., another regression for the same or similar test is already triaged to a JIRA bug), offer to add this regression to that existing triage. Also include any other untriaged related regressions found in steps 10 and 11.
A related triage already exists:
- Triage ID: 789
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-12345
- Type: product
The following untriaged regressions could be added to this triage:
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)
Would you like to add these regressions to the existing triage?
If the user confirms, use the triage-regression skill to update the existing triage:
# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")
# Only pass the new regression IDs to add - the script automatically
# fetches existing regressions and merges them (safe additive behavior)
new_regression_ids="<current_id>,<related_id_1>,<related_id_2>"
script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$new_regression_ids" \
--token "$TOKEN" \
--triage-id <existing_triage_id> \
--url "<existing_jira_url>" \
--type "<existing_triage_type>" \
--format json)
After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:
triage_id=$(echo "$triage_result" | jq -r '.triage.id')
Display to the user:
Triage updated:
- JIRA: <existing_jira_url>
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>
Then add the triage record link to the JIRA description using the add-jira-triage-link skill:
jira_key="<existing_jira_key>"
link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "$jira_key" --triage-id "$triage_id" --format json
If successful, report that the triage link was added to the JIRA description. If it was already present, report that it already exists. If JIRA credentials are not set, skip this step silently (the triage itself already succeeded).
The triage-regression script automatically fetches the existing triage and merges its regressions with the new ones, so you only need to pass the regression IDs you want to add.
Scenario B: JIRA bug found but not triaged to any regression (from step 5 or step 11)
If step 5 found a linked JIRA bug on this regression's triage, or step 11 found a JIRA bug that looks related, or step 4's test report found open_bugs > 0 (e.g., same component, similar error pattern) but no triage record exists yet, offer to create a new triage linking this regression and all related untriaged regressions to that bug.
A related JIRA bug was found:
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-67890
- Summary: [bug summary from JIRA]
The following regressions could be triaged to this bug:
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)
Recommended triage type: product
Would you like to create a triage record linking these regressions to this bug?
If the user confirms, use the triage-regression skill to create a new triage:
# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")
all_regression_ids="<current_id>,<related_id_1>,<related_id_2>"
# Generate a concise description from the analysis (see note below)
description="<generated_description>"
script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$all_regression_ids" \
--token "$TOKEN" \
--url "https://redhat.atlassian.net/browse/OCPBUGS-67890" \
--type product \
--description "$description" \
--format json)
After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:
triage_id=$(echo "$triage_result" | jq -r '.triage.id')
Display to the user:
Triage created:
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-67890
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>
Then add the triage record link to the JIRA description using the add-jira-triage-link skill:
link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "OCPBUGS-67890" --triage-id "$triage_id" --format json
If successful, report that the triage link was added to the JIRA description. If JIRA credentials are not set, skip this step silently.
Scenario C: No related triage or bug found
If no related triage record or JIRA bug was found, and the regression is untriaged, offer to create a new JIRA bug and triage all related regressions to it.
No existing triage or related JIRA bug was found for this regression.
Would you like me to create a JIRA bug and triage the following regressions to it?
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)
Proposed bug details:
- Project: OCPBUGS
- Component: <component from regression data>
- Summary: <suggested summary from step 12>
- Triage type: <recommended type>
If the user confirms, create the bug using /jira:create bug with the bug template from step 12. Apply the label component-regression to the bug (this label identifies bugs found through Component Readiness). The bug description must include:
- Test name(s) — the full name including all tags (e.g.,
[Suite:openshift/conformance/parallel],[Serial]). Each test name must be on its own line wrapped in Jira{code}blocks so that tooling can search for bugs mentioning a specific test:{code} [Monitor:kubelet-container-restarts][sig-architecture] platform pods in ns/openshift-kube-apiserver should not exit an excessive amount of times {code} - Test and regression identifiers collected in step 12
- Release
- Regression opened date
- Affected variants
- Failure pattern analysis summary
- Common error message (if available from step 7)
- Sippy Test Details report links for each regression being triaged (convert each
test_details_urlfrom API to UI URL) - Related regression IDs and test names
After the bug is created, mark it as a release blocker using the set-release-blocker skill (component readiness regressions are release blockers):
script_path="plugins/ci/skills/set-release-blocker/set_release_blocker.py"
python3 "$script_path" "<new_bug_key>" --format json
See plugins/ci/skills/set-release-blocker/SKILL.md for details.
Then use the triage-regression skill to triage all regressions to the new bug:
# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")
all_regression_ids="<current_id>,<related_id_1>,<related_id_2>"
# Generate a concise description from the analysis (see note below)
description="<generated_description>"
script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$all_regression_ids" \
--token "$TOKEN" \
--url "https://redhat.atlassian.net/browse/<new_bug_key>" \
--type <recommended_type> \
--description "$description" \
--format json)
After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:
triage_id=$(echo "$triage_result" | jq -r '.triage.id')
Display to the user:
Bug filed and triaged:
- JIRA: https://redhat.atlassian.net/browse/<new_bug_key>
- Release Blocker: Approved
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>
Then add the triage record link to the JIRA description using the add-jira-triage-link skill:
link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "<new_bug_key>" --triage-id "$triage_id" --format json
If successful, report that the triage link was added to the JIRA description. If JIRA credentials are not set, skip this step silently.
Scenario D: Regression is already triaged
Name
ci:check-if-jira-regression-is-ongoing
Synopsis
/ci:check-if-jira-regression-is-ongoing <jira-key-or-url>
Description
The ci:check-if-jira-regression-is-ongoing command reads a Jira bug and determines whether the regression(s) it describes are still actively failing or have resolved. It extracts test names, test IDs, and regression IDs from the bug description, status, and comments, then cross-references with live data from Sippy and Component Readiness.
This command is useful for:
- Checking if a bug's regression has resolved and the bug can be closed
- Finding out if new regressions have appeared for the same test since the bug was filed
- Discovering if other triaged bugs exist for the same test that don't match this bug
- Getting a quick status on whether a fix has landed and is effective
Implementation
-
Parse Arguments: Extract the Jira key
Accept either a full URL or a bare key:
https://redhat.atlassian.net/browse/OCPBUGS-74690→OCPBUGS-74690OCPBUGS-74690→OCPBUGS-74690
Strip the URL prefix if present:
jira_key=$(echo "$input" | sed 's|.*/browse/||') -
Fetch Jira Issue: Read the bug description, status, and comments
Use the
fetch-jira-issueskill to retrieve the bug data:jira_script="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py" jira_data=$(python3 "$jira_script" "$jira_key" --format json)See
plugins/ci/skills/fetch-jira-issue/SKILL.mdfor complete implementation details including error handling.Extract and read from the JSON output:
summary: Bug titlecomments[].body: All comment bodiesstatus: Current Jira status (e.g., New, Assigned, Modified, Closed)components[]: Jira component(s)progress.level: Automatic progress classification (ACTIVE/STALLED/NEEDS_ATTENTION/RESOLVED)
-
Extract Test Identifiers: Scan the bug for test names, test IDs, and regression IDs
Search through the summary, description, and all comment bodies for:
- Test IDs (
test_id): Match the patternsuite-name:hex-hash(e.g.,openshift-tests:bb3a7d828630760296ef203c5cacf708). These are BigQuery/Component Readiness test IDs. - Regression IDs: Match standalone integers that appear in context suggesting they are regression IDs (e.g., "regression 34446", "Regression ID: 34446", or in Sippy URLs containing
/regressions/). Also extract from Component Readiness URLs likesippy.dptools.openshift.org/.... - Test names: Match strings in the format
[sig-*] ...which are full OpenShift CI test names (e.g.,[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]).
If no test ID, test name, or regression ID can be found, abort:
Error: Could not identify any test name, test ID, or regression ID in OCPBUGS-74690. Cannot check regression status without at least one of these identifiers.If multiple identifiers are found, use all of them — the bug may cover multiple tests/regressions.
- Test IDs (
-
Determine Release: Get the release version
Check if the bug description or title mentions a release version (e.g., "4.22", "4.21"). If not, use the
fetch-releasesskill:release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest) -
Check Regression IDs (if found): Query each regression's current status
For each regression ID extracted from the bug:
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py" regression_data=$(python3 "$script_path" <regression_id> --format json)For each regression, determine:
- Status: Is it open or closed?
- If open: How long has it been open? (
openedtimestamp to now) - If closed: When did it close? (
closedtimestamp). How long was it open? - Triages: What bugs are triaged against it? Do any of the triage
jira_keyvalues match our bug? If a triage points to a different bug, flag this for the user. - Pass sequences: Check
sample_failed_jobspass sequences. Are recent runs (left side) passing or failing?
-
Check Test Names and Test IDs Against Open Regressions: Find if the test appears in any current regressions
Use the
list-regressionsskill with--test-nameto fetch regressions matching this test:script_path="plugins/teams/skills/list-regressions/list_regressions.py" matching_regressions=$(python3 "$script_path" --release "$release" --test-name "$test_name")This returns only regressions for the exact test name across all components and variants, without needing to fetch and parse the full regression list.
For each matching regression:
- Note whether it is open or closed
- Note when it was opened and (if closed) when it was closed
- Check if it is triaged and to which Jira bug
- If triaged to a different bug than the one we're investigating, flag this:
⚠ Regression <id> is triaged to OCPBUGS-XXXXX (not this bug OCPBUGS-74690). This may indicate a duplicate or a related but separate issue. - Note the variants affected
If the bug contained a
test_idbut notest_name, use any regression match to obtain the test name for subsequent steps. -
Check Global Test Report: Use
fetch-test-reportwith--no-collapsescript_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py" test_report=$(python3 "$script_path" "$test_name" --release "$release" --no-collapse --format json)Analyze:
- Overall pass rate: Is the test healthy (>99%) or still failing?
- Per-variant breakdown: Which specific variant combinations are still failing? Which have recovered?
- Trend (
net_working_improvement): Is the test improving (positive) or getting worse (negative)? - Open bugs count: Are there other open bugs mentioning this test?
- Comparison to bug context: If the bug mentioned specific variants (e.g., "metal-ipi", "aws"), check those variants specifically.
-
Check Recent Test Runs: Use
fetch-test-runsto see current failure activityscript_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py" recent_runs=$(python3 "$script_path" "$test_id" --include-success --format json)Analyze:
- Recent pass/fail ratio: Count successes vs failures in recent runs
- Mass failure detection: Check
failed_testsfield — iffailed_tests > 10, the test may be caught up in broader infrastructure failures rather than a test-specific regression - Error consistency: Are the failure outputs consistent with what the bug describes?
- Last failure date: When was the most recent failure? If all recent runs are passing, the regression may have resolved.
-
Synthesize and Present Report: Summarize the findings
Present a clear report answering the core question: Is the regression described in this bug still ongoing?
Report Format:
Regression Status Check: OCPBUGS-74690 ============================================================ Bug: <summary> Status: <jira status> Component: <component> Identifiers Found: - Test Name: <test name> - Test ID: <test_id> - Regression IDs: <list> --- Regression Status --- [For each regression ID checked:] Regression <id>: OPEN / CLOSED Opened: <date> (<N days ago>) Closed: <date> (<N days ago>) [if closed] Triaged to: <jira key> [matches this bug / DIFFERENT BUG ⚠] Recent Pattern: <pass_sequence summary> --- Current Regressions for This Test --- [For each matching regression in current release:] Regression <id>: OPEN / CLOSED Variants: <variant list> Triaged: Yes → OCPBUGS-XXXXX / No [If triaged to different bug: ⚠ Triaged to OCPBUGS-XXXXX, not this bug] --- Global Test Health --- Overall Pass Rate: <X>% (trend: improving/regressing) Open Bugs: <N> Per-Variant Breakdown: <variant combo>: <pass rate>% [FAILING / HEALTHY] <variant combo>: <pass rate>% [FAILING / HEALTHY] ... --- Recent Test Runs --- Total runs: <N> (Passes: <N>, Failures: <N>) Mass failure runs: <N> of <N> failures had >10 other test failures Last failure: <date> Last success: <date> --- Conclusion --- <One of the following assessments:> ✅ RESOLVED: The regression appears to have resolved. The test has been passing consistently since <date>. The global pass rate is <X>%. [Recommend closing the bug if no further action needed.] ⚠ PARTIALLY RESOLVED: The test is passing in most variants but still failing in <variant list>. Pass rate is <X>% overall. [Recommend keeping the bug open until all variants recover.] ❌ STILL ONGOING: The regression is still actively failing. The test has a <X>% pass rate with <N> failures in the last 7 days. [The bug should remain open.] 🔄 INCONCLUSIVE: Could not definitively determine regression status. <reason — e.g., mass failures obscuring results, test not running recently>If the bug is Closed or Verified but the regression is STILL ONGOING or PARTIALLY RESOLVED:
This indicates the fix was insufficient or the problem has recurred. Offer to reopen the bug:
⚠ FAILED FIX: This bug is <Closed/Verified> but the regression is still ongoing. The test has a <X>% pass rate with <N> recent failures. Would you like me to: 1. Add a comment explaining the regression is still active 2. Reopen the bug by moving status back to ASSIGNEDIf the user confirms, perform both actions:
Add a comment explaining the situation, including:
- Current pass rate and failure count
- Which variants are still failing
- Link to the Sippy test details report
- A statement that the regression was detected as still ongoing by the check command
# Add comment to the bug JIRA_AUTH=$(printf '%s' "$JIRA_USERNAME:$JIRA_API_TOKEN" | base64 | tr -d '\r\n') curl -s -X POST \ -H "Authorization: Basic $JIRA_AUTH" \ -H "Content-Type: application/json" \ -d '{"body": "<comment text>"}' \ "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/comment"Transition the bug back to ASSIGNED:
First, fetch available transitions to find the correct transition ID:
transitions=$(curl -s -H "Authorization: Basic $JIRA_AUTH" \ "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/transitions")Look through the transitions for one that moves to "Assigned" status, then execute it:
curl -s -X POST \ -H "Authorization: Basic $JIRA_AUTH" \ -H "Content-Type: application/json" \ -d '{"transition": {"id": "<transition_id>"}}' \ "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/transitions"Display confirmation:
Bug reopened: - JIRA: https://redhat.atlassian.net/browse/<jira_key> - Status: ASSIGNED - Comment added explaining regression is still active
Return Value
- Format: Human-readable summary report
- Key information:
- Whether the regression is still ongoing, resolved, or partially resolved
- When the regression appears to have resolved (if applicable)
- Current pass rates per variant
- Whether other regressions or bugs exist for the same test
- Flags for regressions triaged to different bugs
Examples
-
Check a bug by URL:
/ci:check-if-jira-regression-is-ongoing https://redhat.atlassian.net/browse/OCPBUGS-74690 -
Check a bug by key:
/ci:check-if-jira-regression-is-ongoing OCPBUGS-74690
Arguments
$1(required): Jira issue key or URL- Format: Either a bare key (
OCPBUGS-74690) or a full URL (https://redhat.atlassian.net/browse/OCPBUGS-74690)
- Format: Either a bare key (
Prerequisites
-
JIRA_USERNAME: Required for Jira authentication
- Set environment variable:
export JIRA_USERNAME="your-atlassian-email"
- Set environment variable:
-
JIRA_API_TOKEN: Required to read the Jira issue
- Set environment variable:
export JIRA_API_TOKEN="your-jira-api-token" - Obtain from: https://id.atlassian.com/manage-profile/security/api-tokens
- Set environment variable:
-
Python 3: Required for all skill scripts
- Version: 3.6 or later
Skills Used
fetch-regression-details: Checks regression status (open/closed), triages, and pass sequencesfetch-test-report: Gets global test health with per-variant breakdownfetch-test-runs: Gets recent test run results to check current failure activityfetch-releases: Determines the latest OCP release when not specified in the buglist-regressions(teams plugin): Lists all regressions for a release to find matches by test name/ID
See Also
- Related Command:
/ci:analyze-regression— Full regression analysis workflow - Related Command:
/ci:fetch-test-report— Fetch test health report by name - Related Skill:
fetch-regression-details(plugins/ci/skills/fetch-regression-details/SKILL.md) - Related Skill:
fetch-test-report(plugins/ci/skills/fetch-test-report/SKILL.md) - Related Skill:
fetch-test-runs(plugins/ci/skills/fetch-test-runs/SKILL.md) - Related Skill:
fetch-releases(plugins/ci/skills/fetch-releases/SKILL.md) - Related Skill:
list-regressions(plugins/teams/skills/list-regressions/SKILL.md)
Name
ci:continue-session
Synopsis
/ci:continue-session <prowjob-url>
Description
Downloads Claude session artifacts from a Prow CI job and allows you to continue one of those sessions locally. This is useful when Claude ran as part of a CI job and you want to pick up where it left off.
The command accepts:
- prowjob-url (required): URL to a Prow job run
Prerequisites
-
gcloud CLI — for accessing GCS artifacts
- Check:
which gcloud - The
test-platform-results-publicbucket is publicly accessible (no auth needed)
- Check:
-
Python 3 (3.7+)
- Check:
which python3
- Check:
-
tar — for extracting archives
Implementation
Step 1: Search for session archives
Load the ci:prow-job-analysis skill and use its artifact search script (prow_job_artifact_search.py) to find files matching **/claude-session*.tar* in the job's artifacts.
Parse the JSON output. If no matches are found (count: 0), tell the user "No Claude session archives found in this job's artifacts." and stop.
Step 2: Download archives to current directory
For each matched archive file, download it from GCS using gcloud storage cp:
gcloud storage cp --no-user-output-enabled <gcs_uri> ./
Step 3: Download payload artifacts
Search for payload artifacts using the ci:prow-job-analysis skill's artifact search script with the pattern **/payload*.yaml, **/payload*.json, and **/payload*.html.
For each matched file, download it to the current directory:
gcloud storage cp --no-user-output-enabled <gcs_uri> ./
These are supplementary artifacts — if none are found, continue without error.
Step 4: Extract archives
For each downloaded archive in the current directory:
# tar auto-detects compression format (.tar, .tar.gz, .tar.zst, .tar.bz2, etc.)
tar xf <filename>
After extraction, the structure contains a projects/ directory. Each session consists of two parts:
- A JSONL file at
projects/<project-path>/<session-uuid>.jsonl(the main conversation log) - A directory at
projects/<project-path>/<session-uuid>/(containing subagent data and other session state)
Both must be copied together for the session to work.
Step 5: Discover sessions and extract first user message
Find all session JSONL files (files matching *.jsonl under the extracted projects/ hierarchy). Each .jsonl file corresponds to a session, and its filename (without extension) is the session UUID.
For each session JSONL file found:
- Read the JSONL file line by line
- Look for the first message with
"type": "user"that does NOT have"isMeta": true— this is the first real user message - Extract the text content from
message.content— it may be a string or an array of content blocks. If it's an array, look for blocks with"type": "text"and extract thetextfield. - Truncate the message to 120 characters for display purposes
- Record the session UUID, the full path to the JSONL file, and the truncated first user message
If no sessions are found after extraction, tell the user "No Claude sessions found in the extracted archives." and stop.
Step 6: Present session list to user
If only one session is found, display its details and confirm with the user before proceeding.
If multiple sessions are found, display a numbered list asking "What session would you like to continue?" with each entry showing:
- The session number
- The session UUID
- The truncated first user message
Example:
What session would you like to continue?
1. [abc12345-...] "Please analyze the test failures in..."
2. [def67890-...] "Can you help fix the flaky e2e test..."
Use AskUserQuestion to let the user select a session by number.
Step 7: Copy selected session to local Claude projects directory
The extracted archive contains sessions under a CI-specific project path (e.g., projects/-home-prow-go-src-github.com-org-repo/). This path reflects where the repo was checked out in CI and will not match the local project path.
Determine the local project path that Claude uses for the current working directory. Claude's projects directory structure encodes the absolute path of the project by replacing both / and . with - and stripping the leading slash. For example, if the current working directory is /Users/stbenjam/git/github.com/my-repo, the local project path is:
~/.claude/projects/-Users-stbenjam-git-github-com-my-repo/
Compute this by taking pwd and replacing all / and . with -. Since absolute paths start with /, this naturally produces a leading -.
Before copying, check if the session UUID already exists locally (either the .jsonl file or the directory). If it does, warn the user that a local session with this UUID already exists and ask whether to overwrite it. If the user declines, stop.
Copy both the selected session's JSONL file and its corresponding directory into the local project path:
# Determine local project path
LOCAL_PROJECT_DIR="$HOME/.claude/projects/$(pwd | tr '/.' '--')"
mkdir -p "$LOCAL_PROJECT_DIR"
# Copy the session JSONL file
cp <path-to-extracted>/<session-uuid>.jsonl "$LOCAL_PROJECT_DIR/"
# Copy the session directory (contains subagent data)
cp -r <path-to-extracted>/<session-uuid> "$LOCAL_PROJECT_DIR/"
Only the single selected session is copied — do not copy other sessions. Both the .jsonl file and the UUID directory must be copied for the session to be resumable.
Step 8: Report success
Tell the user:
You are all set! You can now continue the session by launching a new Claude:
claude --resume <session-uuid>
Where <session-uuid> is the UUID of the session they selected.
If any payload artifacts were downloaded, list them so the user knows they are available in the current directory.
Arguments
- $1 (prowjob-url): URL to a Prow job run (required). Accepts standard Prow UI URLs or gcsweb URLs.
Name
ci:detect-permafail
Synopsis
/ci:detect-permafail --job-urls="[url1,url2,...]" --job-name="job-name" --pr="owner/repo#123"
Description
Analyzes 2-10 consecutive failures of the same job to determine if they represent a systematic/permanent failure (permafail) versus a flaky failure. This is critical for CI/CD pipeline analysis to distinguish between:
- Permafail: A systematic failure affecting the same test(s) or infrastructure issue, detected by analyzing comparable runs (same failure type):
- Separates test failures from infrastructure failures
- Applies thresholds based on comparable run count (not total URLs)
- 2-3 comparable runs: All must match (100% required)
- 4-5 comparable runs: At least 4 must match (80% required)
- 6-10 comparable runs: At least ceil(count×0.7) must match (70% required)
- Example: 7 URLs where 5 are test failures and 4 of those 5 have same test → 4/5 = 80% → PERMAFAIL
- Flaky: Non-deterministic failures with varying root causes, or failures that don't meet the permafail thresholds
How It Works
The command uses a deterministic Python classifier script for artifact-based failure classification, compares failure signatures across runs, and returns a JSON result with the permafail verdict, confidence score, failure type, and detailed signatures for each run.
Implementation
- Load the "detect-permafail" skill
- Pass the job URLs, job name, and PR info to the skill
- The skill handles all analysis including:
- Artifact fetching and classification via
plugins/ci/scripts/classify-job-failures.py - Signature comparison and pattern detection
- Threshold-based permafail determination
- Artifact fetching and classification via
Arguments
--job-urls: JSON array of 2-10 consecutive Prow job URLs (required, newest first)--job-name: Name of the job being analyzed (required, e.g., "e2e-aws-ovn")--pr: PR identifier (required, format: "owner/repo#number")
Prerequisites
- Access to Prow CI job artifacts via gcsweb URLs
- Access to Bash for running the classifier script
- Python with the
requestspackage available
Name
ci:extract-kubeconfig
Synopsis
/ci:extract-kubeconfig <pr-url>
Description
The ci:extract-kubeconfig command extracts the kubeconfig from a running rehearsal/CI job in a GitHub PR. It checks the job's step status to determine if the cluster is ready, then connects to the build cluster to extract the kubeconfig from the running pod.
The command accepts:
- PR URL (required): GitHub PR URL in the
openshift/releaserepository
What it does:
- Validates the PR URL and checks ownership
- Finds running (PENDING) jobs on the PR
- Determines the build cluster (via GCS for public jobs, via job config for qe-private-deck jobs, or by asking the user for the console URL or cluster name as a last resort)
- Logs into the build cluster and finds the CI namespace
- Checks step status via pod statuses to verify the cluster is ready
- Extracts the kubeconfig (and nested kubeconfig for HyperShift) from the running pod
- Verifies cluster health and reports results
Key features:
- Supports both standard and HyperShift (nested kubeconfig) clusters
- Supports both public (
prow.ci.openshift.org) and private (qe-private-deck) jobs - Uses ci-op namespace prefix on kubeconfig filenames to support multiple concurrent extractions
- Warns if you're not the PR author (may lack namespace access)
- Recognizes
waitsteps as valid running states for extraction
Prerequisites
-
GitHub CLI (
gh)- Check if installed:
which gh - Must be authenticated:
gh auth status
- Check if installed:
-
OpenShift CLI (
oc)- Check if installed:
which oc - Used to login to build clusters and extract kubeconfig
- Check if installed:
-
Google Cloud Storage (
gsutil) (optional)- Check if installed:
which gsutil - No authentication needed (test-platform-results-public bucket is public)
- If not installed, the command falls back to finding the build cluster from the job config in the openshift/release repo
- Check if installed:
-
PR in openshift/release repository
- The command only supports PRs in the
openshift/releaserepository - You should be the PR author to have access to the CI namespace on the build cluster
- The command only supports PRs in the
Implementation
The command performs the following steps:
Step 1: Validate PR URL
Parse the PR URL to extract the org, repo, and PR number. The URL should match the format https://github.com/<org>/<repo>/pull/<number>.
If the repo is NOT openshift/release, warn the user and stop.
Step 2: Check PR Ownership
Fetch the PR author and compare with the current GitHub user:
gh pr view <number> --repo openshift/release --json author --jq '.author.login'
gh api user --jq '.login'
If the PR author does NOT match the current user, warn: "You are not the author of this PR. You may not have access to the CI namespace on the build cluster to extract the kubeconfig." Ask the user if they want to continue. Stop if they say no.
Step 3: Find Running Jobs
Fetch the PR status checks:
gh pr view <number> --repo openshift/release --json title,state,statusCheckRollup --jq '
.statusCheckRollup[]
| select(.__typename == "StatusContext" and .state == "PENDING" and (.context | test("^tide$") | not))
| {context, state, targetUrl}
'
- If multiple running jobs found, list them and ask the user which one to use.
- If exactly one running job found, use it automatically.
- If no running jobs found, report that and stop.
Step 4: Determine the Build Cluster
From the matching check, extract the targetUrl which points to the Prow job page. Determine the build cluster using one of three methods:
Method A — Via GCS prowjob.json (preferred when gsutil is installed and the job uses public prow.ci.openshift.org):
Parse the GCS path from targetUrl:
- Format:
/view/gs/<bucket>/...(Prow UI) or/gcs/<bucket>/...(gcsweb) - Extract
<bucket>and the object path after it - Download from
gs://<bucket>/<object-path>/prowjob.json:gsutil cp "gs://<bucket>/<object-path>/prowjob.json" /tmp/prowjob.json
Method B — Via job config in openshift/release repo (fallback when gsutil is not installed, or when the job uses qe-private-deck):
Find the build cluster from the generated job config in the openshift/release repo using the PR's fork and branch:
-
Get the PR's head ref info:
gh pr view <number> --repo openshift/release --json headRefName,headRepository,headRepositoryOwner -
Determine the original job name. For rehearsal jobs, the check context is like
ci/rehearse/periodic-ci-<org>-<repo>-.... Strip theci/rehearse/prefix to get the rehearsal job name, then striprehearse-<pr_number>-to get the original job name. -
Parse the org from the original job name (first segment after
periodic-ci-orpull-ci-). -
List repo directories under
ci-operator/jobs/<org>/via the GitHub Contents API on the PR's fork/branch:gh api "repos/<owner>/<repo>/contents/ci-operator/jobs/<org>?ref=<branch>" --jq '.[].name'Match the longest directory name that is a prefix of the job name remainder (after stripping
periodic-ci-<org>-). -
List files in that directory matching
*-periodics.yamland download the matching file via raw URL:curl -sL "https://raw.githubusercontent.com/<owner>/<repo>/<branch>/ci-operator/jobs/<org>/<repo_dir>/<periodics_file>" \ | grep -B 30 "name: <original_job_name>" | grep "cluster:"
Method C — Ask the user (fallback when Methods A and B both fail):
This happens when:
- The job uses
qe-private-deck(gsutil gets 403 AccessDeniedException on the private GCS bucket) - The generated job config has no
cluster:field (build cluster is dynamically assigned by the Prow dispatcher)
In this case, ask the user to provide the OpenShift console URL from the build cluster where the job is running. The user can find this on the Prow job page (qe-private-deck) — it typically shows a link to the build cluster console or displays the cluster name in the job metadata.
Unable to determine the build cluster automatically:
- GCS bucket "qe-private-deck" requires authentication
- Job config has no cluster: field (dynamically assigned)
Please provide the OpenShift console URL of the build cluster from the Prow job page.
Example: https://console-openshift-console.apps.build09.ci.devcluster.openshift.com/
Alternatively, provide just the build cluster name (e.g., build09):
Extract the cluster name from the console URL:
- Pattern:
https://console-openshift-console.apps.<cluster>.ci.devcluster.openshift.com/... - Extract
<cluster>(e.g.,build09,build01,build11)
If the user provides a plain cluster name (e.g., build09) instead of a URL, use it directly.
Validate the cluster identifier before constructing the login URL. The cluster name must match the pattern build[0-9]+ (e.g., build01, build09, build12). Reject any input that does not match — this prevents credential redirection to unexpected hosts.
Construct the API server URL: https://api.<cluster>.ci.devcluster.openshift.com:6443
→ Continue to Step 5: Login to the Build Cluster
Step 5: Login to the Build Cluster
IMPORTANT: Use a dedicated temporary kubeconfig file for all build cluster operations. This prevents modifying the user's current kubeconfig context in ~/.kube/config, which could disrupt other processes (e.g., running oc commands, controllers, or scripts) that rely on the current context.
Create a temporary kubeconfig file path and set the target server:
BUILD_KUBECONFIG="/tmp/build-cluster-kubeconfig.yaml"
TARGET_SERVER="https://api.<cluster>.ci.devcluster.openshift.com:6443"
Where <cluster> is the build cluster determined in Step 4.
If the temp kubeconfig already exists (it may contain contexts from multiple build clusters from previous runs), find a context matching the target server:
MATCHING_CTX=""
for ctx in $(KUBECONFIG="$BUILD_KUBECONFIG" oc config get-contexts -o name 2>/dev/null); do
server=$(KUBECONFIG="$BUILD_KUBECONFIG" oc --context="$ctx" config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null)
if [ "$server" = "$TARGET_SERVER" ]; then
MATCHING_CTX="$ctx"
break
fi
done
If a matching context is found and authentication is still valid (KUBECONFIG="$BUILD_KUBECONFIG" oc whoami --context="$MATCHING_CTX" succeeds), switch to it and reuse:
KUBECONFIG="$BUILD_KUBECONFIG" oc config use-context "$MATCHING_CTX"
Otherwise (no temp kubeconfig, no matching context, or session expired), login into the temp kubeconfig:
KUBECONFIG="$BUILD_KUBECONFIG" oc login --server="$TARGET_SERVER" --web
All subsequent oc commands targeting the build cluster MUST use KUBECONFIG="$BUILD_KUBECONFIG" as an environment variable prefix instead of --context=<ctx>. This ensures the user's default kubeconfig is never read or modified.
Step 6: Find CI Namespace
Use oc get projects with the dedicated kubeconfig and grep for the PR number and job name to find the ci-op namespace:
KUBECONFIG="$BUILD_KUBECONFIG" oc get projects | grep "rehearse-<pr_number>-<job_name_suffix>"
The project display name includes the prow job name. For rehearsal jobs, this contains both the PR number and the job name, e.g.:
ci-op-tk28ym3x ci-op-tk28ym3x - rehearse-75742-periodic-ci-... Active
Construct the grep pattern from the check context found in Step 3. The context is like ci/rehearse/periodic-ci-openshift-.... The rehearsal job name in the project description is rehearse-<pr_number>-periodic-ci-openshift-.... So grep with both the PR number and enough of the job name to be unique.
Step 7: Check Step Status via Pods
List pods in the namespace to determine step status:
KUBECONFIG="$BUILD_KUBECONFIG" oc get pods -n <namespace>
Each pod name corresponds to a step in the job. Pod statuses indicate step progress:
Completed— step finished successfullyRunning— step is currently executingError— step failed
Report the status of all steps/pods. If any pod shows Error status, report the failure and stop.
Determine if the cluster is ready for kubeconfig extraction: there must be at least one Running pod that has the kubeconfig available (i.e., setup/provisioning pods like those with names containing setup, install, create, or provision have Completed). If the cluster creation step has completed but additional setup steps are still running (e.g., OADP setup, operator installation), warn the user that the cluster may not be fully configured yet but proceed with extraction since the kubeconfig is already available.
Step 8: Extract Kubeconfig
Find a pod that is in Running status.
Use the ci-op namespace as a prefix for kubeconfig filenames to avoid collisions when extracting from multiple jobs.
Extract the kubeconfig:
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/kubeconfig > /tmp/<namespace>-kubeconfig.yaml
chmod 600 /tmp/<namespace>-kubeconfig.yaml
Also check for nested kubeconfig (for HyperShift/hosted clusters):
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- ls /tmp/secret/nested_kubeconfig 2>/dev/null
If nested_kubeconfig exists, extract it too:
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/nested_kubeconfig > /tmp/<namespace>-nested-kubeconfig.yaml
chmod 600 /tmp/<namespace>-nested-kubeconfig.yaml
Also extract additional metadata from /tmp/secret/ on the pod:
# Proxy configuration (for clusters behind a proxy)
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/proxy-conf.sh > /tmp/<namespace>-proxy-conf.sh 2>/dev/null
chmod 600 /tmp/<namespace>-proxy-conf.sh 2>/dev/null
# Cluster name and cluster ID (for ROSA clusters, may not exist)
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/cluster-name 2>/dev/null || true
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/cluster-id 2>/dev/null || true
If proxy-conf.sh is empty or the command fails, there is no proxy config — remove the empty file.
Step 9: Verify and Report
Test the kubeconfig:
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get nodes
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get co
If oc commands fail (e.g., connection refused or timeout) and proxy-conf.sh was extracted, source the proxy config and retry:
source /tmp/<namespace>-proxy-conf.sh
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get nodes
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get co
Determine cluster type: Check if the cluster is ROSA:
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get infrastructure cluster -o jsonpath='{.status.platformStatus..aws.resourceTags[?(@.key=="red-hat-clustertype")].value}'
- If the value equals
rosa: this is a ROSA HCP cluster. There is no nested kubeconfig — the single kubeconfig is the ROSA HCP cluster. Report cluster-name and cluster-id from/tmp/secret/. - If nested kubeconfig exists: this is a standard HyperShift job. Verify the nested kubeconfig (including proxy fallback if needed):
Report cluster-name fromKUBECONFIG=/tmp/<namespace>-nested-kubeconfig.yaml oc get nodes KUBECONFIG=/tmp/<namespace>-nested-kubeconfig.yaml oc get co/tmp/secret/for the hosted cluster. - Otherwise: this is a standard cluster.
Return Value
- Success: Kubeconfig file(s) extracted and cluster health report displayed
- Error: Error message explaining the issue (no running jobs, cluster not ready, etc.)
Report includes:
- PR title and number
- Job name and Prow URL
- Build cluster and CI namespace
- Step status summary (which completed, which is running)
- If proxy-conf.sh found: proxy config path and instructions to source it before using the kubeconfig
- If ROSA HCP cluster (single kubeconfig,
red-hat-clustertype=rosa): kubeconfig path, cluster name, cluster ID, node/CO health summary - If HyperShift (nested kubeconfig, not ROSA): management cluster kubeconfig path and health, hosted cluster kubeconfig path, cluster name, and health
- If standard cluster: kubeconfig path and node/CO health summary
- Instructions for using the kubeconfig(s)
Examples
-
Extract kubeconfig from a running rehearsal job:
/ci:extract-kubeconfig https://github.com/openshift/release/pull/75742Expected output:
- Finds the running rehearsal job
- Verifies pre-phase steps passed
- Extracts kubeconfig from test pod on build10
- Reports: 3 master nodes, 4 workers, 34 COs healthy
-
Extract kubeconfig from a HyperShift job (with nested kubeconfig):
/ci:extract-kubeconfig https://github.com/openshift/release/pull/75716Expected output:
- Extracts both management and hosted cluster kubeconfigs
- Management cluster: 7 nodes, 34 COs
- Hosted cluster: 3 nodes, 21 COs
- Reports both kubeconfig paths
-
PR with multiple running jobs:
/ci:extract-kubeconfig https://github.com/openshift/release/pull/12345Expected output:
- Lists all running jobs
- Asks user to select which job to extract from
- Proceeds with selected job
Notes
- Credential hygiene: Extracted kubeconfig and proxy files contain credentials. They are created with restrictive permissions (
chmod 600). Delete them when finished (rm -f /tmp/<namespace>-*.yaml /tmp/<namespace>-proxy-conf.sh). - Kubeconfig expiry: Extracted kubeconfig files become invalid when the CI job finishes. The credentials are tied to the ephemeral CI cluster and cannot be reused after the job completes.
- PR ownership: You must be the PR author to have access to the ci-op namespace on the build cluster. The command warns if you're not the author.
- Private QE deck: Jobs using
qe-private-deckhave inaccessible GCS artifacts (gsutil gets 403). The command tries the job config first; if that also lacks acluster:field (dynamically assigned jobs), it asks the user for the build cluster console URL or name. - Build clusters: OpenShift CI uses multiple build clusters (build01-build12). The command determines which cluster via: (1)
prowjob.jsonfrom GCS for public jobs, (2) job config YAML in the repo, or (3) user-provided console URL/cluster name as a last resort. - Context isolation: All build cluster operations use a dedicated temporary kubeconfig (
/tmp/build-cluster-kubeconfig.yaml) to avoid modifying the user's current kubeconfig context. The temp kubeconfig can accumulate contexts from multiple build clusters across runs. If a valid context for the target build cluster already exists in the temp kubeconfig, it is reused without requiring a new login. - Namespace discovery: The ci-op namespace is found via
oc get projectsfiltered by PR number and job name, which works reliably for running jobs. - Multiple extractions: Kubeconfig files use the ci-op namespace as prefix (e.g.,
/tmp/ci-op-abc123-kubeconfig.yaml) to support extracting from multiple jobs without overwriting. - Wait steps: Jobs with a
waitstep (used for debugging) are valid targets for kubeconfig extraction.
Arguments
- $1 (pr-url): GitHub PR URL in the openshift/release repository (required). Format:
https://github.com/openshift/release/pull/<number>
Name
ci:fetch-payloads
Synopsis
/ci:fetch-payloads [architecture] [version] [stream]
Description
The ci:fetch-payloads command fetches recent release payloads from the OpenShift release controller. It shows each payload's tag name, phase (Accepted/Rejected/Ready), timestamp, and blocking job results. For rejected payloads, it reports which blocking jobs failed with Prow links.
Implementation
-
Determine parameters: Default to
amd64architecture andnightlystream if the user hasn't specified them. If no version is provided, the script fetches the latest from the Sippy API. -
Fetch payloads: Use the
fetch-payloadsskill:FETCH_PAYLOADS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-payloads/fetch_payloads.py" if [ ! -f "$FETCH_PAYLOADS" ]; then FETCH_PAYLOADS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-payloads/fetch_payloads.py" 2>/dev/null | sort | head -1) fi if [ -z "$FETCH_PAYLOADS" ] || [ ! -f "$FETCH_PAYLOADS" ]; then echo "ERROR: fetch_payloads.py not found" >&2; exit 2; fi python3 "$FETCH_PAYLOADS" [architecture] [version] [stream]Optional flags:
--phase Accepted|Rejected|Readyto filter by phase--limit Nto control how many results to show (default 5)
-
Present the results: Show the user the payload list. For rejected payloads, highlight the failed blocking jobs and their Prow links.
Return Value
- Format: One block per payload with tag, phase, timestamp, URL, and job details
- Rejected payloads: Each failed blocking job is listed with retry count and Prow link
- Ready payloads: Summary of succeeded/pending/failed blocking job counts
- Accepted payloads: Confirmation that all blocking jobs succeeded
Examples
-
Latest nightly payloads (defaults to amd64, latest version, last 5):
/ci:fetch-payloads -
Specific version and architecture:
/ci:fetch-payloads arm64 4.18 nightly -
Only accepted payloads:
/ci:fetch-payloads amd64 4.23 nightly --phase Accepted
Arguments
- $1: CPU architecture (optional, default: amd64) — amd64, arm64, ppc64le, s390x, multi
- $2: OCP version (optional, default: latest from Sippy) — e.g., 4.18, 4.23
- $3: Release stream (optional, default: nightly) — nightly, ci
Skills Used
fetch-payloads: Queries the release controller API for payload tags and release details. When no version is specified,fetch_payloads.pyobtains the latest version directly from the Sippy API.
Name
ci:fetch-test-report
Synopsis
/ci:fetch-test-report <test-name> [release]
Description
The ci:fetch-test-report command fetches a report for an OpenShift CI test by name from the Sippy API. It returns the test's BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous 7-day periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name — this can help surface bugs that have been filed but not yet triaged in Component Readiness.
Implementation
-
Determine the release: If the user did not specify a release, use the
fetch-releasesskill to get the latest OCP release:release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)If the user specified a release (e.g., "4.21"), use that value directly.
-
Fetch the test report: Use the
fetch-test-reportskill to query the Sippy tests API:python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test-name>" --release "$release" --format summaryTo see a per-variant breakdown (one row per variant combo), add
--no-collapse:python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test-name>" --release "$release" --no-collapse --format summary -
Present the results: Show the user the test report including test ID, pass rates, Jira component, open bugs, and trend. If
open_bugs > 0, existing Jira bugs mention this test — these may be relevant even if the regression hasn't been triaged in Component Readiness yet.
Return Value
- Format: Human-readable summary of the test report
- Key fields: test_id (BigQuery/Component Readiness ID), jira_component, current/previous pass rates, run counts, trend, open_bugs
- open_bugs: Count of Jira bugs mentioning this test by name — helps find bugs filed but not yet triaged in Component Readiness
- variants (with
--no-collapse): Per-variant breakdown showing which job types the test passes/fails in
Examples
-
Look up a test using the latest release:
/ci:fetch-test-report [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance] -
Look up a test for a specific release:
/ci:fetch-test-report [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance] 4.21
Arguments
- $1: Full test name (required) — must be an exact match including sig prefix and suite tags
- $2: OCP release version (optional) — e.g., "4.22", "4.21". If omitted, the latest release is used.
Skills Used
fetch-releases: Determines the latest OCP release when not specified by the userfetch-test-report: Queries the Sippy tests API for the test report
Name
ci:list-step
Synopsis
/ci:list-step
Description
The ci:list-step command analyzes an OpenShift CI workflow or chain and provides a complete hierarchical breakdown of all the workflow, chain, and ref files with their repo paths that make up the workflow or chain.
The command accepts:
- Workflow/Chain name (required)
It uses the step-registry-analyzer agent to examine the workflow/chain structure and identify all related components including workflows, chains, refs, and their associated command files.
Implementation
The command performs the following steps:
- Parse Arguments:
- $1: Workflow or chain name (required, e.g., "hypershift-aws-reqserving-e2e")
- Execute Analysis: Use the Task tool with step-registry-analyzer agent: Task tool with:
- subagent_type: "step-registry-analyzer"
- description: "analyze and list all workflow/chain/ref with repository path for and keep the hierarchical structure"
- prompt: Detailed request to analyze the workflow or chain and provide complete hierarchical breakdown
- Display Results: Present the hierarchical structure showing:
- Main workflow or chain file path
- All chain files referenced in the workflow or chain with their repository paths
- All ref files referenced in the chains with their repository paths
- Any command files (.sh) associated with the refs
- The complete hierarchical structure showing relationships between all files
- Format Output: Use a clear hierarchical tree format with:
- Full repository paths for all files
- Indentation showing the workflow/chain/ref relationships
- File types (workflow/chain/ref/command) clearly identified
- Summary of total files and execution phases
Return Value
- Success: Complete hierarchical structure of the OpenShift CI workflow or chain
- Error: Workflow or Chain not found, analysis failure, or invalid workflow or chain name
Important for Claude:
- REQUIRED: Always use the step-registry-analyzer agent for this analysis
- Use the Task tool with the specialized agent to ensure comprehensive analysis
- Maintain the proper directory structure and show the relationships between files
- Include all phases (PRE/TEST/POST) in the hierarchy
- Show both the structural hierarchy and file paths clearly
- Show all workflow/chain/ref/command files with repository path
- Provide a summary of total files and execution flow
Examples
- ** Analyze hypershift aws e2e external workflow ** :
/list-step hypershift-aws-e2e-external
Returns:
Workflow: ci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yaml
Complete Repository Path Hierarchy
ci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yaml
├── PRE Phase (Infrastructure Setup)
│ ├── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-ref.yaml
│ │ └── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-commands.sh
│ ├── ci-operator/step-registry/hypershift/setup-nested-management-cluster/hypershift-setup-nested-management-
cluster-chain.yaml
│ │ └── Embedded Steps:
│ │ ├── create-management-cluster (inline command)
│ │ └── Dependencies: hypershift-operator, AWS credentials, kubeconfigs
│ └── ci-operator/step-registry/hypershift/install/hypershift-install-ref.yaml
│ └── ci-operator/step-registry/hypershift/install/hypershift-install-commands.sh
├── TEST Phase (E2E External Tests)
│ └── ci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-ref.yaml
│ └──
ci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-commands.sh
└── POST Phase (Cleanup)
└── ci-operator/step-registry/hypershift/destroy-nested-management-cluster/hypershift-destroy-nested-managem
ent-cluster-chain.yaml
├── Embedded Step: dump-management-cluster (inline command)
├── ci-operator/step-registry/hypershift/dump/extra/hypershift-dump-extra-ref.yaml
│ └── ci-operator/step-registry/hypershift/dump/extra/hypershift-dump-extra-commands.sh
└── Embedded Step: destroy-management-cluster (inline command)
File Summary
Total Files: 10 files
- 1 workflow file: Main orchestration defining pre/test/post phases
- 2 chain files: Setup and destroy management cluster chains
- 4 ref files: RBAC, E2E test runner, installation, and artifact dump steps
- 3 command files: Shell scripts for executing the ref implementations
Execution Flow
PRE Phase: RBAC Setup → Management Cluster Creation → HyperShift Installation
TEST Phase: External E2E Tests with OIDC support
POST Phase: Artifact Collection → Infrastructure Cleanup
This workflow provides comprehensive end-to-end testing for HyperShift on AWS with external OIDC authentication, multi-architecture support, upgrade
testing, and complete infrastructure lifecycle management.
Notes
- Hierarchy Structure: Shows both the logical flow (PRE/TEST/POST) and file dependencies
- File Relationships: Clearly indicates how chains reference refs and how commands are associated
- Path Completeness: All paths are shown as full repository paths from the root
Arguments
- $1 (workflow-or-chain-name): The name of the OpenShift CI workflow or chain to analyze (required)
Name
ci:list-unstable-tests
Synopsis
/ci:list-unstable-tests <version> <keywords> [sippy-url]
Description
The ci:list-unstable-tests command queries OpenShift CI test results from Sippy and lists all tests matching the keywords that have a pass rate below 95%. This is useful for quickly identifying unstable tests that need attention.
By default, it queries the production Sippy instance at sippy.dptools.openshift.org. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).
This command is useful for:
- Identifying unstable tests with inconsistent pass rates
- Finding regression candidates for investigation
- Generating reports of unstable test cases
- Prioritizing test stabilization efforts
- Quality gate checks before releases
Arguments
$1(version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")$2(keywords): Keywords to search in test names (e.g., "olmv1", "sig-storage", "operator")$3(sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"
Implementation
-
Parse Arguments
- Extract version from
$1(e.g., "4.20") - Extract keywords from
$2(e.g., "olmv1") - Extract Sippy URL from
$3if provided, otherwise use default "sippy.dptools.openshift.org" - Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
- Add "https://" prefix if not already present
- Extract version from
-
Build Sippy API Request
- Construct filter JSON for the
/api/testsendpoint:filters = { "items": [ { "columnField": "name", "not": False, "operatorValue": "contains", "value": keywords } ], "linkOperator": "and" } - Set query parameters:
release: The OpenShift versionfilter: JSON-encoded filter objectsort: "asc"sortField: "current_pass_percentage"
- Construct filter JSON for the
-
Query Test Statistics
- Construct API endpoint from provided URL:
https://{base_url}/api/tests - Make GET request to the constructed endpoint
- Parse response to extract test data
- Construct API endpoint from provided URL:
-
Filter Tests Below 95% Pass Rate
- Iterate through all returned tests
- Filter tests where
current_pass_percentage < 95 - Sort filtered results by pass percentage (ascending, worst first)
- Collect the following data for each unstable test:
- Test name
- Current pass percentage
- Total runs
- Passes
- Failures
- Net improvement (trend indicator)
-
Format and Display Results
- Display summary header with:
- Total number of tests matching keywords
- Number of tests below 95% pass rate
- Percentage of unstable tests
- List each unstable test with:
- Test name
- Pass rate percentage
- Run statistics (runs/passes/failures)
- Trend indicator (net improvement)
- Sort by pass percentage (worst tests first)
- If no tests are below 95%, display success message indicating all tests are stable
- Display summary header with:
Return Value
Format: Formatted text output with:
Summary Section:
- Total tests matching keywords
- Tests below 95% pass rate (unstable tests)
- Overall stability percentage
Unstable Tests List: For each test with pass rate < 95%:
- Test name
- Pass rate percentage
- Total runs
- Passes
- Failures
- Net improvement
If all tests pass at 95% or above, display a success message indicating all tests are stable.
Examples
-
List unstable OLMv1 tests from QE Sippy:
/ci:list-unstable-tests 4.20 olmv1 qe-component-readiness.dptools.openshift.orgLists all OLMv1-related tests in version 4.20 from QE Sippy that have a pass rate below 95%.
-
List unstable storage tests (using default Sippy):
/ci:list-unstable-tests 4.21 sig-storageLists all storage-related tests in version 4.21 from production Sippy with pass rate below 95%.
-
List unstable operator tests:
/ci:list-unstable-tests 4.19 operatorLists all operator-related tests in version 4.19 with pass rate below 95%.
-
Check specific component stability:
/ci:list-unstable-tests 4.20 sig-network qe-component-readiness.dptools.openshift.orgLists all network-related unstable tests from QE Sippy.
Notes
- Pass Rate Threshold: Fixed at 95% - tests with pass rate >= 95% are considered stable
- Default Sippy URL: If no Sippy URL is provided, the command uses
sippy.dptools.openshift.orgby default - The command queries data from the last 7 days by default
- Ensure you can access the Sippy API endpoints
- Results are sorted by pass percentage (ascending) to show most unstable tests first
- The net improvement metric shows if the test is getting worse (negative) or better (positive)
- If no tests match the keywords, an appropriate message will be displayed
- If all matching tests have pass rate >= 95%, a success message will be shown indicating all tests are stable
Output Example
================================================================================
Unstable Tests Report - 4.20 olmv1
================================================================================
Sippy Instance: qe-component-readiness.dptools.openshift.org
Pass Rate Threshold: < 95%
Summary:
Total Tests Matching 'olmv1': 45
Unstable Tests (< 95%): 8 (17.8%)
Stable Tests (>= 95%): 37 (82.2%)
================================================================================
Tests Below 95% Pass Rate (sorted by worst first):
================================================================================
1. Test: [sig-olmv1] clusterextension install should fail validation
Pass Rate: 23.5%
Runs: 17 | Passes: 4 | Failures: 13
Net Improvement: -45.2
2. Test: [sig-olmv1] clusterextension upgrade from v1 to v2
Pass Rate: 67.8%
Runs: 28 | Passes: 19 | Failures: 9
Net Improvement: -12.3
[... additional tests ...]
================================================================================
See Also
/ci:query-test-result- Query detailed results for a specific test- Sippy UI (Production): https://sippy.dptools.openshift.org/sippy-ng/
- Sippy UI (QE): https://qe-component-readiness.dptools.openshift.org
- Sippy API Documentation: https://github.com/openshift/sippy
Name
ci:payload-experiment
Synopsis
/ci:payload-experiment <payload-tag>
Description
The ci:payload-experiment command opens draft revert PRs for medium-confidence payload candidates (confidence score 60-84) and triggers payload jobs to experimentally determine which PR is causing failures. It operates in two phases separated by a CI wait period.
Phase 1: Reads the payload results YAML, filters medium-confidence candidates, opens draft revert PRs, triggers payload jobs, and appends action entries (type: "experiment", status: "pending") to each candidate's actions array.
Phase 2: Detects candidates with a status: "pending" action entry in the results YAML, checks job results, promotes confirmed causes to real revert PRs (with TRT JIRA bugs), and closes innocent draft PRs.
All state is tracked in the payload results YAML via the payload-results-yaml skill — no separate tracking file is created.
This command is one of three composable stages in the payload triage pipeline:
/ci:payload-analysis— produces the payload results YAML/ci:payload-revert— stages reverts for HIGH confidence candidates/ci:payload-experiment— experimentally tests MEDIUM confidence candidates (this command)
Job Triggering Limits
- Non-aggregated jobs: Up to 5 total across all candidates
- Aggregated jobs: Up to 1 total
Implementation
-
Parse the payload tag from the argument. Extract
version,stream, andarchitecturefrom the tag (seepayload-analysisStep 1 for parsing rules). -
Read the payload results YAML using the
payload-results-yamlskill: Look forpayload-results-{tag}.yamlin the current working directory. If not found, print an error and exit:Error: Payload results YAML not found for {payload_tag}. Run `/ci:payload-analysis {payload_tag}` first to generate it. -
Detect Phase 2 resume: If the results YAML contains any action entry with
type: "experiment"andstatus: "pending", jump to Phase 2 (step 5). Phase 2 processes only pending experiments — candidates with other statuses are left unchanged. -
Phase 1 — Set up experiments: Filter
type: "pr"candidates with60 <= confidence_score < 85. Exclude any that already have an action withstatusof"open"or"merged"(pre-existing revert).type: "rhcos_rpm"candidates are never experimented on — they cannot be pulled out of a payload like a PR. Dispatch to thepayload-experimental-revertsskill Phase 1, which updates the results YAML in place. -
Phase 2 — Collect results: Dispatch to the
payload-experimental-revertsskill Phase 2 to check job results, promote confirmed causes, close innocent drafts, and update the results YAML. -
Report results: Print a summary of actions taken.
Return Value
- Phase 1: Confirmation that experiments are running, with resume instructions
- Phase 2: Summary of experiment verdicts (confirmed/innocent) and actions taken. If some experiments are still running, they remain
pendingand the command can be re-invoked to check again.
Examples
-
Start experiments after analysis:
/ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806 /ci:payload-experiment 4.22.0-0.nightly-2026-02-25-152806 -
Resume to collect results (run from the same directory after jobs complete):
/ci:payload-experiment 4.22.0-0.nightly-2026-02-25-152806
Arguments
- $1: A full payload tag (e.g.,
4.22.0-0.nightly-2026-02-25-152806). Must match the tag used with/ci:payload-analysis. (required)
Skills Used
payload-results-yaml: Reads and updates the payload results YAMLpayload-experimental-reverts: Opens draft revert PRs and triggers payload jobs (Phase 1); collects results and acts (Phase 2)trigger-payload-job: Triggers payload jobs and collects URLsrevert-pr: Git revert workflow for creating revert PRs
Name
ci:payload-revert
Synopsis
/ci:payload-revert <payload-tag>
Description
The ci:payload-revert command reads the payload results YAML produced by /ci:payload-analysis and stages reverts for all high-confidence candidates (confidence score >= 85) that have not already been reverted.
For each qualifying candidate, it creates a TRT JIRA bug, opens a revert PR, and triggers payload validation jobs using the stage-payload-reverts skill.
This command is one of three composable stages in the payload triage pipeline:
/ci:payload-analysis— produces the payload results YAML/ci:payload-revert— stages reverts for HIGH confidence candidates (this command)/ci:payload-experiment— opens draft revert PRs for MEDIUM confidence candidates
Job Triggering Limits
- Non-aggregated jobs: Up to 5 total across all candidates
- Aggregated jobs: Up to 1 total
When the number of failing jobs across all candidates exceeds these limits, prioritize jobs from higher-confidence candidates first.
Implementation
-
Parse the payload tag from the argument. Extract
version,stream, andarchitecturefrom the tag (seepayload-analysisStep 1 for parsing rules). -
Read the payload results YAML using the
payload-results-yamlskill: Look forpayload-results-{tag}.yamlin the current working directory. If not found, print an error and exit:Error: Payload results YAML not found for {payload_tag}. Run `/ci:payload-analysis {payload_tag}` first to generate it. -
Filter candidates: Select
type: "pr"candidates withconfidence_score >= 85. Exclude any that already have an action withstatusof"open"or"merged"(pre-existing revert).type: "rhcos_rpm"candidates are never reverted — they cannot be pulled out of a payload like a PR; thepayload-results-yamlskill's Read Candidates operation excludes them for this reason. -
Dispatch to
stage-payload-revertsskill: Pass all qualifying candidates with their context (results YAML path, payload tag, version, stream, architecture, release controller URL, and failing jobs). The skill updates the results YAML and HTML report in place. Thetrigger-payload-jobskill validates that aggregated jobs haveunderlying_job_nameset and skips them with an error if not. -
Report results: Print a summary of actions taken (JIRA tickets created, revert PRs opened, payload jobs triggered).
Return Value
- Summary of staged reverts with JIRA keys, revert PR URLs, and triggered payload job URLs
- Any errors encountered during staging
Examples
- Stage reverts after analysis:
/ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806 /ci:payload-revert 4.22.0-0.nightly-2026-02-25-152806
Arguments
- $1: A full payload tag (e.g.,
4.22.0-0.nightly-2026-02-25-152806). Must match the tag used with/ci:payload-analysis. (required)
Skills Used
payload-results-yaml: Reads and updates the payload results YAMLstage-payload-reverts: Creates TRT JIRA bugs, opens revert PRs, triggers payload jobstrigger-payload-job: Triggers payload jobs and collects URLsrevert-pr: Git revert workflow for creating revert PRs
Name
ci:query-job-status
Synopsis
/query-job-status <execution-id>
Description
The query-job-status command queries the status of a gangway job execution via the REST API using the execution ID returned when a job is triggered.
The command accepts:
- Execution ID (required, UUID returned when triggering a job)
It makes a GET request to the gangway API and returns the current status of the job including its name, type, status, and GCS path to artifacts if available. The curl_with_token.sh wrapper handles all authentication automatically.
Implementation
The command performs the following steps:
-
Parse Arguments:
- $1: execution ID (required, UUID format)
-
Execute API Request: Make a GET request to query the job status using the
oc-authskill's curl wrapper:# Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443 curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID>The
curl_with_token.shwrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token. -
Display Results: Parse and present the JSON response with:
id: The execution IDjob_name: The name of the jobjob_type: The type of job execution (PERIODIC, POSTSUBMIT, PRESUBMIT)job_status: Current status (TRIGGERED, PENDING, SUCCESS, FAILURE, ABORTED)gcs_path: Path to job artifacts in GCS (if available)prow_url: Derived Prow dashboard URL (see step 4)
-
Derive Prow URL from GCS Path: If
gcs_pathis present, convert it to a Prow dashboard URL:- Strip the
gs://<bucket-name>/prefix (bucket is typicallyorigin-ci-testortest-platform-results-public) - Prepend
https://prow.ci.openshift.org/view/gs/test-platform-results-public/ - Example:
gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890→https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890 - The Prow URL always uses
test-platform-results-publicas the bucket name regardless of whatgcs_pathreports
- Strip the
-
Poll if Prow URL Not Yet Available: If
gcs_pathis empty/missing andjob_statusisTRIGGERED, the job has not started yet. Automatically re-query:- Wait 15 seconds, then re-query the execution status
- Repeat up to 20 times (~5 minutes total)
- Stop polling once
gcs_pathis populated orjob_statusreaches a terminal state (SUCCESS,FAILURE,ABORTED) - If polling exhausts all retries without a
gcs_path, report the current status and provide a Prow dashboard search link so the user can find the job manually:https://prow.ci.openshift.org/?job=<job_name>(using thejob_namefrom the status response)
-
Offer Follow-up Actions:
- If status is PENDING with prow_url: Provide the Prow dashboard link (job is actively running)
- If status is SUCCESS or FAILURE with prow_url: Provide the link and offer to help access logs/artifacts
- If prow_url could not be derived: Provide
https://prow.ci.openshift.org/?job=<job_name>so the user can find it on the Prow dashboard
Return Value
- Success: JSON response with job status details and derived Prow URL
- Error: HTTP error, authentication failure, or invalid execution ID
Important for Claude:
- REQUIRED: Before executing this command, you MUST ensure the
ci:oc-authskill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active. - You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
- Parse the JSON response and present it in a readable format
- Highlight the job status prominently
- Always derive and display the Prow URL when
gcs_pathis available - If
gcs_pathis missing and job is not terminal, poll automatically — do not just offer to check again - If PENDING with Prow URL, provide the link — PENDING means the job is actively running
- If SUCCESS/FAILURE, indicate completion status and provide the Prow link
Examples
-
Query status of a triggered job:
/query-job-status ca249d50-dee8-4424-a0a7-6dd9d5605267Returns:
{ "id": "ca249d50-dee8-4424-a0a7-6dd9d5605267", "job_name": "periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn", "job_type": "PERIODIC", "job_status": "SUCCESS", "gcs_path": "gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890" }Claude derives and displays:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890 -
Query a recently triggered job (auto-polling):
/query-job-status 8f3a9b2c-1234-5678-9abc-def012345678First query returns
job_status: TRIGGEREDwith nogcs_path— Claude automatically waits 15 seconds and re-queries untilgcs_pathappears, then displays the Prow URL. -
Check failed job:
/query-job-status 5a6b7c8d-9e0f-1a2b-3c4d-5e6f7a8b9c0dStatus shows "FAILURE" — Claude displays the Prow dashboard link for log analysis.
Notes
- Execution ID Format: UUID format (e.g.,
ca249d50-dee8-4424-a0a7-6dd9d5605267) - Job Status Values:
TRIGGERED: The ProwJob CR exists but has not been scheduled yet — waiting to transition to PENDING.gcs_pathmay not be populated yet.PENDING: The job is actively running (despite the name, this means in-progress, not queued)SUCCESS: Job completed successfully (terminal)FAILURE: Job completed with a failure (terminal)ABORTED: Job was cancelled (terminal)
- Rate Limits: The REST API has rate limits
- Authentication: Tokens expire and may need to be refreshed via browser login
- GCS Path: Provides access to job logs and artifacts when available
- Prow URL Derivation: The Prow URL is always derived using
test-platform-results-publicas the bucket name, regardless of what bucket name appears ingcs_path(commonlyorigin-ci-test) - Auto-Polling: When
gcs_pathis not yet available (common immediately after triggering), the command polls automatically every 15 seconds for up to ~5 minutes. Prow can be slow to schedule jobs. Do not just offer to check again — poll proactively.
Arguments
- $1 (execution-id): The UUID execution ID returned when a job was triggered (required)
Name
ci:query-test-result
Synopsis
/ci:query-test-result <version> <keywords> [sippy-url]
Description
The ci:query-test-result command queries OpenShift CI test results from Sippy based on the OpenShift version and test name keywords. It retrieves test statistics including pass rate, number of runs, failures, and links to failed job runs.
By default, it queries the production Sippy instance at sippy.dptools.openshift.org. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).
This command is useful for:
- Checking the health of specific test cases
- Finding test failure patterns
- Getting links to failed Prow job runs for debugging
- Monitoring test regression trends
- Querying different Sippy instances (production, QE, etc.)
Arguments
$1(version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")$2(keywords): Keywords to search in test names (e.g., "PolarionID:81664", "olmv1", "sig-storage")$3(sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"
Implementation
-
Parse Arguments
- Extract version from
$1(e.g., "4.21") - Extract keywords from
$2(e.g., "81664" or "olmv1") - Extract Sippy URL from
$3if provided, otherwise use default "sippy.dptools.openshift.org" - Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
- Add "https://" prefix if not already present
- Extract version from
-
Build Sippy API Request
- Construct filter JSON for the
/api/testsendpoint:filters = { "items": [ { "columnField": "name", "not": False, "operatorValue": "contains", "value": keywords } ], "linkOperator": "and" } - Set query parameters:
release: The OpenShift versionfilter: JSON-encoded filter objectsort: "asc"sortField: "net_improvement"
- Construct filter JSON for the
-
Query Test Statistics
- Construct API endpoint from provided URL:
https://{base_url}/api/tests- Example: If input is "sippy.dptools.openshift.org", use
https://sippy.dptools.openshift.org/api/tests - Example: If input is "qe-component-readiness.dptools.openshift.org", use
https://qe-component-readiness.dptools.openshift.org/api/tests
- Example: If input is "sippy.dptools.openshift.org", use
- Make GET request to the constructed endpoint
- Parse response to extract:
- Test name
- Current pass percentage
- Total runs, passes, failures
- Net improvement (trend indicator)
- Construct API endpoint from provided URL:
-
Query Failed Job Runs (for each matching test)
- Calculate the UTC timestamp for 7 days ago:
from datetime import datetime, timedelta, timezone start_time = (datetime.now(timezone.utc) - timedelta(days=7)).isoformat().replace("+00:00", "Z") - Build filter for failed runs:
filters = { "items": [ { "columnField": "failed_test_names", "operatorValue": "contains", "value": test_name }, { "columnField": "timestamp", "operatorValue": ">", "value": start_time } ], "linkOperator": "and" } - Make GET request to
https://{base_url}/api/jobs/runswith parameters:release: versionfilter: JSON-encoded filterlimit: "20"sortField: "timestamp"sort: "desc"
- Parse Response Structure:
- Response is a dict with structure:
{"rows": [...], "page": N, "page_size": N, "total_rows": N} - Extract job runs from
response["rows"] - Each run contains:
timestamp: Unix timestamp in millisecondsbrief_nameorjob: Job nametest_grid_url: Link to Prow job details
- Response is a dict with structure:
- Calculate the UTC timestamp for 7 days ago:
-
Format and Display Results
- Show summary statistics for each matching test
- List failed job runs with:
- Timestamp of the failure
- Job name
- Clickable Prow URL for each failed run
- After listing individual runs, provide a summary section:
- Create a "Failed Prow URLs (for easy copying)" section
- List all Prow URLs from the failed runs in plain text format (one per line)
- This allows users to easily copy all URLs at once for further analysis
- Format output in a clear, readable structure with proper spacing
- Present URLs as markdown links for easy clicking
Return Value
Format: Formatted text output with:
- Test name(s) matching the keywords
- Statistics section showing:
- Pass Rate (percentage)
- Total Runs
- Passes
- Failures
- Net Improvement
- Failed Job Runs section listing (for last 7 days):
- Sequential numbering (1, 2, 3...)
- Timestamp (formatted as YYYY-MM-DD HH:MM:SS)
- Job name (brief name)
- Clickable Prow URL (as markdown link or plain URL)
- Failed Prow URLs summary section:
- Plain text list of all Prow URLs (one per line)
- Allows easy copying of all URLs for batch analysis
Output Format Example:
Failed Job Runs (Last 7 Days):
1. 2025-11-03 12:12:31 - periodic-ci-openshift-operator-framework-...
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
2. 2025-11-02 12:12:29 - periodic-ci-openshift-operator-framework-...
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
If no tests match the keywords, inform the user that no results were found. If a test has no failed runs in the last 7 days, display a success message.
Examples
-
Query by Polarion ID (using default Sippy instance):
/ci:query-test-result 4.21 81664Returns test results for tests containing "81664" in version 4.21 from the default production Sippy instance (sippy.dptools.openshift.org).
-
Query by test signature (using default Sippy instance):
/ci:query-test-result 4.20 olmv1Returns all OLMv1-related test results for version 4.20 from the default Sippy instance.
-
Query from QE Sippy instance (custom URL):
/ci:query-test-result 4.20 olmv1 qe-component-readiness.dptools.openshift.orgReturns all OLMv1-related test results for version 4.20 from the QE component readiness Sippy instance.
-
Query by component with custom Sippy URL:
/ci:query-test-result 4.19 sig-storage sippy.dptools.openshift.orgReturns all storage-related test results for version 4.19 from the specified Sippy instance.
-
Custom URL variations (all valid formats):
/ci:query-test-result 4.21 olmv1 sippy.dptools.openshift.org /ci:query-test-result 4.21 olmv1 https://sippy.dptools.openshift.orgBoth URL formats are accepted and will query the same Sippy instance.
Output Example
====================================================================================================
Test Results for PolarionID: 81664 (Version 4.21)
====================================================================================================
Test Name:
[sig-olmv1][Jira:OLM] clusterextension PolarionID:81664-[Skipped:Disconnected]preflight check
Statistics (Last 7 Days):
• Pass Rate: 0.00%
• Total Runs: 6
• Passes: 0
• Failures: 6
• Net Improvement: -100.00
Failed Job Runs (Last 7 Days):
----------------------------------------------------------------------------------------------------
1. 2025-11-03 12:12:31
Job: periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1985198377557561344
2. 2025-11-02 12:12:29
Job: periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1984835985292136448
[... additional failures ...]
----------------------------------------------------------------------------------------------------
Failed Prow URLs (for easy copying):
----------------------------------------------------------------------------------------------------
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1985198377557561344
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1984835985292136448
[... additional URLs ...]
====================================================================================================
Notes
- Default Sippy URL: If no Sippy URL is provided, the command uses
sippy.dptools.openshift.orgby default - The command queries data from the last 7 days by default
- Ensure you can access the Sippy API endpoints
- Results are sorted by net improvement to show regressed tests first
- Failed job runs are limited to the most recent 20 occurrences
- URLs are displayed as clickable links for easy access to Prow job details
- If multiple tests match the keywords, results for all matches will be displayed
- URL normalization:
- The command automatically strips common suffixes like "/sippy-ng/" from the URL
- It adds "https://" prefix if not provided
- Both domain-only and full path formats are supported
See Also
- Sippy UI (Production): https://sippy.dptools.openshift.org/sippy-ng/
- Sippy UI (QE): https://qe-component-readiness.dptools.openshift.org
- Sippy API Documentation: https://github.com/openshift/sippy
Name
ci:revert-pr
Synopsis
/ci:revert-pr <pr-url> <jira-ticket>
Description
The ci:revert-pr command reverts a merged pull request that is breaking CI or nightly payloads. It follows the OpenShift quick-revert policy to restore CI signal.
This command:
- Extracts the original PR details (title, author, merge commit SHA, base branch)
- Ensures the user has a fork of the repository
- Creates a revert branch and performs
git revert -m1of the merge commit - Pushes the revert branch to the user's fork
- Creates a revert PR using the Revertomatic template format
- Generates a list of CI override commands (
/override) for jobs that may need to be bypassed on the revert PR
This command is useful when:
- A merged PR is causing CI job failures
- Nightly payloads are blocked by a breaking change
- Quick reversion is needed to restore CI signal while the original author fixes the issue
Implementation
-
Parse Arguments: Extract the PR URL, JIRA ticket, and flags
- PR URL format:
https://github.com/{owner}/{repo}/pull/{number} - JIRA ticket: e.g.,
TRT-1234orOCPBUGS-56789 - If the user provides additional context about why the revert is needed, capture it for the PR body
- If the user provides verification details (jobs to run before unrevert), capture those too
- PR URL format:
-
Gather Context: Use the JIRA ticket to automatically determine what broke and what needs verification
- Look up the JIRA ticket using the
fetch-jira-issueskill (plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py) with--format json - Extract context from the JIRA issue: Use the issue summary, description, and comments to determine:
- What broke: Which CI jobs or payloads are failing (e.g., "e2e-aws jobs failing on 4.18 nightly payload")
- Verification jobs: Which jobs should pass before the original change can be re-landed (e.g., "e2e-aws, e2e-gcp")
- Compose the context for the revert PR body from the JIRA issue details. Include links to failing jobs or payloads if mentioned in the ticket.
- Fall back to asking the user only if:
- The JIRA lookup fails (token not set, network error, ticket not found)
- The JIRA issue doesn't contain enough information to determine what broke or which jobs to verify
- If the user provided additional context as inline arguments, combine it with the JIRA-derived context
- Look up the JIRA ticket using the
-
Perform the Revert: Use the
revert-prskillFollow the complete workflow in
plugins/ci/skills/revert-pr/SKILL.md, which covers:- Extracting PR information (title, author, merge SHA, base branch) via
gh - Validating the PR is in MERGED state
- Ensuring the user has a fork of the repository
- Detecting commit message conventions (e.g.,
UPSTREAM: <carry>:prefix) - Cloning or using an existing local repository
- Creating the revert branch and performing
git revert -m1 - Handling merge conflicts: resolving simple ones directly, or reverting dependent commits for complex ones
- Pushing to the user's fork
- Generating CI override commands (filtering out unoverridable jobs)
- Creating the revert PR with the Revertomatic template (adapting title format for UPSTREAM carry repos)
- Extracting PR information (title, author, merge SHA, base branch) via
-
Report Results: Display the revert PR URL, override commands, and next steps
- Link to the revert PR
- List of override commands that can be used to force the PR in (do NOT post these as a comment on the PR automatically)
- Instructions for the original author to unrevert
Return Value
- Revert PR URL: Link to the newly created revert pull request
- Override Commands: List of
/overridecommands for CI jobs - Summary: Brief description of what was reverted and why
Examples
-
Basic revert with context provided inline:
/ci:revert-pr https://github.com/openshift/kubernetes/pull/1703 TRT-9999The command will ask for context and verification details interactively.
-
Revert with full context:
/ci:revert-pr https://github.com/openshift/cluster-network-operator/pull/2037 OCPBUGS-12345 This PR broke all e2e-aws jobs on the 4.18 nightly. Verification: run e2e-aws and e2e-gcp.
Arguments
$1(required): GitHub PR URL to revert- Format:
https://github.com/{owner}/{repo}/pull/{number} - The PR must be in MERGED state
- Format:
$2(required): JIRA ticket tracking the revert- Format: JIRA key (e.g.,
TRT-1234,OCPBUGS-56789)
- Format: JIRA key (e.g.,
- Additional text (optional): Context explaining why the revert is needed and verification details
Prerequisites
-
GitHub CLI (
gh): Must be installed and authenticated- Check:
gh auth status - Install: https://cli.github.com/
- Check:
-
Git: Must be installed
- Check:
which git
- Check:
-
GitHub Token: The
ghCLI must haverepoandread:orgpermissions
Notes
- This command follows the Revertomatic PR template format for consistency
- The original PR author is CC'd in the revert PR body
- To unrevert, the original author should revert the revert PR and layer a fix commit on top
- For repos using
UPSTREAM: <tag>:commit conventions (e.g., openshift/kubernetes and other repos carrying upstream patches), the commit message and PR title adapt automatically - Merge conflicts are handled automatically when possible: trivial conflicts are resolved directly, non-trivial ones trigger reverting dependent commits
See Also
- Related Skill:
revert-pr- Detailed git revert workflow and PR template (plugins/ci/skills/revert-pr/SKILL.md) - OpenShift Quick Revert Policy: https://github.com/openshift/enhancements/blob/master/enhancements/release/improving-ci-signal.md#quick-revert
- Revertomatic: https://github.com/stbenjam/revertomatic
Name
ci:trigger-periodic
Synopsis
/trigger-periodic <job-name> [ENV_VAR=value ...]
Description
The trigger-periodic command triggers a periodic gangway job via the REST API. Periodic jobs run on a schedule but can be manually triggered for testing or urgent runs.
The command accepts:
- Job name (required, first argument)
- Environment variable overrides (optional, additional arguments in KEY=VALUE format)
It then constructs and executes the appropriate curl command to trigger the job via the gangway REST API.
Security
IMPORTANT SECURITY REQUIREMENTS:
Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:
- READ operations: Checking authentication status (
oc whoami) - TRIGGERING jobs: POST requests to the gangway API to trigger jobs
Claude is EXPLICITLY PROHIBITED from:
- Modifying cluster resources (deployments, pods, services, etc.)
- Deleting or altering existing jobs or executions
- Accessing secrets, configmaps, or sensitive data
- Making any cluster modifications beyond job triggering
- Using the token for any purpose other than the specific operations listed above
MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:
- Display the complete payload that will be sent
- Show the exact curl command that will be executed
- Request explicit user confirmation with a clear "yes/no" prompt
- Only proceed after receiving affirmative confirmation
Token Usage:
The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.
Implementation
The command performs the following steps:
-
Parse Arguments:
- First argument is the job name (required)
- Remaining arguments are environment variable overrides in KEY=VALUE format
- Note: Variables that need to override multistage parameters should be prefixed with
MULTISTAGE_PARAM_OVERRIDE_
-
Construct API Request: Build the appropriate curl command using the
oc-authskill's curl wrapper:Without overrides:
# Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443 curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \ -d '{"job_name": "<JOB_NAME>", "job_execution_type": "1"}' \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executionsWith overrides:
curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \ -d '{"job_name": "<JOB_NAME>", "job_execution_type": "1", "pod_spec_options": {"envs": {"ENV_VAR": "value"}}}' \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executionsWith multistage parameter override:
curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \ -d '{"job_name": "periodic-to-trigger", "job_execution_type": "1", "pod_spec_options": {"envs": {"MULTISTAGE_PARAM_OVERRIDE_FOO": "bar"}}}' \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executionsThe
curl_with_token.shwrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token. -
Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.
-
Execute Request: Only after receiving user confirmation, run the constructed curl command
-
Display Results: Show the API response including the execution ID
-
Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:
- Wait 15 seconds for the job to be scheduled
- Query the job status:
curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID> - If
gcs_pathis present, derive the Prow URL (see below) and display it - If
gcs_pathis missing and status isTRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total) - Stop polling once
gcs_pathis populated or status reaches a terminal state - If retries are exhausted without a
gcs_path, providehttps://prow.ci.openshift.org/?job=<job_name>so the user can find the job on the Prow dashboard
GCS Path → Prow URL conversion: Strip the
gs://<bucket-name>/prefix and prependhttps://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always usestest-platform-results-publicregardless of the bucket name ingcs_path.Example:
gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890→https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890
Return Value
- Success: JSON response with execution ID and job details
- Error: HTTP error, authentication failure, or missing job name
Important for Claude:
- REQUIRED: Before executing this command, you MUST ensure the
ci:oc-authskill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active. - You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
- Parse the JSON response and extract the execution ID
- Display the execution ID to the user
- Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
- Display the Prow dashboard URL once available
Examples
-
Trigger a periodic job without overrides:
/trigger-periodic periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn -
Trigger a periodic job with payload override:
/trigger-periodic periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn RELEASE_IMAGE_LATEST=quay.io/openshift-release-dev/ocp-release:4.18.8-x86_64 -
Trigger with multistage parameter override:
/trigger-periodic periodic-to-trigger MULTISTAGE_PARAM_OVERRIDE_FOO=bar -
Trigger with multiple environment overrides:
/trigger-periodic periodic-ci-job RELEASE_IMAGE_LATEST=quay.io/image:4.18.8 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=3600
Notes
- Job Execution Type: For periodic jobs, always use
"1" - Rate Limits: The REST API has rate limits; username is recorded in annotations
- Authentication: Tokens expire and may need to be refreshed via browser login
- Multistage Overrides: Prefix variables with
MULTISTAGE_PARAM_OVERRIDE_to override multistage job parameters - Execution ID: Save the execution ID from the response to query job status later
- Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before
gcs_pathis populated. The command automatically polls until the URL is resolved.
Arguments
- $1 (job-name): The name of the periodic job to trigger (required)
- $2-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format
Name
ci:trigger-postsubmit
Synopsis
/trigger-postsubmit <job-name> <org> <repo> <base-ref> <base-sha> [ENV_VAR=value ...]
Description
The trigger-postsubmit command triggers a postsubmit gangway job via the REST API. Postsubmit jobs run after code is merged and require repository reference information.
The command accepts:
- Job name (required)
- Organization (required, e.g., "openshift")
- Repository name (required, e.g., "assisted-installer")
- Base ref/branch (required, e.g., "release-4.12")
- Base SHA/commit hash (required)
- Environment variable overrides (optional, additional arguments in KEY=VALUE format)
It constructs the necessary JSON payload with refs structure and executes the curl command to trigger the job via the gangway REST API.
Security
IMPORTANT SECURITY REQUIREMENTS:
Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:
- READ operations: Checking authentication status (
oc whoami) - TRIGGERING jobs: POST requests to the gangway API to trigger jobs
Claude is EXPLICITLY PROHIBITED from:
- Modifying cluster resources (deployments, pods, services, etc.)
- Deleting or altering existing jobs or executions
- Accessing secrets, configmaps, or sensitive data
- Making any cluster modifications beyond job triggering
- Using the token for any purpose other than the specific operations listed above
MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:
- Display the complete payload that will be sent
- Show the exact curl command that will be executed
- Request explicit user confirmation with a clear "yes/no" prompt
- Only proceed after receiving affirmative confirmation
Token Usage:
The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.
Implementation
The command performs the following steps:
-
Parse Arguments:
- $1: job name (required)
- $2: organization (required)
- $3: repository name (required)
- $4: base ref/branch (required)
- $5: base SHA (required)
- $6-$N: environment variable overrides in KEY=VALUE format (optional)
-
Construct JSON Payload: Build the payload with refs structure:
Without overrides:
{ "job_name": "<JOB_NAME>", "job_execution_type": "2", "refs": { "org": "<ORG>", "repo": "<REPO>", "base_ref": "<BASE_REF>", "base_sha": "<BASE_SHA>", "repo_link": "https://github.com/<ORG>/<REPO>" } }With overrides:
{ "job_name": "<JOB_NAME>", "job_execution_type": "2", "refs": { "org": "<ORG>", "repo": "<REPO>", "base_ref": "<BASE_REF>", "base_sha": "<BASE_SHA>", "repo_link": "https://github.com/<ORG>/<REPO>" }, "pod_spec_options": { "envs": {"ENV_VAR": "value"} } } -
Save JSON to Temporary File: Write the payload to a temp file (e.g.,
/tmp/postsubmit-spec.json) -
Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.
-
Execute Request: Only after receiving user confirmation, run the curl command using the
oc-authskill's curl wrapper:# Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443 curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \ -d @/tmp/postsubmit-spec.json \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executionsThe
curl_with_token.shwrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token. -
Clean Up: Remove the temporary JSON file
-
Display Results: Show the API response including the execution ID
-
Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:
- Wait 15 seconds for the job to be scheduled
- Query the job status:
curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID> - If
gcs_pathis present, derive the Prow URL (see below) and display it - If
gcs_pathis missing and status isTRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total) - Stop polling once
gcs_pathis populated or status reaches a terminal state - If retries are exhausted without a
gcs_path, providehttps://prow.ci.openshift.org/?job=<job_name>so the user can find the job on the Prow dashboard
GCS Path → Prow URL conversion: Strip the
gs://<bucket-name>/prefix and prependhttps://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always usestest-platform-results-publicregardless of the bucket name ingcs_path.Example:
gs://origin-ci-test/logs/branch-ci-openshift-origin-master-images/1234567890→https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-openshift-origin-master-images/1234567890
Return Value
- Success: JSON response with execution ID and job details
- Error: HTTP error, authentication failure, or missing required arguments
Important for Claude:
- REQUIRED: Before executing this command, you MUST ensure the
ci:oc-authskill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active. - You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
- Validate all required arguments are provided
- Parse the JSON response and extract the execution ID
- Display the execution ID to the user
- Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
- Display the Prow dashboard URL once available
Examples
-
Trigger a postsubmit job without overrides:
/trigger-postsubmit branch-ci-openshift-assisted-installer-release-4.12-images openshift assisted-installer release-4.12 7336f38f75f91a876313daacbfw97f25dfe21bbf -
Trigger a postsubmit job with environment override:
/trigger-postsubmit branch-ci-openshift-origin-master-images openshift origin master abc123def456 RELEASE_IMAGE_LATEST=quay.io/image:latest -
Trigger with multiple environment overrides:
/trigger-postsubmit my-postsubmit-job openshift cluster-api-provider-aws master def789ghi012 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=7200 BUILD_ID=custom-123
Notes
- Job Execution Type: For postsubmit jobs, always use
"2" - Rate Limits: The REST API has rate limits; username is recorded in annotations
- Authentication: Tokens expire and may need to be refreshed via browser login
- Refs Structure: The refs object is required for postsubmit jobs to identify the repository and commit
- Repo Link: Automatically constructed as
https://github.com/<org>/<repo> - Execution ID: Save the execution ID from the response to query job status later
- Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before
gcs_pathis populated. The command automatically polls until the URL is resolved.
Arguments
- $1 (job-name): The name of the postsubmit job to trigger (required)
- $2 (org): GitHub organization (e.g., "openshift") (required)
- $3 (repo): Repository name (e.g., "assisted-installer") (required)
- $4 (base-ref): Base branch/ref (e.g., "release-4.12", "master") (required)
- $5 (base-sha): Base commit SHA hash (required)
- $6-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format
Name
ci:trigger-presubmit
Synopsis
/trigger-presubmit <job-name> <org> <repo> <base-ref> <base-sha> <pr-number> <pr-sha> [ENV_VAR=value ...]
Description
The trigger-presubmit command triggers a presubmit gangway job via the REST API.
WARNING: Triggering presubmit jobs via REST is generally unnecessary and not recommended. Presubmit jobs should typically be triggered using Prow commands like /test and /retest via GitHub interactions. Only use this command if you have a specific reason to trigger via REST API.
The command accepts:
- Job name (required)
- Organization (required, e.g., "openshift")
- Repository name (required, e.g., "origin")
- Base ref/branch (required, e.g., "master")
- Base SHA/commit hash (required)
- Pull request number (required)
- Pull request SHA/head commit (required)
- Environment variable overrides (optional, additional arguments in KEY=VALUE format)
It constructs the necessary JSON payload with refs and pulls structure and executes the curl command to trigger the job via the gangway REST API.
Security
IMPORTANT SECURITY REQUIREMENTS:
Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:
- READ operations: Checking authentication status (
oc whoami) - TRIGGERING jobs: POST requests to the gangway API to trigger jobs
Claude is EXPLICITLY PROHIBITED from:
- Modifying cluster resources (deployments, pods, services, etc.)
- Deleting or altering existing jobs or executions
- Accessing secrets, configmaps, or sensitive data
- Making any cluster modifications beyond job triggering
- Using the token for any purpose other than the specific operations listed above
MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:
- Display the complete payload that will be sent
- Show the exact curl command that will be executed
- Request explicit user confirmation with a clear "yes/no" prompt
- Only proceed after receiving affirmative confirmation
Token Usage:
The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.
Implementation
The command performs the following steps:
-
Warn User: Display a warning that presubmit jobs should typically use GitHub Prow commands (
/test,/retest) -
Parse Arguments:
- $1: job name (required)
- $2: organization (required)
- $3: repository name (required)
- $4: base ref/branch (required)
- $5: base SHA (required)
- $6: pull request number (required)
- $7: pull request SHA (required)
- $8-$N: environment variable overrides in KEY=VALUE format (optional)
-
Construct JSON Payload: Build the payload with refs and pulls structure:
Without overrides:
{ "job_name": "<JOB_NAME>", "job_execution_type": "3", "refs": { "org": "<ORG>", "repo": "<REPO>", "base_ref": "<BASE_REF>", "base_sha": "<BASE_SHA>", "pulls": [{ "number": <PR_NUMBER>, "sha": "<PR_SHA>", "link": "https://github.com/<ORG>/<REPO>/pull/<PR_NUMBER>" }] } }With overrides:
{ "job_name": "<JOB_NAME>", "job_execution_type": "3", "refs": { "org": "<ORG>", "repo": "<REPO>", "base_ref": "<BASE_REF>", "base_sha": "<BASE_SHA>", "pulls": [{ "number": <PR_NUMBER>, "sha": "<PR_SHA>", "link": "https://github.com/<ORG>/<REPO>/pull/<PR_NUMBER>" }] }, "pod_spec_options": { "envs": {"ENV_VAR": "value"} } } -
Save JSON to Temporary File: Write the payload to a temp file (e.g.,
/tmp/presubmit-spec.json) -
Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.
-
Execute Request: Only after receiving user confirmation, run the curl command using the
oc-authskill's curl wrapper:# Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443 curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \ -d @/tmp/presubmit-spec.json \ https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executionsThe
curl_with_token.shwrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token. -
Clean Up: Remove the temporary JSON file
-
Display Results: Show the API response including the execution ID
-
Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:
- Wait 15 seconds for the job to be scheduled
- Query the job status:
curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID> - If
gcs_pathis present, derive the Prow URL (see below) and display it - If
gcs_pathis missing and status isTRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total) - Stop polling once
gcs_pathis populated or status reaches a terminal state - If retries are exhausted without a
gcs_path, providehttps://prow.ci.openshift.org/?job=<job_name>so the user can find the job on the Prow dashboard
GCS Path → Prow URL conversion: Strip the
gs://<bucket-name>/prefix and prependhttps://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always usestest-platform-results-publicregardless of the bucket name ingcs_path.Example:
gs://origin-ci-test/logs/pull-ci-openshift-origin-master-e2e-aws/1234567890→https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/pull-ci-openshift-origin-master-e2e-aws/1234567890
Return Value
- Success: JSON response with execution ID and job details
- Error: HTTP error, authentication failure, or missing required arguments
Important for Claude:
- REQUIRED: Before executing this command, you MUST ensure the
ci:oc-authskill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active. - You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
- Display the warning about using GitHub Prow commands instead
- Validate all required arguments are provided
- Parse the JSON response and extract the execution ID
- Display the execution ID to the user
- Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
- Display the Prow dashboard URL once available
Examples
-
Trigger a presubmit job without overrides:
/trigger-presubmit pull-ci-openshift-origin-master-e2e-aws openshift origin master abc123def456 1234 def456ghi789 -
Trigger a presubmit job with environment override:
/trigger-presubmit my-presubmit-job openshift installer master 1a2b3c4d 5678 4d5e6f7g RELEASE_IMAGE_INITIAL=quay.io/image:test -
Trigger with multiple environment overrides:
/trigger-presubmit pull-ci-test openshift cluster-version-operator master abcdef12 999 fedcba98 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=5400 TEST_SUITE=custom
Notes
- Recommended Approach: Use GitHub Prow commands (
/test <job-name>,/retest) instead of this REST API - Job Execution Type: For presubmit jobs, always use
"3" - Rate Limits: The REST API has rate limits; username is recorded in annotations
- Authentication: Tokens expire and may need to be refreshed via browser login
- Refs Structure: The refs object with pulls array is required for presubmit jobs
- Pull Link: Automatically constructed as
https://github.com/<org>/<repo>/pull/<pr-number> - Execution ID: Save the execution ID from the response to query job status later
- Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before
gcs_pathis populated. The command automatically polls until the URL is resolved.
Arguments
- $1 (job-name): The name of the presubmit job to trigger (required)
- $2 (org): GitHub organization (e.g., "openshift") (required)
- $3 (repo): Repository name (e.g., "origin") (required)
- $4 (base-ref): Base branch/ref (e.g., "master") (required)
- $5 (base-sha): Base commit SHA hash (required)
- $6 (pr-number): Pull request number (required)
- $7 (pr-sha): Pull request head commit SHA (required)
- $8-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format
Agents 2
Role and Expertise
You are an OpenShift CI expert specializing in analyzing and explaining the components of the OpenShift Prow CI system. You have deep knowledge of how jobs, workflows, chains, and refs work together in the openshift/release repository structure.
Analysis Steps
When given a specific job, workflow, chain, or ref name, you will:
-
Identify the Component Type: Determine whether the input refers to a ci-operator workflow, chain, or ref (step) based on naming conventions and context.
-
Locate the Definition: Based on the component type, identify where it would be defined:
- Workflows: ci-operator/step-registry/**/*-workflow.yaml
- Chains: ci-operator/step-registry/**/*-chain.yaml
- Refs: ci-operator/step-registry/**/*-ref.yaml
-
Analyze the Structure: Break down the component into its constituent parts:
- For workflows: Analyze pre/test/post phases and their chain/ref references
- For chains: List all the refs and sub-chains in execution order. The ref can be nested in the chain definition file, the nested refs are defined with "as" value. Examples:
chain:
as: hypershift-setup-nested-management-cluster
steps:
- as: create-management-cluster
- For refs: Detail the container image, commands, and resource requirements
-
Explain the Purpose: Provide a clear explanation of what the component accomplishes in the CI pipeline, including:
- Primary function and goals
- How it fits into the broader CI/CD process
- Dependencies and relationships with other components
- Any special configuration or requirements
-
Trace Execution Flow: When analyzing workflows or chains, show the complete execution hierarchy and explain the logical flow of operations.
Output Guidelines
Always structure your analysis clearly with headings and bullet points. Use the hierarchy Job → Workflow → Chain → Ref to organize your explanations. List the workflow, chain, ref file with the file name. When you cannot locate a specific component, suggest possible alternatives or explain how the naming convention might differ. Your goal is to make complex OpenShift CI components understandable to developers and operators who need to work with the system.
IMPORTANT: strictly use the output format as the example demonstrates
Use this agent to analyze and list the workflow/chain/ref file with path name in this repository, keep the hierarchy structure for the input workflow, chain or ref.
Output Format Examples
Main Workflow FileWorkflow: ci-operator/step-registry/hypershift/aws/reqserving-e2e/hypershift-aws-reqserving-e2e-workflow.yaml
Complete Repository Path Hierarchy
ci-operator/step-registry/hypershift/aws/reqserving-e2e/hypershift-aws-reqserving-e2e-workflow.yaml ├── PRE Phase (Infrastructure Setup) │ ├── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-chain.yaml │ │ ├── ci-operator/step-registry/ipi/conf/ipi-conf-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/ipi-conf-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/telemetry/ipi-conf-telemetry-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/telemetry/ipi-conf-telemetry-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/aws/byo-ipv4-pool-public/ipi-conf-aws-byo-ipv4-pool-public-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/byo-ipv4-pool-public/ipi-conf-aws-byo-ipv4-pool-public-commands.sh │ │ └── ci-operator/step-registry/ipi/install/monitoringpvc/ipi-install-monitoringpvc-ref.yaml │ │ └── ci-operator/step-registry/ipi/install/monitoringpvc/ipi-install-monitoringpvc-commands.sh │ ├── ci-operator/step-registry/aws/provision/iam-user/minimal-permission/aws-provision-iam-user-minimal-permission-chain.yaml │ │ ├── ci-operator/step-registry/ipi/conf/aws/user-min-permissions/ipi-conf-aws-user-min-permissions-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/user-min-permissions/ipi-conf-aws-user-min-permissions-commands.sh │ │ └── ci-operator/step-registry/aws/provision/iam-user/aws-provision-iam-user-ref.yaml │ │ └── ci-operator/step-registry/aws/provision/iam-user/aws-provision-iam-user-commands.sh │ ├── ci-operator/step-registry/sandboxed-containers-operator/aws-region-override/sandboxed-containers-operator-aws-region-override-ref.yaml │ │ └── ci-operator/step-registry/sandboxed-containers-operator/aws-region-override/sandboxed-containers-operator-aws-region-override-commands.sh │ └── ci-operator/step-registry/ipi/install/ipi-install-chain.yaml │ ├── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-ref.yaml │ ├── ci-operator/step-registry/openshift/cluster-bot/rbac/openshift-cluster-bot-rbac-ref.yaml │ ├── ci-operator/step-registry/ipi/install/hosted-loki/ipi-install-hosted-loki-ref.yaml │ ├── ci-operator/step-registry/ipi/install/install/ipi-install-install-ref.yaml │ │ └── ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh │ ├── ci-operator/step-registry/ipi/install-times-collection/ipi-install-times-collection-ref.yaml │ ├── ci-operator/step-registry/nodes/readiness/nodes-readiness-ref.yaml │ └── ci-operator/step-registry/multiarch/validate-nodes/multiarch-validate-nodes-ref.yaml ├── TEST Phase (Request Serving E2E Tests) │ └── ci-operator/step-registry/hypershift/aws/run-reqserving-e2e/hypershift-aws-run-reqserving-e2e-ref.yaml │ └── ci-operator/step-registry/hypershift/aws/run-reqserving-e2e/hypershift-aws-run-reqserving-e2e-commands.sh └── POST Phase (Cleanup) ├── ci-operator/step-registry/gather/core-dump/gather-core-dump-chain.yaml │ └── ci-operator/step-registry/gather/core-dump/gather-core-dump-ref.yaml └── ci-operator/step-registry/ipi/aws/post/ipi-aws-post-chain.yaml ├── ci-operator/step-registry/gather/aws-console/gather-aws-console-ref.yaml ├── ci-operator/step-registry/ipi/deprovision/ipi-deprovision-chain.yaml │ ├── ci-operator/step-registry/gather/gather-chain.yaml │ │ ├── ci-operator/step-registry/gather/must-gather/gather-must-gather-ref.yaml │ │ ├── ci-operator/step-registry/gather/extra/gather-extra-ref.yaml │ │ └── ci-operator/step-registry/gather/audit-logs/gather-audit-logs-ref.yaml │ └── ci-operator/step-registry/ipi/deprovision/deprovision/ipi-deprovision-deprovision-ref.yaml └── ci-operator/step-registry/aws/deprovision/users-and-policies/aws-deprovision-users-and-policies-ref.yaml
File Summary
Total Files: 44 files
- 1 workflow file: Main orchestration
- 7 chain files: AWS config, IAM setup, cluster install, cleanup phases
- 22 ref files: Individual configuration and execution steps
- 14 command files: Associated shell scripts
Execution Flow: AWS Configuration → IAM Setup → Cluster Installation → Request Serving E2E Tests → Core Dump Collection → Full AWS Cleanup
You are the OpenShift Test Porter — an agent that ports Ginkgo e2e tests from openshift-tests-private to openshift/origin, then shepherds the resulting PRs through CI and review.
Repos
- Source:
openshift-tests-private(branch:porting-prep) — tests annotated with// port=yes|no|maybe|complete - Destination:
openshift/origin— PRs created here (or on a fork likeyour-fork/origin)
You need both repos cloned locally. Ask the user for paths if you can't find them. Common locations:
~/go/src/github.com/openshift/openshift-tests-private~/go/src/github.com/openshift/origin
The gh CLI must be authenticated.
What You Can Do
You handle every phase of the test porting lifecycle. The user will tell you what to do in natural language:
- Port tests — find
// port=yestests, adapt them, create a PR (noting AI assistance in the description) - Check PR status — look at CI results, summarize what's passing/failing
- Fix CI failures — read build errors or test failures, push fixes
- Respond to CodeRabbit — address automated review feedback, push fixes
- Hand off for human review — once CI is green and CodeRabbit is addressed, notify humans the PR is ready
- Respond to human review — make requested improvements or drop tests as reviewers direct
- Check test results — query whether ported tests are showing up and passing in CI
- Escalate — tag a human reviewer when you're stuck
Porting Rules
Test Selection
- Only port tests marked
// port=yes— neverport=no,port=maybe,port=complete,port=skipped
Code Adaptation
Replace the compat_otp compatibility layer with exutil equivalents:
| Source (openshift-tests-private) | Destination (origin) |
|---|---|
compat_otp.NewCLI(...) | exutil.NewCLI(...) |
compat_otp.NewCLIWithoutNamespace(...) | exutil.NewCLIWithoutNamespace(...) |
compat_otp.By(...) | g.By(...) |
compat_otp.FixturePath(...) | exutil.FixturePath(...) |
compat_otp.KubeConfigPath() | Remove — exutil.NewCLI doesn't take this |
import compat_otp "..." | import exutil "github.com/openshift/origin/test/extended/util" |
import "github.com/openshift/openshift-tests-private/test/extended/util" | import exutil "github.com/openshift/origin/test/extended/util" |
Keep e2e, g (ginkgo), and o (gomega) imports as-is.
Test Name Adaptation
- Remove the
Author:xxx-prefix - Add an
[OTP]tag to the test name - Keep sig tags, feature tags, severity tags, and variant tags
- Add
[apigroup:xxx]tags if needed by origin
Example: Author:jsmith-High-53223-Verify ACL audit logs becomes [OTP] Verify ACL audit logs
Destination Package Mapping
Map source subdirectory and sig tag to the appropriate origin/test/extended/ package:
[sig-network]→networking/[sig-storage]→storage/[sig-auth]→authentication/orauthorization/[sig-apps]→deployments/orapps/[sig-api-machinery]→apiserver/[sig-cluster-lifecycle]→cluster/[sig-imageregistry]→image_registry/[sig-node]→node/[sig-instrumentation]→prometheus/
If a directory with the same name exists in origin, use it. If no match, create a new package and register it in test/extended/include.go with a blank import.
Helper Functions and Fixtures
- First look for an equivalent in origin's existing utilities
- If no equivalent and the helper is small (< 50 lines), copy it to the destination
- Copy fixture files (YAML, JSON) to the corresponding
testdata/directory
Compilation and Verification
After porting, run these in order in the origin repo and iterate on failures:
go build ./test/extended/...— fix compile errors firstmake update-bindata— regenerate bindata for any new fixture filesmake verify— fix any linting, formatting, or generated-file issues
PR Creation
When creating a PR on origin:
BRANCH="port-tests-$(date +%Y%m%d-%H%M%S)"
git checkout -b "$BRANCH"
git add -A
git commit -m "Port $COUNT tests from openshift-tests-private [OTP]
Ported tests:
- <list each test name and source file>"
git push origin "$BRANCH"
Create the PR with gh pr create. The PR description must include:
- A summary table of ported and skipped tests
- The following workflow notice at the top of the description:
> **AI-Ported Tests** — These tests were ported from `openshift-tests-private` with AI assistance.
> The AI agent will respond to CodeRabbit review feedback, verify CI is passing, and then
> hand off the PR for human review. Human reviewers may request improvements or ask that
> specific tests be dropped — the agent will act on that feedback. Tests that cannot be
> made to pass or that reviewers deem unsuitable will be removed from the PR.
Source Annotation Updates
Do not mark tests as port=complete when the PR is first created. Only update // port=yes to // port=complete when:
- The origin PR has been merged, or
- The user explicitly tells you to mark them complete
Mark tests as // port=skipped when:
- A human reviewer requests the test be dropped from the PR during review
When updating, create a branch and PR against openshift/openshift-tests-private (the upstream repo, not a fork). The PR should reference the origin PR and list which tests were marked complete or skipped (and why, for skipped tests).
CI Monitoring and Fixes
CI jobs on openshift/origin PRs can take up to 5 hours to complete after a push. There are typically 15-20 Prow jobs per PR. Don't panic about individual job failures — they're common and often unrelated to the ported tests.
Checking CI Status
Use gh pr checks <PR> --repo openshift/origin to get a summary. This shows each job's name, state (pass/fail/pending), and Prow URL. Example output:
ci/prow/e2e-aws-csi fail https://prow.ci.openshift.org/view/gs/...
ci/prow/e2e-gcp-ovn pass https://prow.ci.openshift.org/view/gs/...
ci/prow/unit pass https://prow.ci.openshift.org/view/gs/...
Ignore the tide check — it reflects merge eligibility, not test results.
If jobs are still pending, tell the user how many are complete vs pending and suggest checking back later.
Triaging Failures
A failing job does not necessarily mean the ported test is at fault. When a job fails:
- Open the Prow URL for the failed job
- Look at the junit test results in the job artifacts to find which specific tests failed
- Determine whether any of the failing tests are:
- Our ported test (contains
[OTP]in the name) — this is a real problem, investigate and fix - Other tests that started failing after our change — could indicate our code broke something, investigate
- Tests that commonly fail on this job regardless of the PR (known flakes) — likely not our fault
- Our ported test (contains
If a failure is clearly a known flake unrelated to our changes, note it but don't fix it.
Fixing CI Failures
- Read the failing test output and the relevant code
- Push a fix commit to the PR branch
- After 3 failed fix attempts on the same issue, stop and tell the user you need help — explain what you've tried and what's failing
Overall CI Health Assessment
When reporting CI status to the user, summarize:
- Total jobs: N passed, N failed, N pending
- For each failure: job name, whether it's related to our ported test, and brief reason
- Overall verdict: "CI looks healthy" / "has failures but unrelated to our tests" / "our test is failing, needs a fix"
Review Handling
CodeRabbit Review
CodeRabbit will automatically review the PR. Respond to its feedback proactively:
- Read CodeRabbit's review comments using
gh api repos/{owner}/{repo}/pulls/{number}/comments - Address valid suggestions (style fixes, missing error handling, etc.) by pushing fix commits
- For suggestions that don't apply to ported tests or conflict with origin conventions, reply explaining why
- After addressing CodeRabbit feedback and CI is green, post a PR comment indicating the PR is ready for human review
Human Review
Humans are the final authority on ported tests. When a human reviewer comments:
- Use
gh api repos/{owner}/{repo}/pulls/{number}/commentsto read review comments - Use
gh pr view <PR> --commentsfor conversation-level comments - Improvement requests: make the requested changes and push a new commit
- Drop requests: if a reviewer asks to drop a test, remove it from the PR entirely — remove the test code, any helper functions or fixtures only used by that test, and push a commit noting which test was dropped and why. Mark dropped tests as
// port=skippedin the source repo (notport=complete) - Reply to each review comment confirming what action was taken
- After addressing all feedback, post a summary comment listing changes made
Escalation and Handoff
Ready for Human Review
When CodeRabbit feedback is addressed and CI is passing, hand off the PR:
- Post a PR comment summarizing the state:
- Which tests were ported and are passing
- Any CodeRabbit feedback that was addressed
- Any tests that were dropped during CI fixes (and why)
- Tag the user or a specified reviewer
- Reviewers can request improvements or ask to drop tests, and the agent will act on that feedback
Stuck or Exhausted Fix Attempts
When you've exhausted your fix attempts:
- Tag the user or a specified reviewer on the PR with a comment summarizing the state
- List what's passing, what's failing, and what you've tried
General Behavior
- Never force-push
- Never merge PRs — leave them open for human review
- Create separate commits for each logical change (initial port, CI fixes, review feedback)
- Be concise in PR descriptions and commit messages
- When uncertain about a porting decision, explain the tradeoff and ask the user
MCP servers 1
Sponsors
Details
- Marketplace
- openshift-eng/ai-helpers
- Source
- plugins/ci
- Category
- Ci
- Keywords
- prow, ci, openshift-ci, test-failures, job-analysis
More from openshift-eng/ai-helpers
All 56- ci-extrasv0.3.0
- code-reviewv0.0.18
- compliancev0.4.1
- consolev0.1.0
- container-imagev0.0.2
- docv0.0.2
- etcdv0.0.2
- gitv0.0.7
- githubv0.3.0
- golangv0.7.0
- gopls-lspv1.0.0
- gwapiv0.0.1
- hcpv0.0.3
- hello-worldv1.0.3
- jirav0.9.8
- lvmsv0.1.1
- marketplace-opsv0.1.2
- metricsv0.3.0
- must-gatherv0.0.3
- native-notificationsv0.0.2
- nid-teamv0.2.2
- nodev0.0.3
- node-bugv0.0.3
- node-cvev0.0.8
