A plugin to work with OpenShift CI and analyze Prow job results

version
0.0.111
GitHub stars
123

Install

/plugin install ci@ai-helpers

Run it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.

Contents

Discovered from source

Skills 36

Add JIRA Triage Link

This skill adds a Component Readiness triage record link to a JIRA issue description so that anyone viewing the bug can monitor the on-going status of all regressions triaged to it.

When to Use This Skill

Use this skill after creating or updating a triage record via the triage-regression skill. The link gives bug viewers a way to see all regressions associated with the bug in Component Readiness.

Prerequisites

  1. Jira credentials in the environment, either:
    • JIRA_USERNAME and JIRA_API_TOKEN (Atlassian account email and API token), or
    • JIRA_CLIENT_ID, JIRA_CLIENT_SECRET and JIRA_CLOUD_ID (service-account OAuth credential; used whenever the client ID or secret is set, with no fallback to the API token)
  2. Python 3.6+

Implementation Steps

Step 1: Run the Script

script_path="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"

python3 "$script_path" OCPBUGS-12345 \
  --triage-id 456 \
  --format json

Arguments:

  • issue_key: JIRA issue key (e.g., OCPBUGS-12345)

Required Options:

  • --triage-id <id>: Triage record ID from Component Readiness

Options:

  • --format json|text: Output format (default: text)

Step 2: Parse the Output

output=$(python3 "$script_path" OCPBUGS-12345 --triage-id 456 --format json)
success=$(echo "$output" | jq -r '.success')

if [ "$success" = "true" ]; then
  already=$(echo "$output" | jq -r '.already_present')
  if [ "$already" = "true" ]; then
    echo "Triage link already in description"
  else
    echo "Triage link added"
  fi
fi

Script Output Format

JSON Format (--format json)

Success (link added):

{
  "success": true,
  "issue_key": "OCPBUGS-12345",
  "triage_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/456",
  "already_present": false
}

Success (link already present):

{
  "success": true,
  "issue_key": "OCPBUGS-12345",
  "triage_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/456",
  "already_present": true
}

Error:

{
  "success": false,
  "error": "Failed to update description: HTTP 403: ...",
  "issue_key": "OCPBUGS-12345"
}

Notes

  • The script is idempotent — if the triage link is already in the description, it skips the update
  • Uses Atlassian Document Format (ADF) to append a horizontal rule and the triage link section
  • Appends to the existing description without modifying existing content
  • Requires JIRA API v3 write access to the issue

See Also

  • Related Skill: triage-regression (creates triage records that produce the triage ID)
  • Related Skill: set-release-blocker (similar pattern for updating JIRA issue fields)
  • Related Command: /ci:analyze-regression (calls this skill after triaging)

Bulk Triage Regressions

Input

bulk-triage-regressions <view> [--components comp1 comp2 ...] [--auto-triage] [--existing-only]
bulk-triage-regressions --views <v1> <v2> ... --policy post-ga [--write-mode report|auto]

Example: bulk-triage-regressions 5.0-main --components Installer Unknown

Description

This skill implements the Component Readiness triage duty workflow: it fetches all untriaged regressions for a set of components in a view (e.g., 5.0-main, components Installer and Unknown), analyzes them as a batch, clusters them into root-cause buckets, and then triages each bucket to a single JIRA bug (existing or new).

This differs from /ci:analyze-regression (which analyzes a single regression in depth). Triage duty requires a holistic view, because:

  1. Many regressions, few root causes. One product bug commonly opens 5–30 regressions across variants (different platforms, arches, featuresets, upgrade modes) and across "wrapper" tests (install should succeed: overall, : cluster bootstrap, : cluster creation, verify the cluster readiness and stability, mass-failure tests, etc.). Analyzing regressions one-by-one wastes effort and risks filing duplicate bugs. Cluster first, deep-dive once per cluster.

  2. Component attribution is often wrong. Regressions in Installer and Unknown are catch-all attributions. A failed installation or bootstrap is frequently caused by a specific component — e.g., a monitoring operator failing to go available blocks cluster creation, an etcd slowness issue breaks bootstrap, an MCO bug degrades nodes during install. The Sippy component label tells you which test failed, not whose bug it is. The real owner must be determined from artifacts (cluster operator status, log bundle, operator logs), and the JIRA bug must be filed against the actual owning component, not Installer.

Use this skill when doing triage duty for a view, or whenever a user asks to "look at all untriaged regressions from " rather than a single regression ID.

Implementation

Script invocation rules: Run Python skill scripts directly and analyze their JSON output with your own reasoning (pass --format json where the script offers the flag; scripts without it, such as list_regressions.py, emit JSON by default). Do not pipe script output through inline Python one-liners. Do not suppress stderr: if a script exits non-zero or returns invalid/empty JSON, stop and surface the error — an authentication or API failure must never be mistaken for an empty inventory ("nothing to triage").

Authentication: Read steps (listing, fetching details, test runs, GCS artifacts) require no auth — Phases 1–3 and a read-only report must work without any credentials. Two Sippy hosts exist and the split matters: https://sippy.dptools.openshift.org is public and read-only (regressions, test details, and the full symptom/label catalog — see the list-symptoms skill), while everything on https://sippy-auth.dptools.openshift.org requires a DPCR Bearer token, including the reevaluate probe, which has no public equivalent (the route returns 404 on the public host). The only Phase 3 step that needs a token is therefore the active symptom probe. When it is unavailable (no token, or a 401/403), treat the result as an unknown — never as "no symptoms matched", the same error as reading a failed listing script as an empty inventory, and work down this fallback: (1) passive label_summary / job_labels from fetch-regression-details; (2) list the catalog from the public API with the list-symptoms skill and grep each plausible symptom's match_string against its file_pattern in the GCS artifacts you are already fetching — this reproduces the probe locally for the handful of symptoms related to the bucket's stage and platform; (3) only then the full evidence ladder, recording that the active probe was unavailable so the next shift knows the catalog was checked by hand. Write credentials are validated only when writes are going to happen: with --auto-triage, validate both up front (so an expired token surfaces before hours of analysis); otherwise validate at the start of Phase 4, before the first write. Sippy writes (creating/updating triage records) require a Bearer token from the DPCR cluster (api.cr.j7t7.p1.openshiftapps.com:6443) — see the oc-auth skill and the token-extraction snippet in /ci:analyze-regression; check with an authenticated GET against https://sippy-auth.dptools.openshift.org/api/component_readiness/triages (200 vs 401/403).

JIRA writes (filing bugs, set-release-blocker, add-jira-triage-link) additionally require JIRA credentials in the environment: either JIRA_USERNAME and JIRA_API_TOKEN (API token from https://id.atlassian.com/manage-profile/security/api-tokens), or a service-account OAuth credential in JIRA_CLIENT_ID, JIRA_CLIENT_SECRET and JIRA_CLOUD_ID, which takes precedence when set and never falls back to the API token. Verify either kind with python3 plugins/ci/scripts/jira_auth.py check (exit 0 and the account vs an error). With a service-account credential, Basic auth against https://redhat.atlassian.net does not work: send every direct JIRA REST call (searches, comments, issue creation when the Jira MCP tools are unavailable) through python3 plugins/ci/scripts/jira_auth.py request <METHOD> '/rest/api/3/...' [--data-file body.json], which obtains the access token itself and prints only the response. If a required write credential is missing or invalid, pause before Phase 4 and ask the user to fix it — the analysis and report so far remain valid and must still be presented.

Phase 0: Run the pre-pass (post-ga only)

With --policy post-ga, first run scripts/prepass.py with the given views. Treat the prepass.json clusters as the complete batch inventory: do not re-derive clusters, and do not call the list/details/related APIs for inventory. Analyse clusters in file order (HIGH first) within the time budget. A symptom_probe.status other than ok means unknown, never "no symptoms matched". Read and follow references/post-ga-policy.md for everything else in this mode, and skip Phase 1's collection steps.

Phase 1: Collect the full batch

  1. Load CI context: Read the files in plugins/ci/references/ (jobs.md, tests.md, sippy-apis.md) for conventions on tests, jobs, and Sippy APIs.

  2. Parse arguments:

    • view: required, e.g. 5.0-main
    • --components: component filter list, e.g. Installer Unknown. Matching is case-insensitive and hierarchy-aware: a filter matches the full component name or any /-separated segment of it, so Installer also covers Installer / openshift-installer, and Networking covers Networking / ovn-kubernetes, Networking / router, and every other Networking / * component. If omitted, ask the user which components the duty covers.
    • --auto-triage: if present, triage buckets without per-bucket confirmation when confidence is high (see Phase 4). Default is to present findings and confirm before writing.
    • --existing-only: never file new JIRA issues; write only for matches to existing issues (see "Existing-only mode" in Phase 4).
  3. List regressions with the list-regressions skill:

    python3 plugins/teams/skills/list-regressions/list_regressions.py \
      --view <view> --components <components...>
    

    Keep only open, untriaged regressions (empty triages array), but note recently-triaged ones — they are prime candidates for absorbing untriaged siblings.

    Closed regressions are out of scope — even when untriaged. A regression whose closed field is set has already resolved itself; do not inventory it, cluster it, deep-dive it, or recommend retroactive triage for it. The duty batch consists solely of open untriaged regressions. Closed regressions may be consulted as evidence (e.g., a closed sibling that shares a root cause with an open bucket, or a closed sibling whose existing triage/JIRA an open bucket should reuse — see Pitfalls), but they must never appear as bucket members, action items, or "leftovers" in the report. The only exception is the explicit closed-set audit mode (--audit-closed, see below), which the user must request by name — it is never part of a normal duty run.

  4. Build a batch inventory table: For every open untriaged regression record: regression ID, test name, component/capability, variants (Platform/Arch/Network/Topology/FeatureSet/Upgrade), opened date, failure/run counts. Present this table to the user up front so the scope of the duty run is visible. Do not include closed regressions in the inventory.

  5. Stale-triage sweep (mandatory) — a 100% triaged board can still hide live defects. For every open, already-triaged regression, compare its last_failure against the state of its triage's JIRA: a regression whose bug is Closed/Verified/resolved but which has failed after the resolution date is an alarm, not a statistic. Its fresh failure window is either a failed fix or — more often — a different cause hiding behind the old triage record. For each such regression, verify the recent runs' signature (armed Sippy symptoms are a cheap first oracle: dry-run reevaluate the newest runs — label hits map windows to known causes instantly; unlabeled recent runs mean a new, uninvestigated cause; a hit label whose bugs differ from the triage's JIRA suggests a new cause, one listing the same closed bug a failed fix; a label linking several bugs is inconclusive — compare the runs' signature and timing with each) and add the correct triage(s) for the new window rather than trusting the stale one. (Case 4 in case-notes.md: three "triaged" regressions pointing at a Closed bug while failing daily from two new causes.)

  6. Long-lived wrapper regressions are cause timelines, not single buckets. A wrapper regression that stays open for weeks accumulates causes, each window separately verified and separately triaged. When a previously-analyzed regression shows new last_failure dates, re-verify the new window from scratch — never assume the existing triage covers it. (Case 13: one record carried four causes.)

Phase 2: Cluster into candidate buckets (cheap signals first)

Before any deep log analysis, group regressions using signals already in hand:

  • Same test, different variants — almost always one bucket.
  • Same variant fingerprint, different tests — e.g., install should succeed: overall + : cluster creation + verify the cluster readiness and stability all failing on azure/amd64/techpreview starting the same day is one bucket. Wrapper tests fail together.
  • Same opened date — regressions opened the same day across components often share one payload-level cause.
  • Shared job runs — fetch details for each regression (fetch-regression-details skill) and compare job_runs prowjob_run_ids. Regressions observed in the same failed runs are strong candidates for one bucket — but this is a clustering signal, not proof: the same run can contain independent defects or be a mass failure, so Phase 3 validation is still required. Also run the fetch-related-triages skill per regression; same_last_failure and similarly_named_test matches feed the clustering, and triaged_matches with confidence ≥5 immediately suggest an existing triage/bug for the whole bucket.
  • Symptom labels — fetch-regression-details returns label_summary (per job) and job_labels (per run). A label shared by most failed runs of several regressions is a strong bucket signal, and labels are precise (human-written matchers over artifacts). label_bugs maps each label to its linked Jira keys: labels sharing a bug are a bucket signal, and the bug is a candidate triage target. An empty label list means nothing was detected — not that nothing is wrong, and often just that the runs were never swept: Phase 3 opens with an active dry-run reevaluation that closes this gap.
  • Mass-failure marker: high test_failures counts in job_runs mean the regression is likely collateral of a bigger event, not an independent issue.

Output of this phase: a draft bucket list, each bucket with member regression IDs, the shared fingerprint (test/variant/date/job-run overlap), and any candidate existing triage or JIRA bug.

Treat buckets as hypotheses — Phase 3 must confirm or split them. Do not merge buckets merely because both are "install failures"; installs fail at different stages for unrelated reasons.

Phase 3: Deep-dive each bucket (confirm root cause and real owner)

For each bucket, pick 2–5 representative failed job runs (spread across jobs/variants; include the newest) and analyze. 2–5 runs are sufficient only when they yield a consistent result (same error signature / failure stage across all of them). If the sample is mixed or unclear — different errors, different stages, or an inconclusive owner — extend the sample to 10–20 runs before concluding; a small ambiguous sample must never be the basis for splitting/merging a bucket or attributing an owner.

Cross-cutting analysis principles — most specific rules below (and most Pitfalls) are instances of these three; when a situation matches none of the specific rules, fall back to the principle:

  • Messenger vs. owner. The component that reports a failure is presumed the messenger until artifacts prove it is the producer. This covers: the installer for install failures, wrapper tests for whatever blocked them, a test whose setup failed on another component's error (Case 9 in case-notes.md), a crash-looper whose error names a missing upstream dependency, and any component that names a bad value someone else computed (Case 12). Ask "who produced the condition this error describes?" and walk upstream until the answer is "this component itself".
  • Population claims require population evidence. Any claim quantifying over runs — "resolved", "stopped", "all runs show X", "only variant Y" — must enumerate the population it quantifies over: all of the regression's job_runs by date (never a 3–4 run sample; Case 1), only runs from the regression's own job_runs list (never a broad job-filter query; Case 10), and source-code gating for any "X-only" scope claim.
  • Consult known-cause catalogs before deriving from scratch. Cheap indices exist for almost every question and must be queried before artifact dives or new-bug filings: the armed symptom catalog (dry-run reevaluate), existing triage records, the owning component's recent JIRA bugs regardless of keywords, the owning repo's merge history at onset/cessation boundaries, and closed/dropped sibling regressions. (Case 3: one dry-run call resolved a board that cost a parallel run 264 shell commands.)

Symptom-catalog oracle first (cheap, decisive — run before any artifact dive or subagent dispatch). The passive label_summary from Phase 2 only shows labels already applied by past sweeps; recent runs are typically unswept and show nothing. Actively probe the catalog instead: POST https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate with {"prow_job_build_ids": [...], "dry_run": true} (DPCR Bearer token) for 3–5 representative runs per bucket, spread across platforms. Reevaluation scans the runs' GCS artifacts server-side against every armed symptom matcher and returns symptoms_matched per run — one call can attribute an entire bucket to a known incident in seconds (Case 3 in case-notes.md). A matched label's bugs (see list-symptoms --labels) name the candidate triage target. Matched symptoms are a hypothesis with a named cause — still verify per the steps below (the label names the incident; confirm the causal chain to the bucket's specific test) — but they dictate where to look first and usually collapse the deep-dive to a single confirmatory artifact read. No matches means the cause is not yet cataloged — proceed with the full evidence ladder.

No token? The catalog is still public — degrade, never skip. A 401/403 from the probe is an unknown, never "no symptoms matched": use the credential-free fallback in the Authentication section above, and say in the report which rung was used.

Deep-dive at least one CI sample per bucket — always, even at high confidence. A Sippy triaged_matches confidence of 10 or a same-day triaged sibling is a hypothesis, not a verdict: Sippy matches on test names and shared job runs, which produces conf=10 for the same test name across unrelated platforms and root causes (see Pitfalls). Before accepting any disposition — including "extend existing triage" — read the actual failure evidence for at least one representative run of the bucket (failure output at minimum; installer logs / artifacts for install wrappers) and confirm it matches the target triage's root cause. You have deeper analysis capabilities than Sippy's heuristics — use your own judgement on the raw evidence, and if it contradicts the Sippy categorization, trust the evidence and re-bucket.

  1. Failure outputs: fetch-test-runs skill with the bucket's test IDs and job run IDs — check whether error messages are consistent within the bucket. >90% same error is strong evidence of a single cause, not confirmation: wrapper tests and mass failures print identical error text for unrelated defects (e.g., "nodes not ready" covers disk, memory, and network deaths alike). Confirm with items 2–3 below before treating the bucket as one cause; inconsistent errors ⇒ split the bucket.

  2. Job run context: fetch-job-run-summary skill per representative run — is the regressed test isolated, part of a consistent co-failure set, or one of hundreds of random failures? For Unknown-component and mass-failure regressions this is where the real component reveals itself: read the names of the co-failing tests.

  3. Install/bootstrap failures — mandatory artifact dig: For any bucket whose tests include install should succeed (any stage) or bootstrap/cluster-readiness wrappers, invoke the prow-job-analysis skill per representative run. Do not stop at Sippy's generic "install failed" wrapper. From the GCS artifacts determine:

    • Failure stage: infrastructure provisioning / bootstrap / cluster creation (operators rolling out) / stability window.
    • The blocking condition: for cluster-creation failures, read clusteroperators.json (or the installer log's "Cluster operator X is not available" lines) and the failing operator's pod logs from the log bundle. For bootstrap failures, read the bootstrap log bundle (etcd, bootkube, release-image pulls).
    • Bootstrap-era logs are full of normal transients — validate every causal theory against the gathered end-state. During any bootstrap, operator logs contain scary-looking messages that resolve on their own ("the server could not find the requested resource (post routes.route.openshift.io)" before openshift-apiserver is up, static pods "0 nodes at revision 0" early on). Quoting one of these as the root cause is only valid if the final cluster state agrees. The end-state oracle for bootstrap-stage failures is clusteroperators.json, not pod existence: check etcd's StaticPodsAvailable/revision status and openshift-apiserver's APIServicesAvailable directly — a bootstrap-timed-out run whose gather still shows etcd Available=False: 0 nodes are active; 3 nodes are at revision 0 was blocked by etcd no matter what else looks broken. Do not infer "etcd/apiserver were fine" from workload pods running: during bootstrap the bootstrap-node etcd/apiserver serve the control plane, so pods run happily while cluster etcd never deploys. A crash-looping container's *_previous.log is primary evidence for that container's failure — but before assigning ownership, check whether its error names a missing upstream dependency: such a component is a victim, not an owner.
    • "Operators were not stable" — count condition transitions before classifying. "Healthy at gather time" does NOT imply a harmless transient. Grep the install log for the operator's condition lines and look at LastTransitionTime/DurationSinceTransition: hundreds of transitions with DurationSinceTransition=1s across the stability window means the operator is flapping continuously (a sync-loop product bug that will never pass the stability check — permafail), whereas a single long Progressing=True window that eventually clears is a slow rollout (flaky race against the timeout). These two have opposite classifications and dispositions. (Case 1: a ~1 Hz flap, 1600+ transitions in 30 minutes, mislabeled "self-resolved slow rollout" from a 3-run sample.)
    • Race/ordering theories require timestamp proof — from artifacts, not from the error string. Any "X was not ready when Y ran" mechanism must be proven with timestamps the artifacts already contain (creationTimestamps, Established/Available conditions, phase logs, first/last occurrence of the error). If X was ready before the errors began, that refutes this ordering theory — but not every timing theory: readiness at the source does not mean visibility at the consumer, so the next candidates are cache/informer warmup, an in-progress rollout, or a stale client. Name which one the artifacts support rather than dropping the timing angle entirely. Also measure the failure window's duration: it is part of the mechanism claim and belongs in the bug.
    • Bad-value failures: trace the value to its producer — the error-emitter is usually just the messenger. When a component rejects or fails writing a value (a mapping, an ID range, a path, a quota number), walk the data flow upstream until you reach whoever computed it, however many hops away: the component that names the bad value in its error is the victim's messenger. The producer is then the prime suspect, not the automatic owner — before filing, confirm the value was already wrong as produced: check that no intermediate hop transformed a valid value into an invalid one, and that the value does not violate a documented input contract the consumer was itself obliged to enforce. Whichever hop first turned a valid value into an invalid one owns the bug; name the hops you cleared. (Case 12 in case-notes.md: a GID-mapping error filed against CRI-O when the producer was kubelet's ID allocator two hops upstream — pinns and CRI-O were verified pass-throughs.)
    • Decode suspicious constants before writing "malformed" or "overflow". A weird-looking number is a mechanism clue, not evidence of corruption: check whether it is semantically meaningful — 2^32 − size, a type maximum, a page/alignment boundary, an epoch. Do the arithmetic; never pattern-match "big hex-ish number ⇒ overflow bug". (Case 12: 4294901760 + 65536 = 2^32 — an allocator boundary, handing you the whole mechanism.)
    • The claimed mechanism must predict the observed frequency. A deterministic-bug theory cannot explain n=1 across many runs exercising the same path — a low rate demands a stochastic or boundary mechanism, and the expected probability should be stated in the bug. A mechanism/frequency mismatch means the theory is wrong; resolve it before filing, and mark any remaining mechanism prose as an explicit unverified hypothesis ("unverified: possibly …"), never as "suggesting X bug in component Y". (Case 12: the filed theory predicted every-pod failure; the real mechanism predicted ~1/65535 per pod — matching n=1.)
    • Differential recovery check — when the same error hits many objects but only some stay broken, explain the asymmetry before finalizing. Survivors show the failure was not uniformly permanent; before concluding the trigger was transient, rule out the alternatives (survivors had different exposure, a retry path, or different state). Either way the lasting failure lives in whatever failed to recover, and that component owns the bug. Pattern: transient trigger vs. persistent damage — ask which candidate defect explains the permanence of the failure, and lead with that one.
    • Read the owning repo's source before claiming a missing safeguard. A claim that "component X lacks ordering/gating/protection Y" is only valid after locating (or proving absent) that safeguard in the repo — the same rule as reading a test's skip/gates before describing its scope, applied to product code.
    • The real owner: the component whose operator/pods are actually failing. Examples from past duty: "cluster creation failed" ⇒ monitoring operator degraded ⇒ Monitoring bug; "bootstrap failed" ⇒ etcdserver: request timed out on Azure ⇒ etcd bug; nodes degraded during install ⇒ MCO bug; quota/DNS/cloud-API errors ⇒ ci-infra, not a product bug at all.
    • Stability-window failures — "operator X not available" is a symptom, not a root cause. For cluster-readiness/stability wrappers that fail on a ClusterVersion or ClusterOperator condition, two extra steps are mandatory before assigning an owner:
      1. Recover the underlying condition message when the junit output is vague. Outputs like clusterversion not available: False (empty reason) carry no cause. Run this literal check against the e2e step's build-log.txt (and, if present, the monitor-intervals JSON under the step's artifacts/junit/):

        curl -s <gcs-url-of-e2e-step>/build-log.txt | grep -oE '(Failing|Available|Degraded)=(True|False)[:,][^"\\]{0,160}' | sort | uniq -c | sort -rn | head
        

        The condition messages recovered this way (e.g., a controller error string) name the real culprit. This step is complete only when the report quotes the recovered message for every vague-output run. Never attribute such a run from its co-failing tests: co-failures on techpreview jobs are usually unrelated background noise, and correlation with them has produced wrong owners in past duty runs.

      2. Ask why the operator went unavailable, not just which operator. If everything is healthy at gather time, the wrapper caught a transient flap: read the operator's own pod log around the transition timestamps and find the trigger. Distinguish (a) the operator genuinely failing (⇒ product bug for that operator) from (b) a routine reconciliation/rollout triggered by a cluster mutation (config/secret change, node roll) — a rollout of a single-replica deployment flaps Available by design. A rollout-flap disposition must name the mutator, not stop at "the operator detected a configuration change": quote the operator-log line identifying which object changed (grep the operator pod log for object changed / secret and config names and read the surrounding lines), then identify who wrote it — audit logs (verb update/patch on that object: user + userAgent) or the job's test-harness step logs (search them for the object name or the value being set). If a test step caused the mutation mid-run, the bucket is test owned by that suite, not a product bug against the flapping operator; filing "spurious rollouts" against the operator without naming the mutator is an incomplete deep-dive. (Case 2 in case-notes.md: an image-registry flap drafted as a product bug until the operator log showed the harness had replaced the pull secret mid-run.)

  4. Test-interference buckets — read the offending test's source before describing its scope. When the root cause is another test's or tool's behavior (a test creates pods/namespaces that break an invariant check, leaks resources, reboots nodes, etc.), do not infer why the failure is confined to certain variants from the regression's variant labels — correlation with a variant (techpreview, platform, upgrade mode) is not evidence of how the offending code is gated. Instead:

    • Locate the offending code (gh search code in openshift/origin or the relevant repo for the test name, namespace prefix, or error string) and read its skip/gate conditions (skipIf..., e2eskipper.Skipf, feature-gate checks, platform checks, suite membership).
    • State the actual gating in the report (e.g., "gated on baremetal platform via skipIfNotBaremetal"), and if that gating is broader than the regressed variants, say so — the regression's variant slice then reflects job-scheduling or sample-size effects, not the blast radius, and other variants are also at risk.
    • Check the file's merge history (gh api repos/<org>/<repo>/commits?path=...) — a merge date matching the regression onset both confirms the attribution and identifies the owning team.
    • Never write "X-only" (techpreview-only, platform-only, arch-only) about a test or tool in the report or a bug unless the source code gating has been read and confirms it.
  5. Onset and suspect PRs (when the bucket has a crisp start date): follow the "Determine Regression Start Date" and "Identify Suspect PRs in Payload" procedures from /ci:analyze-regression (first failing run → payload tag via fetch-prowjob-json → fetch-new-prs-in-payload → up to 5 candidate PRs vetted with gh). A LIKELY PR both strengthens the bucket and tells you the owning component/repo.

  6. Cross-check globally: fetch-test-report skill (with --no-collapse) for the bucket's main test — confirms whether the issue is variant-specific or global, and surfaces open_bugs that may already cover the bucket.

  7. Check Slack context (optional — only when Slack access is available): The TRT/release-oversight team discusses ongoing payload and CI issues in #forum-ocp-release-oversight (https://redhat.enterprise.slack.com/archives/C01CQA76KMX). Search/read the last 14 days of messages there for the bucket's signature (test name, error message, operator, platform, payload tag) — known payload-wide events, infra outages, and in-flight fixes are usually discussed there before triages/bugs exist, and a thread often names the owning team or an existing OCPBUGS ticket. If the agent has no Slack access (no Slack tooling/credentials), omit this step entirely — do not block or ask for access.

Mechanism self-review (mandatory before finalizing any bucket). Every rule above is an instance of one behavior: challenge your own conclusion before filing it. After drafting a bucket's root cause, run this adversarial pass on the mechanism claim itself:

  1. Enumerate the load-bearing assumptions in the claim — every "because", "not yet", "race", "missing", "computes wrong", "X-only". For each, cite a specific, checkable evidence locator: an artifact path plus the quoted line, a source-code location, a job-run ID, a structured API/JSON field, a condition's LastTransitionTime, or the query that returned the result. A line number is one acceptable form, not a requirement — much of the evidence here (run metadata, API fields, object IDs, timestamps) has no stable line to cite, and inventing one is worse than quoting the field. What matters is that the next reader can go to the same place and see the same thing. An assumption with no such locator is downgraded in the filed text to an explicit unverified hypothesis ("unverified: possibly …") — never stated as fact.
  2. Check the theory's predictions against everything observed, not just the failing line: does it predict the observed frequency (deterministic theory vs. n=1), the measured window duration, the recovery asymmetry (who healed vs. who stayed broken), and the variant/platform spread? Any mismatch means the theory is wrong or incomplete — resolve it before filing.
  3. Argue the strongest alternative. Spend one honest paragraph on the best competing mechanism (warmup instead of ordering, upstream producer instead of the error-emitter, boundary condition instead of corruption, unrecovered state instead of the transient trigger) and state the artifact evidence that discriminates between them. If nothing discriminates, the confidence is not HIGH.

A bucket whose mechanism claim fails any of these checks is not finalizable — keep digging or downgrade honestly.

After deep-dive, finalize buckets. Depth is mandatory, not optional: no bucket may be finalized at LOW or MEDIUM confidence, and the duty run must not end with open "action items" like "needs artifact deep-dive" or "spot-check installer logs first". When one regression's failed runs split into multiple distinct failure signatures, every signature must be root-caused independently — each may get its own triage record (a regression can carry several), and no run may be left labeled "ambiguous"/"unclear" in a bucket claimed at HIGH confidence: an unexplained run either gets dug into until it joins a signature, or the bucket's confidence is honestly downgraded and the digging continues. Never bundle an unexplained sub-pattern into another pattern's triage "pragmatically" or "since the regression is open anyway" — triage covers only the runs whose root cause it actually explains; attaching unexplained runs to it hides them from the next duty shift and mis-scopes the bug. If a sub-pattern remains unexplained, the whole bucket is not finalizable — keep digging. If confidence is not HIGH after the steps above, keep digging until it is — escalate through the evidence ladder yourself: raw failure outputs → job-run summaries → GCS artifacts (prow-job-artifact-search, install/test-failure analysis skills) → audit logs (grep for the failing object/namespace to identify the creating user and userAgent — this reliably resolves "who created this pod/namespace" questions) → junit timing correlation (what else ran in the same window) → suspect-PR vetting. Triage taking longer is acceptable; leaving an unexplained bucket is not. The only permitted low-confidence outcome is when the evidence is genuinely exhausted (artifacts expired, logs missing), and then the report must say exactly what was checked and what was missing.

Additional finalization rules (each has caused a wrong disposition in a real duty run):

  • "Resolved"/"stopped"/"no recurrence" claims require the full run list, not a sample. Before classifying a signature as resolved or transient, enumerate all of the regression's job_runs by date and confirm the newest runs' signature. A signature absent from a 3–4 run sample of a 20-run regression is not evidence it stopped. And when a sub-test starts "passing" recently, verify recent runs actually reach that stage: a bootstrap failure cannot "self-resolve" while newer runs of the same job die earlier at infrastructure provisioning — the earlier failure masks the later stage, it does not fix it.
  • "Leave untriaged" is not a permitted disposition for a bucket with an identified root cause and owner. If the deep-dive named the mechanism and the responsible component/suite, the bucket gets a triage (to an existing or new issue; under --existing-only, a new-issue bucket instead gets a "Proposed new bugs (not filed)" entry) — "collateral of noisy runs" is only a valid leftover justification when the failure has no independent mechanism (pure co-occurrence). A failure that is deterministically produced by another test's behavior has an independent mechanism and must be triaged as test.
  • Extend, don't duplicate. When an existing triage record already covers the bucket's bug, extend that triage with the new regression IDs (--triage-id); do not create a second triage record pointing at the same JIRA.
  • Subagent outputs must be verified, not trusted. If bucket deep-dives are delegated (subagents, parallel tasks), the orchestrator must check each returned bucket against this section's requirements before accepting it — in particular that the mandatory quotes are present (recovered condition messages for vague wrappers, mutator identification for rollout flaps, transition counts for stability failures, *_previous.log reads for crash-looping containers). A missing mandatory quote means the sub-analysis is incomplete and must be redone, regardless of how confident its prose sounds. Batch size is never a reason to skip mandatory steps.
  • Never end your turn to "wait" for dispatched subagents — the session ends the moment you stop. There is no background execution across turns: an assistant message that ends with a status narration ("waiting for X analysis to complete") and no tool call terminates the run, and in CI the harness will tear the process down at that point. Collect every subagent's result within the turn that needs it, and treat the duty report as a hard checkpoint: if the run were killed right after your current message, the report file must already exist on disk — write intermediate versions early and update them, rather than deferring all writing to a final step that may never come. (Case 5 in case-notes.md: $12 of analysis, no report.)

Each bucket must have:

  • Member regression IDs (re-check the untriaged list — new siblings may have opened during analysis)
  • Root cause summary (one paragraph) and failure classification (permafail / flaky / resolved / recent)
  • Owning component (may differ from the Sippy component — state both)
  • Triage type: product / test / ci-infra / product-infra
  • Disposition: existing triage to extend / existing JIRA to create a triage for / new JIRA needed / no action (resolved or pure infra noise — say so explicitly and leave untriaged only with justification)

Phase 4: Search for existing bugs, then triage each bucket

For each bucket, before filing anything new:

  1. Check bugs linked to the bucket's fired labels, then triaged_matches from fetch-related-triages (confidence ≥5 with an open JIRA is the default target) — but never act on a match, even conf=10, without the Phase 3 per-bucket CI-sample verification confirming the root cause actually matches.

  2. Check open_bugs from the test report.

  3. Search Jira for the root-cause signature (error message, operator name, component-regression label) in OCPBUGS against the owning component — the right bug may exist under Monitoring/etcd/MCO even though the regression sits under Installer.

  4. Component-scoped JIRA listing (mandatory — keyword search is not enough). Owning teams describe defects in developer vocabulary that shares no keywords with the CI-side symptom. After determining the owning component, list its recent bugs regardless of keywords and read the summaries:

    project = OCPBUGS AND component = "<owning component>" AND created >= -21d ORDER BY created DESC
    

    A bug whose creation date falls inside the bucket's failure window, on the owning component, is a duplicate candidate even with zero keyword overlap — open it and compare mechanisms.

  5. Owning-repo merge-history check (mandatory when onset or cessation is dated). Query the owning repo for PRs merged around the bucket's onset and cessation dates:

    gh pr list --repo <org>/<repo> --state merged --search "merged:<window>" --json number,title,mergedAt
    

    A merge at the cessation boundary is likely the fix — its OCPBUGS-* title prefix names the existing bug: triage to that bug instead of filing a new one. A merge at the onset boundary is a suspect trigger. The phrase "resolved by unidentified payload change" is banned from reports and bugs unless this check was run and came back empty. For currently-live breakage, also scan the owning repo's newest merges for revert PRs: a fresh Revert "..." title citing a TRT/OCPBUGS key hands you the trigger PR, the tracking ticket, and the expected recovery time in one query.

Then act (this is where --auto-triage applies; without it, confirm each bucket with the user):

  • Extend existing triage: triage-regression skill with --triage-id (additive merge is automatic; pass only the new IDs).
  • New triage to existing bug: triage-regression skill with --url, --type, and a one-sentence --description (<120 chars).
  • New bug (under --existing-only: do not file or triage; add a Phase 5 proposal instead): file with /jira:create bug (the create skill from the jira plugin) against the owning component, label component-regression, description per the bug-filing template in /ci:analyze-regression ("Prepare Bug Filing Recommendations" section: full test names in {code} blocks, test IDs, regression IDs, variants, error signature, Sippy test-details UI links for every member regression, suspect PRs). Every JIRA issue or comment created by this workflow must end with an AI-attribution footer as a separate, visually marked block — not a sentence buried in the text: place it after a divider, as its own paragraph or note panel, e.g. a rule followed by a panel (type note) in ADF containing "AI-generated content: This bug was filed by AI as part of Component Readiness triage duty. Please verify before acting on it." Release blocker is conditional on the triage type and impact, not automatic: mark the bug a release blocker (set-release-blocker skill) only for product bugs whose failures block or materially degrade blocking/informing payload jobs; test bugs (races, invariant-scan interference) and ci-infra issues (cloud capacity, registry outages) are not release blockers — state the blocker decision and its one-line justification in the report. Then create the triage record.
  • Suggested fixes carry a higher evidentiary bar than attribution — and triage duty never opens fix PRs. A "suggested fix" in a filed bug is optional; when included it requires (a) the mechanism timeline-proven per Phase 3 (timestamps, window duration, alternatives refuted), and (b) a target that matches what the measured window actually proves. A seconds-long trigger window does not by itself justify changing steady-state behaviour for everyone — but it very often does justify a permanent resilience change (retry, re-discovery, gating, idempotent recovery) in whatever failed to survive that window. Aim the fix at the thing that stayed broken, not at the thing that blinked. Never propose weakening a safety mechanism (failurePolicy, validation, admission gating, health checks) as the primary fix; if failing open seems attractive, flag it as a question for the owning team instead. If the analysis surfaces a secondary resilience defect (e.g., a controller that cannot recover from one failed patch), do not demote it to a "defense-in-depth" footnote — re-run the differential-recovery check: it is often the actual primary bug. Opening code-change PRs is out of scope for this workflow entirely.
  • Always finish a triage by running the add-jira-triage-link skill to put the triage URL into the JIRA description. The skill appends to the description — if the description ends with the AI-attribution footer, the appended link would land after it. After running the skill, verify the footer is still the final block; if not, move the footer back to the end of the description (or insert the triage link before the footer in the same update).
  • Comments on existing bugs for added regressions must use only jira-comment.md, never freeform: at most one comment per bug per run, with a linked triage, linked test-details UI page for each regression, and the AI-attribution footer last.
  • If a label that fired on the bucket's runs does not list the triaged bug, propose manage-labels update --bugs <existing>,<new> (replaces the list; needs user confirmation, not covered by --auto-triage).
  • While triaging each bucket, note whether its signature is symptom-worthy (crisp grep-able line in a durable artifact) — the Phase 5 report must carry a symptom proposal for every bucket where it is (see "Label the bucket's signature in Sippy").

With --auto-triage, only act autonomously when confidence is high: consistent error signature across the bucket, and either a confidence ≥5 triaged match or an unambiguous existing open bug. Buckets requiring a new bug, or with mixed signals, are always presented for confirmation.

Existing-only mode (--existing-only)

This mode records matches to existing bugs but never files new JIRA issues. Analysis is unchanged. --policy post-ga implies --existing-only. These rules apply to every write in any phase, including the Phase 1 stale-triage sweep:

  • Allowed writes: only for a verified match to an existing issue:

    • extend a triage, or create one for the existing issue;
    • add-jira-triage-link;
    • comment on that issue, using the jira-comment.md template.

    Everything else, including creating, reopening, or transitioning issues, goes in the report as a recommendation.

  • Otherwise, propose. New-bug buckets, failed fixes, and uncertain matches get a draft under "Proposed new bugs (not filed)" in the Phase 5 report. Their regressions stay untriaged. Never lower the match bar to avoid a proposal: a wrong triage hides the real defect.

  • Subagents do analysis only; the orchestrator does every write.

Phase 5: Duty report

Present a final report:

  1. Inventory: N untriaged regressions found → M buckets.
  2. Per bucket: member regression IDs, root cause, owning component (vs. Sippy component), classification, evidence highlights (error signature, stage, representative run links), action taken (triage ID + JIRA link) or recommendation awaiting confirmation.
  3. Leftovers: regressions deliberately left untriaged (resolved / one-off flake / inconclusive) with justification and what evidence would change the call.
  4. Proposed Sippy symptoms (mandatory section, even if empty): for every bucket whose root cause has a crisp, grep-able single-line signature in a durable artifact, include a concrete symptom proposal per the "Label the bucket's signature in Sippy" section below — label name, bugs, matcher, file pattern, validation pair, retro-apply run set. Proposing costs nothing (creation still requires explicit user confirmation); a duty shift that root-caused a bucket and did not propose a symptom for an obviously grep-able signature has left cheap future-triage value on the table. If no bucket qualifies, say so and why (e.g., signature only visible via timing correlation, artifact expires, no unique string).
  5. Cross-cutting observations: payload-wide events, infra instability windows, techpreview-only patterns — useful context for the next duty shift.
  6. Session usage (when available): if the run is orchestrated by a harness that captures usage telemetry (e.g. the CI job appends a "Session usage" section with model, turns, token counts, and cost after the session ends), do not fabricate these numbers yourself — the model cannot observe its own final token totals mid-session. In interactive runs simply omit the section.
  7. Proposed new bugs (not filed) (--existing-only only, mandatory even if empty): one draft per proposal, following the Phase 4 new-bug template, with member regression IDs and the release-blocker recommendation. A human must be able to file it as written.

Label the bucket's signature in Sippy (propose always, create on confirmation)

When a bucket has a crisp, grep-able artifact signature (an error string or log line unique to the root cause), define a Sippy symptom so future runs are labeled automatically and the signature can be applied retroactively. Symptoms have repeatedly paid for themselves in real duty runs — a validated matcher has adjudicated a disputed root-cause attribution and machine-verified a 19-vs-1 bucket split (Case 6 in case-notes.md). Good candidates: recurring infra events that spawn regressions across many job families, single-occurrence product signatures worth a tripwire (n=1 bugs), and any signature that a "known recurring family" keeps regenerating. These are writes: always present the proposed label/symptom definitions and the target run list to the user and get explicit confirmation first — --auto-triage does not authorize them.

  1. Check for existing coverage before creating anything. List the current catalog — this read is public and needs no token (GET https://sippy.dptools.openshift.org/api/jobs/labels, GET .../api/jobs/symptoms, or the list-symptoms skill) — and search it for the bucket's signature: the label/symptom may already exist from the shift that first triaged the bug (Case 6: a proposed console-crash symptom fully duplicated an existing one; the duplicate-key 400 from the label POST was what revealed it). Also search by the bug key (list-symptoms --labels --search OCPBUGS-NNN); a label linked to the bug counts as coverage only if its matcher targets this bucket's signature. A dry-run reevaluate of one bucket run is an equivalent cheap probe when a DPCR token is available: if an existing symptom targeting this signature fires on the run, the bucket is already covered and nothing should be created.

  2. Create a label and symptom via the authenticated API (DPCR Bearer token, same as triage writes): POST https://sippy-auth.dptools.openshift.org/api/jobs/labels ({"label_title": ..., "explanation": ..., "bugs": ["<JIRA key>"]} — the title becomes the ID), then POST .../api/jobs/symptoms ({"summary": "<descriptive name>", "matcher_type": "string|regex|file", "file_pattern": "<glob under the job-run GCS root>", "match_string": ..., "label_ids": [...]}). The symptom ID is derived from the summary; gzipped artifacts are decompressed transparently. Keep Jira keys in the label's bugs array rather than embedding them in label explanations or symptom summaries. To change a symptom, use its full-replacement PUT endpoint.

  3. Beware benign-transient false positives — presence is not pathology. Before finalizing a matcher, grep the target file in a healthy run: many error lines appear benignly during normal startup — sometimes once, sometimes in bulk (Case 6: a proposed match string appeared 120× in a healthy install's etcd-operator log as a normal warm-up transient; the failed run differed only in that the line persisted to log-end). String/regex matchers cannot count occurrences or check persistence, so if the line can appear benignly at all, re-anchor to an artifact that only exists — or only contains the string — in the failure mode: a container's *_previous.log (only present after a crash/restart) instead of its current log, an events.json string that healthy runs never emit, or best of all the gather-time end state in clusteroperators.json (e.g., StaticPodsAvailable: 0 nodes are active appears there only when etcd truly never deployed — zero hits in healthy runs by construction, because gather runs after the failure window).

  4. Validate with a dry run against one known-positive and one known-negative run — the best negative is a near miss: a failed run of the same job family with a different root cause (it catches over-broad matchers that a green run would not). POST .../api/jobs/runs/reevaluate with {"prow_job_build_ids": [...], "dry_run": true}. Proceed only if both outcomes are correct; otherwise refine the matcher and re-validate. Note: reevaluation runs all symptoms, so unrelated pre-existing labels may legitimately appear on your negative — check only that your symptom's matching is correct.

  5. Then apply with "dry_run": false to the bucket's prowjob_run_ids. The API caps requests at 50 IDs, but reevaluation scans GCS artifacts server-side and large batches time out at the gateway (504) — use batches of 5–10 with a retry, check each result's status, and stop and report on the first failed batch or any non-success result rather than continuing.

  6. Match the underlying error, not a transient side effect: a matcher keyed on a crash/panic goes silently false-negative the moment a partial fix removes the crash while the defect persists. If the primary evidence lives in pod logs that sometimes fail to gather (dying clusters), add a fallback symptom against an artifact that survives (e.g., pods.json state/reason strings) mapped to the same label. When the same signature can surface under different step names (e.g., devscripts-driven vs plain IPI installs), create one symptom per file pattern, all mapped to the same label.

Closed-set audit mode (--audit-closed) — justify why every closed regression closed

bulk-triage-regressions <view> --components ... --audit-closed

An explicitly-requested, read-only mode that answers a different question than triage duty: not "who owns this failure?" but "why did this regression close, and can we prove it?" Every closed untriaged regression must end the audit in exactly one of these justification classes:

ClassMeaningProof required
fixedA fix merged; closure follows itFix PR/bug with merge/resolution date at or before the close boundary
event-endedAn infra/payload event endedEvent window bracketing the failures (bad payload, repo outage, cloud capacity, credentials)
intermittent-cause-openPass rate recovered but the defect is still openSignature matched to an open bug; state explicitly that closure ≠ resolution
collateralClosed with the window of a tracked sibling eventSignature or run overlap with the tracked event
evidence-expiredRun history and artifacts are goneList exactly what was checked (regression details, GCS) and the statistical argument (age, sibling consistency) — flagged, never silently absorbed

No regression may be left as "unknown". evidence-expired is the only permitted terminal state without a mechanism, and it must be flagged in its own report section.

Scale technique (a closed set is 5–30x a duty batch — per-regression deep-dives do not scale)

  1. Inventory and cluster first: fetch details for all members (parallel, ~8 workers is safe for the details endpoint), then cluster by (test-family, platform, featureset, close-window). Expect heavy super-clusters — always look for same-day open/close waves across platforms before analyzing anything individually (Case 7 in case-notes.md: one bad nightly payload explained 21% of an entire closed set).
  2. Bypass the Sippy runs API for bulk signature extraction — it rate-limits (HTTP 429) far below audit volume, and backoff does not help at this scale. Go to GCS directly (unthrottled, parallel-safe):
    • Install-family tests: classify from the installer log (artifacts/*/ipi-install-install*/build-log.txt, devscripts equivalent on metal) with an ordered signature catalog (most-specific first). Seed the catalog from the view's known events and extend it with whatever the duty history has verified (ign-push 403s, cipherSuites/rev-0, toolchain download, quay pulls, registry.ci 5xx, quota, capacity, provisioning, per-operator stabilization). Guard against substring traps (Case 7: lease matching inside release mis-binned 31 regressions).
    • Everything else: parse the junit XML under the e2e step (artifacts/*/<e2e-step>/artifacts/junit/junit_e2e*.xml) with a real XML parser (regex over XML mis-matched ~95% of testcases in practice) and take the <failure> of the exact testcase.
    • Sample the 2 newest failed runs per regression; extend only where signatures disagree.
  3. Match against the full triage catalog: fetch all existing triage records once (/api/component_readiness/triages) and index their regressions by test name — a closed untriaged record is very often the untriaged sibling of a triaged one, which supplies the bug and the closure mechanism for free.
  4. Date-anchor every closure claim: fixed requires the fix date ≤ close boundary (JIRA resolutiondate, gh pr list --search "merged:<window>" on the owning repo); event-ended requires the last failing run to fall inside the event window. The Phase 4 component-scoped JIRA listing and merge-history checks apply here unchanged.
  5. Checkpoint long-running collection to disk and run it detached (nohup + periodic JSON dumps): bulk GCS extraction takes tens of minutes, harness timeouts will kill foreground loops, and two concurrent writers on one results file destroy each other — one writer, atomic-ish checkpoints, verify counts after every stage.

Audit report structure

  1. Executive summary: category table (name, member count, one-line verdict).
  2. Method: exactly which artifact paths / APIs were used and why.
  3. Per-category narrative: mechanism, proof, closure explanation, and for intermittent-cause-open the explicit warning that siblings will reopen.
  4. Per-regression appendix — one row per member, no exceptions (ID, test, platform/featureset, opened→closed, category, evidence sample).
  5. Honest-limitations section listing every evidence-expired member.

Audit-specific pitfalls

  • Sippy retention creates a hard evidence horizon: regressions from the view's first tracking week may have no run history at all, and GCS artifacts expire (~3 months). Do not let the horizon silently shrink the audit — count and flag such members.
  • The view-baseline start date masquerades as a mass event: many unrelated regressions "open" on the view's first tracking day. Check whether a suspicious same-day open wave is simply the earliest date in the dataset before hunting for a common cause.
  • "Closed" is not a verdict: a closed regression whose signature maps to a still-open bug (intermittent-cause-open) is a prediction of future regressions — surface these in the summary so the next duty shift expects them.

Pitfalls (learned from real duty runs)

Principles distilled from real mis-dispositions; full narratives in case-notes.md.


View the full bulk-triage-regressions on GitHub

Detect Permafail

When to Use This Skill

Use this skill when you have 2-10 consecutive failures of the same job and need to determine if the failures represent a systematic/permanent failure (permafail) versus a flaky failure. This is critical for CI/CD pipeline analysis to distinguish between:

  • Permafail: A systematic failure affecting the same test(s) or infrastructure issue, detected by analyzing comparable runs (same failure type):
    • 2-3 comparable runs: All must have the same failure (100% match)
    • 4-5 comparable runs: At least 4 must have the same failure (80% match)
    • 6-10 comparable runs: At least ceil(count × 0.7) must have the same failure (70% match)
  • Flaky: Non-deterministic failures with varying root causes, or failures that don't meet the permafail thresholds

Prerequisites

  • Access to OpenShift CI Prow job artifacts via gcsweb URLs
  • Access to the Bash tool for running plugins/ci/scripts/classify-job-failures.py
  • Python with the requests package available for artifact fetching
  • Knowledge of Prow artifact structure (from fetch-prowjob-json and prow-job-artifact-search skills)
  • 2-10 URLs pointing to consecutive job failures (from Prow/OpenShift CI, ordered newest to oldest)
  • Job name context to verify consistency across all failures
  • PR information to provide context for analysis

Implementation Steps

Step 1: Validate Inputs

Standard Mode (URL-based):

Verify that all required inputs are present with expected types and constraints:

  • failure_urls: Array of 2-10 strings matching Prow job URL pattern https://prow.ci.openshift.org/view/gs/<bucket>/<path>/<job-name>/<build-id> where path may be logs/, pr-logs/pull/, or other GCS paths (must be consecutive runs, ordered newest to oldest)
  • job_name: Non-empty string identifier of the job being analyzed
  • pr_info: Object containing PR number (integer) and repository context (string)
  • Each URL must match the Prow job URL pattern above

Reject requests if:

  • URLs count is less than 2 or more than 10
  • Job names don't match across all URLs (validate via prowjob.json metadata, not path position)
  • PR context is missing

Notes:

  • URL ordering (newest first) is assumed but not validated - the frontend provides them in this order
  • Job name validation should check prowjob.json metadata or verify the name appears in the extracted GCS path, not assume a fixed path position (presubmit URLs include org/repo/pr segments)

Offline/Eval Mode (Pre-normalized Signatures):

For testing or offline analysis, the skill can accept pre-normalized failure signatures directly instead of fetching from URLs. This mode skips Steps 2-4 (artifact fetching and classification) and proceeds directly to Step 5 (threshold analysis).

Input format:

  • signatures: Array of 2-10 pre-normalized signature objects (see Step 4 output format)
  • Each signature must be either test_failure format {type: "test_failure", url: "...", tests: [...], test_count: N} or infra_failure format {type: "infra_failure", url: "...", error: "...", error_hash: "..."}

When signatures are provided, skip to Step 5 and apply threshold logic directly. Use this mode only for evals or when signatures have been extracted by another tool.

Step 2: Classify Job Failures with Script

For standard URL-based analysis, call the deterministic classifier script to fetch artifacts and produce normalized signatures:

"$SKILL_DIR/../../scripts/classify-job-failures.py" --json-input '{
  "failure_urls": ["https://prow.ci.openshift.org/view/gs/..."],
  "job_name": "pull-ci-openshift-origin-master-e2e-aws",
  "pr_info": {"pr_number": 12345, "repository": "openshift/origin"}
}'

The script performs the artifact-based work that does not require AI judgment:

  1. Parses the full GCS path from each Prow URL.
  2. Fetches prowjob.json from gcsweb.
  3. Browses the artifacts directory.
  4. Classifies each run as test_failure or infra_failure based on artifacts, not error text alone.
  5. Extracts failing test names from junit/build logs or normalized infrastructure errors.
  6. Returns the Step 4 normalized signature array.

If the script exits non-zero, report its JSON error and do not continue to threshold analysis unless at least two valid signatures are available from another trusted source.

Step 3: Artifact Classification Rules

The classifier follows these rules:

  • TEST_FAILURE when artifacts contain openshift-e2e-test/, e2e-*, junit/, junit XML files, openshift-tests-*, or monitor-test-*.
  • INFRA_FAILURE only when artifacts show setup/build/gather output without test artifacts, or artifacts are empty/unavailable after a valid job fetch.
  • MonitorTest failures count as test failures because MonitorTests run during e2e execution.
  • Classification must be artifact-based. Do not classify from Prow descriptions or error messages alone.

Step 4: Use Normalized Failure Signatures

Use the script output directly as the normalized signature array.

For test failures:

{
  "type": "test_failure",
  "url": "job_url",
  "tests": ["failing_test_name1", "failing_test_name2"],
  "test_count": 2
}

For infrastructure failures:

{
  "type": "infra_failure",
  "url": "job_url",
  "error": "normalized_error_message",
  "error_hash": "md5_of_error_message"
}

Step 5: Compare Signatures for Permafail Pattern

Apply permafail detection logic using failure-type-based thresholds - the denominator is the count of comparable runs (same failure type), not the matching group size:

Detection Thresholds (based on comparable run count):

  • 2-3 comparable runs: All must match (100% match required)
  • 4-5 comparable runs: At least 4 must match (80% match required)
  • 6-10 comparable runs: At least ceil(count × 0.7) must match (70% threshold)

For Test Failures:

  1. Count total test_failure signatures (this is the denominator)
  2. Extract all unique test names from test_failure signatures
  3. For each unique test name, count how many test_failure signatures contain it (this is the numerator)
  4. Apply threshold based on the denominator (total test_failure count):
    • If denominator is 2-3: numerator must equal denominator (100%)
    • If denominator is 4-5: numerator must be ≥4 (80%)
    • If denominator is 6-10: numerator must be ≥ceil(denominator × 0.7) (70%)
  5. If ANY test name meets its threshold: PERMAFAIL = TRUE
    • Report which test(s) met the threshold and their occurrence count
  6. If no test meets the threshold: PERMAFAIL = FALSE

Example: 8 test_failure signatures, "TestNetworkPolicy" appears in 6 → 6/8 = 75% → needs ceil(8×0.7)=6 → 6≥6 ✓ PERMAFAIL

For Infrastructure Failures:

  1. Count total infra_failure signatures (this is the denominator)
  2. Extract error messages and group similar errors (exact match or >70% character similarity using normalized Levenshtein: similarity = 1 - (levenshtein_distance(a, b) / max(len(a), len(b))). Treat two normalized errors as similar when similarity > 0.70. If both strings are empty, do not group them as similar; require a non-empty normalized error.)
  3. For each error group, count how many infra_failure signatures contain it (this is the numerator)
  4. Apply threshold based on the denominator (total infra_failure count):
    • If denominator is 2-3: numerator must equal denominator (100%)
    • If denominator is 4-5: numerator must be ≥4 (80%)
    • If denominator is 6-10: numerator must be ≥ceil(denominator × 0.7) (70%)
  5. If ANY error group meets its threshold: PERMAFAIL = TRUE
    • Report the error message and occurrence count
  6. If no error group meets the threshold: PERMAFAIL = FALSE

Example: 5 infra_failure signatures - 3 have "operator X timeout", 2 have random errors → 3/5 = 60% → needs 4/5 (80%) → 3<4 → NOT PERMAFAIL

For Mixed Failure Types:

  1. Separate signatures by type (test_failure vs infra_failure)
  2. For each type, count total signatures (the denominator for that type)
  3. Apply threshold logic to each type independently:
    • Test failures: If ≥2 test_failure signatures exist, check if any test appears frequently enough to meet the threshold based on test_failure count
    • Infra failures: If ≥2 infra_failure signatures exist, check if any error appears frequently enough to meet the threshold based on infra_failure count
  4. Minimum runs requirement: Need at least 2 runs of the same type to establish a permafail pattern. A single failure of either type is insufficient.
  5. If either type meets the threshold AND has ≥2 runs: PERMAFAIL = TRUE
    • Report the dominant pattern (the one that triggered permafail)
    • Explain which runs contributed and which were ignored (e.g., "All 4 runs that reached e2e tests failed on TestNetworkPolicy. 3 other runs failed during cluster setup and are not relevant to this test failure pattern.")
  6. If neither type meets criteria: PERMAFAIL = FALSE

Example Scenario 1: PERMAFAIL (4 test, 3 infra):

  • 7 total runs provided
  • 3 runs: infra_failure (cluster creation failed)
  • 4 runs: test_failure (all failing on TestNetworkPolicy)

Analysis:

  • Test failures: 4 runs → use 4/5 threshold (80%) → need 4 matching → have 4/4 (100%) ✓ PERMAFAIL
  • Infra failures: 3 runs → use 3/3 threshold (100%) → different errors → NOT permafail
  • Verdict: PERMAFAIL = TRUE (test failure group met criteria)
  • Reason: "All 4 runs that reached e2e tests failed on TestNetworkPolicy (100% match). 3 additional runs failed during infrastructure setup and are not relevant to this test failure pattern."

Example Scenario 2: NOT PERMAFAIL (1 test, 6 diverse infra):

  • 7 total runs provided
  • 6 runs: infra_failure (each with different errors: cluster creation timeout, AWS quota, network issue, pod eviction, storage failure, DNS timeout)
  • 1 run: test_failure (failed on TestNetworkPolicy)

Analysis:

  • Test failures: 1 run → INSUFFICIENT (need minimum 2 runs to establish pattern)
  • Infra failures: 6 runs with all different errors → no single error appears ≥5 times → NOT permafail
  • Verdict: PERMAFAIL = FALSE
  • Reason: "Only 1 out of 7 runs reached e2e tests and failed on '[sig-arch] daemonset cni-sysctl-allowlist-ds maxUnavailable requirement'. The other 6 runs failed during infrastructure setup, each with different unrelated issues (no consistent infra pattern). A single test failure is insufficient to establish a permafail pattern - we need multiple runs with consistent failures to confirm systematic breakage."

Step 6: Generate Verdict and Return JSON

Construct the final response object with:

  • Boolean verdict: permafail (true/false)
  • Reason string explaining the determination. Always include explicit slash-format ratios for the dominant pattern and strongest non-matching pattern, such as 7/10, 4/4, 5/6, or 2/10. Do not write only 7 of 10; include 7/10.
  • Complete failure signatures array
  • Common tests array (if applicable and permafail=true)
  • Confidence score as a JSON number from 0.0 to 1.0, representing confidence in the final verdict, not the raw match ratio. Never use strings such as "high", "medium", or "0.95". For example, a clearly non-permafail 2/10 pattern can still have confidence around 0.70 because the verdict is well supported.
  • Stable failure_type value:
    • test_failure when the final verdict is driven by test-failure analysis, or only test failures are present
    • infra_failure when the final verdict is driven by infrastructure-failure analysis, or only infrastructure failures are present
    • mixed when both test and infrastructure failures are present and no single type meets the permafail threshold
  • match_ratio: Slash-format string for the dominant pattern, such as "7/10", "4/4", "5/6", or "2/10".
  • threshold_required: Integer count required for the comparable-run group to qualify as permafail.
  • matching_runs: Integer numerator for the dominant or strongest pattern.
  • comparable_runs: Integer denominator for the comparable-run group.

The final JSON must be machine-checkable. The ratio-bearing fields must be top-level fields, not nested under an analysis object. Use this exact style:

{
  "permafail": true,
  "confidence": 0.95,
  "reason": "7/10 test_failure runs failed TestNetworkPolicy, meeting the required 7/10 threshold.",
  "failure_type": "test_failure",
  "match_ratio": "7/10",
  "matching_runs": 7,
  "comparable_runs": 10,
  "threshold_required": 7,
  "signatures": []
}

For a mixed non-permafail, use failure_type: "mixed" and include the word insufficient in the reason when one failure type has fewer than 2 comparable runs.

Do not set confidence equal to the match percentage. Confidence means confidence in the verdict:

  • Clear non-permafail with complete data and a strongest pattern below threshold, such as 2/10, should use confidence: 0.70 or higher.
  • Exact threshold permafail, such as 7/10 test failures or 5/6 infra failures, should use confidence: 0.85 or higher.
  • All comparable runs matching, such as 4/4, should use confidence: 0.99.
  • Ambiguous or incomplete data should use lower confidence.

Output Format

The skill returns a JSON object with this schema:

{
  "permafail": true,
  "confidence": 0.95,
  "reason": "3/3 test_failure runs show the same failing test 'test_node_scale' - consistent permanent failure",
  "failure_type": "test_failure",
  "match_ratio": "3/3",
  "matching_runs": 3,
  "comparable_runs": 3,
  "threshold_required": 3,
  "signatures": [
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "tests": ["test_node_scale"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "tests": ["test_node_scale"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "tests": ["test_node_scale"],
      "test_count": 1
    }
  ],
  "common_tests": ["test_node_scale"]
}

For Infrastructure Failure (permafail=true)

{
  "permafail": true,
  "confidence": 0.92,
  "reason": "All 3 runs fail at cluster creation with identical error: 'Insufficient quota for machine type n1-standard-4'",
  "failure_type": "infra_failure",
  "signatures": [
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "error": "Insufficient quota for machine type n1-standard-4",
      "error_hash": "a1b2c3d4e5f6g7h8"
    },
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "error": "Insufficient quota for machine type n1-standard-4",
      "error_hash": "a1b2c3d4e5f6g7h8"
    },
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/bucket/logs/...",
      "error": "Insufficient quota for machine type n1-standard-4",
      "error_hash": "a1b2c3d4e5f6g7h8"
    }
  ]
}

For Non-Permafail (mixed or varying failures)

{
  "permafail": false,
  "confidence": 0.88,
  "reason": "Mixed failure types detected: 2/3 runs are test_failure, 1/3 is infra_failure. Inconsistent pattern indicates flaky behavior, not a systematic permafail. Test failures: 2/3 (threshold: 2/3). Infra failures: 1/3 (threshold: 1/3).",
  "failure_type": "mixed",
  "match_ratio": "2/3",
  "matching_runs": 2,
  "comparable_runs": 3,
  "threshold_required": 2,
  "signatures": [
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["test_networking"],
      "test_count": 1
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "Pod evicted due to memory pressure",
      "error_hash": "x1y2z3a4b5c6d7e8"
    },
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["test_storage", "test_deployment"],
      "test_count": 2
    }
  ]
}

Failure Signature Format

Test Failure Signature

{
  "type": "test_failure",
  "url": "string (job URL)",
  "tests": ["array", "of", "failing_test_names"],
  "test_count": "integer (length of tests array)"
}

Fields:

  • type: Always "test_failure"
  • url: The Prow job URL for this run
  • tests: Array of test names extracted from failure logs (deduplicated)
  • test_count: Count of unique failing tests

Infrastructure Failure Signature

{
  "type": "infra_failure",
  "url": "string (job URL)",
  "error": "string (normalized error message)",
  "error_hash": "string (MD5 hash of normalized error)"
}

Fields:

  • type: Always "infra_failure"
  • url: The Prow job URL for this run
  • error: Normalized error message with timestamps and build IDs removed
  • error_hash: MD5 hash for fast similarity comparison

Permafail Detection Logic

Test Failure Logic

Apply the threshold rules from Step 5 based on the number of test_failure signatures (the comparable run count):

  1. Count total test_failure signatures (this is N, the denominator)
  2. Collect all unique test names from test_failure signatures
  3. For each unique test name, count how many test_failure signatures contain it (the numerator)
  4. Check if any test meets the threshold:
    • N=2-3: Test must appear in ALL N runs (100% match required)
    • N=4-5: Test must appear in ≥4 runs (80% match required)
    • N=6-10: Test must appear in ≥ceil(N × 0.7) runs (70% match required)
  5. Permafail = TRUE if ANY test meets the threshold
  6. Set confidence based on match strength:
    • 0.99 if all test_failure runs have identical test set
    • 0.85 or higher if threshold is met exactly (e.g., 3/3, 4/5, 7/10)
    • 0.92 if threshold is exceeded (e.g., 5/5, 8/10)
    • 0.70 or higher if no test meets threshold but the non-permafail verdict is clear
  7. In the reason, include the strongest test ratio in slash form, for example 7/10 test_failure runs failed TestNetworkPolicy or 2/10 test_failure runs failed TestNetworkPolicy.
  8. Include top-level match_ratio, matching_runs, comparable_runs, and threshold_required fields for the strongest test pattern.

Infrastructure Failure Logic

Apply the threshold rules from Step 5 based on the number of infra_failure signatures (the comparable run count):

  1. Count total infra_failure signatures (this is N, the denominator)
  2. Extract error messages and group similar errors (exact hash match or >70% string similarity)
  3. For each error group, count how many infra_failure signatures contain it (the numerator)
  4. Apply threshold based on N (total infra_failure count, not the error group size):
    • N=2-3: Error must appear in ALL N runs (100% required)
    • N=4-5: Error must appear in ≥4 runs (80% required)
    • N=6-10: Error must appear in ≥ceil(N × 0.7) runs (70% required)
  5. Permafail = TRUE if ANY error group meets its threshold
    • Example: 5 total infra_failure signatures, 3 with "operator X timeout", 2 with random errors
    • Denominator N=5 → needs 4/5 (80%) → 3/5 = 60% < 80% → NOT PERMAFAIL
  6. Set confidence based on match strength:
    • 0.99 if all infra_failure runs have identical error hash
    • 0.85 or higher if the threshold is met exactly
    • 0.92 if threshold is met with >80% string similarity
    • 0.88 if threshold is met with >70% string similarity
    • 0.70 or higher if no error group meets threshold but the non-permafail verdict is clear
  7. In the reason, include the strongest infra ratio in slash form, for example 5/6 infra_failure runs share operator authentication timeout or 1/6 infra_failure runs share the strongest error.
  8. Include top-level match_ratio, matching_runs, comparable_runs, and threshold_required fields for the strongest infra pattern.

Mixed Type Logic

When both test_failure and infra_failure types are present:

  1. Analyze each group independently using their respective thresholds
  2. Test failures: Check if test_failure signatures have common failing tests
  3. Infra failures: Check if infra_failure signatures have common errors
  4. If either group meets the permafail criteria: PERMAFAIL = TRUE
    • Report the pattern that triggered permafail (tests or infra)
    • Explain the breakdown (e.g., "4 of 4 test runs failed on the same test; 3 other runs failed during setup")
    • Set failure_type to the type that triggered the permafail: test_failure or infra_failure
    • Use the triggering type's top-level ratio fields. For example, if 4/4 test failures match and infra failures are noise, set failure_type: "test_failure", match_ratio: "4/4", matching_runs: 4, comparable_runs: 4, and the test threshold.
  5. If neither group meets criteria: PERMAFAIL = FALSE
    • Reason: "No consistent pattern found in test failures or infrastructure failures"
    • If both test_failure and infra_failure signatures are present, set failure_type to mixed, not flaky
  6. In mixed cases, include slash-format ratios for the evaluated groups in the reason, such as 1/1 test_failure runs and 1/6 infra_failure runs.

Key Principle: Infrastructure failures (cluster setup, resource quota, network issues) are orthogonal to test failures. A PR can have a systematic test failure (permafail) even if some runs fail during infrastructure setup. Analyze each type separately and detect permafails in either category.

Error Handling

Scenario 1: Artifact Fetch Failure

If artifact fetching fails for a job URL:

  • Return status: "error"
  • Return error message with specific failure reason (network error, 404, timeout, etc.)
  • Continue with remaining jobs when at least 2 jobs have not yet been attempted
  • Do NOT return a permafail verdict when fewer than 2 jobs completed successfully

Response:

{
  "status": "error",
  "error": "Failed to fetch artifacts for job: 404 Not Found at gcsweb URL",
  "action": "verify_job_url_validity"
}

Scenario 2: Timeout on Job Analysis

If the classifier script times out or returns an incomplete result:

  • Report the script error.
  • Continue only if at least 2 valid signatures are available from another trusted source.
  • If only 1 or 0 signatures are available: Return error.

Response:

{
  "status": "error",
  "error": "Analysis timeout: Only 2 of 3 jobs analyzed successfully. Insufficient data for permafail determination.",
  "completed_jobs": 2,
  "action": "retry_with_single_job"
}

Scenario 3: Invalid Job URLs

If URL validation fails:

  • Return status: "error"
  • Return specific validation error message
  • Do NOT attempt analysis

Response:

{
  "status": "error",
  "error": "Invalid job URL format: 'url3' is not a valid Prow job URL",
  "invalid_url": "url3",
  "action": "provide_valid_urls"
}

Scenario 4: Job Names Don't Match

If the job_name parameter doesn't match the actual job names extracted from URLs:

  • Return status: "error"
  • Return the expected vs actual job names

Response:

{
  "status": "error",
  "error": "Job name mismatch. Expected 'pull-ci-job-xyz' but found 'pull-ci-job-abc' in run 2",
  "expected_job": "pull-ci-job-xyz",
  "actual_job": "pull-ci-job-abc",
  "action": "provide_matching_job_urls"
}

Scenario 5: Insufficient Comparable Runs (Different Types)

If analysis completes but there are insufficient comparable runs for permafail determination (e.g., 1 test_failure + 1 infra_failure):

  • Return permafail: false
  • Explain that neither failure type has ≥2 comparable runs
  • This is NOT an error - the analysis succeeded but found no same-type pattern

Response:

{
  "permafail": false,
  "confidence": 0.70,
  "reason": "Insufficient comparable runs: 1/2 test failures, 1/2 infra failures. Cannot establish a permafail pattern with only one run of each type. Need at least 2 runs of the same failure type to determine if failures are systematic. Test: 1/2 (threshold: 2/2). Infra: 1/2 (threshold: 2/2).",
  "failure_type": "mixed",
  "match_ratio": "1/2",
  "matching_runs": 1,
  "comparable_runs": 2,
  "threshold_required": 2,
  "signatures": [
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["[sig-network] test"]
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "cluster creation failed"
    }
  ]
}

Scenario 6: Failure to Extract Failure Details

If the classifier output doesn't contain expected failure information:

  • Mark this run as "incomplete"
  • Continue with other runs
  • If ≥2 runs have valid failure data, proceed with analysis
  • Otherwise, return error

Response:

{
  "status": "incomplete",
  "warning": "Run 1 analysis incomplete: could not extract failure details",
  "completed_jobs": 2,
  "incomplete_jobs": 1,
  "permafail": "unknown",
  "recommendation": "Review job logs manually or retry analysis"
}

Examples

Example 1: Permafail - Identical Failing Test

Input:

{
  "failure_urls": [
    "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234567",
    "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234568",
    "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234569"
  ],
  "job_name": "pull-ci-openshift-origin-master-e2e-aws",
  "pr_info": {
    "pr_number": 12345,
    "repository": "openshift/origin"
  }
}

Subagent analysis results for all 3 runs:

  • Run 1: Failed tests = ["[sig-api] API discovery should provide capability information"]
  • Run 2: Failed tests = ["[sig-api] API discovery should provide capability information"]
  • Run 3: Failed tests = ["[sig-api] API discovery should provide capability information"]

Output:

{
  "permafail": true,
  "confidence": 0.99,
  "reason": "3/3 consecutive runs fail with identical test: '[sig-api] API discovery should provide capability information'. This is a systematic permanent failure.",
  "failure_type": "test_failure",
  "match_ratio": "3/3",
  "matching_runs": 3,
  "comparable_runs": 3,
  "threshold_required": 3,
  "signatures": [
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234567",
      "tests": ["[sig-api] API discovery should provide capability information"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234568",
      "tests": ["[sig-api] API discovery should provide capability information"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/pull-ci-openshift-origin-master-e2e-aws/1234569",
      "tests": ["[sig-api] API discovery should provide capability information"],
      "test_count": 1
    }
  ],
  "common_tests": ["[sig-api] API discovery should provide capability information"]
}

Example 2: Permafail Despite Mixed Failure Types

Input:

{
  "failure_urls": [
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1111",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1112",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1113",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1114",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1115",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1116",
    "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1117"
  ],
  "job_name": "periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node",
  "pr_info": {
    "pr_number": 3186,
    "repository": "openshift/ovn-kubernetes"
  }
}

Subagent analysis results:

  • Run 1: Infrastructure failure = "Cluster creation timeout" (infra_failure)
  • Run 2: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
  • Run 3: Infrastructure failure = "AWS quota exceeded" (infra_failure)
  • Run 4: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
  • Run 5: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)
  • Run 6: Infrastructure failure = "Cluster creation timeout" (infra_failure)
  • Run 7: Failed tests = ["[sig-network] Networking should provide connectivity"] (test_failure)

Analysis:

  • 7 total runs: 3 infra_failures, 4 test_failures
  • Test failures: 4/4 (100%) have identical failing test
  • Infra failures: 3 runs, but different errors (not a permafail pattern in infra)
  • Verdict: PERMAFAIL = TRUE based on test failure group

Output:

{
  "permafail": true,
  "confidence": 0.99,
  "reason": "All 4 runs that reached e2e tests failed on '[sig-network] Networking should provide connectivity' (100% match). 3 additional runs failed during infrastructure setup (cluster creation, AWS quota) and are not relevant to this test failure pattern. This is a systematic test failure caused by the PR changes.",
  "failure_type": "test_failure",
  "signatures": [
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1111",
      "error": "Cluster creation timeout",
      "error_hash": "a1b2c3d4"
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1112",
      "tests": ["[sig-network] Networking should provide connectivity"],
      "test_count": 1
    },
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1113",
      "error": "AWS quota exceeded",
      "error_hash": "e5f6g7h8"
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1114",
      "tests": ["[sig-network] Networking should provide connectivity"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1115",
      "tests": ["[sig-network] Networking should provide connectivity"],
      "test_count": 1
    },
    {
      "type": "infra_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1116",
      "error": "Cluster creation timeout",
      "error_hash": "a1b2c3d4"
    },
    {
      "type": "test_failure",
      "url": "https://prow.ci.openshift.org/view/gs/..../logs/periodic-ci-openshift-release-main-ci-4.19-e2e-aws-upgrade-ovn-single-node/1117",
      "tests": ["[sig-network] Networking should provide connectivity"],
      "test_count": 1
    }
  ],
  "common_tests": ["[sig-network] Networking should provide connectivity"]
}

Example 3: NOT Permafail - Insufficient Matching Infra Errors

Input: 5 runs where 3 have identical operator installation failure, 2 have random infra issues

Subagent analysis results:

  • Run 1: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
  • Run 2: Infrastructure failure = "AWS quota exceeded for instance type m5.xlarge" (infra_failure)
  • Run 3: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
  • Run 4: Infrastructure failure = "operator authentication timeout waiting for operator to reach Available=True" (infra_failure)
  • Run 5: Infrastructure failure = "Pod evicted due to memory pressure on node ip-10-0-1-2" (infra_failure)

Analysis:

  • 5 total infra_failure signatures (denominator = 5)
  • "operator authentication timeout" appears in 3 runs (numerator = 3)
  • Threshold for N=5 (4-5 runs): need at least 4 matching (80% required)
  • 3/5 = 60% < 80% → NOT PERMAFAIL

Output:

{
  "permafail": false,
  "confidence": 0.70,
  "reason": "Infrastructure failures are not consistent enough. 3/5 runs failed with 'operator authentication timeout waiting for operator to reach Available=True' (60% match), but with 5 infra failures at least 4/5 must match (80% required). The presence of 2 different random errors indicates flaky infrastructure rather than systematic failure.",
  "failure_type": "infra_failure",
  "match_ratio": "3/5",
  "matching_runs": 3,
  "comparable_runs": 5,
  "threshold_required": 4,
  "signatures": [
    {
      "type": "infra_failure",
      "url": "...",
      "error": "operator authentication timeout waiting for operator to reach Available=True",
      "error_hash": "a1b2c3d4"
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "AWS quota exceeded for instance type m5.xlarge",
      "error_hash": "e5f6g7h8"
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "operator authentication timeout waiting for operator to reach Available=True",
      "error_hash": "a1b2c3d4"
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "operator authentication timeout waiting for operator to reach Available=True",
      "error_hash": "a1b2c3d4"
    },
    {
      "type": "infra_failure",
      "url": "...",
      "error": "Pod evicted due to memory pressure on node ip-10-0-1-2",
      "error_hash": "i9j0k1l2"
    }
  ]
}

Example 4: Non-Permafail - No Consistent Pattern

Input: 3 runs with different test failures

Subagent analysis results:

  • Run 1: Failed tests = ["[sig-network] networking should support networking"] (test_failure)
  • Run 2: Failed tests = ["[sig-storage] storage should support volumes"] (test_failure)
  • Run 3: Failed tests = ["[sig-api] API discovery should work"] (test_failure)

Output:

{
  "permafail": false,
  "confidence": 0.70,
  "reason": "No consistent failure pattern detected. 0/3 runs show matching tests - each of the 3 runs failed with different tests: networking, storage, API discovery. This indicates flaky/non-deterministic behavior rather than a systematic permafail.",
  "failure_type": "test_failure",
  "match_ratio": "0/3",
  "matching_runs": 0,
  "comparable_runs": 3,
  "threshold_required": 3,
  "signatures": [
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["[sig-network] networking should support networking"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["[sig-storage] storage should support volumes"],
      "test_count": 1
    },
    {
      "type": "test_failure",
      "url": "...",
      "tests": ["[sig-api] API discovery should work"],
      "test_count": 1
    }
  ]
}

Threshold Logic Validation Examples

Table-driven test cases demonstrating correct threshold behavior:

Total URLsTest FailuresInfra FailuresMatching Test CountVerdictReason
101002 same testNOT permafail2/10 = 20% < 70% threshold (need ≥7)
101007 same testPERMAFAIL7/10 = 70% ≥ 70% threshold ✓
7434 same test (in test bucket)PERMAFAIL4/4 = 100% ≥ 80% threshold (4-5 runs need ≥4) ✓
7161 test, all 6 infra diverseNOT permafailOnly 1 test_failure (need ≥2), and infra errors are all different
7161 test, 5 same infra errorPERMAFAIL5/6 = 83% ≥ ceil(6×0.7)=5 (70% threshold) → infra bucket meets threshold ✓

Technical Details

Artifact Analysis Approach

This skill analyzes Prow job artifacts directly using techniques from existing CI skills:

  1. URL parsing - Extract job name and build ID from Prow URLs
  2. Artifact fetching - Use plugins/ci/scripts/classify-job-failures.py to fetch prowjob.json and browse artifacts
  3. Classification - Use artifact-based detection (junit files, test directories) to classify failures
  4. Failure extraction - Parse junit XML or build logs to extract test names or error messages

This approach follows patterns from fetch-prowjob-json, prow-job-artifact-search, and prow-job-analyze-test-failure skills.

Script Execution Strategy

The classifier script performs deterministic artifact analysis before AI threshold reasoning:

failure_urls ━━━ classify-job-failures.py ━━━ normalized signatures ━━━ Step 5 threshold analysis

Benefits:

  • Keeps artifact fetching and parsing deterministic.
  • Avoids spending agent reasoning on mechanical classification.
  • Produces the same normalized signature schema used by offline eval cases.

Synchronization:

  • Run the script once with all failure URLs.
  • Require at least 2 valid normalized signatures to proceed.
  • Stop and report the script's JSON error if URL validation or artifact fetching fails.

Error Message Normalization

Normalize infrastructure error messages for comparison:

  1. Remove timestamps: 2025-05-12T14:32:10Z → ""
  2. Remove build IDs: build-12345-xyz → ""
  3. Remove resource names with IDs: pod-abc123xyz → "pod-*"
  4. Remove request/limit values: Numbers in memory/CPU specs → ""
  5. Keep: Error classification, core message, error type

Example normalization:

Input:  "Pod evicted at 2025-05-12T14:32:10Z (build-12345): insufficient memory (512M < 1Gi required)"
Output: "Pod evicted: insufficient memory (* < *Gi required)"

Confidence Scoring

Confidence reflects how certain the permafail verdict is:

  • 0.99: All 3 runs have identical failure signature (test names or error hashes)
  • 0.95: All 3 runs have ≥1 common failing test in test_failure
  • 0.92: All 3 runs have >80% similar error messages in infra_failure
  • 0.85: 2 of 3 runs share common failure or mixed types detected
  • 0.70: Insufficient data or ambiguous failure patterns

Use confidence to determine remediation priority:

  • Confidence ≥ 0.95: High priority permafail, block PR merge
  • Confidence 0.85-0.94: Medium priority, warn but allow manual override
  • Confidence < 0.85: Low confidence verdict, require manual review

Diagnose Job Run Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

This skill takes a Prow job run URL and explains which symptoms matched the run and what each applied label means — including the matched file and text.

When to Use This Skill

Use this skill when:

  • A CI job failed and you want to know if it is a known failure mode before debugging it from scratch
  • You are about to create a new symptom and want to check what already matched the run (see manage-symptoms)
  • You want a plain-language explanation of the labels shown on a run in the Sippy UI or Spyglass

Prerequisites

  1. Default mode (already-applied labels): only network access to public GCS (https://storage.googleapis.com) and the public Sippy API (https://sippy.dptools.openshift.org) — no authentication required.

  2. Deep mode (--deep, server-side dry-run rescan): a Bearer token from the DPCR cluster.

    • Must be logged into the DPCR cluster via oc login
    • Cluster API: https://api.cr.j7t7.p1.openshiftapps.com:6443
    • Use the oc-auth skill to obtain the token (see the token-acquisition snippet in reevaluate-job-runs/SKILL.md); prefer export SIPPY_TOKEN=... over --token — argv is visible in process listings
  3. Python 3: Python 3.6 or later, standard library only.

Implementation Steps

Step 1: Default mode — explain already-applied labels

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id>"

For each applied label the summary output shows: the label title and explanation, links to associated Jira issues, the symptom that applied it (summary, matcher type, file pattern, match string), and the actual matched file and matched text from the run's artifacts.

Step 2: Deep mode — server-side rescan

Use when the run predates the current symptom set or shows no labels. This asks Sippy to re-scan the run with dry_run: true — it writes nothing and reports what would match now:

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "<prow_url>" --deep

Note: deep mode reports label IDs only — the matched file/text detail is only available in default mode (it comes from the GCS artifacts). Deep mode submits an asynchronous batch, polls its same-origin status link through pending, processing, and running, and reports results after the batch reaches complete. A failed or cancelled batch is an error.

Step 3: Nothing matched?

If default mode finds no labels, first suggest --deep: the run may simply never have been scanned, and default mode cannot distinguish that from "scanned, nothing matched". If a deep rescan also finds nothing and the user has identified the failure cause, guide them to the manage-symptoms skill to create a new symptom (and reevaluate-job-runs to apply it retroactively to past runs).

Arguments:

  • prow_url: Prow job run URL (https://prow.ci.openshift.org/view/gs/..., positional, required)

Options:

  • --deep: Server-side dry-run rescan via the reevaluate API (requires a token)
  • --token <token>: Bearer token from the oc-auth skill (only needed with --deep; optional if the SIPPY_TOKEN environment variable is set, which is preferred — argv is visible in process listings; --token takes precedence)
  • --format json|summary: Output format (default: summary)

API Details

Default mode uses the public GCS JSON API (no auth):

  • List label artifacts: GET https://storage.googleapis.com/storage/v1/b/{bucket}/o?prefix={path}/artifacts/job_labels/
  • Download each object: GET https://storage.googleapis.com/storage/v1/b/{bucket}/o/{object}?alt=media

GCS job_labels artifact schema (verified live 2026-07): each *.json object under artifacts/job_labels/ contains one wrapped entry (a label-summary.html file is also present and skipped):

{
  "symptom_label_v1": {
    "symptom": {"id": "...", "summary": "...", "matcher_type": "string",
                "file_pattern": "...", "match_string": "...", "label_ids": ["..."]},
    "label": {"id": "...", "label_title": "...", "explanation": "...",
              "bugs": ["OCPBUGS-12345"]},
    "file_match": "artifacts/.../nodes.json",
    "text_match": "the exact text that matched"
  }
}

The script also cross-references GET /api/jobs/labels and GET /api/jobs/symptoms on the public Sippy API to enrich entries with current explanations and Jira associations (embedded copies are snapshots from labeling time).

Deep mode calls POST https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate with {"prow_job_build_ids": ["<build_id>"], "dry_run": true}. Sippy returns HTTP 202 with batch_id, requested, and links.status. The client validates that status link, polls it with authenticated GET requests, and reads labels from each matching items[].result.labels_applied. Cross-origin status URLs and HTTP redirects are rejected rather than followed with or without credentials.

Error Handling

  • Bad / non-Prow URL: Must contain /view/gs/ and end in a numeric build ID (exit 1 client-side).
  • GCS 404 or no job_labels artifacts: The run may be too old (artifacts pruned) or was never scanned — suggest --deep to rescan server-side.
  • No labels found: Not an error — the script prints guidance (try --deep first, then create a new symptom via manage-symptoms).
  • 401/403 or HTML login page in deep mode: Token missing/expired (the SSO proxy may return a login page instead of 401) — refresh via the oc-auth skill.
  • HTML gateway error page or non-JSON body in deep mode: Transient gateway error (likely 504) — retry later.
  • Unknown or malformed batch status: Rejected as an API error instead of polling indefinitely.
  • Failed or cancelled deep batch: Reported as an error (exit 1).
  • Sippy API unreachable: exit 1 with a clear message.

Exit Codes:

  • 0: Diagnosis completed (including "no labels found")
  • 1: Validation, network, or API error

See Also

  • Related Skill: list-symptoms (browse/search the symptom and label catalogs)
  • Related Skill: manage-symptoms (create a new symptom when nothing matched)
  • Related Skill: reevaluate-job-runs (apply symptoms retroactively to past runs)
  • Related Skill: oc-auth (token for deep mode)
  • Related Skill: prow-job-analyze-test-failure (deeper manual failure analysis)

Fetch JIRA Issue

This skill fetches detailed information about a JIRA issue from the Red Hat JIRA REST API. It retrieves status, assignee, priority, resolution, comments, linked PRs, and classifies the issue's progress as ACTIVE, STALLED, or NEEDS_ATTENTION.

When to Use This Skill

Use this skill when you need to:

  • Check the status and progress of a JIRA bug linked to a regression triage
  • Determine if a bug is being actively worked on or needs attention
  • Extract PR links from JIRA comments
  • Get assignee and component information for a JIRA issue
  • Analyze recent comment activity on a bug

Prerequisites

  1. JIRA credentials: Required for authentication, one of:

    • API token:
    • Service account (OAuth client credentials): set JIRA_CLIENT_ID, JIRA_CLIENT_SECRET and JIRA_CLOUD_ID. Used whenever the client ID or secret is set and neither --token nor --username is passed; there is no fallback to the API token. Requests then go to api.atlassian.com instead of redhat.atlassian.net
  2. Network Access: Must be able to reach redhat.atlassian.net

    • Check: curl -s -o /dev/null -w '%{http_code}' https://redhat.atlassian.net
  3. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Run the Python Script

# Path to the Python script
script_path="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py"

# Fetch issue data in JSON format (default)
python3 "$script_path" OCPBUGS-74401 --format json

# Or fetch as human-readable summary
python3 "$script_path" OCPBUGS-74401 --format summary

# Pass token explicitly instead of using JIRA_API_TOKEN env var
python3 "$script_path" OCPBUGS-74401 --token "your-token" --format json

Step 2: Parse the Output

The script outputs structured JSON data that can be further processed:

# Store result in variable
jira_data=$(python3 "$script_path" OCPBUGS-74401 --format json)

# Extract key fields
status=$(echo "$jira_data" | jq -r '.status')
assignee=$(echo "$jira_data" | jq -r '.assignee.display_name // "Unassigned"')
progress_level=$(echo "$jira_data" | jq -r '.progress.level')
progress_reason=$(echo "$jira_data" | jq -r '.progress.reason')

# Check progress classification
echo "Status: $status"
echo "Assignee: $assignee"
echo "Progress: $progress_level - $progress_reason"

Step 3: Use Progress Classification

The skill automatically classifies issue progress:

  • ACTIVE: Issue is being worked on (assigned with recent activity, PRs linked, or active status)
  • STALLED: Issue is assigned but has no activity in 14+ days
  • NEEDS_ATTENTION: Issue is NEW/unassigned or has no progress
  • RESOLVED: Issue is closed/verified
# Example: Check if a triaged regression's bug needs attention
progress_level=$(echo "$jira_data" | jq -r '.progress.level')

case "$progress_level" in
  "ACTIVE")
    echo "Bug is being worked on - no action needed"
    ;;
  "STALLED")
    echo "Bug may need attention - consider commenting or reassigning"
    ;;
  "NEEDS_ATTENTION")
    echo "Bug needs someone to pick it up"
    ;;
  "RESOLVED")
    echo "Bug is closed"
    ;;
esac

Error Handling

The script exits with code 1 and prints errors to stderr for these cases:

  • No credentials: neither the API token variables (or --token) nor the service-account variables are set, or the service-account set is incomplete
  • Auth failure (401): Token is invalid or expired, or the service-account credential was rejected
  • Access denied (403): Token lacks permissions for the project
  • Not found (404): Issue key doesn't exist
  • Network error: Cannot reach redhat.atlassian.net
# Example: Handle missing token gracefully
if [ -z "$JIRA_CLIENT_ID" ] && { [ -z "$JIRA_API_TOKEN" ] || [ -z "$JIRA_USERNAME" ]; }; then
  echo "Warning: JIRA credentials not set. Skipping JIRA analysis."
else
  jira_data=$(python3 "$script_path" "$jira_key" --format json 2>/dev/null)
  if [ $? -ne 0 ]; then
    echo "Warning: Failed to fetch JIRA issue. Continuing without JIRA data."
  fi
fi

Output Format

JSON Format (--format json)

{
  "key": "OCPBUGS-74401",
  "url": "https://redhat.atlassian.net/browse/OCPBUGS-74401",
  "summary": "ovn-ipsec-host creates duplicate openssl attribute",
  "status": "Modified",
  "resolution": null,
  "priority": "Critical",
  "assignee": {
    "display_name": "Jane Developer",
    "email": "jdeveloper@redhat.com"
  },
  "reporter": {
    "display_name": "John Reporter",
    "email": "jreporter@redhat.com"
  },
  "components": ["Networking / cluster-network-operator"],
  "labels": ["Regression"],
  "fix_versions": ["4.22"],
  "created": "2026-01-20T14:30:00.000+0000",
  "updated": "2026-02-10T09:15:00.000+0000",
  "comment_count": 5,
  "comments": [
    {
      "author": "Jane Developer",
      "created": "2026-02-08T10:00:00.000+0000",
      "body": "PR submitted: https://github.com/openshift/ovn-kubernetes/pull/1234"
    }
  ],
  "linked_prs": [
    "https://github.com/openshift/ovn-kubernetes/pull/1234"
  ],
  "progress": {
    "level": "ACTIVE",
    "label": "ACTIVE",
    "reason": "PR in progress (1 linked)",
    "days_since_update": 2,
    "days_since_last_comment": 4
  }
}

Summary Format (--format summary)

JIRA Issue: OCPBUGS-74401
============================================================

Summary: ovn-ipsec-host creates duplicate openssl attribute
URL: https://redhat.atlassian.net/browse/OCPBUGS-74401
Status: Modified
Priority: Critical

Assignee: Jane Developer
Reporter: John Reporter

Components: Networking / cluster-network-operator
Labels: Regression
Fix Versions: 4.22

Created: 2026-01-20
Updated: 2026-02-10

Progress: ACTIVE - PR in progress (1 linked)
  Days since update: 2
  Days since last comment: 4

Linked PRs (1):
  - https://github.com/openshift/ovn-kubernetes/pull/1234

Recent Comments (1 of 5 total):
  [2026-02-08] Jane Developer:
    PR submitted: https://github.com/openshift/ovn-kubernetes/pull/1234

Notes

  • The script uses only Python standard library modules (no pip dependencies)
  • All comments are returned in the JSON output; the summary format shows only the last 3
  • PR links are extracted from comment bodies using pattern matching for GitHub PR URLs
  • Progress classification uses configurable thresholds: 7 days for "recent", 14 days for "stale"
  • The --token flag takes precedence over the JIRA_API_TOKEN environment variable and over service-account credentials

See Also

  • Related Command: /ci:analyze-regression - Uses this skill to check JIRA progress on triaged regressions
  • Related Command: /ci:check-if-jira-regression-is-ongoing - Uses this skill for JIRA bug analysis
  • Related Skill: fetch-regression-details - Fetches regression data that may link to JIRA bugs
  • Related Skill: triage-regression - Creates triage records linking regressions to JIRA bugs
  • JIRA REST API: https://developer.atlassian.com/server/jira/platform/rest-apis/

Fetch Job Run Summary

This skill fetches a summary of a Prow job run from the Sippy API, listing all tests that failed (excluding flakes) along with their error messages.

When to Use This Skill

Use this skill when you need to:

  • List all tests that failed in a specific Prow job run
  • Understand which components are affected by failures in a job
  • Identify dominant error patterns across multiple test failures
  • Investigate mass test failure regressions by examining the underlying test failures
  • Get a quick overview of a job run's health (pass rate, infrastructure failure, etc.)

Prerequisites

  1. Python 3: Version 3.6 or later
  2. Network Access: Must be able to reach https://sippy.dptools.openshift.org

Implementation

script_path="plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py"

# Text output (AI-readable)
python3 "$script_path" <prow_job_run_id>

# JSON output (structured)
python3 "$script_path" <prow_job_run_id> --format json

Parameters

ParameterRequiredDescription
prow_job_run_idYesThe Prow job run ID (numeric, e.g., 2030845545290928128)
--formatNoOutput format: text (default) or json

Extracting a Job Run ID

The job run ID can be extracted from a Prow job URL:

https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job-name>/<job_run_id>

The last path segment is the job run ID.

API Details

Endpoint: https://sippy.dptools.openshift.org/api/job/run/summary

Query Parameter: prow_job_run_id (required)

Response Fields:

FieldTypeDescription
idintProw job run ID
namestringFull job name
releasestringOpenShift release (e.g., 4.22)
clusterstringBuild cluster (e.g., build01)
urlstringProw job URL
startTimestringISO 8601 start time
durationSecondsintJob duration in seconds
overallResultstringS (success) or F (failure)
reasonstringHuman-readable result reason
infrastructureFailureboolWhether this was an infrastructure failure
testCountintTotal number of tests run
testFailureCountintNumber of failed tests
testFailuresdictMap of test name to error message (excludes flakes)
variantslistJob variant tags

Flake Handling

The Sippy API automatically handles flake detection. When a test has both a failure and a pass in the same job run's junit results (the standard flake pattern), it is not included in testFailures. Only true failures — tests that failed without a corresponding pass — appear in the results.

Output

Text Format

The text output is structured for AI consumption with:

  1. Job Run Summary: Metadata (job name, release, duration, result, variants)
  2. Test Statistics: Total tests, failures, pass rate
  3. Failed Tests: Failure count (sorted by count descending)
  4. Dominant Error Patterns: Error messages appearing in >5% of failures, with counts
  5. Detailed Failures: Each failed test with its error message, grouped by SIG

JSON Format

Structured JSON with the same data:

{
  "success": true,
  "job_name": "periodic-ci-openshift-hypershift-...",
  "job_run_id": "2030845545290928128",
  "release": "4.22",
  "test_count": 3827,
  "failure_count": 400,
  "pass_rate": 89.5,
  "failed_tests": [
    {"test_name": "[sig-cli] example test name", "error": "error message..."}
  ],
  "dominant_error_patterns": [
    {"pattern": "stale GroupVersion discovery: ...", "count": 50, "percentage": 12.5}
  ]
}

Examples

Example 1: Investigate a Mass Test Failure

# Get the summary of a job run that triggered mass test failures
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845545290928128

Example 2: Compare Two Runs

# Fetch summaries for a failing and passing run to diff
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845545290928128 --format json > failing.json
python3 plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py 2030845542774345728 --format json > passing.json

Notes

  • The testFailureCount may be slightly higher than the number of entries in testFailures due to internal deduplication
  • Error messages are truncated to 300 characters in text output to keep output manageable
  • The dominant error pattern detection strips UUIDs and timestamps for better grouping
  • For test-level output analysis (raw test failure output for a specific test across runs), use the fetch-test-runs skill instead

See Also

  • Related Skill: fetch-test-runs - Fetch raw outputs for a specific test across multiple job runs
  • Related Skill: fetch-regression-details - Fetch Component Readiness regression details
  • Related Skill: ci:prow-job-analysis - Comprehensive analysis of test and install failures in a Prow job

Fetch New PRs in Payload

This skill fetches the list of pull requests that are new in a given OpenShift payload tag compared to the previous payload. It tries the Sippy payload diff API first, and falls back to the release controller API when Sippy has not yet ingested the payload (e.g., in-progress or very recent payloads).

When to Use This Skill

Use this skill when you need to:

  • Determine what changed between two consecutive payloads
  • Identify which PRs were included in a specific payload
  • Investigate whether a specific PR or component change landed in a payload
  • Correlate a regression or test failure with newly merged PRs
  • Build a changelog of what shipped in a payload

The payload tag can be obtained from the fetch-prowjob-json skill (release.openshift.io/tag annotation) or from release controller pages.

Prerequisites

  1. Network Access: Must be able to reach the Sippy API and/or the release controller

    • Sippy: curl -s https://sippy.dptools.openshift.org/api/health
    • Release controller: curl -s https://amd64.ocp.releases.ci.openshift.org/api/v1/releasestream/4-stable/latest
    • No authentication required for either
  2. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Run the Python Script

The skill uses a Python script to fetch and format the payload diff data:

# Locate the Python script
FETCH_NEW_PRS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py"
if [ ! -f "$FETCH_NEW_PRS" ]; then
  FETCH_NEW_PRS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py" 2>/dev/null | sort | head -1)
fi
if [ -z "$FETCH_NEW_PRS" ] || [ ! -f "$FETCH_NEW_PRS" ]; then echo "ERROR: fetch_new_prs_in_payload.py not found" >&2; exit 2; fi

# Fetch new PRs in JSON format
python3 "$FETCH_NEW_PRS" <payload_tag> --format json

# Or fetch as human-readable summary grouped by component
python3 "$FETCH_NEW_PRS" <payload_tag> --format summary

Step 2: Parse the Output

The script outputs structured JSON data that can be further processed:

# Store JSON output in a variable for processing
pr_data=$(python3 "$script_path" 4.22.0-0.ci-2026-02-06-195709 --format json)

# Extract specific fields using jq if needed
total=$(echo "$pr_data" | jq '.total_prs')
pr_urls=$(echo "$pr_data" | jq -r '.pull_requests[].url')
components=$(echo "$pr_data" | jq -r '[.pull_requests[].component] | unique[]')

# Find PRs for a specific component
echo "$pr_data" | jq '.pull_requests[] | select(.component == "hypershift")'

# Find PRs with associated bugs
echo "$pr_data" | jq '.pull_requests[] | select(.bug_url != "")'

Step 3: Use the Data

The structured data includes all PR details from the payload diff:

{
  "payload_tag": "4.22.0-0.nightly-2026-01-15-114134",
  "total_prs": 17,
  "pull_requests": [
    {
      "url": "https://github.com/openshift/assisted-service/pull/8594",
      "pull_request_id": "8594",
      "component": "agent-installer-api-server",
      "description": "Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer",
      "bug_url": "https://redhat.atlassian.net/browse/MGMT-22584"
    },
    {
      "url": "https://github.com/openshift/hypershift/pull/7470",
      "pull_request_id": "7470",
      "component": "hypershift",
      "description": "use InfraStatus.APIPort for custom DNS kubeconfig",
      "bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-72258"
    }
  ]
}

Error Handling

The Python script handles common error cases automatically:

Case 1: Payload Not Found (404)

python3 fetch_new_prs_in_payload.py 4.22.0-0.ci-9999-99-99-000000
# Error: Payload '4.22.0-0.ci-9999-99-99-000000' not found.
# Verify the payload tag exists (e.g., 4.22.0-0.ci-2026-02-06-195709).

Case 2: Network Error

python3 fetch_new_prs_in_payload.py 4.22.0-0.ci-2026-02-06-195709
# Error: Failed to connect to Sippy API: [Errno -2] Name or service not known
# Check network connectivity.

Case 3: Missing Arguments

python3 fetch_new_prs_in_payload.py
# usage: fetch_new_prs_in_payload.py [-h] [--format {json,summary}] payload_tag
# fetch_new_prs_in_payload.py: error: the following arguments are required: payload_tag

Exit Codes:

  • 0: Success
  • 1: Error (invalid input, API error, network error, etc.)

Data Sources

The script tries two APIs in order:

  1. Sippy (primary) — has longer history but only includes completed payloads
  2. Release controller (fallback) — available immediately for in-progress and recent payloads, but has shorter retention

The output format is identical regardless of which source is used. The fallback is automatic and transparent.

API Details

Sippy Endpoint

GET https://sippy.dptools.openshift.org/api/payloads/diff?toPayload={payload_tag}

Parameters

  • toPayload (required): The payload tag to diff against its predecessor (e.g., 4.22.0-0.ci-2026-02-06-195709)

Response Schema

The API returns a JSON array of PR objects:

[
  {
    "id": 0,
    "created_at": "0001-01-01T00:00:00Z",
    "updated_at": "0001-01-01T00:00:00Z",
    "deleted_at": null,
    "url": "https://github.com/openshift/hypershift/pull/7470",
    "pull_request_id": "7470",
    "name": "hypershift",
    "description": "use InfraStatus.APIPort for custom DNS kubeconfig",
    "bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-72258"
  }
]

Raw API Fields:

  • id, created_at, updated_at, deleted_at: Database metadata (not useful for consumers)
  • url: Full GitHub pull request URL
  • pull_request_id: PR number as a string
  • name: Component name(s) affected by this PR (may contain comma-separated values for multi-component PRs, e.g., "olm-catalogd, olm-operator-controller")
  • description: PR title/description
  • bug_url: Associated Jira bug URL (empty string if none)

The Python script remaps name to component in its output for clarity.

Examples

Example 1: Fetch PRs as JSON

python3 "$FETCH_NEW_PRS" 4.22.0-0.nightly-2026-01-15-114134 --format json

Expected Output:

{
  "payload_tag": "4.22.0-0.nightly-2026-01-15-114134",
  "total_prs": 17,
  "pull_requests": [
    {
      "url": "https://github.com/openshift/assisted-service/pull/8594",
      "pull_request_id": "8594",
      "component": "agent-installer-api-server",
      "description": "Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer",
      "bug_url": "https://redhat.atlassian.net/browse/MGMT-22584"
    },
    {
      "url": "https://github.com/openshift/machine-config-operator/pull/5509",
      "pull_request_id": "5509",
      "component": "machine-config-operator",
      "description": "Set `NodeDegraded` MCN condition when node state annotation is set to `Degraded`",
      "bug_url": "https://redhat.atlassian.net/browse/OCPBUGS-67229"
    }
  ]
}

Example 2: Fetch PRs as Summary

python3 "$FETCH_NEW_PRS" 4.22.0-0.nightly-2026-01-15-114134 --format summary

Expected Output:

New PRs in payload 4.22.0-0.nightly-2026-01-15-114134
============================================================
Total: 17 new pull requests

  agent-installer-api-server (1 PRs):
    - Create Enhancement Document for 3rd Party CNI / No CNI Support in Assisted Installer [https://redhat.atlassian.net/browse/MGMT-22584]
      https://github.com/openshift/assisted-service/pull/8594

  hypershift (4 PRs):
    - [kubevirt] Make L3 migration labeling conditional [https://redhat.atlassian.net/browse/OCPBUGS-66205]
      https://github.com/openshift/hypershift/pull/7308
    - feat(api): add support for graceful service account signing key rotation [https://redhat.atlassian.net/browse/CNTRLPLANE-1768]
      https://github.com/openshift/hypershift/pull/7324
    - Scaffold OpenShiftManager controller [https://redhat.atlassian.net/browse/API-1835]
      https://github.com/openshift/hypershift/pull/7445
    - use InfraStatus.APIPort for custom DNS kubeconfig [https://redhat.atlassian.net/browse/OCPBUGS-72258]
      https://github.com/openshift/hypershift/pull/7470

  machine-config-operator (2 PRs):
    - Set `NodeDegraded` MCN condition when node state annotation is set to `Degraded` [https://redhat.atlassian.net/browse/OCPBUGS-67229]
      https://github.com/openshift/machine-config-operator/pull/5509
    - Prevent unnecessary systemd unit disable [https://redhat.atlassian.net/browse/OCPBUGS-58023]
      https://github.com/openshift/machine-config-operator/pull/5527

Example 3: Use with fetch-prowjob-json Skill

Combine with the fetch-prowjob-json skill to get the payload tag from a Prow job, then find what PRs were new in that payload:

# 1. Get payload tag from prowjob.json (via fetch-prowjob-json skill)
#    payload_tag = metadata.annotations["release.openshift.io/tag"]
#    e.g., "4.22.0-0.ci-2026-02-06-195709"

# 2. Fetch new PRs in that payload
python3 "$FETCH_NEW_PRS" "$payload_tag" --format json

Notes

  • Neither API requires authentication for read-only access
  • The Python script uses only standard library modules (no external dependencies)
  • The previous payload is determined automatically; you only provide the target payload
  • Payload tags follow the format: {version}-0.{stream}-{date}-{time} (e.g., 4.22.0-0.ci-2026-02-06-195709 or 4.22.0-0.nightly-2026-01-15-114134)
  • The component field (called name in the raw API) may contain multiple comma-separated component names for PRs that affect multiple components
  • PRs without an associated bug will have an empty string for bug_url
  • An empty response array means no new PRs were found (the payload may be identical to its predecessor)

See Also

  • Related Skill: fetch-prowjob-json (provides payload tag from Prow job metadata)
  • Related Skill: fetch-regression-details (for correlating regressions with payload changes)
  • Related Command: /ci:analyze-regression (analyzes regressions that may be caused by new PRs)

Fetch Payloads

This skill fetches recent release payloads from the OpenShift release controller, showing their tag name, acceptance phase, timestamp, blocking job results, and a link to the release details page.

When to Use This Skill

Use this skill when you need to:

  • Find the latest accepted (or rejected) nightly or CI payloads for a given OCP version
  • Check the current state of release payloads for any architecture
  • See which blocking jobs failed for rejected payloads
  • Get a link to the release controller page for a specific payload

Implementation Steps

Step 1: Determine defaults

If the user did not specify an architecture or stream, default to amd64 and nightly. If no version is specified, the script automatically fetches the latest from the Sippy API.

Step 2: Fetch payloads

FETCH_PAYLOADS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-payloads/fetch_payloads.py"
if [ ! -f "$FETCH_PAYLOADS" ]; then
  FETCH_PAYLOADS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-payloads/fetch_payloads.py" 2>/dev/null | sort | head -1)
fi
if [ -z "$FETCH_PAYLOADS" ] || [ ! -f "$FETCH_PAYLOADS" ]; then echo "ERROR: fetch_payloads.py not found" >&2; exit 2; fi
python3 "$FETCH_PAYLOADS" [architecture] [version] [stream]

Examples:

# Latest amd64 nightly payloads (all defaults, last 5)
python3 "$FETCH_PAYLOADS"

# arm64 4.18 nightly
python3 "$FETCH_PAYLOADS" arm64 4.18 nightly

# Only accepted payloads
python3 "$FETCH_PAYLOADS" amd64 4.18 nightly --phase Accepted

# Show more results
python3 "$FETCH_PAYLOADS" amd64 4.18 nightly --limit 20

Step 3: Present results

The script outputs one block per payload to stdout with job details. Present to the user as-is or summarize.

Output Format

The script outputs a JSON object to stdout:

{
  "hours_since_last_accepted": 23.5,
  "last_accepted_tag": "4.22.0-0.nightly-2026-02-24-030944",
  "payloads": [ ... ]
}
  • hours_since_last_accepted: Hours since the most recent Accepted payload in the stream (from the full unfiltered history), or null if none found.
  • last_accepted_tag: Tag name of the most recent Accepted payload, or null if none found.
  • payloads: Array of payload objects, each containing tag, phase, release controller URL, and job results (blocking and async jobs with Prow URLs and retry details).

Error Handling

  1. Unknown architecture: Exits with error listing valid architectures
  2. CI stream on non-amd64: Exits with error (CI stream is amd64-only)
  3. Network error: Prints connection failure to stderr, exits 1
  4. No payloads found: Prints message to stderr, exits 1

Fetch Prow Job Runs

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

This skill finds Prow job runs (and their prow_id values) via the Sippy /api/jobs/runs endpoint. It exists because this API does not accept ad-hoc query parameters like ?job_name= or ?period= — those do not exist and will be silently ignored or rejected. Filtering is done through a JSON filter parameter with a fixed set of column fields and operators (listed below), and this skill's script builds that JSON with verified fields and operators so you never have to guess.

When to Use This Skill

Use this skill when you need to:

  • Find run IDs (prow_id) to feed into reevaluate-job-runs after creating or updating a symptom
  • Check how widespread a failure is (e.g. all failed runs of a job family in the last day)
  • List recent runs matching a platform/network/etc. variant
  • Get the Prow URL of recent runs of a given job

Prerequisites

  1. Network Access: The Sippy API must be accessible at https://sippy.dptools.openshift.org
    • No authentication required
  2. Python 3: Python 3.6 or later, standard library only

Implementation Steps

Invoke the script with flags matching the question:

script_path="plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py"

# Metal-platform runs in the last 24 hours
python3 "$script_path" --release 5.0 --variant Platform:metal --since-hours 24 --format summary

# Failed runs of jobs whose name contains e2e-metal
python3 "$script_path" --release 5.0 --job-contains e2e-metal --result F --format summary

# Pipe run IDs into symptom reevaluation
python3 "$script_path" --release 5.0 --job-contains e2e-metal --result F --since-hours 48 --ids-only \
  | xargs python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py --dry-run

Flags:

  • --release <release> (required): OpenShift release, e.g. 5.0
  • --job-contains <substr> (repeatable): job name must contain each substring
  • --variant <Key:Value> (repeatable): run must have each variant entry, e.g. Platform:metal
  • --result <code>: overall result code (S success, F failure, n infra failure — lowercase n)
  • --since-hours <n>: only runs newer than N hours ago
  • --filter-json <raw>: escape hatch — raw JSON array of extra filter items merged with the generated ones
  • --limit <n>: max rows (default 100)
  • --ids-only: print just prow_id per line (pipeline-friendly)
  • --format {json,summary}: output format (default json)

All filter items are combined with AND.

API Details

Endpoint

GET https://sippy.dptools.openshift.org/api/jobs/runs?release={release}&filter={urlencoded_json}&limit={n}&sortField=timestamp&sort=desc

No authentication required.

Filter JSON Shape

{"items": [{"columnField": "name", "operatorValue": "contains", "value": "e2e-metal"}], "linkOperator": "and"}

Verified Column Fields and Operators

Do NOT invent other query parameters or fields — only these are verified to work:

columnFieldoperatorValuevalue
namecontainsjob-name substring
variantshas entryKey:Value, e.g. Platform:metal
timestamp> / <RFC 3339 UTC timestamp, e.g. 2026-10-14T12:00:00Z
overall_resultequalsS (success), F (failure), n (infra failure — lowercase)

An unknown columnField yields HTTP 400 with a "column does not exist" message; the script surfaces that message verbatim.

Response Row Fields

{"rows": [...]}; each row has prow_id (string), job, url, variants[], overall_result, failed, succeeded, infrastructure_failure, test_failures, timestamp (epoch millis), cluster.

Error Handling

Case 1: Unknown Column (HTTP 400)

Error: HTTP 400 from Sippy API: {"code":400,"message":"could not filter or sort data: error updating query for filter ... column does not exist"}

Use only the column fields in the table above. Exits 1.

Case 2: Network Error

Error: failed to connect to Sippy API: [Errno -2] Name or service not known

Exits 1. Check connectivity to sippy.dptools.openshift.org.

Case 3: Empty Results

No matching runs prints [] (JSON), Total: 0 (summary), or nothing (--ids-only) and exits 0 — not an error.

Exit Codes:

  • 0: Success (including empty results)
  • 1: Error (HTTP error, network error, invalid --filter-json)

Examples

Example 1: Metal Runs in the Last 24 Hours

python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
  --release 5.0 --variant Platform:metal --since-hours 24 --limit 5 --format summary

Expected Output (excerpt):

n  periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-ha-cert-rotation-shutdown-5y-age-90d  2080684960133419008  1784908873000
Total: 5

Example 2: Failed Runs of a Job Family

python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
  --release 5.0 --job-contains e2e-metal --result F --format summary

Example 3: Feed Run IDs to Reevaluation

python3 plugins/ci/skills/fetch-prow-job-runs/fetch_prow_job_runs.py \
  --release 5.0 --job-contains e2e-metal --since-hours 48 --ids-only \
  | xargs python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py --dry-run

Notes

  • Results are sorted newest-first (sortField=timestamp&sort=desc)
  • timestamp filter values must be RFC 3339 UTC strings (the script computes this for --since-hours)
  • Repeatable flags AND together; there is no OR support in this script (use --filter-json if you need custom items)

See Also

  • Related Skill: reevaluate-job-runs (re-run symptom detection on the run IDs found here)
  • Related Skill: diagnose-job-run-symptoms (explain which symptoms/labels apply to a run)
  • Related Skill: list-symptoms (browse the symptom/label catalog)
  • Related Skill: fetch-regression-details (another source of job run IDs, from regressions)

Fetch ProwJob JSON

This skill fetches the prowjob.json artifact for a given Prow job and returns key fields including job metadata, spec details, status, and release annotations.

When to Use This Skill

Use this skill when you need to:

  • Get metadata about a specific Prow job run (status, timing, cluster, pod name)
  • Determine what payload a job was tested against (release.openshift.io/tag)
  • Determine the upgrade source version for upgrade jobs (release.openshift.io/from-tag)
  • Get the release images (RELEASE_IMAGE_LATEST, RELEASE_IMAGE_INITIAL) used in a job
  • Check the ci-operator target and variant for a job
  • Get the GCS bucket path for a job's artifacts

Prerequisites

  1. Network Access: Must be able to reach the GCS web proxy
    • Check: curl -s -o /dev/null -w '%{http_code}' https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/
    • No authentication required

Implementation Steps

Step 1: Parse the Prow Job URL

Extract the GCS path from the Prow job URL. Prow URLs follow this pattern:

https://prow.ci.openshift.org/view/gs/<bucket>/<path>/<job-name>/<build-id>

Extract everything after /view/ to get the GCS reference:

gs/<bucket>/<path>/<job-name>/<build-id>

Step 2: Construct the GCS Web URL

Convert the Prow URL to a gcsweb URL that serves the raw prowjob.json:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/<bucket>/<path>/<job-name>/<build-id>/prowjob.json

Example:

  • Input Prow URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672
  • GCS Web URL: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672/prowjob.json

The conversion is: replace https://prow.ci.openshift.org/view/ with https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/ and append /prowjob.json.

Step 3: Fetch the prowjob.json

Use curl or WebFetch to retrieve the JSON content from the constructed URL.

curl -s "https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results-public/logs/<job-name>/<build-id>/prowjob.json"

Step 4: Extract and Return Key Fields

From the full prowjob.json, extract and return the following fields:

Required Fields
FieldJSON PathDescription
Job Namespec.jobFull periodic/presubmit/postsubmit job name
Job Typespec.typeperiodic, presubmit, or postsubmit
Build IDstatus.build_idUnique build identifier
Statestatus.statesuccess, failure, aborted, pending, triggered
Descriptionstatus.descriptionHuman-readable status description
Start Timestatus.startTimeISO 8601 timestamp
Completion Timestatus.completionTimeISO 8601 timestamp (null if still running)
Clusterspec.clusterBuild cluster (e.g., build02, build05)
Pod Namestatus.pod_nameKubernetes pod name
Prow URLstatus.urlLink back to the Prow job view
Payload Tagmetadata.annotations["release.openshift.io/tag"]The payload version tested or upgraded to (e.g., 4.22.0-0.ci-2026-02-06-195709). This is the payload the cluster was tested against or upgraded to.
Upgrade From Tagmetadata.annotations["release.openshift.io/from-tag"]The original payload version before upgrade (e.g., 4.22.0-0.ci-2026-02-05-195709). Only present for upgrade jobs. This is the version the cluster starts at before upgrading to the payload tag.
Release Sourcemetadata.annotations["release.openshift.io/source"]Release stream source (e.g., ocp/4.22)
Architecturemetadata.annotations["release.openshift.io/architecture"]CPU architecture (e.g., amd64, arm64, multi)
Refsspec.extra_refsArray of repo references (org, repo, base_ref)
TargetFrom spec.pod_spec.containers[0].argsThe --target= value from ci-operator args
VariantFrom spec.pod_spec.containers[0].argsThe --variant= value from ci-operator args
RELEASE_IMAGE_LATESTFrom spec.pod_spec.containers[0].envThe release image the job upgrades to or installs
RELEASE_IMAGE_INITIALFrom spec.pod_spec.containers[0].envThe initial release image (for upgrade jobs)
Notes on Release Annotations
  • release.openshift.io/tag: Present on release-controller-triggered jobs. Identifies the payload being validated. For upgrade jobs, this is the version the cluster upgrades to.
  • release.openshift.io/from-tag: Only present on upgrade jobs. Identifies the version the cluster starts at before the upgrade. If absent, the job is not an upgrade job.
  • These annotations may not be present on manually triggered or presubmit jobs.
  • The RELEASE_IMAGE_LATEST and RELEASE_IMAGE_INITIAL environment variables correspond to these tags and contain the full registry pull spec.

Error Handling

Case 1: Invalid Prow URL Format

If the URL doesn't match the expected pattern https://prow.ci.openshift.org/view/gs/...:

  • Report the error to the user
  • Show the expected URL format

Case 2: prowjob.json Not Found (404)

If the GCS web proxy returns a 404:

  • The job may have been garbage collected or the URL may be incorrect
  • Suggest verifying the Prow job URL is correct and the job exists

Case 3: Network Error

If the GCS web proxy is unreachable:

  • Check connectivity to gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com
  • The proxy does not require VPN or authentication

Examples

Example 1: Fetch a Periodic Upgrade Job

Input:

https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade/2019864414127132672

Key output fields:

Job Name:             periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn-upgrade
Job Type:             periodic
Build ID:             2019864414127132672
State:                failure
Start Time:           2026-02-06T20:02:58Z
Completion Time:      2026-02-07T00:36:09Z
Cluster:              build02
Payload Tag:          4.22.0-0.ci-2026-02-06-195709
Upgrade From Tag:     4.22.0-0.ci-2026-02-05-195709
Architecture:         amd64
Target:               e2e-gcp-ovn-upgrade
Variant:              ci-4.22
RELEASE_IMAGE_LATEST: registry.ci.openshift.org/ocp/release:4.22.0-0.ci-2026-02-06-195709
RELEASE_IMAGE_INITIAL:registry.ci.openshift.org/ocp/release:4.22.0-0.ci-2026-02-05-195709

Example 2: Non-Upgrade Job (No from-tag)

For a non-upgrade job, the Upgrade From Tag field will be absent:

Job Name:             periodic-ci-openshift-release-master-ci-4.22-e2e-gcp-ovn
Job Type:             periodic
Build ID:             2019864414127132999
State:                success
Payload Tag:          4.22.0-0.ci-2026-02-06-195709
Upgrade From Tag:     (not present - not an upgrade job)
Architecture:         amd64
Target:               e2e-gcp-ovn
Variant:              ci-4.22

Notes

  • The prowjob.json is uploaded to GCS by Prow's sidecar after the job completes (or is aborted). It may not be available for jobs still in progress.
  • The managedFields section of the metadata can be omitted from output as it contains only field ownership bookkeeping.
  • The metadata.labels section is truncated for some fields (e.g., job name) due to Kubernetes label length limits. Always use metadata.annotations or spec.job for the full job name.
  • The GCS web proxy at gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com serves raw file content without authentication.
  • Duration can be calculated from completionTime - startTime.
  • The spec.decoration_config.timeout field shows the maximum allowed runtime for the job.

See Also

  • Related Skill: ci:prow-job-analysis (comprehensive Prow CI job failure analysis — test, install, disruption, and more)

Fetch Regression Details

This skill fetches detailed regression information from the Component Readiness API. It retrieves comprehensive data about a specific regression including test name, affected variants, release information, triage status, and related metadata.

When to Use This Skill

Use this skill when you need to retrieve complete details about a Component Readiness regression, such as:

  • Analyzing a specific regression from Component Readiness
  • Getting test name, release, and variant information for a regression
  • Checking triage status and existing bug assignments
  • Building regression analysis reports
  • Automating regression workflows

Prerequisites

  1. Network Access: Must be able to reach the Sippy API

    • Check: curl -s https://sippy.dptools.openshift.org/api/health
    • No authentication required for public API endpoints
  2. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Should be available on most systems
    • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Run the Python Script

The skill uses a Python script to fetch and parse regression data (including sample failed jobs):

# Path to the Python script
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"

# Fetch regression data in JSON format (includes failed jobs)
python3 "$script_path" <regression_id> --format json

# Or fetch as human-readable summary
python3 "$script_path" <regression_id> --format summary

Step 2: Parse the Output

The script outputs structured JSON data that can be further processed:

# Store JSON output in a variable for processing
regression_data=$(python3 "$script_path" 34446 --format json)

# Extract specific fields using jq if needed
test_name=$(echo "$regression_data" | jq -r '.test_name')
component=$(echo "$regression_data" | jq -r '.component')
jira_keys=$(echo "$regression_data" | jq -r '.triages[].jira_key')

# Extract failed job URLs for analysis
failed_job_urls=$(echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_url')

Step 3: Use the Data

The structured data includes all necessary regression details:

{
  "regression_id": 34446,
  "test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
  "test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
  "release": "4.22",
  "base_release": "4.21",
  "component": "Build",
  "capability": "Builds",
  "view": "4.22-main",
  "opened": "2026-01-28T08:02:45.127153Z",
  "closed": null,
  "status": "open",
  "last_failure": "2026-01-30T07:03:46Z",
  "variants": [
    "Architecture:amd64",
    "Platform:metal",
    "Network:ovn",
    "Upgrade:none"
  ],
  "max_failures": 18,
  "triages": [
    {
      "id": 344,
      "url": "https://redhat.atlassian.net/browse/OCPBUGS-74651",
      "jira_key": "OCPBUGS-74651",
      "type": "product",
      "description": "RHCOS 10 metal ipv4 job permafailing on ipv6 network access",
      "bug_id": 17817794,
      "resolved": false,
      "created_at": "2026-01-29T17:58:20.858051Z",
      "updated_at": "2026-01-30T13:28:33.429963Z"
    }
  ],
  "analysis_status": -2,
  "analysis_explanations": [
    "Test is failing consistently across multiple job runs",
    "Regression detected compared to baseline release"
  ],
  "sample_failed_jobs": {
    "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": {
      "pass_sequence": "FFFFFFFFFFFFFFFFFF",
      "label_summary": {
        "ErrImagePullQuay502BadGateway": 3
      },
      "failed_runs": [
        {
          "job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
          "job_run_id": "2017184460591599616",
          "start_time": "2026-01-30T10:33:47",
          "test_failures": 3,
          "job_labels": ["ErrImagePullQuay502BadGateway"]
        }
      ]
    },
    "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-techpreview": {
      "pass_sequence": "SSFSSSSSSSS",
      "label_summary": {},
      "failed_runs": [
        {
          "job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
          "job_run_id": "2016460830022832128",
          "start_time": "2026-01-28T10:37:27",
          "test_failures": 1,
          "job_labels": []
        }
      ]
    }
  },
  "job_runs": [
    {
      "id": 2600,
      "regression_id": 34446,
      "prowjob_run_id": "2044678206610477056",
      "prowjob_name": "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview",
      "prowjob_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2044678206610477056",
      "start_time": "2026-04-16T07:23:39Z",
      "test_failures": 3
    }
  ],
  "test_details_url": "https://sippy.dptools.openshift.org/api/component_readiness/test_details?...",
  "api_url": "https://sippy.dptools.openshift.org/api/component_readiness/regressions/34446"
}

Note:

  • analysis_status: Integer status code from the regression analysis. Negative numbers indicate problems, with lower numbers representing more severe issues.
  • analysis_explanations: List of human-readable explanations describing the status of the regression.
  • sample_failed_jobs: Dictionary keyed by job name. Each job contains:
    • pass_sequence: Chronological success/fail pattern for this specific job (newest to oldest). "S" = successful run, "F" = failing run. Example: "FFFFFSSS" shows 5 recent failures, then 3 older successes.
    • label_summary: Dictionary of {label: count} aggregated across all failed runs for this job. A label appearing in many/all failed runs is a strong signal that the labelled symptom is the root cause. Labels are human-defined: a team member wrote a regex that matches specific artifact content and named the symptom (e.g., "ErrImagePullQuay502BadGateway"). If many or all failed runs share a label, prioritize investigating that symptom.
    • failed_runs: List of failed runs for this job, sorted by start_time (newest first). Each run includes:
      • test_failures: Total number of tests failing in the entire job run (not just the regressed test). High values (>10) indicate a mass failure run where the regressed test may be collateral damage rather than the primary problem.
      • job_labels: List of symptom labels applied to this specific run. Labels are precise — they fire only when a human-defined regex matches job artifacts. An empty list means no known symptom was detected for this run.
  • label_bugs: {label: [Jira keys]} for labels seen in sample_failed_jobs, from the public label catalog. On fetch failure, label_bugs_error is set instead — treat it as unknown, not "no bugs".
  • job_runs: Complete list of all job runs where the failure was observed throughout the entire life of the regression (not just the last reporting period). Sorted by start_time (newest first). Each entry contains:
    • id: Unique ID for this job run record
    • regression_id: The regression this run belongs to
    • prowjob_run_id: The Prow job run ID
    • prowjob_name: The Prow job name
    • prowjob_url: Direct URL to the Prow job run
    • start_time: ISO 8601 timestamp when the job started
    • test_failures: Integer — total number of test failures in this job run. High values (e.g., >10) indicate a mass failure run where the regressed test may just be caught up in a larger issue rather than being the primary problem
    • This data is useful for: determining when the regression started (earliest entries), how frequently it occurs, whether failures cluster in certain jobs, identifying mass failure runs where the regression may be collateral damage, and linking related regressions that share the same job runs. Sippy uses this data automatically in the fetch_related_triages API to find regressions with shared job runs.

Error Handling

The Python script handles common error cases automatically:

Case 1: Regression Not Found (404)

python3 fetch_regression_details.py 99999999
# Error: Regression ID 99999999 not found.
# Verify the regression ID exists in Component Readiness.

Case 2: Invalid Regression ID

python3 fetch_regression_details.py abc
# Error: Regression ID must be a positive integer, got 'abc'

Case 3: Network Error

python3 fetch_regression_details.py 34446
# Error: Failed to connect to Sippy API: [Errno -2] Name or service not known
# Check network connectivity and VPN settings.

Case 4: Missing Arguments

python3 fetch_regression_details.py
# Usage: fetch_regression_details.py <regression_id> [--format json|summary]

Exit Codes:

  • 0: Success
  • 1: Error (invalid input, API error, network error, etc.)

API Response Schema

The API returns a JSON object with the following structure:

{
  "id": 34446,
  "view": "4.22-main",
  "release": "4.22",
  "base_release": "4.21",
  "component": "Build",
  "capability": "Builds",
  "test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
  "test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
  "variants": [
    "Upgrade:none",
    "Architecture:amd64",
    "Platform:metal",
    "Network:ovn"
  ],
  "opened": "2026-01-28T08:02:45.127153Z",
  "closed": {
    "Time": "0001-01-01T00:00:00Z",
    "Valid": false
  },
  "triages": [
    {
      "id": 344,
      "created_at": "2026-01-29T17:58:20.858051Z",
      "updated_at": "2026-01-30T13:28:33.429963Z",
      "url": "https://redhat.atlassian.net/browse/OCPBUGS-74651",
      "description": "Description of the triage",
      "type": "product",
      "bug_id": 17817794,
      "resolved": {
        "Time": "0001-01-01T00:00:00Z",
        "Valid": false
      }
    }
  ],
  "last_failure": {
    "Time": "2026-01-30T07:03:46Z",
    "Valid": true
  },
  "max_failures": 18,
  "job_runs": [
    {
      "id": 2600,
      "regression_id": 34446,
      "prowjob_run_id": "2044678206610477056",
      "prowjob_name": "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview",
      "prowjob_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
      "start_time": "2026-04-16T07:23:39Z",
      "test_failures": 3
    }
  ],
  "links": {
    "self": "https://sippy.dptools.openshift.org/api/component_readiness/regressions/34446",
    "test_details": "https://sippy.dptools.openshift.org/api/component_readiness/test_details?..."
  }
}

Key Fields:

  • id: Regression ID
  • test_name: Full test name with signature groups
  • release: Sample release (e.g., "4.22")
  • base_release: Baseline release for comparison (e.g., "4.21")
  • component: Component owning the test
  • capability: Higher-level capability area
  • variants: Array of variant key:value pairs (e.g., "Platform:metal", "Network:ovn")
  • opened: ISO 8601 timestamp when regression was first detected
  • closed: Object with Valid boolean and Time (valid if regression is closed)
  • triages: Array of triage objects with JIRA URL, type, and resolution status
  • job_runs: Array of all job runs where the failure was observed throughout the regression's entire life. Each entry includes prowjob_run_id, prowjob_name, prowjob_url, start_time, and test_failures (total failures in the run — high values indicate mass failure runs where the regression may be collateral damage)
  • links.test_details: Direct URL to Sippy test details page with sample failures

Examples

Example 1: Fetch Regression as JSON

# Fetch regression 34446 in JSON format (includes failed jobs)
python3 plugins/ci/skills/fetch-regression-details/fetch_regression_details.py 34446 --format json

Expected Output:

{
  "regression_id": 34446,
  "test_name": "[sig-builds][Feature:Builds] custom build with buildah...",
  "test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
  "release": "4.22",
  "base_release": "4.21",
  "component": "Build",
  "capability": "Builds",
  "status": "open",
  "variants": ["Architecture:amd64", "Platform:metal", "Network:ovn"],
  "triages": [
    {
      "jira_key": "OCPBUGS-74651",
      "type": "product",
      "resolved": false
    }
  ],
  "analysis_status": -2,
  "analysis_explanations": [
    "Test is failing consistently across multiple job runs",
    "Regression detected compared to baseline release"
  ],
  "sample_failed_jobs": {
    "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": {
      "pass_sequence": "FFFFFFFFFFFFFFFFFF",
      "failed_runs": [
        {
          "job_url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
          "job_run_id": "2017184460591599616",
          "start_time": "2026-01-30T10:33:47"
        }
      ]
    }
  }
}

Example 2: Fetch Regression as Human-Readable Summary

# Fetch regression 34446 as formatted summary
python3 plugins/ci/skills/fetch-regression-details/fetch_regression_details.py 34446 --format summary

Expected Output:

Regression #34446 Details:
============================================================

Test Name: [sig-builds][Feature:Builds] custom build with buildah...
Release: 4.22 (baseline: 4.21)
Component: Build

Status: Open (opened: 2026-01-28)
Max Failures: 19

Triages:
  - OCPBUGS-74651 (product, active): Description of the issue

Sample Failed Jobs (19 runs):
  - periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview
    Run ID: 2017184460591599616
    Started: 2026-01-30
    URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...

Example 3: Extract Failed Job URLs

# Get all failed job URLs for analysis
script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"
data=$(python3 "$script_path" 34446 --format json)

# Extract failed job URLs
echo "$data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_url'

Expected Output:

https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2017184460591599616
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview/2017131608699572224
...

Output Format

The script supports two output formats:

JSON Format (--format json)

Returns structured JSON data with all regression fields:

{
  "regression_id": 34446,
  "test_name": "...",
  "test_id": "...",
  "release": "4.22",
  "base_release": "4.21",
  "component": "Build",
  "capability": "Builds",
  "view": "4.22-main",
  "opened": "2026-01-28T08:02:45.127153Z",
  "closed": null,
  "status": "open",
  "last_failure": "2026-01-30T07:03:46Z",
  "variants": [...],
  "max_failures": 19,
  "triages": [...],
  "analysis_status": -2,
  "analysis_explanations": ["...", "..."],
  "sample_failed_jobs": {...},
  "test_details_url": "...",
  "api_url": "..."
}

Summary Format (--format summary)

Returns human-readable formatted summary:

Regression #34446 Details:
============================================================

Test Name: [sig-builds][Feature:Builds] custom build with buildah...
Release: 4.22 (baseline: 4.21)
Component: Build
Capability: Builds

Status: Open (opened: 2026-01-28)
Last Failure: 2026-01-30
Max Failures: 19

Analysis Status: -2
  (Negative status indicates a problem - lower is more severe)
Analysis Explanations:
  - Test is failing consistently across multiple job runs
  - Regression detected compared to baseline release

Affected Variants:
  - Architecture:amd64
  - Platform:metal
  - Network:ovn
  - Upgrade:none

Triages:
  - OCPBUGS-74651 (product, active): Description...

Test Details: https://sippy.dptools.openshift.org/...
API URL: https://sippy.dptools.openshift.org/api/...

Notes

  • The Python script uses only standard library modules (no external dependencies)
  • The API does not require authentication for read-only access
  • Regression IDs are persistent and do not change
  • The test_details_url provides links to sample job failures for further analysis
  • Closed regressions will have status: "closed" and a non-null closed timestamp
  • The variants array shows all platform/topology combinations where the test is failing
  • Default output format is JSON; use --format summary for human-readable output
  • analysis_status is an integer where negative values indicate problems (lower = more severe)
  • analysis_explanations provides human-readable context for the analysis status
  • sample_failed_jobs is a dictionary keyed by job name, containing only jobs with at least one failed run
  • Each job includes a pass_sequence string (newest to oldest) with "S" for successful runs and "F" for failed runs
  • Each job includes a label_summary dict aggregating job_labels across all failed runs for that job. This is the first place to look for a shared root cause: if most or all failed runs share a label, that symptom is almost certainly the root cause. Labels are human-defined and precise — they only fire when a regex written by a team member matches specific artifact content.
  • Each failed run includes test_failures (total tests failing in the job run) and job_labels (symptom labels for that specific run). Use test_failures > 10 as a signal for mass failure runs where the regressed test may be collateral damage.
  • job_runs contains the complete history of all job runs where the failure was observed across the regression's entire lifetime, sorted by start_time (newest first). Use this to determine when failures started, how common they were, and which jobs are most affected. The test_failures field shows the total failure count in each run — high values (e.g., >10) indicate mass failure runs where the regression may just be caught up in a larger issue rather than being the primary problem

See Also

  • Component Readiness API Documentation: https://sippy.dptools.openshift.org/api/docs
  • Related Skill: ci:prow-job-analysis (comprehensive Prow CI job failure analysis)
  • Related Command: /ci:analyze-regression (uses this skill)

Fetch Related Triages

Queries the Sippy API to find existing triage records and untriaged regressions that may be related to a given Component Readiness regression. The API matches based on similarly named tests and shared job runs (using the complete job_runs history tracked across each regression's entire lifetime).

When to Use This Skill

  • During regression analysis to find existing triages that could cover the current regression
  • To find untriaged regressions that share the same root cause and should be triaged together
  • To avoid filing duplicate bugs when a related triage already exists

API Endpoint

GET https://sippy.dptools.openshift.org/api/component_readiness/regressions/{id}/matches

No authentication required (uses production Sippy URL).

Usage

# Fetch all related triages (default: all confidence levels)
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479

# Filter to high confidence matches only
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479 --min-confidence 5

# Human-readable summary
python3 plugins/ci/skills/fetch-related-triages/fetch_related_triages.py 35479 --format summary

Arguments

  • regression_id (required): The Component Readiness regression ID
  • --min-confidence (optional): Minimum confidence level to include (1-10, default: 1)
  • --format (optional): Output format — json (default) or summary

Confidence Levels

The API assigns a confidence level (1-10) to each match:

  • 10: High confidence — the triage's regressions include the same test (exact test_id match) or very closely related tests with shared failure patterns
  • 5: Medium confidence — similarly named tests matched by edit distance, suggesting related tests in the same area
  • 2: Low confidence — regressions share the same job runs (identified via the job_runs history tracked across each regression's lifetime), which may indicate a shared root cause but could be coincidental in mass failure scenarios

Output Format

JSON output

{
  "success": true,
  "regression_id": 35479,
  "triaged_matches": [
    {
      "triage_id": 370,
      "triage_ui_url": "https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/370",
      "jira_key": "OCPBUGS-76612",
      "jira_url": "https://redhat.atlassian.net/browse/OCPBUGS-76612",
      "jira_status": "New",
      "jira_summary": "Component Readiness: [Networking / ovn-kubernetes] ...",
      "triage_type": "test",
      "triage_description": "...",
      "confidence_level": 10,
      "regressions_on_triage": [
        {
          "id": 35217,
          "test_name": "...",
          "test_id": "...",
          "component": "Networking / ovn-kubernetes",
          "variants": ["..."],
          "opened": "...",
          "closed": null,
          "triages": null,
          "last_failure": "..."
        }
      ]
    }
  ],
  "untriaged_regressions": [
    {
      "id": 35480,
      "test_name": "...",
      "test_id": "...",
      "component": "...",
      "variants": ["..."],
      "opened": "...",
      "closed": null,
      "triages": null,
      "last_failure": "...",
      "match_reason": "similarly_named_test",
      "edit_distance": 1
    }
  ]
}

Key fields

  • triaged_matches: Existing triage records that look related, sorted by confidence (highest first). Each includes the JIRA bug info and the regressions already on the triage.
  • untriaged_regressions: Open regressions that are not yet triaged but appear related. These are candidates to be triaged together with the current regression. Each includes a match_reason field:
    • similarly_named_test: Matched by test name edit distance (lower edit_distance = more similar)
    • same_last_failure: Matched by sharing the same job runs (identified via the job_runs history tracked across each regression's lifetime)

How the API Matches Work

The API uses two strategies to find related triages:

  1. Similarly named tests: Compares the regression's test name against test names on existing triages using edit distance. An edit distance of 0 means the same test name (different variants); 1-2 means very similar test names (e.g., L2 vs L3 variant of the same test).

  2. Shared job runs: Finds regressions that were observed in the same job runs. The regression data now includes a complete job_runs list covering all runs where the failure was observed throughout the regression's entire life (not just the last reporting period). Sippy uses this data to find other regressions that failed in the same Prow job runs, which is a strong signal they share the same root cause (e.g., a mass CI infrastructure failure or a broad product regression). Regressions with high test_failures counts in their shared runs are more likely to be collateral damage from a larger issue.

Notes

  • Uses the production Sippy URL (no port-forward needed)
  • No authentication required
  • The script deduplicates untriaged regressions by ID
  • Only open, untriaged regressions are included in untriaged_regressions
  • Triaged matches are sorted by confidence level (highest first)
  • The 60-second timeout handles the potentially large response

Fetch Releases

This skill fetches the list of available OpenShift releases from the Sippy API. It can return all OCP releases or just the latest version.

When to Use This Skill

Use this skill when you need to:

  • Determine the latest OCP release version
  • Get a list of all available OCP releases
  • Provide a default release version when the user hasn't specified one

Prerequisites

  1. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)
  2. Network Access: Must be able to reach the public Sippy API

    • Check: curl -s https://sippy.dptools.openshift.org/api/releases | head -c 100

Implementation Steps

Step 1: Get the Latest Release

To get just the latest OCP release version:

release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)
echo "$release"
# Output: 4.22

Step 2: List All Releases

To list all available OCP releases:

# JSON format (includes GA dates)
python3 plugins/ci/skills/fetch-releases/fetch_releases.py --format json

# Simple list, one per line
python3 plugins/ci/skills/fetch-releases/fetch_releases.py --format list

Step 3: Use with Other Skills

The primary use case is providing a default release to other skills:

# Determine release
release="${user_specified_release:-$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)}"

# Use with fetch-test-report
python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test_name>" --release "$release"

Output Format

--latest

Prints a single release version string:

4.22

--format json (default)

[
  {"release": "4.22"},
  {"release": "4.21", "ga": "2025-06-17T00:00:00Z"},
  {"release": "4.20", "ga": "2025-10-21T00:00:00Z"},
  {"release": "4.19", "ga": "2025-06-17T00:00:00Z"}
]

Releases without a ga field are still in development.

--format list

4.22
4.21
4.20
4.19
4.18

Error Handling

  1. Network Error: If the Sippy API is unreachable, the script prints an error to stderr and exits with code 1.
  2. No Releases: If no standard OCP releases are found, exits with code 1.

Exit Codes:

  • 0: Success
  • 1: Error

Notes

  • Uses the public Sippy API at https://sippy.dptools.openshift.org/api/releases (no port-forward needed)
  • Filters out non-standard releases (okd, aro, presubmits, etc.) — only returns releases matching X.Y format
  • The API returns releases in order with the latest first
  • This skill is based on the same API as the teams plugin's get-release-dates skill

See Also

  • Related Skill: fetch-test-report (uses release to query test data)
  • Related Skill (teams plugin): get-release-dates (fetches detailed release dates and metadata)

Fetch Test Report

This skill fetches a report for an OpenShift CI test by its full name using the Sippy tests API. It returns test metadata including the BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous reporting periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name, which can help surface bugs that have been filed but not yet triaged in Component Readiness.

When to Use This Skill

Use this skill when you need to:

  • Look up a test's Component Readiness / BigQuery test ID (test_id)
  • Check current and previous pass rates for a test
  • Find the Jira component associated with a test
  • Determine if a test is flaking, failing, or stable
  • Get run counts and failure/flake breakdowns for a test
  • Check if there are open Jira bugs mentioning this test (may not yet be triaged in Component Readiness)
  • See a per-variant breakdown of pass rates to identify if the test fails only in certain job types

Prerequisites

  1. Network Access: The Sippy API must be accessible at https://sippy.dptools.openshift.org

    • No authentication required
    • Check: curl -s https://sippy.dptools.openshift.org/api/health?release=4.22
  2. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Determine the Release

If the user did not specify a release, use the fetch-releases skill to determine the latest OCP release:

release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)

If the user specified a release, use that directly.

Step 2: Run the Python Script

The skill uses a Python script to query the Sippy tests API:

# Path to the Python script
script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"

# Fetch test report in JSON format (collapsed — one row for the test across all variants)
python3 "$script_path" "<test_name>" --release "$release" --format json

# Or fetch as human-readable summary
python3 "$script_path" "<test_name>" --release "$release" --format summary

# Fetch per-variant breakdown (one row per variant combo — useful for identifying
# if the test is failing only in certain job types, e.g., aws vs metal, upgrade vs new-install)
python3 "$script_path" "<test_name>" --release "$release" --no-collapse --format json

Step 3: Parse the Output

The script outputs a JSON array (typically with one element for an exact name match):

# Store JSON output in a variable for processing
test_data=$(python3 "$script_path" "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" --format json)

# Extract specific fields using jq
test_id=$(echo "$test_data" | jq -r '.[0].test_id')
component=$(echo "$test_data" | jq -r '.[0].jira_component')
pass_rate=$(echo "$test_data" | jq -r '.[0].current_pass_percentage')
runs=$(echo "$test_data" | jq -r '.[0].current_runs')

API Details

Endpoint

GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}
GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}&collapse=false

When collapse=false is specified, the API returns one row per variant combination instead of a single collapsed row. Each row includes a variants array showing the specific variant combo (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]). This helps identify if the test is failing in certain types of jobs and not others.

Filter Format

The filter parameter is a JSON object using Sippy's standard filter syntax:

{
  "items": [
    {
      "columnField": "name",
      "operatorValue": "equals",
      "value": "<test_name>"
    }
  ]
}

Response Schema

The API returns a JSON array of test objects:

[
  {
    "id": 5987,
    "test_id": "openshift-tests:e8f7491a505095b6356dfd8d4cf7218b",
    "name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
    "suite_name": "openshift-tests",
    "variants": null,
    "jira_component": "kube-apiserver",
    "jira_component_id": "12367637",
    "current_successes": 3425,
    "current_failures": 22,
    "current_flakes": 0,
    "current_pass_percentage": 99.36,
    "current_failure_percentage": 0.64,
    "current_flake_percentage": 0,
    "current_working_percentage": 99.36,
    "current_runs": 3447,
    "previous_successes": 2570,
    "previous_failures": 43,
    "previous_flakes": 0,
    "previous_pass_percentage": 98.35,
    "previous_failure_percentage": 1.65,
    "previous_flake_percentage": 0,
    "previous_working_percentage": 98.35,
    "previous_runs": 2613,
    "net_failure_improvement": 1.01,
    "net_flake_improvement": 0,
    "net_working_improvement": 1.01,
    "net_improvement": 1.01,
    "tags": null,
    "open_bugs": 0
  }
]

Key Fields:

  • id: Sippy PostgreSQL internal ID (less commonly used)
  • test_id: BigQuery / Component Readiness test ID (format: suite:hash). This is the ID used with modern skills like fetch-regression-details and fetch-test-runs.
  • name: Full test name
  • suite_name: Test suite (e.g., openshift-tests)
  • variants: Array of variant strings (only present when --no-collapse / collapse=false is used). Shows the specific variant combo for this row (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]).
  • jira_component: Associated OCPBUGS Jira component
  • jira_component_id: Jira component numeric ID
  • current_*: Metrics for the current 7-day reporting period
  • previous_*: Metrics for the 7 days before the current period
  • net_working_improvement: Change in working percentage (positive = improving)
  • open_bugs: Number of open Jira bugs that mention this test by name. This can surface bugs that have been filed but not yet triaged in Component Readiness. Useful for finding existing work before filing a duplicate.

Error Handling

Case 1: Sippy Not Reachable

python3 fetch_test_report.py "[sig-api-machinery] ..."
# Error: Failed to connect to Sippy API: [Errno 61] Connection refused
# Check network connectivity to sippy.dptools.openshift.org.

Case 2: No Test Found

If the test name doesn't match any test, the script returns an empty JSON array [] or "No tests found matching the given name." in summary mode.

Case 3: Missing Arguments

python3 fetch_test_report.py
# usage: fetch_test_report.py [-h] [--release RELEASE] [--format {json,summary}] test_name
# fetch_test_report.py: error: the following arguments are required: test_name

Exit Codes:

  • 0: Success
  • 1: Error (API error, network error, etc.)

Examples

Example 1: Fetch Test Report in JSON Format

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --format json

Example 2: Get a Human-Readable Summary

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --format summary

Expected Output:

Test: [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]
  Test ID:         openshift-tests:abc123def456
  Suite:           openshift-tests
  Jira Component:  kube-apiserver
  Open Bugs:       0

  Current Period (last 7 days):
    Runs:       3447
    Pass Rate:  99.36%
    Failures:   22 (0.64%)
    Flakes:     0 (0.00%)

  Previous Period (7 days before current):
    Runs:       2613
    Pass Rate:  98.35%
    Failures:   43 (1.65%)
    Flakes:     0 (0.00%)

  Trend:  improved (+1.01%)

Example 3: Fetch Per-Variant Breakdown

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.22 --no-collapse --format summary

This returns one row per variant combination, showing which specific job types the test is passing or failing in. Useful for regression analysis to determine if the failure is platform-specific, upgrade-specific, etc.

Example 4: Fetch Test Report for a Different Release

python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py \
  "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" \
  --release 4.21 --format json

Notes

  • The test name must be an exact match — use the full test name including sig prefix and suite tags
  • The API is accessed via the production Sippy URL; no authentication required
  • If --release is not specified, use the fetch-releases skill to determine the latest release
  • current_* fields cover the last 7 days; previous_* fields cover the 7 days before that
  • current_working_percentage = pass rate + flake rate (tests that ultimately passed, possibly after retries)
  • The test_id field is the one used in Component Readiness URLs and with the fetch-regression-details and fetch-test-runs skills
  • An empty result ([]) means no test matched — double-check the exact test name spelling

See Also

  • Related Skill: fetch-releases (determines the latest OCP release)
  • Related Skill: fetch-regression-details (uses test_id to fetch regression details)
  • Related Skill: fetch-test-runs (uses test name to fetch individual test run results)
  • Related Command: /ci:fetch-test-report (command that invokes this skill)
  • Related Command: /ci:analyze-regression (full regression analysis workflow)

Fetch Test Runs

This skill fetches test runs from the Sippy API. It can return both failed and successful test runs, including JUnit output for AI-based analysis.

When to Use This Skill

Use this skill when you need to:

  • Fetch test run data for a specific test across all jobs
  • Get raw test failure outputs for AI-based similarity analysis
  • Compare error messages across runs to determine if they share the same root cause
  • Include successful runs in addition to failures (optional)
  • Filter runs by job name substrings (e.g., only GCP techpreview jobs)
  • Access JUnit test output for debugging and investigation

Prerequisites

  1. Network Access: Must be able to reach the Sippy test runs API

    • No authentication required
    • Check: curl -s https://sippy.dptools.openshift.org/api/tests/v2/runs?test_id=test
  2. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)
  3. Input Data: Requires test_id (job_run_ids are optional)

    • Get from fetch-regression-details skill output
    • test_id: Found in regression data (e.g., "openshift-tests:71c053c318c11cfc47717b9cf711c326")
    • job_run_ids: Optional - extracted from sample_failed_jobs[].failed_runs[].job_run_id

Implementation Steps

Step 1: Run the Python Script

# Path to the Python script
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"

# Fetch all test runs (failures only, by default)
python3 "$script_path" "$test_id" --format json

# Include successful runs as well
python3 "$script_path" "$test_id" --include-success --format json

# Filter to a specific Prow job (exact name works as substring of itself)
python3 "$script_path" "$test_id" --job-contains "periodic-ci-openshift-release-..." --format json

# Filter by multiple substrings (AND logic, case-insensitive, server-side)
python3 "$script_path" "$test_id" --job-contains gcp --job-contains techpreview --format json

# Filter to specific job run IDs (backward compatible with analyze-regression)
python3 "$script_path" "$test_id" "$job_run_ids" --format json

# Get human-readable summary
python3 "$script_path" "$test_id" --format summary

Arguments:

  • test_id: Required test identifier (e.g., "openshift-tests:abc123")
  • job_run_ids: Optional comma-separated list of Prow job run IDs to filter by

Options:

  • --include-success: Include successful test runs (default: failures only)
  • --job-contains <name>: Filter by job name substring (server-side, case-insensitive). Repeatable for AND logic — all substrings must appear in the job name. Full job names also work since they are substrings of themselves. E.g., --job-contains gcp --job-contains techpreview matches jobs containing both "gcp" and "techpreview".
  • --start-days-ago <days>: Number of days to look back (default API is 7 days)
  • --exclude-output: Strip the output field from each run to reduce response size. Use when you only need pass/fail status (e.g., regression start analysis). Significantly reduces JSON output size for large result sets.
  • --output <path>: Write output to a file instead of stdout. Use when fetching large result sets (e.g., --include-success --start-days-ago 28) that may exceed stdout buffer limits. The script prints a summary line to stderr confirming the write.
  • --format json|summary: Output format (default: json)

Step 2: Prepare Input Data (for analyze-regression)

When used with regression analysis, extract required data from regression details:

# Assuming you have regression_data from fetch-regression-details skill
test_id=$(echo "$regression_data" | jq -r '.test_id')

# Collect all job_run_ids from sample_failed_jobs
# This creates a comma-separated list of all failed job run IDs
job_run_ids=$(echo "$regression_data" | jq -r '
  .sample_failed_jobs
  | to_entries[]
  | .value.failed_runs[]
  | .job_run_id
' | tr '\n' ',' | sed 's/,$//')

echo "Test ID: $test_id"
echo "Job Run IDs: $job_run_ids"

Step 3: Parse the Output

The script outputs structured JSON data:

# Store JSON output for processing
output_data=$(python3 "$script_path" "$test_id" --format json)

# Check if fetch was successful
success=$(echo "$output_data" | jq -r '.success')

if [ "$success" = "true" ]; then
  # Extract runs array
  runs=$(echo "$output_data" | jq -r '.runs')

  # The runs array contains objects with: url, output, test_name, success
  # The AI command will analyze these runs for similarity
  echo "Fetched $(echo "$runs" | jq 'length') runs"
else
  # Handle error case
  error=$(echo "$output_data" | jq -r '.error')
  echo "Error: $error"
  echo "Test runs API may not be available"
fi

API Response Schema

The Sippy API returns a JSON array of test run objects:

[
  {
    "url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.22-e2e-aws-ovn-techpreview/2016123858595090432",
    "output": "fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access: /apis/stable.e2e-validating-admission-policy-1181/: the server could not find the requested resource",
    "test_name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
    "success": false,
    "failed_tests": 3
  },
  {
    "url": "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...",
    "output": "",
    "test_name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
    "success": true,
    "failed_tests": 0
  }
]

Script Output Format

The script supports two output formats:

JSON Format (--format json)

Returns structured JSON with raw runs:

{
  "success": true,
  "test_id": "openshift-tests:71c053c318c11cfc47717b9cf711c326",
  "requested_job_runs": 0,
  "include_success": false,
  "job_name_filters": ["gcp", "techpreview"],
  "runs": [
    {
      "url": "https://prow.ci.openshift.org/...",
      "output": "fail [...]: error message",
      "test_name": "[sig-api-machinery] test name",
      "success": false,
      "failed_tests": 3
    }
  ],
  "api_url": "https://sippy.dptools.openshift.org/api/tests/v2/runs?test_id=...&prowjob_name=gcp&prowjob_name=techpreview"
}

Field Descriptions:

  • success: Boolean indicating if the API call succeeded
  • test_id: The test identifier that was queried
  • requested_job_runs: Number of job run IDs requested (0 if none specified)
  • include_success: Whether successful runs were requested
  • job_name_filters: List of job name substrings used for server-side filtering (null if not specified)
  • runs: Raw array of test run objects from Sippy API
    • url: Prow job URL for this specific run
    • output: The actual JUnit test failure output text (empty for successes)
    • test_name: Full test name
    • success: Boolean indicating if this run passed
    • failed_tests: Count of all tests that failed in this job run. If failed_tests > 10, this is a mass failure job — the test may be caught up in a larger issue (e.g., infrastructure failure, installer failure) that needs further investigation to root cause. When many runs show mass failures, the regression may not be caused by a change specific to this test.
  • api_url: The API URL that was called

Error Response (when success is false):

{
  "success": false,
  "error": "Failed to connect to test runs API: Connection refused",
  "test_id": "openshift-tests:abc123",
  "requested_job_runs": 0,
  "include_success": false,
  "job_name_filters": null
}

Summary Format (--format summary)

Returns human-readable formatted output with sample runs:

Test Runs
============================================================

Test ID: openshift-tests:71c053c318c11cfc47717b9cf711c326
Job Contains: ['gcp', 'techpreview']
Include Successes: False
Runs Fetched: 18

Successes: 0, Failures: 18

Mass Failure Runs (>10 test failures in job): 4 of 18
  ⚠ These runs had many other test failures — this test may be caught up in a larger issue.

Sample Runs:

1. [FAIL] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
   Failed Tests in Job: 3
   Output: fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access...

2. [FAIL] [MASS FAILURE] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
   Failed Tests in Job: 47
   Output: fail [k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145]: Fail to access...

3. [PASS] Job URL: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...

... and 15 more runs

Error Handling

Case 1: API Not Available

python3 fetch_test_runs.py "openshift-tests:abc"

Output (JSON format):

{
  "success": false,
  "error": "Failed to connect to test runs API: Connection refused.",
  "test_id": "openshift-tests:abc",
  "requested_job_runs": 0,
  "include_success": false,
  "job_name_filters": null
}

Output (summary format):

Test Runs - FETCH FAILED
============================================================

Error: Failed to connect to test runs API: Connection refused.

The test runs API may not be available.

Case 2: No Runs Returned

If the API returns an empty array:

{
  "success": true,
  "test_id": "openshift-tests:abc",
  "requested_job_runs": 0,
  "include_success": false,
  "job_name_filters": null,
  "runs": []
}

Case 3: Invalid Arguments

python3 fetch_test_runs.py

Output:

Usage: fetch_test_runs.py <test_id> [job_run_ids] [options]

Arguments:
  test_id       Test identifier (e.g., 'openshift-tests:abc123')
  job_run_ids   Optional comma-separated list of Prow job run IDs

Options:
  --include-success    Include successful test runs (default: failures only)
  --job-contains       Filter by job name substring (repeatable for AND logic)
  --exclude-output     Strip output text from runs to reduce response size
  --output <path>      Write output to file instead of stdout
  --format json|summary   Output format (default: json)

Exit Codes:

  • 0: Success
  • 1: Error (invalid input, API error, network error, etc.)

Examples

Example 1: Fetch All Failures for a Test

script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
python3 "$script_path" "openshift-tests:bb3a7d828630760296ef203c5cacf708" --format json

Example 2: Fetch All Runs Including Successes

script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
python3 "$script_path" "openshift-tests:bb3a7d828630760296ef203c5cacf708" --include-success --format json

Example 3: Fetch Specific Job Runs (Backward Compatible)

Used by analyze-regression command:

# Assume regression_data is already fetched
test_id=$(echo "$regression_data" | jq -r '.test_id')
job_run_ids=$(echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries[] | .value.failed_runs[] | .job_run_id' | tr '\n' ',' | sed 's/,$//')

# Fetch outputs for specific job runs
script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
output_data=$(python3 "$script_path" "$test_id" "$job_run_ids" --format json)

# Check success
if [ "$(echo "$output_data" | jq -r '.success')" = "true" ]; then
  echo "Successfully fetched runs"
fi

Example 4: Get Summary Report

python3 plugins/ci/skills/fetch-test-runs/fetch_test_runs.py \
  "openshift-tests:71c053c318c11cfc47717b9cf711c326" \
  --format summary

Example 5: Extract Output Messages for AI Analysis

# Fetch runs
output_data=$(python3 "$script_path" "$test_id" --format json)

# Extract all failure output messages
if [ "$(echo "$output_data" | jq -r '.success')" = "true" ]; then
  # Get all output texts from failed runs
  echo "$output_data" | jq -r '.runs[] | select(.success == false) | .output'

  # AI command will analyze these for:
  # - Similarity/consistency
  # - Common error patterns
  # - File references and API paths
  # - Root cause determination
fi

Example 6: Filter Runs by Job Name Substrings

Filter to runs from jobs matching multiple criteria (e.g., GCP + techpreview):

script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"

# Get only GCP techpreview runs (both substrings must match, server-side)
python3 "$script_path" "openshift-tests:abc123" --include-success \
  --job-contains gcp --job-contains techpreview \
  --start-days-ago 28 --format json

# Get only metal upgrade runs
python3 "$script_path" "openshift-tests:abc123" \
  --job-contains metal --job-contains upgrade \
  --format summary

# Full job name also works (it's a substring of itself)
python3 "$script_path" "openshift-tests:abc123" \
  --job-contains "periodic-ci-openshift-release-master-nightly-4.22-e2e-gcp-ovn-techpreview" \
  --format json

Example 7: Determine Regression Start Date

Used by analyze-regression command to find when failures began:

script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"

# Get the job with the most failures
most_failed_job="periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn"

# Fetch all runs (including successes) for this specific job, going back 28 days
job_history=$(python3 "$script_path" "$test_id" \
  --include-success \
  --job-contains "$most_failed_job" \
  --start-days-ago 28 \
  --format json)

# Analyze the run history
if [ "$(echo "$job_history" | jq -r '.success')" = "true" ]; then
  # Runs are returned newest to oldest
  # Iterate to find where failures started
  echo "$job_history" | jq -r '.runs[] | "\(.success) \(.url)"'

  # Look for transition from passing to failing
  # Find the first failure that's part of the current regression
fi

Notes

  • The script uses only Python standard library - no external dependencies required
  • Uses the production Sippy URL
  • Handles API unavailability by returning a JSON error response with success: false and a descriptive error message
  • Returns raw outputs for AI-based interpretation and similarity analysis
  • Job run IDs are optional - can fetch all runs for a test
  • --include-success allows analyzing both passing and failing runs
  • --job-contains filters results server-side using case-insensitive substring matching. Repeat for AND logic (all substrings must appear in the job name). Full job names work too since they are substrings of themselves.
  • --start-days-ago allows looking back further than the default 7 days (e.g., --start-days-ago 28)
  • Combine --include-success, --job-contains, and --start-days-ago to get full test history for regression analysis
  • Backward compatible with analyze-regression command (accepts job_run_ids)
  • Summary format shows first 5 runs only, to keep output manageable
  • Runs are returned in order from most recent to least recent
  • Each run includes a failed_tests count — the total number of tests that failed in that job. If failed_tests > 10, it indicates a mass failure job where many tests failed together, suggesting the test may be caught up in a larger issue (infrastructure failure, installer failure, etc.) rather than a regression specific to this test

See Also

  • Related Skill: fetch-regression-details (provides test_id and job_run_ids)
  • Related Command: /ci:analyze-regression (uses this skill for failure analysis)

Fetch Tide PR Status

CLI equivalent of the Prow PR Status page. Focuses on labels and required jobs — the most common merge blockers. Tide also checks other things (author, milestone) which may appear in the tide verdict but are not broken out here.

Only outputs open, non-draft PRs.

Prerequisites

  • gh CLI authenticated
  • python3 with pyyaml (pip install pyyaml)

Usage

# All open PRs by an author
python3 "${SKILL_DIR}/fetch_tide_pr_status.py" openshift/cluster-monitoring-operator machine424

# Specific PRs
python3 "${SKILL_DIR}/fetch_tide_pr_status.py" openshift/cluster-monitoring-operator 3057,3044

Output

JSON array, one object per PR:

[
  {
    "number": 3044,
    "title": "OCPBUGS-105305: [release-4.21] wrap ...",
    "author": "machine424",
    "url": "https://github.com/openshift/cluster-monitoring-operator/pull/3044",
    "branch": "release-4.21",
    "state": "open",
    "mergeable": true,
    "tide": "Not mergeable. Needs jira/valid-bug label.",
    "blockers": [
      "missing required label: jira/valid-bug",
      "has forbidden label: jira/invalid-bug",
      "job not passing: ci/prow/e2e-aws-ovn (running)"
    ],
    "labels": {
      "met": false,
      "required": {
        "have": ["lgtm", "approved"],
        "missing": ["jira/valid-bug"]
      },
      "forbidden": {
        "have": ["jira/invalid-bug"],
        "clear": ["needs-rebase", "do-not-merge/hold"]
      }
    },
    "required_jobs": [
      {"name": "ci/prow/unit", "state": "success"},
      {"name": "ci/prow/e2e-aws-ovn", "state": "running", "description": "Job triggered."}
    ]
  }
]

tide — Tide's own verdict. blockers — what's preventing merge. required_jobs — non-optional jobs from openshift/release config; if the config could not be fetched, contains a single {"error": "..."} entry instead of jobs (not reflected in blockers).

Data Sources

  • https://prow.ci.openshift.org/tide.js — required/forbidden labels (public)
  • gh api repos/{o}/{r}/pulls/{n} — PR metadata
  • gh api repos/{o}/{r}/commits/{sha}/status — job statuses + Tide context
  • raw.githubusercontent.com/openshift/release/master/ci-operator/jobs/... — presubmit config (required vs optional)

Generate OKD/SCOS ci-operator configuration YAML files for the openshift/release repository from ART image metadata using the bundled Python generator.

Arguments

Parse from: $ARGUMENTS

  • First argument: path to the ocp-build-data directory (must contain group.yml, streams.yml, images/).
  • Second argument: OKD version string (e.g. 4.18, 5.0).
  • --output-dir <path>: Directory to write ci-operator configs (default for this skill: ./okd-ci-configs). In an openshift/release checkout, use its ci-operator/config/ directory.
  • --github-token <token>: Optional GitHub token for downloading upstream Dockerfiles. Requests are unauthenticated when omitted.
  • --dry-run: Print generated YAML without writing files. Upstream Dockerfiles are still downloaded.

If no arguments are provided, look for ocp-build-data at ./tmp-dir/ocp-build-data/ or ../ocp-build-data/, and derive the OKD version from group.yml vars (MAJOR.MINOR). Use build data for the requested release; the OKD version argument controls the destination imagestream and does not change the metadata's branches.

Execution

1. Locate the bundled helper

Set SKILL_DIR to the absolute directory containing this SKILL.md. The entry point is generate_okd_ci_configs.py alongside this file.

  • In this repository, the skill directory is .claude/skills/generate-okd-ci-operator-configs-with-helper-script/.
  • In the ai-helpers CI plugin, it is ${CLAUDE_PLUGIN_ROOT}/skills/generate-okd-ci-operator-configs/.

The entry point imports the sibling okd_ci_configs package. Keep the whole skill directory together when installing or copying it. Running the entry point by its absolute path works from any working directory without setting PYTHONPATH or installing the helper package.

Python 3, PyYAML, and requests are required. Use the project's virtual environment if available. Check dependencies with:

python3 -c 'import yaml, requests'

2. Clone ocp-build-data (if not already present)

If the ocp-build-data directory is not provided or does not exist locally, clone it from GitHub:

git clone https://github.com/openshift-eng/ocp-build-data.git --branch openshift-$OKD_VERSION

The branch should match the target OKD version (e.g. openshift-5.0 for OKD 5.0, openshift-4.18 for OKD 4.18).

3. Validate inputs and run the generator

Verify the build-data directory contains group.yml, streams.yml, and images/. Resolve the arguments into OCP_BUILD_DATA_PATH, OKD_VERSION, and OUTPUT_DIR; pass ./okd-ci-configs explicitly when the skill's output default is used, since the Python CLI requires --output-dir.

python3 "$SKILL_DIR/generate_okd_ci_configs.py" \
    --ocp-build-data "$OCP_BUILD_DATA_PATH" \
    --okd-version "$OKD_VERSION" \
    --output-dir "$OUTPUT_DIR"

Append --github-token with the provided token or --dry-run when requested.

The generator reads and substitutes ART metadata, resolves payload dependencies, downloads upstream Dockerfiles, and writes one config per (org, repo, branch) to:

{output-dir}/{org}/{repo}/{org}-{repo}-{branch}__okd-scos.yaml

It handles okd_alignment overrides for parent images, Dockerfiles, source paths, context directories, build arguments, build roots, and RPM repository injection.

4. Report results

Report the generated config count, output paths, skipped images with reasons, and warnings or errors. A successful invocation can still skip images or refuse incomplete repository configs; include those limitations in the report.

For configs generated into an openshift/release checkout, the follow-up generation commands are make ci-operator-configs and make jobs.

Helper layout

The entry point owns argument parsing and input validation. Read the relevant module when diagnosing or changing a specific part of generation:

ModuleResponsibility
helpers.pyGit URL normalization, public upstream mappings, variable substitution, and CI image coordinates
metadata.pyLoad image metadata, resolve OKD tags and pullspecs, and find payload dependencies
dockerfiles.pyDownload upstream Dockerfiles and parse FROM instructions
ci_operator.pyBuild image/repository configurations and serialize ci-operator YAML
generator.pyOrchestrate loading, reconciliation, generation, and result reporting

When copying this skill into ai-helpers, use plugins/ci/skills/generate-okd-ci-operator-configs/ and set the frontmatter name to generate-okd-ci-operator-configs to match that directory. The helper package needs no plugin manifest registration.

Key resolution rules

  • Streams resolve in order: okd.resolve_as.image, upstream_image, then image.
  • Images with okd_alignment.resolve_as.stream or .image use the resolved pullspec and are not built by this generator.
  • Otherwise, images use registry.ci.openshift.org/origin/scos-{version}:{tag}. The tag comes from okd_alignment.tag_name, or payload_name/name with the ose- prefix stripped.
  • Public upstream URLs use the longest matching public_upstreams mapping in group.yml. A mapping's public_branch overrides the source branch; otherwise the source branch is used, with main as the fallback when absent.
  • Images explicitly disabled for OKD, lacking usable source or parent metadata, or unnecessary for the payload are skipped. Dockerfile download failures and parent-count mismatches also skip images.

List Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags that categorize failures — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

This skill lists, searches, and fetches symptoms and labels using the public (read-only, no auth) Sippy API.

When to Use This Skill

Use this skill when you need to:

  • Browse the catalog of known CI failure signatures (symptoms)
  • Check whether a failure pattern already has a symptom before creating a new one (avoid duplicates)
  • Look up what a label means, including its title, explanation, and associated Jira issues
  • Find which symptoms apply a given label
  • Inspect a single symptom or label by its ID

Prerequisites

  1. Network Access: The Sippy API must be accessible at https://sippy.dptools.openshift.org

    • No authentication required
    • Check: curl -s https://sippy.dptools.openshift.org/api/jobs/labels | head -c 200
  2. Python 3: Python 3.6 or later

    • Check: python3 --version
    • Uses only standard library (no external dependencies)

Implementation Steps

Invoke the script with the flags that match the user's question:

script_path="plugins/ci/skills/list-symptoms/list_symptoms.py"

# List all symptoms (JSON by default)
python3 "$script_path"

# List all symptoms as a human-readable summary
python3 "$script_path" --format summary

# Search symptoms by text (matches id, summary, and match_string, case-insensitive)
python3 "$script_path" --search "credentials" --format summary

# Find all symptoms that apply a given label
python3 "$script_path" --label AWSAuthFailure --format summary

# Fetch a single symptom by ID
python3 "$script_path" --id AWSCouldNotValidateAccessCredentials

# List all labels instead of symptoms
python3 "$script_path" --labels --format summary

# Fetch a single label by ID
python3 "$script_path" --labels --id AWSAuthFailure

Flags:

  • --id <id>: fetch a single symptom (or label with --labels) by ID
  • --search <text>: case-insensitive text search over id/summary/match_string (for labels: id/label_title/explanation/bugs)
  • --label <label_id>: only symptoms that apply this label ID
  • --matcher-type {string,regex,none,cel}: only symptoms of this matcher type
  • --labels: operate on labels instead of symptoms
  • --format {json,summary}: output format (default json)

API Details

Endpoints

GET https://sippy.dptools.openshift.org/api/jobs/symptoms
GET https://sippy.dptools.openshift.org/api/jobs/symptoms/{id}
GET https://sippy.dptools.openshift.org/api/jobs/labels
GET https://sippy.dptools.openshift.org/api/jobs/labels/{id}

Example Symptom JSON

{
  "id": "AWSCouldNotValidateAccessCredentials",
  "summary": "AWS could not validate access credentials",
  "matcher_type": "string",
  "file_pattern": "build-log.txt",
  "match_string": "api error AuthFailure: AWS was not able to validate the provided access credentials",
  "label_ids": ["AWSAuthFailure"],
  "created_by": "kenzhang",
  "updated_by": "kenzhang",
  "updated_at": "2026-04-27T16:09:12.660547Z"
}

Symptom Fields

FieldDescription
idImmutable identifier (generated from the summary on create)
summaryShort, unique human-readable description (≤200 chars)
matcher_typeOne of string, regex, none (file exists), cel (expression over label names)
file_patternGlob over the job run's artifact files, e.g. **/build-log.txt (not used for cel)
match_stringSubstring, regular expression, or CEL expression depending on matcher_type
label_idsLabel IDs applied to a run when the symptom matches
created_by, updated_by, updated_atMetadata about who created/last modified the symptom and when

Label Fields

FieldDescription
idImmutable identifier (≤80 chars)
label_titleUnique human-readable title
explanationMarkdown explanation of what the label means
bugsJira issue keys associated with the label
hide_display_contextsUI contexts where the label is hidden (subset of spyglass, metrics, jaq-options)

Error Handling

Case 1: Not Found (404)

python3 list_symptoms.py --id NoSuchSymptom
# Error: not found: https://sippy.dptools.openshift.org/api/jobs/symptoms/NoSuchSymptom (use list mode to see valid IDs)

Exits 1 and suggests listing symptoms/labels to find valid IDs.

Case 2: Network Error

# Error: failed to connect to Sippy API: [Errno -2] Name or service not known

Exits 1. Check network connectivity to sippy.dptools.openshift.org.

Case 3: Empty Results

Filters that match nothing print [] (JSON) or No results. (summary) and exit 0 — this is not an error.

Exit Codes:

  • 0: Success (including empty results)
  • 1: Error (404, network error, etc.)

Examples

Example 1: List All Symptoms

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --format summary

Expected Output (excerpt):

Symptom: AWSCouldNotValidateAccessCredentials
  Summary:      AWS could not validate access credentials
  Matcher:      string
  File pattern: build-log.txt
  Match string: api error AuthFailure: AWS was not able to validate the provided access credentials
  Labels:       AWSAuthFailure
  Updated by:   kenzhang at 2026-04-27T16:09:12.660547Z

Total: 42

Example 2: Search for an Existing Symptom Before Creating One

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --search "credentials" --format summary

Example 3: Symptoms That Apply a Label

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --label AWSAuthFailure --format summary

Example 4: Fetch One Symptom by ID

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --id AWSCouldNotValidateAccessCredentials

Example 5: List All Labels

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --labels --format summary

Notes

  • --search, --label, and --matcher-type filtering is performed client-side: the script fetches the full list and filters locally
  • This skill is read-only; to create, update, or delete symptoms or labels use the manage-symptoms and manage-labels skills
  • No authentication is required — the public Sippy instance serves all read endpoints

See Also

  • Related Skill: manage-symptoms (create/update/delete symptoms)
  • Related Skill: manage-labels (create/update/delete labels)
  • Related Skill: diagnose-job-run-symptoms (explain which symptoms/labels apply to a job run)
manage-labelsmanage-symptomsoc-authpayload-analysispayload-autodl-jsonpayload-experimental-revertspayload-results-yamlpayload-snapshotprow-job-analysisprow-job-analyze-resourceprow-job-prometheusreevaluate-job-runsrevert-prset-release-blockerstage-payload-revertstriage-regressiontrigger-payload-jobunderstanding-openshift-prs

Commands 20

Name

ci:add-debug-wait

Synopsis

/ci:add-debug-wait <workflow-or-job-name> [timeout]

Description

The ci:add-debug-wait command adds a wait step to a CI job/workflow for debugging test failures.

What it does:

  1. Takes job name, OCP version, and optional timeout as input
  2. Finds and edits the job config or workflow file
  3. Asks if any existing test steps should be removed (and cleans up orphaned env vars)
  4. Adds - ref: wait before the last remaining test step (with optional timeout configuration)
  5. Runs make update to validate and regenerate Prow jobs
  6. Creates a git commit, pushes to your fork, and creates a PR via gh

That's it! Simple, fast, and automated.

Implementation

The command performs the following steps:

Step 1: Gather Required Information

Prompt user for (in this order):

  1. Workflow/Job Name: (from command argument $1 or prompt)

    Workflow or job name: <user-input>
    Example: aws-c2s-ipi-disc-priv-fips-f7
    Example: baremetalds-two-node-arbiter-e2e-openshift-test-private-tests
    
  2. Timeout (optional, from command argument $2):

    Wait timeout in hours (optional, default: 3h):
    Examples: "1h", "2h", "8h", "24h", "72h"
    Valid range: 1h to 72h
    
    • If not provided, uses the wait step's default behavior (3 hours)
    • Format: Integer followed by 'h' (e.g., "1h", "2h", "8h")
    • Valid range: 1h to 72h (maximum enforced by wait step's timeout setting)
    • For job config files (ci-operator/config/): Set as the TIMEOUT env var (e.g., TIMEOUT: +8 hours) in the job's env: section
    • For workflow files (ci-operator/step-registry/): Set as timeout: (normalized to Go duration format, e.g., "8h" → "8h0m0s") and best_effort: true properties on the ref step
  3. OCP Version: (prompt - REQUIRED for searching job configs)

    OCP version for debugging (e.g., 4.18, 4.19, 4.20, 4.21, 4.22):
    

    This is used to:

    • Search the correct job config file (e.g., release-4.21)
    • Document which version needs debugging
    • Add context to the PR
  4. OpenShift Release Repo Path: (prompt if not in current directory)

    Path to openshift/release repository:
    Default: ~/repos/openshift-release
    

Step 2: Validate Environment and Prepare Branch

Silently validate and prepare (no user prompts):

cd <repo-path>

# Check 1: Repository exists and is correct
git remote -v | grep "openshift/release" || exit 1

# Check 2: Determine the default branch name (main or master)
default_branch=$(git remote show upstream 2>/dev/null | grep "HEAD branch" | awk '{print $NF}')
if [ -z "$default_branch" ]; then
  # Fallback: check which branch exists locally
  for branch in main master; do
    if git rev-parse --verify "$branch" >/dev/null 2>&1; then
      default_branch="$branch"
      break
    fi
  done
fi

# Check 3: Checkout default branch and update from upstream
# Use --track -B to handle fresh clones where the local branch may not exist
git checkout --track -B "${default_branch}" "upstream/${default_branch}" 2>/dev/null \
  || git checkout "${default_branch}"
git pull --rebase upstream "${default_branch}"

Step 3: Search for Job/Test Configuration

Priority 1: Search job configs first (more specific and targeted):

cd <repo-path>

# Search for job config files matching the OCP version
# The job name could be in various config files, so search broadly
grep -r "as: ${job_name}" ci-operator/config/ --include="*release-${ocp_version}*.yaml" -l

Example searches:

  • For aws-c2s-ipi-disc-priv-fips-f7 and OCP 4.21:
    grep -r "as: aws-c2s-ipi-disc-priv-fips-f7" ci-operator/config/ --include="*release-4.21*.yaml" -l
    

Handle job config search results:

  • 1 file found:

    ✅ Found job configuration:
    ${file_path}
    
    Type: Job configuration file
    
    Proceeding with job config modification...
    

    → Continue to Step 4a: Analyze Job Configuration

  • Multiple files found:

    Found ${count} matching job config files:
    
    1. ci-operator/config/.../release-4.21__amd64-nightly.yaml
    2. ci-operator/config/.../release-4.21__arm64-nightly.yaml
    3. ci-operator/config/.../release-4.21__ppc64le-nightly.yaml
    
    Select file (1-${count}) or 'q' to quit:
    

    Prompt user to select which file to modify, then continue to Step 4a: Analyze Job Configuration

  • 0 files found:

    ℹ️  No job config found for: ${job_name} (OCP ${ocp_version})
    
    Searching for workflow files instead...
    

    → Continue to Priority 2 below

Priority 2: Search workflow files (if job config not found):

cd <repo-path>

# Search for workflow files
find ci-operator/step-registry -type f -name "*${workflow_name}*workflow*.yaml"

Handle workflow search results:

  • 0 files found:

    ❌ No job config or workflow file found for: ${job_name}
    
    Suggestions:
    1. Check spelling of job/workflow name
    2. Verify OCP version (${ocp_version})
    3. Try with partial name
    4. Search manually:
       - Job configs: grep -r "as: ${job_name}" ci-operator/config/
       - Workflows: find ci-operator/step-registry -name "*workflow*.yaml" | grep <partial-name>
    
  • 1 file found:

    ✅ Found workflow file:
    ${file_path}
    
    Type: Workflow file
    
    Proceeding with workflow modification...
    

    → Continue to Step 4b: Analyze Workflow File

  • Multiple files found:

    Found ${count} matching workflow files:
    
    1. ci-operator/step-registry/.../workflow1.yaml
    2. ci-operator/step-registry/.../workflow2.yaml
    3. ci-operator/step-registry/.../workflow3.yaml
    
    Select file (1-${count}) or 'q' to quit:
    

    Prompt user to select which file to modify, then continue to Step 4b: Analyze Workflow File

Step 4a: Analyze Job Configuration

Read and parse the job config YAML:

# Find the specific test definition
grep -A 30 "as: ${job_name}" <job-config-file>

Check for:

  1. ✅ Has steps: section
  2. ✅ Has test: section inside steps
  3. ❌ Does NOT already have - ref: wait

Example current structure:

- as: aws-c2s-ipi-disc-priv-fips-f7
  cron: 36 16 3,12,19,26 * *
  steps:
    cluster_profile: aws-c2s-qe
    env:
      BASE_DOMAIN: qe.devcluster.openshift.com
      FIPS_ENABLED: "true"
    test:
    - chain: openshift-e2e-test-qe
    workflow: cucushift-installer-rehearse-aws-c2s-ipi-disconnected-private

If wait already exists:

ℹ️  Wait step already configured in job config

Current test section:
  test:
  - ref: wait
  - chain: openshift-e2e-test-qe

No changes needed. The job is already set up for debugging.

If no test section found:

ℹ️  Job config found but no test: section

This job uses only the workflow's test steps.
Searching for the workflow: ${workflow_name}

→ Fall back to searching for workflow (Priority 2 in Step 3)

Ask about removing existing test steps: Display the test: steps and ask the user if any should be removed (common when only the cluster + wait step is needed). If steps are removed, also remove any env vars in the env: section that were only used by those steps (e.g., OPERATORS, SKIP_TESTS) to avoid make update validation errors about undeclared parameters.

→ Continue to Step 5a: Show Diff for Job Config

Step 4b: Analyze Workflow File

Read and parse the workflow YAML:

cat <workflow-file>

Check for:

  1. ✅ Has workflow: section
  2. ✅ Has test: section
  3. ❌ Does NOT already have - ref: wait

Example current structure:

workflow:
  as: baremetalds-two-node-arbiter-upgrade
  steps:
    pre:
      - chain: baremetalds-ipi-pre
    test:
      - chain: baremetalds-ipi-test
    post:
      - chain: baremetalds-ipi-post

If wait already exists:

ℹ️  Wait step already configured in workflow

Current test section:
  test:
    - ref: wait
    - chain: baremetalds-ipi-test

No changes needed. The workflow is already set up for debugging.

If no test section exists: Add a test: section with the wait step before the post: section. Since these are debug PRs (never merged), it is safe to add a test section even if the workflow didn't have one.

    test:
      - ref: wait
    post:
      - chain: cucushift-installer-rehearse-aws-ipi-deprovision

Or with custom timeout:

    test:
      - ref: wait
        timeout: 8h0m0s
        best_effort: true
    post:
      - chain: cucushift-installer-rehearse-aws-ipi-deprovision

Ask about removing existing test steps: Same as Step 4a — display and ask. For workflow files, warn that removing steps affects ALL jobs using this workflow.

→ Continue to Step 5b: Modify Workflow File

Step 5a: Modify Job Config File

Edit the job config file directly - no confirmation needed:

# Add wait step before the last test step
# If timeout is provided, set it via the TIMEOUT env var in the job's env section
# See Step 6 for the YAML modification algorithm

After editing, validate the file by running the ref-step linter:

python3 <ai-helpers-repo>/plugins/ci/scripts/validate-ref-steps.py <job-config-file>

This catches invalid properties (e.g., timeout:, best_effort:) on ref: steps in job config files. If validation fails, fix the file before committing — use the TIMEOUT env var instead.

Two scenarios:

  1. Without custom timeout (uses wait step's built-in default of 3h):

    test:
    - ref: wait
    - chain: openshift-e2e-test-qe
    

    Note: No timeout or env change needed - the wait step will use its default TIMEOUT env var (3 hours)

  2. With custom timeout (user-provided timeout parameter): Add TIMEOUT to the existing env: section within the job's steps: block, and add - ref: wait to the test: section:

    env:
      BASE_DOMAIN: qe.devcluster.openshift.com
      TIMEOUT: +8 hours
    test:
    - ref: wait
    - chain: openshift-e2e-test-qe
    

    Note: The wait ref reads the TIMEOUT env var to determine how long to wait. Format: +N hours where N is the number of hours (e.g., +1 hours, +8 hours, +24 hours, +72 hours). Do NOT quote the value — the determinize-ci-operator tool strips quotes. If the job already has an env: section, add the TIMEOUT key to it. If not, create the env: section under steps:.

Show brief confirmation:

✅ Modified: ${job_name} (OCP ${ocp_version})
   File: <job-config-file-path>
   Added: - ref: wait${timeout:+ (TIMEOUT: +${hours} hours)}

Step 5b: Modify Workflow File

Edit the workflow file directly - no confirmation needed:

# Add wait step before the last test step
# If timeout is provided, add it as a step property
# See Step 6 for the YAML modification algorithm

Two scenarios:

  1. Without custom timeout (uses wait step's built-in default of 3h):

    test:
    - ref: wait
    - chain: baremetalds-ipi-test
    

    Note: No timeout or best_effort needed - the wait step will use its default TIMEOUT env var (3 hours)

  2. With custom timeout (user-provided timeout parameter):

    test:
    - ref: wait
      timeout: 8h0m0s
      best_effort: true
    - chain: baremetalds-ipi-test
    

    Note: best_effort: true is required when timeout is customized to prevent the wait step from failing the job if it times out

Show brief confirmation:

✅ Modified: ${workflow_name} workflow
   File: <workflow-file-path>
   Added: - ref: wait${timeout:+ (timeout: ${timeout})}
   ⚠️  Impact: Affects ALL jobs using this workflow

Step 6: Create Branch and Commit

Branch naming:

debug-${workflow_name}-${ocp_version}-$(date +%Y%m%d)

Example: debug-baremetalds-two-node-arbiter-4.21-20250131

Git operations:

# Create branch from the updated default branch (already checked out and updated in Step 2)
git checkout -b "${branch_name}"

# Modify the file (add wait step using the implementation below)
# Add '- ref: wait' before the last remaining step in the test: section

# Run make update to validate config and regenerate Prow jobs
make update

# Stage all changes (config file + any generated files from make update)
git add <config-file> <any-other-changed-files>

# Commit
git commit -m "[Debug] Add wait step to ${workflow_name} for OCP ${ocp_version}

This adds a wait step to enable debugging of test failures in OCP ${ocp_version}.
${removed_steps:+Removes ${removed_steps} steps and their associated orphaned env vars.}

The wait step pauses the workflow before tests run, allowing QE to:
- SSH into the test environment
- Inspect system state and logs
- Debug configuration issues
- Investigate test failures

OCP Version: ${ocp_version}
Workflow: ${workflow_name}"

YAML Modification Algorithm:

The modification process for both job configs and workflow files follows the same pattern:

  1. Locate the target: Find the test: section

    • For job configs: Within the specific job definition (- as: ${job_name})
    • For workflows: At the workflow level
  2. Find test steps: Identify all steps (lines with - ref: or - chain:)

  3. Check for duplicates: Ensure - ref: wait doesn't already exist

  4. Insert wait step: Add before the last test step with matching indentation

  5. Handle timeout (differs between job configs and workflow files):

    For job config files (ci-operator/config/):

    • Without timeout: Add simple - ref: wait
    • With timeout: Add simple - ref: wait AND add TIMEOUT: +N hours to the job's env: section

    For workflow files (ci-operator/step-registry/):

    • Without timeout: Add simple - ref: wait
    • With timeout: Add - ref: wait with timeout: and best_effort: properties (valid in workflow schema)

Example transformation (job config):

Before:

env:
  BASE_DOMAIN: qe.devcluster.openshift.com
test:
- chain: openshift-e2e-test-qe

After (without timeout):

env:
  BASE_DOMAIN: qe.devcluster.openshift.com
test:
- ref: wait
- chain: openshift-e2e-test-qe

After (with timeout=8h):

env:
  BASE_DOMAIN: qe.devcluster.openshift.com
  TIMEOUT: +8 hours
test:
- ref: wait
- chain: openshift-e2e-test-qe

Example transformation (workflow file):

Before:

test:
- chain: baremetalds-ipi-test

After (with timeout=8h):

test:
- ref: wait
  timeout: 8h0m0s
  best_effort: true
- chain: baremetalds-ipi-test
  1. Validate: Run validate-ref-steps.py on the modified file (see Step 5a). Fix any errors before committing.

Critical constraints:

  • Preserve exact YAML indentation (typically 2 spaces per level)
  • Insert BEFORE the last step, not after
  • Normalize timeout format: for job configs use +N hours, for workflow files use Go duration (e.g., "8h" → "8h0m0s")

Step 7: Push to Fork

Push the branch to the user's fork (origin remote):

git push origin "${branch_name}"

Step 8: Create PR

Create the PR using gh:

gh pr create --repo openshift/release --title "[Debug] Add wait step to ${job_name} for OCP ${ocp_version}" --body "$(cat <<'EOF'
## Summary
- Adds a wait step to enable debugging of test failures in OCP ${ocp_version}
- Job: ${job_name}
- Timeout: ${timeout:-default (3h)}
${removed_steps:+- Removed steps: ${removed_steps} and their associated orphaned env vars}

The wait step pauses the workflow before tests run, allowing QE to:
- SSH into the test environment
- Inspect system state and logs
- Debug configuration issues
- Investigate test failures

⚠️ **DO NOT MERGE** — close this PR after debugging is complete.
EOF
)"

Display the PR URL:

✅ PR created successfully!

PR: <pr_url>
Branch: ${branch_name}
Job: ${job_name}
OCP: ${ocp_version}

⚠️  Remember to close PR after debugging (DO NOT MERGE)

That's it! Simple and clean.

Error Handling

Error: Repository Not Found

❌ Error: Repository not found at ${repo_path}

Please provide the correct path to openshift/release repository.

To clone:
git clone https://github.com/openshift/release.git

Error: Not in openshift/release Repo

❌ Error: This doesn't appear to be the openshift/release repository

Remote URL: ${current_remote}
Expected: github.com/openshift/release

Please navigate to the correct repository.

Error: Workflow File Not Found

❌ Error: Workflow file not found

Searched for: *${workflow_name}*workflow*.yaml
Location: ci-operator/step-registry/

Suggestions:
1. Verify the workflow name
2. Try a partial match
3. Search manually: find ci-operator/step-registry -name "*workflow*.yaml"

Error: Wait Step Already Exists

ℹ️  Wait step already configured in this workflow

No action needed - you can proceed with debugging using the existing wait step.

Error: Invalid OCP Version

❌ Invalid OCP version: ${version}

Valid versions: 4.18, 4.19, 4.20, 4.21, 4.22, master

Please provide a valid version.

Error: Invalid Timeout Format

❌ Invalid timeout format: ${timeout}

Valid format: Integer followed by 'h' (e.g., "1h", "2h", "8h", "24h", "72h")
Valid range: 1h to 72h

Examples:
- "1h" (1 hour)
- "8h" (8 hours)
- "24h" (24 hours)
- "72h" (72 hours, maximum)

Please provide a valid timeout in hours.

Note: Timeout Normalization

When a user provides a timeout like "8h", the implementation should normalize it to the standard Go duration format "8h0m0s" for consistency with existing configurations in the codebase.

Return Value

  • Success: Created PR URL and debugging instructions
  • Error: Error message with suggestions for resolution
  • Format: Text output with emoji indicators for status

Examples

Example 1: Without Timeout (Default 3h)

/ci:add-debug-wait aws-ipi-f7-longduration-workload

Prompts for: OCP version (4.21), repo path

Result:

test:
- ref: wait
- chain: openshift-e2e-test-qe

Returns: PR URL

Example 2: With Custom Timeout (Job Config)

/ci:add-debug-wait aws-ipi-f7-longduration-workload 8h

Prompts for: OCP version (4.21), repo path

Result (adds TIMEOUT env var + ref: wait):

env:
  BASE_DOMAIN: qe.devcluster.openshift.com
  TIMEOUT: +8 hours
test:
- ref: wait
- chain: openshift-e2e-test-qe

Returns: PR URL

Example 3: With Custom Timeout (Workflow File)

/ci:add-debug-wait baremetalds-two-node-arbiter-upgrade 24h

Behavior: Searches job config first, falls back to workflow if not found. Warns that workflow changes affect ALL jobs using it.

Result (timeout/best_effort on ref is valid in workflow files):

test:
- ref: wait
  timeout: 24h0m0s
  best_effort: true
- chain: baremetalds-ipi-test

Returns: PR URL

Example 4: With Step Removal (Job Config)

/ci:add-debug-wait tempo-upstream-tests 8h

User selects to remove - ref: install-operators and - ref: distributed-tracing-tests-tempo-upstream. The - as: install step (operator bundle install) is kept since the user did not select it for removal.

Result (removes selected refs + their orphaned env vars OPERATORS and SKIP_TESTS):

env:
  BASE_DOMAIN: devobscluster.devcluster.openshift.com
  TIMEOUT: +8 hours
test:
- as: install
  cli: latest
  commands: |
    oc create namespace openshift-tempo-operator
    ...
- ref: wait

Returns: PR URL

Arguments

  • $1 (workflow-or-job-name): The name of the CI workflow or job to add the wait step to (required)
  • $2 (timeout): Optional timeout in hours (1h-72h). Examples: "1h", "8h", "24h", "72h". If not provided, uses wait step's default (3h)

Notes

Best Practices for QE

Before Running Command:

  • ✅ Confirm test is actually failing
  • ✅ Check existing debug PRs
  • ✅ Know which OCP version is affected

During Debugging:

  • 📝 Take detailed notes
  • 💾 Save logs and screenshots
  • 🔍 Document root cause
  • 📊 Record all findings

After Debugging:

  • ✅ Document findings
  • ✅ Close the debug PR
  • ✅ Delete the branch
  • ✅ Share learnings with team
  • ✅ Create fix PR if needed

Future Enhancements

Add companion commands:

  • /ci:close-debug-pr - Lists open debug PRs, prompts for findings, closes PR
  • /ci:list-debug-prs - Show all open debug PRs
  • /ci:revert-debug-pr - Revert a debug PR that was merged by mistake

Name

ci:analyze-pr-reverts

Synopsis

/ci:analyze-pr-reverts [limit]

Description

The ci:analyze-pr-reverts command fetches recently reverted pull requests from the Sippy API and analyzes them to identify common failure patterns, the most affected repositories, and recommends concrete preventive measures such as new presubmit jobs, org-wide CodeRabbit review rules, or process changes.

This command is intended to be run periodically (e.g., monthly) to track revert trends over time.

Implementation

  1. Fetch Revert PRs from Sippy API: Query the Sippy API for PRs with "revert" in the title.

    curl -s 'https://sippy.dptools.openshift.org/api/pull_requests?release=Presubmits&filter=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22title%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22revert%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&sortField=merged_at&sort=desc&limit=<LIMIT>'
    
    • The default limit is 50 but can be overridden by the user's argument
    • The API returns JSON with fields including: merged_at, link, author, title
    • Parse the JSON response and extract all PR entries
  2. Deduplicate: Remove duplicate entries (same PR URL appearing multiple times).

  3. Scrape PR Details: For each unique PR URL, fetch details using gh:

    gh pr view <url> --json title,body,files,labels
    

    Extract from each PR:

    • Repository: Parse org/repo from the URL
    • Description: The PR body explaining why the revert was needed
    • Files changed: What areas of code were affected
    • Root cause category: Classify based on the description (see categories below)

    If a PR cannot be fetched (permissions, deleted, etc.), note it and continue.

  4. Categorize Each Revert: Classify each revert into one or more of these root cause categories based on the PR description and the original PR that was reverted:

    • Test/platform coverage gap: New or migrated tests that fail on untested platforms (microshift, single-node, IPv6, FIPS, real-time, upgrade jobs). Look for mentions of specific failing job profiles.
    • IPv6/network incompatibility: Changes that work on IPv4 but break IPv6/dual-stack clusters. Look for hardcoded IPv4 addresses, network bind issues.
    • Operator/controller regression: Core operator logic changes causing CrashLoopBackOff, degraded states, install failures, or functional breakage.
    • Build/CI tooling: Renovate configs, dependency automation, build pipeline changes that break CI.
    • API/validation side effects: API changes, CRD validation changes, or policy enforcement that has unexpected downstream impact.
    • Feature flag/config change: Premature feature gate removal, default value changes, config toggles.
    • Cross-component dependency: Change that depends on another component's change that hasn't landed or built yet.
    • Flaky/unstable new test: New test that passes sometimes but flakes at a high enough rate to block payloads.
  5. Generate Statistics:

    • Reverts by repository: Count and rank repos by number of reverts
    • Reverts by category: Count and rank root cause categories
    • Repeat offenders: PRs that were reverted, re-merged, and reverted again
    • Most common failing job profiles: Which CI job profiles appear most in revert descriptions (e.g., e2e-metal-ipi-ovn-ipv6, e2e-aws-ovn-microshift)
    • Reverts by author: Identify if certain authors/teams have recurring patterns
    • TRT-initiated vs. author-initiated: How many were quick-reverts by TRT (Revertomatic) vs. self-reverts
  6. Generate Preventive Recommendations: Based on the patterns found, produce actionable recommendations in three areas:

    a. New Presubmit Jobs / CI Coverage

    For each pattern of reverts caused by missing platform coverage, recommend specific presubmit or payload jobs that should be added. Common recommendations include:

    • If IPv6 reverts are frequent: Recommend repos add /payload-job e2e-metal-ipi-ovn-ipv6 to presubmit or require it before merge
    • If microshift reverts are frequent: Recommend repos add microshift conformance jobs
    • If single-node reverts are frequent: Recommend single-node upgrade jobs
    • If upgrade reverts are frequent: Recommend upgrade-from-previous-version jobs
    • If FIPS reverts are frequent: Recommend FIPS-specific jobs

    Be specific about which repos need which jobs based on the data.

    b. CodeRabbit Review Rules

    Recommend new review rules for the openshift/coderabbit repository's .coderabbit.yaml configuration. The coderabbit repo contains org-wide review guidelines that apply to all OpenShift repositories.

    The existing .coderabbit.yaml already has checks for:

    • Stable/deterministic test names
    • Test structure and quality
    • MicroShift compatibility

    Based on the revert patterns, recommend NEW pre_merge_checks.custom_checks entries. For example:

    • IPv6 compatibility check: Flag hardcoded IPv4 addresses (0.0.0.0, 127.0.0.1) in network-related code, recommend dual-stack testing
    • Multi-node assumptions check: Flag tests that assume multiple nodes without checking node count
    • Feature gate removal check: Flag removal of feature gates without verifying downstream consumers
    • Cross-component dependency check: Flag changes that import or depend on unreleased changes in other repos
    • Upgrade compatibility check: Flag changes to operator behavior that could break upgrade paths

    For each recommendation, provide the complete YAML block that could be added to .coderabbit.yaml in the same format as the existing checks (name, mode, instructions).

    c. Process / Documentation Recommendations

    • Recommend documentation updates, team process changes, or other non-automated improvements
    • Identify if specific teams need targeted guidance
    • Suggest improvements to the revert process itself if patterns indicate issues
  7. Format the Report: Output a well-structured markdown report with the following sections:

    ## PR Revert Analysis Report
    ### Date Range: [earliest] to [latest merged_at]
    ### Summary
    ### Reverts by Repository
    ### Reverts by Root Cause Category
    ### Repeat Reverts (PRs reverted more than once)
    ### Most Common Failing Job Profiles
    ### Detailed Revert List (table with: repo, PR link, category, brief reason)
    ### Recommendations
    #### New Presubmit Jobs
    #### New CodeRabbit Review Rules
    #### Process Improvements
    

    All PR links in the report MUST use full GitHub URLs (e.g., https://github.com/openshift/machine-config-operator/pull/5703), never shorthand references like #5703 which would resolve relative to whatever repo the reader is viewing.

Return Value

  • Markdown report: Full analysis report as described above
  • Recommended CodeRabbit YAML: Ready-to-use YAML blocks for .coderabbit.yaml
  • Summary statistics: Key numbers (total reverts, top repo, top category)

Examples

  1. Basic usage (fetches last 50 revert PRs):

    /ci:analyze-pr-reverts
    
  2. With custom limit:

    /ci:analyze-pr-reverts 100
    

Arguments

  • $1 (optional): Maximum number of revert PRs to fetch from the Sippy API. Defaults to 50.

Prerequisites

  1. GitHub CLI (gh): Must be installed and authenticated

    • Check: gh auth status
    • Needed to fetch PR details (description, files, labels) for each revert
  2. Network access: Must be able to reach sippy.dptools.openshift.org

Notes

  • Some PRs may not be fetchable via gh (private repos, deleted PRs, permission issues). These are noted in the report but don't block analysis.
  • The query captures PRs with "revert" or "Revert" in the title. Some legitimate non-revert PRs may be included (e.g., "unrevert" PRs that restore previously reverted changes). Filter these out during analysis.
  • PRs from non-openshift orgs (e.g., Azure/ARO-HCP) may have limited gh access. Include them in stats but note if details couldn't be fetched.
  • The recommended CodeRabbit rules should be conservative (mode: warning not error) to avoid blocking legitimate PRs while the rules are tuned.

See Also

Name

ci:analyze-prow-job-resource

Synopsis

Generate a report showing the lifecycle of a single resource:

/ci:analyze-prow-job-resource <prowjob-url> [namespace:][kind/][resource-name]

Generate a report showing multiple resource lifecycles:

/ci:analyze-prow-job-resource <prowjob-url> [namespace:][kind/][resource-name],[namespace:][kind/][resource-name],...

Description

Analyze the lifecycle of Kubernetes resource(s) in a Prow CI job by invoking the "prow-job-analyze-resource" skill.

Implementation

Pass the user's request to the skill, which will:

  • Download Prow job artifacts from Google Cloud Storage
  • Parse audit logs and pod logs
  • Generate an interactive HTML report with timeline visualization

The skill handles all the implementation details including URL parsing, artifact downloading, log parsing, and HTML report generation.

Name

ci:analyze-regression

Synopsis

/ci:analyze-regression <regression id>

Description

The ci:analyze-regression command analyzes details for a specific Component Readiness regression and suggests next steps for investigation.

The command performs a full analysis regardless of whether the regression has been triaged. For triaged regressions, it also fetches the linked JIRA issue to analyze whether someone is actively working on the fix or if the issue needs attention.

This command is useful for:

  • Understanding regression patterns and failure modes
  • Checking if a triaged regression is being actively worked on or needs attention
  • Identifying related regressions that might be caused by the same issue
  • Getting pointers on where to investigate next

Implementation

Important: Avoiding user permission prompts when running scripts

When calling Python skill scripts via the Bash tool, always run the script directly without piping the output through inline Python (python3 -c "..."). Complex piped commands trigger user permission prompts, while simple python3 script.py args calls are auto-approved.

  • Do: python3 script.py args --format json 2>/dev/null — run the script directly and process the JSON output in your reasoning
  • Don't: python3 script.py args | python3 -c "import json; ..." — piped inline Python triggers permission prompts

Parse and analyze the JSON output from scripts using your own reasoning capabilities rather than shell pipelines.

Obtaining the DPCR authentication token from mounted kubeconfig

The triage and bug filing steps (step 14) require a Bearer token from the DPCR cluster (api.cr.j7t7.p1.openshiftapps.com:6443). When running in a container with ~/.kube mounted (read-only), the token is extracted directly from the mounted kubeconfig using oc:

# Find the oc context for the DPCR cluster from the mounted kubeconfig
DPCR_CONTEXT=$(oc config get-contexts -o name 2>/dev/null | while read -r ctx; do
  server=$(oc config view -o jsonpath="{.clusters[?(@.name=='$(oc config view -o jsonpath="{.contexts[?(@.name=='$ctx')].context.cluster}" 2>/dev/null)')].cluster.server}" 2>/dev/null || echo "")
  server_clean=$(echo "$server" | sed -E 's|^https?://||')
  if [ "$server_clean" = "api.cr.j7t7.p1.openshiftapps.com:6443" ]; then
    echo "$ctx"
    break
  fi
done)

# Extract the token from the DPCR context
if [ -z "$DPCR_CONTEXT" ]; then
  echo "ERROR: Could not find a DPCR cluster context in kubeconfig. Set DPCR_CONTEXT manually or run: oc login https://api.cr.j7t7.p1.openshiftapps.com:6443"
  exit 1
fi
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT" 2>/dev/null)

This works because oc reads from ~/.kube/config which is bind-mounted from the host. The token stored in the kubeconfig was obtained when the user previously ran oc login to the DPCR cluster on the host. If the token is expired, instruct the user to re-authenticate on the host: oc login https://api.cr.j7t7.p1.openshiftapps.com:6443.

  1. Load CI Context: Read all documentation files in plugins/ci/references/ for context on tests, jobs, and CI conventions. These contain important notes on specific test frameworks, job ownership, and debugging guidance that should inform the analysis.

    ls plugins/ci/references/
    

    Read each file found. Keep this context in mind throughout the analysis — it may affect how you interpret failure patterns, who to recommend contacting, or what the root cause is likely to be.

  2. Parse Arguments: Extract regression ID

    • Regression ID format: Integer ID from Component Readiness
    • Example: 34446
  3. Fetch Regression Details: Use the fetch-regression-details skill

    Run the Python script to fetch comprehensive regression data:

    script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"
    regression_data=$(python3 "$script_path" <regression_id> --format json)
    

    The skill automatically:

    Extract the following from the JSON response:

    • test_name: Full test name
    • release and base_release: Version information
    • component and capability: Ownership mapping
    • variants: Platform/topology combinations where test is failing
    • opened and closed: Regression timeline
    • triages: Existing JIRA tickets
    • analysis_status: Integer status code (negative indicates problems, -1000 indicates failed fix)
    • analysis_explanations: Human-readable explanations for the status
    • test_details_url: Link to Sippy test details (API URL - must be converted to UI URL before displaying, see note below)
    • sample_failed_jobs: Dictionary keyed by job name, each containing:
      • pass_sequence: Chronological S/F pattern (newest to oldest)
      • failed_runs: List of failed runs with job_url, job_run_id, start_time
    • job_runs: Complete list of all job runs where the failure was observed throughout the regression's entire life (not just the last reporting period). Each entry contains:
      • prowjob_run_id, prowjob_name, prowjob_url: Job identification and link
      • start_time: When the job ran
      • test_failures: Total number of test failures in the run. High values (e.g., >10) indicate mass failure runs where the regressed test may just be caught up in a larger issue rather than being the primary problem

    Converting test_details_url to UI URL: The test_details_url from the API is an API endpoint not suitable for display or bug reports. Convert it to the UI URL by replacing the base path. The query parameters are identical:

    # Convert API URL to UI URL
    test_details_ui_url=$(echo "$test_details_url" | sed 's|https://sippy.dptools.openshift.org/api/component_readiness/test_details|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details|')
    

    Always use the converted test_details_ui_url when displaying the link in the report or including it in bug descriptions.

    See plugins/ci/skills/fetch-regression-details/SKILL.md for complete implementation details.

  4. Fetch Global Test Report: Use the fetch-test-report skill to check how this test is doing globally

    Use the test name and release from the regression data to fetch the global test report with per-variant breakdown:

    # Extract test name and release from regression data
    test_name=$(echo "$regression_data" | jq -r '.test_name')
    release=$(echo "$regression_data" | jq -r '.release')
    
    # Fetch per-variant breakdown to see which job types are affected
    script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"
    test_report=$(python3 "$script_path" "$test_name" --release "$release" --no-collapse --format json)
    

    Analyze the test report:

    • Global pass rate: Check the overall pass rate across all variants. A low pass rate confirms a widespread issue; a high pass rate with the regression suggests the problem is variant-specific.
    • Per-variant breakdown: With --no-collapse, each row shows a specific variant combo (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]). Compare pass rates across variants to identify if the failure is platform-specific, network-specific, or upgrade-specific.
    • Open bugs: If open_bugs > 0, someone has already filed a Jira bug mentioning this test. This bug may not yet be triaged in Component Readiness. Note these bugs in the report — they could be used to triage the regression without filing a duplicate.
    • Trend: Check net_working_improvement — positive means improving, negative means getting worse.

    Include the global test report findings in Section 1 of the final report.

  5. Check Triage Status and Fetch JIRA Progress: Determine triage state and analyze bug progress

    Check if the regression has already been triaged and fetch JIRA details:

    # Check if regression has triages
    triage_count=$(echo "$regression_data" | jq '.triages | length')
    

    If triage_count > 0 (regression is triaged):

    This means a human has already attributed this regression to a specific bug. For each triage entry, fetch the JIRA issue to analyze progress. Also check if step 4's test report found open bugs that may be related.

    # Check if JIRA auth environment variables are set
    if [ -z "$JIRA_USERNAME" ] || [ -z "$JIRA_API_TOKEN" ]; then
      echo "Warning: JIRA_USERNAME and/or JIRA_API_TOKEN not set. Skipping JIRA progress analysis."
    else
      # For each triage, fetch JIRA details using the fetch-jira-issue skill
      jira_script="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py"
      for jira_key in $(echo "$regression_data" | jq -r '.triages[].jira_key'); do
        jira_data=$(python3 "$jira_script" "$jira_key" --format json)
    
        # The fetch-jira-issue skill automatically classifies progress. Extract the classification:
        progress_level=$(echo "$jira_data" | jq -r '.progress.level')
        progress_reason=$(echo "$jira_data" | jq -r '.progress.reason')
        assignee=$(echo "$jira_data" | jq -r '.assignee.display_name // "Unassigned"')
        linked_prs=$(echo "$jira_data" | jq -r '.linked_prs | length')
      done
    fi
    

    See plugins/ci/skills/fetch-jira-issue/SKILL.md for complete implementation details.

    Progress levels returned by the skill:

    • ACTIVE: Status is ASSIGNED/IN PROGRESS with recent activity, PRs linked, or recent comments
    • STALLED: Status is ASSIGNED but no activity in 14+ days
    • NEEDS_ATTENTION: Status is NEW/OPEN with no assignee or no recent progress
    • RESOLVED: Status is Closed/Verified

    Classification Output:

    JIRA Progress Analysis:
    - OCPBUGS-12345: 🟢 ACTIVE - Assigned to user@redhat.com, PR in review (2 days ago)
    - OCPBUGS-12345: 🟡 STALLED - Assigned but no activity in 21 days
    - OCPBUGS-12345: 🔴 NEEDS ATTENTION - Status NEW, no assignee, no comments
    

    Note for Failed Fixes (analysis_status == -1000):

    • This indicates the test is still failing AFTER the triage was resolved
    • Flag: "⚠️ FAILED FIX: Test continues to fail after triage was resolved"
    • Recommend: Re-opening the bug or filing a new one

    If triage_count == 0 (regression is NOT triaged):

    • No bug has been filed yet
    • If step 4's test report found open_bugs > 0, list these — someone may have filed a bug that hasn't been triaged yet
    • Continue with full investigation (steps 6-11)
    • Bug filing recommendations will be provided in step 12

    Always continue to step 6 regardless of triage status to provide full analysis.

  6. Interpret Regression Data: Analyze failure patterns from sample_failed_jobs

    Parse the sample_failed_jobs dictionary from the regression data:

    # Extract and analyze each job
    echo "$regression_data" | jq -r '.sample_failed_jobs | to_entries | .[] | "\(.key)|\(.value.pass_sequence)|\(.value.failed_runs | length)"'
    
    • Identify Jobs with Most Failures:

      • Iterate through the sample_failed_jobs dictionary keys (job names)
      • Count failed_runs array length for each job: echo "$regression_data" | jq '.sample_failed_jobs["job-name"].failed_runs | length'
      • Sort jobs by failure count (descending) to identify the most impacted jobs
      • Example: Job A has 18 failures, Job B has 1 failure → Job A is the primary concern
    • Analyze Pass Sequence Patterns: For each job, examine the pass_sequence string.

      CRITICAL: Reading Direction

      • The pass_sequence is ordered newest to oldest (left = most recent, right = oldest)
      • First character = most recent job run
      • Last character = oldest job run
      • Example: FFFSSSSSSS means: 3 most recent runs failed, 7 older runs passed

      Pattern 1: Permafailing Test

      • pass_sequence starts with many "F"s at the LEFT (e.g., FFFFFFFFFF or FFFFFFFSS)
      • The LEFT side (most recent runs) shows consistent failures
      • Example: FFFFFFFFFFFFFFFFFF - leftmost 18 characters are all F = 18 most recent runs all failed
      • Interpretation: Test is currently broken and consistently failing
      • Action: High priority - test is completely broken for this job variant

      Pattern 2: Resolved Issue

      • pass_sequence starts with "S"s at the LEFT, with "F"s toward the RIGHT (e.g., SSSSSFFFFF or SSSSSFFFFFSS)
      • The LEFT side (most recent runs) shows successes, RIGHT side (older runs) shows failures
      • Example: SSSSFFFFFFFFFFFF - leftmost 4 chars are S (recent passes), rightward chars are F (older failures)
      • Example: SSSSSSSSSSSSSSSSSSFFSFFF - many S's on left (recent passes), F's on right (older failures)
      • Interpretation: The problem existed in older runs but has been resolved in recent runs
      • Action: Lower priority - verify if issue is truly resolved or if more monitoring is needed

      Pattern 3: Flaky Test

      • pass_sequence shows "F"s and "S"s interspersed throughout (e.g., SFSFSFSFSF or FSSFFSSFF)
      • No consistent block of failures - failures are scattered across the sequence
      • Example: SSFSSFSSFSSF shows intermittent failures mixed with successes
      • Interpretation: Test appears flaky rather than consistently failing
      • Action: May require test stabilization or flake investigation rather than product bug

      Pattern 4: Recent Regression

      • pass_sequence starts with "F"s at the LEFT, followed by "S"s toward the RIGHT (e.g., FFSSSSSSSSS)
      • The LEFT side (most recent runs) shows new failures, RIGHT side (older runs) shows the test was passing
      • Example: FFFFFSSSSSSSSSS - leftmost 5 chars are F (recent failures), rightward chars are S (older successes)
      • Interpretation: Test was passing historically but has recently started failing
      • Action: High priority - recent regression, investigate recent code changes
    • Generate Pattern Summary: Create a summary for each job:

      • Job name
      • Total failed runs
      • Pass sequence pattern classification (permafail / resolved / flaky / recent regression)
      • Recommended priority level
      • Suggested next steps

    Example Analysis:

    Given this sample_failed_jobs structure:

    {
      "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv4-rhcos10-techpreview": {
        "pass_sequence": "FFFFFFFFFFFFFFFFFF",
        "failed_runs": [/* 18 failed runs */]
      },
      "periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-techpreview": {
        "pass_sequence": "SSFSSSSSSSS",
        "failed_runs": [/* 1 failed run */]
      },
      "periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-example": {
        "pass_sequence": "SSSSSSSSSSSSSSSSSSFFSFFF",
        "failed_runs": [/* 5 failed runs */]
      }
    }
    

    Remember: LEFT = newest, RIGHT = oldest

    Pattern analysis output:

    • Job 1 (18 failures): FFFFFFFFFFFFFFFFFF
      • Reading: All 18 characters are F, starting from the LEFT (newest)
      • Classification: Permafail - most recent runs are all failing
      • Priority: High
      • Action: "Test is completely broken for this metal+ovn+ipv4+rhcos10 variant. Investigate immediately."
    • Job 2 (1 failure): SSFSSSSSSSS
      • Reading: LEFT (newest) starts with SS, then one F in position 3, rest are S
      • Classification: Flaky - isolated failure in mostly passing runs
      • Priority: Low
      • Action: "Single recent failure in mostly passing job. Monitor for pattern or investigate if recurring."
    • Job 3 (5 failures): SSSSSSSSSSSSSSSSSSFFSFFF
      • Reading: LEFT (newest) has 18 S's = most recent 18 runs passed; RIGHT (oldest) has FFSFFF = older runs had failures
      • Classification: Resolved - failures were in OLDER runs, recent runs are passing
      • Priority: Low
      • Action: "Issue appears to have been resolved. The failures occurred in older runs, not recent ones."
  7. Analyze Failure Output Consistency: Use the fetch-test-runs skill

    Use the fetch-test-runs skill to fetch and analyze actual test failure outputs from all failed job runs. This helps determine if all failures have the same root cause or if there are multiple issues.

    Implementation:

    # Extract test_id from regression data (already fetched in step 3)
    test_id=$(echo "$regression_data" | jq -r '.test_id')
    
    # Collect all job_run_ids from sample_failed_jobs
    # This creates a comma-separated list of all failed job run IDs across all jobs
    job_run_ids=$(echo "$regression_data" | jq -r '
      .sample_failed_jobs
      | to_entries[]
      | .value.failed_runs[]
      | .job_run_id
    ' | tr '\n' ',' | sed 's/,$//')
    
    # Use the fetch-test-runs skill
    script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
    output_analysis=$(python3 "$script_path" "$test_id" "$job_run_ids" --format json)
    

    The skill fetches raw test failure outputs from Sippy API.

    See plugins/ci/skills/fetch-test-runs/SKILL.md for complete implementation details.

    Parse Results and Analyze with AI:

    # Check if fetch was successful
    success=$(echo "$output_analysis" | jq -r '.success')
    
    if [ "$success" = "true" ]; then
      # Extract outputs array
      outputs=$(echo "$output_analysis" | jq -r '.runs')
      num_outputs=$(echo "$outputs" | jq 'length')
    
      echo "Fetched $num_outputs test failure runs"
    
      # AI ANALYSIS: Compare the runs for similarity
      # Examine each output message to determine:
      # 1. How many failures show the same or very similar error messages
      # 2. What percentage of failures are consistent
      # 3. What the common error pattern is (if any)
      # 4. Extract file references, API paths, or resource names from error messages
      #
      # The runs are in format: {"url": "...", "output": "error text", "test_name": "..."}
      #
      # Classify consistency:
      # - Highly Consistent (>90%): All/nearly all show same error -> single root cause
      # - Moderately Consistent (50-90%): Most share patterns -> primary issue with variation
      # - Inconsistent (<50%): Different errors -> multiple causes or environmental issues
      #
      # Extract from error messages:
      # - File/line references (e.g., "discovery.go:145")
      # - API/resource paths (e.g., "/apis/stable.e2e-validating-admission-policy-1181/")
      # - Common error phrases (e.g., "server could not find the requested resource")
    
    else
      # API not available - this is acceptable, continue without output analysis
      error=$(echo "$output_analysis" | jq -r '.error')
      echo "Note: Test output analysis unavailable - $error"
      echo "Continuing with other analysis steps..."
    fi
    

    How to Analyze Outputs with AI:

    Read through the failure outputs and identify patterns:

    1. Compare Error Messages: Count how many outputs have identical or very similar messages

      • Example: If 17 out of 18 say "the server could not find the requested resource", that's 94% consistency
    2. Extract Common Elements: Look for shared components in the error messages

      • File references: "k8s.io/kubernetes/test/e2e/apimachinery/discovery.go:145"
      • API paths: "/apis/stable.e2e-validating-admission-policy-1181/"
      • Error phrases: "server could not find the requested resource"
    3. Classify Consistency:

      • Highly Consistent (>90%): Single root cause - all failures show same error
      • Moderately Consistent (50-90%): Primary issue - most share patterns with some variation
      • Inconsistent (<50%): Multiple causes - failures show different error types
    4. Determine Root Cause: Based on the common error message and extracted information, infer what the underlying issue likely is

      • Example: "API endpoint not available" if errors mention missing API resources
      • Example: "Timeout issue" if errors mention timeouts or waiting conditions

7a. Analyze Job Run Context: Use the fetch-job-run-summary skill

For each failed job run, the regressed test is just one of potentially many test results. Understanding the broader job run context — how many other tests failed, which ones, and whether they share error patterns — is critical for diagnosing the root cause.

  • If the regressed test is the only failure (or one of very few) in each run, it is likely a targeted issue specific to that test or its component.
  • If the regressed test fails alongside a consistent set of other tests, those co-failures point to a shared root cause (e.g., an operator failing that breaks multiple dependent tests).
  • If the regressed test is caught up in mass failures with different tests failing each time, it may be collateral damage from infrastructure instability or a foundational issue rather than a specific regression in the test's component.

This is especially important for mass test failure regressions ([Jira:"Test Framework"] there should not be mass test failures), where the fetch-test-runs output from step 7 only shows the mass failure count breakdown and the actual failing tests must be examined to find the real root cause.

Implementation:

Select a representative sample of failed job runs to analyze. Pick up to 3 runs from the sample_failed_jobs data — if failure counts vary, choose runs with different severities (e.g., the run with the most failures, one with a moderate count, and one with the fewest):

script_path="plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py"

# For each selected job run ID from the sample_failed_jobs data:
summary=$(python3 "$script_path" <job_run_id> --format json)

See plugins/ci/skills/fetch-job-run-summary/SKILL.md for complete implementation details.

Analyze the results across runs:

  1. Assess isolation vs. co-failure: For each run, check failure_count. If the regressed test is the only failure (or one of 2-3), the issue is isolated to this test. If it fails alongside 10+ other tests, it may be a symptom of a broader problem.

  2. Compare failed test lists across runs: Check whether the same set of other tests co-fail with the regressed test. Consistent co-failures suggest a shared root cause. Random co-failures suggest environmental issues.

  3. Check dominant error patterns: The skill automatically detects error messages appearing in >5% of failures. A single dominant error across many tests (e.g., "stale GroupVersion discovery" at 94%) indicates one root cause cascading across many tests, and the regressed test is likely collateral damage. If the regressed test has a unique error distinct from other failures, it may be an independent issue.

  4. Assess failure scale consistency: Compare failure_count across runs. If counts vary dramatically (e.g., 12 in one run, 397 in another), the issue may be intermittent or timing-dependent. Consistent failure counts suggest a deterministic problem.

  5. Identify the true failing component (especially for mass test failure regressions): The mass test failure regression is attributed to "Test Framework" in Component Readiness, but the actual product issue is in whatever component's tests are failing. Use the failed test names to identify the real affected component(s).

Classification:

  • Isolated failure: The regressed test is the only failure or one of very few in each run. The issue is specific to this test's component. File the bug against that component.
  • Consistent co-failures: The same set of tests fail together across runs, with a dominant error pattern. This points to a specific product bug affecting multiple tests. The bug should be filed against the component responsible for the shared failure mode.
  • Inconsistent/random co-failures: The regressed test fails in runs with many other failures, but the set of co-failing tests varies. This suggests infrastructure instability, cluster health issues, or a foundational problem (e.g., API server instability) causing cascading and unpredictable failures.
  • Scaling pattern: A consistent core set of failures plus a variable number of additional failures. The core failures are the root cause; the additional failures are secondary effects.

Output for Report: Include in the report:

  • Number of runs analyzed and their failure counts
  • Whether the regressed test is an isolated failure or part of broader co-failures
  • Whether the same tests consistently co-fail across runs
  • The dominant error pattern (if any) with percentage
  • For mass test failure regressions: the real affected component(s) based on failing test names
  • Assessment: isolated test issue, targeted product bug with co-failures, or collateral damage from widespread instability

7b. Deep Install Failure Analysis (install should succeed regressions only)

Only perform this step if the test name contains "install should succeed".

Install regressions require deeper per-run analysis because different runs may fail at entirely different installation stages (bootstrap, infrastructure, cluster creation, etc.) for different reasons. This step invokes the ci:prow-job-analysis skill on a representative sample of failed runs to determine whether all failures share a single root cause or have multiple independent causes.

Select runs to analyze:

Select up to 5 failed job runs to analyze. Choose from the jobs with the most failures (identified in step 6). If failure counts vary across jobs, spread the selection across jobs to get a representative sample. Extract job_url values from sample_failed_jobs:

# Example: extract up to 5 job URLs from sample_failed_jobs (one per job, preferring high-failure jobs)
echo "$regression_data" | jq -r '
  .sample_failed_jobs
  | to_entries
  | sort_by(.value.failed_runs | length)
  | reverse
  | .[0:5]
  | .[]
  | .value.failed_runs[0].job_url
'

Invoke the install failure skill for each selected run:

For each selected job_url, use the Skill tool to invoke ci:prow-job-analysis with the job URL. The skill downloads and analyzes GCS artifacts (installer logs, log bundles, junit XML) and returns a structured report including:

  • Failure stage (bootstrap, infrastructure, cluster creation, cluster operator stability, configuration, other)
  • Root cause summary
  • Key error messages

Synthesize findings across runs:

After analyzing all runs, compare the failure stages and root causes:

  • Consistent — single root cause: All runs fail at the same stage with the same error pattern (e.g., all fail at cluster bootstrap with etcd not forming). High confidence this is one bug.
  • Consistent stage, multiple root causes: All runs fail at the same stage but with different error messages (e.g., all infrastructure failures but different cloud API errors). May be a single unstable area with varied surface failures.
  • Multiple causes: Runs fail at different stages (e.g., some bootstrap, some cluster creation). Likely multiple independent issues or an intermittently unstable environment. Each failure mode may need separate investigation.
  • Inconclusive: Skill could not retrieve artifacts or determine root cause for most runs. Note this and continue.

Output for report: Include in Section 4b:

  • Number of runs analyzed
  • Failure stage breakdown (e.g., "2/3 runs: cluster bootstrap, 1/3 runs: infrastructure")
  • Root cause consistency classification
  • Per-run summary: job URL, failure stage, key error message
  • Overall assessment and recommended next steps
  1. Determine Regression Start Date: Analyze job_runs data from the regression

    The regression data from step 3 includes a job_runs array containing all job runs where the failure was observed throughout the entire life of the regression. Use this data to determine when the regression started without needing a separate API call.

    Implementation:

    # Extract job_runs from regression data (already fetched in step 3)
    # job_runs are sorted newest-first; reverse to walk oldest-first for start date analysis
    job_runs=$(echo "$regression_data" | jq '.job_runs')
    num_job_runs=$(echo "$job_runs" | jq 'length')
    

    How to Determine Regression Start Date from job_runs:

    1. Walk from oldest to newest: The job_runs array is sorted newest-first (index 0 = most recent). Reverse the order conceptually or iterate from the end to find the earliest entries.

    2. Find the first failure: The earliest job_run entry (last in the array) represents the earliest observed occurrence of this failure. Its start_time gives the approximate regression start date and prowjob_url links to that first failing run.

    3. Analyze the pattern: Walk through the job_runs chronologically to understand:

      • How quickly failures ramped up after the first occurrence
      • Whether failures are concentrated in specific jobs (prowjob_name)
      • Whether test_failures counts are high in many runs — high values (e.g., >10) indicate mass failure runs where the regressed test may just be caught up in a larger issue (infrastructure instability, a foundational bug, etc.) rather than being the primary problem. If most job_runs show high test_failures, flag this as a potential mass failure regression.
    4. Identify the most affected jobs: Group job_runs by prowjob_name and count occurrences per job. This complements the sample_failed_jobs data from step 6 with full historical coverage.

    5. Handle Edge Cases:

      • If job_runs spans the entire tracked period with no gap → the regression may predate the tracked data; note this
      • If runs are scattered sparsely → likely a flaky test, not a clear regression start
      • If pattern is unclear → do not include this in the report
    6. Report the Findings (only if a clear start date can be determined):

      • Report the prowjob_url of the first failure
      • Report the start_time date when that run occurred
      • Report the total number of runs tracked
      • Flag if most runs have high test_failures counts (mass failure pattern)
      • Example: "Regression appears to have started on 2026-01-15 with job run: https://prow.ci.openshift.org/view/gs/..."

    Fallback: If job_runs is empty or missing (e.g., regression predates this feature), fall back to using the fetch-test-runs skill with --include-success as before:

    most_failed_job=$(echo "$regression_data" | jq -r '
      .sample_failed_jobs
      | to_entries
      | sort_by(.value.failed_runs | length)
      | reverse
      | .[0].key
    ')
    
    script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
    job_history=$(python3 "$script_path" "$test_id" --include-success --job-contains "$most_failed_job" --start-days-ago 28 --exclude-output --format json)
    

    Output Format (only include if start date can be determined):

    Regression Start Analysis:
    - First Failing Run: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...
    - Approximate Start Date: 2026-01-15
    - Total Tracked Runs: 47
    - Most Affected Job: periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn (18 occurrences)
    - Mass Failure Pattern: 35/47 runs had >10 test failures — regression may be collateral damage from a larger issue
    - Pattern: Failures began 2026-01-15, became consistent by 2026-01-17
    

    Note: If no clear start date can be determined, skip this section entirely. Do not include inconclusive results.

  2. Identify Suspect PRs in Payload: Use fetch-prowjob-json and fetch-new-prs-in-payload skills

    Only perform this step if step 8 successfully identified a clear regression start point (i.e., a first failing run URL exists). If step 8 was skipped or inconclusive, skip this step entirely.

    This step identifies pull requests that may have caused the regression by examining what was new in the payload where failures began.

    Step 9a: Get the payload tag from the first failing run

    Use the fetch-prowjob-json skill to fetch the prowjob.json for the first failing run identified in step 8.

    # The first_failing_run_url comes from step 8
    # Use the fetch-prowjob-json skill to convert to gcsweb URL and fetch
    # See plugins/ci/skills/fetch-prowjob-json/SKILL.md for URL conversion details
    #
    # Extract these annotations:
    payload_tag=$( ... )       # metadata.annotations["release.openshift.io/tag"]
    from_tag=$( ... )          # metadata.annotations["release.openshift.io/from-tag"] (may not exist)
    
    • payload_tag: The payload the cluster was tested against (or upgraded to for upgrade jobs)
    • from_tag: The original version before upgrade (only present on upgrade jobs). If present, note this in the report as it helps distinguish whether a regression is in the upgrade path itself vs the target version.

    If the prowjob.json cannot be fetched or the release.openshift.io/tag annotation is missing (e.g., manually triggered jobs), skip the rest of this step.

    Step 9b: Fetch new PRs in the payload

    script_path="plugins/ci/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py"
    pr_data=$(python3 "$script_path" "$payload_tag" --format json)
    

    See plugins/ci/skills/fetch-new-prs-in-payload/SKILL.md for complete implementation details.

    Step 9c: Identify potentially related PRs

    From the list of new PRs, identify candidates that might be related to the regression. Filter based on:

    1. Component match: PR component name matches or overlaps with the regression's component or capability
    2. Repository match: PR is from a repo related to the failing test (e.g., a test in [sig-network] and a PR from openshift/ovn-kubernetes)
    3. Title keywords: PR description contains keywords related to the test name, error messages from step 7, or the affected subsystem
    4. Bug association: PR has a bug_url referencing a fix for something in the same area

    Select a maximum of 5 candidate PRs to investigate (to avoid excessive API calls). Prioritize PRs whose component or repo most closely matches the regression.

    If no PRs look related based on filtering, state that in the report and skip step 9d.

    Step 9d: Check PR details with GitHub CLI

    For each candidate PR (up to 5), use the gh CLI to fetch the PR description and diff summary:

    # Check if gh CLI is available
    if command -v gh &>/dev/null; then
      # Extract org/repo and PR number from the PR URL
      # e.g., https://github.com/openshift/machine-config-operator/pull/5509
      # → owner=openshift, repo=machine-config-operator, pr_number=5509
    
      # Fetch PR details (title, body, changed files)
      gh pr view "$pr_number" --repo "$owner/$repo" --json title,body,files,labels
    
      # Fetch the diff summary (file names and change counts)
      gh pr diff "$pr_number" --repo "$owner/$repo" --stat
    else
      echo "Note: gh CLI not available. Showing PR URLs only - install gh for deeper analysis."
    fi
    

    Analyze each PR for relevance:

    • Read the PR description and diff to determine if the changes could plausibly affect the failing test
    • Look for changes to files, packages, or APIs referenced in the test error messages (from step 7)
    • Note if the PR modifies test infrastructure, API schemas, or operator behavior relevant to the regression
    • Classify each PR as:
      • Likely related: Changes directly affect the area where the test is failing
      • Possibly related: Changes are in a related subsystem but not directly in the failure path
      • Unlikely related: Changes appear unrelated to the test failure

    Output Format:

    Suspect PRs in Payload (payload: 4.22.0-0.ci-2026-02-06-195709):
    Upgrade From: 4.22.0-0.ci-2026-02-05-195709
    
    Investigated 3 of 17 new PRs in this payload:
    
    1. [LIKELY] openshift/machine-config-operator#5509
       "Set NodeDegraded MCN condition when node state annotation is set to Degraded"
       Bug: OCPBUGS-67229
       Relevance: Changes MCN condition logic which is tested by the failing test
       Files changed: pkg/controller/node/status.go (+45/-12)
    
    2. [POSSIBLY] openshift/hypershift#7470
       "use InfraStatus.APIPort for custom DNS kubeconfig"
       Bug: OCPBUGS-72258
       Relevance: Modifies API port handling, test involves API connectivity
    
    3. [UNLIKELY] openshift/router#707
       "Updating openshift-enterprise-haproxy-router-container image"
       Relevance: Router image update, unrelated to test failure area
    
  3. Identify Related Regressions: Use the list-regressions skill to find similar failing tests

Use the list-regressions skill with test name filtering to find related regressions that may share the same root cause. Run two queries: one for the exact same test (different variants) and one for similar tests (same namespace/sig).

Implementation:

# Extract test name and release from regression data
test_name=$(echo "$regression_data" | jq -r '.test_name')
release=$(echo "$regression_data" | jq -r '.release')
script_path="plugins/teams/skills/list-regressions/list_regressions.py"

# Query 1: Find regressions for the exact same test across all variants
same_test_regressions=$(python3 "$script_path" --release "$release" --test-name "$test_name")

# Query 2: Find regressions for similar tests (e.g., same namespace)
# Extract the namespace from the test name (e.g., "openshift-machine-config-operator")
# and search for other tests mentioning it
# Choose a distinctive substring from the test name that identifies related tests
similar_test_regressions=$(python3 "$script_path" --release "$release" --test-name-contains "<distinctive_substring>")

--test-name and --test-name-contains cannot be combined with --components or --team — they search across all components automatically.

Analyze Related Regressions:

From the filtered results, identify regressions related to the current one:

  • Same test, different variants: From the --test-name query — other regressions for the same test but with different variant combinations (e.g., same test failing on both aws and metal platforms)
  • Similar test names: From the --test-name-contains query — regressions with test names that share a common prefix or test suite (e.g., same [sig-api-machinery] tests)

For each related regression found, note:

  • Regression ID
  • Test name (if different from the current one)
  • Variants where it is failing
  • Whether it is triaged (has entries in triages array) and to which JIRA bug
  • Whether it is open or closed

For each related regression that will be included in a bug report or triage, fetch its details to get the test_details_url:

# Fetch details for a related regression to get its test_details_url
related_data=$(python3 "plugins/ci/skills/fetch-regression-details/fetch_regression_details.py" <related_regression_id> --format json)
related_test_details_url=$(echo "$related_data" | jq -r '.test_details_url')
# Convert to UI URL
related_test_details_ui_url=$(echo "$related_test_details_url" | sed 's|https://sippy.dptools.openshift.org/api/component_readiness/test_details|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details|')

Summarize variant patterns:

  • Identify if the test is failing across all jobs of one Platform variant (e.g., all metal jobs)
  • Identify if failures cluster around a specific Upgrade, Network, or Topology variant
  • Note any variant combinations that are NOT failing (helps narrow root cause)
  1. Find Related Triages and Untriaged Regressions: Use the fetch-related-triages skill

Query the Sippy API to find existing triage records and untriaged regressions related to this regression:

script_path="plugins/ci/skills/fetch-related-triages/fetch_related_triages.py"
related=$(python3 "$script_path" <regression_id> --format json)

See plugins/ci/skills/fetch-related-triages/SKILL.md for complete implementation details.

The API returns matches based on similarly named tests and shared job runs (using the complete job_runs history tracked across each regression's lifetime), each with a confidence level (1-10):

  • 10: High confidence — same or very closely related tests
  • 5: Medium confidence — similarly named tests matched by edit distance
  • 2: Low confidence — regressions sharing the same job runs (may indicate shared root cause, but could be coincidental in mass failure scenarios)

Analyze the results:

  • triaged_matches: Existing triage records that look related. For each match:

    • Note the triage_id, jira_key, jira_status, and confidence_level
    • High confidence matches (>=5) with open JIRA bugs are strong candidates for adding this regression to
    • Low confidence matches or closed JIRA bugs are informational
    • Present these in the report as potential triage targets
  • untriaged_regressions: Open regressions not yet triaged that appear related. These are candidates to be triaged together with the current regression under one bug. For each:

    • Note the match_reason (similarly_named_test or same_last_failure)
    • similarly_named_test with low edit_distance (0-2) are likely the same or very similar tests in different variants
    • same_last_failure regressions may share the same root cause (same failing job runs) but could be from different components

Combine with step 10 results: Merge these findings with the related regressions found via --test-name in step 10. The two sources are complementary:

  • Step 10 finds regressions for the exact same test name (different variants)
  • This step finds regressions with similar test names AND existing triages that may already cover the issue
  1. Prepare Bug Filing Recommendations or Existing Bug Status: Generate actionable information

Important: Component Readiness regressions are treated as release blockers. Any bug filed for a regression should be conveyed to the user as a release blocker. See the release blocker definition for details on the criteria and process.

If regression is NOT triaged (no existing JIRA):

  • Component assignment (from test mappings)
  • If step 4's test report found open_bugs > 0, note these existing bugs — one may be suitable for triaging this regression without filing a duplicate
  • Bug summary suggestion based on failure pattern (informed by step 6 pattern analysis)
  • Bug description template including:
    • Test name(s) and release — use the full test name including all tags (e.g., [Suite:openshift/conformance/parallel], [Serial], [Conformance]). Format each test name on its own line wrapped in Jira {code} blocks so that tooling can find bugs by test name:
      {code}
      [Jira:"Test Framework"] there should not be mass test failures
      {code}
      
    • Test ID (test_id — the BigQuery/Component Readiness ID, e.g., openshift-tests:abc123)
    • Regression ID(s) — the Component Readiness regression ID(s) being triaged
    • Regression opened date
    • Affected variants
    • Failure patterns identified (permafail/flaky/resolved/recent)
    • Pass sequence analysis from step 6
    • Global test report from step 4: overall pass rate, per-variant breakdown, open bugs
    • Failure output consistency analysis from step 7 (if available):
      • Common error message
      • Consistency percentage
      • Key debugging information (file references, resources, stack traces)
    • Sippy Test Details report links - this is critical for debugging:
      • Link for the current regression (converted test_details_ui_url)
      • Links for each related regression found in step 10 (each regression has its own test_details_url from the list-regressions data - convert each to UI URL)
    • Regression start date (if determined in step 8)
    • Suspect PRs from payload analysis (if determined in step 9) — include PR URLs and relevance classification for LIKELY and POSSIBLY related PRs
    • Related regressions (if any) with their regression IDs and test names
  • Triage type recommendation:
    • product: actual product issues (default)
    • test: clear issues in the test itself (especially for flaky patterns)
    • ci-infra: CI outages
    • product-infra: customer-facing outages (e.g., quay)

If regression IS triaged (existing JIRA):

  • Note: "A bug already exists for this regression - do not file a duplicate"
  • Display JIRA key(s) with links
  • Show JIRA progress analysis from step 5:
    • 🟢 ACTIVE: Bug is being worked on, no action needed
    • 🟡 STALLED: Bug may need attention, consider commenting or reassigning
    • 🔴 NEEDS ATTENTION: Bug appears abandoned, consider taking ownership or escalating
  • If analysis_status == -1000 (failed fix):
    • Recommend: Re-open the existing bug OR file a new bug if the failure mode is different
    • Provide new bug template if failure analysis suggests a different root cause
  1. Display Comprehensive Report: Present findings in clear format

Section 1: Regression Summary

  • Test name
  • Component
  • Regression opened/closed dates
  • Affected variants
  • Analysis status and explanations
  • Current triage status (none for untriaged regressions)
  • Global test report (from step 4): overall pass rate, trend, open bugs count
  • If open bugs exist, note them as potential triage targets

Section 2: Global Test Health (from fetch-test-report skill with --no-collapse)

  • Overall pass rate across all variants for this release
  • Per-variant breakdown highlighting variants with low pass rates
  • Open bugs count — if > 0, list them as potential triage targets
  • Trend direction (improving/regressing/unchanged)

Section 3: Failure Pattern Analysis

  • Jobs ranked by number of failures (most to least impacted)
  • For each job:
    • Job name
    • Total failed runs
    • Pass sequence visualization
    • Pattern classification:
      • Permafail: All or most recent runs failing
      • Resolved: Recent runs succeeding after a block of failures
      • Flaky: Sporadic failures interspersed with successes
      • Recent Regression: Recently started failing after consistent successes
    • Priority level (High/Medium/Low)
    • Recommended action
  • Overall assessment of regression severity

Section 4: Failure Output Analysis (from fetch-test-runs skill)

  • Number of test outputs analyzed
  • Consistency classification (Highly Consistent / Moderately Consistent / Inconsistent)
  • Most common error message with occurrence count
  • Key debugging information:
    • File/line references
    • Resource or API paths
    • Error messages
  • Sample job URLs for manual inspection
  • If the test outputs API is not available, this section states: "Test output analysis not available"

Section 4a: Job Run Context (from fetch-job-run-summary skill)

  • Number of job runs analyzed and their individual failure counts
  • Whether the regressed test is an isolated failure or part of broader co-failures
  • Whether the same tests consistently co-fail across runs
  • Dominant error pattern with percentage (if applicable)
  • For mass test failure regressions: real affected component(s) based on which tests are failing
  • Assessment: isolated test issue, targeted product bug with co-failures, or collateral damage from widespread instability

Section 5: Regression Start Analysis (only if determinable)

  • Job analyzed (the job with most failures)
  • Approximate start date of the regression
  • URL of the first failing job run
  • Pattern description (e.g., "18 consecutive failures followed by 12 successes")
  • Note: This section is omitted if no clear start date can be determined

Section 6: Suspect PRs in Payload (only if regression start was determined)

  • Payload tag and upgrade-from tag (if applicable)
  • Total number of new PRs in the payload
  • Number of PRs investigated (up to 5)
  • For each investigated PR:
    • Relevance classification: LIKELY, POSSIBLY, or UNLIKELY
    • PR URL, title, and associated bug (if any)
    • Brief explanation of why it may or may not be related
    • Key files changed (from gh pr diff --stat)
  • This section is omitted if step 8 did not determine a clear regression start, if the prowjob.json lacks release annotations, or if no PRs looked potentially related. It is also omitted if gh CLI is not available, though PR URLs are still listed.

Section 7: Related Regressions and Existing Triages (from fetch-related-triages skill and list-regressions --test-name)

  • Existing triages that may cover this regression, ranked by confidence level (10 = high, 5 = medium, 2 = low)
  • For each existing triage: JIRA key, status, summary, triage type, confidence level, triage UI link
  • Untriaged regressions that appear related (same/similar tests, shared failure times)
  • Regressions for the exact same test in different variants (from step 10)
  • Recommendation: whether to add to an existing triage or file a new bug
  1. Offer to Triage: After presenting the report, offer to triage the regression

Based on the analysis, determine the appropriate triage action and ask the user if they want to proceed.

Generating the triage description (for new triage records only): Every new triage record must include a --description. Generate a single short sentence (under 120 characters) summarizing the failure — similar to a JIRA bug summary title. Be concise; do not use more than one sentence. Example: "InsightsDataGather CRD not found - InsightsRuntimeExtractor tests failing since Feb 6"

Scenario A: Related triage record found on another regression (from step 11)

If step 11 found that a related regression already has a triage record (i.e., another regression for the same or similar test is already triaged to a JIRA bug), offer to add this regression to that existing triage. Also include any other untriaged related regressions found in steps 10 and 11.

A related triage already exists:
- Triage ID: 789
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-12345
- Type: product

The following untriaged regressions could be added to this triage:
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)

Would you like to add these regressions to the existing triage?

If the user confirms, use the triage-regression skill to update the existing triage:

# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")

# Only pass the new regression IDs to add - the script automatically
# fetches existing regressions and merges them (safe additive behavior)
new_regression_ids="<current_id>,<related_id_1>,<related_id_2>"

script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$new_regression_ids" \
  --token "$TOKEN" \
  --triage-id <existing_triage_id> \
  --url "<existing_jira_url>" \
  --type "<existing_triage_type>" \
  --format json)

After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:

triage_id=$(echo "$triage_result" | jq -r '.triage.id')

Display to the user:

Triage updated:
- JIRA: <existing_jira_url>
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>

Then add the triage record link to the JIRA description using the add-jira-triage-link skill:

jira_key="<existing_jira_key>"
link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "$jira_key" --triage-id "$triage_id" --format json

If successful, report that the triage link was added to the JIRA description. If it was already present, report that it already exists. If JIRA credentials are not set, skip this step silently (the triage itself already succeeded).

The triage-regression script automatically fetches the existing triage and merges its regressions with the new ones, so you only need to pass the regression IDs you want to add.

Scenario B: JIRA bug found but not triaged to any regression (from step 5 or step 11)

If step 5 found a linked JIRA bug on this regression's triage, or step 11 found a JIRA bug that looks related, or step 4's test report found open_bugs > 0 (e.g., same component, similar error pattern) but no triage record exists yet, offer to create a new triage linking this regression and all related untriaged regressions to that bug.

A related JIRA bug was found:
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-67890
- Summary: [bug summary from JIRA]

The following regressions could be triaged to this bug:
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)

Recommended triage type: product

Would you like to create a triage record linking these regressions to this bug?

If the user confirms, use the triage-regression skill to create a new triage:

# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")

all_regression_ids="<current_id>,<related_id_1>,<related_id_2>"

# Generate a concise description from the analysis (see note below)
description="<generated_description>"

script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$all_regression_ids" \
  --token "$TOKEN" \
  --url "https://redhat.atlassian.net/browse/OCPBUGS-67890" \
  --type product \
  --description "$description" \
  --format json)

After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:

triage_id=$(echo "$triage_result" | jq -r '.triage.id')

Display to the user:

Triage created:
- JIRA: https://redhat.atlassian.net/browse/OCPBUGS-67890
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>

Then add the triage record link to the JIRA description using the add-jira-triage-link skill:

link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "OCPBUGS-67890" --triage-id "$triage_id" --format json

If successful, report that the triage link was added to the JIRA description. If JIRA credentials are not set, skip this step silently.

Scenario C: No related triage or bug found

If no related triage record or JIRA bug was found, and the regression is untriaged, offer to create a new JIRA bug and triage all related regressions to it.

No existing triage or related JIRA bug was found for this regression.

Would you like me to create a JIRA bug and triage the following regressions to it?
- Regression <current_regression_id> (this regression)
- Regression <related_id_1> (related, same test, different variant)
- Regression <related_id_2> (related, same test, different variant)

Proposed bug details:
- Project: OCPBUGS
- Component: <component from regression data>
- Summary: <suggested summary from step 12>
- Triage type: <recommended type>

If the user confirms, create the bug using /jira:create bug with the bug template from step 12. Apply the label component-regression to the bug (this label identifies bugs found through Component Readiness). The bug description must include:

  • Test name(s) — the full name including all tags (e.g., [Suite:openshift/conformance/parallel], [Serial]). Each test name must be on its own line wrapped in Jira {code} blocks so that tooling can search for bugs mentioning a specific test:
    {code}
    [Monitor:kubelet-container-restarts][sig-architecture] platform pods in ns/openshift-kube-apiserver should not exit an excessive amount of times
    {code}
    
  • Test and regression identifiers collected in step 12
  • Release
  • Regression opened date
  • Affected variants
  • Failure pattern analysis summary
  • Common error message (if available from step 7)
  • Sippy Test Details report links for each regression being triaged (convert each test_details_url from API to UI URL)
  • Related regression IDs and test names

After the bug is created, mark it as a release blocker using the set-release-blocker skill (component readiness regressions are release blockers):

script_path="plugins/ci/skills/set-release-blocker/set_release_blocker.py"
python3 "$script_path" "<new_bug_key>" --format json

See plugins/ci/skills/set-release-blocker/SKILL.md for details.

Then use the triage-regression skill to triage all regressions to the new bug:

# Obtain auth token from DPCR cluster (oc-auth skill)
TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT")

all_regression_ids="<current_id>,<related_id_1>,<related_id_2>"

# Generate a concise description from the analysis (see note below)
description="<generated_description>"

script_path="plugins/ci/skills/triage-regression/triage_regression.py"
triage_result=$(python3 "$script_path" "$all_regression_ids" \
  --token "$TOKEN" \
  --url "https://redhat.atlassian.net/browse/<new_bug_key>" \
  --type <recommended_type> \
  --description "$description" \
  --format json)

After triaging, extract the triage ID from the response and display both the JIRA URL and the triage UI link:

triage_id=$(echo "$triage_result" | jq -r '.triage.id')

Display to the user:

Bug filed and triaged:
- JIRA: https://redhat.atlassian.net/browse/<new_bug_key>
- Release Blocker: Approved
- Triage: https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/triages/<triage_id>

Then add the triage record link to the JIRA description using the add-jira-triage-link skill:

link_script="plugins/ci/skills/add-jira-triage-link/add_jira_triage_link.py"
python3 "$link_script" "<new_bug_key>" --triage-id "$triage_id" --format json

If successful, report that the triage link was added to the JIRA description. If JIRA credentials are not set, skip this step silently.

Scenario D: Regression is already triaged


View the full analyze-regression.md on GitHub

Name

ci:check-if-jira-regression-is-ongoing

Synopsis

/ci:check-if-jira-regression-is-ongoing <jira-key-or-url>

Description

The ci:check-if-jira-regression-is-ongoing command reads a Jira bug and determines whether the regression(s) it describes are still actively failing or have resolved. It extracts test names, test IDs, and regression IDs from the bug description, status, and comments, then cross-references with live data from Sippy and Component Readiness.

This command is useful for:

  • Checking if a bug's regression has resolved and the bug can be closed
  • Finding out if new regressions have appeared for the same test since the bug was filed
  • Discovering if other triaged bugs exist for the same test that don't match this bug
  • Getting a quick status on whether a fix has landed and is effective

Implementation

  1. Parse Arguments: Extract the Jira key

    Accept either a full URL or a bare key:

    • https://redhat.atlassian.net/browse/OCPBUGS-74690 → OCPBUGS-74690
    • OCPBUGS-74690 → OCPBUGS-74690

    Strip the URL prefix if present:

    jira_key=$(echo "$input" | sed 's|.*/browse/||')
    
  2. Fetch Jira Issue: Read the bug description, status, and comments

    Use the fetch-jira-issue skill to retrieve the bug data:

    jira_script="plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py"
    jira_data=$(python3 "$jira_script" "$jira_key" --format json)
    

    See plugins/ci/skills/fetch-jira-issue/SKILL.md for complete implementation details including error handling.

    Extract and read from the JSON output:

    • summary: Bug title
    • comments[].body: All comment bodies
    • status: Current Jira status (e.g., New, Assigned, Modified, Closed)
    • components[]: Jira component(s)
    • progress.level: Automatic progress classification (ACTIVE/STALLED/NEEDS_ATTENTION/RESOLVED)
  3. Extract Test Identifiers: Scan the bug for test names, test IDs, and regression IDs

    Search through the summary, description, and all comment bodies for:

    • Test IDs (test_id): Match the pattern suite-name:hex-hash (e.g., openshift-tests:bb3a7d828630760296ef203c5cacf708). These are BigQuery/Component Readiness test IDs.
    • Regression IDs: Match standalone integers that appear in context suggesting they are regression IDs (e.g., "regression 34446", "Regression ID: 34446", or in Sippy URLs containing /regressions/). Also extract from Component Readiness URLs like sippy.dptools.openshift.org/....
    • Test names: Match strings in the format [sig-*] ... which are full OpenShift CI test names (e.g., [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]).

    If no test ID, test name, or regression ID can be found, abort:

    Error: Could not identify any test name, test ID, or regression ID in OCPBUGS-74690.
    Cannot check regression status without at least one of these identifiers.
    

    If multiple identifiers are found, use all of them — the bug may cover multiple tests/regressions.

  4. Determine Release: Get the release version

    Check if the bug description or title mentions a release version (e.g., "4.22", "4.21"). If not, use the fetch-releases skill:

    release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)
    
  5. Check Regression IDs (if found): Query each regression's current status

    For each regression ID extracted from the bug:

    script_path="plugins/ci/skills/fetch-regression-details/fetch_regression_details.py"
    regression_data=$(python3 "$script_path" <regression_id> --format json)
    

    For each regression, determine:

    • Status: Is it open or closed?
    • If open: How long has it been open? (opened timestamp to now)
    • If closed: When did it close? (closed timestamp). How long was it open?
    • Triages: What bugs are triaged against it? Do any of the triage jira_key values match our bug? If a triage points to a different bug, flag this for the user.
    • Pass sequences: Check sample_failed_jobs pass sequences. Are recent runs (left side) passing or failing?
  6. Check Test Names and Test IDs Against Open Regressions: Find if the test appears in any current regressions

    Use the list-regressions skill with --test-name to fetch regressions matching this test:

    script_path="plugins/teams/skills/list-regressions/list_regressions.py"
    matching_regressions=$(python3 "$script_path" --release "$release" --test-name "$test_name")
    

    This returns only regressions for the exact test name across all components and variants, without needing to fetch and parse the full regression list.

    For each matching regression:

    • Note whether it is open or closed
    • Note when it was opened and (if closed) when it was closed
    • Check if it is triaged and to which Jira bug
    • If triaged to a different bug than the one we're investigating, flag this:
      ⚠ Regression <id> is triaged to OCPBUGS-XXXXX (not this bug OCPBUGS-74690).
        This may indicate a duplicate or a related but separate issue.
      
    • Note the variants affected

    If the bug contained a test_id but no test_name, use any regression match to obtain the test name for subsequent steps.

  7. Check Global Test Report: Use fetch-test-report with --no-collapse

    script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"
    test_report=$(python3 "$script_path" "$test_name" --release "$release" --no-collapse --format json)
    

    Analyze:

    • Overall pass rate: Is the test healthy (>99%) or still failing?
    • Per-variant breakdown: Which specific variant combinations are still failing? Which have recovered?
    • Trend (net_working_improvement): Is the test improving (positive) or getting worse (negative)?
    • Open bugs count: Are there other open bugs mentioning this test?
    • Comparison to bug context: If the bug mentioned specific variants (e.g., "metal-ipi", "aws"), check those variants specifically.
  8. Check Recent Test Runs: Use fetch-test-runs to see current failure activity

    script_path="plugins/ci/skills/fetch-test-runs/fetch_test_runs.py"
    recent_runs=$(python3 "$script_path" "$test_id" --include-success --format json)
    

    Analyze:

    • Recent pass/fail ratio: Count successes vs failures in recent runs
    • Mass failure detection: Check failed_tests field — if failed_tests > 10, the test may be caught up in broader infrastructure failures rather than a test-specific regression
    • Error consistency: Are the failure outputs consistent with what the bug describes?
    • Last failure date: When was the most recent failure? If all recent runs are passing, the regression may have resolved.
  9. Synthesize and Present Report: Summarize the findings

    Present a clear report answering the core question: Is the regression described in this bug still ongoing?

    Report Format:

    Regression Status Check: OCPBUGS-74690
    ============================================================
    
    Bug: <summary>
    Status: <jira status>
    Component: <component>
    
    Identifiers Found:
    - Test Name: <test name>
    - Test ID: <test_id>
    - Regression IDs: <list>
    
    --- Regression Status ---
    
    [For each regression ID checked:]
    Regression <id>: OPEN / CLOSED
      Opened: <date> (<N days ago>)
      Closed: <date> (<N days ago>) [if closed]
      Triaged to: <jira key> [matches this bug / DIFFERENT BUG ⚠]
      Recent Pattern: <pass_sequence summary>
    
    --- Current Regressions for This Test ---
    
    [For each matching regression in current release:]
    Regression <id>: OPEN / CLOSED
      Variants: <variant list>
      Triaged: Yes → OCPBUGS-XXXXX / No
      [If triaged to different bug: ⚠ Triaged to OCPBUGS-XXXXX, not this bug]
    
    --- Global Test Health ---
    
    Overall Pass Rate: <X>% (trend: improving/regressing)
    Open Bugs: <N>
    
    Per-Variant Breakdown:
      <variant combo>: <pass rate>% [FAILING / HEALTHY]
      <variant combo>: <pass rate>% [FAILING / HEALTHY]
      ...
    
    --- Recent Test Runs ---
    
    Total runs: <N> (Passes: <N>, Failures: <N>)
    Mass failure runs: <N> of <N> failures had >10 other test failures
    Last failure: <date>
    Last success: <date>
    
    --- Conclusion ---
    
    <One of the following assessments:>
    
    ✅ RESOLVED: The regression appears to have resolved. The test has been
       passing consistently since <date>. The global pass rate is <X>%.
       [Recommend closing the bug if no further action needed.]
    
    ⚠ PARTIALLY RESOLVED: The test is passing in most variants but still
       failing in <variant list>. Pass rate is <X>% overall.
       [Recommend keeping the bug open until all variants recover.]
    
    ❌ STILL ONGOING: The regression is still actively failing. The test
       has a <X>% pass rate with <N> failures in the last 7 days.
       [The bug should remain open.]
    
    🔄 INCONCLUSIVE: Could not definitively determine regression status.
       <reason — e.g., mass failures obscuring results, test not running recently>
    

    If the bug is Closed or Verified but the regression is STILL ONGOING or PARTIALLY RESOLVED:

    This indicates the fix was insufficient or the problem has recurred. Offer to reopen the bug:

    ⚠ FAILED FIX: This bug is <Closed/Verified> but the regression is still ongoing.
    The test has a <X>% pass rate with <N> recent failures.
    
    Would you like me to:
    1. Add a comment explaining the regression is still active
    2. Reopen the bug by moving status back to ASSIGNED
    

    If the user confirms, perform both actions:

    Add a comment explaining the situation, including:

    • Current pass rate and failure count
    • Which variants are still failing
    • Link to the Sippy test details report
    • A statement that the regression was detected as still ongoing by the check command
    # Add comment to the bug
    JIRA_AUTH=$(printf '%s' "$JIRA_USERNAME:$JIRA_API_TOKEN" | base64 | tr -d '\r\n')
    curl -s -X POST \
      -H "Authorization: Basic $JIRA_AUTH" \
      -H "Content-Type: application/json" \
      -d '{"body": "<comment text>"}' \
      "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/comment"
    

    Transition the bug back to ASSIGNED:

    First, fetch available transitions to find the correct transition ID:

    transitions=$(curl -s -H "Authorization: Basic $JIRA_AUTH" \
      "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/transitions")
    

    Look through the transitions for one that moves to "Assigned" status, then execute it:

    curl -s -X POST \
      -H "Authorization: Basic $JIRA_AUTH" \
      -H "Content-Type: application/json" \
      -d '{"transition": {"id": "<transition_id>"}}' \
      "https://redhat.atlassian.net/rest/api/3/issue/<jira_key>/transitions"
    

    Display confirmation:

    Bug reopened:
    - JIRA: https://redhat.atlassian.net/browse/<jira_key>
    - Status: ASSIGNED
    - Comment added explaining regression is still active
    

Return Value

  • Format: Human-readable summary report
  • Key information:
    • Whether the regression is still ongoing, resolved, or partially resolved
    • When the regression appears to have resolved (if applicable)
    • Current pass rates per variant
    • Whether other regressions or bugs exist for the same test
    • Flags for regressions triaged to different bugs

Examples

  1. Check a bug by URL:

    /ci:check-if-jira-regression-is-ongoing https://redhat.atlassian.net/browse/OCPBUGS-74690
    
  2. Check a bug by key:

    /ci:check-if-jira-regression-is-ongoing OCPBUGS-74690
    

Arguments

  • $1 (required): Jira issue key or URL
    • Format: Either a bare key (OCPBUGS-74690) or a full URL (https://redhat.atlassian.net/browse/OCPBUGS-74690)

Prerequisites

  1. JIRA_USERNAME: Required for Jira authentication

    • Set environment variable: export JIRA_USERNAME="your-atlassian-email"
  2. JIRA_API_TOKEN: Required to read the Jira issue

  3. Python 3: Required for all skill scripts

    • Version: 3.6 or later

Skills Used

  • fetch-regression-details: Checks regression status (open/closed), triages, and pass sequences
  • fetch-test-report: Gets global test health with per-variant breakdown
  • fetch-test-runs: Gets recent test run results to check current failure activity
  • fetch-releases: Determines the latest OCP release when not specified in the bug
  • list-regressions (teams plugin): Lists all regressions for a release to find matches by test name/ID

See Also

  • Related Command: /ci:analyze-regression — Full regression analysis workflow
  • Related Command: /ci:fetch-test-report — Fetch test health report by name
  • Related Skill: fetch-regression-details (plugins/ci/skills/fetch-regression-details/SKILL.md)
  • Related Skill: fetch-test-report (plugins/ci/skills/fetch-test-report/SKILL.md)
  • Related Skill: fetch-test-runs (plugins/ci/skills/fetch-test-runs/SKILL.md)
  • Related Skill: fetch-releases (plugins/ci/skills/fetch-releases/SKILL.md)
  • Related Skill: list-regressions (plugins/teams/skills/list-regressions/SKILL.md)

Name

ci:continue-session

Synopsis

/ci:continue-session <prowjob-url>

Description

Downloads Claude session artifacts from a Prow CI job and allows you to continue one of those sessions locally. This is useful when Claude ran as part of a CI job and you want to pick up where it left off.

The command accepts:

  • prowjob-url (required): URL to a Prow job run

Prerequisites

  1. gcloud CLI — for accessing GCS artifacts

    • Check: which gcloud
    • The test-platform-results-public bucket is publicly accessible (no auth needed)
  2. Python 3 (3.7+)

    • Check: which python3
  3. tar — for extracting archives

Implementation

Step 1: Search for session archives

Load the ci:prow-job-analysis skill and use its artifact search script (prow_job_artifact_search.py) to find files matching **/claude-session*.tar* in the job's artifacts.

Parse the JSON output. If no matches are found (count: 0), tell the user "No Claude session archives found in this job's artifacts." and stop.

Step 2: Download archives to current directory

For each matched archive file, download it from GCS using gcloud storage cp:

gcloud storage cp --no-user-output-enabled <gcs_uri> ./

Step 3: Download payload artifacts

Search for payload artifacts using the ci:prow-job-analysis skill's artifact search script with the pattern **/payload*.yaml, **/payload*.json, and **/payload*.html.

For each matched file, download it to the current directory:

gcloud storage cp --no-user-output-enabled <gcs_uri> ./

These are supplementary artifacts — if none are found, continue without error.

Step 4: Extract archives

For each downloaded archive in the current directory:

# tar auto-detects compression format (.tar, .tar.gz, .tar.zst, .tar.bz2, etc.)
tar xf <filename>

After extraction, the structure contains a projects/ directory. Each session consists of two parts:

  • A JSONL file at projects/<project-path>/<session-uuid>.jsonl (the main conversation log)
  • A directory at projects/<project-path>/<session-uuid>/ (containing subagent data and other session state)

Both must be copied together for the session to work.

Step 5: Discover sessions and extract first user message

Find all session JSONL files (files matching *.jsonl under the extracted projects/ hierarchy). Each .jsonl file corresponds to a session, and its filename (without extension) is the session UUID.

For each session JSONL file found:

  1. Read the JSONL file line by line
  2. Look for the first message with "type": "user" that does NOT have "isMeta": true — this is the first real user message
  3. Extract the text content from message.content — it may be a string or an array of content blocks. If it's an array, look for blocks with "type": "text" and extract the text field.
  4. Truncate the message to 120 characters for display purposes
  5. Record the session UUID, the full path to the JSONL file, and the truncated first user message

If no sessions are found after extraction, tell the user "No Claude sessions found in the extracted archives." and stop.

Step 6: Present session list to user

If only one session is found, display its details and confirm with the user before proceeding.

If multiple sessions are found, display a numbered list asking "What session would you like to continue?" with each entry showing:

  • The session number
  • The session UUID
  • The truncated first user message

Example:

What session would you like to continue?

1. [abc12345-...] "Please analyze the test failures in..."
2. [def67890-...] "Can you help fix the flaky e2e test..."

Use AskUserQuestion to let the user select a session by number.

Step 7: Copy selected session to local Claude projects directory

The extracted archive contains sessions under a CI-specific project path (e.g., projects/-home-prow-go-src-github.com-org-repo/). This path reflects where the repo was checked out in CI and will not match the local project path.

Determine the local project path that Claude uses for the current working directory. Claude's projects directory structure encodes the absolute path of the project by replacing both / and . with - and stripping the leading slash. For example, if the current working directory is /Users/stbenjam/git/github.com/my-repo, the local project path is:

~/.claude/projects/-Users-stbenjam-git-github-com-my-repo/

Compute this by taking pwd and replacing all / and . with -. Since absolute paths start with /, this naturally produces a leading -.

Before copying, check if the session UUID already exists locally (either the .jsonl file or the directory). If it does, warn the user that a local session with this UUID already exists and ask whether to overwrite it. If the user declines, stop.

Copy both the selected session's JSONL file and its corresponding directory into the local project path:

# Determine local project path
LOCAL_PROJECT_DIR="$HOME/.claude/projects/$(pwd | tr '/.' '--')"
mkdir -p "$LOCAL_PROJECT_DIR"

# Copy the session JSONL file
cp <path-to-extracted>/<session-uuid>.jsonl "$LOCAL_PROJECT_DIR/"

# Copy the session directory (contains subagent data)
cp -r <path-to-extracted>/<session-uuid> "$LOCAL_PROJECT_DIR/"

Only the single selected session is copied — do not copy other sessions. Both the .jsonl file and the UUID directory must be copied for the session to be resumable.

Step 8: Report success

Tell the user:

You are all set! You can now continue the session by launching a new Claude:

claude --resume <session-uuid>

Where <session-uuid> is the UUID of the session they selected.

If any payload artifacts were downloaded, list them so the user knows they are available in the current directory.

Arguments

  • $1 (prowjob-url): URL to a Prow job run (required). Accepts standard Prow UI URLs or gcsweb URLs.

Name

ci:detect-permafail

Synopsis

/ci:detect-permafail --job-urls="[url1,url2,...]" --job-name="job-name" --pr="owner/repo#123"

Description

Analyzes 2-10 consecutive failures of the same job to determine if they represent a systematic/permanent failure (permafail) versus a flaky failure. This is critical for CI/CD pipeline analysis to distinguish between:

  • Permafail: A systematic failure affecting the same test(s) or infrastructure issue, detected by analyzing comparable runs (same failure type):
    • Separates test failures from infrastructure failures
    • Applies thresholds based on comparable run count (not total URLs)
    • 2-3 comparable runs: All must match (100% required)
    • 4-5 comparable runs: At least 4 must match (80% required)
    • 6-10 comparable runs: At least ceil(count×0.7) must match (70% required)
    • Example: 7 URLs where 5 are test failures and 4 of those 5 have same test → 4/5 = 80% → PERMAFAIL
  • Flaky: Non-deterministic failures with varying root causes, or failures that don't meet the permafail thresholds

How It Works

The command uses a deterministic Python classifier script for artifact-based failure classification, compares failure signatures across runs, and returns a JSON result with the permafail verdict, confidence score, failure type, and detailed signatures for each run.

Implementation

  • Load the "detect-permafail" skill
  • Pass the job URLs, job name, and PR info to the skill
  • The skill handles all analysis including:
    • Artifact fetching and classification via plugins/ci/scripts/classify-job-failures.py
    • Signature comparison and pattern detection
    • Threshold-based permafail determination

Arguments

  • --job-urls: JSON array of 2-10 consecutive Prow job URLs (required, newest first)
  • --job-name: Name of the job being analyzed (required, e.g., "e2e-aws-ovn")
  • --pr: PR identifier (required, format: "owner/repo#number")

Prerequisites

  • Access to Prow CI job artifacts via gcsweb URLs
  • Access to Bash for running the classifier script
  • Python with the requests package available

Name

ci:extract-kubeconfig

Synopsis

/ci:extract-kubeconfig <pr-url>

Description

The ci:extract-kubeconfig command extracts the kubeconfig from a running rehearsal/CI job in a GitHub PR. It checks the job's step status to determine if the cluster is ready, then connects to the build cluster to extract the kubeconfig from the running pod.

The command accepts:

  • PR URL (required): GitHub PR URL in the openshift/release repository

What it does:

  1. Validates the PR URL and checks ownership
  2. Finds running (PENDING) jobs on the PR
  3. Determines the build cluster (via GCS for public jobs, via job config for qe-private-deck jobs, or by asking the user for the console URL or cluster name as a last resort)
  4. Logs into the build cluster and finds the CI namespace
  5. Checks step status via pod statuses to verify the cluster is ready
  6. Extracts the kubeconfig (and nested kubeconfig for HyperShift) from the running pod
  7. Verifies cluster health and reports results

Key features:

  • Supports both standard and HyperShift (nested kubeconfig) clusters
  • Supports both public (prow.ci.openshift.org) and private (qe-private-deck) jobs
  • Uses ci-op namespace prefix on kubeconfig filenames to support multiple concurrent extractions
  • Warns if you're not the PR author (may lack namespace access)
  • Recognizes wait steps as valid running states for extraction

Prerequisites

  1. GitHub CLI (gh)

    • Check if installed: which gh
    • Must be authenticated: gh auth status
  2. OpenShift CLI (oc)

    • Check if installed: which oc
    • Used to login to build clusters and extract kubeconfig
  3. Google Cloud Storage (gsutil) (optional)

    • Check if installed: which gsutil
    • No authentication needed (test-platform-results-public bucket is public)
    • If not installed, the command falls back to finding the build cluster from the job config in the openshift/release repo
  4. PR in openshift/release repository

    • The command only supports PRs in the openshift/release repository
    • You should be the PR author to have access to the CI namespace on the build cluster

Implementation

The command performs the following steps:

Step 1: Validate PR URL

Parse the PR URL to extract the org, repo, and PR number. The URL should match the format https://github.com/<org>/<repo>/pull/<number>.

If the repo is NOT openshift/release, warn the user and stop.

Step 2: Check PR Ownership

Fetch the PR author and compare with the current GitHub user:

gh pr view <number> --repo openshift/release --json author --jq '.author.login'
gh api user --jq '.login'

If the PR author does NOT match the current user, warn: "You are not the author of this PR. You may not have access to the CI namespace on the build cluster to extract the kubeconfig." Ask the user if they want to continue. Stop if they say no.

Step 3: Find Running Jobs

Fetch the PR status checks:

gh pr view <number> --repo openshift/release --json title,state,statusCheckRollup --jq '
  .statusCheckRollup[]
  | select(.__typename == "StatusContext" and .state == "PENDING" and (.context | test("^tide$") | not))
  | {context, state, targetUrl}
'
  • If multiple running jobs found, list them and ask the user which one to use.
  • If exactly one running job found, use it automatically.
  • If no running jobs found, report that and stop.

Step 4: Determine the Build Cluster

From the matching check, extract the targetUrl which points to the Prow job page. Determine the build cluster using one of three methods:

Method A — Via GCS prowjob.json (preferred when gsutil is installed and the job uses public prow.ci.openshift.org):

Parse the GCS path from targetUrl:

  • Format: /view/gs/<bucket>/... (Prow UI) or /gcs/<bucket>/... (gcsweb)
  • Extract <bucket> and the object path after it
  • Download from gs://<bucket>/<object-path>/prowjob.json:
    gsutil cp "gs://<bucket>/<object-path>/prowjob.json" /tmp/prowjob.json
    

Method B — Via job config in openshift/release repo (fallback when gsutil is not installed, or when the job uses qe-private-deck):

Find the build cluster from the generated job config in the openshift/release repo using the PR's fork and branch:

  1. Get the PR's head ref info:

    gh pr view <number> --repo openshift/release --json headRefName,headRepository,headRepositoryOwner
    
  2. Determine the original job name. For rehearsal jobs, the check context is like ci/rehearse/periodic-ci-<org>-<repo>-.... Strip the ci/rehearse/ prefix to get the rehearsal job name, then strip rehearse-<pr_number>- to get the original job name.

  3. Parse the org from the original job name (first segment after periodic-ci- or pull-ci-).

  4. List repo directories under ci-operator/jobs/<org>/ via the GitHub Contents API on the PR's fork/branch:

    gh api "repos/<owner>/<repo>/contents/ci-operator/jobs/<org>?ref=<branch>" --jq '.[].name'
    

    Match the longest directory name that is a prefix of the job name remainder (after stripping periodic-ci-<org>-).

  5. List files in that directory matching *-periodics.yaml and download the matching file via raw URL:

    curl -sL "https://raw.githubusercontent.com/<owner>/<repo>/<branch>/ci-operator/jobs/<org>/<repo_dir>/<periodics_file>" \
      | grep -B 30 "name: <original_job_name>" | grep "cluster:"
    

Method C — Ask the user (fallback when Methods A and B both fail):

This happens when:

  • The job uses qe-private-deck (gsutil gets 403 AccessDeniedException on the private GCS bucket)
  • The generated job config has no cluster: field (build cluster is dynamically assigned by the Prow dispatcher)

In this case, ask the user to provide the OpenShift console URL from the build cluster where the job is running. The user can find this on the Prow job page (qe-private-deck) — it typically shows a link to the build cluster console or displays the cluster name in the job metadata.

Unable to determine the build cluster automatically:
- GCS bucket "qe-private-deck" requires authentication
- Job config has no cluster: field (dynamically assigned)

Please provide the OpenShift console URL of the build cluster from the Prow job page.
Example: https://console-openshift-console.apps.build09.ci.devcluster.openshift.com/

Alternatively, provide just the build cluster name (e.g., build09):

Extract the cluster name from the console URL:

  • Pattern: https://console-openshift-console.apps.<cluster>.ci.devcluster.openshift.com/...
  • Extract <cluster> (e.g., build09, build01, build11)

If the user provides a plain cluster name (e.g., build09) instead of a URL, use it directly.

Validate the cluster identifier before constructing the login URL. The cluster name must match the pattern build[0-9]+ (e.g., build01, build09, build12). Reject any input that does not match — this prevents credential redirection to unexpected hosts.

Construct the API server URL: https://api.<cluster>.ci.devcluster.openshift.com:6443

→ Continue to Step 5: Login to the Build Cluster

Step 5: Login to the Build Cluster

IMPORTANT: Use a dedicated temporary kubeconfig file for all build cluster operations. This prevents modifying the user's current kubeconfig context in ~/.kube/config, which could disrupt other processes (e.g., running oc commands, controllers, or scripts) that rely on the current context.

Create a temporary kubeconfig file path and set the target server:

BUILD_KUBECONFIG="/tmp/build-cluster-kubeconfig.yaml"
TARGET_SERVER="https://api.<cluster>.ci.devcluster.openshift.com:6443"

Where <cluster> is the build cluster determined in Step 4.

If the temp kubeconfig already exists (it may contain contexts from multiple build clusters from previous runs), find a context matching the target server:

MATCHING_CTX=""
for ctx in $(KUBECONFIG="$BUILD_KUBECONFIG" oc config get-contexts -o name 2>/dev/null); do
  server=$(KUBECONFIG="$BUILD_KUBECONFIG" oc --context="$ctx" config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null)
  if [ "$server" = "$TARGET_SERVER" ]; then
    MATCHING_CTX="$ctx"
    break
  fi
done

If a matching context is found and authentication is still valid (KUBECONFIG="$BUILD_KUBECONFIG" oc whoami --context="$MATCHING_CTX" succeeds), switch to it and reuse:

KUBECONFIG="$BUILD_KUBECONFIG" oc config use-context "$MATCHING_CTX"

Otherwise (no temp kubeconfig, no matching context, or session expired), login into the temp kubeconfig:

KUBECONFIG="$BUILD_KUBECONFIG" oc login --server="$TARGET_SERVER" --web

All subsequent oc commands targeting the build cluster MUST use KUBECONFIG="$BUILD_KUBECONFIG" as an environment variable prefix instead of --context=<ctx>. This ensures the user's default kubeconfig is never read or modified.

Step 6: Find CI Namespace

Use oc get projects with the dedicated kubeconfig and grep for the PR number and job name to find the ci-op namespace:

KUBECONFIG="$BUILD_KUBECONFIG" oc get projects | grep "rehearse-<pr_number>-<job_name_suffix>"

The project display name includes the prow job name. For rehearsal jobs, this contains both the PR number and the job name, e.g.:

ci-op-tk28ym3x    ci-op-tk28ym3x - rehearse-75742-periodic-ci-...    Active

Construct the grep pattern from the check context found in Step 3. The context is like ci/rehearse/periodic-ci-openshift-.... The rehearsal job name in the project description is rehearse-<pr_number>-periodic-ci-openshift-.... So grep with both the PR number and enough of the job name to be unique.

Step 7: Check Step Status via Pods

List pods in the namespace to determine step status:

KUBECONFIG="$BUILD_KUBECONFIG" oc get pods -n <namespace>

Each pod name corresponds to a step in the job. Pod statuses indicate step progress:

  • Completed — step finished successfully
  • Running — step is currently executing
  • Error — step failed

Report the status of all steps/pods. If any pod shows Error status, report the failure and stop.

Determine if the cluster is ready for kubeconfig extraction: there must be at least one Running pod that has the kubeconfig available (i.e., setup/provisioning pods like those with names containing setup, install, create, or provision have Completed). If the cluster creation step has completed but additional setup steps are still running (e.g., OADP setup, operator installation), warn the user that the cluster may not be fully configured yet but proceed with extraction since the kubeconfig is already available.

Step 8: Extract Kubeconfig

Find a pod that is in Running status.

Use the ci-op namespace as a prefix for kubeconfig filenames to avoid collisions when extracting from multiple jobs.

Extract the kubeconfig:

KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/kubeconfig > /tmp/<namespace>-kubeconfig.yaml
chmod 600 /tmp/<namespace>-kubeconfig.yaml

Also check for nested kubeconfig (for HyperShift/hosted clusters):

KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- ls /tmp/secret/nested_kubeconfig 2>/dev/null

If nested_kubeconfig exists, extract it too:

KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/nested_kubeconfig > /tmp/<namespace>-nested-kubeconfig.yaml
chmod 600 /tmp/<namespace>-nested-kubeconfig.yaml

Also extract additional metadata from /tmp/secret/ on the pod:

# Proxy configuration (for clusters behind a proxy)
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/proxy-conf.sh > /tmp/<namespace>-proxy-conf.sh 2>/dev/null
chmod 600 /tmp/<namespace>-proxy-conf.sh 2>/dev/null

# Cluster name and cluster ID (for ROSA clusters, may not exist)
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/cluster-name 2>/dev/null || true
KUBECONFIG="$BUILD_KUBECONFIG" oc exec -n <namespace> <running_pod> -c test -- cat /tmp/secret/cluster-id 2>/dev/null || true

If proxy-conf.sh is empty or the command fails, there is no proxy config — remove the empty file.

Step 9: Verify and Report

Test the kubeconfig:

KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get nodes
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get co

If oc commands fail (e.g., connection refused or timeout) and proxy-conf.sh was extracted, source the proxy config and retry:

source /tmp/<namespace>-proxy-conf.sh
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get nodes
KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get co

Determine cluster type: Check if the cluster is ROSA:

KUBECONFIG=/tmp/<namespace>-kubeconfig.yaml oc get infrastructure cluster -o jsonpath='{.status.platformStatus..aws.resourceTags[?(@.key=="red-hat-clustertype")].value}'
  • If the value equals rosa: this is a ROSA HCP cluster. There is no nested kubeconfig — the single kubeconfig is the ROSA HCP cluster. Report cluster-name and cluster-id from /tmp/secret/.
  • If nested kubeconfig exists: this is a standard HyperShift job. Verify the nested kubeconfig (including proxy fallback if needed):
    KUBECONFIG=/tmp/<namespace>-nested-kubeconfig.yaml oc get nodes
    KUBECONFIG=/tmp/<namespace>-nested-kubeconfig.yaml oc get co
    
    Report cluster-name from /tmp/secret/ for the hosted cluster.
  • Otherwise: this is a standard cluster.

Return Value

  • Success: Kubeconfig file(s) extracted and cluster health report displayed
  • Error: Error message explaining the issue (no running jobs, cluster not ready, etc.)

Report includes:

  • PR title and number
  • Job name and Prow URL
  • Build cluster and CI namespace
  • Step status summary (which completed, which is running)
  • If proxy-conf.sh found: proxy config path and instructions to source it before using the kubeconfig
  • If ROSA HCP cluster (single kubeconfig, red-hat-clustertype=rosa): kubeconfig path, cluster name, cluster ID, node/CO health summary
  • If HyperShift (nested kubeconfig, not ROSA): management cluster kubeconfig path and health, hosted cluster kubeconfig path, cluster name, and health
  • If standard cluster: kubeconfig path and node/CO health summary
  • Instructions for using the kubeconfig(s)

Examples

  1. Extract kubeconfig from a running rehearsal job:

    /ci:extract-kubeconfig https://github.com/openshift/release/pull/75742
    

    Expected output:

    • Finds the running rehearsal job
    • Verifies pre-phase steps passed
    • Extracts kubeconfig from test pod on build10
    • Reports: 3 master nodes, 4 workers, 34 COs healthy
  2. Extract kubeconfig from a HyperShift job (with nested kubeconfig):

    /ci:extract-kubeconfig https://github.com/openshift/release/pull/75716
    

    Expected output:

    • Extracts both management and hosted cluster kubeconfigs
    • Management cluster: 7 nodes, 34 COs
    • Hosted cluster: 3 nodes, 21 COs
    • Reports both kubeconfig paths
  3. PR with multiple running jobs:

    /ci:extract-kubeconfig https://github.com/openshift/release/pull/12345
    

    Expected output:

    • Lists all running jobs
    • Asks user to select which job to extract from
    • Proceeds with selected job

Notes

  • Credential hygiene: Extracted kubeconfig and proxy files contain credentials. They are created with restrictive permissions (chmod 600). Delete them when finished (rm -f /tmp/<namespace>-*.yaml /tmp/<namespace>-proxy-conf.sh).
  • Kubeconfig expiry: Extracted kubeconfig files become invalid when the CI job finishes. The credentials are tied to the ephemeral CI cluster and cannot be reused after the job completes.
  • PR ownership: You must be the PR author to have access to the ci-op namespace on the build cluster. The command warns if you're not the author.
  • Private QE deck: Jobs using qe-private-deck have inaccessible GCS artifacts (gsutil gets 403). The command tries the job config first; if that also lacks a cluster: field (dynamically assigned jobs), it asks the user for the build cluster console URL or name.
  • Build clusters: OpenShift CI uses multiple build clusters (build01-build12). The command determines which cluster via: (1) prowjob.json from GCS for public jobs, (2) job config YAML in the repo, or (3) user-provided console URL/cluster name as a last resort.
  • Context isolation: All build cluster operations use a dedicated temporary kubeconfig (/tmp/build-cluster-kubeconfig.yaml) to avoid modifying the user's current kubeconfig context. The temp kubeconfig can accumulate contexts from multiple build clusters across runs. If a valid context for the target build cluster already exists in the temp kubeconfig, it is reused without requiring a new login.
  • Namespace discovery: The ci-op namespace is found via oc get projects filtered by PR number and job name, which works reliably for running jobs.
  • Multiple extractions: Kubeconfig files use the ci-op namespace as prefix (e.g., /tmp/ci-op-abc123-kubeconfig.yaml) to support extracting from multiple jobs without overwriting.
  • Wait steps: Jobs with a wait step (used for debugging) are valid targets for kubeconfig extraction.

Arguments

  • $1 (pr-url): GitHub PR URL in the openshift/release repository (required). Format: https://github.com/openshift/release/pull/<number>

Name

ci:fetch-payloads

Synopsis

/ci:fetch-payloads [architecture] [version] [stream]

Description

The ci:fetch-payloads command fetches recent release payloads from the OpenShift release controller. It shows each payload's tag name, phase (Accepted/Rejected/Ready), timestamp, and blocking job results. For rejected payloads, it reports which blocking jobs failed with Prow links.

Implementation

  1. Determine parameters: Default to amd64 architecture and nightly stream if the user hasn't specified them. If no version is provided, the script fetches the latest from the Sippy API.

  2. Fetch payloads: Use the fetch-payloads skill:

    FETCH_PAYLOADS="${CLAUDE_PLUGIN_ROOT}/skills/fetch-payloads/fetch_payloads.py"
    if [ ! -f "$FETCH_PAYLOADS" ]; then
      FETCH_PAYLOADS=$(find ~/.claude/plugins -type f -path "*/ci/skills/fetch-payloads/fetch_payloads.py" 2>/dev/null | sort | head -1)
    fi
    if [ -z "$FETCH_PAYLOADS" ] || [ ! -f "$FETCH_PAYLOADS" ]; then echo "ERROR: fetch_payloads.py not found" >&2; exit 2; fi
    python3 "$FETCH_PAYLOADS" [architecture] [version] [stream]
    

    Optional flags:

    • --phase Accepted|Rejected|Ready to filter by phase
    • --limit N to control how many results to show (default 5)
  3. Present the results: Show the user the payload list. For rejected payloads, highlight the failed blocking jobs and their Prow links.

Return Value

  • Format: One block per payload with tag, phase, timestamp, URL, and job details
  • Rejected payloads: Each failed blocking job is listed with retry count and Prow link
  • Ready payloads: Summary of succeeded/pending/failed blocking job counts
  • Accepted payloads: Confirmation that all blocking jobs succeeded

Examples

  1. Latest nightly payloads (defaults to amd64, latest version, last 5):

    /ci:fetch-payloads
    
  2. Specific version and architecture:

    /ci:fetch-payloads arm64 4.18 nightly
    
  3. Only accepted payloads:

    /ci:fetch-payloads amd64 4.23 nightly --phase Accepted
    

Arguments

  • $1: CPU architecture (optional, default: amd64) — amd64, arm64, ppc64le, s390x, multi
  • $2: OCP version (optional, default: latest from Sippy) — e.g., 4.18, 4.23
  • $3: Release stream (optional, default: nightly) — nightly, ci

Skills Used

  • fetch-payloads: Queries the release controller API for payload tags and release details. When no version is specified, fetch_payloads.py obtains the latest version directly from the Sippy API.

Name

ci:fetch-test-report

Synopsis

/ci:fetch-test-report <test-name> [release]

Description

The ci:fetch-test-report command fetches a report for an OpenShift CI test by name from the Sippy API. It returns the test's BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous 7-day periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name — this can help surface bugs that have been filed but not yet triaged in Component Readiness.

Implementation

  1. Determine the release: If the user did not specify a release, use the fetch-releases skill to get the latest OCP release:

    release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)
    

    If the user specified a release (e.g., "4.21"), use that value directly.

  2. Fetch the test report: Use the fetch-test-report skill to query the Sippy tests API:

    python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test-name>" --release "$release" --format summary
    

    To see a per-variant breakdown (one row per variant combo), add --no-collapse:

    python3 plugins/ci/skills/fetch-test-report/fetch_test_report.py "<test-name>" --release "$release" --no-collapse --format summary
    
  3. Present the results: Show the user the test report including test ID, pass rates, Jira component, open bugs, and trend. If open_bugs > 0, existing Jira bugs mention this test — these may be relevant even if the regression hasn't been triaged in Component Readiness yet.

Return Value

  • Format: Human-readable summary of the test report
  • Key fields: test_id (BigQuery/Component Readiness ID), jira_component, current/previous pass rates, run counts, trend, open_bugs
  • open_bugs: Count of Jira bugs mentioning this test by name — helps find bugs filed but not yet triaged in Component Readiness
  • variants (with --no-collapse): Per-variant breakdown showing which job types the test passes/fails in

Examples

  1. Look up a test using the latest release:

    /ci:fetch-test-report [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]
    
  2. Look up a test for a specific release:

    /ci:fetch-test-report [sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance] 4.21
    

Arguments

  • $1: Full test name (required) — must be an exact match including sig prefix and suite tags
  • $2: OCP release version (optional) — e.g., "4.22", "4.21". If omitted, the latest release is used.

Skills Used

  • fetch-releases: Determines the latest OCP release when not specified by the user
  • fetch-test-report: Queries the Sippy tests API for the test report

Name

ci:list-step

Synopsis

/ci:list-step

Description

The ci:list-step command analyzes an OpenShift CI workflow or chain and provides a complete hierarchical breakdown of all the workflow, chain, and ref files with their repo paths that make up the workflow or chain.

The command accepts:

  • Workflow/Chain name (required)

It uses the step-registry-analyzer agent to examine the workflow/chain structure and identify all related components including workflows, chains, refs, and their associated command files.

Implementation

The command performs the following steps:

  1. Parse Arguments:
    • $1: Workflow or chain name (required, e.g., "hypershift-aws-reqserving-e2e")
  2. Execute Analysis: Use the Task tool with step-registry-analyzer agent: Task tool with:
  • subagent_type: "step-registry-analyzer"
  • description: "analyze and list all workflow/chain/ref with repository path for and keep the hierarchical structure"
  • prompt: Detailed request to analyze the workflow or chain and provide complete hierarchical breakdown
  1. Display Results: Present the hierarchical structure showing:
    • Main workflow or chain file path
    • All chain files referenced in the workflow or chain with their repository paths
    • All ref files referenced in the chains with their repository paths
    • Any command files (.sh) associated with the refs
    • The complete hierarchical structure showing relationships between all files
  2. Format Output: Use a clear hierarchical tree format with:
    • Full repository paths for all files
    • Indentation showing the workflow/chain/ref relationships
    • File types (workflow/chain/ref/command) clearly identified
    • Summary of total files and execution phases

Return Value

  • Success: Complete hierarchical structure of the OpenShift CI workflow or chain
  • Error: Workflow or Chain not found, analysis failure, or invalid workflow or chain name

Important for Claude:

  1. REQUIRED: Always use the step-registry-analyzer agent for this analysis
  2. Use the Task tool with the specialized agent to ensure comprehensive analysis
  3. Maintain the proper directory structure and show the relationships between files
  4. Include all phases (PRE/TEST/POST) in the hierarchy
  5. Show both the structural hierarchy and file paths clearly
  6. Show all workflow/chain/ref/command files with repository path
  7. Provide a summary of total files and execution flow

Examples

  1. ** Analyze hypershift aws e2e external workflow ** :
   /list-step hypershift-aws-e2e-external

Returns:

     Workflow: ci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yaml

Complete Repository Path Hierarchy

ci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yaml
├── PRE Phase (Infrastructure Setup)
│   ├── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-ref.yaml
│   │   └── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-commands.sh
│   ├── ci-operator/step-registry/hypershift/setup-nested-management-cluster/hypershift-setup-nested-management-
cluster-chain.yaml
│   │   └── Embedded Steps:
│   │       ├── create-management-cluster (inline command)
│   │       └── Dependencies: hypershift-operator, AWS credentials, kubeconfigs
│   └── ci-operator/step-registry/hypershift/install/hypershift-install-ref.yaml
│       └── ci-operator/step-registry/hypershift/install/hypershift-install-commands.sh
├── TEST Phase (E2E External Tests)
│   └── ci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-ref.yaml
│       └──
ci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-commands.sh
└── POST Phase (Cleanup)
└── ci-operator/step-registry/hypershift/destroy-nested-management-cluster/hypershift-destroy-nested-managem
ent-cluster-chain.yaml
├── Embedded Step: dump-management-cluster (inline command)
├── ci-operator/step-registry/hypershift/dump/extra/hypershift-dump-extra-ref.yaml
│   └── ci-operator/step-registry/hypershift/dump/extra/hypershift-dump-extra-commands.sh
└── Embedded Step: destroy-management-cluster (inline command)

File Summary

Total Files: 10 files
- 1 workflow file: Main orchestration defining pre/test/post phases
- 2 chain files: Setup and destroy management cluster chains
- 4 ref files: RBAC, E2E test runner, installation, and artifact dump steps
- 3 command files: Shell scripts for executing the ref implementations

Execution Flow

PRE Phase: RBAC Setup → Management Cluster Creation → HyperShift Installation
TEST Phase: External E2E Tests with OIDC support
POST Phase: Artifact Collection → Infrastructure Cleanup

This workflow provides comprehensive end-to-end testing for HyperShift on AWS with external OIDC authentication, multi-architecture support, upgrade
testing, and complete infrastructure lifecycle management.

Notes

  • Hierarchy Structure: Shows both the logical flow (PRE/TEST/POST) and file dependencies
  • File Relationships: Clearly indicates how chains reference refs and how commands are associated
  • Path Completeness: All paths are shown as full repository paths from the root

Arguments

  • $1 (workflow-or-chain-name): The name of the OpenShift CI workflow or chain to analyze (required)

Name

ci:list-unstable-tests

Synopsis

/ci:list-unstable-tests <version> <keywords> [sippy-url]

Description

The ci:list-unstable-tests command queries OpenShift CI test results from Sippy and lists all tests matching the keywords that have a pass rate below 95%. This is useful for quickly identifying unstable tests that need attention.

By default, it queries the production Sippy instance at sippy.dptools.openshift.org. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).

This command is useful for:

  • Identifying unstable tests with inconsistent pass rates
  • Finding regression candidates for investigation
  • Generating reports of unstable test cases
  • Prioritizing test stabilization efforts
  • Quality gate checks before releases

Arguments

  • $1 (version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")
  • $2 (keywords): Keywords to search in test names (e.g., "olmv1", "sig-storage", "operator")
  • $3 (sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"

Implementation

  1. Parse Arguments

    • Extract version from $1 (e.g., "4.20")
    • Extract keywords from $2 (e.g., "olmv1")
    • Extract Sippy URL from $3 if provided, otherwise use default "sippy.dptools.openshift.org"
    • Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
    • Add "https://" prefix if not already present
  2. Build Sippy API Request

    • Construct filter JSON for the /api/tests endpoint:
      filters = {
          "items": [
              {
                  "columnField": "name",
                  "not": False,
                  "operatorValue": "contains",
                  "value": keywords
              }
          ],
          "linkOperator": "and"
      }
      
    • Set query parameters:
      • release: The OpenShift version
      • filter: JSON-encoded filter object
      • sort: "asc"
      • sortField: "current_pass_percentage"
  3. Query Test Statistics

    • Construct API endpoint from provided URL: https://{base_url}/api/tests
    • Make GET request to the constructed endpoint
    • Parse response to extract test data
  4. Filter Tests Below 95% Pass Rate

    • Iterate through all returned tests
    • Filter tests where current_pass_percentage < 95
    • Sort filtered results by pass percentage (ascending, worst first)
    • Collect the following data for each unstable test:
      • Test name
      • Current pass percentage
      • Total runs
      • Passes
      • Failures
      • Net improvement (trend indicator)
  5. Format and Display Results

    • Display summary header with:
      • Total number of tests matching keywords
      • Number of tests below 95% pass rate
      • Percentage of unstable tests
    • List each unstable test with:
      • Test name
      • Pass rate percentage
      • Run statistics (runs/passes/failures)
      • Trend indicator (net improvement)
    • Sort by pass percentage (worst tests first)
    • If no tests are below 95%, display success message indicating all tests are stable

Return Value

Format: Formatted text output with:

Summary Section:

  • Total tests matching keywords
  • Tests below 95% pass rate (unstable tests)
  • Overall stability percentage

Unstable Tests List: For each test with pass rate < 95%:

  • Test name
  • Pass rate percentage
  • Total runs
  • Passes
  • Failures
  • Net improvement

If all tests pass at 95% or above, display a success message indicating all tests are stable.

Examples

  1. List unstable OLMv1 tests from QE Sippy:

    /ci:list-unstable-tests 4.20 olmv1 qe-component-readiness.dptools.openshift.org
    

    Lists all OLMv1-related tests in version 4.20 from QE Sippy that have a pass rate below 95%.

  2. List unstable storage tests (using default Sippy):

    /ci:list-unstable-tests 4.21 sig-storage
    

    Lists all storage-related tests in version 4.21 from production Sippy with pass rate below 95%.

  3. List unstable operator tests:

    /ci:list-unstable-tests 4.19 operator
    

    Lists all operator-related tests in version 4.19 with pass rate below 95%.

  4. Check specific component stability:

    /ci:list-unstable-tests 4.20 sig-network qe-component-readiness.dptools.openshift.org
    

    Lists all network-related unstable tests from QE Sippy.

Notes

  • Pass Rate Threshold: Fixed at 95% - tests with pass rate >= 95% are considered stable
  • Default Sippy URL: If no Sippy URL is provided, the command uses sippy.dptools.openshift.org by default
  • The command queries data from the last 7 days by default
  • Ensure you can access the Sippy API endpoints
  • Results are sorted by pass percentage (ascending) to show most unstable tests first
  • The net improvement metric shows if the test is getting worse (negative) or better (positive)
  • If no tests match the keywords, an appropriate message will be displayed
  • If all matching tests have pass rate >= 95%, a success message will be shown indicating all tests are stable

Output Example

================================================================================
Unstable Tests Report - 4.20 olmv1
================================================================================
Sippy Instance: qe-component-readiness.dptools.openshift.org
Pass Rate Threshold: < 95%

Summary:
  Total Tests Matching 'olmv1': 45
  Unstable Tests (< 95%): 8 (17.8%)
  Stable Tests (>= 95%): 37 (82.2%)

================================================================================
Tests Below 95% Pass Rate (sorted by worst first):
================================================================================

1. Test: [sig-olmv1] clusterextension install should fail validation
   Pass Rate: 23.5%
   Runs: 17 | Passes: 4 | Failures: 13
   Net Improvement: -45.2

2. Test: [sig-olmv1] clusterextension upgrade from v1 to v2
   Pass Rate: 67.8%
   Runs: 28 | Passes: 19 | Failures: 9
   Net Improvement: -12.3

[... additional tests ...]

================================================================================

See Also

Name

ci:payload-experiment

Synopsis

/ci:payload-experiment <payload-tag>

Description

The ci:payload-experiment command opens draft revert PRs for medium-confidence payload candidates (confidence score 60-84) and triggers payload jobs to experimentally determine which PR is causing failures. It operates in two phases separated by a CI wait period.

Phase 1: Reads the payload results YAML, filters medium-confidence candidates, opens draft revert PRs, triggers payload jobs, and appends action entries (type: "experiment", status: "pending") to each candidate's actions array.

Phase 2: Detects candidates with a status: "pending" action entry in the results YAML, checks job results, promotes confirmed causes to real revert PRs (with TRT JIRA bugs), and closes innocent draft PRs.

All state is tracked in the payload results YAML via the payload-results-yaml skill — no separate tracking file is created.

This command is one of three composable stages in the payload triage pipeline:

  1. /ci:payload-analysis — produces the payload results YAML
  2. /ci:payload-revert — stages reverts for HIGH confidence candidates
  3. /ci:payload-experiment — experimentally tests MEDIUM confidence candidates (this command)

Job Triggering Limits

  • Non-aggregated jobs: Up to 5 total across all candidates
  • Aggregated jobs: Up to 1 total

Implementation

  1. Parse the payload tag from the argument. Extract version, stream, and architecture from the tag (see payload-analysis Step 1 for parsing rules).

  2. Read the payload results YAML using the payload-results-yaml skill: Look for payload-results-{tag}.yaml in the current working directory. If not found, print an error and exit:

    Error: Payload results YAML not found for {payload_tag}.
    Run `/ci:payload-analysis {payload_tag}` first to generate it.
    
  3. Detect Phase 2 resume: If the results YAML contains any action entry with type: "experiment" and status: "pending", jump to Phase 2 (step 5). Phase 2 processes only pending experiments — candidates with other statuses are left unchanged.

  4. Phase 1 — Set up experiments: Filter type: "pr" candidates with 60 <= confidence_score < 85. Exclude any that already have an action with status of "open" or "merged" (pre-existing revert). type: "rhcos_rpm" candidates are never experimented on — they cannot be pulled out of a payload like a PR. Dispatch to the payload-experimental-reverts skill Phase 1, which updates the results YAML in place.

  5. Phase 2 — Collect results: Dispatch to the payload-experimental-reverts skill Phase 2 to check job results, promote confirmed causes, close innocent drafts, and update the results YAML.

  6. Report results: Print a summary of actions taken.

Return Value

  • Phase 1: Confirmation that experiments are running, with resume instructions
  • Phase 2: Summary of experiment verdicts (confirmed/innocent) and actions taken. If some experiments are still running, they remain pending and the command can be re-invoked to check again.

Examples

  1. Start experiments after analysis:

    /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806
    /ci:payload-experiment 4.22.0-0.nightly-2026-02-25-152806
    
  2. Resume to collect results (run from the same directory after jobs complete):

    /ci:payload-experiment 4.22.0-0.nightly-2026-02-25-152806
    

Arguments

  • $1: A full payload tag (e.g., 4.22.0-0.nightly-2026-02-25-152806). Must match the tag used with /ci:payload-analysis. (required)

Skills Used

  • payload-results-yaml: Reads and updates the payload results YAML
  • payload-experimental-reverts: Opens draft revert PRs and triggers payload jobs (Phase 1); collects results and acts (Phase 2)
  • trigger-payload-job: Triggers payload jobs and collects URLs
  • revert-pr: Git revert workflow for creating revert PRs

Name

ci:payload-revert

Synopsis

/ci:payload-revert <payload-tag>

Description

The ci:payload-revert command reads the payload results YAML produced by /ci:payload-analysis and stages reverts for all high-confidence candidates (confidence score >= 85) that have not already been reverted.

For each qualifying candidate, it creates a TRT JIRA bug, opens a revert PR, and triggers payload validation jobs using the stage-payload-reverts skill.

This command is one of three composable stages in the payload triage pipeline:

  1. /ci:payload-analysis — produces the payload results YAML
  2. /ci:payload-revert — stages reverts for HIGH confidence candidates (this command)
  3. /ci:payload-experiment — opens draft revert PRs for MEDIUM confidence candidates

Job Triggering Limits

  • Non-aggregated jobs: Up to 5 total across all candidates
  • Aggregated jobs: Up to 1 total

When the number of failing jobs across all candidates exceeds these limits, prioritize jobs from higher-confidence candidates first.

Implementation

  1. Parse the payload tag from the argument. Extract version, stream, and architecture from the tag (see payload-analysis Step 1 for parsing rules).

  2. Read the payload results YAML using the payload-results-yaml skill: Look for payload-results-{tag}.yaml in the current working directory. If not found, print an error and exit:

    Error: Payload results YAML not found for {payload_tag}.
    Run `/ci:payload-analysis {payload_tag}` first to generate it.
    
  3. Filter candidates: Select type: "pr" candidates with confidence_score >= 85. Exclude any that already have an action with status of "open" or "merged" (pre-existing revert). type: "rhcos_rpm" candidates are never reverted — they cannot be pulled out of a payload like a PR; the payload-results-yaml skill's Read Candidates operation excludes them for this reason.

  4. Dispatch to stage-payload-reverts skill: Pass all qualifying candidates with their context (results YAML path, payload tag, version, stream, architecture, release controller URL, and failing jobs). The skill updates the results YAML and HTML report in place. The trigger-payload-job skill validates that aggregated jobs have underlying_job_name set and skips them with an error if not.

  5. Report results: Print a summary of actions taken (JIRA tickets created, revert PRs opened, payload jobs triggered).

Return Value

  • Summary of staged reverts with JIRA keys, revert PR URLs, and triggered payload job URLs
  • Any errors encountered during staging

Examples

  1. Stage reverts after analysis:
    /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806
    /ci:payload-revert 4.22.0-0.nightly-2026-02-25-152806
    

Arguments

  • $1: A full payload tag (e.g., 4.22.0-0.nightly-2026-02-25-152806). Must match the tag used with /ci:payload-analysis. (required)

Skills Used

  • payload-results-yaml: Reads and updates the payload results YAML
  • stage-payload-reverts: Creates TRT JIRA bugs, opens revert PRs, triggers payload jobs
  • trigger-payload-job: Triggers payload jobs and collects URLs
  • revert-pr: Git revert workflow for creating revert PRs

Name

ci:query-job-status

Synopsis

/query-job-status <execution-id>

Description

The query-job-status command queries the status of a gangway job execution via the REST API using the execution ID returned when a job is triggered.

The command accepts:

  • Execution ID (required, UUID returned when triggering a job)

It makes a GET request to the gangway API and returns the current status of the job including its name, type, status, and GCS path to artifacts if available. The curl_with_token.sh wrapper handles all authentication automatically.

Implementation

The command performs the following steps:

  1. Parse Arguments:

    • $1: execution ID (required, UUID format)
  2. Execute API Request: Make a GET request to query the job status using the oc-auth skill's curl wrapper:

    # Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token
    # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443
    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID>
    

    The curl_with_token.sh wrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token.

  3. Display Results: Parse and present the JSON response with:

    • id: The execution ID
    • job_name: The name of the job
    • job_type: The type of job execution (PERIODIC, POSTSUBMIT, PRESUBMIT)
    • job_status: Current status (TRIGGERED, PENDING, SUCCESS, FAILURE, ABORTED)
    • gcs_path: Path to job artifacts in GCS (if available)
    • prow_url: Derived Prow dashboard URL (see step 4)
  4. Derive Prow URL from GCS Path: If gcs_path is present, convert it to a Prow dashboard URL:

    • Strip the gs://<bucket-name>/ prefix (bucket is typically origin-ci-test or test-platform-results-public)
    • Prepend https://prow.ci.openshift.org/view/gs/test-platform-results-public/
    • Example: gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890 → https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890
    • The Prow URL always uses test-platform-results-public as the bucket name regardless of what gcs_path reports
  5. Poll if Prow URL Not Yet Available: If gcs_path is empty/missing and job_status is TRIGGERED, the job has not started yet. Automatically re-query:

    • Wait 15 seconds, then re-query the execution status
    • Repeat up to 20 times (~5 minutes total)
    • Stop polling once gcs_path is populated or job_status reaches a terminal state (SUCCESS, FAILURE, ABORTED)
    • If polling exhausts all retries without a gcs_path, report the current status and provide a Prow dashboard search link so the user can find the job manually: https://prow.ci.openshift.org/?job=<job_name> (using the job_name from the status response)
  6. Offer Follow-up Actions:

    • If status is PENDING with prow_url: Provide the Prow dashboard link (job is actively running)
    • If status is SUCCESS or FAILURE with prow_url: Provide the link and offer to help access logs/artifacts
    • If prow_url could not be derived: Provide https://prow.ci.openshift.org/?job=<job_name> so the user can find it on the Prow dashboard

Return Value

  • Success: JSON response with job status details and derived Prow URL
  • Error: HTTP error, authentication failure, or invalid execution ID

Important for Claude:

  1. REQUIRED: Before executing this command, you MUST ensure the ci:oc-auth skill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active.
  2. You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
  3. Parse the JSON response and present it in a readable format
  4. Highlight the job status prominently
  5. Always derive and display the Prow URL when gcs_path is available
  6. If gcs_path is missing and job is not terminal, poll automatically — do not just offer to check again
  7. If PENDING with Prow URL, provide the link — PENDING means the job is actively running
  8. If SUCCESS/FAILURE, indicate completion status and provide the Prow link

Examples

  1. Query status of a triggered job:

    /query-job-status ca249d50-dee8-4424-a0a7-6dd9d5605267
    

    Returns:

    {
      "id": "ca249d50-dee8-4424-a0a7-6dd9d5605267",
      "job_name": "periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn",
      "job_type": "PERIODIC",
      "job_status": "SUCCESS",
      "gcs_path": "gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890"
    }
    

    Claude derives and displays: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890

  2. Query a recently triggered job (auto-polling):

    /query-job-status 8f3a9b2c-1234-5678-9abc-def012345678
    

    First query returns job_status: TRIGGERED with no gcs_path — Claude automatically waits 15 seconds and re-queries until gcs_path appears, then displays the Prow URL.

  3. Check failed job:

    /query-job-status 5a6b7c8d-9e0f-1a2b-3c4d-5e6f7a8b9c0d
    

    Status shows "FAILURE" — Claude displays the Prow dashboard link for log analysis.

Notes

  • Execution ID Format: UUID format (e.g., ca249d50-dee8-4424-a0a7-6dd9d5605267)
  • Job Status Values:
    • TRIGGERED: The ProwJob CR exists but has not been scheduled yet — waiting to transition to PENDING. gcs_path may not be populated yet.
    • PENDING: The job is actively running (despite the name, this means in-progress, not queued)
    • SUCCESS: Job completed successfully (terminal)
    • FAILURE: Job completed with a failure (terminal)
    • ABORTED: Job was cancelled (terminal)
  • Rate Limits: The REST API has rate limits
  • Authentication: Tokens expire and may need to be refreshed via browser login
  • GCS Path: Provides access to job logs and artifacts when available
  • Prow URL Derivation: The Prow URL is always derived using test-platform-results-public as the bucket name, regardless of what bucket name appears in gcs_path (commonly origin-ci-test)
  • Auto-Polling: When gcs_path is not yet available (common immediately after triggering), the command polls automatically every 15 seconds for up to ~5 minutes. Prow can be slow to schedule jobs. Do not just offer to check again — poll proactively.

Arguments

  • $1 (execution-id): The UUID execution ID returned when a job was triggered (required)

Name

ci:query-test-result

Synopsis

/ci:query-test-result <version> <keywords> [sippy-url]

Description

The ci:query-test-result command queries OpenShift CI test results from Sippy based on the OpenShift version and test name keywords. It retrieves test statistics including pass rate, number of runs, failures, and links to failed job runs.

By default, it queries the production Sippy instance at sippy.dptools.openshift.org. You can optionally specify a different Sippy instance URL to query alternative environments (e.g., QE component readiness).

This command is useful for:

  • Checking the health of specific test cases
  • Finding test failure patterns
  • Getting links to failed Prow job runs for debugging
  • Monitoring test regression trends
  • Querying different Sippy instances (production, QE, etc.)

Arguments

  • $1 (version): OpenShift version to query (e.g., "4.21", "4.20", "4.19")
  • $2 (keywords): Keywords to search in test names (e.g., "PolarionID:81664", "olmv1", "sig-storage")
  • $3 (sippy-url) [optional]: Sippy instance base URL. Defaults to "sippy.dptools.openshift.org" if not provided. Examples: "qe-component-readiness.dptools.openshift.org"

Implementation

  1. Parse Arguments

    • Extract version from $1 (e.g., "4.21")
    • Extract keywords from $2 (e.g., "81664" or "olmv1")
    • Extract Sippy URL from $3 if provided, otherwise use default "sippy.dptools.openshift.org"
    • Normalize URL to extract base domain for API endpoint (strip "/sippy-ng/" suffix if present)
    • Add "https://" prefix if not already present
  2. Build Sippy API Request

    • Construct filter JSON for the /api/tests endpoint:
      filters = {
          "items": [
              {
                  "columnField": "name",
                  "not": False,
                  "operatorValue": "contains",
                  "value": keywords
              }
          ],
          "linkOperator": "and"
      }
      
    • Set query parameters:
      • release: The OpenShift version
      • filter: JSON-encoded filter object
      • sort: "asc"
      • sortField: "net_improvement"
  3. Query Test Statistics

    • Construct API endpoint from provided URL: https://{base_url}/api/tests
      • Example: If input is "sippy.dptools.openshift.org", use https://sippy.dptools.openshift.org/api/tests
      • Example: If input is "qe-component-readiness.dptools.openshift.org", use https://qe-component-readiness.dptools.openshift.org/api/tests
    • Make GET request to the constructed endpoint
    • Parse response to extract:
      • Test name
      • Current pass percentage
      • Total runs, passes, failures
      • Net improvement (trend indicator)
  4. Query Failed Job Runs (for each matching test)

    • Calculate the UTC timestamp for 7 days ago:
      from datetime import datetime, timedelta, timezone
      
      start_time = (datetime.now(timezone.utc) - timedelta(days=7)).isoformat().replace("+00:00", "Z")
      
    • Build filter for failed runs:
      filters = {
          "items": [
              {
                  "columnField": "failed_test_names",
                  "operatorValue": "contains",
                  "value": test_name
              },
              {
                  "columnField": "timestamp",
                  "operatorValue": ">",
                  "value": start_time
              }
          ],
          "linkOperator": "and"
      }
      
    • Make GET request to https://{base_url}/api/jobs/runs with parameters:
      • release: version
      • filter: JSON-encoded filter
      • limit: "20"
      • sortField: "timestamp"
      • sort: "desc"
    • Parse Response Structure:
      • Response is a dict with structure: {"rows": [...], "page": N, "page_size": N, "total_rows": N}
      • Extract job runs from response["rows"]
      • Each run contains:
        • timestamp: Unix timestamp in milliseconds
        • brief_name or job: Job name
        • test_grid_url: Link to Prow job details
  5. Format and Display Results

    • Show summary statistics for each matching test
    • List failed job runs with:
      • Timestamp of the failure
      • Job name
      • Clickable Prow URL for each failed run
    • After listing individual runs, provide a summary section:
      • Create a "Failed Prow URLs (for easy copying)" section
      • List all Prow URLs from the failed runs in plain text format (one per line)
      • This allows users to easily copy all URLs at once for further analysis
    • Format output in a clear, readable structure with proper spacing
    • Present URLs as markdown links for easy clicking

Return Value

Format: Formatted text output with:

  • Test name(s) matching the keywords
  • Statistics section showing:
    • Pass Rate (percentage)
    • Total Runs
    • Passes
    • Failures
    • Net Improvement
  • Failed Job Runs section listing (for last 7 days):
    • Sequential numbering (1, 2, 3...)
    • Timestamp (formatted as YYYY-MM-DD HH:MM:SS)
    • Job name (brief name)
    • Clickable Prow URL (as markdown link or plain URL)
  • Failed Prow URLs summary section:
    • Plain text list of all Prow URLs (one per line)
    • Allows easy copying of all URLs for batch analysis

Output Format Example:

Failed Job Runs (Last 7 Days):
1. 2025-11-03 12:12:31 - periodic-ci-openshift-operator-framework-...
   https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...

2. 2025-11-02 12:12:29 - periodic-ci-openshift-operator-framework-...
   https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/...

If no tests match the keywords, inform the user that no results were found. If a test has no failed runs in the last 7 days, display a success message.

Examples

  1. Query by Polarion ID (using default Sippy instance):

    /ci:query-test-result 4.21 81664
    

    Returns test results for tests containing "81664" in version 4.21 from the default production Sippy instance (sippy.dptools.openshift.org).

  2. Query by test signature (using default Sippy instance):

    /ci:query-test-result 4.20 olmv1
    

    Returns all OLMv1-related test results for version 4.20 from the default Sippy instance.

  3. Query from QE Sippy instance (custom URL):

    /ci:query-test-result 4.20 olmv1 qe-component-readiness.dptools.openshift.org
    

    Returns all OLMv1-related test results for version 4.20 from the QE component readiness Sippy instance.

  4. Query by component with custom Sippy URL:

    /ci:query-test-result 4.19 sig-storage sippy.dptools.openshift.org
    

    Returns all storage-related test results for version 4.19 from the specified Sippy instance.

  5. Custom URL variations (all valid formats):

    /ci:query-test-result 4.21 olmv1 sippy.dptools.openshift.org
    /ci:query-test-result 4.21 olmv1 https://sippy.dptools.openshift.org
    

    Both URL formats are accepted and will query the same Sippy instance.

Output Example

====================================================================================================
Test Results for PolarionID: 81664 (Version 4.21)
====================================================================================================

Test Name:
[sig-olmv1][Jira:OLM] clusterextension PolarionID:81664-[Skipped:Disconnected]preflight check

Statistics (Last 7 Days):
  • Pass Rate: 0.00%
  • Total Runs: 6
  • Passes: 0
  • Failures: 6
  • Net Improvement: -100.00

Failed Job Runs (Last 7 Days):
----------------------------------------------------------------------------------------------------

1. 2025-11-03 12:12:31
   Job: periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1
   https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1985198377557561344

2. 2025-11-02 12:12:29
   Job: periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1
   https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1984835985292136448

[... additional failures ...]

----------------------------------------------------------------------------------------------------
Failed Prow URLs (for easy copying):
----------------------------------------------------------------------------------------------------
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1985198377557561344
https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-operator-framework-operator-controller-release-4.21-periodics-e2e-aws-ovn-techpreview-extended-f1/1984835985292136448
[... additional URLs ...]

====================================================================================================

Notes

  • Default Sippy URL: If no Sippy URL is provided, the command uses sippy.dptools.openshift.org by default
  • The command queries data from the last 7 days by default
  • Ensure you can access the Sippy API endpoints
  • Results are sorted by net improvement to show regressed tests first
  • Failed job runs are limited to the most recent 20 occurrences
  • URLs are displayed as clickable links for easy access to Prow job details
  • If multiple tests match the keywords, results for all matches will be displayed
  • URL normalization:
    • The command automatically strips common suffixes like "/sippy-ng/" from the URL
    • It adds "https://" prefix if not provided
    • Both domain-only and full path formats are supported

See Also

Name

ci:revert-pr

Synopsis

/ci:revert-pr <pr-url> <jira-ticket>

Description

The ci:revert-pr command reverts a merged pull request that is breaking CI or nightly payloads. It follows the OpenShift quick-revert policy to restore CI signal.

This command:

  • Extracts the original PR details (title, author, merge commit SHA, base branch)
  • Ensures the user has a fork of the repository
  • Creates a revert branch and performs git revert -m1 of the merge commit
  • Pushes the revert branch to the user's fork
  • Creates a revert PR using the Revertomatic template format
  • Generates a list of CI override commands (/override) for jobs that may need to be bypassed on the revert PR

This command is useful when:

  • A merged PR is causing CI job failures
  • Nightly payloads are blocked by a breaking change
  • Quick reversion is needed to restore CI signal while the original author fixes the issue

Implementation

  1. Parse Arguments: Extract the PR URL, JIRA ticket, and flags

    • PR URL format: https://github.com/{owner}/{repo}/pull/{number}
    • JIRA ticket: e.g., TRT-1234 or OCPBUGS-56789
    • If the user provides additional context about why the revert is needed, capture it for the PR body
    • If the user provides verification details (jobs to run before unrevert), capture those too
  2. Gather Context: Use the JIRA ticket to automatically determine what broke and what needs verification

    • Look up the JIRA ticket using the fetch-jira-issue skill (plugins/ci/skills/fetch-jira-issue/fetch_jira_issue.py) with --format json
    • Extract context from the JIRA issue: Use the issue summary, description, and comments to determine:
      • What broke: Which CI jobs or payloads are failing (e.g., "e2e-aws jobs failing on 4.18 nightly payload")
      • Verification jobs: Which jobs should pass before the original change can be re-landed (e.g., "e2e-aws, e2e-gcp")
    • Compose the context for the revert PR body from the JIRA issue details. Include links to failing jobs or payloads if mentioned in the ticket.
    • Fall back to asking the user only if:
      • The JIRA lookup fails (token not set, network error, ticket not found)
      • The JIRA issue doesn't contain enough information to determine what broke or which jobs to verify
    • If the user provided additional context as inline arguments, combine it with the JIRA-derived context
  3. Perform the Revert: Use the revert-pr skill

    Follow the complete workflow in plugins/ci/skills/revert-pr/SKILL.md, which covers:

    • Extracting PR information (title, author, merge SHA, base branch) via gh
    • Validating the PR is in MERGED state
    • Ensuring the user has a fork of the repository
    • Detecting commit message conventions (e.g., UPSTREAM: <carry>: prefix)
    • Cloning or using an existing local repository
    • Creating the revert branch and performing git revert -m1
    • Handling merge conflicts: resolving simple ones directly, or reverting dependent commits for complex ones
    • Pushing to the user's fork
    • Generating CI override commands (filtering out unoverridable jobs)
    • Creating the revert PR with the Revertomatic template (adapting title format for UPSTREAM carry repos)
  4. Report Results: Display the revert PR URL, override commands, and next steps

    • Link to the revert PR
    • List of override commands that can be used to force the PR in (do NOT post these as a comment on the PR automatically)
    • Instructions for the original author to unrevert

Return Value

  • Revert PR URL: Link to the newly created revert pull request
  • Override Commands: List of /override commands for CI jobs
  • Summary: Brief description of what was reverted and why

Examples

  1. Basic revert with context provided inline:

    /ci:revert-pr https://github.com/openshift/kubernetes/pull/1703 TRT-9999
    

    The command will ask for context and verification details interactively.

  2. Revert with full context:

    /ci:revert-pr https://github.com/openshift/cluster-network-operator/pull/2037 OCPBUGS-12345
    This PR broke all e2e-aws jobs on the 4.18 nightly. Verification: run e2e-aws and e2e-gcp.
    

Arguments

  • $1 (required): GitHub PR URL to revert
    • Format: https://github.com/{owner}/{repo}/pull/{number}
    • The PR must be in MERGED state
  • $2 (required): JIRA ticket tracking the revert
    • Format: JIRA key (e.g., TRT-1234, OCPBUGS-56789)
  • Additional text (optional): Context explaining why the revert is needed and verification details

Prerequisites

  1. GitHub CLI (gh): Must be installed and authenticated

  2. Git: Must be installed

    • Check: which git
  3. GitHub Token: The gh CLI must have repo and read:org permissions

Notes

  • This command follows the Revertomatic PR template format for consistency
  • The original PR author is CC'd in the revert PR body
  • To unrevert, the original author should revert the revert PR and layer a fix commit on top
  • For repos using UPSTREAM: <tag>: commit conventions (e.g., openshift/kubernetes and other repos carrying upstream patches), the commit message and PR title adapt automatically
  • Merge conflicts are handled automatically when possible: trivial conflicts are resolved directly, non-trivial ones trigger reverting dependent commits

See Also

Name

ci:trigger-periodic

Synopsis

/trigger-periodic <job-name> [ENV_VAR=value ...]

Description

The trigger-periodic command triggers a periodic gangway job via the REST API. Periodic jobs run on a schedule but can be manually triggered for testing or urgent runs.

The command accepts:

  • Job name (required, first argument)
  • Environment variable overrides (optional, additional arguments in KEY=VALUE format)

It then constructs and executes the appropriate curl command to trigger the job via the gangway REST API.

Security

IMPORTANT SECURITY REQUIREMENTS:

Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:

  • READ operations: Checking authentication status (oc whoami)
  • TRIGGERING jobs: POST requests to the gangway API to trigger jobs

Claude is EXPLICITLY PROHIBITED from:

  • Modifying cluster resources (deployments, pods, services, etc.)
  • Deleting or altering existing jobs or executions
  • Accessing secrets, configmaps, or sensitive data
  • Making any cluster modifications beyond job triggering
  • Using the token for any purpose other than the specific operations listed above

MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:

  1. Display the complete payload that will be sent
  2. Show the exact curl command that will be executed
  3. Request explicit user confirmation with a clear "yes/no" prompt
  4. Only proceed after receiving affirmative confirmation

Token Usage: The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.

Implementation

The command performs the following steps:

  1. Parse Arguments:

    • First argument is the job name (required)
    • Remaining arguments are environment variable overrides in KEY=VALUE format
    • Note: Variables that need to override multistage parameters should be prefixed with MULTISTAGE_PARAM_OVERRIDE_
  2. Construct API Request: Build the appropriate curl command using the oc-auth skill's curl wrapper:

    Without overrides:

    # Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token
    # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443
    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \
      -d '{"job_name": "<JOB_NAME>", "job_execution_type": "1"}' \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions
    

    With overrides:

    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \
      -d '{"job_name": "<JOB_NAME>", "job_execution_type": "1", "pod_spec_options": {"envs": {"ENV_VAR": "value"}}}' \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions
    

    With multistage parameter override:

    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \
      -d '{"job_name": "periodic-to-trigger", "job_execution_type": "1", "pod_spec_options": {"envs": {"MULTISTAGE_PARAM_OVERRIDE_FOO": "bar"}}}' \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions
    

    The curl_with_token.sh wrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token.

  3. Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.

  4. Execute Request: Only after receiving user confirmation, run the constructed curl command

  5. Display Results: Show the API response including the execution ID

  6. Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:

    • Wait 15 seconds for the job to be scheduled
    • Query the job status: curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID>
    • If gcs_path is present, derive the Prow URL (see below) and display it
    • If gcs_path is missing and status is TRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total)
    • Stop polling once gcs_path is populated or status reaches a terminal state
    • If retries are exhausted without a gcs_path, provide https://prow.ci.openshift.org/?job=<job_name> so the user can find the job on the Prow dashboard

    GCS Path → Prow URL conversion: Strip the gs://<bucket-name>/ prefix and prepend https://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always uses test-platform-results-public regardless of the bucket name in gcs_path.

    Example: gs://origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890 → https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn/1234567890

Return Value

  • Success: JSON response with execution ID and job details
  • Error: HTTP error, authentication failure, or missing job name

Important for Claude:

  1. REQUIRED: Before executing this command, you MUST ensure the ci:oc-auth skill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active.
  2. You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
  3. Parse the JSON response and extract the execution ID
  4. Display the execution ID to the user
  5. Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
  6. Display the Prow dashboard URL once available

Examples

  1. Trigger a periodic job without overrides:

    /trigger-periodic periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn
    
  2. Trigger a periodic job with payload override:

    /trigger-periodic periodic-ci-openshift-release-master-ci-4.14-e2e-aws-ovn RELEASE_IMAGE_LATEST=quay.io/openshift-release-dev/ocp-release:4.18.8-x86_64
    
  3. Trigger with multistage parameter override:

    /trigger-periodic periodic-to-trigger MULTISTAGE_PARAM_OVERRIDE_FOO=bar
    
  4. Trigger with multiple environment overrides:

    /trigger-periodic periodic-ci-job RELEASE_IMAGE_LATEST=quay.io/image:4.18.8 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=3600
    

Notes

  • Job Execution Type: For periodic jobs, always use "1"
  • Rate Limits: The REST API has rate limits; username is recorded in annotations
  • Authentication: Tokens expire and may need to be refreshed via browser login
  • Multistage Overrides: Prefix variables with MULTISTAGE_PARAM_OVERRIDE_ to override multistage job parameters
  • Execution ID: Save the execution ID from the response to query job status later
  • Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before gcs_path is populated. The command automatically polls until the URL is resolved.

Arguments

  • $1 (job-name): The name of the periodic job to trigger (required)
  • $2-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format

Name

ci:trigger-postsubmit

Synopsis

/trigger-postsubmit <job-name> <org> <repo> <base-ref> <base-sha> [ENV_VAR=value ...]

Description

The trigger-postsubmit command triggers a postsubmit gangway job via the REST API. Postsubmit jobs run after code is merged and require repository reference information.

The command accepts:

  • Job name (required)
  • Organization (required, e.g., "openshift")
  • Repository name (required, e.g., "assisted-installer")
  • Base ref/branch (required, e.g., "release-4.12")
  • Base SHA/commit hash (required)
  • Environment variable overrides (optional, additional arguments in KEY=VALUE format)

It constructs the necessary JSON payload with refs structure and executes the curl command to trigger the job via the gangway REST API.

Security

IMPORTANT SECURITY REQUIREMENTS:

Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:

  • READ operations: Checking authentication status (oc whoami)
  • TRIGGERING jobs: POST requests to the gangway API to trigger jobs

Claude is EXPLICITLY PROHIBITED from:

  • Modifying cluster resources (deployments, pods, services, etc.)
  • Deleting or altering existing jobs or executions
  • Accessing secrets, configmaps, or sensitive data
  • Making any cluster modifications beyond job triggering
  • Using the token for any purpose other than the specific operations listed above

MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:

  1. Display the complete payload that will be sent
  2. Show the exact curl command that will be executed
  3. Request explicit user confirmation with a clear "yes/no" prompt
  4. Only proceed after receiving affirmative confirmation

Token Usage: The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.

Implementation

The command performs the following steps:

  1. Parse Arguments:

    • $1: job name (required)
    • $2: organization (required)
    • $3: repository name (required)
    • $4: base ref/branch (required)
    • $5: base SHA (required)
    • $6-$N: environment variable overrides in KEY=VALUE format (optional)
  2. Construct JSON Payload: Build the payload with refs structure:

    Without overrides:

    {
      "job_name": "<JOB_NAME>",
      "job_execution_type": "2",
      "refs": {
        "org": "<ORG>",
        "repo": "<REPO>",
        "base_ref": "<BASE_REF>",
        "base_sha": "<BASE_SHA>",
        "repo_link": "https://github.com/<ORG>/<REPO>"
      }
    }
    

    With overrides:

    {
      "job_name": "<JOB_NAME>",
      "job_execution_type": "2",
      "refs": {
        "org": "<ORG>",
        "repo": "<REPO>",
        "base_ref": "<BASE_REF>",
        "base_sha": "<BASE_SHA>",
        "repo_link": "https://github.com/<ORG>/<REPO>"
      },
      "pod_spec_options": {
        "envs": {"ENV_VAR": "value"}
      }
    }
    
  3. Save JSON to Temporary File: Write the payload to a temp file (e.g., /tmp/postsubmit-spec.json)

  4. Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.

  5. Execute Request: Only after receiving user confirmation, run the curl command using the oc-auth skill's curl wrapper:

    # Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token
    # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443
    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \
      -d @/tmp/postsubmit-spec.json \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions
    

    The curl_with_token.sh wrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token.

  6. Clean Up: Remove the temporary JSON file

  7. Display Results: Show the API response including the execution ID

  8. Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:

    • Wait 15 seconds for the job to be scheduled
    • Query the job status: curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID>
    • If gcs_path is present, derive the Prow URL (see below) and display it
    • If gcs_path is missing and status is TRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total)
    • Stop polling once gcs_path is populated or status reaches a terminal state
    • If retries are exhausted without a gcs_path, provide https://prow.ci.openshift.org/?job=<job_name> so the user can find the job on the Prow dashboard

    GCS Path → Prow URL conversion: Strip the gs://<bucket-name>/ prefix and prepend https://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always uses test-platform-results-public regardless of the bucket name in gcs_path.

    Example: gs://origin-ci-test/logs/branch-ci-openshift-origin-master-images/1234567890 → https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-openshift-origin-master-images/1234567890

Return Value

  • Success: JSON response with execution ID and job details
  • Error: HTTP error, authentication failure, or missing required arguments

Important for Claude:

  1. REQUIRED: Before executing this command, you MUST ensure the ci:oc-auth skill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active.
  2. You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
  3. Validate all required arguments are provided
  4. Parse the JSON response and extract the execution ID
  5. Display the execution ID to the user
  6. Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
  7. Display the Prow dashboard URL once available

Examples

  1. Trigger a postsubmit job without overrides:

    /trigger-postsubmit branch-ci-openshift-assisted-installer-release-4.12-images openshift assisted-installer release-4.12 7336f38f75f91a876313daacbfw97f25dfe21bbf
    
  2. Trigger a postsubmit job with environment override:

    /trigger-postsubmit branch-ci-openshift-origin-master-images openshift origin master abc123def456 RELEASE_IMAGE_LATEST=quay.io/image:latest
    
  3. Trigger with multiple environment overrides:

    /trigger-postsubmit my-postsubmit-job openshift cluster-api-provider-aws master def789ghi012 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=7200 BUILD_ID=custom-123
    

Notes

  • Job Execution Type: For postsubmit jobs, always use "2"
  • Rate Limits: The REST API has rate limits; username is recorded in annotations
  • Authentication: Tokens expire and may need to be refreshed via browser login
  • Refs Structure: The refs object is required for postsubmit jobs to identify the repository and commit
  • Repo Link: Automatically constructed as https://github.com/<org>/<repo>
  • Execution ID: Save the execution ID from the response to query job status later
  • Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before gcs_path is populated. The command automatically polls until the URL is resolved.

Arguments

  • $1 (job-name): The name of the postsubmit job to trigger (required)
  • $2 (org): GitHub organization (e.g., "openshift") (required)
  • $3 (repo): Repository name (e.g., "assisted-installer") (required)
  • $4 (base-ref): Base branch/ref (e.g., "release-4.12", "master") (required)
  • $5 (base-sha): Base commit SHA hash (required)
  • $6-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format

Name

ci:trigger-presubmit

Synopsis

/trigger-presubmit <job-name> <org> <repo> <base-ref> <base-sha> <pr-number> <pr-sha> [ENV_VAR=value ...]

Description

The trigger-presubmit command triggers a presubmit gangway job via the REST API.

WARNING: Triggering presubmit jobs via REST is generally unnecessary and not recommended. Presubmit jobs should typically be triggered using Prow commands like /test and /retest via GitHub interactions. Only use this command if you have a specific reason to trigger via REST API.

The command accepts:

  • Job name (required)
  • Organization (required, e.g., "openshift")
  • Repository name (required, e.g., "origin")
  • Base ref/branch (required, e.g., "master")
  • Base SHA/commit hash (required)
  • Pull request number (required)
  • Pull request SHA/head commit (required)
  • Environment variable overrides (optional, additional arguments in KEY=VALUE format)

It constructs the necessary JSON payload with refs and pulls structure and executes the curl command to trigger the job via the gangway REST API.

Security

IMPORTANT SECURITY REQUIREMENTS:

Claude is granted LIMITED and SPECIFIC access to the app.ci cluster token for the following AUTHORIZED operations ONLY:

  • READ operations: Checking authentication status (oc whoami)
  • TRIGGERING jobs: POST requests to the gangway API to trigger jobs

Claude is EXPLICITLY PROHIBITED from:

  • Modifying cluster resources (deployments, pods, services, etc.)
  • Deleting or altering existing jobs or executions
  • Accessing secrets, configmaps, or sensitive data
  • Making any cluster modifications beyond job triggering
  • Using the token for any purpose other than the specific operations listed above

MANDATORY USER CONFIRMATION: Before executing ANY POST operation (job trigger), Claude MUST:

  1. Display the complete payload that will be sent
  2. Show the exact curl command that will be executed
  3. Request explicit user confirmation with a clear "yes/no" prompt
  4. Only proceed after receiving affirmative confirmation

Token Usage: The app.ci cluster token is used solely for authentication with the gangway REST API. This token grants the same permissions as the authenticated user and must be handled with appropriate care. The curl_with_token.sh wrapper handles all authentication automatically.

Implementation

The command performs the following steps:

  1. Warn User: Display a warning that presubmit jobs should typically use GitHub Prow commands (/test, /retest)

  2. Parse Arguments:

    • $1: job name (required)
    • $2: organization (required)
    • $3: repository name (required)
    • $4: base ref/branch (required)
    • $5: base SHA (required)
    • $6: pull request number (required)
    • $7: pull request SHA (required)
    • $8-$N: environment variable overrides in KEY=VALUE format (optional)
  3. Construct JSON Payload: Build the payload with refs and pulls structure:

    Without overrides:

    {
      "job_name": "<JOB_NAME>",
      "job_execution_type": "3",
      "refs": {
        "org": "<ORG>",
        "repo": "<REPO>",
        "base_ref": "<BASE_REF>",
        "base_sha": "<BASE_SHA>",
        "pulls": [{
          "number": <PR_NUMBER>,
          "sha": "<PR_SHA>",
          "link": "https://github.com/<ORG>/<REPO>/pull/<PR_NUMBER>"
        }]
      }
    }
    

    With overrides:

    {
      "job_name": "<JOB_NAME>",
      "job_execution_type": "3",
      "refs": {
        "org": "<ORG>",
        "repo": "<REPO>",
        "base_ref": "<BASE_REF>",
        "base_sha": "<BASE_SHA>",
        "pulls": [{
          "number": <PR_NUMBER>,
          "sha": "<PR_SHA>",
          "link": "https://github.com/<ORG>/<REPO>/pull/<PR_NUMBER>"
        }]
      },
      "pod_spec_options": {
        "envs": {"ENV_VAR": "value"}
      }
    }
    
  4. Save JSON to Temporary File: Write the payload to a temp file (e.g., /tmp/presubmit-spec.json)

  5. Request User Confirmation: Display the complete JSON payload and curl command to the user, then explicitly ask for confirmation before proceeding. Wait for affirmative user response.

  6. Execute Request: Only after receiving user confirmation, run the curl command using the oc-auth skill's curl wrapper:

    # Use curl_with_token.sh from oc-auth skill - it automatically adds the OAuth token
    # app.ci cluster API: https://api.ci.l2s4.p1.openshiftapps.com:6443
    curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -v -X POST \
      -d @/tmp/presubmit-spec.json \
      https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions
    

    The curl_with_token.sh wrapper retrieves the OAuth token from the app.ci cluster and adds it as an Authorization header automatically, without exposing the token.

  7. Clean Up: Remove the temporary JSON file

  8. Display Results: Show the API response including the execution ID

  9. Poll for Prow URL: After a successful trigger, automatically resolve the Prow dashboard URL:

    • Wait 15 seconds for the job to be scheduled
    • Query the job status: curl_with_token.sh https://api.ci.l2s4.p1.openshiftapps.com:6443 -X GET https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<EXECUTION_ID>
    • If gcs_path is present, derive the Prow URL (see below) and display it
    • If gcs_path is missing and status is TRIGGERED, wait 15 seconds and retry (up to 20 retries, ~5 minutes total)
    • Stop polling once gcs_path is populated or status reaches a terminal state
    • If retries are exhausted without a gcs_path, provide https://prow.ci.openshift.org/?job=<job_name> so the user can find the job on the Prow dashboard

    GCS Path → Prow URL conversion: Strip the gs://<bucket-name>/ prefix and prepend https://prow.ci.openshift.org/view/gs/test-platform-results-public/. The Prow URL always uses test-platform-results-public regardless of the bucket name in gcs_path.

    Example: gs://origin-ci-test/logs/pull-ci-openshift-origin-master-e2e-aws/1234567890 → https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/pull-ci-openshift-origin-master-e2e-aws/1234567890

Return Value

  • Success: JSON response with execution ID and job details
  • Error: HTTP error, authentication failure, or missing required arguments

Important for Claude:

  1. REQUIRED: Before executing this command, you MUST ensure the ci:oc-auth skill is loaded by invoking it with the Skill tool. The curl_with_token.sh script depends on this skill being active.
  2. You must locate and verify curl_with_token.sh before running it, you (Claude Code) have a bug that tries to use the script from the wrong directory!
  3. Display the warning about using GitHub Prow commands instead
  4. Validate all required arguments are provided
  5. Parse the JSON response and extract the execution ID
  6. Display the execution ID to the user
  7. Automatically poll for the Prow URL — do NOT just offer to check status. Poll until the Prow URL is resolved or retries are exhausted.
  8. Display the Prow dashboard URL once available

Examples

  1. Trigger a presubmit job without overrides:

    /trigger-presubmit pull-ci-openshift-origin-master-e2e-aws openshift origin master abc123def456 1234 def456ghi789
    
  2. Trigger a presubmit job with environment override:

    /trigger-presubmit my-presubmit-job openshift installer master 1a2b3c4d 5678 4d5e6f7g RELEASE_IMAGE_INITIAL=quay.io/image:test
    
  3. Trigger with multiple environment overrides:

    /trigger-presubmit pull-ci-test openshift cluster-version-operator master abcdef12 999 fedcba98 MULTISTAGE_PARAM_OVERRIDE_TIMEOUT=5400 TEST_SUITE=custom
    

Notes

  • Recommended Approach: Use GitHub Prow commands (/test <job-name>, /retest) instead of this REST API
  • Job Execution Type: For presubmit jobs, always use "3"
  • Rate Limits: The REST API has rate limits; username is recorded in annotations
  • Authentication: Tokens expire and may need to be refreshed via browser login
  • Refs Structure: The refs object with pulls array is required for presubmit jobs
  • Pull Link: Automatically constructed as https://github.com/<org>/<repo>/pull/<pr-number>
  • Execution ID: Save the execution ID from the response to query job status later
  • Prow URL Delay: The Prow URL may not be available immediately after triggering — the job needs to be scheduled and transition from TRIGGERED to PENDING before gcs_path is populated. The command automatically polls until the URL is resolved.

Arguments

  • $1 (job-name): The name of the presubmit job to trigger (required)
  • $2 (org): GitHub organization (e.g., "openshift") (required)
  • $3 (repo): Repository name (e.g., "origin") (required)
  • $4 (base-ref): Base branch/ref (e.g., "master") (required)
  • $5 (base-sha): Base commit SHA hash (required)
  • $6 (pr-number): Pull request number (required)
  • $7 (pr-sha): Pull request head commit SHA (required)
  • $8-$N (ENV_VAR=value): Optional environment variable overrides in KEY=VALUE format

Agents 2

Role and Expertise

You are an OpenShift CI expert specializing in analyzing and explaining the components of the OpenShift Prow CI system. You have deep knowledge of how jobs, workflows, chains, and refs work together in the openshift/release repository structure.

Analysis Steps

When given a specific job, workflow, chain, or ref name, you will:

  1. Identify the Component Type: Determine whether the input refers to a ci-operator workflow, chain, or ref (step) based on naming conventions and context.

  2. Locate the Definition: Based on the component type, identify where it would be defined:

    • Workflows: ci-operator/step-registry/**/*-workflow.yaml
    • Chains: ci-operator/step-registry/**/*-chain.yaml
    • Refs: ci-operator/step-registry/**/*-ref.yaml
  3. Analyze the Structure: Break down the component into its constituent parts:

    • For workflows: Analyze pre/test/post phases and their chain/ref references
    • For chains: List all the refs and sub-chains in execution order. The ref can be nested in the chain definition file, the nested refs are defined with "as" value. Examples: chain: as: hypershift-setup-nested-management-cluster steps:
      • as: create-management-cluster
    • For refs: Detail the container image, commands, and resource requirements
  4. Explain the Purpose: Provide a clear explanation of what the component accomplishes in the CI pipeline, including:

    • Primary function and goals
    • How it fits into the broader CI/CD process
    • Dependencies and relationships with other components
    • Any special configuration or requirements
  5. Trace Execution Flow: When analyzing workflows or chains, show the complete execution hierarchy and explain the logical flow of operations.

Output Guidelines

Always structure your analysis clearly with headings and bullet points. Use the hierarchy Job → Workflow → Chain → Ref to organize your explanations. List the workflow, chain, ref file with the file name. When you cannot locate a specific component, suggest possible alternatives or explain how the naming convention might differ. Your goal is to make complex OpenShift CI components understandable to developers and operators who need to work with the system.

IMPORTANT: strictly use the output format as the example demonstrates

Use this agent to analyze and list the workflow/chain/ref file with path name in this repository, keep the hierarchy structure for the input workflow, chain or ref.

Output Format Examples

Main Workflow File

Workflow: ci-operator/step-registry/hypershift/aws/reqserving-e2e/hypershift-aws-reqserving-e2e-workflow.yaml

Complete Repository Path Hierarchy

ci-operator/step-registry/hypershift/aws/reqserving-e2e/hypershift-aws-reqserving-e2e-workflow.yaml ├── PRE Phase (Infrastructure Setup) │ ├── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-chain.yaml │ │ ├── ci-operator/step-registry/ipi/conf/ipi-conf-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/ipi-conf-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/telemetry/ipi-conf-telemetry-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/telemetry/ipi-conf-telemetry-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-commands.sh │ │ ├── ci-operator/step-registry/ipi/conf/aws/byo-ipv4-pool-public/ipi-conf-aws-byo-ipv4-pool-public-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/byo-ipv4-pool-public/ipi-conf-aws-byo-ipv4-pool-public-commands.sh │ │ └── ci-operator/step-registry/ipi/install/monitoringpvc/ipi-install-monitoringpvc-ref.yaml │ │ └── ci-operator/step-registry/ipi/install/monitoringpvc/ipi-install-monitoringpvc-commands.sh │ ├── ci-operator/step-registry/aws/provision/iam-user/minimal-permission/aws-provision-iam-user-minimal-permission-chain.yaml │ │ ├── ci-operator/step-registry/ipi/conf/aws/user-min-permissions/ipi-conf-aws-user-min-permissions-ref.yaml │ │ │ └── ci-operator/step-registry/ipi/conf/aws/user-min-permissions/ipi-conf-aws-user-min-permissions-commands.sh │ │ └── ci-operator/step-registry/aws/provision/iam-user/aws-provision-iam-user-ref.yaml │ │ └── ci-operator/step-registry/aws/provision/iam-user/aws-provision-iam-user-commands.sh │ ├── ci-operator/step-registry/sandboxed-containers-operator/aws-region-override/sandboxed-containers-operator-aws-region-override-ref.yaml │ │ └── ci-operator/step-registry/sandboxed-containers-operator/aws-region-override/sandboxed-containers-operator-aws-region-override-commands.sh │ └── ci-operator/step-registry/ipi/install/ipi-install-chain.yaml │ ├── ci-operator/step-registry/ipi/install/rbac/ipi-install-rbac-ref.yaml │ ├── ci-operator/step-registry/openshift/cluster-bot/rbac/openshift-cluster-bot-rbac-ref.yaml │ ├── ci-operator/step-registry/ipi/install/hosted-loki/ipi-install-hosted-loki-ref.yaml │ ├── ci-operator/step-registry/ipi/install/install/ipi-install-install-ref.yaml │ │ └── ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh │ ├── ci-operator/step-registry/ipi/install-times-collection/ipi-install-times-collection-ref.yaml │ ├── ci-operator/step-registry/nodes/readiness/nodes-readiness-ref.yaml │ └── ci-operator/step-registry/multiarch/validate-nodes/multiarch-validate-nodes-ref.yaml ├── TEST Phase (Request Serving E2E Tests) │ └── ci-operator/step-registry/hypershift/aws/run-reqserving-e2e/hypershift-aws-run-reqserving-e2e-ref.yaml │ └── ci-operator/step-registry/hypershift/aws/run-reqserving-e2e/hypershift-aws-run-reqserving-e2e-commands.sh └── POST Phase (Cleanup) ├── ci-operator/step-registry/gather/core-dump/gather-core-dump-chain.yaml │ └── ci-operator/step-registry/gather/core-dump/gather-core-dump-ref.yaml └── ci-operator/step-registry/ipi/aws/post/ipi-aws-post-chain.yaml ├── ci-operator/step-registry/gather/aws-console/gather-aws-console-ref.yaml ├── ci-operator/step-registry/ipi/deprovision/ipi-deprovision-chain.yaml │ ├── ci-operator/step-registry/gather/gather-chain.yaml │ │ ├── ci-operator/step-registry/gather/must-gather/gather-must-gather-ref.yaml │ │ ├── ci-operator/step-registry/gather/extra/gather-extra-ref.yaml │ │ └── ci-operator/step-registry/gather/audit-logs/gather-audit-logs-ref.yaml │ └── ci-operator/step-registry/ipi/deprovision/deprovision/ipi-deprovision-deprovision-ref.yaml └── ci-operator/step-registry/aws/deprovision/users-and-policies/aws-deprovision-users-and-policies-ref.yaml

File Summary

Total Files: 44 files

  • 1 workflow file: Main orchestration
  • 7 chain files: AWS config, IAM setup, cluster install, cleanup phases
  • 22 ref files: Individual configuration and execution steps
  • 14 command files: Associated shell scripts

Execution Flow: AWS Configuration → IAM Setup → Cluster Installation → Request Serving E2E Tests → Core Dump Collection → Full AWS Cleanup

You are the OpenShift Test Porter — an agent that ports Ginkgo e2e tests from openshift-tests-private to openshift/origin, then shepherds the resulting PRs through CI and review.

Repos

  • Source: openshift-tests-private (branch: porting-prep) — tests annotated with // port=yes|no|maybe|complete
  • Destination: openshift/origin — PRs created here (or on a fork like your-fork/origin)

You need both repos cloned locally. Ask the user for paths if you can't find them. Common locations:

  • ~/go/src/github.com/openshift/openshift-tests-private
  • ~/go/src/github.com/openshift/origin

The gh CLI must be authenticated.

What You Can Do

You handle every phase of the test porting lifecycle. The user will tell you what to do in natural language:

  1. Port tests — find // port=yes tests, adapt them, create a PR (noting AI assistance in the description)
  2. Check PR status — look at CI results, summarize what's passing/failing
  3. Fix CI failures — read build errors or test failures, push fixes
  4. Respond to CodeRabbit — address automated review feedback, push fixes
  5. Hand off for human review — once CI is green and CodeRabbit is addressed, notify humans the PR is ready
  6. Respond to human review — make requested improvements or drop tests as reviewers direct
  7. Check test results — query whether ported tests are showing up and passing in CI
  8. Escalate — tag a human reviewer when you're stuck

Porting Rules

Test Selection

  • Only port tests marked // port=yes — never port=no, port=maybe, port=complete, port=skipped

Code Adaptation

Replace the compat_otp compatibility layer with exutil equivalents:

Source (openshift-tests-private)Destination (origin)
compat_otp.NewCLI(...)exutil.NewCLI(...)
compat_otp.NewCLIWithoutNamespace(...)exutil.NewCLIWithoutNamespace(...)
compat_otp.By(...)g.By(...)
compat_otp.FixturePath(...)exutil.FixturePath(...)
compat_otp.KubeConfigPath()Remove — exutil.NewCLI doesn't take this
import compat_otp "..."import exutil "github.com/openshift/origin/test/extended/util"
import "github.com/openshift/openshift-tests-private/test/extended/util"import exutil "github.com/openshift/origin/test/extended/util"

Keep e2e, g (ginkgo), and o (gomega) imports as-is.

Test Name Adaptation

  • Remove the Author:xxx- prefix
  • Add an [OTP] tag to the test name
  • Keep sig tags, feature tags, severity tags, and variant tags
  • Add [apigroup:xxx] tags if needed by origin

Example: Author:jsmith-High-53223-Verify ACL audit logs becomes [OTP] Verify ACL audit logs

Destination Package Mapping

Map source subdirectory and sig tag to the appropriate origin/test/extended/ package:

  • [sig-network] → networking/
  • [sig-storage] → storage/
  • [sig-auth] → authentication/ or authorization/
  • [sig-apps] → deployments/ or apps/
  • [sig-api-machinery] → apiserver/
  • [sig-cluster-lifecycle] → cluster/
  • [sig-imageregistry] → image_registry/
  • [sig-node] → node/
  • [sig-instrumentation] → prometheus/

If a directory with the same name exists in origin, use it. If no match, create a new package and register it in test/extended/include.go with a blank import.

Helper Functions and Fixtures

  1. First look for an equivalent in origin's existing utilities
  2. If no equivalent and the helper is small (< 50 lines), copy it to the destination
  3. Copy fixture files (YAML, JSON) to the corresponding testdata/ directory

Compilation and Verification

After porting, run these in order in the origin repo and iterate on failures:

  1. go build ./test/extended/... — fix compile errors first
  2. make update-bindata — regenerate bindata for any new fixture files
  3. make verify — fix any linting, formatting, or generated-file issues

PR Creation

When creating a PR on origin:

BRANCH="port-tests-$(date +%Y%m%d-%H%M%S)"
git checkout -b "$BRANCH"
git add -A
git commit -m "Port $COUNT tests from openshift-tests-private [OTP]

Ported tests:
- <list each test name and source file>"
git push origin "$BRANCH"

Create the PR with gh pr create. The PR description must include:

  1. A summary table of ported and skipped tests
  2. The following workflow notice at the top of the description:
> **AI-Ported Tests** — These tests were ported from `openshift-tests-private` with AI assistance.
> The AI agent will respond to CodeRabbit review feedback, verify CI is passing, and then
> hand off the PR for human review. Human reviewers may request improvements or ask that
> specific tests be dropped — the agent will act on that feedback. Tests that cannot be
> made to pass or that reviewers deem unsuitable will be removed from the PR.

Source Annotation Updates

Do not mark tests as port=complete when the PR is first created. Only update // port=yes to // port=complete when:

  • The origin PR has been merged, or
  • The user explicitly tells you to mark them complete

Mark tests as // port=skipped when:

  • A human reviewer requests the test be dropped from the PR during review

When updating, create a branch and PR against openshift/openshift-tests-private (the upstream repo, not a fork). The PR should reference the origin PR and list which tests were marked complete or skipped (and why, for skipped tests).

CI Monitoring and Fixes

CI jobs on openshift/origin PRs can take up to 5 hours to complete after a push. There are typically 15-20 Prow jobs per PR. Don't panic about individual job failures — they're common and often unrelated to the ported tests.

Checking CI Status

Use gh pr checks <PR> --repo openshift/origin to get a summary. This shows each job's name, state (pass/fail/pending), and Prow URL. Example output:

ci/prow/e2e-aws-csi          fail   https://prow.ci.openshift.org/view/gs/...
ci/prow/e2e-gcp-ovn          pass   https://prow.ci.openshift.org/view/gs/...
ci/prow/unit                  pass   https://prow.ci.openshift.org/view/gs/...

Ignore the tide check — it reflects merge eligibility, not test results.

If jobs are still pending, tell the user how many are complete vs pending and suggest checking back later.

Triaging Failures

A failing job does not necessarily mean the ported test is at fault. When a job fails:

  1. Open the Prow URL for the failed job
  2. Look at the junit test results in the job artifacts to find which specific tests failed
  3. Determine whether any of the failing tests are:
    • Our ported test (contains [OTP] in the name) — this is a real problem, investigate and fix
    • Other tests that started failing after our change — could indicate our code broke something, investigate
    • Tests that commonly fail on this job regardless of the PR (known flakes) — likely not our fault

If a failure is clearly a known flake unrelated to our changes, note it but don't fix it.

Fixing CI Failures

  1. Read the failing test output and the relevant code
  2. Push a fix commit to the PR branch
  3. After 3 failed fix attempts on the same issue, stop and tell the user you need help — explain what you've tried and what's failing

Overall CI Health Assessment

When reporting CI status to the user, summarize:

  • Total jobs: N passed, N failed, N pending
  • For each failure: job name, whether it's related to our ported test, and brief reason
  • Overall verdict: "CI looks healthy" / "has failures but unrelated to our tests" / "our test is failing, needs a fix"

Review Handling

CodeRabbit Review

CodeRabbit will automatically review the PR. Respond to its feedback proactively:

  1. Read CodeRabbit's review comments using gh api repos/{owner}/{repo}/pulls/{number}/comments
  2. Address valid suggestions (style fixes, missing error handling, etc.) by pushing fix commits
  3. For suggestions that don't apply to ported tests or conflict with origin conventions, reply explaining why
  4. After addressing CodeRabbit feedback and CI is green, post a PR comment indicating the PR is ready for human review

Human Review

Humans are the final authority on ported tests. When a human reviewer comments:

  1. Use gh api repos/{owner}/{repo}/pulls/{number}/comments to read review comments
  2. Use gh pr view <PR> --comments for conversation-level comments
  3. Improvement requests: make the requested changes and push a new commit
  4. Drop requests: if a reviewer asks to drop a test, remove it from the PR entirely — remove the test code, any helper functions or fixtures only used by that test, and push a commit noting which test was dropped and why. Mark dropped tests as // port=skipped in the source repo (not port=complete)
  5. Reply to each review comment confirming what action was taken
  6. After addressing all feedback, post a summary comment listing changes made

Escalation and Handoff

Ready for Human Review

When CodeRabbit feedback is addressed and CI is passing, hand off the PR:

  1. Post a PR comment summarizing the state:
    • Which tests were ported and are passing
    • Any CodeRabbit feedback that was addressed
    • Any tests that were dropped during CI fixes (and why)
  2. Tag the user or a specified reviewer
  3. Reviewers can request improvements or ask to drop tests, and the agent will act on that feedback

Stuck or Exhausted Fix Attempts

When you've exhausted your fix attempts:

  • Tag the user or a specified reviewer on the PR with a comment summarizing the state
  • List what's passing, what's failing, and what you've tried

General Behavior

  • Never force-push
  • Never merge PRs — leave them open for human review
  • Create separate commits for each logical change (initial port, CI fixes, review feedback)
  • Be concise in PR descriptions and commit messages
  • When uncertain about a porting decision, explain the tradeoff and ask the user

MCP servers 1

Sponsors

Details

Marketplace
openshift-eng/ai-helpers
Source
plugins/ci
Category
Ci
Keywords
prow, ci, openshift-ci, test-failures, job-analysis

More from openshift-eng/ai-helpers

All 56