OpenShift development utilities and helpers

version
0.0.8
GitHub stars
123

Install

/plugin install openshift@ai-helpers

Run it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.

Contents

Discovered from source

Skills 2

Quick Start - OVN Topology Generation

IMMEDIATE ACTIONS (follow these steps in order):

  1. Detect Cluster: Find the OVN-Kubernetes cluster kubeconfig

    Run: scripts/detect-cluster.sh 2>/dev/null

    The script discovers OVN-Kubernetes clusters:

    • Scans all kubeconfig files: current KUBECONFIG env, ~/.kube/kind-config, ~/ovn.conf, ~/.kube/config
    • Tests ALL contexts in each kubeconfig (not just current-context)
    • Returns parseable list to stdout: index|kubeconfig|cluster_name|node_count|namespace
    • Diagnostics go to stderr
    • Exit code: 0=success, 1=no clusters found

    How to handle the output:

    The script returns pipe-delimited lines to stdout, one per cluster found, e.g.:

    1|/home/user/.kube/kind-config|kind-ovn|3|ovn-kubernetes
    2|/home/user/.kube/config|prod-cluster|12|openshift-ovn-kubernetes
    

    Decision logic:

    • If one cluster found → automatically use it (extract kubeconfig path from column 2)
    • If multiple clusters found → show the list to user and ask them to choose by number
    • After selection, extract the kubeconfig path from column 2 of the chosen line
    • Store the selected kubeconfig path in variable KC for use in subsequent steps

    Example output format parsing:

    • Column 1: Index number (for user selection)
    • Column 2: Kubeconfig file path (this is what you need for $KC)
    • Column 3: Cluster display name
    • Column 4: Number of nodes
    • Column 5: OVN namespace name

    Important: Parse the output using standard text processing. The exact implementation is up to you - use whatever approach works best (awk, Python, inline parsing, etc.).

  2. Check Permissions: Verify user's Kubernetes access level and inform about write permissions

    Run: scripts/check_permissions.py "$KC"

    The script returns:

    • Exit 0: Read-only access or user confirmed → proceed
    • Exit 1: Error or user cancelled → stop
    • Exit 2: Write permissions detected → AI must ask user for confirmation

    When exit code 2 is returned:

    1. Parse the stdout to get the list of write permissions
    2. Display the permissions clearly to the user using a formatted message
    3. Explain that:
      • This skill performs ONLY read-only operations
      • No cluster modifications will be made
      • The warning is for transparency about their access level
      • List read-only operations: kubectl get, kubectl exec (ovn-nbctl list), local file writes
      • List forbidden operations: kubectl create/delete/patch, ovn-nbctl modifications
    4. Ask the user explicitly: "You have cluster admin permissions. This command will only perform read-only operations. Do you want to proceed?"
    5. If user says yes → continue, if no → stop

    Example of proper user communication:

    ⚠️  WARNING: Write Permissions Detected
    
    Your kubeconfig has cluster admin permissions:
      • Delete pods, deployments, services
      • Create and modify resources
      • Full cluster access
    
    📋 IMPORTANT:
    This command will ONLY perform read-only operations:
      ✅ kubectl get (pods, nodes)
      ✅ kubectl exec (to run read-only ovn-nbctl list commands)
      ✅ Local file writes (topology diagram)
    
    Operations that will NEVER be performed:
      ❌ kubectl create/delete/patch/apply
      ❌ ovn-nbctl modifications
      ❌ Any cluster state changes
    
    Do you want to proceed with read-only topology generation?
    

    Security Note: This step ensures informed consent. The user must be explicitly aware that their cluster admin credentials are accessible to the AI agent (acting on their behalf), even though only read-only operations will be performed. This transparency is critical for security and trust.

  3. Check Output File: Ask user if ovn-topology-diagram.md exists:

    • (1) Overwrite, (2) Custom path, (3) Timestamp, (4) Cancel
  4. Create Private Temp Directory: Create a private temporary directory using mkdtemp and use it for all temporary files.

    TMPDIR=$(mktemp -d)
    
  5. Collect OVN Data: Get full topology data from the cluster

    Run: scripts/collect_ovn_data.py "$KC" "$TMPDIR"

    Detail files written to $TMPDIR:

    • ovn_switches_detail.txt - node|uuid|name|other_config
    • ovn_routers_detail.txt - node|uuid|name|external_ids|options
    • ovn_lsps_detail.txt - node|name|addresses|type|options
    • ovn_lrps_detail.txt - node|name|mac|networks|options
    • ovn_pods_detail.txt - namespace|name|ip|node
  6. Analyze Placement: Determine per-node vs cluster-wide components

    Run: scripts/analyze_placement.py "$TMPDIR"

    Placement results written to $TMPDIR:

    • ovn_switch_placement.txt - name|placement (per-node|cluster-wide|cluster-wide-visual)
    • ovn_router_placement.txt - name|placement (per-node|cluster-wide|cluster-wide-visual)
  7. Generate Diagram: Create Mermaid graph BT diagram

    • Read $TMPDIR/ovn_switch_placement.txt to determine where each switch goes
    • Read $TMPDIR/ovn_router_placement.txt to determine where each router goes
    • Read detail files directly (ovn_switches_detail.txt, ovn_routers_detail.txt, etc.)
    • Skip UUID column when parsing switches/routers detail files
    • If placement is per-node → put inside node subgraph
    • If placement is cluster-wide or cluster-wide-visual → put outside subgraphs
  8. Save & Report: Write diagram to file, show summary, clean up temporary files

CRITICAL RULES:

  • ❌ NO codebase searching for IPs/MACs
  • ❌ NO synthetic/example data
  • ❌ NO inline multi-line bash (use helper scripts)
  • ❌ NO direct kubectl commands (must use helper scripts only)
  • ✅ Use helper scripts for all kubectl interactions and architecture discovery
  • ✅ For helper scripts only: If kubectl is required, use KUBECONFIG="$KC" kubectl --kubeconfig="$KC"
  • ✅ SECURITY: Create private temp directory with TMPDIR=$(mktemp -d) - never use /tmp directly
  • ✅ Temporary files use $TMPDIR (private directory created with mkdtemp)
  • ✅ Clean up temporary files when done: rm -rf "$TMPDIR"

Safety & Security Guarantees

🔒 Read-Only Operations

This skill performs ONLY read-only operations against your Kubernetes cluster. No cluster state is modified.

Allowed Operations:

  • ✅ kubectl get - Query resources
  • ✅ kubectl exec ... ovn-nbctl list - Query OVN database (read-only)
  • ✅ Local file writes (temporary files in $TMPDIR, output diagram)

Forbidden Operations (NEVER used):

  • ❌ kubectl create/apply/delete/patch - No resource modifications
  • ❌ kubectl scale/drain/cordon - No cluster operations
  • ❌ ovn-nbctl create/set/add/remove/destroy - No OVN modifications
  • ❌ No pod restarts or service disruptions

Privacy Consideration: The generated diagram contains network topology information. Control sharing based on your organization's security policies and data classification requirements.


Architecture Concepts

Interconnect Mode (Distributed NBDB)

In interconnect mode, each node runs its own NBDB with local copies of components:

Per-Node Components (different UUIDs on each node):

  • ovn_cluster_router - Each node has its own cluster router instance
  • join switch - Each node has its own join switch instance
  • transit_switch - Each node has its own transit switch instance
  • Node switches (e.g., ovn-control-plane, ovn-worker)
  • External switches (e.g., ext_ovn-control-plane)
  • Gateway routers (e.g., GR_ovn-control-plane)

Visualization Overrides:

  • transit_switch: PER-NODE in reality → shown CLUSTER-WIDE for visualization clarity
  • join: PER-NODE → kept PER-NODE (no override)

Helper Scripts

All helper scripts are in the scripts/ directory.

ScriptPurposeInputOutput
detect-cluster.shFind OVN cluster kubeconfig across all contexts. Scans multiple kubeconfig files and all their contexts. Returns parseable list.NoneParseable list to stdout: index|kubeconfig|cluster|nodes|namespace. Exit: 0=success, 1=none found
check_permissions.pyCheck user permissions and warn if write access detected.KUBECONFIG pathExit: 0=proceed, 1=cancelled/error, 2=write perms (needs user confirmation)
collect_ovn_data.pyData collector: Queries each node for all data, with graceful degradation (continues on node failures). Writes detail files.KUBECONFIG path, TMPDIRDetail files: ovn_switches_detail.txt, ovn_routers_detail.txt, ovn_lsps_detail.txt, ovn_lrps_detail.txt, ovn_pods_detail.txt
analyze_placement.pyPlacement analyzer: Analyzes UUID patterns from detail files to determine per-node vs cluster-wide placement.TMPDIR (reads detail files)Placement files: ovn_switch_placement.txt, ovn_router_placement.txt

Diagram Generation Rules

Structure

graph BT
    subgraph node1["<b style='color:black'>Node: name (node_ip)</b>"]
        direction BT

        %% LAYER 1 (Bottom): Pods and Management Ports
        POD_example["Pod: pod-name<br/>Namespace: ns<br/>IP: x.x.x.x"]
        MGMT["Management Port: k8s-node<br/>IP: x.x.x.x"]

        %% LAYER 2: Pod LSPs
        LSP_pod["LSP: namespace_pod-name<br/>MAC: xx:xx:xx:xx:xx:xx<br/>IP: x.x.x.x"]
        LSP_mgmt["LSP: k8s-node<br/>MAC: xx:xx:xx:xx:xx:xx<br/>IP: x.x.x.x"]

        %% LAYER 3: Node Switch
        LS_node["Logical Switch: node-name<br/>Subnet: x.x.x.x/24"]

        %% LAYER 4: Node Switch LSPs
        LSP_stor["LSP: stor-node<br/>Type: router"]

        %% LAYER 5: Cluster Router
        LR_cluster["Logical Router: ovn_cluster_router"]
        LRP_rtos["LRP: rtos-node<br/>MAC: xx:xx<br/>IP: x.x.x.x/24"]
        LRP_rtoj["LRP: rtoj-ovn_cluster_router<br/>IP: x.x.x.x/16"]
        LRP_rtots["LRP: rtots-node<br/>IP: x.x.x.x/16"]

        %% LAYER 6: Join Switch
        LS_join["Logical Switch: join"]
        LSP_jtor_cr["LSP: jtor-ovn_cluster_router<br/>Type: router"]
        LSP_jtor_gr["LSP: jtor-GR_node<br/>Type: router"]

        %% LAYER 7: Gateway Router
        LR_gr["Gateway Router: GR_node"]
        LRP_rtoj_gr["LRP: rtoj-GR_node<br/>IP: x.x.x.x/16"]
        LRP_rtoe["LRP: rtoe-GR_node<br/>IP: x.x.x.x/24"]

        %% LAYER 8: External Switch
        LS_ext["Logical Switch: ext_node"]
        LSP_etor["LSP: etor-GR_node<br/>Type: router"]
        LSP_breth0["LSP: breth0_node<br/>Type: localnet"]

        %% LAYER 9 (Top): External Network
        EXT_NET["External Network<br/>Physical bridge: breth0<br/>Node IP: x.x.x.x"]

        %% Connections (bottom-to-top flow)
        POD_example --> LSP_pod --> LS_node
        MGMT --> LSP_mgmt --> LS_node
        LS_node --> LSP_stor
        LSP_stor -.->|peer| LRP_rtos --> LR_cluster
        LR_cluster --> LRP_rtoj -.->|peer| LSP_jtor_cr --> LS_join
        LS_join --> LSP_jtor_gr -.->|peer| LRP_rtoj_gr --> LR_gr
        LR_gr --> LRP_rtoe -.->|peer| LSP_etor --> LS_ext
        LS_ext --> LSP_breth0 -.->|physical| EXT_NET
        LR_cluster --> LRP_rtots
    end

    %% Cluster-wide components (AFTER nodes to appear on top)
    %% Only include components with placement=cluster-wide or cluster-wide-visual
    %% Example: transit_switch in interconnect mode
    LS_cluster_component["Component Name<br/>Details"]

    %% Connections from nodes to cluster-wide components
    LRP_from_node -.->|connects to| LSP_cluster_port --> LS_cluster_component

Key Requirements

  1. Graph Direction: Always graph BT (bottom-to-top)

  2. Component Placement: Determined by $TMPDIR/ovn_*_placement.txt

    • per-node → INSIDE node subgraph
    • cluster-wide or cluster-wide-visual → OUTSIDE all subgraphs, defined AFTER all node subgraphs
    • CRITICAL: Define ALL cluster-wide components AFTER all nodes to position them at the TOP
      • Prevents connection lines from overlapping with node subgraphs
      • Applies to ANY component with cluster-wide or cluster-wide-visual placement
  3. Node Subgraphs: Each physical node gets a subgraph with direction BT

    • Node Ordering: ALWAYS order nodes as: control-plane node first, then worker nodes sorted alphabetically by name
    • Title format: "<b style='color:black'>Node: {node_name} ({external_ip})</b>"
    • Get external IP from rtoe-GR_{node} router port network field
    • Use different background colors for each node for visual distinction
    • CRITICAL: Define components in BOTTOM-TO-TOP order (matches packet flow):
      1. Bottom Layer: Pods and Management Ports (traffic originates here)
      2. Layer 2: Pod LSPs
      3. Layer 3: Node Switch (where pods connect)
      4. Layer 4: Node Switch LSPs (stor, breth0)
      5. Layer 5: Cluster Router + Router Ports (rtos, rtoj, rtots)
      6. Layer 6: Join Switch + Join LSPs
      7. Layer 7: Gateway Router + Router Ports (rtoj, rtoe)
      8. Layer 8: External Switch + External LSPs (etor, breth0)
      9. Top Layer: External Network (physical bridge)
  4. Pod Representation:

    • CRITICAL: Show ALL pods from $TMPDIR/ovn_pods_detail.txt as SEPARATE entities
    • DO NOT discover pods from LSPs - many pods (host-network) don't have individual LSPs
    • Pod format: POD_{id}["Pod: {name}<br/>Namespace: {ns}<br/>IP: {ip}"]
    • Connect pods to their respective LSPs:
      • Host-network pods (IP == node IP): POD_{id} --> MGMT_{node} --> LSP_k8s_{node}
      • Pod-network pods (IP in 10.244.x.x): POD_{id} --> LSP_{namespace}_{podname}
  5. Physical Network Layer:

    • Show explicit external network entities per node
    • Format: EXT_NET_{node}["External Network<br/>Physical bridge: breth0<br/>Node IP: {external_ip}"]
    • Connect localnet LSPs to external network: LSP_breth0_{node} -.->|physical| EXT_NET_{node}
  6. Colors (apply using classDef with color:#000 for black text):

    • Pods: fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000
    • Switches: fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#000
    • Routers: fill:#fff9c4,stroke:#f57f17,stroke-width:2px,color:#000
    • LSPs (ALL types): fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#000
    • LRPs: fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#000
    • External Network: fill:#e0e0e0,stroke:#424242,stroke-width:2px,color:#000
    • Management Ports (MGMT entities only, not LSPs): fill:#fff8e1,stroke:#f57c00,stroke-width:2px,color:#000
    • Node Subgraph Backgrounds (use style statements, NOT classDef): Apply at END of diagram after all class assignments. Rotate through these 3 colors:
      • Node 1 (index 0): style node1 fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,stroke-dasharray: 5 5,color:#000 (green)
      • Node 2 (index 1): style node2 fill:#e1f5fe,stroke:#0277bd,stroke-width:3px,stroke-dasharray: 5 5,color:#000 (blue)
      • Node 3 (index 2): style node3 fill:#fff3e0,stroke:#ef6c00,stroke-width:3px,stroke-dasharray: 5 5,color:#000 (orange)
      • Node 4+ repeats: Use index % 3 to rotate colors (0=green, 1=blue, 2=orange)
  7. Required Info:

    • Pods: Name, Namespace, IP
    • Switches: Name, Subnet (from other_config), Gateway (optional)
    • LSPs: Name, MAC, IP (or Type + Router Port for router ports)
    • LRPs: Name, MAC, Network (IP/CIDR)
    • Routers: Name, Description
    • External Network: Physical bridge name (breth0), Node IP
  8. Connections:

    • Solid arrows: Layer connections
    • Dashed arrows with |peer|: Peer port relationships

Pod Discovery

CRITICAL: Discover pods from $TMPDIR/ovn_pods_detail.txt, NOT from LSPs

The file $TMPDIR/ovn_pods_detail.txt contains ALL running pods in the cluster (populated by collect_ovn_data.py). Format: namespace|pod_name|pod_ip|node_name

Pod-to-LSP Mapping Logic:

  1. Read ALL pods from $TMPDIR/ovn_pods_detail.txt

  2. For each pod, determine which LSP it connects to:

    a. Host-Network Pods (pod IP == node IP, e.g., 10.89.0.10):

    • Connect to management port LSP: k8s-{node}
    • These pods share the management port's IP address
    • MUST be included in diagram despite not having individual LSPs

    b. Pod-Network Pods (pod IP in pod network range, e.g., 10.244.x.x):

    • Connect to individual LSP: {namespace}_{pod-name-with-hash}
    • LSP name format: kube-system_coredns-674b8bbfcf-qhfrq
    • Filter LSPs where type="" (empty string)
    • Extract MAC and IP from LSP addresses field
  3. Diagram representation:

    POD_id["Pod: {name}<br/>Namespace: {ns}<br/>IP: {ip}"]
    POD_id --> LSP_id
    

Special LSPs (NOT pods, treat as infrastructure):

  • k8s-{node}: Management port LSP (multiple host-network pods connect to this)
  • stor-{node}: Router port (type="router")
  • breth0_{node}: LocalNet port (type="localnet")
  • jtor-*, etor-*, tstor-*: Router ports (type="router")

Example: Control Plane Node with Host-Network Pods

%% 8 host-network pods sharing management port
POD_etcd["Pod: etcd-ovn-control-plane<br/>Namespace: kube-system<br/>IP: 10.89.0.10"]
POD_apiserver["Pod: kube-apiserver-ovn-control-plane<br/>Namespace: kube-system<br/>IP: 10.89.0.10"]
POD_controller["Pod: kube-controller-manager-ovn-control-plane<br/>Namespace: kube-system<br/>IP: 10.89.0.10"]
POD_scheduler["Pod: kube-scheduler-ovn-control-plane<br/>Namespace: kube-system<br/>IP: 10.89.0.10"]
POD_ovnkube_cp["Pod: ovnkube-control-plane-ovn-control-plane<br/>Namespace: ovn-kubernetes<br/>IP: 10.89.0.10"]
POD_ovnkube_id["Pod: ovnkube-identity-ovn-control-plane<br/>Namespace: ovn-kubernetes<br/>IP: 10.89.0.10"]
POD_ovnkube_node["Pod: ovnkube-node-xyz<br/>Namespace: ovn-kubernetes<br/>IP: 10.89.0.10"]
POD_ovs_node["Pod: ovs-node-xyz<br/>Namespace: ovn-kubernetes<br/>IP: 10.89.0.10"]

%% Management port LSP (shared by all host-network pods)
MGMT_cp["Management Port: k8s-ovn-control-plane<br/>IP: 10.89.0.10"]
LSP_mgmt["LSP: k8s-ovn-control-plane<br/>MAC: xx:xx:xx:xx:xx:xx<br/>IP: 10.244.0.2"]

%% All host-network pods connect to same management port
POD_etcd --> MGMT_cp
POD_apiserver --> MGMT_cp
POD_controller --> MGMT_cp
POD_scheduler --> MGMT_cp
POD_ovnkube_cp --> MGMT_cp
POD_ovnkube_id --> MGMT_cp
POD_ovnkube_node --> MGMT_cp
POD_ovs_node --> MGMT_cp

MGMT_cp --> LSP_mgmt --> LS_node

Final Steps

  1. Generate complete Mermaid diagram following structure above
  2. Save to file chosen by user
  3. Show summary: nodes, switches, routers, ports, mode
  4. Clean up temporary directory:
    rm -rf "$TMPDIR"
    
  5. Tell user to open file in IDE to view rendered diagram

OpenShift Node Kernel Inspection Skill

This skill provides utilities for inspecting kernel-level networking configuration on OpenShift/Kubernetes nodes using oc debug.

Overview

The skill enables interaction with kernel networking tools on Kubernetes nodes without requiring SSH access. It uses oc debug to create ephemeral containers with host network access and executes kernel commands in the host's namespace.

Commands Provided

node-kernel-ip

Executes ip commands to inspect routing tables, network devices, and interfaces.

Script: node-kernel-ip.sh

Usage:

./node-kernel-ip.sh <node> <image> --command <cmd> [--options <opts>] [--filter <params>]

Example:

./node-kernel-ip.sh worker-1 registry.redhat.io/rhel9/support-tools --command "route show"

node-kernel-iptables

Executes iptables or ip6tables commands to inspect packet filter rules.

Script: node-kernel-iptables.sh

Usage:

./node-kernel-iptables.sh <node> <image> --command <cmd> [--table <table>] [--filter <params>]

Example:

./node-kernel-iptables.sh worker-1 registry.redhat.io/rhel9/support-tools --command "-L POSTROUTING" --table nat --filter "-nv4"

node-kernel-nft

Executes nft commands to inspect nftables packet filtering and classification rules.

Script: node-kernel-nft.sh

Usage:

./node-kernel-nft.sh <node> <image> --command <cmd> [--family <family>]

Example:

./node-kernel-nft.sh worker-1 registry.redhat.io/rhel9/support-tools --command "list tables" --family inet

node-kernel-conntrack

Executes conntrack commands or reads /proc/net/nf_conntrack to inspect connection tracking entries.

Script: node-kernel-conntrack.sh

Usage:

./node-kernel-conntrack.sh <node> <image> [--command <cmd>] [--filter <params>]

Example:

./node-kernel-conntrack.sh worker-1 registry.redhat.io/rhel9/support-tools --command "-L" --filter "-s 1.2.3.4"

Helper Functions

The kernel-helper.sh script provides shared functions:

  • check_utility_exists: Verifies a utility exists in the debug image
  • execute_kernel_command: Executes commands on a node via oc debug
  • filter_warnings: Removes common oc debug warning messages from output
  • validate_node_exists: Validates node name exists in cluster
  • detect_and_set_kubeconfig: Auto-detects and configures kubeconfig

Output Handling

All commands ensure:

  • stdout contains only the actual command output (routing tables, interface info, etc.)
  • stderr contains warnings, debug messages, and errors
  • Common oc debug warnings are filtered out automatically using improved regex patterns
  • Output is explicitly captured and returned to ensure proper data flow to calling processes

Improvements

The execute_kernel_command() function explicitly captures all output from oc debug and filters warnings before returning results, ensuring:

  1. Complete output capture: All stdout/stderr is captured in a variable
  2. Reliable filtering: Uses single grep -E with multiple patterns for efficiency
  3. Guaranteed output: The || cat fallback ensures output passes through even if no warnings are found
  4. Explicit return: Output is explicitly sent to stdout using printf for reliable data flow

Commands 17

Name

openshift:add-enhancement

Synopsis

/openshift:add-enhancement [area] <name> <description> <jira>

Description

The openshift:add-enhancement command creates a new OpenShift Enhancement Proposal (EP) based on the official template from the openshift/enhancements repository. It generates a comprehensive enhancement document with all required sections, metadata, and guidance following OpenShift's enhancement process.

This command automates the creation of enhancement proposals by:

  • Fetching the latest enhancement template from the openshift/enhancements repository
  • Analyzing the provided description to extract what, why, and who information
  • Generating user stories, goals, and non-goals based on the requirements
  • Creating enhancement files with valid YAML metadata, correct heading structure, and all required sections
  • Applying OpenShift-specific conventions for feature gates, API design, and testing requirements

Arguments

  • area (optional): Enhancement area (subdirectory under enhancements/). If not provided, the command will attempt to infer the best area from the enhancement description and ask for confirmation.
  • name: One-line title describing the enhancement
  • description: Detailed description (what, why, who)
  • jira: JIRA ticket URL for tracking

Implementation

Act as an experienced software architect to create a comprehensive enhancement proposal. Follow these steps:

Important: Reference the guidance in the OpenShift enhancements repository at https://github.com/openshift/enhancements/blob/master/dev-guide/feature-zero-to-hero.md, particularly the section "Writing an OpenShift Enhancement", when creating enhancement proposals. This guide provides essential context on the OpenShift Enhancement Proposal process, feature gates, API design conventions, testing requirements, and promotion criteria.

  1. Verify Repository: Before proceeding, ensure you are operating in the correct repository:

    • Check if the current directory is a git repository: git rev-parse --is-inside-work-tree
    • Verify the repository is openshift/enhancements or a fork:
      • Get the remote URL: git config --get remote.origin.url
      • Check if the URL contains "openshift/enhancements" (e.g., git@github.com:openshift/enhancements.git or https://github.com/openshift/enhancements.git)
      • For forks, the URL might be different (e.g., git@github.com:username/enhancements.git) - in this case, check if it's a fork by examining the repository structure (presence of enhancements/ directory and guidelines/ directory with enhancement_template.md)
    • If verification fails, exit with a clear error message: "Error: This command must be run in the openshift/enhancements repository or a fork. Please clone the repository first or navigate to the correct directory."
    • If successful, continue to the next step
  2. Determine Enhancement Area: Identify or validate the area where the enhancement should be created:

    • List available areas by examining subdirectories in enhancements/: ls -d enhancements/*/ (extract just the directory names)
    • If the <area> argument WAS provided by the user:
      • Check if enhancements/<area>/ exists
      • If it exists: proceed with this area
      • If it does NOT exist:
        • Show the list of available areas
        • Use AskUserQuestion to ask: "The area '' does not exist. Do you want to create a new area called ''?" with options:
          • "Yes, create new area ''" (then confirm they're sure about creating a new area)
          • "No, let me choose from existing areas" (then show available areas and ask them to select)
    • If the <area> argument was NOT provided:
      • Analyze the enhancement name and description to infer the most appropriate area
      • Look for keywords that match existing area names (e.g., "authentication", "network", "storage", "installer", "api", "cluster-logging", "monitoring", etc.)
      • Use AskUserQuestion to confirm: "Based on your enhancement description, I think the best area is ''. Is this correct?" with options:
        • "Yes, use ''"
        • "No, let me choose a different area" (then list available areas and ask them to select or specify a new one)
      • If the user chooses a different area that doesn't exist, ask if they want to create it (same flow as above)
    • Creating a new area (if chosen):
      • Create the directory: mkdir -p enhancements/<new-area>/
      • Add a README.md in the new area directory to document its purpose
    • Proceed with the validated/created area
  3. Fetch the Enhancement Template: Before starting, fetch the latest template from the openshift/enhancements repository:

  4. Parse the Description: Extract the following from the description:

    • What: What is this enhancement about
    • Why: Why this change is required (motivation)
    • Who: Which personas this applies to (use this to generate user stories)
  5. Ask Clarifying Questions (if needed): Use the AskUserQuestion tool to gather:

    • Specific user stories or motivations if not clear from the description
    • Explicit Goals or Non-Goals the user wants to be included
    • Any specific technical constraints or requirements
    • Topology considerations (Hypershift, SNO, MicroShift, OKE relevance)
    • Whether this proposal adds/changes CRDs, admission and conversion webhooks, ValidatingAdmissionPlugin, MutatingAdmissionPlugin, aggregated API servers, or finalizers (needed for API Extensions section)
    • Feature gate information: According to the OpenShift enhancement dev guide (https://github.com/openshift/enhancements/blob/master/dev-guide/feature-zero-to-hero.md), ALL new OpenShift features must start disabled by default using feature gates. Ask about the proposed feature gate name and initial feature set (DevPreviewNoUpgrade or TechPreviewNoUpgrade).
    • Ask clarifying questions about telemetry, security, upgrade and downgrade process, rollbacks, dependencies, in case it is not possible to assert these fields.
  6. Generate the Enhancement File:

    • Create the file at enhancements/<area>/<filename>.md where filename is the kebab-case version of the name argument
    • Fill in the template with:
      • Title: Use the provided name
      • Summary: One paragraph describing what this enhancement is about
      • Motivation: Explain why this change is required based on the description
      • User Stories: Generate 2-4 user stories based on the "who" information using the format:

        "As a role, I want to take some action so that I can accomplish a goal." Include a story on how the proposal will be operationalized: life-cycled, monitored and remediated at scale.

      • Goals: List specific, measurable goals (3-5 items). Goals should describe what users want from their perspective, not implementation details.
      • Non-Goals: List what is explicitly out of scope (2-3 items)
      • Proposal: High-level description of the proposed solution
      • Workflow Description: Detailed workflow with actors and steps
      • Mermaid Diagram: Add a sequence diagram when the workflow involves multiple actors or complex interactions between components (e.g., user -> API server -> controller -> operator). Simple single-actor workflows may not need a diagram.
      • API Extensions: Only fill this section if the user confirms the proposal adds/changes CRDs, admission and conversion webhooks, ValidatingAdmissionPlugin, MutatingAdmissionPlugin, aggregated API servers, or finalizers. Per the template, name the API extensions and describe if this enhancement modifies the behaviour of existing resources. Otherwise, add a TODO comment asking the user to complete this section if applicable.
      • Topology Considerations: Include subsections for Hypershift/Hosted Control Planes, Standalone Clusters, Single-node Deployments or MicroShift, and OKE (OpenShift Kubernetes Engine). Address how the proposal affects each topology.
      • Implementation Details/Notes/Constraints: Provide a high-level overview of the code changes required. Follow the guidance from the template: "While it is useful to go into the details of the code changes required, it is not necessary to show how the code will be rewritten in the enhancement." Keep it as an overview; the developer should fill in the specific implementation details. Include a reminder about creating a feature gate: Per the OpenShift dev guide (https://github.com/openshift/enhancements/blob/master/dev-guide/feature-zero-to-hero.md), all new features must be gated behind a feature gate in https://github.com/openshift/api/blob/master/features/features.go with the appropriate feature set (DevPreviewNoUpgrade or TechPreviewNoUpgrade initially).
      • Test Plan: Add a TODO comment with guidance on required test labels per the OpenShift dev guide (https://github.com/openshift/enhancements/blob/master/dev-guide/feature-zero-to-hero.md): Tests must include [OCPFeatureGate:FeatureName] label for the feature gate, [Jira:"Component Name"] for the component, and appropriate test type labels like [Suite:...], [Serial] for tests that cannot run in parallel, [Slow] for tests exceeding 5 minutes, or [Disruptive] for tests that impact other workloads. Reference the test conventions guide (https://github.com/openshift/enhancements/blob/master/dev-guide/test-conventions.md) for details.
      • Graduation Criteria: Add a TODO comment referencing the specific promotion requirements from the OpenShift dev guide (https://github.com/openshift/enhancements/blob/master/dev-guide/feature-zero-to-hero.md): minimum 5 tests, 7 runs per week, 14 runs per supported platform, 95% pass rate, and tests running on all supported platforms (AWS, Azure, GCP, vSphere, Baremetal with various network stacks).
      • Metadata: Fill in creation-date with today's date, tracking-link with the provided JIRA ticket URL, set other fields to TBD. For api-approvers: use "None" if there are no API changes (no new/modified CRDs, webhooks, aggregated API servers, or finalizers); otherwise use "TBD" as a placeholder (the enhancement author will request an API reviewer from the #forum-api-review Slack channel later).
  7. Handle Unfilled Sections: For sections that cannot be filled based on the input:

    • Add a clear comment like <!-- TODO: This section needs to be filled in -->
    • Provide guidance on what should be included
  8. Writing Guidelines:

    • Write in a clear, concise, professional manner
    • Focus on the essential information
    • Use bullet points and structured formatting
    • Avoid unnecessary verbosity
    • Line Length: Keep lines in the generated enhancement at a maximum of 80 characters, but prioritize validity over line length limits. Only break lines at 80 characters if doing so will NOT create:
      • Invalid or broken URLs (URLs themselves should never be split, but the line CAN and SHOULD be broken before or after the URL)
      • Invalid Markdown syntax (e.g., breaking Markdown links, code blocks, or formatting)
      • Invalid code examples (e.g., breaking code in the middle of statements) If breaking at 80 characters would split a URL, code, or Markdown syntax, find the nearest valid break point such as: after a sentence, before a URL starts, after a URL ends, or at a natural paragraph break. For regular prose, it is acceptable to exceed 80 characters by 10-15 characters to avoid breaking words mid-word. Only allow lines >95 characters when the line contains a single unbreakable element (like a standalone URL with no surrounding text, or a single line of code).
  9. Validate:

    • Create a valid filename from the name (lowercase, replace spaces with dashes)
    • Verify all required YAML metadata is present
    • Verify the JIRA ticket URL is included in the tracking-link metadata field
    • Ensure the enhancement file is created in the correct path: enhancements/<area>/<filename>.md

Output

After creating the enhancement file, provide:

  • The full path to the created file
  • A brief summary of what was included
  • A list of sections that need further attention (marked with TODO comments)

Begin by analyzing the inputs and asking any necessary clarifying questions before generating the enhancement proposal.

Name

openshift:bootstrap-om

Synopsis

/openshift:bootstrap-om

Description

The openshift:bootstrap-om command automates the complete integration of OpenShift Manager (OM) into OpenShift operators. OM (formerly Multi-Operator Manager/MOM) is a framework designed to reduce duplicate effort and improve consistency across different OpenShift cluster topologies (standalone/OCP and Hypershift/HCP) by centralizing operator management and enabling comprehensive testing.

This command automates:

  • Automatic resource discovery - Analyzes the codebase to identify all input and output resources
  • Command implementation - Creates the three required OM commands (input-resources, output-resources, apply-configuration)
  • Test infrastructure setup - Configures Makefile targets and test directories
  • Test scenario creation - Generates initial test cases with proper structure
  • Integration validation - Runs tests to ensure commands compile, execute, and produce expected output

Note: These instructions are optimized for operators built with github.com/openshift/library-go.

OpenShift Manager (OM) Overview

OM enables centralized operator management by requiring operators to declare their resource dependencies and configuration logic:

  1. input-resources - Lists all Kubernetes API resources the operator needs to read/watch
  2. output-resources - Maps Kubernetes API resources the operator creates/manages to cluster types (Configuration/Management/UserWorkload)
  3. apply-configuration - Runs operator logic in isolation using a manifestclient (a Kubernetes client that reads from a must-gather-like input directory instead of the API server): syncs once, outputs resulting resources to an output directory, then exits.

These declarations enable:

  • Production Runtime: A single OperatorManager binary can communicate with the Kubernetes API server on behalf of multiple operators, maintaining shared caches and implementing rate limiting
  • Testing: The apply-configuration command validates operator behavior without a live cluster by using the manifestclient for file-based input/output
  • Consistency: Ensure or facilitate identical operator behavior across different cluster topologies (standalone, Hypershift/HCP)

Note: While "OM" (OpenShift Manager) is the new name, libraries and code still use "mom" (Multi-Operator Manager) naming for backwards compatibility.

Implementation

The command executes the following automated workflow:

1. Analyze the Operator's Resource Usage

IMPORTANT: Automatically identify resources by analyzing code. Do NOT ask users to manually list them.

Find Input Resources (Resources the Operator READS)

Input resources are those the operator watches, gets, or lists from the API server.

Search for:

  • Controller informers and listers (e.g., configInformer, deploymentLister)
  • Direct API calls (e.g., client.Get(), client.List())
  • Resource watches in controller registration
  • ConfigMaps, Secrets, CustomResources being read

How to search:

# Find controller files
find pkg -name "*controller*.go" -o -name "*operator*.go"

# Look for informers
grep -r "Informer()" pkg/

# Look for listers
grep -r "Lister" pkg/

# Look for direct API reads
grep -r "client.Get\|client.List" pkg/

Common patterns:

  • configMapInformer.Lister().ConfigMaps("namespace").Get("name") → Input: ConfigMap in namespace
  • deploymentInformer.Informer().AddEventHandler() → Input: Deployment resource
  • operatorClient.Operator() → Input: Operator custom resource
Find Output Resources (Resources the Operator CREATES/MANAGES)

Output resources are categorized by cluster targeting (where they live in different topologies):

ConfigurationResources: Resources targeted at the cluster where configuration is held

  • On standalone: the one cluster
  • On HCP: logically a view into resources in the namespace of the guest cluster
  • Common examples: config.openshift.io resources (OAuth, Ingress, etc.)

ManagementResources: Resources targeted at the cluster where management plane responsibility is held

  • On standalone: the one cluster
  • On HCP: logically resources in the namespace of the guest cluster (control plane aspects)
  • Common examples: Operator's own Deployments, Services, ConfigMaps, Secrets, ServiceAccounts, ClusterOperator status
  • Includes EventingNamespaces: namespaces where the operator emits events

UserWorkloadResources: Resources targeted at the cluster where user workloads run

  • On standalone: the one cluster
  • On HCP: the guest cluster
  • Common examples: ClusterRoles, ClusterRoleBindings, CRDs, resources that run alongside user workloads

How to search:

# Find manifests and bindata
find bindata/ manifests/ -type f 2>/dev/null

# Look for resource creation/apply
grep -r "Apply\|Create\|Update" pkg/ | grep -i "deployment\|service\|configmap"

# Look for manifest files
grep -r "apiVersion:" bindata/ manifests/ 2>/dev/null

Common patterns:

  • Files in bindata/ or manifests/ → Output resources
  • resourceapply.ApplyDeployment() → Output: Deployment
  • resourceapply.ApplyConfigMap() → Output: ConfigMap
  • ClusterOperator status updates → Output: ClusterOperator

2. Prepare Dependencies

Before creating command stubs, manually add the dependency to go.mod:

// In the require section of go.mod, add:
github.com/openshift/multi-operator-manager v0.0.0-20250930141021-05cb0b9abdb4

Note: Don't run go mod tidy yet - it will remove the dependency since no code imports it yet. After creating the stub files that import the library, you'll run go mod tidy && go mod vendor, and the dependency will be committed together with the stubs in the first commit

3. Implement the Three OM Commands (Incrementally)

IMPORTANT: Create command stubs first, then fill them in with actual resource declarations. This creates a clean, logical git history with ~6-8 focused commits:

  1. Create command stub → integrate into CLI → commit
  2. Fill in the stub with discovered resources → commit
  3. Repeat for each command
  4. Set up test infrastructure → commit
  5. (Optional) Add test scenario if apply-configuration works → commit
Common Patterns

Directory structure:

mkdir -p pkg/cmd/mom

Note: We keep the mom directory name for consistency with library imports.

Helper functions for declaring resources (use in both input-resources and output-resources):

Common resources:

  • ExactConfigMap(namespace, name), ExactSecret(namespace, name), ExactDeployment(namespace, name)
  • ExactService(namespace, name), ExactServiceAccount(namespace, name), ExactNamespace(name)
  • ExactRole(namespace, name), ExactRoleBinding(namespace, name)

OpenShift-specific:

  • ExactConfigResource(name) - For config.openshift.io resources (e.g., "oauths", "ingresses")
  • ExactLowLevelOperator(name) - For operator.openshift.io resources
  • ExactClusterOperator(name) - For ClusterOperators

Cluster-scoped RBAC:

  • ExactClusterRole(name), ExactClusterRoleBinding(name)

Other:

  • ExactPDB(namespace, name), ExactOAuthClient(name), GeneratedCSR(prefix)
  • ExactResource(group, version, resource, namespace, name) - For any other resource

CLI integration pattern (add to cmd/*/main.go):

import (
        "github.com/openshift/YOUR-OPERATOR/pkg/cmd/mom"
)

// In your main command function:
cmd.AddCommand(mom.NewInputResourcesCommand(ioStreams))
cmd.AddCommand(mom.NewOutputResourcesCommand(ioStreams))
cmd.AddCommand(mom.NewApplyConfigurationCommand(ioStreams))

Build/test pattern:

make build
./your-operator-binary <command-name>  # Test the command
git add <files>
git commit -m "<title>

<optional description>

Generated with Claude Code"
3.1 Create and Commit input-resources Command Stub

Create pkg/cmd/mom/input_resources_command.go:

package mom

import (
        "context"

        "github.com/openshift/multi-operator-manager/pkg/library/libraryinputresources"
        "github.com/spf13/cobra"
        "k8s.io/cli-runtime/pkg/genericiooptions"
)

func NewInputResourcesCommand(streams genericiooptions.IOStreams) *cobra.Command {
        return libraryinputresources.NewInputResourcesCommand(runInputResources, runOutputResources, streams)
}

func runInputResources(ctx context.Context) (*libraryinputresources.InputResources, error) {
        return &libraryinputresources.InputResources{
                ApplyConfigurationResources: libraryinputresources.ResourceList{
                        ExactResources: []libraryinputresources.ExactResourceID{
                                // TODO: Fill in discovered resources
                        },
                },
        }, nil
}

Integrate into CLI (see Common Patterns), build, test, and commit with title: "Add OM input-resources command stub"

3.2 Fill in input-resources and Commit

Analyze the codebase to discover all input resources (see Section 1 for search patterns).

Update runInputResources function with discovered resources:

func runInputResources(ctx context.Context) (*libraryinputresources.InputResources, error) {
        return &libraryinputresources.InputResources{
                ApplyConfigurationResources: libraryinputresources.ResourceList{
                        ExactResources: []libraryinputresources.ExactResourceID{
                                // Example discovered resources (see Helper Functions):
                                libraryinputresources.ExactLowLevelOperator("youroperators"),
                                libraryinputresources.ExactConfigResource("infrastructures"),
                                libraryinputresources.ExactNamespace("your-operator-namespace"),
                                libraryinputresources.ExactConfigMap("namespace", "config-name"),
                                // ... all discovered input resources
                        },
                },
        }, nil
}

Build, test, and commit with title: "Populate OM input-resources with discovered resources"

3.3 Create and Commit output-resources Command Stub

Create pkg/cmd/mom/output_resources_command.go:

package mom

import (
        "context"

        "github.com/openshift/multi-operator-manager/pkg/library/libraryoutputresources"
        "github.com/spf13/cobra"
        "k8s.io/cli-runtime/pkg/genericiooptions"
)

func NewOutputResourcesCommand(streams genericiooptions.IOStreams) *cobra.Command {
        return libraryoutputresources.NewOutputResourcesCommand(runOutputResources, streams)
}

func runOutputResources(ctx context.Context) (*libraryoutputresources.OutputResources, error) {
        return &libraryoutputresources.OutputResources{
                ConfigurationResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{},
                },
                ManagementResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{},
                        EventingNamespaces: []string{},
                },
                UserWorkloadResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{},
                },
        }, nil
}

Integrate into CLI, build, test, and commit with title: "Add OM output-resources command stub"

3.4 Fill in output-resources and Commit

Analyze the codebase to discover all output resources (see Section 1 for search patterns and category descriptions).

Update runOutputResources function with discovered resources:

func runOutputResources(ctx context.Context) (*libraryoutputresources.OutputResources, error) {
        return &libraryoutputresources.OutputResources{
                ConfigurationResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{
                                libraryoutputresources.ExactConfigResource("ingresses"),
                        },
                },
                ManagementResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{
                                libraryoutputresources.ExactClusterOperator("operatorname"),
                                libraryoutputresources.ExactNamespace("namespace"),
                                libraryoutputresources.ExactDeployment("namespace", "name"),
                                // ... all discovered management resources
                        },
                        EventingNamespaces: []string{
                                "operator-namespace",
                        },
                },
                UserWorkloadResources: libraryoutputresources.ResourceList{
                        ExactResources: []libraryoutputresources.ExactResourceID{
                                libraryoutputresources.ExactClusterRole("name"),
                                libraryoutputresources.ExactClusterRoleBinding("name"),
                        },
                },
        }, nil
}

Commit the verified output-resource implementation as "Populate OM output-resources with discovered resources".

3.5 Create apply-configuration Command Stub and Commit

This is the most complex command. It requires running your operator's reconciliation logic once using the manifestclient.

Create pkg/cmd/mom/apply_configuration_command.go:

package mom

import (
        "context"
        "fmt"

        "github.com/openshift/multi-operator-manager/pkg/library/libraryapplyconfiguration"
        "github.com/spf13/cobra"
        "k8s.io/cli-runtime/pkg/genericiooptions"
)

func NewApplyConfigurationCommand(streams genericiooptions.IOStreams) *cobra.Command {
        return libraryapplyconfiguration.NewApplyConfigurationCommand(RunApplyConfiguration, runOutputResources, streams)
}

func RunApplyConfiguration(ctx context.Context, input libraryapplyconfiguration.ApplyConfigurationInput) (*libraryapplyconfiguration.ApplyConfigurationRunResult, libraryapplyconfiguration.AllDesiredMutationsGetter, error) {
        // TODO: Implement operator reconciliation logic
        //
        // The manifestclient (input.ManagementClient) is a drop-in replacement for standard k8s clients.
        // Pass it to your operator and run sync logic ONCE (not in a loop).
        //
        // Example: op := operator.NewOperator(input.ManagementClient, input.ManagementEventRecorder)
        //          if err := op.Sync(ctx); err != nil { return nil, nil, err }
        //          return &libraryapplyconfiguration.ApplyConfigurationRunResult{}, nil, nil
        //
        // See: github.com/openshift/cluster-authentication-operator/pkg/cmd/mom/apply_configuration_command.go

        return nil, nil, fmt.Errorf("not yet implemented - see TODO above")
}

Integrate into CLI, build (verify it compiles), and commit with title: "Add OM apply-configuration command stub"

Note: If you cannot implement apply-configuration fully, that's acceptable! The stub provides instructions for manual implementation later.

4. Set Up Test Infrastructure and Commit

IMPORTANT: Check if the project uses build-machinery-go before proceeding:

ls vendor/github.com/openshift/build-machinery-go/make/targets/openshift/operator/mom.mk

If build-machinery-go is NOT present:

  • Inform the user that test infrastructure requires build-machinery-go
  • Explain they need to vendor github.com/openshift/build-machinery-go first
  • Skip test infrastructure setup (OM commands are still usable without it)

If build-machinery-go IS present:

4.1 Update Makefile

Add to your Makefile:

include $(addprefix ./vendor/github.com/openshift/build-machinery-go/make/, \
        targets/openshift/operator/mom.mk \
)

This provides:

  • make test-operator-integration - Runs OM tests and validates output
  • make update-test-operator-integration - Updates expected test output
4.2 Create Test Scenarios (Optional - only if apply-configuration is implemented)

Create test directory structure:

mkdir -p test-data/apply-configuration/overall/minimal-cluster/input-dir

Each test scenario needs:

  • test.yaml - Test metadata (binaryName, testName, description, testType, now)
  • input-dir/ - Input resources in must-gather format organized by cluster-scoped-resources/<group>/<kind>.yaml or namespaces/<namespace>/<group>/<kind>/<name>.yaml

Run initial test:

make build
make test-operator-integration
make update-test-operator-integration  # Update expected output
make test-operator-integration  # Should now pass

Commit Makefile (and test-data/ if created) with title: "Add OM test infrastructure"

5. Verify and Iterate

  1. Run make test-operator-integration - Should pass after updating expected output
  2. Create additional test scenarios for different configurations
  3. Verify input-resources and output-resources are complete by checking test output

Reference Implementation

Study these files in cluster-authentication-operator:

  • pkg/cmd/mom/input_resources_command.go - Complete input resource declarations
  • pkg/cmd/mom/output_resources_command.go - Complete output resource declarations
  • pkg/cmd/mom/apply_configuration_command.go - Integration with operator core logic
  • cmd/*/main.go - CLI integration pattern
  • Makefile - OM makefile inclusion
  • test-data/apply-configuration/overall/ - Multiple test scenario examples

Documentation:

Key Concept - manifestclient: Implements standard Kubernetes client interfaces (client.Client, kubernetes.Interface, dynamic.Interface). It's a drop-in replacement that reads from files instead of the API server - no refactoring needed!

Validation Checklist

Before considering OM integration complete:

  • pkg/cmd/mom/input_resources_command.go exists and lists all input resources
  • pkg/cmd/mom/output_resources_command.go exists and lists all output resources
  • pkg/cmd/mom/apply_configuration_command.go exists (stub or implementation)
  • Commands integrated into main CLI
  • Makefile includes mom.mk (if build-machinery-go available)
  • At least one test scenario in test-data/apply-configuration/overall/ (if apply-configuration implemented)
  • make test-operator-integration passes (if tests created)

Usage Guidelines

  1. Separate Dependency Commits - ALWAYS commit dependency changes separately. Any changes to go.mod, go.sum, or vendor/ must go in their own commits. NEVER mix vendor changes with code implementation.

  2. Automatic Analysis - Analyze codebase automatically. Do NOT ask users to list resources.

  3. apply-configuration Can Be a Stub - If you cannot implement it fully, create a comprehensive stub with TODO comments. This is acceptable!

  4. Validate Early - Build and test after each command stub/fill-in to catch issues immediately.

The command should autonomously complete all steps from resource discovery through test infrastructure setup (if build-machinery-go is available), creating focused, logical commits that are easy to review.

Name

openshift:bump-deps

Synopsis

/openshift:bump-deps <dependency> [version] [--create-jira] [--create-pr]

Description

The openshift:bump-deps command automates the process of bumping dependencies in OpenShift organization projects. It analyzes the dependency, determines the appropriate version to bump to, updates the necessary files (go.mod, go.sum, package.json, etc.), runs tests, and optionally creates Jira tickets and pull requests.

This command significantly reduces the manual effort required for dependency updates by automating:

  • Dependency version discovery and analysis
  • Compatibility checking with current codebase
  • File updates (go.mod, package.json, Dockerfile, etc.)
  • Test execution to verify the update
  • Jira ticket creation with comprehensive details
  • Pull request creation with proper formatting
  • Release notes generation

The command intelligently handles different dependency types (Go modules, npm packages, container images, etc.) and can process single or multiple dependencies at once.

Implementation

The command executes the following workflow:

1. Repository Analysis

  • Detects repository type (Go, Node.js, Python, etc.)
  • Identifies dependency management files (go.mod, package.json, requirements.txt, etc.)
  • Determines current project structure and conventions
  • Checks for existing CI/CD configuration

2. Dependency Discovery

For Go Projects:

  • Parses go.mod to find current version
  • Uses go list -m -versions <module> to list available versions
  • Checks for major version compatibility (v0, v1, v2+)
  • Identifies if dependency is direct or indirect

For Node.js Projects:

  • Parses package.json for current version
  • Uses npm/yarn to find latest versions
  • Checks semantic versioning constraints
  • Identifies devDependencies vs dependencies

For Container Images:

  • Parses Dockerfile and related files
  • Checks registry for available tags
  • Verifies image digest and signatures
  • Identifies base images and tool images

For Python Projects:

  • Parses requirements.txt or pyproject.toml
  • Uses pip to find available versions
  • Checks for version constraints

3. Version Selection

If no version is specified:

  • Suggests latest stable version
  • Considers semantic versioning (patch, minor, major)
  • Checks for breaking changes in release notes
  • Validates against project's minimum version requirements

If version is specified:

  • Validates version exists
  • Checks compatibility with current project version
  • Warns about major version jumps

4. Impact Analysis

  • Searches codebase for usage of the dependency
  • Identifies files importing/using the dependency
  • Analyzes API changes between versions
  • Checks for deprecated features being used
  • Reviews upstream changelog and release notes
  • Identifies potential breaking changes

5. File Updates

Go Projects:

  • Updates go.mod with new version
  • Runs go mod tidy to update go.sum
  • Runs go mod vendor if vendor directory exists
  • Updates any version constraints in comments

Node.js Projects:

  • Updates package.json
  • Runs npm install or yarn install
  • Updates package-lock.json or yarn.lock

Container Images:

  • Updates Dockerfile(s)
  • Updates related manifests (kubernetes, etc.)
  • Updates any CI configuration using the image

Python Projects:

  • Updates requirements.txt or pyproject.toml
  • Generates new lock file if applicable

6. Testing Strategy

  • Identifies relevant test suites
  • Runs unit tests: make test or equivalent
  • Runs integration tests if available
  • Runs e2e tests for critical dependencies
  • Checks for test failures and analyzes logs
  • Verifies build succeeds: make build

7. Jira Ticket Creation (if --create-jira)

Creates a Jira ticket with:

  • Summary: Bump {dependency} from {old_version} to {new_version}
  • Type: Task or Bug (if security update)
  • Components: Auto-detected from repository
  • Labels: ["dependencies", "automated-update", "ai-generated"]
    • Adds "security" if CVE-related
    • Adds "breaking-change" if major version bump
  • Description: Includes:
    • Dependency information and type
    • Current and new versions
    • Changelog summary
    • Breaking changes (if any)
    • Files modified
    • Test results
    • Migration steps (if needed)
    • Links to upstream release notes
  • Target Version: Auto-detected from release branches

8. Pull Request Creation (if --create-pr)

Creates a pull request with:

  • Title: [{JIRA-ID}] Bump {dependency} from {old_version} to {new_version}
  • Body: Includes:
    • Link to Jira ticket
    • Summary of changes
    • Breaking changes callout
    • Testing performed
    • Checklist for reviewers
    • Release notes snippet
  • Labels: Auto-applied based on change type
  • Branch naming: deps/{dependency}-{new_version} or {jira-id}-bump-{dependency}

9. Conflict Resolution

If updates cause issues:

  • Identifies conflicting dependencies
  • Suggests resolution strategies
  • Can attempt automatic resolution for common cases
  • Provides manual resolution steps for complex scenarios

Return Value

  • Claude agent text: Processing status, test results, and summary
  • Side effects:
    • Modified dependency files (go.mod, package.json, etc.)
    • Updated lock files
    • Jira ticket created (if --create-jira)
    • Pull request created (if --create-pr)
    • Git branch created with changes

Examples

  1. Bump a Go dependency to latest:

    /openshift:bump-deps k8s.io/api
    

    Output:

    Analyzing dependency: k8s.io/api
    Current version: v0.28.0
    Latest version: v0.29.1
    
    Checking compatibility...
    ✅ No breaking changes detected
    
    Updating go.mod...
    Running go mod tidy...
    
    Running tests...
    ✅ All tests passed
    
    Summary:
    - Dependency: k8s.io/api
    - Old version: v0.28.0
    - New version: v0.29.1
    - Files modified: go.mod, go.sum
    - Tests: ✅ Passed
    
    Changes are ready. Use --create-pr to create a pull request.
    
  2. Bump to a specific version with Jira ticket:

    /openshift:bump-deps golang.org/x/net v0.20.0 --create-jira
    

    Output:

    Analyzing dependency: golang.org/x/net
    Current version: v0.19.0
    Target version: v0.20.0
    
    Reviewing changes...
    ⚠️  Breaking changes detected in v0.20.0:
    - http2: Server.IdleTimeout applies to idle h2 connections
    
    Updating go.mod...
    Running tests...
    ✅ All tests passed
    
    Creating Jira ticket...
    ✅ Created: OCPBUGS-12345
    
    Summary:
    - Jira: https://redhat.atlassian.net/browse/OCPBUGS-12345
    - Dependency: golang.org/x/net
    - Version: v0.19.0 → v0.20.0
    - Breaking changes: Yes
    
  3. Bump and create PR in one step:

    /openshift:bump-deps github.com/spf13/cobra --create-jira --create-pr
    

    Output:

    Processing dependency bump for github.com/spf13/cobra...
    
    [1/7] Analyzing dependency...
    Current: v1.7.0
    Latest: v1.8.0
    
    [2/7] Checking changelog...
    Changes include:
    - New features: Enhanced shell completion
    - Bug fixes: 5 issues resolved
    - No breaking changes
    
    [3/7] Updating files...
    ✅ go.mod updated
    ✅ go.sum updated
    
    [4/7] Running tests...
    ✅ Unit tests: 156/156 passed
    ✅ Integration tests: 23/23 passed
    
    [5/7] Creating Jira ticket...
    ✅ Created: OCPBUGS-12346
    
    [6/7] Creating git branch...
    ✅ Branch: OCPBUGS-12346-bump-cobra
    
    [7/7] Creating pull request...
    ✅ PR created: #1234
    
    Summary:
    - Jira: https://redhat.atlassian.net/browse/OCPBUGS-12346
    - PR: https://github.com/openshift/repo/pull/1234
    - Dependency: github.com/spf13/cobra
    - Version: v1.7.0 → v1.8.0
    - Tests: All passed
    
    Next steps:
    1. Review the PR at the link above
    2. Address any reviewer comments
    3. Merge when approved
    
  4. Bump multiple related dependencies:

    /openshift:bump-deps "k8s.io/*"
    

    Output:

    Found 8 Kubernetes dependencies to update:
    
    [1/8] k8s.io/api: v0.28.0 → v0.29.1
    [2/8] k8s.io/apimachinery: v0.28.0 → v0.29.1
    [3/8] k8s.io/client-go: v0.28.0 → v0.29.1
    [4/8] k8s.io/kubectl: v0.28.0 → v0.29.1
    ...
    
    These should be updated together to maintain compatibility.
    Proceed with batch update? [y/N]
    
  5. Bump a container base image:

    /openshift:bump-deps registry.access.redhat.com/ubi9/ubi-minimal
    

    Output:

    Analyzing container image: ubi9/ubi-minimal
    Current: 9.3-1361
    Latest: 9.4-1194
    
    Checking for security updates...
    ✅ 3 CVEs fixed in new version
    
    Updating Dockerfile...
    Building test image...
    Running container tests...
    ✅ All tests passed
    
    Files modified:
    - Dockerfile
    - .github/workflows/build.yml
    

Arguments

  • $1 (required): Dependency identifier

    • Go module: github.com/org/repo or golang.org/x/net
    • npm package: @types/node or react
    • Container image: registry.access.redhat.com/ubi9/ubi-minimal
    • Wildcard for batch: k8s.io/* (requires confirmation)
  • $2 (optional): Target version

    • Semantic version: v1.2.3, 1.2.3
    • Version range: ^1.2.0, ~1.2.0
    • Special: latest, latest-stable
    • If omitted: suggests latest stable version
  • --create-jira (flag): Create a Jira ticket for the update

    • Auto-detects project from repository
    • Can be configured with JIRA_PROJECT env var
    • Ticket includes full change analysis
  • --create-pr (flag): Create a pull request with the changes

    • Implies creating a git branch
    • Includes --create-jira automatically
    • PR is created as draft if tests fail
  • --jira-project (option): Specify Jira project (default: auto-detect)

    • Example: --jira-project OCPBUGS
  • --component (option): Specify Jira component (default: auto-detect)

    • Example: --component "Control Plane"
  • --branch (option): Specify git branch name (default: auto-generate)

    • Example: --branch feature/update-deps
  • --skip-tests (flag): Skip running tests (not recommended)

    • Use only for non-critical updates
    • PR will be marked as draft
  • --force (flag): Force update even if tests fail

    • Creates PR as draft
    • Includes test failure details in PR

Error Handling

The command handles common error cases:

  • Dependency not found: Lists similar dependencies in project
  • Version not found: Shows available versions
  • Test failures:
    • Provides detailed error logs
    • Suggests potential fixes
    • Asks whether to create draft PR anyway
  • Conflicting dependencies:
    • Identifies conflicts
    • Suggests resolution order
    • Can attempt batch update
  • Breaking changes:
    • Highlights breaking changes
    • Links to migration guides
    • Requires explicit confirmation for major bumps
  • Network failures: Retries with exponential backoff
  • Permission errors: Checks git/GitHub authentication

Notes

  • Repository name and organization are auto-detected from git remote -v
  • For Go dependencies, supports both versioned (v2+) and unversioned modules
  • Automatically detects if running in a fork vs upstream repository
  • Respects .gitignore and doesn't commit generated/vendored files unnecessarily
  • Can handle dependencies with replace directives in go.mod
  • Supports monorepos with multiple go.mod files
  • All Jira tickets are labeled with "ai-generated" for tracking
  • PR creation requires GitHub CLI (gh) to be installed and authenticated
  • For security updates (CVEs), automatically prioritizes and adds "security" and "cve" labels
  • Compatible with Renovate - can be used to customize/enhance Renovate PRs

Environment Variables

  • JIRA_PROJECT: Default Jira project for ticket creation
  • JIRA_COMPONENT: Default component for Jira tickets
  • GITHUB_TOKEN: GitHub authentication (if not using gh auth)
  • DEFAULT_BRANCH: Override default branch detection (default: main)

See Also

  • utils:process-renovate-pr - Process existing Renovate dependency PRs
  • git:create-pr - General PR creation command
  • jira:create - Manual Jira ticket creation

Name

openshift:cluster-health-check

Synopsis

/openshift:cluster-health-check [--verbose] [--output-format json|text]

Description

The cluster-health-check command performs a comprehensive health analysis of an OpenShift/Kubernetes cluster and reports any detected issues. It examines cluster operators, nodes, deployments, pods, persistent volumes, and other critical resources to identify problems that may affect cluster stability or workload availability.

This command is useful for:

  • Quick cluster status assessment
  • Troubleshooting cluster issues
  • Pre-deployment validation
  • Regular health monitoring
  • Identifying degraded components

Prerequisites

Before using this command, ensure you have:

  1. Kubernetes/OpenShift CLI: Either oc (OpenShift) or kubectl (Kubernetes)

  2. Active cluster connection: Must be connected to a running cluster

    • Verify with: oc whoami or kubectl cluster-info
    • Ensure KUBECONFIG is set if needed
  3. Sufficient permissions: Must have read access to cluster resources

    • Cluster-admin or monitoring role recommended for comprehensive checks
    • Minimum: ability to view nodes, pods, and cluster operators

Arguments

  • --verbose (optional): Enable detailed output with additional context

    • Shows resource-level details
    • Includes warning conditions
    • Provides remediation suggestions
  • --output-format (optional): Output format for results

    • text (default): Human-readable text format
    • json: Machine-readable JSON format for automation

Implementation

The command performs the following health checks:

1. Determine CLI Tool

Detect which Kubernetes CLI is available:

if command -v oc &> /dev/null; then
    CLI="oc"
    CLUSTER_TYPE="OpenShift"
elif command -v kubectl &> /dev/null; then
    CLI="kubectl"
    CLUSTER_TYPE="Kubernetes"
else
    echo "Error: Neither 'oc' nor 'kubectl' CLI found. Please install one of them."
    exit 1
fi

2. Verify Cluster Connectivity

Check if connected to a cluster:

if ! $CLI cluster-info &> /dev/null; then
    echo "Error: Not connected to a cluster. Please configure your KUBECONFIG."
    exit 1
fi

# Get cluster version info
if [ "$CLUSTER_TYPE" = "OpenShift" ]; then
    CLUSTER_VERSION=$($CLI version -o json 2>/dev/null | jq -r '.openshiftVersion // "unknown"')
else
    CLUSTER_VERSION=$($CLI version --short 2>/dev/null | grep -i server | awk '{print $3}')
fi

3. Initialize Health Check Report

Create a report structure to collect findings:

REPORT_FILE=".work/cluster-health-check/report-$(date +%Y%m%d-%H%M%S).txt"
mkdir -p .work/cluster-health-check

# Initialize counters
CRITICAL_ISSUES=0
WARNING_ISSUES=0
INFO_MESSAGES=0

4. Check Cluster Operators (OpenShift only)

For OpenShift clusters, check cluster operator health:

if [ "$CLUSTER_TYPE" = "OpenShift" ]; then
    echo "Checking Cluster Operators..."

    # Get all cluster operators
    DEGRADED_COs=$($CLI get clusteroperators -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True")) | .metadata.name')

    UNAVAILABLE_COs=$($CLI get clusteroperators -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="False")) | .metadata.name')

    PROGRESSING_COs=$($CLI get clusteroperators -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Progressing" and .status=="True")) | .metadata.name')

    if [ -n "$DEGRADED_COs" ]; then
        CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$DEGRADED_COs" | wc -l)))
        echo "❌ CRITICAL: Degraded cluster operators found:"
        echo "$DEGRADED_COs" | while read co; do
            echo "  - $co"
            # Get degraded message
            $CLI get clusteroperator "$co" -o json | jq -r '.status.conditions[] | select(.type=="Degraded") | "    Reason: \(.reason)\n    Message: \(.message)"'
        done
    fi

    if [ -n "$UNAVAILABLE_COs" ]; then
        CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$UNAVAILABLE_COs" | wc -l)))
        echo "❌ CRITICAL: Unavailable cluster operators found:"
        echo "$UNAVAILABLE_COs" | while read co; do
            echo "  - $co"
        done
    fi

    if [ -n "$PROGRESSING_COs" ]; then
        WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$PROGRESSING_COs" | wc -l)))
        echo "⚠️  WARNING: Cluster operators in progress:"
        echo "$PROGRESSING_COs" | while read co; do
            echo "  - $co"
        done
    fi
fi

5. Check Node Health

Examine all cluster nodes for issues:

echo "Checking Node Health..."

# Get nodes that are not Ready
NOT_READY_NODES=$($CLI get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status!="True")) | .metadata.name')

if [ -n "$NOT_READY_NODES" ]; then
    CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$NOT_READY_NODES" | wc -l)))
    echo "❌ CRITICAL: Nodes not in Ready state:"
    echo "$NOT_READY_NODES" | while read node; do
        echo "  - $node"
        # Get node conditions
        $CLI get node "$node" -o json | jq -r '.status.conditions[] | "    \(.type): \(.status) - \(.message // "N/A")"'
    done
fi

# Check for SchedulingDisabled nodes
DISABLED_NODES=$($CLI get nodes -o json | jq -r '.items[] | select(.spec.unschedulable==true) | .metadata.name')

if [ -n "$DISABLED_NODES" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$DISABLED_NODES" | wc -l)))
    echo "⚠️  WARNING: Nodes with scheduling disabled:"
    echo "$DISABLED_NODES" | while read node; do
        echo "  - $node"
    done
fi

# Check for node pressure conditions (MemoryPressure, DiskPressure, PIDPressure)
PRESSURE_NODES=$($CLI get nodes -o json | jq -r '.items[] | select(.status.conditions[] | select((.type=="MemoryPressure" or .type=="DiskPressure" or .type=="PIDPressure") and .status=="True")) | .metadata.name')

if [ -n "$PRESSURE_NODES" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$PRESSURE_NODES" | wc -l)))
    echo "⚠️  WARNING: Nodes under resource pressure:"
    echo "$PRESSURE_NODES" | while read node; do
        echo "  - $node"
        $CLI get node "$node" -o json | jq -r '.status.conditions[] | select((.type=="MemoryPressure" or .type=="DiskPressure" or .type=="PIDPressure") and .status=="True") | "    \(.type): \(.message // "N/A")"'
    done
fi

# Check node resource utilization if metrics-server is available
if $CLI top nodes &> /dev/null; then
    echo "Node Resource Utilization:"
    $CLI top nodes
fi

6. Check Pod Health Across All Namespaces

Identify problematic pods:

echo "Checking Pod Health..."

# Get pods that are not Running or Completed
FAILED_PODS=$($CLI get pods --all-namespaces -o json | jq -r '.items[] | select(.status.phase != "Running" and .status.phase != "Succeeded") | "\(.metadata.namespace)/\(.metadata.name) [\(.status.phase)]"')

if [ -n "$FAILED_PODS" ]; then
    CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$FAILED_PODS" | wc -l)))
    echo "❌ CRITICAL: Pods in failed/pending state:"
    echo "$FAILED_PODS"
fi

# Check for pods with restarts
HIGH_RESTART_PODS=$($CLI get pods --all-namespaces -o json | jq -r '.items[] | select(.status.containerStatuses[]? | .restartCount > 5) | "\(.metadata.namespace)/\(.metadata.name) [Restarts: \(.status.containerStatuses[0].restartCount)]"')

if [ -n "$HIGH_RESTART_PODS" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$HIGH_RESTART_PODS" | wc -l)))
    echo "⚠️  WARNING: Pods with high restart count (>5):"
    echo "$HIGH_RESTART_PODS"
fi

# Check for CrashLoopBackOff pods
CRASHLOOP_PODS=$($CLI get pods --all-namespaces -o json | jq -r '.items[] | select(.status.containerStatuses[]? | .state.waiting?.reason == "CrashLoopBackOff") | "\(.metadata.namespace)/\(.metadata.name)"')

if [ -n "$CRASHLOOP_PODS" ]; then
    CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$CRASHLOOP_PODS" | wc -l)))
    echo "❌ CRITICAL: Pods in CrashLoopBackOff:"
    echo "$CRASHLOOP_PODS"
fi

# Check for ImagePullBackOff pods
IMAGE_PULL_PODS=$($CLI get pods --all-namespaces -o json | jq -r '.items[] | select(.status.containerStatuses[]? | .state.waiting?.reason == "ImagePullBackOff" or .state.waiting?.reason == "ErrImagePull") | "\(.metadata.namespace)/\(.metadata.name)"')

if [ -n "$IMAGE_PULL_PODS" ]; then
    CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$IMAGE_PULL_PODS" | wc -l)))
    echo "❌ CRITICAL: Pods with image pull errors:"
    echo "$IMAGE_PULL_PODS"
fi

7. Check Deployment/StatefulSet/DaemonSet Health

Verify workload controllers:

echo "Checking Deployments..."

# Check deployments with unavailable replicas
UNHEALTHY_DEPLOYMENTS=$($CLI get deployments --all-namespaces -o json | jq -r '.items[] | select(.status.unavailableReplicas > 0 or .status.replicas != .status.readyReplicas) | "\(.metadata.namespace)/\(.metadata.name) [Ready: \(.status.readyReplicas // 0)/\(.spec.replicas)]"')

if [ -n "$UNHEALTHY_DEPLOYMENTS" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$UNHEALTHY_DEPLOYMENTS" | wc -l)))
    echo "⚠️  WARNING: Deployments with unavailable replicas:"
    echo "$UNHEALTHY_DEPLOYMENTS"
fi

echo "Checking StatefulSets..."

UNHEALTHY_STATEFULSETS=$($CLI get statefulsets --all-namespaces -o json | jq -r '.items[] | select(.status.replicas != .status.readyReplicas) | "\(.metadata.namespace)/\(.metadata.name) [Ready: \(.status.readyReplicas // 0)/\(.spec.replicas)]"')

if [ -n "$UNHEALTHY_STATEFULSETS" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$UNHEALTHY_STATEFULSETS" | wc -l)))
    echo "⚠️  WARNING: StatefulSets with unavailable replicas:"
    echo "$UNHEALTHY_STATEFULSETS"
fi

echo "Checking DaemonSets..."

UNHEALTHY_DAEMONSETS=$($CLI get daemonsets --all-namespaces -o json | jq -r '.items[] | select(.status.numberReady != .status.desiredNumberScheduled) | "\(.metadata.namespace)/\(.metadata.name) [Ready: \(.status.numberReady)/\(.status.desiredNumberScheduled)]"')

if [ -n "$UNHEALTHY_DAEMONSETS" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$UNHEALTHY_DAEMONSETS" | wc -l)))
    echo "⚠️  WARNING: DaemonSets with unavailable pods:"
    echo "$UNHEALTHY_DAEMONSETS"
fi

8. Check Persistent Volume Claims

Check for storage issues:

echo "Checking Persistent Volume Claims..."

# Get PVCs that are not Bound
PENDING_PVCS=$($CLI get pvc --all-namespaces -o json | jq -r '.items[] | select(.status.phase != "Bound") | "\(.metadata.namespace)/\(.metadata.name) [\(.status.phase)]"')

if [ -n "$PENDING_PVCS" ]; then
    WARNING_ISSUES=$((WARNING_ISSUES + $(echo "$PENDING_PVCS" | wc -l)))
    echo "⚠️  WARNING: PVCs not in Bound state:"
    echo "$PENDING_PVCS"
fi

9. Check Critical Namespace Health

For OpenShift, check critical namespaces:

if [ "$CLUSTER_TYPE" = "OpenShift" ]; then
    echo "Checking Critical Namespaces..."

    CRITICAL_NAMESPACES="openshift-kube-apiserver openshift-etcd openshift-authentication openshift-console openshift-monitoring"

    for ns in $CRITICAL_NAMESPACES; do
        # Check if namespace exists
        if ! $CLI get namespace "$ns" &> /dev/null; then
            CRITICAL_ISSUES=$((CRITICAL_ISSUES + 1))
            echo "❌ CRITICAL: Critical namespace missing: $ns"
            continue
        fi

        # Check for failed pods in critical namespace
        FAILED_IN_NS=$($CLI get pods -n "$ns" -o json | jq -r '.items[] | select(.status.phase != "Running" and .status.phase != "Succeeded") | .metadata.name')

        if [ -n "$FAILED_IN_NS" ]; then
            CRITICAL_ISSUES=$((CRITICAL_ISSUES + $(echo "$FAILED_IN_NS" | wc -l)))
            echo "❌ CRITICAL: Failed pods in critical namespace $ns:"
            echo "$FAILED_IN_NS" | while read pod; do
                echo "  - $pod"
            done
        fi
    done
fi

10. Check Events for Recent Errors

Look for recent warning/error events:

echo "Checking Recent Events..."

# Get events from last 30 minutes with Warning or Error type
RECENT_WARNINGS=$($CLI get events --all-namespaces --field-selector type=Warning -o json | jq -r --arg since "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-30M +%Y-%m-%dT%H:%M:%SZ)" '.items[] | select(.lastTimestamp > $since) | "\(.lastTimestamp) [\(.involvedObject.namespace)/\(.involvedObject.name)]: \(.message)"' | head -20)

if [ -n "$RECENT_WARNINGS" ]; then
    echo "⚠️  Recent Warning Events (last 30 minutes):"
    echo "$RECENT_WARNINGS"
fi

11. Generate Summary Report

Create a summary of findings:

echo ""
echo "==============================================="
echo "Cluster Health Check Summary"
echo "==============================================="
echo "Cluster Type: $CLUSTER_TYPE"
echo "Cluster Version: $CLUSTER_VERSION"
echo "Check Time: $(date)"
echo ""
echo "Results:"
echo "  Critical Issues: $CRITICAL_ISSUES"
echo "  Warnings: $WARNING_ISSUES"
echo ""

if [ $CRITICAL_ISSUES -eq 0 ] && [ $WARNING_ISSUES -eq 0 ]; then
    echo "✅ Cluster is healthy - no issues detected"
    exit 0
elif [ $CRITICAL_ISSUES -gt 0 ]; then
    echo "❌ Cluster has CRITICAL issues requiring immediate attention"
    exit 1
else
    echo "⚠️  Cluster has warnings - monitoring recommended"
    exit 0
fi

12. Optional: Export to JSON Format

If --output-format json is specified, export findings as JSON:

{
  "cluster": {
    "type": "OpenShift",
    "version": "4.21.0",
    "checkTime": "2025-10-31T12:00:00Z"
  },
  "summary": {
    "criticalIssues": 2,
    "warnings": 5,
    "healthy": false
  },
  "findings": {
    "clusterOperators": {
      "degraded": ["authentication", "monitoring"],
      "unavailable": [],
      "progressing": ["network"]
    },
    "nodes": {
      "notReady": ["worker-1"],
      "schedulingDisabled": ["worker-2"],
      "underPressure": []
    },
    "pods": {
      "failed": ["namespace/pod-1", "namespace/pod-2"],
      "crashLooping": [],
      "imagePullErrors": ["namespace/pod-3"]
    },
    "workloads": {
      "unhealthyDeployments": [],
      "unhealthyStatefulSets": [],
      "unhealthyDaemonSets": []
    },
    "storage": {
      "pendingPVCs": []
    }
  }
}

Examples

Example 1: Basic health check

/openshift:cluster-health-check

Output:

Checking Cluster Operators...
✅ All cluster operators healthy

Checking Node Health...
⚠️  WARNING: Nodes with scheduling disabled:
  - ip-10-0-51-201.us-east-2.compute.internal

Checking Pod Health...
✅ All pods healthy

...

===============================================
Cluster Health Check Summary
===============================================
Cluster Type: OpenShift
Cluster Version: 4.21.0
Check Time: 2025-10-31 12:00:00

Results:
  Critical Issues: 0
  Warnings: 1

⚠️  Cluster has warnings - monitoring recommended

Example 2: Verbose health check

/openshift:cluster-health-check --verbose

Example 3: JSON output for automation

/openshift:cluster-health-check --output-format json

Return Value

The command returns different exit codes based on findings:

  • Exit 0: No critical issues found (cluster is healthy or has only warnings)
  • Exit 1: Critical issues detected requiring immediate attention

Output Format:

  • Text (default): Human-readable report with emoji indicators
  • JSON: Structured data suitable for parsing/automation

Common Issues and Remediation

Degraded Cluster Operators

Symptoms: Cluster operators showing Degraded=True or Available=False

Investigation:

oc get clusteroperator <operator-name> -o yaml
oc logs -n openshift-<operator-namespace> -l app=<operator-name>

Remediation: Check operator logs and events for specific errors

Nodes Not Ready

Symptoms: Nodes in NotReady state

Investigation:

oc describe node <node-name>
oc get events --field-selector involvedObject.name=<node-name>

Remediation: Common causes include network issues, disk pressure, or kubelet problems

Pods in CrashLoopBackOff

Symptoms: Pods continuously restarting

Investigation:

oc logs <pod-name> -n <namespace> --previous
oc describe pod <pod-name> -n <namespace>

Remediation: Check application logs, resource limits, and configuration

ImagePullBackOff Errors

Symptoms: Pods unable to pull container images

Investigation:

oc describe pod <pod-name> -n <namespace>

Remediation: Verify image name, registry credentials, and network connectivity

Security Considerations

  • Read-only access: This command only reads cluster state, no modifications
  • Sensitive data: Be cautious when sharing reports as they may contain cluster topology information
  • RBAC requirements: Ensure user has appropriate permissions for all resource types checked

See Also

Notes

  • The command checks cluster state at a point in time; transient issues may not be detected
  • For OpenShift clusters, cluster operator checks are performed
  • For vanilla Kubernetes, cluster operator checks are skipped
  • Resource utilization checks require metrics-server to be installed
  • Some checks may be skipped if user lacks sufficient permissions

Name

openshift:crd-review

Synopsis

/openshift:crd-review [repository-path]

Description

The openshift:crd-review command analyzes Go Kubernetes Custom Resource Definitions (CRDs) in a repository against both:

This command checks CRDs for compliance with these conventions, covering:

  • API naming conventions and patterns
  • Resource structure and field organization
  • Status field design and patterns
  • Field types and validation
  • Documentation standards
  • OpenShift-specific requirements

The review covers Go API type definitions, providing actionable feedback to improve API design.

Key Convention Checks

Kubernetes API Conventions

Naming Conventions
  • Resource Names: Must follow DNS label format (lowercase, alphanumeric, hyphens)
  • Field Names: PascalCase for Go, camelCase for JSON
  • Avoid: Abbreviations, underscores, ambiguous names
  • Include: Units/types in field names when needed (e.g., timeoutSeconds)
API Structure
  • Required Fields: Every API object must embed a k8s.io/apimachinery/pkg/apis/meta/v1 TypeMeta struct
  • Metadata: Every API object must include a k8s.io/apimachinery/pkg/apis/meta/v1 ObjectMeta struct called metadata
  • Spec/Status Separation: Clear separation between desired state (spec) and observed state (status)
Status Field Design
  • Conditions: Must include conditions array with:
    • type: Clear, human-readable condition type
    • status: True, False, or Unknown
    • reason: Machine-readable reason code
    • message: Human-readable message
    • lastTransitionTime: RFC 3339 timestamp
Field Types
  • Integers: Prefer int32 over int64
  • Avoid: Unsigned integers, floating-point values
  • Enums: Use string constants, not numeric values
  • Optional Fields: Use pointers in Go
Versioning
  • Group Names: Use domain format (e.g., myapp.example.com)
  • Version Strings: Must match DNS label format (e.g., v1, v1beta1)
  • Migration: Provide clear paths between versions

OpenShift API Conventions

Configuration vs Workload APIs
  • Configuration APIs: Typically cluster-scoped, manage cluster behavior
  • Workload APIs: Usually namespaced, user-facing resources
Field Design
  • Avoid Boolean Fields: Use enumerations that describe end-user behavior instead of binary true/false
    • ❌ Bad: paused: true
    • ✅ Good: lifecycle: "Paused" with enum values ["Paused", "Active"]
  • Object References: Use specific types, omit "Ref" suffix
  • Clear Semantics: Each field should have one clear purpose
Documentation Requirements
  • Godoc Comments: Comprehensive documentation for all exported types and fields
  • JSON Field Names: Use JSON names in documentation (not Go names)
  • User-Facing: Write for users, not just developers
  • Explain Interactions: Document how fields interact with each other
Validation
  • Kubebuilder Tags: Use validation markers (+kubebuilder:validation:*)
  • Enum Values: Explicitly define allowed values
  • Field Constraints: Define minimums, maximums, patterns
  • Meaningful Errors: Validation messages should guide users
Union Types
  • Discriminated Unions: Use a discriminator field to select variant
  • Optional Pointers: All union members should be optional pointers
  • Validation: Ensure exactly one union member is set

Implementation

The command performs the following analysis workflow:

  1. Repository Discovery

    • Find Go API types (typically in api/, pkg/apis/ directories)
    • Identify CRD generation markers (+kubebuilder comments)
  2. Kubernetes Convention Validation

    • Naming validation: Check resource names, field names, condition types
    • Structure validation: Verify required fields, metadata, spec/status separation
    • Status validation: Ensure conditions array, proper condition structure
    • Field type validation: Check integer types, avoid floats, validate enums
    • Versioning validation: Verify group names and version strings
  3. OpenShift Convention Validation

    • API classification: Identify configuration vs workload APIs
    • Field design: Flag boolean fields, check enumerations
    • Documentation: Verify Godoc comments, user-facing descriptions
    • Validation markers: Check kubebuilder validation tags
    • Union types: Validate discriminated union patterns
  4. Report Generation

    • List all findings with severity levels (Critical, Warning, Info)
    • Provide specific file and line references
    • Include remediation suggestions
    • Highlight whether a suggested change might lead to breaking API changes
    • Link to relevant convention documentation

Output Format

The command generates a structured report with:

  • Summary: Overview of findings by severity
  • Kubernetes Findings: Issues related to upstream conventions
  • OpenShift Findings: Issues related to OpenShift-specific patterns
  • Recommendations: Actionable steps to improve API design
  • openshift/api crd-command reference: Add a prominent note notifying the user of the existence of the openshift/api repository's api-review command (https://github.com/openshift/api/blob/master/.claude/commands/api-review.md) for PR reviews against that repository.

Each finding includes:

  • Severity level (❌ Critical, ⚠️ Warning, 💡 Info)
  • File location and line number
  • Description of the issue
  • Remediation suggestion
  • Link to relevant documentation

Examples

Example 1: Review current repository

/crd-review

Analyzes CRDs in the current working directory.

Example 2: Review specific repository

/crd-review /path/to/operator-project

Analyzes CRDs in the specified directory.

Example 3: Review with detailed output

The command automatically provides detailed output including:

  • All CRD files found
  • Go API type definitions
  • Compliance summary
  • Specific violations with file references

Common Findings

Kubernetes Convention Issues

Boolean vs Enum Fields

Issue: Using boolean where enum is better

// ❌ Bad
type MySpec struct {
    Enabled bool `json:"enabled"`
}

// ✅ Good
type MySpec struct {
    // State defines the operational state
    // Valid values are: "Enabled", "Disabled", "Auto"
    // +kubebuilder:validation:Enum=Enabled;Disabled;Auto
    State string `json:"state"`
}
Missing Status Conditions

Issue: Status without conditions array

// ❌ Bad
type MyStatus struct {
    Ready bool `json:"ready"`
}

// ✅ Good
type MyStatus struct {
    // Conditions represent the latest available observations
    // +listType=map
    // +listMapKey=type
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}
Improper Field Naming

Issue: Ambiguous or abbreviated names

// ❌ Bad
type MySpec struct {
    Timeout int `json:"timeout"` // Ambiguous unit
    Cnt     int `json:"cnt"`     // Abbreviation
}

// ✅ Good
type MySpec struct {
    // TimeoutSeconds is the timeout in seconds
    // +kubebuilder:validation:Minimum=1
    TimeoutSeconds int32 `json:"timeoutSeconds"`

    // Count is the number of replicas
    // +kubebuilder:validation:Minimum=0
    Count int32 `json:"count"`
}

OpenShift Convention Issues

Missing Documentation

Issue: Exported fields without Godoc

// ❌ Bad
type MySpec struct {
    Field string `json:"field"`
}

// ✅ Good
type MySpec struct {
    // field specifies the configuration field for...
    // This value determines how the operator will...
    // Valid values include...
    Field string `json:"field"`
}
Missing Validation

Issue: Fields without kubebuilder validation

// ❌ Bad
type MySpec struct {
    Mode string `json:"mode"`
}

// ✅ Good
type MySpec struct {
    // mode defines the operational mode
    // +kubebuilder:validation:Enum=Standard;Advanced;Debug
    // +kubebuilder:validation:Required
    Mode string `json:"mode"`
}

Best Practices

  1. Start with Conventions: Review conventions before writing APIs
  2. Use Code Generation: Leverage controller-gen and kubebuilder markers
  3. Document Early: Write Godoc comments as you define types
  4. Validate Everything: Add validation markers for all fields
  5. Review Regularly: Run this command during development and before PRs
  6. Follow Examples: Study well-designed APIs in OpenShift core

Arguments

  • repository-path (optional): Path to repository containing CRDs. Defaults to current working directory.

Exit Codes

  • 0: Analysis completed successfully
  • 1: Error during analysis (e.g., invalid path, no CRDs found)

See Also

Name

openshift:create-cluster

Synopsis

/openshift:create-cluster [release-image] [platform] [options]

Description

The create-cluster command automates the process of extracting the OpenShift installer from a release image (if not already present) and creating a new OpenShift Container Platform (OCP) cluster. It handles installer extraction from OCP release images, configuration preparation, and cluster creation in a streamlined workflow.

This command is useful for:

  • Setting up development/test clusters quickly

⚠️ When to Use This Tool

IMPORTANT: This is a last resort tool for advanced use cases. For most development workflows, you should use one of these better alternatives:

Recommended Alternatives

  1. Cluster Bot: Request ephemeral test clusters without managing infrastructure

    • No cloud credentials needed
    • Supports dependent PR testing
    • Automatically cleaned up
  2. Gangway

  3. Multi-PR Testing in CI: Test multiple dependent PRs together using /test-with commands

When to Use create-cluster

Only use this command when:

  • You need full control over cluster configuration
  • You're testing installer changes that aren't suitable for CI
  • You need a long-lived development cluster on your own cloud account
  • The alternatives don't meet your specific requirements

Note: This command requires significant setup (cloud credentials, pull secrets, DNS configuration, understanding of OCP versions). If you're new to OpenShift development, start with Cluster Bot or Gangway instead.

Prerequisites

Before using this command, ensure you have:

  1. OpenShift CLI (oc): Required to extract the installer from the release image

  2. Cloud Provider Credentials configured for your chosen platform:

    • AWS: ~/.aws/credentials configured with appropriate permissions
    • Azure: Azure CLI authenticated (az login)
    • GCP: The command will guide you through service account setup (either using an existing service account JSON or creating a new one)
    • vSphere: vCenter credentials
    • OpenStack: clouds.yaml configured
  3. Pull Secret: Download from Red Hat Console

  4. Domain/DNS Configuration:

    • AWS: Route53 hosted zone
    • Other platforms: Appropriate DNS setup

Arguments

The command accepts arguments in multiple ways:

Positional Arguments

/openshift:create-cluster [release-image] [platform]

Interactive Mode

If arguments are not provided, the command will interactively prompt for:

  • OpenShift release image
  • Platform (aws, azure, gcp, vsphere, openstack, none/baremetal)
  • Cluster name
  • Base domain
  • Pull secret location

Argument Details

  • release-image (required): OpenShift release image to extract the installer from

    • Production release: quay.io/openshift-release-dev/ocp-release:4.21.0-ec.2-x86_64
    • CI build: registry.ci.openshift.org/ocp/release:4.21.0-0.ci-2025-10-27-031915
    • Stable release: quay.io/openshift-release-dev/ocp-release:4.20.1-x86_64
    • The command will prompt for this if not provided
  • platform (optional): Target platform for the cluster

    • aws: Amazon Web Services
    • azure: Microsoft Azure
    • gcp: Google Cloud Platform
    • vsphere: VMware vSphere
    • openstack: OpenStack
    • none: Bare metal / platform-agnostic
    • Default: Prompts user to select
  • cluster-name (optional): Name for the cluster

    • Default: ocp-cluster
    • Must be DNS-compatible
  • base-domain (required): Base domain for the cluster

    • Example: example.com → Cluster API will be api.{cluster-name}.{base-domain}
  • pull-secret (required): Path to pull secret file

    • User will be prompted to provide the path
  • installer-dir (optional): Directory to store/find installer binaries

    • Default: ~/.openshift-installers

Implementation

The command performs the following steps:

1. Validate Prerequisites

Check that required tools and credentials are available:

  • Verify oc CLI is installed and available
  • Verify cloud provider credentials are configured (if applicable)
  • Confirm domain/DNS requirements

If any prerequisites are missing, provide clear instructions on how to configure them.

2. Get Release Image from User

If not provided as an argument, prompt the user for the OpenShift release image:

Please provide the OpenShift release image:

Examples:
  - Production release: quay.io/openshift-release-dev/ocp-release:4.21.0-ec.2-x86_64
  - CI build:          registry.ci.openshift.org/ocp/release:4.21.0-0.ci-2025-10-27-031915
  - Stable release:    quay.io/openshift-release-dev/ocp-release:4.20.1-x86_64

Release image:

Store the user's input as $RELEASE_IMAGE.

Extract version from image for naming:

# Parse version from image tag (e.g., "4.21.0-ec.2" or "4.21.0-0.ci-2025-10-27-031915")
VERSION=$(echo "$RELEASE_IMAGE" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+[^"]*' | head -1)

3. Determine Installer Location and Extract if Needed

INSTALLER_DIR="${installer-dir:-$HOME/.openshift-installers}"
INSTALLER_PATH="$INSTALLER_DIR/openshift-install-${VERSION}"

Check if installer directory exists:

  • If $INSTALLER_DIR does not exist:
    • Ask user for confirmation: "The installer directory $INSTALLER_DIR does not exist. Would you like to create it?"
    • If user confirms (yes): Create the directory with mkdir -p "$INSTALLER_DIR"
    • If user declines (no): Exit with error message suggesting an alternative path

Check if the installer already exists at $INSTALLER_PATH:

  • If present: Verify it works with "$INSTALLER_PATH" version
    • If version matches the release image: Skip extraction
    • If different or fails: Proceed with extraction
  • If not present: Proceed with extraction

Extract installer from release image:

  1. Verify oc CLI is available:

    if ! command -v oc &> /dev/null; then
        echo "Error: 'oc' CLI not found. Please install the OpenShift CLI."
        exit 1
    fi
    
  2. Extract the installer binary:

    oc adm release extract \
        --tools \
        --from="$RELEASE_IMAGE" \
        --to="$INSTALLER_DIR"
    

    This extracts the openshift-install binary and other tools from the release image.

  3. Locate and rename the extracted installer:

    # The extract command creates a tar.gz with the tools
    # Find the most recently extracted openshift-install tar (compatible with both GNU and BSD find)
    INSTALLER_TAR=$(find "$INSTALLER_DIR" -name "openshift-install-*.tar.gz" -type f -exec ls -t {} + | head -1)
    
    # Extract from tar and rename
    cd "$INSTALLER_DIR"
    tar -xzf "$INSTALLER_TAR" openshift-install
    mv openshift-install "openshift-install-${VERSION}"
    chmod +x "openshift-install-${VERSION}"
    
    # Clean up the tar file
    rm "$INSTALLER_TAR"
    
  4. Verify the installer:

    "$INSTALLER_PATH" version
    

    Expected output should show the version matching $VERSION.

4. Prepare Installation Directory

Create a clean installation directory:

INSTALL_DIR="${cluster-name}-install-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$INSTALL_DIR"
cd "$INSTALL_DIR"

5. Collect Required Information and Generate install-config.yaml

IMPORTANT: Do NOT run the installer interactively. Instead, collect all required information from the user and generate the install-config.yaml programmatically.

Step 5.1: Collect Information

Prompt the user for the following information (if not already provided as arguments):

  1. SSH Public Key:

    • Check for existing SSH keys: ls -la ~/.ssh/*.pub
    • Ask user to select from available keys or specify path
    • Default: ~/.ssh/id_rsa.pub
  2. Platform (if not provided as argument):

    • Ask user to select: aws, azure, gcp, vsphere, openstack, none
  3. Platform-specific details:

    • For AWS:
      • Region (e.g., us-east-1, us-west-2)
    • For Azure:
      • Region (e.g., centralus, eastus)
      • Cloud name (e.g., AzurePublicCloud)
    • For GCP:
      • Follow the GCP Service Account Setup (see Step 5.2a below)
      • Project ID
      • Region (e.g., us-central1)
    • For other platforms: collect required platform-specific info
  4. Base Domain:

    • Ask for base domain (e.g., example.com, devcluster.openshift.com)
    • Validate that domain is configured (e.g., Route53 hosted zone for AWS)
  5. Cluster Name:

    • Ask for cluster name or use default: ocp-cluster
    • Validate DNS compatibility (lowercase, hyphens only)
  6. Pull Secret:

    • IMPORTANT: Always ask user to provide the path to their pull secret file
    • Do NOT use default paths like ~/pull-secret.txt or ~/Downloads/pull-secret.txt
    • Prompt: "Please provide the path to your pull secret file (download from https://console.redhat.com/openshift/install/pull-secret):"
    • Read contents of pull secret file from the provided path

Step 5.2a: GCP Service Account Setup (Only for GCP platform)

If the platform is GCP, the installer requires a service account JSON file with appropriate permissions. Present the user with two options:

  1. Use an existing service account JSON file
  2. Create a new service account

Ask the user: "Do you want to use an existing service account JSON file or create a new one?"

Option 1: Use Existing Service Account

If the user chooses to use an existing service account:

  • Prompt: "Please provide the path to your GCP service account JSON file:"
  • Store the path as $GCP_SERVICE_ACCOUNT_PATH
  • Verify the file exists and is valid JSON
  • Set the environment variable:
    export GOOGLE_APPLICATION_CREDENTIALS="$GCP_SERVICE_ACCOUNT_PATH"
    

Option 2: Create New Service Account

If the user chooses to create a new service account:

  1. Verify gcloud CLI is installed:

    if ! command -v gcloud &> /dev/null; then
        echo "Error: 'gcloud' CLI not found. Please install the Google Cloud SDK."
        echo "Visit: https://cloud.google.com/sdk/docs/install"
        exit 1
    fi
    
  2. Prompt for Kerberos ID:

    • Ask: "Please provide your Kerberos ID (e.g., jsmith):"
    • Store as $KERBEROS_ID
    • Validate it's not empty
  3. Set service account name:

    SERVICE_ACCOUNT_NAME="${KERBEROS_ID}-development"
    
  4. Create the service account:

    echo "Creating service account: $SERVICE_ACCOUNT_NAME"
    gcloud iam service-accounts create "$SERVICE_ACCOUNT_NAME" --display-name="$SERVICE_ACCOUNT_NAME"
    
  5. Extract service account details:

    # Get service account information
    SERVICE_ACCOUNT_JSON="$(gcloud iam service-accounts list --format json | jq -r '.[] | select(.name | match("/\(env.SERVICE_ACCOUNT_NAME)@"))')"
    SERVICE_ACCOUNT_EMAIL="$(jq -r .email <<< "$SERVICE_ACCOUNT_JSON")"
    PROJECT_ID="$(jq -r .projectId <<< "$SERVICE_ACCOUNT_JSON")"
    
    echo "Service Account Email: $SERVICE_ACCOUNT_EMAIL"
    echo "Project ID: $PROJECT_ID"
    
  6. Grant required permissions:

    echo "Granting IAM roles to service account..."
    
    while IFS= read -r ROLE_TO_ADD ; do
       echo "Adding role: $ROLE_TO_ADD"
       gcloud projects add-iam-policy-binding "$PROJECT_ID" \
          --condition="None" \
          --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
          --role="$ROLE_TO_ADD"
    done << 'END_OF_ROLES'
    roles/compute.admin
    roles/iam.securityAdmin
    roles/iam.serviceAccountAdmin
    roles/iam.serviceAccountKeyAdmin
    roles/iam.serviceAccountUser
    roles/storage.admin
    roles/dns.admin
    roles/compute.loadBalancerAdmin
    roles/iam.roleAdmin
    END_OF_ROLES
    
    echo "All roles granted successfully."
    
  7. Create and download service account key:

    KEY_FILE="${HOME}/.gcp/${SERVICE_ACCOUNT_NAME}-key.json"
    mkdir -p "$(dirname "$KEY_FILE")"
    
    echo "Creating service account key..."
    gcloud iam service-accounts keys create "$KEY_FILE" \
       --iam-account="$SERVICE_ACCOUNT_EMAIL"
    
    echo "Service account key saved to: $KEY_FILE"
    
  8. Set environment variable:

    export GOOGLE_APPLICATION_CREDENTIALS="$KEY_FILE"
    echo "GOOGLE_APPLICATION_CREDENTIALS set to: $KEY_FILE"
    
  9. Store PROJECT_ID for later use in install-config.yaml generation.

Step 5.2: Generate install-config.yaml

Create the install-config.yaml file programmatically based on collected information:

# Read SSH public key
SSH_KEY=$(cat "$SSH_KEY_PATH")

# Read pull secret
PULL_SECRET=$(cat "$PULL_SECRET_PATH")

# Generate install-config.yaml
cat > install-config.yaml <<EOF
apiVersion: v1
baseDomain: ${BASE_DOMAIN}
metadata:
  name: ${CLUSTER_NAME}
compute:
- name: worker
  replicas: 3
controlPlane:
  name: master
  replicas: 3
networking:
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
platform:
  ${PLATFORM}:
    region: ${REGION}
pullSecret: '${PULL_SECRET}'
sshKey: '${SSH_KEY}'
EOF

Platform-specific configurations:

For AWS:

platform:
  aws:
    region: us-east-1

For Azure:

platform:
  azure:
    region: centralus
    baseDomainResourceGroupName: ${RESOURCE_GROUP_NAME}
    cloudName: AzurePublicCloud

For GCP:

platform:
  gcp:
    projectID: ${PROJECT_ID}
    region: us-central1

For None/Baremetal:

platform:
  none: {}

IMPORTANT: Always backup install-config.yaml after creation:

cp install-config.yaml install-config.yaml.backup

The installer consumes this file, so the backup is essential for reference.

6. Create the Cluster

Run the installer:

"$INSTALLER_PATH" create cluster --dir=.

Monitor the installation progress. This typically takes 30-45 minutes.

7. Post-Installation

Once installation completes:

  1. Display kubeconfig location:

    Kubeconfig: $INSTALL_DIR/auth/kubeconfig
    
  2. Display cluster credentials:

    Console URL: https://console-openshift-console.apps.${cluster-name}.${base-domain}
    Username: kubeadmin
    Password: (from $INSTALL_DIR/auth/kubeadmin-password)
    
  3. Export KUBECONFIG (offer to add to shell profile):

    export KUBECONFIG="$PWD/auth/kubeconfig"
    
  4. Verify cluster access:

    oc get nodes
    oc get co  # cluster operators
    
  5. Save cluster information to a summary file:

    Cluster: ${cluster-name}
    Version: ${VERSION}
    Release Image: ${RELEASE_IMAGE}
    Platform: ${platform}
    Console: https://console-openshift-console.apps.${cluster-name}.${base-domain}
    API: https://api.${cluster-name}.${base-domain}:6443
    Kubeconfig: $INSTALL_DIR/auth/kubeconfig
    Created: $(date)
    

8. Error Handling

If installation fails:

  1. Capture logs: Installation logs are in .openshift_install.log
  2. Provide diagnostics: Check common failure points:
    • Quota limits on cloud provider
    • DNS configuration issues
    • Invalid pull secret
    • Network/firewall issues
  3. Cleanup guidance: Inform user about cleanup:
    "$INSTALLER_PATH" destroy cluster --dir=.
    

Examples

Example 1: Basic cluster creation (interactive)

/openshift:create-cluster

The command will prompt for release image and all necessary information.

Example 2: Create AWS cluster with production release

/openshift:create-cluster quay.io/openshift-release-dev/ocp-release:4.21.0-ec.2-x86_64 aws

Example 3: Create cluster with CI build

/openshift:create-cluster registry.ci.openshift.org/ocp/release:4.21.0-0.ci-2025-10-27-031915 gcp

Cleanup

To destroy the cluster after testing:

cd $INSTALL_DIR
"$INSTALLER_PATH" destroy cluster --dir=.

WARNING: This will permanently delete all cluster resources.

Common Issues

  1. Pull secret not found:

  2. Insufficient cloud quotas:

    • Check cloud provider quota limits
    • Request quota increase if needed
  3. DNS issues:

    • Ensure base domain has a valid DNS zone configured
    • For AWS, verify Route53 hosted zone exists
  4. SSH key not found:

    • Generate with ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa
  5. Unauthorized access to release image:

    • Error: error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:...: unauthorized: access to the requested resource is not authorized
    • For quay.io/openshift-release-dev/ocp-v4.0-art-dev you can get the pull secret from https://console.redhat.com/openshift/install/pull-secret and save it in a file and provide it here.

Security Considerations

  • Pull secret: Contains authentication for Red Hat registries. Keep secure.
  • kubeadmin password: Stored in plaintext in auth directory. Rotate after cluster creation.
  • kubeconfig: Contains cluster admin credentials. Restrict file permissions and do not share.
  • Cloud credentials: Never commit to version control.

Return Value

  • Success: Returns 0 and displays cluster information including kubeconfig path
  • Failure: Returns non-zero and displays error diagnostics

See Also

Arguments:

  • $1 (release-image): OpenShift release image to extract the installer from (e.g., quay.io/openshift-release-dev/ocp-release:4.21.0-ec.2-x86_64)
  • $2 (platform): Target cloud platform for cluster deployment (aws, azure, gcp, vsphere, openstack, none)

Name

openshift:destroy-cluster

Synopsis

/openshift:destroy-cluster [install-dir]

Description

The destroy-cluster command safely destroys an OpenShift Container Platform (OCP) cluster that was previously created using the /openshift:create-cluster command. It locates the appropriate installer binary, verifies the cluster information, and performs cleanup of all cloud resources.

This command is useful for:

  • Cleaning up development/test clusters after testing
  • Removing failed cluster installations
  • Freeing up cloud resources and quotas

⚠️ WARNING: This operation is irreversible and will permanently delete:

  • All cluster resources (VMs, load balancers, storage, etc.)
  • All data stored in the cluster
  • All configuration and credentials
  • DNS records (if managed by the installer)

Prerequisites

Before using this command, ensure you have:

  1. Installation directory from the original cluster creation

    • Contains the cluster metadata and terraform state
    • Located at {cluster-name}-install-{timestamp} by default
  2. OpenShift installer binary that matches the cluster version

    • Should be available at ~/.openshift-installers/openshift-install-{version}
    • Same version used to create the cluster
  3. Cloud Provider Credentials still configured and valid

    • Same credentials used during cluster creation
    • Must have permissions to delete resources
  4. Network connectivity to the cloud provider

    • Required to communicate with cloud APIs

Arguments

  • install-dir (optional): Path to the cluster installation directory
    • Default: Interactive prompt to select from available installation directories
    • Must contain cluster metadata files (metadata.json, terraform.tfstate, etc.)
    • Example: ./my-cluster-install-20251028-120000

Implementation

The command performs the following steps:

1. Locate Installation Directory

If install-dir is not provided:

  • Search for installation directories in the current directory
  • Look for directories matching pattern *-install-* or containing .openshift_install_state.json
  • Present a list of found directories to the user for selection
  • Allow user to manually enter a path if directory not found

If install-dir is provided:

  • Validate the directory exists
  • Verify it contains cluster metadata files

2. Extract Cluster Information

Read cluster details from the installation directory:

# Read cluster metadata
if [ -f "$INSTALL_DIR/metadata.json" ]; then
    CLUSTER_NAME=$(jq -r '.clusterName' "$INSTALL_DIR/metadata.json")
    INFRA_ID=$(jq -r '.infraID' "$INSTALL_DIR/metadata.json")
    PLATFORM=$(jq -r '.platform' "$INSTALL_DIR/metadata.json")
fi

# Try to extract version from cluster-info or log files
VERSION=$(grep -oE 'openshift-install.*v[0-9]+\.[0-9]+\.[0-9]+' "$INSTALL_DIR/.openshift_install.log" | head -1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+[^"]*' | head -1)

3. Display Cluster Information and Confirm

Show the user what will be destroyed:

Cluster Information:
  Name: ${CLUSTER_NAME}
  Infrastructure ID: ${INFRA_ID}
  Platform: ${PLATFORM}
  Installation Directory: ${INSTALL_DIR}
  Version: ${VERSION}

⚠️  WARNING: This will permanently destroy the cluster and all its resources!

This action will delete:
  - All cluster VMs and compute resources
  - Load balancers and networking resources
  - Storage volumes and persistent data
  - DNS records
  - All cluster configuration

Are you sure you want to destroy this cluster? (yes/no):

Important: Require the user to type "yes" (not just "y") to confirm destruction.

4. Locate the Correct Installer

Find the installer binary that matches the cluster version:

INSTALLER_DIR="${HOME}/.openshift-installers"
INSTALLER_PATH="$INSTALLER_DIR/openshift-install-${VERSION}"

# Check if the version-specific installer exists
if [ ! -f "$INSTALLER_PATH" ]; then
    echo "Warning: Installer for version ${VERSION} not found at ${INSTALLER_PATH}"
    echo "Searching for alternative installers..."

    # Look for any installer in the installers directory
    AVAILABLE_INSTALLERS=$(find "$INSTALLER_DIR" -name "openshift-install-*" -type f 2>/dev/null)

    if [ -n "$AVAILABLE_INSTALLERS" ]; then
        echo "Found installers:"
        echo "$AVAILABLE_INSTALLERS"
        echo ""
        echo "You may use a different version installer, but this may cause issues."
        echo "Would you like to:"
        echo "  1. Use an available installer from the list above"
        echo "  2. Extract the correct installer from the release image"
        echo "  3. Cancel the operation"
    else
        echo "No installers found. Would you like to extract the installer? (yes/no):"
    fi
fi

# Verify installer works
"$INSTALLER_PATH" version

5. Backup Important Files (Optional)

Offer to backup key files before destruction:

Would you like to backup cluster information before destroying? (yes/no):

If yes, create a backup:

BACKUP_DIR="${INSTALL_DIR}-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Backup key files
cp "$INSTALL_DIR/metadata.json" "$BACKUP_DIR/" 2>/dev/null
cp "$INSTALL_DIR/auth/kubeconfig" "$BACKUP_DIR/" 2>/dev/null
cp "$INSTALL_DIR/auth/kubeadmin-password" "$BACKUP_DIR/" 2>/dev/null
cp "$INSTALL_DIR/.openshift_install.log" "$BACKUP_DIR/" 2>/dev/null
cp "$INSTALL_DIR/install-config.yaml.backup" "$BACKUP_DIR/" 2>/dev/null

echo "Backup created at: $BACKUP_DIR"

6. Run Cluster Destroy

Execute the destroy command:

cd "$INSTALL_DIR"

echo "Starting cluster destruction..."
echo "This may take 10-15 minutes..."

"$INSTALLER_PATH" destroy cluster --dir=. --log-level=debug

DESTROY_EXIT_CODE=$?

Monitor the destruction progress and display status updates.

7. Verify Cleanup

After the destroy command completes:

  1. Check exit code:

    if [ $DESTROY_EXIT_CODE -eq 0 ]; then
        echo "✅ Cluster destroyed successfully"
    else
        echo "❌ Cluster destruction failed with exit code: $DESTROY_EXIT_CODE"
        echo "Check logs at: $INSTALL_DIR/.openshift_install.log"
    fi
    
  2. Verify cloud resources (platform-specific):

    • AWS: Check for lingering resources with tag kubernetes.io/cluster/${INFRA_ID}
    • Azure: Verify resource group deletion
    • GCP: Check project for remaining resources
  3. List any remaining resources:

    If any resources remain, provide commands to manually clean them up.
    

8. Cleanup Installation Directory (Optional)

Ask the user if they want to remove the installation directory:

The cluster has been destroyed. Would you like to delete the installation directory? (yes/no):
  Directory: $INSTALL_DIR
  Size: $(du -sh "$INSTALL_DIR" | cut -f1)

If yes:

rm -rf "$INSTALL_DIR"
echo "Installation directory removed"

If no:

echo "Installation directory preserved at: $INSTALL_DIR"
echo "You can manually remove it later with: rm -rf $INSTALL_DIR"

9. Display Summary

Show final summary:

Cluster Destruction Summary:
  Cluster Name: ${CLUSTER_NAME}
  Status: Successfully destroyed
  Platform: ${PLATFORM}
  Duration: ${DURATION}
  Backup: ${BACKUP_DIR} (if created)

Next steps:
  - Verify your cloud console for any lingering resources
  - Check your cloud billing to ensure resources are no longer incurring charges
  - Remove installation directory if not already deleted: ${INSTALL_DIR}

Error Handling

If destruction fails, the command should:

  1. Capture error logs from .openshift_install.log
  2. Identify the failure point:
    • Timeout waiting for resource deletion
    • Permission errors
    • API rate limiting
    • Network connectivity issues
    • Resources locked or in use
  3. Provide recovery options:
    • Retry the destroy operation
    • Manual cleanup instructions for specific resources
    • Contact support if critical errors occur

Common failure scenarios:

Timeout errors:

# Some resources may take longer to delete
# Retry the destroy command:
"$INSTALLER_PATH" destroy cluster --dir="$INSTALL_DIR"

Permission errors:

Error: Cloud credentials may have expired or lack permissions
Solution:
  1. Verify cloud credentials are still valid
  2. Check IAM permissions for resource deletion
  3. Re-run the destroy command after fixing credentials

Partial destruction:

Warning: Some resources could not be deleted automatically.

Remaining resources:
  - Load balancer: ${LB_NAME}
  - Security group: ${SG_NAME}
  - S3 bucket: ${BUCKET_NAME}

Manual cleanup commands:
  [Platform-specific commands to delete remaining resources]

Examples

Example 1: Destroy cluster with interactive directory selection

/openshift:destroy-cluster

The command will search for installation directories and prompt you to select one.

Example 2: Destroy cluster with specific directory

/openshift:destroy-cluster ./my-cluster-install-20251028-120000

Example 3: Destroy cluster with full path

/openshift:destroy-cluster /home/user/clusters/test-cluster-install-20251028-120000

Common Issues

  1. Installation directory not found:

    • Ensure you're in the correct directory
    • Provide the full path to the installation directory
    • Check if the directory was moved or renamed
  2. Installer binary not found:

    • The command will help you extract the correct installer
    • Alternatively, manually place the installer in ~/.openshift-installers/
  3. Cloud credentials expired:

    • Refresh your cloud credentials
    • Re-authenticate with the cloud provider CLI
    • Re-run the destroy command
  4. Resources already deleted manually:

    • The destroy command may fail if resources were manually deleted
    • Check the logs and manually clean up any remaining resources
    • Remove the installation directory manually
  5. Destroy hangs or times out:

    • Some resources may take longer to delete (especially load balancers)
    • Wait for the operation to complete (can take 15-30 minutes)
    • If truly stuck, cancel and retry
    • Check cloud console for resource status

Safety Features

This command includes several safety measures:

  1. Confirmation required: Must type "yes" to proceed
  2. Cluster information displayed: Shows what will be destroyed before proceeding
  3. Backup option: Offers to backup important files
  4. Validation checks: Verifies installation directory and metadata
  5. Detailed logging: All operations logged for troubleshooting
  6. Error recovery: Provides manual cleanup instructions if automated cleanup fails

Return Value

  • Success: Returns 0 and displays destruction summary
  • Failure: Returns non-zero and displays error diagnostics with recovery instructions

See Also

Arguments:

  • $1 (install-dir): Path to the cluster installation directory created by create-cluster (optional, interactive if not provided)

Name

openshift:expand-test-case

Synopsis

/openshift:expand-test-case [test-idea-or-file-or-commands] [format]

Description

The expand-test-case command transforms basic test ideas or existing oc commands into comprehensive test scenarios. It accepts three types of input:

  1. Test idea: Simple description of what to test (e.g., "verify pod deployment")
  2. File path: Path to existing test file to expand (e.g., /path/to/test.sh or /path/to/test.go)
  3. oc commands: Direct oc CLI commands to analyze and expand (e.g., oc create pod nginx)

The command expands the input to cover positive flows, negative scenarios, edge cases, and boundary conditions, helping QE engineers ensure thorough test coverage.

Supports two output formats:

  • oc CLI: Shell scripts with oc commands for manual or automated execution
  • Ginkgo: Go test code using Ginkgo/Gomega framework for E2E tests

Implementation

The command analyzes the input and generates comprehensive scenarios:

  1. Parse Input: Determine if input is a test idea, file path, or oc commands
    • If file path: Read and analyze existing test code
    • If oc commands: Parse commands to understand what's being tested
    • If test idea: Understand the core feature or behavior
  2. Identify Test Dimensions: Determine coverage aspects (functionality, security, performance, edge cases)
  3. Generate Positive Tests: Happy path scenarios where everything works
  4. Generate Negative Tests: Error handling, invalid inputs, permission issues
  5. Add Edge Cases: Boundary values, race conditions, resource limits
  6. Define Validation: Clear success criteria and assertions
  7. Format Output: Generate in requested format (oc CLI or Ginkgo) - MUST follow the standards in "Test Coverage Guidelines" section below

CRITICAL: All generated test scenarios MUST adhere to the coverage dimensions, best practices, and standards defined in the "Test Coverage Guidelines" section below. Use the referenced examples and patterns from OpenShift origin repository.

Test Coverage Guidelines

The command generates comprehensive test scenarios following industry best practices:

Test Coverage Dimensions:

  • Positive Tests: Valid inputs and expected workflows
  • Negative Tests: Invalid inputs, permission errors, missing dependencies
  • Edge Cases: Boundary values (0, max values, empty inputs, special characters)
  • Security Tests: RBAC validation, security context enforcement, privilege escalation
  • Resource Tests: Low memory, disk pressure, network issues, rate limiting
  • Concurrency: Multiple operations happening simultaneously
  • Failure Recovery: Restart behavior, cleanup on failure

References:

Best Practices Applied:

  • Use stable, descriptive test names (no dynamic IDs or timestamps)
  • Ensure proper resource cleanup (prevent resource leaks)
  • Include meaningful assertions with clear failure messages
  • Isolate tests (each test creates its own resources)
  • Add appropriate timeouts to prevent hanging tests
  • Follow Ginkgo patterns: Describe/Context/It hierarchy
  • Use framework helpers: e2epod, e2enode, e2enamespace

Arguments

  • $1 (test-idea-or-file-or-commands): One of:
    • Test idea: Description of what to test
    • File path: Path to existing test file
    • oc commands: Set of oc CLI commands to analyze and expand
  • $2 (format): Output format - "oc CLI" or "Ginkgo" (optional, will prompt if not provided)

Name

openshift:ironic-status

Synopsis

/openshift:ironic-status

Description

The openshift:ironic-status command checks the status of Ironic baremetal nodes in an OpenShift cluster.

This command is useful for:

  • Monitoring baremetal node health and provisioning status
  • Troubleshooting node provisioning issues
  • Verifying node enrollment and availability
  • Checking node maintenance states
  • Diagnosing baremetal infrastructure problems

Prerequisites

Before using this command, ensure you have:

  1. OpenShift CLI (oc): Must be installed and configured

  2. Active cluster connection: Must be connected to an OpenShift cluster with baremetal nodes

    • Verify with: oc whoami
    • Ensure KUBECONFIG is set if needed
  3. Sufficient permissions: Must have access to the openshift-machine-api namespace

    • Ability to exec into pods in openshift-machine-api namespace
    • Read access to services and deployments
  4. baremetal cluster: The cluster must be running on baremetal infrastructure with Metal3/Ironic enabled

    • Verify Metal3 components: oc get pods -n openshift-machine-api | grep metal3-ironic

Implementation

The command performs the following steps to retrieve Ironic node status:

1. Detect Ironic Service Endpoint

Locate the Ironic service in the openshift-machine-api namespace:

# Find the Ironic service (typically named metal3-state)
oc get service -n openshift-machine-api metal3-state

# Get the ClusterIP of the Ironic service
IRONIC_SERVICE=$(oc get service -n openshift-machine-api metal3-state)
IRONIC_IP=$(oc get service -n openshift-machine-api $IRONIC_SERVICE -o jsonpath='{.spec.clusterIP}')

# Ironic API typically runs on port 6385
IRONIC_ENDPOINT="https://localhost:6385"

2. Retrieve Ironic Credentials

Extract authentication credentials from the baremetal operator container:

# Find the metal3-baremetal-operator pod
BAREMETAL_OPERATOR_POD=$(oc get pods -n openshift-machine-api -l baremetal.openshift.io/cluster-baremetal-operator=metal3-baremetal-operator -o jsonpath='{.items[0].metadata.name}')

# Extract username from the mounted auth volume
IRONIC_USERNAME=$(oc exec -n openshift-machine-api $BAREMETAL_OPERATOR_POD -c metal3-baremetal-operator -- cat /auth/ironic/username)

# Extract password from the mounted auth volume
IRONIC_PASSWORD=$(oc exec -n openshift-machine-api $BAREMETAL_OPERATOR_POD -c metal3-baremetal-operator -- cat /auth/ironic/password)

3. Query Ironic Node List

Execute the baremetal client command to retrieve node status:

# Find the metal3-ironic pod
# Example: oc get pods -n openshift-machine-api
# Output may show: metal3-65bd97647f-hbd48

# Execute baremetal node list command
oc exec -n openshift-machine-api $METAL3_POD -c metal3-ironic -- \
  baremetal \
    --os-cacert /certs/ironic/tls.crt
    --os-auth-type http_basic \
    --os-username "$IRONIC_USERNAME" \
    --os-password "$IRONIC_PASSWORD" \
    --os-endpoint "$IRONIC_ENDPOINT" \
    node list

4. Parse and Present Node Information

The output typically includes:

  • UUID: Unique identifier for each node
  • Name: Node name (usually matches BareMetalHost name)
  • Instance UUID: Associated instance if provisioned
  • Power State: Current power status (power on, power off)
  • Provisioning State: Current provisioning status (available, active, deploying, etc.)
  • Maintenance: Whether the node is in maintenance mode

5. Error Handling

Handle common error scenarios:

# Check if Metal3 components are running
if ! oc get pods -n openshift-machine-api | grep -q metal3; then
    echo "Error: Metal3 components not found. This may not be a baremetal cluster."
    exit 1
fi

# Check if credentials were retrieved successfully
if [ -z "$IRONIC_USERNAME" ] || [ -z "$IRONIC_PASSWORD" ]; then
    echo "Error: Failed to retrieve Ironic credentials"
    exit 1
fi

# Handle connection failures
if ! oc exec ... 2>&1 | grep -q "UUID"; then
    echo "Error: Failed to connect to Ironic API"
    echo "Check that the Ironic service is accessible"
    exit 1
fi

Adaptation Guidance

If expected values don't match your environment, use these techniques to discover the correct values:

Finding the Ironic API Port

If port 6385 doesn't work, discover the actual port:

# Check service ports
oc get service -n openshift-machine-api metal3-state -o jsonpath='{.spec.ports[*].port}'

# Then update IRONIC_ENDPOINT
IRONIC_ENDPOINT="https://localhost:${DISCOVERED_PORT}"

Finding Services and Pods

If service/pod names differ from expected patterns:

# List all services/pods in the namespace
oc get services -n openshift-machine-api
oc get pods -n openshift-machine-api

# Filter by labels
oc get services -n openshift-machine-api -l app=metal3
oc get pods -n openshift-machine-api -l app=metal3 -o jsonpath='{.items[0].metadata.name}'

Finding Container Names and Credential Paths

If container names or credential paths are different:

# List containers in a pod
oc get pod -n openshift-machine-api <pod-name> -o jsonpath='{.spec.containers[*].name}'

# Find credential files
oc exec -n openshift-machine-api $POD -c $CONTAINER -- find /auth -type f 2>/dev/null

Alternative Commands

If baremetal command is unavailable, try:

# Check what's available
oc exec -n openshift-machine-api $METAL3_POD -c metal3-ironic -- which openstack

# Try alternatives: "openstack baremetal node list" or "ironic node-list"

# Or use curl directly
oc exec -n openshift-machine-api $METAL3_POD -c metal3-ironic -- \
  curl -k -u "$IRONIC_USERNAME:$IRONIC_PASSWORD" "$IRONIC_ENDPOINT/v1/nodes"

General strategy: When commands fail, list resources with oc get, check labels with -l, inspect with oc describe, and explore interactively with oc exec.

Return Value

The command outputs a table with the following columns:

  • Format: ASCII table with columns for UUID, Name, Instance UUID, Power State, Provisioning State, and Maintenance

Example output:

+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name                   | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+
| 12345678-1234-1234-1234-123456789012 | openshift-worker-0     | abcdef12-3456-7890-abcd-ef1234567890 | power on    | active             | False       |
| 23456789-2345-2345-2345-234567890123 | openshift-worker-1     | bcdef123-4567-8901-bcde-f12345678901 | power on    | active             | False       |
| 34567890-3456-3456-3456-345678901234 | openshift-worker-2     | None                                 | power off   | available          | False       |
+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+

Exit codes:

  • 0: Successfully retrieved and displayed node status
  • 1: Error occurred (Metal3 not found, credential retrieval failed, connection error)

Examples

Example 1: Basic usage

/openshift:ironic-status

Output:

Detecting Ironic endpoint...
Found Ironic service: metal3-state at 172.30.123.45:6385

Retrieving Ironic credentials...
Successfully retrieved credentials from baremetal operator

Querying Ironic node list...

+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name                   | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+
| 12345678-1234-1234-1234-123456789012 | openshift-worker-0     | abcdef12-3456-7890-abcd-ef1234567890 | power on    | active             | False       |
| 23456789-2345-2345-2345-234567890123 | openshift-worker-1     | bcdef123-4567-8901-bcde-f12345678901 | power on    | active             | False       |
| 34567890-3456-3456-3456-345678901234 | openshift-worker-2     | None                                 | power off   | available          | False       |
+--------------------------------------+------------------------+--------------------------------------+-------------+--------------------+-------------+

Summary: 3 nodes total (2 active, 1 available)

Common Provisioning States

Understanding the provisioning states:

  • available: Node is ready to be provisioned
  • active: Node is provisioned and in use
  • deploying: Node is currently being provisioned
  • deploy failed: Provisioning attempt failed
  • cleaning: Node is being cleaned for reuse
  • manageable: Node is manageable but not available for deployment
  • inspect failed: Introspection failed
  • error: Node is in an error state

Troubleshooting

Metal3 Components Not Found

If Metal3 components are not running:

oc get pods -n openshift-machine-api
# Look for pods starting with 'metal3-'

# Check BareMetalHost resources
oc get baremetalhosts -n openshift-machine-api

Credential Retrieval Failure

If unable to retrieve credentials:

# Check if the secret exists
oc get secrets -n openshift-machine-api | grep metal3

# Verify the baremetal operator pod is running
oc get pods -n openshift-machine-api -l k8s-app=metal3-baremetal-operator

Connection Timeout

If unable to connect to Ironic API:

# Check if the Ironic pod is running
oc get pods -n openshift-machine-api -l app=metal3

# Check Ironic logs
oc logs -n openshift-machine-api <metal3-pod-name> -c metal3-ironic

Security Considerations

  • Credentials: The command retrieves sensitive Ironic credentials; ensure output is not shared publicly
  • Cluster access: Requires exec permissions into cluster pods
  • Read-only operation: This command only reads node status and does not modify any resources

See Also

Notes

  • This command is specific to OpenShift clusters deployed on baremetal infrastructure
  • The Ironic service endpoint and pod names may vary depending on the OpenShift version
  • Ensure you have network connectivity to the cluster

Name

openshift:new-e2e-test

Synopsis

/new-e2e-test [test-specification]

Description

The new-e2e-test command assists in writing and validating new tests for the OpenShift test suite. It follows best practices for Ginkgo-based testing and ensures test reliability through automated validation.

This command handles the complete lifecycle of test development:

  • Writes tests following Ginkgo patterns and OpenShift conventions
  • Validates tests for reliability through multiple test runs
  • Ensures proper test naming and structure
  • Handles both origin repository and extension tests according to their repository conventions

Test Framework Guidelines

Ginkgo Framework

  • OpenShift-tests uses Ginkgo as its testing framework
  • Tests are organized in a BDD (Behavior-Driven Development) style with Describe/Context/It blocks
  • All tests should follow Ginkgo patterns and conventions except
    • You MUST NOT use BeforeAll, AfterAll hooks
    • MUST NOT use ginkgo.Serial, instead use the [Serial] annotation in the test name if non-parallel execution is required

Repository-Specific Guidelines

Origin Repository Tests

If working in the "origin" code repository:

  • All tests should go into the test/extended directory
  • If creating a new package, import it into test/extended/include.go
  • After writing your test, MUST rebuild the openshift-tests binary using make openshift-tests
Other repositories

Other repositories have have different conventions for locations of tests and how they get imported. Examine the code base and follow the conventions defined.

Critical Test Requirements

Test Names

CRITICAL: Test names must be stable and deterministic.

❌ NEVER Include Dynamic Information:
  • Pod names (e.g., "test-pod-abc123")
  • Timestamps
  • Random UUIDs or generated identifiers
  • Node names
  • Namespace names with random suffixes
  • Limits that may change later
✅ ALWAYS Use Descriptive, Static Names:
  • Good example: "should create a pod with custom security context"

  • Bad example: "should create pod test-pod-xyz123 with custom security context"

  • Good example: "should create a pod within a reasonable timeframe"

  • Bad example: "should create a pod within 15 seconds"

Results

CRITICAL: Tests must always produce a pass, fail or skip result. Do not create tests that only produce pass or only produce a fail result.

Test Structure Guidelines

Best Practices

  • Tests should be focused and test one specific behavior
  • Use proper setup and cleanup in BeforeEach/AfterEach blocks
  • Include appropriate timeouts for operations
  • Add meaningful assertions with clear failure messages
  • Follow existing patterns in the codebase for consistency

Implementation

The command performs the following steps:

  1. Analyze Specification: Parse the test specification provided by the user
  2. Write Test: Create a new test file following Ginkgo and OpenShift conventions
    • Determine correct location
    • Follow proper test structure
    • Use stable, descriptive naming
    • Implement proper setup/cleanup
  3. Build Binary: Rebuild the appropriate test binary (openshift-tests or a test extension)

Arguments

  • $1 (test-specification): Description of the test behavior to validate. Should clearly specify:
    • What feature/behavior to test
    • Expected outcomes
    • Any specific conditions or configurations

Name

openshift:node-kernel-conntrack

Synopsis

/openshift:node-kernel-conntrack <node> <image> [--command <cmd>] [--filter <params>]

Description

The openshift:node-kernel-conntrack command allows interaction with the connection tracking system of a Kubernetes node. Use this command to discover all or a filtered selection of currently tracked connections. This is essential for debugging NAT issues, connection state problems, and understanding packet flows through the kernel's netfilter subsystem.

Connection tracking (conntrack) is a critical component of Linux networking that maintains state information about network connections. In OVN-Kubernetes environments, conntrack is used for:

  • Service load balancing and NAT
  • Network policy enforcement
  • Connection state tracking across network namespace boundaries
  • Debugging intermittent connectivity issues

The command uses oc debug to create an ephemeral container with host network access, then executes conntrack operations. If the provided image doesn't contain the conntrack CLI utility, the command falls back to printing /proc/net/nf_conntrack directly.

Implementation

This command invokes the openshift-node-kernel skill which provides kernel-level connection tracking inspection capabilities:

  1. Parameter Validation: Validates required node and image parameters
  2. Utility Check: Verifies if the debug image contains the conntrack utility
  3. Command Execution: Uses oc debug to create ephemeral container and execute conntrack commands
  4. Fallback Handling: If conntrack utility is not available, falls back to reading /proc/net/nf_conntrack
  5. Output Filtering: Removes common oc debug warnings to provide clean output

Skill Reference:

  • Implementation: plugins/openshift/skills/openshift-node-kernel/node-kernel-conntrack.sh
  • Helper functions: plugins/openshift/skills/openshift-node-kernel/kernel-helper.sh
  • Documentation: plugins/openshift/skills/openshift-node-kernel/SKILL.md

Parameters

Required Parameters

  • node: Name of the Kubernetes node where conntrack entries should be extracted.

    • Must be a valid node name in the cluster
    • Example: ovn-control-plane, ovn-worker, worker-0
  • image: Container image to use for creating a debug connection to the node.

    • Should contain networking utilities (preferably with conntrack CLI)
    • Common images:
      • registry.redhat.io/rhel9/support-tools (Red Hat)
      • nicolaka/netshoot (community)
    • If image lacks conntrack utility, /proc/net/nf_conntrack will be printed

Optional Parameters

  • --command <cmd>: Conntrack operation to perform (requires conntrack CLI in image)

    • -L, --dump: List connection tracking table (default)
    • -C, --count: Show table counter
    • -S, --stats: Show in-kernel connection tracking statistics
  • --filter <params>: Filter parameters to narrow down results

    • -s, --src, --orig-src <IP>: Match source address in original direction
    • -d, --dst, --orig-dst <IP>: Match destination address in original direction
    • -p, --proto <PROTO>: Specify layer 4 protocol (TCP, UDP, etc.)
    • --sport, --orig-port-src <PORT>: Source port in original direction
    • --dport, --orig-port-dst <PORT>: Destination port in original direction

Return Value

The command returns connection tracking entries in the following format:

tcp 6 91 ESTABLISHED src=1.2.3.4 dst=5.6.7.8 sport=32000 dport=10250 src=5.6.7.8 dst=1.2.3.4 sport=10250 dport=32000 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=2

Each entry contains:

  • Protocol (tcp/udp/icmp)
  • Protocol number
  • TTL (time to live in seconds)
  • Connection state (ESTABLISHED, TIME_WAIT, etc.)
  • Original direction: source/destination IP and ports
  • Reply direction: source/destination IP and ports
  • Connection flags ([ASSURED], [UNREPLIED], etc.)
  • Connection mark
  • Security context
  • Usage counter

Examples

Example 1: List all conntrack entries

/openshift:node-kernel-conntrack ovn-worker registry.redhat.io/rhel9/support-tools --command -L

Output:

tcp      6 431999 ESTABLISHED src=10.0.0.5 dst=10.96.0.1 sport=54321 dport=443 src=10.96.0.1 dst=10.0.0.5 sport=443 dport=54321 [ASSURED] mark=0 use=1
udp      17 29 src=10.0.0.5 dst=8.8.8.8 sport=45678 dport=53 src=8.8.8.8 dst=10.0.0.5 sport=53 dport=45678 [ASSURED] mark=0 use=1

Example 2: Show conntrack statistics

/openshift:node-kernel-conntrack ovn-control-plane nicolaka/netshoot --command -S

Output:

cpu=0   found=1234 invalid=0 insert=567 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0
cpu=1   found=2345 invalid=0 insert=678 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0

Example 3: Filter by specific source and destination

/openshift:node-kernel-conntrack ovn-worker registry.redhat.io/rhel9/support-tools --command -L --filter "-s 10.244.0.5 -d 10.96.0.1 -p tcp"

Output:

tcp      6 86399 ESTABLISHED src=10.244.0.5 dst=10.96.0.1 sport=38456 dport=443 src=10.96.0.1 dst=10.244.0.5 sport=443 dport=38456 [ASSURED] mark=0 use=1

Example 4: Filter by port

/openshift:node-kernel-conntrack ovn-worker nicolaka/netshoot --filter "--dport 10250 -p tcp"

Output:

tcp      6 299 ESTABLISHED src=10.0.0.1 dst=10.0.0.5 sport=45678 dport=10250 src=10.0.0.5 dst=10.0.0.1 sport=10250 dport=45678 [ASSURED] mark=0 use=2

Example 5: Count total conntrack entries

/openshift:node-kernel-conntrack ovn-control-plane registry.redhat.io/rhel9/support-tools --command -C

Output:

12847

Troubleshooting

Image doesn't have conntrack utility

If the specified image doesn't contain the conntrack CLI tool, the command automatically falls back to reading /proc/net/nf_conntrack. The output format is the same, but advanced filtering options won't be available.

Solution: Use an image with conntrack utility:

  • nicolaka/netshoot

No entries returned

This could indicate:

  1. No active connections matching the filter criteria
  2. Connection tracking module not loaded

Solution: Check without filters first, then verify kernel modules:

/openshift:node-kernel-ip <node> <image> --command "netns show"

Permission denied errors

The debug pod might not have sufficient privileges.

Solution: Ensure your OpenShift user has permissions to create debug pods with host access.

See Also

  • /openshift:node-kernel-iptables - Inspect packet filter rules
  • /openshift:node-kernel-nft - Inspect nftables configuration
  • /openshift:node-kernel-ip - Inspect network interfaces and routing
  • Netfilter conntrack documentation

Name

openshift:node-kernel-ip

Synopsis

/openshift:node-kernel-ip <node> <image> --command <cmd> [--options <opts>] [--filter <params>]

Description

The openshift:node-kernel-ip command allows interaction with the kernel to list routing tables, network devices, and interfaces. The ip utility is the modern Linux networking configuration tool that shows and manipulates routing, network devices, policy routing, and tunnels.

In OVN-Kubernetes environments, the ip command is essential for:

  • Inspecting OVN overlay network interfaces (genev_sys_*, ovn-k8s-mp0, etc.)
  • Viewing routing tables for pod and service traffic
  • Debugging network namespace configurations
  • Analyzing interface states and configurations
  • Troubleshooting connectivity issues between nodes and pods

The command uses oc debug to create an ephemeral container with host network access, then executes ip operations. The image must contain the ip utility (from iproute2 package).

Implementation

This command invokes the openshift-node-kernel skill which provides kernel-level networking inspection capabilities:

  1. Parameter Validation: Validates required node, image, and command parameters
  2. Utility Check: Verifies the debug image contains the ip utility
  3. Command Execution: Uses oc debug to create ephemeral container and execute ip commands
  4. Output Filtering: Removes common oc debug warnings to provide clean output

Skill Reference:

  • Implementation: plugins/openshift/skills/openshift-node-kernel/node-kernel-ip.sh
  • Helper functions: plugins/openshift/skills/openshift-node-kernel/kernel-helper.sh
  • Documentation: plugins/openshift/skills/openshift-node-kernel/SKILL.md

Parameters

Required Parameters

  • node: Name of the Kubernetes node where network information should be extracted.

    • Must be a valid node name in the cluster
    • Example: ovn-control-plane, ovn-worker, worker-0
  • image: Container image to use for creating a debug connection to the node.

    • Must contain ip utility (iproute2 package)
    • Common images:
      • registry.redhat.io/rhel9/support-tools
      • nicolaka/netshoot
  • --command <cmd>: IP operation to perform. Valid commands:

    • address show: Display protocol (IP/IPv6) addresses on devices
    • link show: Display network devices
    • neighbour show or neighbor show: Display ARP or NDISC cache entries
    • netns show: Display network namespaces
    • route show: Display routing table entries
    • rule show: Display rules in routing policy database
    • vrf show: Display virtual routing and forwarding devices

Optional Parameters

  • --options <opts>: Additional options for the ip command

    • -d, -details: Output more detailed information
    • -4: Shortcut for -family inet (IPv4 only)
    • -6: Shortcut for -family inet6 (IPv6 only)
    • -r, -resolve: Use DNS to resolve names instead of showing IPs
    • -n, -netns <NETNS>: Execute in specified network namespace
    • -a, -all: Execute command over all objects (if supported)
    • -s, -stats: Output statistics
  • --filter <params>: Sub-commands and filters for more specific queries

    • Varies by command type
    • Examples: dev eth0, table all, type veth

Return Value

The command returns ip utility output in standard format. Output varies based on the command:

For address show:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0

For route show:

default via 172.17.0.1 dev eth0
10.244.0.0/16 via 10.244.1.1 dev ovn0 src 10.244.0.1
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2

Examples

Example 1: Show all network interfaces

/openshift:node-kernel-ip ovn-worker registry.redhat.io/rhel9/support-tools --command "link show"

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
3: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN
4: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue state UNKNOWN

Example 2: Show IPv4 routing table

/openshift:node-kernel-ip ovn-control-plane nicolaka/netshoot --command "route show" --options "-4"

Output:

default via 10.0.0.254 dev br-ex proto dhcp src 10.0.0.10 metric 48
10.0.0.0/24 dev br-ex proto kernel scope link src 10.0.0.10 metric 48
10.244.0.0/24 dev ovn-k8s-mp0 proto kernel scope link src 10.244.0.1

Example 3: Show all routing tables (including policy routing)

/openshift:node-kernel-ip ovn-worker registry.redhat.io/rhel9/support-tools --command "route show" --filter "table all"

Output:

default via 10.0.0.254 dev eth0 table default
10.244.0.0/16 dev ovn0 table main
local 127.0.0.0/8 dev lo table local
local 10.0.0.5 dev eth0 table local
broadcast 10.0.0.255 dev eth0 table local

Example 4: Show detailed interface information

/openshift:node-kernel-ip ovn-control-plane registry.redhat.io/rhel9/support-tools --command "link show" --options "-d"

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff promiscuity 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0

Example 5: Show specific interface

/openshift:node-kernel-ip ovn-worker nicolaka/netshoot --command "address show" --filter "dev ovn-k8s-mp0"

Output:

3: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default
    link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 brd 10.244.0.255 scope global ovn-k8s-mp0

Example 6: Show ARP/neighbor cache

/openshift:node-kernel-ip ovn-control-plane registry.redhat.io/rhel9/support-tools --command "neighbour show"

Output:

10.0.0.254 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE
10.244.1.5 dev ovn0 lladdr 0a:58:0a:f4:01:05 STALE
fe80::1 dev eth0 lladdr 52:54:00:12:34:56 router REACHABLE

Example 7: Show network namespaces

/openshift:node-kernel-ip ovn-worker registry.redhat.io/rhel9/support-tools --command "netns show"

Output:

ovnkube-node-12345 (id: 0)

Example 8: Show routing policy rules

/openshift:node-kernel-ip ovn-control-plane nicolaka/netshoot --command "rule show"

Output:

0:      from all lookup local
100:    from 10.244.0.0/16 lookup main
32766:  from all lookup main
32767:  from all lookup default

Example 9: Show interface statistics

/openshift:node-kernel-ip ovn-worker registry.redhat.io/rhel9/support-tools --command "link show" --options "-s"

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    RX: bytes  packets  errors  dropped overrun mcast
    1234567    12345    0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    1234567    12345    0       0       0       0

Example 10: Show IPv6 addresses

/openshift:node-kernel-ip ovn-control-plane registry.redhat.io/rhel9/support-tools --command "address show" --options "-6"

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN
    inet6 ::1/128 scope host
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
    inet6 fe80::42:acff:fe11:2/64 scope link

Understanding Output

Link States

  • UP: Interface is administratively up
  • LOWER_UP: Interface has carrier/link detected
  • UNKNOWN: State cannot be determined
  • DOWN: Interface is administratively down

Link Types

  • loopback: Loopback interface (lo)
  • ether: Ethernet interface
  • veth: Virtual Ethernet pair
  • geneve: GENEVE tunnel interface
  • bridge: Bridge interface

Route Types

  • unicast: Standard route to destination
  • local: Local interface address
  • broadcast: Broadcast address
  • multicast: Multicast route

Route Scopes

  • global: Routes valid everywhere
  • link: Valid only on directly connected network
  • host: Valid only inside this host

Neighbor States

  • REACHABLE: Neighbor is reachable
  • STALE: Neighbor entry is valid but suspicious
  • DELAY: Neighbor entry validation pending
  • INCOMPLETE: ARP resolution in progress
  • FAILED: ARP resolution failed

Troubleshooting

Image doesn't have ip utility

The command will fail if ip utility is not available.

Solution: Use an image with iproute2 package:

/openshift:node-kernel-ip ovn-worker registry.redhat.io/rhel9/support-tools --command "link show"

No OVN interfaces visible

If you don't see ovn-k8s-mp0 or genev_sys_* interfaces:

  1. OVN-Kubernetes might not be running on this node
  2. You might be in wrong network namespace

Solution: Check network namespaces:

/openshift:node-kernel-ip <node> <image> --command "netns show"

Empty routing table

Could indicate:

  1. Wrong routing table queried
  2. No routes configured (unlikely)

Solution: Show all tables:

/openshift:node-kernel-ip <node> <image> --command "route show" --filter "table all"

Permission errors

Debug pod might lack necessary privileges.

Solution: Verify your OpenShift user has permissions for privileged debug pods.

OVN-Kubernetes Network Interfaces

In OVN-Kubernetes environments, you'll commonly see:

Node Interfaces

  • ovn-k8s-mp0: OVN management port (node to pod communication)
  • genev_sys_6081: GENEVE tunnel interface (overlay network)
  • br-ex: External bridge (node external connectivity)
  • breth0: Bridge for physical interface

Pod Interfaces (in pod network namespace)

  • eth0: Pod's primary interface
  • vethXXXXXX: Virtual ethernet pair connecting pod to OVS bridge

Common Interface Patterns

ovn-k8s-mp0: Management port, usually 10.244.X.1
genev_sys_6081: GENEVE tunnel endpoint
br-ex: External bridge for node connectivity

Routing in OVN-Kubernetes

Typical routing setup on OVN-Kubernetes node:

  1. Default route: Via br-ex to external gateway
  2. Pod CIDR routes: Via ovn-k8s-mp0 to local pods
  3. Service CIDR routes: Via OVN overlay to service backends
  4. Local table: Local interface addresses

Example routing table:

default via 10.0.0.254 dev br-ex
10.244.0.0/24 dev ovn-k8s-mp0 proto kernel scope link
10.244.1.0/24 via 10.244.1.1 dev ovn0

Network Namespaces

OVN-Kubernetes uses network namespaces for isolation:

  • Host namespace: Default namespace with all node interfaces
  • Pod namespaces: Each pod gets its own network namespace
  • OVN namespace: Some OVN components may use dedicated namespace

Use --options "-n <namespace>" to query specific namespace.

See Also

  • /openshift:node-kernel-conntrack - Inspect connection tracking entries
  • /openshift:node-kernel-iptables - Inspect packet filter rules
  • /openshift:node-kernel-nft - Inspect nftables configuration
  • ip command manual
  • iproute2 documentation

Name

openshift:node-kernel-iptables

Synopsis

/openshift:node-kernel-iptables <node> <image> --command <cmd> [--table <table>] [--filter <params>]

Description

The openshift:node-kernel-iptables command allows interaction with the kernel to list packet filter rules. Iptables and ip6tables are used to inspect the tables of IPv4 and IPv6 packet filter rules in the Linux kernel.

In OVN-Kubernetes environments, iptables rules are crucial for:

  • Network address translation (NAT) for services
  • Packet filtering and security policies
  • Connection tracking integration
  • SNAT/DNAT for egress and ingress traffic
  • Debugging packet flow issues

The command uses oc debug to create an ephemeral container with host network access, then executes iptables operations. The image must contain iptables or ip6tables CLI utilities.

Implementation

This command invokes the openshift-node-kernel skill which provides kernel-level packet filter inspection capabilities:

  1. Parameter Validation: Validates required node, image, and command parameters
  2. Utility Check: Verifies the debug image contains iptables or ip6tables utilities
  3. Command Execution: Uses oc debug to create ephemeral container and execute iptables commands
  4. Output Filtering: Removes common oc debug warnings to provide clean output

Skill Reference:

  • Implementation: plugins/openshift/skills/openshift-node-kernel/node-kernel-iptables.sh
  • Helper functions: plugins/openshift/skills/openshift-node-kernel/kernel-helper.sh
  • Documentation: plugins/openshift/skills/openshift-node-kernel/SKILL.md

Parameters

Required Parameters

  • node: Name of the Kubernetes node where iptables rules should be extracted.

    • Must be a valid node name in the cluster
    • Example: ovn-control-plane, ovn-worker, worker-0
  • image: Container image to use for creating a debug connection to the node.

    • Must contain iptables and/or ip6tables utilities
    • Common images:
      • registry.redhat.io/rhel9/support-tools
      • nicolaka/netshoot
  • --command <cmd>: Iptables operation to perform

    • -L, --list [chain]: List all rules in selected chain (or all chains if not specified)
    • -S, --list-rules [chain]: Print all rules in iptables-save format

Optional Parameters

  • --table <table>: Iptables table to query (default: filter)

    • filter: Default table for packet filtering
    • nat: Network address translation table
    • mangle: Packet alteration table
    • raw: Connection tracking exemptions table
    • security: Mandatory Access Control (MAC) networking rules
  • --filter <params>: Additional filter parameters

    • -s, --source <address[/mask]>: Source specification
    • -d, --destination <address[/mask]>: Destination specification
    • -v, --verbose: Verbose output with packet and byte counters
    • -n, --numeric: Numeric output (no DNS resolution)
    • -p, --protocol <protocol>: Protocol filter (tcp, udp, icmp, etc.)
    • -4, --ipv4: IPv4 rules only
    • -6, --ipv6: IPv6 rules only

Return Value

The command returns iptables rules in standard iptables format. Output varies based on the command:

For -L (list) command:

Chain POSTROUTING (policy ACCEPT 675K packets, 41M bytes)
 pkts bytes target     prot opt in     out     source               destination
 675K   41M OVN-KUBE-EGRESS-IP-MULTI-NIC  all  --  *      *       0.0.0.0/0            0.0.0.0/0

For -S (list-rules) command:

-P POSTROUTING ACCEPT
-A POSTROUTING -j OVN-KUBE-EGRESS-IP-MULTI-NIC

Examples

Example 1: List NAT table rules with verbose output

/openshift:node-kernel-iptables ovn-control-plane registry.redhat.io/rhel9/support-tools --command -L --table nat --filter "-nv4"

Output:

Chain PREROUTING (policy ACCEPT 1234 packets, 89012 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1234 89012 OVN-KUBE-NODEPORT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 567 packets, 45678 bytes)
 pkts bytes target     prot opt in     out     source               destination
  567 45678 OVN-KUBE-SNAT-MGMTPORT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Example 2: Show filter table in iptables-save format

/openshift:node-kernel-iptables ovn-worker nicolaka/netshoot --command -S --table filter

Output:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N OVN-KUBE-FIREWALL
-A FORWARD -j OVN-KUBE-FIREWALL
-A OVN-KUBE-FIREWALL -m comment --comment "default deny policy" -j DROP

Example 3: List specific chain

/openshift:node-kernel-iptables ovn-control-plane registry.redhat.io/rhel9/support-tools --command "-L FORWARD" --table filter --filter "-n"

Output:

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
OVN-KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0

Example 4: Filter by protocol and destination

/openshift:node-kernel-iptables ovn-worker nicolaka/netshoot --command -L --table nat --filter "-p tcp -d 10.96.0.1"

Output:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             10.96.0.1            tcp dpt:443 to:10.0.0.5:6443

Example 5: List all tables for comprehensive view

/openshift:node-kernel-iptables ovn-control-plane registry.redhat.io/rhel9/support-tools --command -L --table nat
/openshift:node-kernel-iptables ovn-control-plane registry.redhat.io/rhel9/support-tools --command -L --table filter
/openshift:node-kernel-iptables ovn-control-plane registry.redhat.io/rhel9/support-tools --command -L --table mangle

Example 6: IPv6 rules

/openshift:node-kernel-iptables ovn-worker registry.redhat.io/rhel9/support-tools --command -L --table filter --filter "-6 -nv"

Understanding Output

Chain Types

  • PREROUTING: Rules applied before routing decision
  • INPUT: Rules for packets destined to local processes
  • FORWARD: Rules for packets being routed through the system
  • OUTPUT: Rules for locally generated packets
  • POSTROUTING: Rules applied after routing decision

Target Actions

  • ACCEPT: Allow packet to pass through
  • DROP: Silently discard packet
  • REJECT: Discard packet and send error response
  • DNAT: Destination NAT (change destination address)
  • SNAT: Source NAT (change source address)
  • MASQUERADE: Dynamic source NAT
  • Custom chains: Jump to user-defined chain (e.g., OVN-KUBE-*)

Packet Counters

When using -v (verbose):

  • pkts: Number of packets matching this rule
  • bytes: Total bytes of packets matching this rule

Troubleshooting

Image doesn't have iptables utility

The command will fail if the image lacks iptables/ip6tables.

Solution: Use an image with iptables utilities:

/openshift:node-kernel-iptables ovn-worker registry.redhat.io/rhel9/support-tools --command -L

Legacy vs nftables backend

Modern systems may use nftables backend for iptables.

Solution: If you need nftables information, use /openshift:node-kernel-nft command instead.

Empty output or no rules

This could indicate:

  1. The specified table/chain has no rules
  2. Wrong table name
  3. Using IPv6 filters on IPv4 rules (or vice versa)

Solution: Try listing all chains without filters:

/openshift:node-kernel-iptables <node> <image> --command -L --table nat

Permission errors

Debug pod might lack necessary privileges.

Solution: Verify your OpenShift user has permissions for creating privileged debug pods.

OVN-Kubernetes Specific Chains

In OVN-Kubernetes environments, you'll commonly see these custom chains:

  • OVN-KUBE-NODEPORT: Handles NodePort service traffic
  • OVN-KUBE-EXTERNALIP: Manages external IP services
  • OVN-KUBE-SNAT-MGMTPORT: SNAT for management port traffic
  • OVN-KUBE-EGRESS-IP: Egress IP functionality
  • OVN-KUBE-FIREWALL: Network policy enforcement

See Also

  • /openshift:node-kernel-conntrack - Inspect connection tracking entries
  • /openshift:node-kernel-nft - Inspect nftables configuration
  • /openshift:node-kernel-ip - Inspect network interfaces and routing
  • iptables man page

Name

openshift:node-kernel-nft

Synopsis

/openshift:node-kernel-nft <node> <image> --command <cmd> [--family <family>]

Description

The openshift:node-kernel-nft command allows interaction with the kernel to list packet filtering and classification rules managed by nftables. Nftables is the modern successor to iptables, providing a more unified and efficient packet filtering framework in the Linux kernel.

In OVN-Kubernetes environments, nftables may be used for:

  • Modern packet filtering replacing legacy iptables
  • Service load balancing rules
  • Network policy enforcement
  • NAT rules for ingress and egress traffic
  • More efficient rule matching than iptables

The command uses oc debug to create an ephemeral container with host network access, then executes nft operations. The image must contain the nft CLI utility.

Implementation

This command invokes the openshift-node-kernel skill which provides kernel-level nftables inspection capabilities:

  1. Parameter Validation: Validates required node, image, and command parameters
  2. Utility Check: Verifies the debug image contains the nft utility
  3. Command Execution: Uses oc debug to create ephemeral container and execute nft commands
  4. Output Filtering: Removes common oc debug warnings to provide clean output

Skill Reference:

  • Implementation: plugins/openshift/skills/openshift-node-kernel/node-kernel-nft.sh
  • Helper functions: plugins/openshift/skills/openshift-node-kernel/kernel-helper.sh
  • Documentation: plugins/openshift/skills/openshift-node-kernel/SKILL.md

Parameters

Required Parameters

  • node: Name of the Kubernetes node where nftables configuration should be extracted.

    • Must be a valid node name in the cluster
    • Example: ovn-control-plane, ovn-worker, worker-0
  • image: Container image to use for creating a debug connection to the node.

    • Must contain nft utility
    • Common images:
      • registry.redhat.io/rhel9/support-tools
      • nicolaka/netshoot
  • --command <cmd>: Nftables operation to perform. Valid commands:

    • list ruleset: Print the entire ruleset in human-readable format
    • list tables: List all tables
    • list chains: List all chains (optionally filtered by table/family)
    • list sets: Display elements in sets
    • list maps: Display elements in maps
    • list flowtables: List all flowtables

Optional Parameters

  • --family <family>: Address family to filter results. Valid families:
    • ip: IPv4 address family
    • ip6: IPv6 address family
    • inet: Internet (IPv4/IPv6) address family
    • arp: ARP address family (IPv4 ARP packets)
    • bridge: Bridge address family (packets traversing bridge device)
    • netdev: Netdev address family (ingress/egress packet handling)

Return Value

The command returns nftables configuration in nft native format. Output format depends on the command:

For list tables:

table inet ovn-kubernetes
table ip nat

For list ruleset:

table inet ovn-kubernetes {
  chain forward {
    type filter hook forward priority 0; policy accept;
    ct state related,established accept
    ct state invalid drop
  }
}

For list chains:

table inet ovn-kubernetes {
  chain forward {
    type filter hook forward priority 0; policy accept;
  }
  chain input {
    type filter hook input priority 0; policy accept;
  }
}

Examples

Example 1: List all nftables tables

/openshift:node-kernel-nft ovn-control-plane registry.redhat.io/rhel9/support-tools --command "list tables"

Output:

table inet ovn-kubernetes
table ip nat
table ip filter

Example 2: List tables for specific address family

/openshift:node-kernel-nft ovn-worker nicolaka/netshoot --command "list tables" --family inet

Output:

table inet ovn-kubernetes

Example 3: Show complete ruleset

/openshift:node-kernel-nft ovn-control-plane registry.redhat.io/rhel9/support-tools --command "list ruleset"

Output:

table inet ovn-kubernetes {
  chain PREROUTING {
    type nat hook prerouting priority -100; policy accept;
    ip daddr 10.96.0.0/12 dnat to meta mark map @svc_map
  }

  chain FORWARD {
    type filter hook forward priority 0; policy accept;
    ct state related,established accept
    ct state invalid drop
    ip saddr 10.244.0.0/16 accept
  }

  set svc_ips {
    type ipv4_addr
    elements = { 10.96.0.1, 10.96.0.10, 10.96.0.100 }
  }

  map svc_map {
    type mark : ipv4_addr
    elements = { 0x100 : 10.0.0.5, 0x101 : 10.0.0.6 }
  }
}

Example 4: List only chains

/openshift:node-kernel-nft ovn-worker registry.redhat.io/rhel9/support-tools --command "list chains" --family inet

Output:

table inet ovn-kubernetes {
  chain PREROUTING {
    type nat hook prerouting priority -100; policy accept;
  }
  chain FORWARD {
    type filter hook forward priority 0; policy accept;
  }
  chain POSTROUTING {
    type nat hook postrouting priority 100; policy accept;
  }
}

Example 5: List sets (for service IPs, pod IPs, etc.)

/openshift:node-kernel-nft ovn-control-plane nicolaka/netshoot --command "list sets"

Output:

table inet ovn-kubernetes {
  set svc_ips {
    type ipv4_addr
    elements = { 10.96.0.1, 10.96.0.10, 10.96.0.100 }
  }
  set pod_ips {
    type ipv4_addr
    elements = { 10.244.0.5, 10.244.0.10, 10.244.1.5 }
  }
}

Example 6: List flowtables (for connection offloading)

/openshift:node-kernel-nft ovn-worker registry.redhat.io/rhel9/support-tools --command "list flowtables"

Output:

table inet ovn-kubernetes {
  flowtable fastpath {
    hook ingress priority 0;
    devices = { eth0, ovn0 };
  }
}

Understanding Output

Address Families

  • ip: IPv4 only rules
  • ip6: IPv6 only rules
  • inet: Combined IPv4/IPv6 rules (most common in modern setups)
  • arp: ARP protocol handling
  • bridge: Bridge device packet processing
  • netdev: Device-level packet processing

Chain Types and Hooks

  • filter hook input: Process packets destined for local system
  • filter hook forward: Process packets being routed through system
  • filter hook output: Process locally generated packets
  • nat hook prerouting: NAT before routing decision
  • nat hook postrouting: NAT after routing decision

Priority

Lower priority number = earlier execution. Common priorities:

  • -100: Early NAT operations
  • 0: Standard filtering
  • 100: Late NAT operations

Verdicts

  • accept: Allow packet
  • drop: Silently discard packet
  • reject: Discard and send error
  • jump <chain>: Jump to another chain
  • goto <chain>: Go to chain without return
  • return: Return to calling chain

Data Structures

  • sets: Collections of unique values (IPs, ports, etc.)
  • maps: Key-value mappings (marks to IPs, etc.)
  • flowtables: Hardware offload tables for fast path

Troubleshooting

Image doesn't have nft utility

The command will fail if nft is not available.

Solution: Use an image with nft utility:

/openshift:node-kernel-nft ovn-worker registry.redhat.io/rhel9/support-tools --command "list tables"

No tables found

This indicates nftables is not in use on this node.

Solution: The system might be using legacy iptables. Try:

/openshift:node-kernel-iptables <node> <image> --command -L

Invalid command error

Only specific list commands are allowed for safety.

Solution: Use one of the valid commands:

  • list ruleset
  • list tables
  • list chains
  • list sets
  • list maps
  • list flowtables

Permission denied

Debug pod lacks necessary privileges.

Solution: Verify your OpenShift user has permissions for privileged debug pods.

Empty output

Could indicate:

  1. No nftables rules configured
  2. Wrong address family filter
  3. Nftables not in use (system using iptables instead)

Solution: Try without family filter first:

/openshift:node-kernel-nft <node> <image> --command "list tables"

OVN-Kubernetes Nftables Usage

Modern OVN-Kubernetes deployments may use nftables for:

  1. Service load balancing: Maps and sets for service IP to backend mapping
  2. NAT operations: SNAT/DNAT rules in PREROUTING/POSTROUTING
  3. Network policies: Filtering rules in FORWARD chain
  4. Connection tracking: Integration with conntrack for stateful filtering
  5. Flow offloading: Flowtables for hardware-accelerated packet processing

Common table names in OVN-Kubernetes:

  • inet ovn-kubernetes: Main OVN-K table
  • ip nat: IPv4 NAT rules
  • inet filter: Packet filtering rules

Performance Considerations

Nftables offers better performance than iptables:

  • Single rule evaluation (vs. multiple tables in iptables)
  • Better scalability with large rule sets
  • More efficient matching with sets and maps
  • Hardware offload support via flowtables

See Also

  • /openshift:node-kernel-conntrack - Inspect connection tracking entries
  • /openshift:node-kernel-iptables - Inspect legacy iptables rules
  • /openshift:node-kernel-ip - Inspect network interfaces and routing
  • nftables wiki
  • nftables manual

Name

openshift:rebase

Synopsis

/openshift:rebase [tag]

Description

The /openshift:rebase command rebases git repository in the current working directory to a new upstream release specified by [tag]. If no [tag] is specified, the command tries to find the latest stable upstream release.

The repository must follow rules described in https://github.com/openshift/kubernetes/blob/master/REBASE.openshift.md, namely all OpenShift-specific commits must have prefix UPSTREAM:.

Implementation

Pre-requisites

Three local remote repositories should be tracked from a local machine: origin tracking the user's fork of this repository, openshift tracking this repository and upstream tracking the upstream repository.

To verify the correct setup, use

git remote -v

Fail, if there is no upstream, origin or openshift remote.

Rebase to the new upstream version

  1. Fetch all the remote repositories including tags

    git fetch --all
    
  2. Find the main branch of the repository. It's either master or main. In the following steps, we will use master, but replace it with the main branch.

  3. If user did not specify an upstream tag to rebase to as <tag>, find the greatest upstream tag that is not alpha, beta or rc.

  4. Create a new branch based on the newest tag $1 of the upstream repository. Name it after the tag.

    git checkout -b rebase-<tag> <tag>
    
  5. Merge openshift/master branch into the rebase-$1 branch with merge strategy ours:

    git merge -s ours openshift/master
    
  6. Find the last rebase that has been done to openshift/master. We will use the upstream tag used for this rebase as $previous_tag.

  7. Find the merge base of the openshift/master and $previous_tag by running git merge-base openshift/master $previous_tag. We will use this merge base as $mergebase.

  8. Prepare commits.tsv tab-separated values file containing the set of carry commits in the openshift/master branch that need to be considered for picking:

    Create the commits file:

    echo -e 'Sha\tMessage\tDecision' > commits.tsv
    git log ${mergebase}..openshift/master --ancestry-path --reverse --no-merges --pretty="tformat:%h%x09%s%x09" | grep "UPSTREAM:" > commits.tsv
    
  9. Go through the commits in the commits.tsv file and for each of them decide whether to pick, drop or squash it. Commits carried on rebase branches have commit messages prefixed as follows:

    • UPSTREAM: <carry>: Add OpenShift files: ALWAYS carry this commit and mark it as "cherry-pick". This is a persistent carry that contains all OpenShift-specific files and should be present in every rebase.

    • Other UPSTREAM: <carry> commit: A persistent carry that needs to be considered for squashing. Examine what files it modifies using git show --stat <commit-sha>. If it modifies ONLY OpenShift-specific files (Dockerfile, OWNERS, .ci-operator.yaml, .snyk, etc.), mark it as "squash", otherwise mark is as "cherry-pick".

    • UPSTREAM: <drop>: A carry that should probably not be picked for the subsequent rebase branch. In general, these commits are used to maintain the codebase in ways that are branch-specific, like the update of generated files or dependencies. Mark such commit as "drop".

    • UPSTREAM: (upstream PR number): The number identifies a PR in upstream repository (e.g. https://github.com///pull/). A commit with this message should only be picked into the subsequent rebase branch if the commits of the referenced PR are not included in the upstream branch. To check if a given commit is included in the upstream branch, open the referenced upstream PR and check any of its commits for the release tag.

    For each commit:

    • Print the decision you made and why.
    • Update commits.tsv with the decision ("cherry-pick", "drop", or "squash").
  10. Cherry-pick all commits marked as "cherry-pick" in commits.tsv. Then squash ALL commits marked as "squash" into a single commit named "UPSTREAM: : Add OpenShift files" to keep the number of commits as low as possible.

    Use git reset --soft to squash multiple commits together, then create a single commit with all the changes. The commit message should list what was included (e.g., "Additional changes: remove .github files, add .snyk file, update Dockerfile and .ci-operator.yaml").

  11. If the upstream repository DOES NOT include vendor/ directory and the OpenShift fork DOES, then update the vendor directory with go mod tidy and go mod vendor. Amend these vendor updates into the "UPSTREAM: : Add OpenShift files" commit using git commit --amend --no-edit.

  12. As a verification step, see the last rebase and ensure that all changes made in the last rebase are present in the current one. Either as a cherry pick or were part of the rebase. Verify all changes were applied during the rebase. Either as a cherry-picked patch or they were included in the new upstream tag. List all these commits, together with checks you made and their result.

  13. Verify the changes by running make and make test (or a similar command like like go build ./... and go test ./...). Stop here if there are compilation errors or test failures that indicate real code issues. If you make any new commits to fix compilation or tests, let user review these changes and then squash them into the commit "UPSTREAM: : Add OpenShift files" too.

  14. Find links to upstream changelogs between $previous_tag and $1. Make sure they are links to changelogs, not tags. Print list of the links.

  15. Create a github pull request against the OpenShift github repository (openshift/). IMPORTANT: Use --repo openshift/<repo-name> to ensure the PR is created against the correct OpenShift repository, not the upstream. The PR title should be "Rebase to $1 for OCP ". Follow the repository .github/PULL_REQUEST_TEMPLATE.md, if it exists. Description of the PR must look like:

    ## Upstream changelogs
    <List links to all upstream changelogs, as composed in the previous step.>
    
    ## Summary of changes
    <List all new major features and breaking changes that happened between $previous_tag and $1.
    Do not list upstream commits or PRs, make a human readable summary of them.
    Do not include small bug fixes, small updates, or dependency bumps.>
    
    ## Carried commits
    <List of commits from commits.tsv. For each commit print a decision you made - either "drop", "cherry-pick", or "squash".>
    
    Diff to upstream: <link to a diff between the upstream project/upstream repository/tag $1 and this PR (i.e. my personal fork with branch `rebase-$1`>
    
    Previous rebase: <link to the previous rebase PR on github>
    

    When opening the PR, ALWAYS use gh pr create --web --repo openshift/<repo-name> to allow user edit the PR before creation.

Name

openshift:review-test-cases

Synopsis

/openshift:review-test-cases [file-path-or-test-code-or-commands]

Description

The review-test-cases command provides comprehensive review of OpenShift test cases to ensure quality, completeness, and adherence to best practices. It accepts three types of input:

  1. File path: Path to test file (e.g., /path/to/test.sh or /path/to/test.go)
  2. oc commands: Direct oc CLI commands to review (e.g., paste a set of oc commands)
  3. Test code: Pasted Ginkgo test code to analyze

The command analyzes test code in both oc CLI shell scripts and Ginkgo Go tests, helping QE engineers identify gaps in test coverage, improve test reliability, and ensure tests follow OpenShift testing standards.

Implementation

The command analyzes test cases and provides structured feedback:

  1. Parse Test Input: Determine if input is a file path, oc commands, or test code
    • If file path: Read and analyze the test file
    • If oc commands: Parse command sequence
    • If test code: Analyze pasted Ginkgo/test code
  2. Identify Test Format: Detect if it's oc CLI shell script or Ginkgo Go code
  3. Analyze Test Structure: Review organization, naming, and patterns
  4. Check Coverage: Verify positive, negative, and edge case coverage
  5. Review Assertions: Ensure proper validation and error checking
  6. Evaluate Cleanup: Verify resource cleanup and namespace management
  7. Assess Best Practices: MUST follow the standards defined in "Testing Guidelines and References" section below
  8. Generate Recommendations: Provide actionable improvement suggestions based on the guidelines

CRITICAL: All reviews MUST be evaluated against the specific standards, references, and best practices listed in the "Testing Guidelines and References" section below. Do not use generic testing advice - follow the OpenShift-specific guidelines provided.

Testing Guidelines and References

The review follows established testing best practices from:

For Ginkgo/E2E Tests:

For oc CLI Tests:

Key Testing Standards:

  • Use descriptive, stable test names (no timestamps, random IDs)
  • Proper resource cleanup (AfterEach, defer, trap)
  • Meaningful assertions with clear failure messages
  • Test isolation (each test creates own resources)
  • Appropriate timeouts and waits
  • No BeforeAll/AfterAll in Ginkgo tests
  • Use framework helpers (e2epod, e2enode) when available

Arguments

  • $1 (file-path-or-test-code-or-commands): One of:
    • File path: Path to test file (shell script or Go test file)
    • oc commands: Set of oc CLI commands to review
    • Test code: Pasted test code (Ginkgo or shell script)

Name

openshift:visualize-ovn-topology

Synopsis

/openshift:visualize-ovn-topology

Description

The openshift:visualize-ovn-topology command generates a comprehensive Mermaid diagram of the OVN-Kubernetes network topology for a running cluster. The diagram shows:

  • Logical switches and routers
  • Switch and router ports with MAC/IP addresses
  • Pod connectivity
  • External network connections
  • Per-node component placement (interconnect mode) or centralized components (default mode)

The command automatically detects the cluster architecture (interconnect vs default mode) and creates an accurate topology diagram based on real data from the OVN databases.

Implementation

This command invokes the generating-ovn-topology skill which implements a data-driven architecture discovery approach:

  1. Cluster Detection: Automatically finds and connects to an OVN-Kubernetes cluster
  2. Permission Check: Verifies Kubernetes access level and warns if write permissions detected
    • If you have cluster admin permissions, you'll be asked to confirm before proceeding
    • The command only performs read-only operations regardless of your permission level
    • This check ensures informed consent when using admin credentials
  3. Architecture Discovery: Analyzes UUID patterns across node databases to determine component placement (per-node vs cluster-wide)
  4. Data Collection: Queries OVN northbound databases for topology information
  5. Diagram Generation: Creates a Mermaid graph with proper component placement
  6. Output: Saves diagram to ovn-topology-diagram.md (or timestamped/custom path if file exists)

Key Features:

  • Data-driven: Never generates synthetic data - always queries real cluster
  • Architecture-aware: Detects and handles both interconnect and default deployment modes
  • Complete topology: Shows all logical switches, routers, and ports
  • Visual clarity: Uses color-coded components and node subgraphs for organization

Skill Reference:

  • Implementation details: plugins/openshift/skills/generating-ovn-topology/SKILL.md
  • Helper scripts: plugins/openshift/skills/generating-ovn-topology/scripts/

Return Value

  • Format: Mermaid diagram saved to file
  • Location: ./ovn-topology-diagram.md (current directory) or custom path if specified
  • Output: Summary statistics and preview of the generated diagram

Examples

  1. Basic usage (generates topology for detected cluster):

    /openshift:visualize-ovn-topology
    

    Output:

    ✓ Successfully generated OVN-Kubernetes topology diagram
    
    📄 Diagram saved to: ovn-topology-diagram.md
    
    Summary:
    - 3 nodes (ovn-control-plane, ovn-worker, ovn-worker2)
    - 10 logical switches, 4 logical routers
    - 27 logical switch ports, 13 logical router ports
    - 9 running pods
    - Mode: Interconnect (distributed control plane)
    
    💡 Open the file in your IDE to view the full rendered Mermaid diagram!
    
  2. With existing file (prompts for action):

    /openshift:visualize-ovn-topology
    

    You'll be asked:

    File ovn-topology-diagram.md already exists. Would you like to:
    (1) Overwrite it
    (2) Save to a different location
    (3) Append timestamp to filename
    (4) Cancel
    

Prerequisites

  • kubectl: Must be installed and configured with access to an OVN-Kubernetes cluster
  • Cluster: A running Kubernetes cluster with OVN-Kubernetes CNI deployed
  • Access: Permission to exec into pods in the OVN namespace (e.g., ovn-kubernetes or openshift-ovn-kubernetes)

Security & Safety

This command performs ONLY read-only operations:

  • ✅ kubectl get - Query pods and nodes
  • ✅ kubectl exec - Run read-only ovn-nbctl list commands
  • ✅ Local file writes - Save topology diagram

Operations NEVER performed:

  • ❌ No kubectl create/delete/patch/apply
  • ❌ No ovn-nbctl modifications
  • ❌ No cluster state changes

Permission Check: If you have cluster admin permissions, you'll receive a warning message before the command proceeds. This is for transparency - you'll be informed about your access level and asked to confirm. The command will still only perform read-only operations.

Notes

  • The command is cluster-agnostic and works with any OVN-Kubernetes deployment
  • For KIND clusters created via /openshift:create-cluster, the kubeconfig is automatically detected
  • The diagram uses bottom-to-top layout (graph BT) following network topology conventions
  • All component placement is determined by UUID analysis, not hardcoded assumptions

Sponsors

Details

Marketplace
openshift-eng/ai-helpers
Source
plugins/openshift
Category
Openshift
Keywords
openshift, development, helpers

More from openshift-eng/ai-helpers

All 56