A plugin to analyze and report on must-gather data

version
0.0.3
GitHub stars
123

Install

/plugin install must-gather@ai-helpers

Run it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.

Contents

Discovered from source

Skills 1

Must-Gather Analyzer Skill

Comprehensive analysis of OpenShift must-gather diagnostic data with helper scripts that parse YAML and display output in oc-like format.

Overview

This skill provides analysis for:

  • ClusterVersion: Current version, update status, and capabilities
  • Cluster Operators: Status, degradation, and availability
  • Pods: Health, restarts, crashes, and failures across namespaces
  • Nodes: Conditions, capacity, and readiness
  • Network: OVN/SDN diagnostics and connectivity
  • Events: Warning and error events across namespaces
  • etcd: Cluster health, member status, and quorum
  • Storage: PersistentVolumes and PersistentVolumeClaims status

Must-Gather Directory Structure

Important: Must-gather data is contained in a subdirectory with a long hash name:

must-gather/
└── registry-ci-openshift-org-origin-...-sha256-<hash>/
    ├── cluster-scoped-resources/
    │   ├── config.openshift.io/clusteroperators/
    │   └── core/nodes/
    ├── namespaces/
    │   └── <namespace>/
    │       └── pods/
    │           └── <pod-name>/
    │               └── <pod-name>.yaml
    └── network_logs/

The analysis scripts expect the path to the subdirectory (the one with the hash), not the root must-gather folder.

Instructions

1. Get Must-Gather Path

Ask the user for the must-gather directory path if not already provided.

  • If they provide the root directory, look for the subdirectory with the hash name
  • The correct path contains cluster-scoped-resources/ and namespaces/ directories

2. Choose Analysis Type

Based on user's request, run the appropriate helper script:

ClusterVersion Analysis
./scripts/analyze_clusterversion.py <must-gather-path>

Shows cluster version information similar to oc get clusterversion:

  • Current version and update status
  • Progressing state
  • Available updates
  • Version conditions
  • Enabled capabilities
  • Update history
Cluster Operators Analysis
./scripts/analyze_clusteroperators.py <must-gather-path>

Shows cluster operator status similar to oc get clusteroperators:

  • Available, Progressing, Degraded conditions
  • Version information
  • Time since condition change
  • Detailed messages for operators with issues
Pods Analysis
# All namespaces
./scripts/analyze_pods.py <must-gather-path>

# Specific namespace
./scripts/analyze_pods.py <must-gather-path> --namespace <namespace>

# Show only problematic pods
./scripts/analyze_pods.py <must-gather-path> --problems-only

Shows pod status similar to oc get pods -A:

  • Ready/Total containers
  • Status (Running, Pending, CrashLoopBackOff, etc.)
  • Restart counts
  • Age
  • Categorized issues (crashlooping, pending, failed)
Nodes Analysis
./scripts/analyze_nodes.py <must-gather-path>

# Show only nodes with issues
./scripts/analyze_nodes.py <must-gather-path> --problems-only

Shows node status similar to oc get nodes:

  • Ready status
  • Roles (master, worker)
  • Age
  • Kubernetes version
  • Node conditions (DiskPressure, MemoryPressure, etc.)
  • Capacity and allocatable resources
Network Analysis
./scripts/analyze_network.py <must-gather-path>

Shows network health:

  • Network type (OVN-Kubernetes, OpenShift SDN)
  • Network operator status
  • OVN pod health
  • PodNetworkConnectivityCheck results
  • Network-related issues
Events Analysis
# Recent events (last 100)
./scripts/analyze_events.py <must-gather-path>

# Warning events only
./scripts/analyze_events.py <must-gather-path> --type Warning

# Events in specific namespace
./scripts/analyze_events.py <must-gather-path> --namespace openshift-etcd

# Show last 50 events
./scripts/analyze_events.py <must-gather-path> --count 50

Shows cluster events:

  • Event type (Warning, Normal)
  • Last seen timestamp
  • Reason and message
  • Affected object
  • Event count
etcd Analysis
./scripts/analyze_etcd.py <must-gather-path>

Shows etcd cluster health:

  • Member health status
  • Member list with IDs and URLs
  • Endpoint status (leader, version, DB size)
  • Quorum status
  • Cluster summary
Storage Analysis
# All PVs and PVCs
./scripts/analyze_pvs.py <must-gather-path>

# PVCs in specific namespace
./scripts/analyze_pvs.py <must-gather-path> --namespace openshift-monitoring

Shows storage resources:

  • PersistentVolumes (capacity, status, claims)
  • PersistentVolumeClaims (binding, capacity)
  • Storage classes
  • Pending/unbound volumes
Monitoring Analysis
# All alerts.
./scripts/analyze_prometheus.py <must-gather-path>

# Alerts in specific namespace
./scripts/analyze_prometheus.py <must-gather-path> --namespace openshift-monitoring

Shows monitoring information:

  • Alerts (state, namespace, name, active since, labels)
  • Total of pending/firing alerts

3. Interpret and Report

After running the scripts:

  1. Review the summary statistics
  2. Focus on items flagged with issues
  3. Provide actionable insights and next steps
  4. Suggest log analysis for specific components if needed
  5. Cross-reference issues (e.g., degraded operator → failing pods → node issues)

Output Format

All scripts provide:

  • Summary Section: High-level statistics with emoji indicators
  • Table View: oc-like formatted output
  • Issues Section: Detailed breakdown of problems

Example summary format:

================================================================================
SUMMARY: 25/28 operators healthy
  ⚠️  3 operators with issues
  🔄 1 progressing
  ❌ 2 degraded
================================================================================

Helper Scripts Reference

scripts/analyze_clusterversion.py

Parses: cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml Output: ClusterVersion table with detailed version info, conditions, and capabilities

scripts/analyze_clusteroperators.py

Parses: cluster-scoped-resources/config.openshift.io/clusteroperators/ Output: ClusterOperator status table with conditions

scripts/analyze_pods.py

Parses: namespaces/*/pods/*/*.yaml (individual pod directories) Output: Pod status table with issues categorized

scripts/analyze_nodes.py

Parses: cluster-scoped-resources/core/nodes/ Output: Node status table with conditions and capacity

scripts/analyze_network.py

Parses: network_logs/, network operator, OVN resources Output: Network health summary and diagnostics

scripts/analyze_events.py

Parses: namespaces/*/core/events.yaml Output: Event table sorted by last occurrence

scripts/analyze_etcd.py

Parses: etcd_info/ (endpoint_health.json, member_list.json, endpoint_status.json) Output: etcd cluster health and member status

scripts/analyze_pvs.py

Parses: cluster-scoped-resources/core/persistentvolumes/, namespaces/*/core/persistentvolumeclaims.yaml Output: PV and PVC status tables

scripts/analyze_ovn_dbs.py

Parses OVN Northbound and Southbound database dumps and supports targeted OVSDB queries.

scripts/analyze_windows_logs.py

Parses Windows node component logs for kube-proxy, hybrid-overlay, kubelet, containerd, WICD, and CSI proxy.

Tips for Analysis

  1. Start with Cluster Operators: They often reveal system-wide issues
  2. Check Timing: Look at "SINCE" columns to understand when issues started
  3. Follow Dependencies: Degraded operator → check its namespace pods → check hosting nodes
  4. Look for Patterns: Multiple pods failing on same node suggests node issue
  5. Cross-reference: Use multiple scripts together for complete picture

Common Scenarios

"Why is my cluster degraded?"

  1. Run analyze_clusteroperators.py - identify degraded operators
  2. Run analyze_pods.py --namespace <operator-namespace> - check operator pods
  3. Run analyze_nodes.py - verify node health

"Pods keep crashing"

  1. Run analyze_pods.py --problems-only - find crashlooping pods
  2. Check which nodes they're on
  3. Run analyze_nodes.py - verify node conditions
  4. Suggest checking pod logs in must-gather data

"Network connectivity issues"

  1. Run analyze_network.py - check network health
  2. Run analyze_pods.py --namespace openshift-ovn-kubernetes
  3. Check PodNetworkConnectivityCheck results

Next Steps After Analysis

Based on findings, suggest:

  • Examining specific pod logs in namespaces/<ns>/pods/<pod>/<container>/logs/
  • Reviewing events in namespaces/<ns>/core/events.yaml
  • Checking audit logs in audit_logs/
  • Analyzing metrics data if available
  • Looking at host service logs in host_service_logs/

Commands 3

Name

must-gather:analyze

Synopsis

/must-gather:analyze [must-gather-path] [component]

Description

The analyze command performs comprehensive analysis of OpenShift must-gather diagnostic data. It runs specialized Python analysis scripts to extract and summarize cluster health information across multiple components.

The command can analyze:

  • Cluster version and update status
  • Cluster operator health (degraded, progressing, unavailable)
  • Node conditions and resource status
  • Pod failures, restarts, and crash loops
  • Network configuration and OVN health
  • OVN databases - logical topology, ACLs, pods
  • Kubernetes events (warnings and errors)
  • etcd cluster health and quorum status
  • Persistent volume and claim status
  • Prometheus alerts

You can request analysis of the entire cluster or focus on a specific component.

Prerequisites

Required Directory Structure:

Must-gather data typically has this structure:

must-gather/
└── registry-ci-openshift-org-origin-...-sha256-<hash>/
    ├── cluster-scoped-resources/
    ├── namespaces/
    └── ...

The actual must-gather directory is the subdirectory with the hash name, not the parent directory.

Required Scripts:

Analysis scripts are bundled with this plugin at:

<plugin-root>/skills/must-gather-analyzer/scripts/
├── analyze_clusterversion.py
├── analyze_clusteroperators.py
├── analyze_nodes.py
├── analyze_pods.py
├── analyze_network.py
├── analyze_ovn_dbs.py
├── analyze_events.py
├── analyze_etcd.py
└── analyze_pvs.py

Where <plugin-root> is the directory where this plugin is installed (typically ~/.cursor/commands/ai-helpers/plugins/must-gather/ or similar).

Error Handling

CRITICAL: Script-Only Analysis

  • NEVER attempt to analyze must-gather data directly using bash commands, grep, or manual file reading
  • ONLY use the provided Python scripts in plugins/must-gather/skills/must-gather-analyzer/scripts/
  • If scripts are missing or not found:
    1. Stop immediately
    2. Inform the user that the analysis scripts are not available
    3. Ask the user to ensure the scripts are installed at the correct path
    4. Do NOT attempt alternative approaches

Script Availability Check:

Before running any analysis:

  1. Locate the scripts directory by searching for a known script:

    SCRIPT_PATH=$(find ~ -name "analyze_clusteroperators.py" -path "*/must-gather/skills/must-gather-analyzer/scripts/*" 2>/dev/null | head -1)
    
    if [ -z "$SCRIPT_PATH" ]; then
        echo "ERROR: Must-gather analysis scripts not found."
        echo "Please ensure the must-gather plugin from ai-helpers is properly installed."
        exit 1
    fi
    
    # All scripts are in the same directory, so just get the directory
    SCRIPTS_DIR=$(dirname "$SCRIPT_PATH")
    
  2. If scripts cannot be found, STOP and report to the user:

    The must-gather analysis scripts could not be located. Please ensure the must-gather plugin from openshift-eng/ai-helpers is properly installed in your Claude Code plugins directory.
    

Implementation

The command performs the following steps:

  1. Validate Must-Gather Path:

    • If path not provided as argument, ask the user
    • Check if path contains cluster-scoped-resources/ and namespaces/ directories
    • If user provides root directory, automatically find the correct subdirectory
    • Verify the path exists and is readable
  2. Determine Analysis Scope:

    STEP 1: Check for SPECIFIC component keywords

    If the user mentions a specific component, run ONLY that script:

    • "pods", "pod status", "containers", "crashloop", "failing pods" → analyze_pods.py ONLY
    • "etcd", "etcd health", "quorum" → analyze_etcd.py ONLY
    • "network", "networking", "ovn", "connectivity" → analyze_network.py ONLY
    • "ovn databases", "ovn-dbs", "ovn db", "logical switches", "acls" → analyze_ovn_dbs.py ONLY
    • "nodes", "node status", "node conditions" → analyze_nodes.py ONLY
    • "operators", "cluster operators", "degraded" → analyze_clusteroperators.py ONLY
    • "version", "cluster version", "update", "upgrade" → analyze_clusterversion.py ONLY
    • "events", "warnings", "errors" → analyze_events.py ONLY
    • "storage", "pv", "pvc", "volumes", "persistent" → analyze_pvs.py ONLY
    • "alerts", "prometheus", "monitoring" → analyze_prometheus.py ONLY
    • "windows", "windows nodes", "windows logs", "hns", "containerd", "hybrid-overlay" → analyze_windows_logs.py ONLY

    STEP 2: No specific component mentioned

    If generic request like "analyze must-gather", "/must-gather:analyze", or "check the cluster", run ALL scripts in this order:

    1. ClusterVersion (analyze_clusterversion.py)
    2. Cluster Operators (analyze_clusteroperators.py)
    3. Nodes (analyze_nodes.py)
    4. Pods - problems only (analyze_pods.py --problems-only)
    5. Network (analyze_network.py)
    6. Events - warnings only (analyze_events.py --type Warning --count 50)
    7. etcd (analyze_etcd.py)
    8. Storage (analyze_pvs.py)
    9. Monitoring (analyze_prometheus.py)
    10. Windows Logs - ONLY if Windows nodes detected:
      • First check if host_service_logs/windows/ directory exists in the must-gather
      • If directory exists, run: analyze_windows_logs.py <must-gather-path>
      • If directory does not exist, skip silently (cluster has no Windows nodes)
  3. Locate Plugin Scripts:

    • Use the script availability check from the Error Handling section to find the plugin root
    • Store the scripts directory path in $SCRIPTS_DIR
  4. Execute Analysis Scripts:

    python3 "$SCRIPTS_DIR/<script>.py" <must-gather-path>
    

    Example:

    python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" ./must-gather.local.123/quay-io-...
    
  5. Synthesize Results: Generate findings and recommendations based on script output

Return Value

The command outputs structured analysis results to stdout:

For Component-Specific Analysis:

  • Script output for the requested component only
  • Focused findings and recommendations

For Full Analysis:

  • Organized sections for each component
  • Executive summary of overall cluster health
  • Prioritized list of critical issues
  • Actionable recommendations
  • Suggested log files to review

Output Structure

================================================================================
MUST-GATHER ANALYSIS SUMMARY
================================================================================

[Script outputs organized by component]

CLUSTER VERSION:
[output from analyze_clusterversion.py]

CLUSTER OPERATORS:
[output from analyze_clusteroperators.py]

NODES:
[output from analyze_nodes.py]

PROBLEMATIC PODS:
[output from analyze_pods.py --problems-only]

NETWORK STATUS:
[output from analyze_network.py]

WARNING EVENTS (Last 50):
[output from analyze_events.py --type Warning --count 50]

ETCD CLUSTER HEALTH:
[output from analyze_etcd.py]

STORAGE (PVs/PVCs):
[output from analyze_pvs.py]

MONITORING (Alerts):
[output from analyze_prometheus.py]

================================================================================
FINDINGS AND RECOMMENDATIONS
================================================================================

Critical Issues:
- [Critical problems requiring immediate attention]

Warnings:
- [Potential issues or degraded components]

Recommendations:
- [Specific next steps for investigation]

Logs to Review:
- [Specific log files to examine based on findings]

Examples

  1. Full cluster analysis:

    /must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/
    

    Runs all analysis scripts and provides comprehensive cluster diagnostics.

  2. Analyze pod issues only:

    /must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/ analyze the pod statuses
    

    Runs only analyze_pods.py to focus on pod-related issues.

  3. Check etcd health:

    /must-gather:analyze check etcd health
    

    Asks for must-gather path, then runs only analyze_etcd.py.

  4. Network troubleshooting:

    /must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/ show me network issues
    

    Runs only analyze_network.py for network-specific analysis.

Notes

  • Must-Gather Path: Always use the subdirectory containing cluster-scoped-resources/ and namespaces/, not the parent directory
  • Script Dependencies: Analysis scripts must be executable and have required Python dependencies installed
  • Error Handling: If scripts are not found or must-gather path is invalid, clear error messages are displayed
  • Cross-Referencing: The analysis attempts to correlate issues across components (e.g., degraded operator → failing pods)
  • Pattern Detection: Identifies patterns like multiple pod failures on the same node
  • Actionable Output: Focuses on insights and recommendations rather than raw data dumps
  • Priority: Issues are prioritized by severity (Critical > Warning > Info)

Arguments

  • $1 (must-gather-path): Optional. Path to the must-gather directory (the subdirectory with the hash name). If not provided, the user will be asked.
  • $2+ (component): Optional. If keywords for a specific component are detected, only that component's analysis script will run. Otherwise, all scripts run.

Name

must-gather:ovn-dbs

Synopsis

/must-gather:ovn-dbs [must-gather-path] [--node <node-name>] [--query <json>]

Description

The ovn-dbs command analyzes OVN Northbound and Southbound databases collected from clusters. It uses ovsdb-tool to query the binary database files (.db) collected per-node, providing detailed information about the logical network topology, pods, ACLs, and routers on each node.

The command automatically maps ovnkube pods to their corresponding nodes by reading pod specifications from the must-gather data.

Two modes of operation:

  1. Standard Analysis (default): Runs pre-built analysis showing switches, ports, ACLs, and routers
  2. Query Mode (--query): Run custom OVSDB JSON queries for specific data extraction

What it analyzes:

  • Per-zone logical network topology
  • Logical Switches and their ports
  • Pod Logical Switch Ports with namespace, pod name, and IP addresses
  • Access Control Lists (ACLs) with priorities, directions, and match rules
  • Logical Routers and their ports

Important: This command only works with must-gathers from clusters, where each node/zone has its own database files.

Prerequisites

The must-gather should contain:

network_logs/
└── ovnk_database_store.tar.gz

Required Tools:

  • ovsdb-tool must be installed (from openvswitch package)
    • Check with: which ovsdb-tool
    • Install: sudo dnf install openvswitch or sudo apt install openvswitch-common

Analysis Script:

The script is bundled with this plugin:

<plugin-root>/skills/must-gather-analyzer/scripts/analyze_ovn_dbs.py

Where <plugin-root> is the directory where this plugin is installed (typically ~/.cursor/commands/ai-helpers/plugins/must-gather/ or similar).

Claude will automatically locate it by searching for the script in the plugin installation directory, regardless of your current working directory.

Implementation

The command performs the following steps:

  1. Locate Analysis Script:

    SCRIPT_PATH=$(find ~ -name "analyze_ovn_dbs.py" -path "*/must-gather/skills/must-gather-analyzer/scripts/*" 2>/dev/null | head -1)
    
    if [ -z "$SCRIPT_PATH" ]; then
        echo "ERROR: analyze_ovn_dbs.py script not found."
        echo "Please ensure the must-gather plugin from ai-helpers is properly installed."
        exit 1
    fi
    
    SCRIPTS_DIR=$(dirname "$SCRIPT_PATH")
    
  2. Extract Database Tarball:

    • Locate network_logs/ovnk_database_store.tar.gz
    • Extract if not already extracted
    • Find all *_nbdb and *_sbdb files
  3. Query Each Zone's Database: For each zone (node), query the Northbound database using ovsdb-tool query:

    ovsdb-tool query <zone>_nbdb '["OVN_Northbound", {"op":"select", "table":"<table>", "where":[], "columns":[...]}]'
    
  4. Analyze and Display:

    • Logical Switches: Names and port counts
    • Logical Switch Ports: Filter for pods (external_ids.pod=true), show namespace, pod name, and IP
    • ACLs: Priority, direction, match rules, and actions
    • Logical Routers: Names and port counts
  5. Present Zone Summary:

    • Total counts per zone
    • Detailed breakdowns
    • Sorted and formatted output

Return Value

The command outputs structured analysis for each node:

Found 6 node(s)

================================================================================
Node: ip-10-0-26-145.us-east-2.compute.internal
Pod:  ovnkube-node-79cbh
================================================================================
  Logical Switches:      4
  Logical Switch Ports:  55
  ACLs:                  7
  Logical Routers:       2

  LOGICAL SWITCHES (4):
  NAME                                                         PORTS
  --------------------------------------------------------------------------------
  transit_switch                                               6
  ip-10-0-1-10.us-east-2.compute.internal                      7
  ext_ip-10-0-1-10.us-east-2.compute.internal                  2
  join                                                         2

  POD LOGICAL SWITCH PORTS (5):
  NAMESPACE                                POD                                           IP
  ------------------------------------------------------------------------------------------------------------------------
  openshift-dns                            dns-default-abc123                            10.128.0.5
  openshift-monitoring                     prometheus-k8s-0                              10.128.0.10
  openshift-etcd                           etcd-master-0                                 10.128.0.3
  ...

  ACCESS CONTROL LISTS (7):
  PRIORITY   DIRECTION       ACTION          MATCH
  ------------------------------------------------------------------------------------------------------------------------
  1012       from-lport      allow           inport == @a4743249366342378346 && (ip4.mcast ...
  1011       to-lport        drop            (ip4.mcast || mldv1 || mldv2 || ...
  1001       to-lport        allow-related   ip4.src==10.128.0.2
  ...

  LOGICAL ROUTERS (2):
  NAME                                                         PORTS
  --------------------------------------------------------------------------------
  ovn_cluster_router                                           3
  GR_ip-10-0-1-10.us-east-2.compute.internal                   2

Examples

  1. Analyze all nodes in a must-gather:

    /must-gather:ovn-dbs ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/
    

    Shows logical network topology for all nodes.

  2. Analyze specific node:

    /must-gather:ovn-dbs ./must-gather/.../ --node ip-10-0-26-145
    

    Shows OVN database information only for the specified node (supports partial name matching).

  3. Analyze master node:

    /must-gather:ovn-dbs ./must-gather/.../ --node master-0
    

    Filter to a specific master node using partial name matching.

  4. Interactive usage without path:

    /must-gather:ovn-dbs
    

    The command will ask for the must-gather path.

  5. Check if pod exists in OVN:

    /must-gather:ovn-dbs ./must-gather/.../
    

    Then search the output for the pod name to see which node it's on and its IP allocation.

  6. Investigate ACL rules on a specific node:

    /must-gather:ovn-dbs ./must-gather/.../ --node worker-1
    

    Review the ACL section for a specific node to understand traffic filtering rules.

  7. Run custom OVSDB query (Query Mode):

    /must-gather:ovn-dbs ./must-gather/.../ --query '["OVN_Northbound", {"op":"select", "table":"ACL", "where":[["priority", ">", 1000]], "columns":["priority","match","action"]}]'
    

    Query ACLs with priority > 1000 across all nodes. Claude can construct the JSON query for any OVSDB table.

  8. Query specific node with custom query:

    /must-gather:ovn-dbs ./must-gather/.../ --node master-0 --query '["OVN_Northbound", {"op":"select", "table":"Logical_Switch", "where":[], "columns":["name","ports"]}]'
    

    List all logical switches with their ports on master-0.

  9. Query specific table (Claude constructs JSON): Just ask Claude to query a specific OVSDB table and it will construct the appropriate JSON query. For example:

    • "Show all Logical_Router_Static_Route entries"
    • "Find ACLs with action 'drop'"
    • "List Logical_Switch_Port entries where external_ids contains 'openshift-etcd'"

Error Handling

Missing ovsdb-tool:

Error: ovsdb-tool not found. Please install openvswitch package.

Solution: Install openvswitch: sudo dnf install openvswitch

Missing database tarball:

Error: Database tarball not found: network_logs/ovnk_database_store.tar.gz

Solution: Ensure this is a must-gather from an OVN cluster.

Node not found:

Error: No databases found for node matching 'master-5'

Available nodes:
  - ip-10-0-77-117.us-east-2.compute.internal
  - ip-10-0-26-145.us-east-2.compute.internal
  - ip-10-0-1-194.us-east-2.compute.internal

Solution: Use one of the listed node names or a partial match.

Notes

  • Binary Database Format: Uses ovsdb-tool to read OVSDB binary files directly
  • Per-Node Analysis: Each node in IC mode has its own database (one NB and one SB per zone)
  • Node Mapping: Automatically correlates ovnkube pods to nodes by reading pod specs from must-gather
  • Pod Discovery: Pods are identified by external_ids with pod=true
  • IP Extraction: Pod IPs are parsed from the addresses field (format: "MAC IP")
  • ACL Priorities: Higher priority ACLs are processed first (shown at top)
  • Node Filtering: Supports partial name matching for convenience (e.g., "--node master" matches all masters)
  • Query Mode: Accepts raw OVSDB JSON queries in the format ["OVN_Northbound", {"op":"select", "table":"...", ...}]
  • Claude Query Construction: Claude can automatically construct OVSDB JSON queries based on natural language requests
  • Performance: Querying large databases may take a few seconds per node

Use Cases

  1. Verify Pod Network Configuration:

    • Check if pods are registered in OVN
    • Verify IP address assignments
    • Confirm logical switch port creation
  2. Troubleshoot Connectivity Issues:

    • Review ACL rules blocking traffic
    • Check if pods are in correct logical switches
    • Verify router configurations
  3. Understand Topology:

    • See how zones are interconnected via transit_switch
    • Review gateway router configurations
    • Understand logical network structure
  4. Audit Network Policies:

    • See ACL rules generated from NetworkPolicies
    • Identify overly permissive or restrictive rules
    • Check rule priorities and match conditions

Arguments

  • $1 (must-gather-path): Optional. Path to the must-gather directory containing network_logs/. If not provided, user will be prompted.
  • --node, -n (node-name): Optional. Filter analysis to a specific node. Supports partial name matching (e.g., "master-0", "ip-10-0-26-145"). If no match is found, displays list of available nodes.
  • --query, -q (json-query): Optional. Run a raw OVSDB JSON query instead of standard analysis. Claude can construct the JSON query based on OVSDB transaction format. When provided, outputs raw JSON results instead of formatted analysis.

Name

must-gather:windows

Synopsis

/must-gather:windows [must-gather-path] [--component COMPONENT] [--errors-only]

Description

The must-gather:windows command analyzes Windows-specific logs collected during must-gather from Windows nodes in OpenShift clusters. It parses logs from Windows-specific components and identifies common Windows node issues.

This command analyzes logs from:

  • kube-proxy - Windows networking service
  • hybrid-overlay - OVN-Kubernetes hybrid networking (Linux-Windows pod communication)
  • kubelet - Windows node agent
  • containerd - Container runtime for Windows
  • WICD - Windows Instance Config Daemon (node configuration)
  • csi-proxy - Storage plugin for Windows

Use this command when:

  • Troubleshooting Windows node issues
  • Investigating Windows pod failures
  • Analyzing Windows container runtime problems
  • Debugging hybrid-overlay networking issues
  • Reviewing HNS (Host Network Service) failures

Prerequisites

Windows Node Logs Collection:

Windows node logs must be collected during must-gather. This requires:

  1. The cluster has Windows nodes (labeled with kubernetes.io/os=windows)
  2. Must-gather was run with Windows log collection enabled

Expected Directory Structure:

must-gather/
└── host_service_logs/
    └── windows/
        └── log_files/
            ├── kube-proxy/kube-proxy.log
            ├── hybrid-overlay/hybrid-overlay.log
            ├── kubelet/kubelet.log
            ├── containerd/containerd.log
            ├── wicd/
            │   ├── windows-instance-config-daemon.exe.INFO
            │   ├── windows-instance-config-daemon.exe.ERROR
            │   └── windows-instance-config-daemon.exe.WARNING
            └── csi-proxy/csi-proxy.log

Implementation

  1. Locate Windows logs:

    • Check for host_service_logs/windows/log_files/ directory
    • If not found, inform user that cluster may not have Windows nodes or logs weren't collected
  2. Run Windows log analyzer:

    python3 plugins/must-gather/skills/must-gather-analyzer/scripts/analyze_windows_logs.py \
      <must-gather-path>
    
  3. Parse logs for errors and warnings:

    • Search for error patterns specific to each component
    • Categorize errors (HNS, Containerd, Hybrid-Overlay, etc.)
    • Count occurrences and identify trends
  4. Detect common Windows issues:

    • HNS failures - Pods stuck in ContainerCreating
    • Containerd errors - Container runtime failures
    • Hybrid-overlay issues - Linux-Windows connectivity problems
    • Kubelet failures - Pod scheduling and management issues
    • WICD errors - Node configuration problems
    • CSI-Proxy failures - Storage mount issues
  5. Generate report with:

    • Component status summary
    • Error and warning counts
    • Detected issues with severity (CRITICAL, HIGH, etc.)
    • Recommendations for remediation
    • Detailed error messages categorized by type

Return Value

The command outputs:

================================================================================
WINDOWS NODE LOGS ANALYSIS
================================================================================
Log directory: <path>

Components analyzed: 6/8
Total log lines:     125,432
Total errors found:  23
Total warnings:      15

COMPONENT STATUS:
COMPONENT                 LINES      ERRORS     WARNINGS   STATUS
--------------------------------------------------------------------------------
kube-proxy                15,234     0          2          ✅ OK
hybrid-overlay            8,912      5          3          ❌ ERRORS
kubelet                   45,123     12         5          ❌ ERRORS
containerd                32,456     6          4          ❌ ERRORS
wicd-info                 12,345     0          1          ✅ OK
wicd-error                234        0          0          ✅ OK
wicd-warning              456        0          0          ✅ OK
csi-proxy                 10,672     0          0          ✅ OK

================================================================================
DETECTED ISSUES
================================================================================

1. [CRITICAL] HNS (Host Network Service) Failures Detected
   Found 5 HNS-related errors. This typically causes pods to fail in ContainerCreating state.
   → Check Windows node networking configuration. May need to restart HNS service or reboot node.

2. [CRITICAL] Container Runtime Failures
   Found 6 containerd errors. Containers may fail to start.
   → Check containerd service status on Windows node. Review container image compatibility.

================================================================================
DETAILED ERRORS BY CATEGORY
================================================================================

HNS ERRORS (5):
--------------------------------------------------------------------------------
  [hybrid-overlay:1234] failed to create HNS endpoint for pod default/test-pod
  [hybrid-overlay:2345] HNS network attach failed: endpoint not found
  ...

CONTAINERD ERRORS (6):
--------------------------------------------------------------------------------
  [containerd:567] failed to start container: runtime error
  [kubelet:890] failed to create pod sandbox: containerd timeout
  ...

Examples

  1. Analyze all Windows logs:

    /must-gather:windows ./must-gather.local.123456789
    

    Analyzes all Windows component logs and provides comprehensive report.

  2. Analyze specific component:

    /must-gather:windows ./must-gather.local.123456789 --component kubelet
    

    Analyzes only kubelet logs from Windows nodes.

  3. Summary only (skip detailed errors):

    /must-gather:windows ./must-gather.local.123456789 --errors-only
    

    Shows component status and detected issues without detailed error listing.

  4. Analyze hybrid-overlay networking:

    /must-gather:windows ./must-gather.local.123456789 --component hybrid-overlay
    

    Focuses on hybrid-overlay logs to troubleshoot Linux-Windows pod connectivity.

  5. Analyze containerd runtime:

    /must-gather:windows ./must-gather.local.123456789 --component containerd
    

    Reviews container runtime logs for Windows container failures.

Common Windows Issues Detected

HNS (Host Network Service) Failures

Symptoms:

  • Pods stuck in ContainerCreating
  • failed to create HNS endpoint errors
  • Network attach failures

Recommendations:

  • Restart HNS service: Restart-Service hns
  • Check Windows Firewall rules
  • Verify network adapter configuration
  • Consider node reboot if HNS is unresponsive

Containerd Runtime Errors

Symptoms:

  • Containers fail to start
  • runtime error messages
  • Image pull failures

Recommendations:

  • Verify image OS matches (Windows vs Linux)
  • Check containerd service status
  • Review image platform compatibility
  • Inspect containerd configuration

Hybrid-Overlay Networking Issues

Symptoms:

  • Linux pods cannot reach Windows pods
  • Tunnel setup failures
  • OVN errors

Recommendations:

  • Verify OVN-Kubernetes hybrid-overlay configuration
  • Check tunnel connectivity between nodes
  • Review network policies
  • Validate VXLAN configuration

Kubelet Issues

Symptoms:

  • Pods not scheduling to Windows nodes
  • Pod sandbox creation failures
  • Runtime not ready errors

Recommendations:

  • Check kubelet service status
  • Review node conditions
  • Verify container runtime connectivity
  • Check pod node selectors

WICD Configuration Errors

Symptoms:

  • Node configuration failures
  • Instance config daemon errors

Recommendations:

  • Review WICD error logs
  • Check node bootstrap configuration
  • Verify cloud provider integration

Notes

  • No Windows Nodes: If the command reports no Windows logs found, the cluster either has no Windows nodes or the logs weren't collected during must-gather
  • Log Location: Windows logs are collected from nodes labeled kubernetes.io/os=windows
  • Collection Script: Windows logs are gathered by the gather_windows_nodes script in the must-gather collection phase
  • Error Patterns: The analyzer uses regex patterns specific to each Windows component
  • Max Errors: By default, shows first 50 errors per log file; use --max-errors to adjust
  • Cross-Reference: Combine with /must-gather:analyze for full cluster analysis including Windows resources

Arguments

  • $1 (must-gather-path): Optional. Path to must-gather directory. If not provided, user will be prompted.
  • --component COMPONENT: Optional. Analyze only a specific component (kubelet, containerd, hybrid-overlay, kube-proxy, wicd, csi-proxy).
  • --errors-only: Optional. Show summary and detected issues only, skip detailed error listings.
  • --max-errors N: Optional. Maximum number of errors to collect per log file (default: 50).

Sponsors

Details

Marketplace
openshift-eng/ai-helpers
Source
plugins/must-gather
Category
Debugging
Keywords
must-gather, diagnostics, cluster-analysis

More from openshift-eng/ai-helpers

All 56