
must-gather
A plugin to analyze and report on must-gather data
- version
- 0.0.3
- GitHub stars
- 123
Install
/plugin install must-gather@ai-helpersRun it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.
Contents
Discovered from sourceSkills 1
Must-Gather Analyzer Skill
Comprehensive analysis of OpenShift must-gather diagnostic data with helper scripts that parse YAML and display output in oc-like format.
Overview
This skill provides analysis for:
- ClusterVersion: Current version, update status, and capabilities
- Cluster Operators: Status, degradation, and availability
- Pods: Health, restarts, crashes, and failures across namespaces
- Nodes: Conditions, capacity, and readiness
- Network: OVN/SDN diagnostics and connectivity
- Events: Warning and error events across namespaces
- etcd: Cluster health, member status, and quorum
- Storage: PersistentVolumes and PersistentVolumeClaims status
Must-Gather Directory Structure
Important: Must-gather data is contained in a subdirectory with a long hash name:
must-gather/
└── registry-ci-openshift-org-origin-...-sha256-<hash>/
├── cluster-scoped-resources/
│ ├── config.openshift.io/clusteroperators/
│ └── core/nodes/
├── namespaces/
│ └── <namespace>/
│ └── pods/
│ └── <pod-name>/
│ └── <pod-name>.yaml
└── network_logs/
The analysis scripts expect the path to the subdirectory (the one with the hash), not the root must-gather folder.
Instructions
1. Get Must-Gather Path
Ask the user for the must-gather directory path if not already provided.
- If they provide the root directory, look for the subdirectory with the hash name
- The correct path contains
cluster-scoped-resources/andnamespaces/directories
2. Choose Analysis Type
Based on user's request, run the appropriate helper script:
ClusterVersion Analysis
./scripts/analyze_clusterversion.py <must-gather-path>
Shows cluster version information similar to oc get clusterversion:
- Current version and update status
- Progressing state
- Available updates
- Version conditions
- Enabled capabilities
- Update history
Cluster Operators Analysis
./scripts/analyze_clusteroperators.py <must-gather-path>
Shows cluster operator status similar to oc get clusteroperators:
- Available, Progressing, Degraded conditions
- Version information
- Time since condition change
- Detailed messages for operators with issues
Pods Analysis
# All namespaces
./scripts/analyze_pods.py <must-gather-path>
# Specific namespace
./scripts/analyze_pods.py <must-gather-path> --namespace <namespace>
# Show only problematic pods
./scripts/analyze_pods.py <must-gather-path> --problems-only
Shows pod status similar to oc get pods -A:
- Ready/Total containers
- Status (Running, Pending, CrashLoopBackOff, etc.)
- Restart counts
- Age
- Categorized issues (crashlooping, pending, failed)
Nodes Analysis
./scripts/analyze_nodes.py <must-gather-path>
# Show only nodes with issues
./scripts/analyze_nodes.py <must-gather-path> --problems-only
Shows node status similar to oc get nodes:
- Ready status
- Roles (master, worker)
- Age
- Kubernetes version
- Node conditions (DiskPressure, MemoryPressure, etc.)
- Capacity and allocatable resources
Network Analysis
./scripts/analyze_network.py <must-gather-path>
Shows network health:
- Network type (OVN-Kubernetes, OpenShift SDN)
- Network operator status
- OVN pod health
- PodNetworkConnectivityCheck results
- Network-related issues
Events Analysis
# Recent events (last 100)
./scripts/analyze_events.py <must-gather-path>
# Warning events only
./scripts/analyze_events.py <must-gather-path> --type Warning
# Events in specific namespace
./scripts/analyze_events.py <must-gather-path> --namespace openshift-etcd
# Show last 50 events
./scripts/analyze_events.py <must-gather-path> --count 50
Shows cluster events:
- Event type (Warning, Normal)
- Last seen timestamp
- Reason and message
- Affected object
- Event count
etcd Analysis
./scripts/analyze_etcd.py <must-gather-path>
Shows etcd cluster health:
- Member health status
- Member list with IDs and URLs
- Endpoint status (leader, version, DB size)
- Quorum status
- Cluster summary
Storage Analysis
# All PVs and PVCs
./scripts/analyze_pvs.py <must-gather-path>
# PVCs in specific namespace
./scripts/analyze_pvs.py <must-gather-path> --namespace openshift-monitoring
Shows storage resources:
- PersistentVolumes (capacity, status, claims)
- PersistentVolumeClaims (binding, capacity)
- Storage classes
- Pending/unbound volumes
Monitoring Analysis
# All alerts.
./scripts/analyze_prometheus.py <must-gather-path>
# Alerts in specific namespace
./scripts/analyze_prometheus.py <must-gather-path> --namespace openshift-monitoring
Shows monitoring information:
- Alerts (state, namespace, name, active since, labels)
- Total of pending/firing alerts
3. Interpret and Report
After running the scripts:
- Review the summary statistics
- Focus on items flagged with issues
- Provide actionable insights and next steps
- Suggest log analysis for specific components if needed
- Cross-reference issues (e.g., degraded operator → failing pods → node issues)
Output Format
All scripts provide:
- Summary Section: High-level statistics with emoji indicators
- Table View:
oc-like formatted output - Issues Section: Detailed breakdown of problems
Example summary format:
================================================================================
SUMMARY: 25/28 operators healthy
⚠️ 3 operators with issues
🔄 1 progressing
❌ 2 degraded
================================================================================
Helper Scripts Reference
scripts/analyze_clusterversion.py
Parses: cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml
Output: ClusterVersion table with detailed version info, conditions, and capabilities
scripts/analyze_clusteroperators.py
Parses: cluster-scoped-resources/config.openshift.io/clusteroperators/
Output: ClusterOperator status table with conditions
scripts/analyze_pods.py
Parses: namespaces/*/pods/*/*.yaml (individual pod directories)
Output: Pod status table with issues categorized
scripts/analyze_nodes.py
Parses: cluster-scoped-resources/core/nodes/
Output: Node status table with conditions and capacity
scripts/analyze_network.py
Parses: network_logs/, network operator, OVN resources
Output: Network health summary and diagnostics
scripts/analyze_events.py
Parses: namespaces/*/core/events.yaml
Output: Event table sorted by last occurrence
scripts/analyze_etcd.py
Parses: etcd_info/ (endpoint_health.json, member_list.json, endpoint_status.json)
Output: etcd cluster health and member status
scripts/analyze_pvs.py
Parses: cluster-scoped-resources/core/persistentvolumes/, namespaces/*/core/persistentvolumeclaims.yaml
Output: PV and PVC status tables
scripts/analyze_ovn_dbs.py
Parses OVN Northbound and Southbound database dumps and supports targeted OVSDB queries.
scripts/analyze_windows_logs.py
Parses Windows node component logs for kube-proxy, hybrid-overlay, kubelet, containerd, WICD, and CSI proxy.
Tips for Analysis
- Start with Cluster Operators: They often reveal system-wide issues
- Check Timing: Look at "SINCE" columns to understand when issues started
- Follow Dependencies: Degraded operator → check its namespace pods → check hosting nodes
- Look for Patterns: Multiple pods failing on same node suggests node issue
- Cross-reference: Use multiple scripts together for complete picture
Common Scenarios
"Why is my cluster degraded?"
- Run
analyze_clusteroperators.py- identify degraded operators - Run
analyze_pods.py --namespace <operator-namespace>- check operator pods - Run
analyze_nodes.py- verify node health
"Pods keep crashing"
- Run
analyze_pods.py --problems-only- find crashlooping pods - Check which nodes they're on
- Run
analyze_nodes.py- verify node conditions - Suggest checking pod logs in must-gather data
"Network connectivity issues"
- Run
analyze_network.py- check network health - Run
analyze_pods.py --namespace openshift-ovn-kubernetes - Check PodNetworkConnectivityCheck results
Next Steps After Analysis
Based on findings, suggest:
- Examining specific pod logs in
namespaces/<ns>/pods/<pod>/<container>/logs/ - Reviewing events in
namespaces/<ns>/core/events.yaml - Checking audit logs in
audit_logs/ - Analyzing metrics data if available
- Looking at host service logs in
host_service_logs/
Commands 3
Name
must-gather:analyze
Synopsis
/must-gather:analyze [must-gather-path] [component]
Description
The analyze command performs comprehensive analysis of OpenShift must-gather diagnostic data. It runs specialized Python analysis scripts to extract and summarize cluster health information across multiple components.
The command can analyze:
- Cluster version and update status
- Cluster operator health (degraded, progressing, unavailable)
- Node conditions and resource status
- Pod failures, restarts, and crash loops
- Network configuration and OVN health
- OVN databases - logical topology, ACLs, pods
- Kubernetes events (warnings and errors)
- etcd cluster health and quorum status
- Persistent volume and claim status
- Prometheus alerts
You can request analysis of the entire cluster or focus on a specific component.
Prerequisites
Required Directory Structure:
Must-gather data typically has this structure:
must-gather/
└── registry-ci-openshift-org-origin-...-sha256-<hash>/
├── cluster-scoped-resources/
├── namespaces/
└── ...
The actual must-gather directory is the subdirectory with the hash name, not the parent directory.
Required Scripts:
Analysis scripts are bundled with this plugin at:
<plugin-root>/skills/must-gather-analyzer/scripts/
├── analyze_clusterversion.py
├── analyze_clusteroperators.py
├── analyze_nodes.py
├── analyze_pods.py
├── analyze_network.py
├── analyze_ovn_dbs.py
├── analyze_events.py
├── analyze_etcd.py
└── analyze_pvs.py
Where <plugin-root> is the directory where this plugin is installed (typically ~/.cursor/commands/ai-helpers/plugins/must-gather/ or similar).
Error Handling
CRITICAL: Script-Only Analysis
- NEVER attempt to analyze must-gather data directly using bash commands, grep, or manual file reading
- ONLY use the provided Python scripts in
plugins/must-gather/skills/must-gather-analyzer/scripts/ - If scripts are missing or not found:
- Stop immediately
- Inform the user that the analysis scripts are not available
- Ask the user to ensure the scripts are installed at the correct path
- Do NOT attempt alternative approaches
Script Availability Check:
Before running any analysis:
-
Locate the scripts directory by searching for a known script:
SCRIPT_PATH=$(find ~ -name "analyze_clusteroperators.py" -path "*/must-gather/skills/must-gather-analyzer/scripts/*" 2>/dev/null | head -1) if [ -z "$SCRIPT_PATH" ]; then echo "ERROR: Must-gather analysis scripts not found." echo "Please ensure the must-gather plugin from ai-helpers is properly installed." exit 1 fi # All scripts are in the same directory, so just get the directory SCRIPTS_DIR=$(dirname "$SCRIPT_PATH") -
If scripts cannot be found, STOP and report to the user:
The must-gather analysis scripts could not be located. Please ensure the must-gather plugin from openshift-eng/ai-helpers is properly installed in your Claude Code plugins directory.
Implementation
The command performs the following steps:
-
Validate Must-Gather Path:
- If path not provided as argument, ask the user
- Check if path contains
cluster-scoped-resources/andnamespaces/directories - If user provides root directory, automatically find the correct subdirectory
- Verify the path exists and is readable
-
Determine Analysis Scope:
STEP 1: Check for SPECIFIC component keywords
If the user mentions a specific component, run ONLY that script:
- "pods", "pod status", "containers", "crashloop", "failing pods" →
analyze_pods.pyONLY - "etcd", "etcd health", "quorum" →
analyze_etcd.pyONLY - "network", "networking", "ovn", "connectivity" →
analyze_network.pyONLY - "ovn databases", "ovn-dbs", "ovn db", "logical switches", "acls" →
analyze_ovn_dbs.pyONLY - "nodes", "node status", "node conditions" →
analyze_nodes.pyONLY - "operators", "cluster operators", "degraded" →
analyze_clusteroperators.pyONLY - "version", "cluster version", "update", "upgrade" →
analyze_clusterversion.pyONLY - "events", "warnings", "errors" →
analyze_events.pyONLY - "storage", "pv", "pvc", "volumes", "persistent" →
analyze_pvs.pyONLY - "alerts", "prometheus", "monitoring" →
analyze_prometheus.pyONLY - "windows", "windows nodes", "windows logs", "hns", "containerd", "hybrid-overlay" →
analyze_windows_logs.pyONLY
STEP 2: No specific component mentioned
If generic request like "analyze must-gather", "/must-gather:analyze", or "check the cluster", run ALL scripts in this order:
- ClusterVersion (
analyze_clusterversion.py) - Cluster Operators (
analyze_clusteroperators.py) - Nodes (
analyze_nodes.py) - Pods - problems only (
analyze_pods.py --problems-only) - Network (
analyze_network.py) - Events - warnings only (
analyze_events.py --type Warning --count 50) - etcd (
analyze_etcd.py) - Storage (
analyze_pvs.py) - Monitoring (
analyze_prometheus.py) - Windows Logs - ONLY if Windows nodes detected:
- First check if
host_service_logs/windows/directory exists in the must-gather - If directory exists, run:
analyze_windows_logs.py <must-gather-path> - If directory does not exist, skip silently (cluster has no Windows nodes)
- First check if
- "pods", "pod status", "containers", "crashloop", "failing pods" →
-
Locate Plugin Scripts:
- Use the script availability check from the Error Handling section to find the plugin root
- Store the scripts directory path in
$SCRIPTS_DIR
-
Execute Analysis Scripts:
python3 "$SCRIPTS_DIR/<script>.py" <must-gather-path>Example:
python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" ./must-gather.local.123/quay-io-... -
Synthesize Results: Generate findings and recommendations based on script output
Return Value
The command outputs structured analysis results to stdout:
For Component-Specific Analysis:
- Script output for the requested component only
- Focused findings and recommendations
For Full Analysis:
- Organized sections for each component
- Executive summary of overall cluster health
- Prioritized list of critical issues
- Actionable recommendations
- Suggested log files to review
Output Structure
================================================================================
MUST-GATHER ANALYSIS SUMMARY
================================================================================
[Script outputs organized by component]
CLUSTER VERSION:
[output from analyze_clusterversion.py]
CLUSTER OPERATORS:
[output from analyze_clusteroperators.py]
NODES:
[output from analyze_nodes.py]
PROBLEMATIC PODS:
[output from analyze_pods.py --problems-only]
NETWORK STATUS:
[output from analyze_network.py]
WARNING EVENTS (Last 50):
[output from analyze_events.py --type Warning --count 50]
ETCD CLUSTER HEALTH:
[output from analyze_etcd.py]
STORAGE (PVs/PVCs):
[output from analyze_pvs.py]
MONITORING (Alerts):
[output from analyze_prometheus.py]
================================================================================
FINDINGS AND RECOMMENDATIONS
================================================================================
Critical Issues:
- [Critical problems requiring immediate attention]
Warnings:
- [Potential issues or degraded components]
Recommendations:
- [Specific next steps for investigation]
Logs to Review:
- [Specific log files to examine based on findings]
Examples
-
Full cluster analysis:
/must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/Runs all analysis scripts and provides comprehensive cluster diagnostics.
-
Analyze pod issues only:
/must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/ analyze the pod statusesRuns only
analyze_pods.pyto focus on pod-related issues. -
Check etcd health:
/must-gather:analyze check etcd healthAsks for must-gather path, then runs only
analyze_etcd.py. -
Network troubleshooting:
/must-gather:analyze ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/ show me network issuesRuns only
analyze_network.pyfor network-specific analysis.
Notes
- Must-Gather Path: Always use the subdirectory containing
cluster-scoped-resources/andnamespaces/, not the parent directory - Script Dependencies: Analysis scripts must be executable and have required Python dependencies installed
- Error Handling: If scripts are not found or must-gather path is invalid, clear error messages are displayed
- Cross-Referencing: The analysis attempts to correlate issues across components (e.g., degraded operator → failing pods)
- Pattern Detection: Identifies patterns like multiple pod failures on the same node
- Actionable Output: Focuses on insights and recommendations rather than raw data dumps
- Priority: Issues are prioritized by severity (Critical > Warning > Info)
Arguments
- $1 (must-gather-path): Optional. Path to the must-gather directory (the subdirectory with the hash name). If not provided, the user will be asked.
- $2+ (component): Optional. If keywords for a specific component are detected, only that component's analysis script will run. Otherwise, all scripts run.
Name
must-gather:ovn-dbs
Synopsis
/must-gather:ovn-dbs [must-gather-path] [--node <node-name>] [--query <json>]
Description
The ovn-dbs command analyzes OVN Northbound and Southbound databases collected from clusters. It uses ovsdb-tool to query the binary database files (.db) collected per-node, providing detailed information about the logical network topology, pods, ACLs, and routers on each node.
The command automatically maps ovnkube pods to their corresponding nodes by reading pod specifications from the must-gather data.
Two modes of operation:
- Standard Analysis (default): Runs pre-built analysis showing switches, ports, ACLs, and routers
- Query Mode (
--query): Run custom OVSDB JSON queries for specific data extraction
What it analyzes:
- Per-zone logical network topology
- Logical Switches and their ports
- Pod Logical Switch Ports with namespace, pod name, and IP addresses
- Access Control Lists (ACLs) with priorities, directions, and match rules
- Logical Routers and their ports
Important: This command only works with must-gathers from clusters, where each node/zone has its own database files.
Prerequisites
The must-gather should contain:
network_logs/
└── ovnk_database_store.tar.gz
Required Tools:
ovsdb-toolmust be installed (from openvswitch package)- Check with:
which ovsdb-tool - Install:
sudo dnf install openvswitchorsudo apt install openvswitch-common
- Check with:
Analysis Script:
The script is bundled with this plugin:
<plugin-root>/skills/must-gather-analyzer/scripts/analyze_ovn_dbs.py
Where <plugin-root> is the directory where this plugin is installed (typically ~/.cursor/commands/ai-helpers/plugins/must-gather/ or similar).
Claude will automatically locate it by searching for the script in the plugin installation directory, regardless of your current working directory.
Implementation
The command performs the following steps:
-
Locate Analysis Script:
SCRIPT_PATH=$(find ~ -name "analyze_ovn_dbs.py" -path "*/must-gather/skills/must-gather-analyzer/scripts/*" 2>/dev/null | head -1) if [ -z "$SCRIPT_PATH" ]; then echo "ERROR: analyze_ovn_dbs.py script not found." echo "Please ensure the must-gather plugin from ai-helpers is properly installed." exit 1 fi SCRIPTS_DIR=$(dirname "$SCRIPT_PATH") -
Extract Database Tarball:
- Locate
network_logs/ovnk_database_store.tar.gz - Extract if not already extracted
- Find all
*_nbdband*_sbdbfiles
- Locate
-
Query Each Zone's Database: For each zone (node), query the Northbound database using
ovsdb-tool query:ovsdb-tool query <zone>_nbdb '["OVN_Northbound", {"op":"select", "table":"<table>", "where":[], "columns":[...]}]' -
Analyze and Display:
- Logical Switches: Names and port counts
- Logical Switch Ports: Filter for pods (external_ids.pod=true), show namespace, pod name, and IP
- ACLs: Priority, direction, match rules, and actions
- Logical Routers: Names and port counts
-
Present Zone Summary:
- Total counts per zone
- Detailed breakdowns
- Sorted and formatted output
Return Value
The command outputs structured analysis for each node:
Found 6 node(s)
================================================================================
Node: ip-10-0-26-145.us-east-2.compute.internal
Pod: ovnkube-node-79cbh
================================================================================
Logical Switches: 4
Logical Switch Ports: 55
ACLs: 7
Logical Routers: 2
LOGICAL SWITCHES (4):
NAME PORTS
--------------------------------------------------------------------------------
transit_switch 6
ip-10-0-1-10.us-east-2.compute.internal 7
ext_ip-10-0-1-10.us-east-2.compute.internal 2
join 2
POD LOGICAL SWITCH PORTS (5):
NAMESPACE POD IP
------------------------------------------------------------------------------------------------------------------------
openshift-dns dns-default-abc123 10.128.0.5
openshift-monitoring prometheus-k8s-0 10.128.0.10
openshift-etcd etcd-master-0 10.128.0.3
...
ACCESS CONTROL LISTS (7):
PRIORITY DIRECTION ACTION MATCH
------------------------------------------------------------------------------------------------------------------------
1012 from-lport allow inport == @a4743249366342378346 && (ip4.mcast ...
1011 to-lport drop (ip4.mcast || mldv1 || mldv2 || ...
1001 to-lport allow-related ip4.src==10.128.0.2
...
LOGICAL ROUTERS (2):
NAME PORTS
--------------------------------------------------------------------------------
ovn_cluster_router 3
GR_ip-10-0-1-10.us-east-2.compute.internal 2
Examples
-
Analyze all nodes in a must-gather:
/must-gather:ovn-dbs ./must-gather/registry-ci-openshift-org-origin-4-20-...-sha256-abc123/Shows logical network topology for all nodes.
-
Analyze specific node:
/must-gather:ovn-dbs ./must-gather/.../ --node ip-10-0-26-145Shows OVN database information only for the specified node (supports partial name matching).
-
Analyze master node:
/must-gather:ovn-dbs ./must-gather/.../ --node master-0Filter to a specific master node using partial name matching.
-
Interactive usage without path:
/must-gather:ovn-dbsThe command will ask for the must-gather path.
-
Check if pod exists in OVN:
/must-gather:ovn-dbs ./must-gather/.../Then search the output for the pod name to see which node it's on and its IP allocation.
-
Investigate ACL rules on a specific node:
/must-gather:ovn-dbs ./must-gather/.../ --node worker-1Review the ACL section for a specific node to understand traffic filtering rules.
-
Run custom OVSDB query (Query Mode):
/must-gather:ovn-dbs ./must-gather/.../ --query '["OVN_Northbound", {"op":"select", "table":"ACL", "where":[["priority", ">", 1000]], "columns":["priority","match","action"]}]'Query ACLs with priority > 1000 across all nodes. Claude can construct the JSON query for any OVSDB table.
-
Query specific node with custom query:
/must-gather:ovn-dbs ./must-gather/.../ --node master-0 --query '["OVN_Northbound", {"op":"select", "table":"Logical_Switch", "where":[], "columns":["name","ports"]}]'List all logical switches with their ports on master-0.
-
Query specific table (Claude constructs JSON): Just ask Claude to query a specific OVSDB table and it will construct the appropriate JSON query. For example:
- "Show all Logical_Router_Static_Route entries"
- "Find ACLs with action 'drop'"
- "List Logical_Switch_Port entries where external_ids contains 'openshift-etcd'"
Error Handling
Missing ovsdb-tool:
Error: ovsdb-tool not found. Please install openvswitch package.
Solution: Install openvswitch: sudo dnf install openvswitch
Missing database tarball:
Error: Database tarball not found: network_logs/ovnk_database_store.tar.gz
Solution: Ensure this is a must-gather from an OVN cluster.
Node not found:
Error: No databases found for node matching 'master-5'
Available nodes:
- ip-10-0-77-117.us-east-2.compute.internal
- ip-10-0-26-145.us-east-2.compute.internal
- ip-10-0-1-194.us-east-2.compute.internal
Solution: Use one of the listed node names or a partial match.
Notes
- Binary Database Format: Uses
ovsdb-toolto read OVSDB binary files directly - Per-Node Analysis: Each node in IC mode has its own database (one NB and one SB per zone)
- Node Mapping: Automatically correlates ovnkube pods to nodes by reading pod specs from must-gather
- Pod Discovery: Pods are identified by
external_idswithpod=true - IP Extraction: Pod IPs are parsed from the
addressesfield (format: "MAC IP") - ACL Priorities: Higher priority ACLs are processed first (shown at top)
- Node Filtering: Supports partial name matching for convenience (e.g., "--node master" matches all masters)
- Query Mode: Accepts raw OVSDB JSON queries in the format
["OVN_Northbound", {"op":"select", "table":"...", ...}] - Claude Query Construction: Claude can automatically construct OVSDB JSON queries based on natural language requests
- Performance: Querying large databases may take a few seconds per node
Use Cases
-
Verify Pod Network Configuration:
- Check if pods are registered in OVN
- Verify IP address assignments
- Confirm logical switch port creation
-
Troubleshoot Connectivity Issues:
- Review ACL rules blocking traffic
- Check if pods are in correct logical switches
- Verify router configurations
-
Understand Topology:
- See how zones are interconnected via transit_switch
- Review gateway router configurations
- Understand logical network structure
-
Audit Network Policies:
- See ACL rules generated from NetworkPolicies
- Identify overly permissive or restrictive rules
- Check rule priorities and match conditions
Arguments
- $1 (must-gather-path): Optional. Path to the must-gather directory containing network_logs/. If not provided, user will be prompted.
- --node, -n (node-name): Optional. Filter analysis to a specific node. Supports partial name matching (e.g., "master-0", "ip-10-0-26-145"). If no match is found, displays list of available nodes.
- --query, -q (json-query): Optional. Run a raw OVSDB JSON query instead of standard analysis. Claude can construct the JSON query based on OVSDB transaction format. When provided, outputs raw JSON results instead of formatted analysis.
Name
must-gather:windows
Synopsis
/must-gather:windows [must-gather-path] [--component COMPONENT] [--errors-only]
Description
The must-gather:windows command analyzes Windows-specific logs collected during must-gather from Windows nodes in OpenShift clusters. It parses logs from Windows-specific components and identifies common Windows node issues.
This command analyzes logs from:
- kube-proxy - Windows networking service
- hybrid-overlay - OVN-Kubernetes hybrid networking (Linux-Windows pod communication)
- kubelet - Windows node agent
- containerd - Container runtime for Windows
- WICD - Windows Instance Config Daemon (node configuration)
- csi-proxy - Storage plugin for Windows
Use this command when:
- Troubleshooting Windows node issues
- Investigating Windows pod failures
- Analyzing Windows container runtime problems
- Debugging hybrid-overlay networking issues
- Reviewing HNS (Host Network Service) failures
Prerequisites
Windows Node Logs Collection:
Windows node logs must be collected during must-gather. This requires:
- The cluster has Windows nodes (labeled with
kubernetes.io/os=windows) - Must-gather was run with Windows log collection enabled
Expected Directory Structure:
must-gather/
└── host_service_logs/
└── windows/
└── log_files/
├── kube-proxy/kube-proxy.log
├── hybrid-overlay/hybrid-overlay.log
├── kubelet/kubelet.log
├── containerd/containerd.log
├── wicd/
│ ├── windows-instance-config-daemon.exe.INFO
│ ├── windows-instance-config-daemon.exe.ERROR
│ └── windows-instance-config-daemon.exe.WARNING
└── csi-proxy/csi-proxy.log
Implementation
-
Locate Windows logs:
- Check for
host_service_logs/windows/log_files/directory - If not found, inform user that cluster may not have Windows nodes or logs weren't collected
- Check for
-
Run Windows log analyzer:
python3 plugins/must-gather/skills/must-gather-analyzer/scripts/analyze_windows_logs.py \ <must-gather-path> -
Parse logs for errors and warnings:
- Search for error patterns specific to each component
- Categorize errors (HNS, Containerd, Hybrid-Overlay, etc.)
- Count occurrences and identify trends
-
Detect common Windows issues:
- HNS failures - Pods stuck in ContainerCreating
- Containerd errors - Container runtime failures
- Hybrid-overlay issues - Linux-Windows connectivity problems
- Kubelet failures - Pod scheduling and management issues
- WICD errors - Node configuration problems
- CSI-Proxy failures - Storage mount issues
-
Generate report with:
- Component status summary
- Error and warning counts
- Detected issues with severity (CRITICAL, HIGH, etc.)
- Recommendations for remediation
- Detailed error messages categorized by type
Return Value
The command outputs:
================================================================================
WINDOWS NODE LOGS ANALYSIS
================================================================================
Log directory: <path>
Components analyzed: 6/8
Total log lines: 125,432
Total errors found: 23
Total warnings: 15
COMPONENT STATUS:
COMPONENT LINES ERRORS WARNINGS STATUS
--------------------------------------------------------------------------------
kube-proxy 15,234 0 2 ✅ OK
hybrid-overlay 8,912 5 3 ❌ ERRORS
kubelet 45,123 12 5 ❌ ERRORS
containerd 32,456 6 4 ❌ ERRORS
wicd-info 12,345 0 1 ✅ OK
wicd-error 234 0 0 ✅ OK
wicd-warning 456 0 0 ✅ OK
csi-proxy 10,672 0 0 ✅ OK
================================================================================
DETECTED ISSUES
================================================================================
1. [CRITICAL] HNS (Host Network Service) Failures Detected
Found 5 HNS-related errors. This typically causes pods to fail in ContainerCreating state.
→ Check Windows node networking configuration. May need to restart HNS service or reboot node.
2. [CRITICAL] Container Runtime Failures
Found 6 containerd errors. Containers may fail to start.
→ Check containerd service status on Windows node. Review container image compatibility.
================================================================================
DETAILED ERRORS BY CATEGORY
================================================================================
HNS ERRORS (5):
--------------------------------------------------------------------------------
[hybrid-overlay:1234] failed to create HNS endpoint for pod default/test-pod
[hybrid-overlay:2345] HNS network attach failed: endpoint not found
...
CONTAINERD ERRORS (6):
--------------------------------------------------------------------------------
[containerd:567] failed to start container: runtime error
[kubelet:890] failed to create pod sandbox: containerd timeout
...
Examples
-
Analyze all Windows logs:
/must-gather:windows ./must-gather.local.123456789Analyzes all Windows component logs and provides comprehensive report.
-
Analyze specific component:
/must-gather:windows ./must-gather.local.123456789 --component kubeletAnalyzes only kubelet logs from Windows nodes.
-
Summary only (skip detailed errors):
/must-gather:windows ./must-gather.local.123456789 --errors-onlyShows component status and detected issues without detailed error listing.
-
Analyze hybrid-overlay networking:
/must-gather:windows ./must-gather.local.123456789 --component hybrid-overlayFocuses on hybrid-overlay logs to troubleshoot Linux-Windows pod connectivity.
-
Analyze containerd runtime:
/must-gather:windows ./must-gather.local.123456789 --component containerdReviews container runtime logs for Windows container failures.
Common Windows Issues Detected
HNS (Host Network Service) Failures
Symptoms:
- Pods stuck in
ContainerCreating failed to create HNS endpointerrors- Network attach failures
Recommendations:
- Restart HNS service:
Restart-Service hns - Check Windows Firewall rules
- Verify network adapter configuration
- Consider node reboot if HNS is unresponsive
Containerd Runtime Errors
Symptoms:
- Containers fail to start
runtime errormessages- Image pull failures
Recommendations:
- Verify image OS matches (Windows vs Linux)
- Check containerd service status
- Review image platform compatibility
- Inspect containerd configuration
Hybrid-Overlay Networking Issues
Symptoms:
- Linux pods cannot reach Windows pods
- Tunnel setup failures
- OVN errors
Recommendations:
- Verify OVN-Kubernetes hybrid-overlay configuration
- Check tunnel connectivity between nodes
- Review network policies
- Validate VXLAN configuration
Kubelet Issues
Symptoms:
- Pods not scheduling to Windows nodes
- Pod sandbox creation failures
- Runtime not ready errors
Recommendations:
- Check kubelet service status
- Review node conditions
- Verify container runtime connectivity
- Check pod node selectors
WICD Configuration Errors
Symptoms:
- Node configuration failures
- Instance config daemon errors
Recommendations:
- Review WICD error logs
- Check node bootstrap configuration
- Verify cloud provider integration
Notes
- No Windows Nodes: If the command reports no Windows logs found, the cluster either has no Windows nodes or the logs weren't collected during must-gather
- Log Location: Windows logs are collected from nodes labeled
kubernetes.io/os=windows - Collection Script: Windows logs are gathered by the
gather_windows_nodesscript in the must-gather collection phase - Error Patterns: The analyzer uses regex patterns specific to each Windows component
- Max Errors: By default, shows first 50 errors per log file; use
--max-errorsto adjust - Cross-Reference: Combine with
/must-gather:analyzefor full cluster analysis including Windows resources
Arguments
- $1 (must-gather-path): Optional. Path to must-gather directory. If not provided, user will be prompted.
- --component COMPONENT: Optional. Analyze only a specific component (kubelet, containerd, hybrid-overlay, kube-proxy, wicd, csi-proxy).
- --errors-only: Optional. Show summary and detected issues only, skip detailed error listings.
- --max-errors N: Optional. Maximum number of errors to collect per log file (default: 50).
Sponsors
Details
- Marketplace
- openshift-eng/ai-helpers
- Source
- plugins/must-gather
- Category
- Debugging
- Keywords
- must-gather, diagnostics, cluster-analysis
More from openshift-eng/ai-helpers
All 56- native-notificationsv0.0.2
- nid-teamv0.2.2
- nodev0.0.3
- node-bugv0.0.3
- node-cvev0.0.8
- node-onboardingv0.0.3
- node-rpmv0.0.3
- node-teamv1.0.1
- node-tuningv1.0.1
- olmv0.1.3
- olm-teamv0.2.1
- openshiftv0.0.8
- openshift-developerv1.1.42
- openshift-tls-profilev0.0.2
- operator-dashboardv0.0.1
- originv0.0.2
- ote-migrationv0.0.3
- patternfly
- prodsec-skillsv1.0.0
- prow-agentv0.0.7
- rds-analyzerv0.0.2
- sessionv0.0.2
- snowflakev0.5.1
- sosreportv0.0.2