
sosreport
Analyze sosreport archives for system diagnostics and troubleshooting
- version
- 0.0.2
- GitHub stars
- 123
Install
/plugin install sosreport@ai-helpersRun it inside Claude Code after adding the marketplace with /plugin marketplace add openshift-eng/ai-helpers.
Contents
Discovered from sourceSkills 5
Logs Analysis Skill
This skill provides detailed guidance for analyzing logs from sosreport archives, including journald logs, system logs, kernel messages, and application logs.
When to Use This Skill
Use this skill when:
- Analyzing the
/sosreport:analyzecommand's log analysis phase - Investigating specific log-related errors or warnings in a sosreport
- Performing deep-dive analysis of system failures from logs
- Identifying patterns and root causes in system logs
Prerequisites
- Sosreport archive must be extracted to a working directory
- Path to the sosreport root directory must be known
- Basic understanding of Linux log structure and journald
Key Log Locations in Sosreport
Sosreports contain logs in several locations:
-
Journald logs:
sos_commands/logs/journalctl_*journalctl_--no-pager_--boot- Current boot logsjournalctl_--no-pager- All available logsjournalctl_--no-pager_--priority_err- Error priority logs
-
Traditional system logs:
var/log/messages- System-level messagesdmesg- Kernel ring buffersecure- Authentication and security logscron- Cron job logs
-
Application logs:
var/log/(varies by application)httpd/- Apache logsnginx/- Nginx logsaudit/audit.log- SELinux audit logs
Implementation Steps
Step 1: Identify Available Log Sources
-
Check for journald logs:
ls -la sos_commands/logs/journalctl_* 2>/dev/null || echo "No journald logs found" -
Check for traditional system logs:
ls -la var/log/{messages,dmesg,secure} 2>/dev/null || echo "No traditional logs found" -
Identify application-specific logs:
find var/log/ -type f -name "*.log" 2>/dev/null | head -20
Step 2: Analyze Journald Logs
-
Parse journalctl output for error patterns:
# Look for common error indicators grep -iE "(error|failed|failure|critical|panic|segfault|oom)" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -100 -
Identify OOM (Out of Memory) killer events:
grep -i "out of memory\|oom.*kill" sos_commands/logs/journalctl_--no-pager 2>/dev/null -
Find kernel panics:
grep -i "kernel panic\|bug:\|oops:" sos_commands/logs/journalctl_--no-pager 2>/dev/null -
Check for segmentation faults:
grep -i "segfault\|sigsegv\|core dump" sos_commands/logs/journalctl_--no-pager 2>/dev/null -
Extract service failures:
grep -i "failed to start\|failed with result" sos_commands/logs/journalctl_--no-pager 2>/dev/null
Step 3: Analyze System Logs (var/log)
-
Check messages for errors:
# If file exists and is readable if [ -f var/log/messages ]; then grep -iE "(error|failed|failure|critical)" var/log/messages | tail -100 fi -
Check dmesg for hardware issues:
if [ -f var/log/dmesg ]; then grep -iE "(error|fail|warning|i/o error|bad sector)" var/log/dmesg fi -
Analyze authentication logs:
if [ -f var/log/secure ]; then grep -iE "(failed|failure|invalid|denied)" var/log/secure | tail -50 fi
Step 4: Count and Categorize Errors
-
Count errors by severity:
# Critical errors grep -ic "critical\|panic\|fatal" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0" # Errors grep -ic "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0" # Warnings grep -ic "warning\|warn" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0" -
Find most frequent error messages:
grep -iE "(error|failed)" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \ sed 's/^.*\]: //' | \ sort | uniq -c | sort -rn | head -10 -
Extract timestamps for error timeline:
# Get first and last error timestamps grep -i "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \ head -1 | awk '{print $1, $2, $3}' grep -i "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \ tail -1 | awk '{print $1, $2, $3}'
Step 5: Analyze Application-Specific Logs
-
Identify application logs:
find var/log/ -type f \( -name "*.log" -o -name "*_log" \) 2>/dev/null -
Check for stack traces and exceptions:
# Python tracebacks grep -A 10 "Traceback (most recent call last)" var/log/*.log 2>/dev/null | head -50 # Java exceptions grep -B 2 -A 10 "Exception\|Error:" var/log/*.log 2>/dev/null | head -50 -
Look for common application errors:
# Database connection errors grep -i "connection.*refused\|connection.*timeout\|database.*error" var/log/*.log 2>/dev/null # HTTP/API errors grep -E "HTTP [45][0-9]{2}|status.*[45][0-9]{2}" var/log/*.log 2>/dev/null | head -20
Step 6: Generate Log Analysis Summary
Create a structured summary with the following information:
-
Error Statistics:
- Total critical errors
- Total errors
- Total warnings
- Time range of errors (first to last)
-
Critical Findings:
- Kernel panics (with timestamps)
- OOM killer events (with victim processes)
- Segmentation faults (with process names)
- Service failures (with service names)
-
Top Error Messages (sorted by frequency):
- Error message
- Count
- First occurrence timestamp
- Affected component/service
-
Application-Specific Issues:
- Stack traces found
- Database errors
- Network/connectivity errors
- Authentication failures
-
Log File Locations:
- Provide paths to specific log files for manual investigation
- Indicate which logs contain the most relevant information
Error Handling
-
Missing log files:
- If journalctl logs are missing, fall back to var/log/* files
- If traditional logs are missing, document this in the summary
- Some sosreports may have limited logs due to collection parameters
-
Large log files:
- For files larger than 100MB, sample the beginning and end
- Use
head -n 10000andtail -n 10000to avoid memory issues - Inform user that analysis is based on sampling
-
Compressed logs:
- Check for
.gzfiles invar/log/ - Use
zgrepinstead ofgrepfor compressed files - Example:
zgrep -i "error" var/log/messages*.gz
- Check for
-
Binary log formats:
- Some logs may be in binary format (e.g., journald binary logs)
- Rely on
sos_commands/logs/journalctl_*text outputs - Do not attempt to parse binary files directly
Output Format
The log analysis should produce:
LOG ANALYSIS SUMMARY
====================
Time Range: {first_log_entry} to {last_log_entry}
ERROR STATISTICS
----------------
Critical: {count}
Errors: {count}
Warnings: {count}
CRITICAL FINDINGS
-----------------
Kernel Panics: {count}
- {timestamp}: {panic_message}
OOM Killer Events: {count}
- {timestamp}: Killed {process_name} (PID: {pid})
Segmentation Faults: {count}
- {timestamp}: {process_name} segfaulted
Service Failures: {count}
- {service_name}: {failure_reason}
TOP ERROR MESSAGES
------------------
1. [{count}x] {error_message}
First seen: {timestamp}
Component: {component}
2. [{count}x] {error_message}
First seen: {timestamp}
Component: {component}
APPLICATION ERRORS
------------------
Stack Traces: {count} found in {log_files}
Database Errors: {count}
Network Errors: {count}
Auth Failures: {count}
LOG FILES FOR INVESTIGATION
---------------------------
- Primary: {sosreport_path}/sos_commands/logs/journalctl_--no-pager
- System: {sosreport_path}/var/log/messages
- Kernel: {sosreport_path}/var/log/dmesg
- Security: {sosreport_path}/var/log/secure
- Application: {sosreport_path}/var/log/{app_specific}
RECOMMENDATIONS
---------------
1. {actionable_recommendation_based_on_findings}
2. {actionable_recommendation_based_on_findings}
Examples
Example 1: OOM Killer Analysis
# Detect OOM events
grep -B 5 -A 15 "Out of memory" sos_commands/logs/journalctl_--no-pager
# Output interpretation:
# - Which process was killed
# - Memory state at the time
# - What triggered the OOM
Example 2: Service Failure Pattern
# Find failed services
grep "failed to start\|Failed with result" sos_commands/logs/journalctl_--no-pager | \
awk -F'[][]' '{print $2}' | sort | uniq -c | sort -rn
# This shows which services failed most frequently
Example 3: Timeline of Errors
# Create error timeline
grep -i "error\|fail" sos_commands/logs/journalctl_--no-pager | \
awk '{print $1, $2, $3}' | sort | uniq -c
# Shows error frequency over time
Tips for Effective Analysis
- Start with critical errors: Focus on panics, OOMs, and segfaults first
- Look for patterns: Repeated errors often indicate systemic issues
- Check timestamps: Correlate errors with the reported incident time
- Consider context: Read surrounding log lines for context
- Cross-reference: Correlate log findings with resource analysis
- Check all log sources: Check both journald and traditional logs, as some events may only appear in one
- Document findings: Note file paths and line numbers for reference
Common Log Patterns to Look For
- OOM Killer: "Out of memory: Kill process" → Memory pressure issue
- Segfault: "segfault at" → Application crash, possible bug
- I/O Error: "I/O error" in dmesg → Hardware or filesystem issue
- Connection Refused: "Connection refused" → Service not running or firewall
- Permission Denied: "Permission denied" → SELinux, file permissions, or ACL issue
- Timeout: "timeout" → Network or resource contention
- Failed to start: "Failed to start" → Service configuration or dependency issue
See Also
- Resource Analysis Skill: For correlating log errors with resource constraints
- System Configuration Analysis Skill: For investigating service failures
- Network Analysis Skill: For investigating connectivity errors
Network Analysis Skill
This skill provides detailed guidance for analyzing network configuration and connectivity from sosreport archives, including interfaces, routing, firewall rules, and DNS configuration.
When to Use This Skill
Use this skill when:
- Analyzing the
/sosreport:analyzecommand's network analysis phase - Investigating network connectivity issues
- Diagnosing firewall or routing problems
- Verifying network configuration
Prerequisites
- Sosreport archive must be extracted to a working directory
- Path to the sosreport root directory must be known
- Understanding of Linux networking concepts
Key Network Data Locations in Sosreport
-
Network Interfaces:
sos_commands/networking/ip_-o_addr- IP addressessos_commands/networking/ip_link- Link statussos_commands/networking/ip_-s_link- Link statistics with errorsetc/sysconfig/network-scripts/- Network configuration files (RHEL)
-
Routing:
sos_commands/networking/ip_route- Routing tablesos_commands/networking/ip_-6_route- IPv6 routing tableproc/net/route- Kernel routing table
-
Network Connections:
sos_commands/networking/netstat_-neopa- Active connectionssos_commands/networking/ss_-tupna- Socket statisticsproc/net/tcp- TCP connectionsproc/net/udp- UDP connections
-
Firewall:
sos_commands/firewalld/- Firewalld configurationsos_commands/iptables/iptables_-vnxL- iptables rulessos_commands/nftables/- nftables configuration
-
DNS and Resolution:
etc/resolv.conf- DNS serversetc/hosts- Static hostname mappingsetc/nsswitch.conf- Name resolution order
-
Network Services:
sos_commands/networking/networkmanager_info- NetworkManager statussystemctl status NetworkManageroutput
Implementation Steps
Step 1: Analyze Network Interfaces
-
List all network interfaces:
if [ -f sos_commands/networking/ip_-o_addr ]; then cat sos_commands/networking/ip_-o_addr fi -
Check interface states:
if [ -f sos_commands/networking/ip_link ]; then # Look for interface states (UP/DOWN) grep -E "^[0-9]+:" sos_commands/networking/ip_link fi -
Parse interface information:
- Interface name (eth0, ens192, etc.)
- State (UP/DOWN)
- IP addresses (IPv4 and IPv6)
- MAC address
- MTU size
-
Check for interface errors:
if [ -f sos_commands/networking/ip_-s_link ]; then # Look for RX/TX errors, drops, overruns cat sos_commands/networking/ip_-s_link fi -
Identify interface issues:
- Interfaces with no IP address (when expected)
- Interfaces in DOWN state (when should be UP)
- High error counts (RX/TX errors, drops)
- Duplicate IP addresses
- MTU mismatches
Step 2: Analyze Routing Configuration
-
Check default route:
if [ -f sos_commands/networking/ip_route ]; then grep "^default" sos_commands/networking/ip_route || echo "No default route found" fi -
Review routing table:
if [ -f sos_commands/networking/ip_route ]; then cat sos_commands/networking/ip_route fi -
Check IPv6 routing:
if [ -f sos_commands/networking/ip_-6_route ]; then cat sos_commands/networking/ip_-6_route fi -
Identify routing issues:
- Missing default route
- Multiple default routes (conflicting)
- Incorrect gateway addresses
- Route to nowhere (unreachable gateway)
Step 3: Analyze Network Connectivity
-
Check active connections:
if [ -f sos_commands/networking/netstat_-neopa ]; then cat sos_commands/networking/netstat_-neopa elif [ -f sos_commands/networking/ss_-tupna ]; then cat sos_commands/networking/ss_-tupna fi -
Count connections by state:
# Count TCP connection states if [ -f sos_commands/networking/netstat_-neopa ]; then grep "^tcp" sos_commands/networking/netstat_-neopa | awk '{print $6}' | sort | uniq -c fi -
Find listening services:
# Show what's listening on which ports if [ -f sos_commands/networking/netstat_-neopa ]; then grep "LISTEN" sos_commands/networking/netstat_-neopa fi -
Check for connection issues:
- Excessive TIME_WAIT connections
- Many connections in SYN_SENT (connection attempts failing)
- High number of CLOSE_WAIT (application not closing)
- Port conflicts (multiple services on same port)
Step 4: Analyze Firewall Configuration
-
Check if firewalld is active:
if [ -d sos_commands/firewalld ]; then # Firewalld is present if [ -f sos_commands/firewalld/firewall-cmd_--list-all-zones ]; then cat sos_commands/firewalld/firewall-cmd_--list-all-zones fi fi -
Review iptables rules:
if [ -f sos_commands/iptables/iptables_-vnxL ]; then cat sos_commands/iptables/iptables_-vnxL fi -
Check firewall zones and rules:
- Active zones
- Allowed services
- Allowed ports
- Rich rules
- Drop/reject policies
-
Identify firewall issues:
- Required ports blocked
- Overly permissive rules (any any accept)
- Conflicting rules
- Missing rules for services
Step 5: Analyze DNS Configuration
-
Check DNS servers:
if [ -f etc/resolv.conf ]; then cat etc/resolv.conf fi -
Review /etc/hosts:
if [ -f etc/hosts ]; then # Show non-comment, non-empty lines grep -v "^#\|^$" etc/hosts fi -
Check hostname resolution:
# Check hostname if [ -f hostname ]; then cat hostname fi # Check FQDN if [ -f etc/hostname ]; then cat etc/hostname fi -
Verify nsswitch configuration:
if [ -f etc/nsswitch.conf ]; then grep "^hosts:" etc/nsswitch.conf fi -
Identify DNS issues:
- No DNS servers configured
- Unreachable DNS servers (check connectivity in logs)
- Incorrect search domains
- Hostname resolution failures in logs
Step 6: Check for Network Errors in Logs
-
Look for network-related errors:
# Connection refused errors grep -i "connection refused" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 # Timeout errors grep -i "timeout\|timed out" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 # Network unreachable grep -i "network.*unreachable\|no route to host" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 # DNS resolution failures grep -i "could not resolve\|dns.*fail\|name resolution" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 -
Check for link state changes:
grep -i "link.*up\|link.*down\|carrier.*lost" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 -
Look for network device errors:
grep -i "network.*error\|eth[0-9].*error\|transmit.*error" var/log/dmesg 2>/dev/null
Step 7: Generate Network Analysis Summary
Create a structured summary with the following sections:
-
Interface Summary:
- List of all interfaces with status
- IP addresses assigned
- Interface errors/drops
- Link speeds and duplex settings
-
Routing Summary:
- Default gateway
- Number of routes
- Any routing anomalies
-
Connectivity Summary:
- Active connection count by state
- Listening services and ports
- Connection issues detected
-
Firewall Summary:
- Firewall type (firewalld/iptables/nftables)
- Active zones (if firewalld)
- Key allowed services/ports
- Potential blocking rules
-
DNS Summary:
- DNS servers configured
- Search domains
- Hostname configuration
- DNS resolution issues
-
Network Issues:
- Critical network problems
- Warnings and recommendations
- Evidence from logs
Error Handling
-
Missing network files:
- Different sosreport versions may have different file names
- Fall back to alternative files (netstat vs ss)
- Document missing data in summary
-
Multiple network configurations:
- System may use NetworkManager, systemd-networkd, or traditional ifcfg
- Identify which is in use and analyze accordingly
-
IPv6 presence:
- Check if IPv6 is enabled
- Analyze IPv6 configuration if present
- Note if IPv6 is disabled when expected
Output Format
The network analysis should produce:
NETWORK CONFIGURATION SUMMARY
==============================
NETWORK INTERFACES
------------------
Interface: {name}
State: {UP|DOWN}
IP Addresses: {ipv4}, {ipv6}
MAC: {mac_address}
MTU: {mtu}
RX Errors: {rx_errors} packets, {rx_dropped} dropped
TX Errors: {tx_errors} packets, {tx_dropped} dropped
Status: {OK|WARNING|CRITICAL}
ROUTING
-------
Default Gateway: {gateway_ip} via {interface}
Total Routes: {count}
Key Routes:
{destination} via {gateway} dev {interface}
Status: {OK|WARNING|CRITICAL}
Issues:
- {routing_issue_description}
CONNECTIVITY
------------
Total Active Connections: {count}
Connections by State:
ESTABLISHED: {count}
TIME_WAIT: {count}
CLOSE_WAIT: {count}
SYN_SENT: {count}
Listening Services:
{port}/{protocol} - {service_name} (PID {pid})
Status: {OK|WARNING|CRITICAL}
Issues:
- {connectivity_issue_description}
FIREWALL
--------
Type: {firewalld|iptables|nftables|none}
Default Zone: {zone_name} (if firewalld)
Allowed Services: {service1}, {service2}, ...
Allowed Ports: {port1/protocol}, {port2/protocol}, ...
Active Rules Count: {count}
Status: {OK|WARNING|CRITICAL}
Potential Issues:
- {firewall_issue_description}
DNS CONFIGURATION
-----------------
DNS Servers: {dns1}, {dns2}, {dns3}
Search Domains: {domain1}, {domain2}
Hostname: {hostname}
FQDN: {fqdn}
Status: {OK|WARNING|CRITICAL}
Issues:
- {dns_issue_description}
NETWORK ERRORS FROM LOGS
------------------------
Connection Refused: {count} occurrences
Timeouts: {count} occurrences
DNS Failures: {count} occurrences
Link State Changes: {count} occurrences
Recent Network Errors:
{timestamp}: {error_message}
CRITICAL NETWORK ISSUES
-----------------------
{severity}: {issue_description}
Evidence: {file_path_or_log_excerpt}
Impact: {impact_description}
Recommendation: {remediation_action}
RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}
DATA SOURCES
------------
- Interfaces: {sosreport_path}/sos_commands/networking/ip_-o_addr
- Routes: {sosreport_path}/sos_commands/networking/ip_route
- Connections: {sosreport_path}/sos_commands/networking/netstat_-neopa
- Firewall: {sosreport_path}/sos_commands/firewalld/
- DNS: {sosreport_path}/etc/resolv.conf
Examples
Example 1: Interface Analysis
# Check interface IP addresses
$ cat sos_commands/networking/ip_-o_addr
1: lo inet 127.0.0.1/8 scope host lo
2: eth0 inet 192.168.1.100/24 brd 192.168.1.255 scope global eth0
2: eth0 inet6 fe80::a00:27ff:fe4e:66a1/64 scope link
# Check for errors
$ cat sos_commands/networking/ip_-s_link | grep -A 4 "eth0"
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
RX: bytes packets errors dropped overrun mcast
15234567 98234 0 0 0 123
TX: bytes packets errors dropped carrier collsns
8765432 54321 15 0 0 0
# Interpretation: eth0 has 15 TX errors - investigate cable/switch
Example 2: Firewall Rule Analysis
# Check firewalld active zone
$ grep -A 20 "public" sos_commands/firewalld/firewall-cmd_--list-all-zones
public (active)
target: default
services: ssh dhcpv6-client http https
ports: 8080/tcp 9090/tcp
...
# Interpretation: HTTP/HTTPS allowed, custom ports 8080 and 9090 open
Example 3: Connection State Issues
# Count connection states
$ grep "^tcp" sos_commands/networking/netstat_-neopa | awk '{print $6}' | sort | uniq -c
234 ESTABLISHED
1523 TIME_WAIT
12 CLOSE_WAIT
5 SYN_SENT
# Interpretation:
# - Excessive TIME_WAIT (normal after closing connections)
# - CLOSE_WAIT suggests application not properly closing sockets
# - SYN_SENT indicates outbound connection attempts failing
Tips for Effective Analysis
- Check interface consistency: Ensure IP addresses match expected configuration
- Verify gateway reachability: Default gateway should be on the same subnet
- Look for asymmetric routing: Packets in/out may take different paths
- Check MTU settings: MTU mismatches can cause packet fragmentation issues
- Correlate with logs: Network errors in logs often explain configuration issues
- Consider network topology: Understand expected network layout
- Check both IPv4 and IPv6: Be sure to check IPv6 if it's in use
Common Network Patterns and Issues
- No default route: "Network unreachable" errors, can't reach internet
- Interface down: "Network is down" errors, no connectivity
- Duplicate IP: ARP conflicts, intermittent connectivity
- Firewall blocking: "Connection refused/timeout" for specific ports
- DNS failure: Can't resolve hostnames, but IP connectivity works
- Port exhaustion: Too many TIME_WAIT connections, can't create new connections
- MTU issues: Large packets fail, small packets work (PMTUD failure)
Network Issue Severity Classification
| Issue Type | Severity | Impact |
|---|---|---|
| No network interface | Critical | Complete loss of connectivity |
| No default route | Critical | No external connectivity |
| Interface errors >1% | Warning | Potential packet loss |
| Excessive TIME_WAIT | Warning | May indicate performance issue |
| Missing DNS server | Critical | Name resolution failure |
| Firewall blocking required port | High | Service unavailable |
| IPv6 autoconfiguration failure | Low | IPv6 connectivity issue |
See Also
- Logs Analysis Skill: For detailed network error log analysis
- System Configuration Analysis Skill: For network service status
- Resource Analysis Skill: For network I/O statistics
OVS Database and Flow Analysis
This skill provides detailed analysis of Open vSwitch (OVS) data collected via sosreport. It operates in four modes:
- Default mode: Full analysis - conf.db + all text files (requires ovsdb-tool, falls back to text files if not available)
- Database mode (
--db): Database only - analyze conf.db (requires ovsdb-tool) - Text files mode (
--flows-only): Text files only - no ovsdb-tool needed - Query mode (
--query): Run custom OVSDB JSON queries (requires ovsdb-tool)
When to Use This Skill
Use this skill when:
- Troubleshooting packet drops - Find drop flows being hit
- Analyzing flow performance - View top flows by packet count
- Analyzing OVS configuration from a sosreport
- Troubleshooting bridge, port, or interface issues
- Investigating DPDK vs kernelspace datapath
- Reviewing tunnel configurations (VXLAN, Geneve, GRE)
- Checking port statistics (drops, errors, traffic)
- Mapping Kubernetes pods to OVS interfaces
Prerequisites
Default mode (full analysis):
- ovsdb-tool must be installed (from openvswitch package)
- Falls back to
--flows-onlyif ovsdb-tool not found - Sosreport with
sos_commands/openvswitch/directory andconf.db
Database mode (--db):
- ovsdb-tool must be installed (from openvswitch package)
- Check:
which ovsdb-tool - Fedora/RHEL:
sudo dnf install openvswitch - Ubuntu/Debian:
sudo apt install openvswitch-common
- Check:
- Sosreport with
conf.dbfile
Text files mode (--flows-only):
- No special tools needed - works out of the box!
- Sosreport with
sos_commands/openvswitch/directory
Query mode (--query):
- ovsdb-tool must be installed
- Incompatible with
--flows-only
File Locations in Sosreport
The sosreport collects:
sosreport-hostname-date/
├── etc/openvswitch/conf.db (OVS database - for --db mode)
├── var/lib/openvswitch/conf.db (alternate location)
└── sos_commands/openvswitch/ (default mode uses these)
├── ovs-vsctl_-t_5_show (topology)
├── ovs-vsctl_-t_5_list_bridge (bridge details)
├── ovs-vsctl_-t_5_list_interface (interface details)
├── ovs-vsctl_-t_5_list_Open_vSwitch (system info)
├── ovs-ofctl_dump-flows_<bridge> (OpenFlow entries)
├── ovs-ofctl_dump-ports_<bridge> (Port statistics)
├── ovs-appctl_coverage.show (internal stats)
├── ovs-appctl_upcall.show (datapath health)
├── ovs-appctl_tnl.ports.show_-v (tunnel ports)
└── ...
Analysis Script
The analysis script is located at:
<plugin-root>/skills/ovs-db-analysis/scripts/analyze_ovs_db.py
Implementation Steps
Step 1: Locate the Analysis Script
SCRIPT_PATH=$(find ~ -name "analyze_ovs_db.py" -path "*/sosreport/skills/ovs-db-analysis/scripts/*" 2>/dev/null | head -1)
if [ -z "$SCRIPT_PATH" ]; then
echo "ERROR: analyze_ovs_db.py script not found."
exit 1
fi
Step 2: Run the Analysis
# Default: Full analysis - conf.db + text files (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date/
# Analyze from archive
python3 "$SCRIPT_PATH" /path/to/sosreport-hostname-date.tar.xz
# Database only mode (requires ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --db
# Text files only mode (no ovsdb-tool needed)
python3 "$SCRIPT_PATH" /path/to/sosreport/ --flows-only
Step 3: Custom Queries (Optional, requires --db)
For specific investigations, use raw OVSDB queries:
# Query all bridges
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Bridge", "where":[], "columns":["name","datapath_type","fail_mode"]}]'
# Query VXLAN interfaces
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[["type","==","vxlan"]], "columns":["name","options"]}]'
# Query interfaces with errors
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[], "columns":["name","error"]}]'
Analysis Output
The default mode analyzes:
| Analysis | Source File | Description |
|---|---|---|
| System Info | ovs-vsctl_-t_5_list_Open_vSwitch | OVS/DPDK version, system type, external IDs |
| Topology | ovs-vsctl_-t_5_show | Bridge overview with ports grouped by type |
| Bridge Details | ovs-vsctl_-t_5_list_bridge | Datapath type, fail mode, CT zones |
| Interfaces | ovs-vsctl_-t_5_list_interface | By type, pod interfaces with mapping |
| OpenFlow | ovs-ofctl_dump-flows_* | Flow counts, drops, top flows |
| Port Stats | ovs-ofctl_dump-ports_* | RX/TX drops and errors |
| Tunnels | ovs-appctl_tnl.ports.show_-v | Configured tunnel ports |
| Datapath Health | ovs-appctl_upcall.show | Flow table usage vs limit |
| OVS Stats | ovs-appctl_coverage.show | Internal counters (netlink, OVSDB, etc.) |
OpenFlow Analysis Features
| Feature | Description |
|---|---|
| Flow Count | Total flows per bridge (total, drop, with hits) |
| Drop Detection | Flows with actions=drop that have packet hits |
| Top Flows | Most active flows sorted by n_packets |
| Table Distribution | Flow counts per OpenFlow table |
| Port Drops | RX/TX drop counters per port |
| Port Errors | RX/TX error counters per port |
| Datapath Health | Flow table usage vs limit (from upcall.show) |
| OVS Stats | Internal statistics (netlink, OpenFlow, OVSDB transactions) |
OVN Internal Tables (Ignore Drops in These)
When analyzing drop flows in OVN-managed bridges (br-int), ignore drops in table 44 and tables 64-87 as these are internal OVN mechanics, not security policy or real packet drops.
⚠️ Note: OVN and OVS are complex systems and table mappings can change between releases. Always verify drop analysis against the specific OVN version in use. When in doubt, check the OVN source code for your release.
| Table | Name | Purpose |
|---|---|---|
| 44 | CHK_LB_OUTPUT | Loopback prevention - drops packets that would loop back (high volume, normal) |
| 64 | SAVE_INPORT | Save ingress port for later |
| 65 | LOG_TO_PHY | Logical to physical mapping |
| 66 | MAC_BINDING | MAC binding lookups |
| 67 | MAC_LOOKUP | MAC address table lookups |
| 68-69 | CHK_LB_HAIRPIN | Load balancer hairpin checks |
| 70 | CT_SNAT_HAIRPIN | Conntrack SNAT hairpin |
| 71-72 | GET/LOOKUP_FDB | FDB (forwarding DB) lookups |
| 73-74 | CHK_IN_PORT_SEC | Ingress port security |
| 75 | CHK_OUT_PORT_SEC | Egress port security |
| 76-77 | ECMP_NH | ECMP next-hop handling |
| 78 | CHK_LB_AFFINITY | Load balancer affinity |
| 79 | MAC_CACHE_USE | MAC cache miss (high volume, normal) |
| 80 | CT_ZONE_LOOKUP | Conntrack zone lookup |
| 81-83 | CT_ORIG_*_LOAD | Conntrack original tuple loading |
| 84 | FLOOD_REMOTE_CHASSIS | Remote chassis flooding |
| 85 | CT_STATE_SAVE | Conntrack state save |
| 86 | CT_ORIG_PROTO_LOAD | Conntrack protocol loading |
| 87 | GET_REMOTE_FDB | Remote FDB lookup |
Relevant drop tables for troubleshooting:
- Table 9: Ingress ACLs - actual policy drops (look for
reg0=0x8000) - Table 0: Initial classification drops
- Tables < 64: Generally meaningful drops
Common Analysis Scenarios
1. Troubleshooting Packet Drops
# Quick analysis (default mode, no ovsdb-tool)
python3 "$SCRIPT_PATH" /path/to/sosreport/
Look for:
- ACTIVE DROP FLOWS section showing flows dropping packets
- PORT STATISTICS showing ports with drops/errors
2. Checking DPDK Configuration
# Default mode shows DPDK info from text files
python3 "$SCRIPT_PATH" /path/to/sosreport/
# Or use database mode for custom queries
python3 "$SCRIPT_PATH" /path/to/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Open_vSwitch", "where":[], "columns":["dpdk_initialized","other_config"]}]'
3. Investigating Pod Connectivity
The interface analysis shows pod-to-OVS mapping:
- Interfaces with
external_idscontainingiface-id - Interface names ending with
_h(veth host side)
4. Datapath Health Check
The upcall stats show:
DATAPATH FLOW TABLE HEALTH
Current flows: 155 / 200,000 (0.1% used)
Average: 156, Max seen: 215
✓ Flow table healthy
If usage > 90%, flows are being evicted too aggressively.
Output Example
================================================================================
OVS ANALYSIS - sosreport-hostname-2024-01-15
================================================================================
Mode: Text file analysis (no ovsdb-tool required)
================================================================================
OVS SYSTEM INFORMATION
================================================================================
Field Value
------------------------- --------------------------------------------------
OVS Version "3.3.4-62.el9fdp"
DB Version "8.5.0"
System Type rhcos
System Version "4.16"
DPDK Initialized false
Datapath Types [netdev, system]
External IDs:
hostname: master2.example.com
ovn-encap-ip: 10.32.110.5
ovn-encap-type: geneve
================================================================================
OVS TOPOLOGY
================================================================================
System UUID: 7e9a3f70-86fa-4578-a849-4fd807a64a10
Total Bridges: 2
Bridge: br-ex
ports: 3
internal: br-ex
patch: patch-br-ex-to-br-int
system: nm-bond
Bridge: br-int
fail_mode: secure
datapath_type: system
ports: 12
geneve: 9 ports
internal: ovn-k8s-mp0, br-int
patch: patch-br-int-to-br-ex
================================================================================
OPENFLOW ANALYSIS
================================================================================
Bridge: br-int
----------------------------------------------------------------------
Total flows: 2,017
Flows with hits: 318
Drop flows: 150 (9 actively dropping)
⚠️ ACTIVE DROP FLOWS (9):
table=40, priority=0, packets=8,105
table=79, priority=100, packets=1,356
match: ip,reg14=0x2,metadata=0x5,dl_src=00:62:0b:ea:b5:e0
--------------------------------------------------------------------------------
DATAPATH FLOW TABLE HEALTH
--------------------------------------------------------------------------------
Current flows: 155 / 200,000 (0.1% used)
Average: 156, Max seen: 215
✓ Flow table healthy
Error Handling
| Error | Solution |
|---|---|
ovsdb-tool not found | Install openvswitch package OR use default mode (no --db) |
conf.db not found | Use default mode (analyzes text files instead) |
sos_commands/openvswitch not found | Ensure sosreport has OVS data collected |
See Also
- OVN Database Analysis - For OVN NB/SB databases
- sosreport openvswitch plugin - What sosreport collects
Resource Analysis Skill
This skill provides detailed guidance for analyzing system resource usage from sosreport archives, including memory, CPU, disk space, and process information.
When to Use This Skill
Use this skill when:
- Analyzing the
/sosreport:analyzecommand's resource analysis phase - Investigating performance issues or resource bottlenecks
- Identifying resource exhaustion problems
- Correlating resource usage with system failures
Prerequisites
- Sosreport archive must be extracted to a working directory
- Path to the sosreport root directory must be known
- Understanding of Linux resource management
Key Resource Data Locations in Sosreport
-
Memory Information:
sos_commands/memory/free- Memory usage snapshotproc/meminfo- Detailed memory statisticssos_commands/memory/swapon_-s- Swap usageproc/buddyinfo- Memory fragmentation
-
CPU Information:
sos_commands/processor/lscpu- CPU architecture and featuresproc/cpuinfo- Detailed CPU informationsos_commands/processor/turbostat- CPU frequency and power states (if available)uptime- Load averages
-
Disk Information:
sos_commands/filesys/df_-al- Filesystem usagesos_commands/block/lsblk- Block device informationsos_commands/filesys/mount- Mounted filesystemsproc/diskstats- Disk I/O statistics
-
Process Information:
sos_commands/process/ps_auxwww- Process list with detailssos_commands/process/top- Process snapshot (if available)proc/[pid]/- Per-process information
Implementation Steps
Step 1: Analyze Memory Usage
-
Parse free command output:
# Check if free output exists if [ -f sos_commands/memory/free ]; then cat sos_commands/memory/free fi -
Extract memory metrics:
# Parse /proc/meminfo for detailed stats if [ -f proc/meminfo ]; then grep -E "^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Dirty|Slab):" proc/meminfo fi -
Calculate memory usage percentage:
- Total memory = MemTotal
- Used memory = MemTotal - MemAvailable
- Usage percentage = (Used / Total) * 100
- Parse from
freeoutput or calculate frommeminfo
-
Check for memory pressure indicators:
# Look for OOM events in logs grep -i "out of memory\|oom killer" sos_commands/logs/journalctl_--no-pager 2>/dev/null # Check swap usage if [ -f sos_commands/memory/swapon_-s ]; then cat sos_commands/memory/swapon_-s fi -
Identify memory issues:
- Memory usage > 90% → Critical
- Memory usage > 80% → Warning
- Heavy swap usage (>50% swap used) → Performance issue
- OOM killer events → Critical memory exhaustion
Step 2: Analyze CPU Usage
-
Extract CPU information:
# Get CPU count and model if [ -f sos_commands/processor/lscpu ]; then grep -E "^(CPU\(s\)|Model name|Thread|Core|Socket|CPU MHz):" sos_commands/processor/lscpu fi -
Check load averages:
# Parse uptime for load averages if [ -f uptime ]; then cat uptime fi # Or from proc/loadavg if [ -f proc/loadavg ]; then cat proc/loadavg fi -
Interpret load averages:
- Load average format: 1-min, 5-min, 15-min
- Compare with CPU count from lscpu
- Load > CPU count → System overloaded
- Load >> CPU count (2x or more) → Critical overload
-
Check for CPU throttling:
# Look for thermal throttling in logs grep -i "throttl\|temperature\|thermal" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 -
Identify CPU issues:
- 1-min load > 2x CPU count → Critical
- 5-min load > CPU count → Warning
- Thermal throttling present → Hardware/cooling issue
Step 3: Analyze Disk Usage
-
Parse df output for filesystem usage:
if [ -f sos_commands/filesys/df_-al ]; then # Skip header and special filesystems, show only regular filesystems grep -v "^Filesystem\|tmpfs\|devtmpfs\|overlay" sos_commands/filesys/df_-al | grep -v "^$" fi -
Identify full or nearly-full filesystems:
# Extract filesystems with usage > 85% if [ -f sos_commands/filesys/df_-al ]; then awk 'NR>1 && $5+0 >= 85 {print $5, $6, $1}' sos_commands/filesys/df_-al | grep -v "tmpfs\|devtmpfs" fi -
Check disk I/O errors:
# Look for I/O errors in logs grep -i "i/o error\|read error\|write error\|bad sector" var/log/dmesg 2>/dev/null grep -i "i/o error\|read error\|write error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 -
Analyze block devices:
if [ -f sos_commands/block/lsblk ]; then cat sos_commands/block/lsblk fi -
Identify disk issues:
- Filesystem > 95% full → Critical
- Filesystem > 85% full → Warning
- I/O errors present → Hardware issue
- Root filesystem full → System stability risk
Step 4: Analyze Process Information
-
Parse ps output:
if [ -f sos_commands/process/ps_auxwww ]; then # Show header head -1 sos_commands/process/ps_auxwww fi -
Find top CPU consumers:
# Sort by CPU usage (column 3), show top 10 if [ -f sos_commands/process/ps_auxwww ]; then tail -n +2 sos_commands/process/ps_auxwww | sort -k3 -rn | head -10 fi -
Find top memory consumers:
# Sort by memory usage (column 4), show top 10 if [ -f sos_commands/process/ps_auxwww ]; then tail -n +2 sos_commands/process/ps_auxwww | sort -k4 -rn | head -10 fi -
Check for zombie processes:
# Look for processes in Z state if [ -f sos_commands/process/ps_auxwww ]; then grep " Z " sos_commands/process/ps_auxwww || echo "No zombie processes found" fi -
Count processes by state:
# Count processes by state (R=running, S=sleeping, D=uninterruptible, Z=zombie, T=stopped) if [ -f sos_commands/process/ps_auxwww ]; then tail -n +2 sos_commands/process/ps_auxwww | awk '{print $8}' | cut -c1 | sort | uniq -c fi -
Identify process issues:
- Zombie processes present → Parent process not reaping children
- Many processes in D state → I/O bottleneck
- Single process using >80% memory → Memory leak or expected behavior
- Many processes using high CPU → CPU contention
Step 5: Correlate Resource Usage with Issues
-
Cross-reference with logs:
- If high memory usage, check for OOM events in logs
- If high disk usage, check for disk full errors
- If high load, check for performance-related errors
-
Identify resource exhaustion patterns:
- Memory exhaustion → OOM killer → Service crashes
- Disk full → Write failures → Application errors
- CPU overload → Timeouts → Request failures
-
Build timeline:
- When did resource issues start?
- Correlate with log timestamps
- Identify triggering event when log entries or metric changes indicate a clear cause
Step 6: Generate Resource Analysis Summary
Create a structured summary with the following sections:
-
Memory Summary:
- Total memory
- Used memory (GB and %)
- Available memory
- Swap usage (GB and %)
- Memory pressure indicators (OOM events)
-
CPU Summary:
- CPU count and model
- Load averages (1-min, 5-min, 15-min)
- Load per CPU
- CPU issues (throttling, overload)
-
Disk Summary:
- Filesystems and usage percentages
- Full or nearly-full filesystems
- I/O errors count
- Most full filesystem
-
Process Summary:
- Total process count
- Top CPU consumers (top 5)
- Top memory consumers (top 5)
- Zombie process count
- Processes in uninterruptible sleep (D state)
-
Critical Resource Issues:
- List issues by severity
- Provide evidence (file paths, metrics)
- Suggest remediation
Error Handling
-
Missing resource files:
- If
freeis missing, parseproc/meminfodirectly - If
psis missing, checkproc/for process information - Document missing data in summary
- If
-
Parsing errors:
- Handle different output formats (free -h vs free -m)
- Account for locale differences in number formats
- Validate data before calculations
-
Incomplete data:
- Some sosreports may not include all resource files
- Indicate which metrics are unavailable
- Work with available data only
Output Format
The resource analysis should produce:
RESOURCE USAGE SUMMARY
======================
MEMORY
------
Total: {total_gb} GB
Used: {used_gb} GB ({used_pct}%)
Available: {available_gb} GB ({available_pct}%)
Buffers: {buffers_gb} GB
Cached: {cached_gb} GB
Swap Total: {swap_total_gb} GB
Swap Used: {swap_used_gb} GB ({swap_used_pct}%)
Status: {OK|WARNING|CRITICAL}
Issues:
- {memory_issue_description}
CPU
---
Model: {cpu_model}
CPU Count: {cpu_count}
Threads/Core: {threads_per_core}
Load Averages: {load_1m}, {load_5m}, {load_15m}
Load per CPU: {load_1m_per_cpu}, {load_5m_per_cpu}, {load_15m_per_cpu}
Status: {OK|WARNING|CRITICAL}
Issues:
- {cpu_issue_description}
DISK USAGE
----------
Filesystem Size Used Avail Use% Mounted on
{filesystem} {size} {used} {avail} {pct}% {mount}
Nearly Full Filesystems (>85%):
- {mount}: {pct}% full ({available} available)
I/O Errors: {count} errors found in logs
Status: {OK|WARNING|CRITICAL}
Issues:
- {disk_issue_description}
PROCESSES
---------
Total Processes: {total}
Running: {running}
Sleeping: {sleeping}
Zombie: {zombie}
Uninterruptible: {uninterruptible}
Top CPU Consumers:
1. {process_name} (PID {pid}): {cpu}% CPU, {mem}% MEM
2. {process_name} (PID {pid}): {cpu}% CPU, {mem}% MEM
3. {process_name} (PID {pid}): {cpu}% CPU, {mem}% MEM
Top Memory Consumers:
1. {process_name} (PID {pid}): {mem}% MEM, {cpu}% CPU
2. {process_name} (PID {pid}): {mem}% MEM, {cpu}% CPU
3. {process_name} (PID {pid}): {mem}% MEM, {cpu}% CPU
Status: {OK|WARNING|CRITICAL}
Issues:
- {process_issue_description}
CRITICAL RESOURCE ISSUES
------------------------
{severity}: {issue_description}
Evidence: {file_path}
Impact: {impact_description}
Recommendation: {remediation_action}
RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}
DATA SOURCES
------------
- Memory: {sosreport_path}/sos_commands/memory/free
- Memory: {sosreport_path}/proc/meminfo
- CPU: {sosreport_path}/sos_commands/processor/lscpu
- Load: {sosreport_path}/uptime
- Disk: {sosreport_path}/sos_commands/filesys/df_-al
- Processes: {sosreport_path}/sos_commands/process/ps_auxwww
Examples
Example 1: Memory Analysis
# Parse free command output
$ cat sos_commands/memory/free
total used free shared buff/cache available
Mem: 16277396 8123456 2145678 123456 6008262 7654321
Swap: 8388604 512000 7876604
# Interpretation:
# - Total RAM: ~16 GB
# - Used: ~8 GB (50%)
# - Available: ~7.6 GB (47%)
# - Swap used: ~500 MB (6%)
# Status: OK - healthy memory usage
Example 2: Disk Full Detection
# Find filesystems > 85% full
$ awk 'NR>1 && $5+0 >= 85' sos_commands/filesys/df_-al
/dev/sda1 50G 45G 5G 90% /
/dev/sdb1 100G 96G 4G 96% /var/log
# Critical: Root filesystem at 90%, /var/log at 96%
# Action required: Clean up disk space
Example 3: High Load Investigation
# Check load averages
$ cat uptime
14:23:45 up 10 days, 3:42, 2 users, load average: 8.45, 7.23, 6.12
# With lscpu showing 4 CPUs:
# Load per CPU: 2.1, 1.8, 1.5
# System is overloaded (load > 2x CPU count)
Tips for Effective Analysis
- Context matters: High resource usage isn't always bad - consider the workload
- Look for trends: Compare 1-min, 5-min, 15-min loads to see if issues are growing
- Correlate metrics: High load + high memory + disk full = multiple issues
- Check ratios: Usage percentages are more meaningful than absolute values
- Validate findings: Cross-reference with log analysis for confirmation
- Consider capacity: Does the system have enough CPU, memory, and disk for its workload?
Common Resource Patterns
- Memory leak: Steadily increasing memory usage, eventual OOM
- Disk full: Application writes failing, log rotation issues
- CPU spike: Load average spike, potentially from runaway process
- I/O bottleneck: High load but low CPU usage, many D-state processes
- Swap thrashing: High swap usage, poor performance
- Zombie accumulation: Parent process bug not reaping children
Severity Classification
| Metric | OK | Warning | Critical |
|---|---|---|---|
| Memory Usage | < 80% | 80-90% | > 90% |
| Swap Usage | < 20% | 20-50% | > 50% |
| Disk Usage | < 85% | 85-95% | > 95% |
| Load (per CPU) | < 1.0 | 1.0-2.0 | > 2.0 |
| Root FS Usage | < 80% | 80-90% | > 90% |
See Also
- Logs Analysis Skill: For finding resource-related errors in logs
- System Configuration Analysis Skill: For investigating service resource limits
- Network Analysis Skill: For network-related performance issues
System Configuration Analysis Skill
This skill provides detailed guidance for analyzing system configuration from sosreport archives, including OS information, installed packages, systemd services, and SELinux/AppArmor settings.
When to Use This Skill
Use this skill when:
- Analyzing the
/sosreport:analyzecommand's system configuration phase - Investigating service failures or misconfigurations
- Verifying package versions and updates
- Checking security policy settings (SELinux/AppArmor)
- Understanding system state and configuration
Prerequisites
- Sosreport archive must be extracted to a working directory
- Path to the sosreport root directory must be known
- Understanding of Linux system administration
Key Configuration Data Locations in Sosreport
-
System Information:
uname- Kernel versionetc/os-release- OS distribution and versionuptime- System uptimeproc/uptime- Uptime in secondssos_commands/release/- Release information
-
Package Information:
installed-rpms- RPM packages (RHEL/Fedora/CentOS)installed-debs- DEB packages (Debian/Ubuntu)sos_commands/yum/- Yum/DNF informationsos_commands/rpm/- RPM database queries
-
Service Status:
sos_commands/systemd/systemctl_list-units- All unitssos_commands/systemd/systemctl_list-units_--failed- Failed unitssos_commands/systemd/systemctl_status_--all- Detailed service statussos_commands/systemd/systemctl_list-unit-files- Unit files
-
SELinux:
sos_commands/selinux/sestatus- SELinux statussos_commands/selinux/getenforce- Current enforcement modesos_commands/selinux/selinux-policy- Policy informationvar/log/audit/audit.log- SELinux denials
-
AppArmor (if applicable):
sos_commands/apparmor/- AppArmor configurationetc/apparmor.d/- AppArmor profiles
-
System Configuration Files:
etc/- System-wide configurationetc/sysctl.conforetc/sysctl.d/- Kernel parametersetc/security/limits.conf- Resource limits
Implementation Steps
Step 1: Analyze System Information
-
Check OS version and distribution:
if [ -f etc/os-release ]; then cat etc/os-release fi -
Get kernel version:
if [ -f uname ]; then cat uname elif [ -f proc/version ]; then cat proc/version fi -
Check system uptime:
if [ -f uptime ]; then cat uptime elif [ -f proc/uptime ]; then # Parse uptime from proc/uptime (seconds) awk '{printf "%.2f days\n", $1/86400}' proc/uptime fi -
Extract key system details:
- OS name and version
- Kernel version
- System architecture (x86_64, aarch64, etc.)
- Uptime (days)
-
Check for outdated kernel or OS:
- Compare kernel version with current stable
- Note if system hasn't been rebooted in a very long time (>365 days)
- Identify if OS version is EOL
Step 2: Analyze Installed Packages
-
List installed packages:
# For RPM-based systems if [ -f installed-rpms ]; then cat installed-rpms fi # For DEB-based systems if [ -f installed-debs ]; then cat installed-debs fi -
Extract key package versions:
# Important system packages grep -E "^(kernel|systemd|glibc|openssh|openssl)" installed-rpms 2>/dev/null # Or use awk to parse package name and version awk '{print $1}' installed-rpms | head -20 -
Check for known problematic versions:
- Security vulnerabilities (if known CVEs)
- Buggy package versions
- Compatibility issues
-
Identify package manager issues:
# Check yum/dnf logs for errors if [ -d sos_commands/yum ]; then grep -i "error\|fail" sos_commands/yum/* 2>/dev/null fi -
Count packages and categorize:
- Total packages installed
- Key package versions (kernel, systemd, glibc, etc.)
- Recently updated packages (if timestamps available)
Step 3: Analyze Service Status
-
List all systemd units:
if [ -f sos_commands/systemd/systemctl_list-units ]; then cat sos_commands/systemd/systemctl_list-units fi -
Identify failed services:
if [ -f sos_commands/systemd/systemctl_list-units_--failed ]; then cat sos_commands/systemd/systemctl_list-units_--failed elif [ -f sos_commands/systemd/systemctl_list-units ]; then grep "failed" sos_commands/systemd/systemctl_list-units fi -
Check service details:
# Parse detailed status for failed services if [ -f sos_commands/systemd/systemctl_status_--all ]; then # Extract service names and their status grep -E "●|Active:" sos_commands/systemd/systemctl_status_--all | head -50 fi -
Count services by state:
# Count running, failed, inactive services if [ -f sos_commands/systemd/systemctl_list-units ]; then awk '{print $4}' sos_commands/systemd/systemctl_list-units | sort | uniq -c fi -
Identify critical service failures:
- System services (systemd-*, dbus, NetworkManager)
- Application services (httpd, nginx, postgresql, etc.)
- Custom services
-
Extract failure reasons from logs:
# For each failed service, find related log entries grep -i "failed to start\|service.*failed" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20
Step 4: Analyze SELinux Configuration
-
Check SELinux status:
if [ -f sos_commands/selinux/sestatus ]; then cat sos_commands/selinux/sestatus fi -
Get SELinux mode:
if [ -f sos_commands/selinux/getenforce ]; then cat sos_commands/selinux/getenforce fi -
Check for SELinux denials:
# Look for AVC denials in audit log if [ -f var/log/audit/audit.log ]; then grep "avc.*denied" var/log/audit/audit.log | head -50 fi # Or in journald logs grep -i "selinux.*denied\|avc.*denied" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20 -
Parse denial information:
- Extract denied operations (read, write, execute, etc.)
- Identify source and target contexts
- Note which services are affected
-
Check for SELinux booleans:
if [ -f sos_commands/selinux/getsebool_-a ]; then cat sos_commands/selinux/getsebool_-a fi -
Identify SELinux issues:
- SELinux in permissive mode (may hide errors)
- SELinux disabled (security concern)
- Frequent AVC denials (policy may need adjustment)
- Context mismatches
Step 5: Check System Configuration
-
Review kernel parameters:
# Check sysctl settings if [ -f sos_commands/kernel/sysctl_-a ]; then cat sos_commands/kernel/sysctl_-a elif [ -d etc/sysctl.d ]; then cat etc/sysctl.d/*.conf 2>/dev/null fi -
Check resource limits:
if [ -f etc/security/limits.conf ]; then grep -v "^#\|^$" etc/security/limits.conf fi # Check limits.d directory if [ -d etc/security/limits.d ]; then cat etc/security/limits.d/*.conf 2>/dev/null fi -
Review boot parameters:
if [ -f sos_commands/boot/grub2-editenv_list ]; then cat sos_commands/boot/grub2-editenv_list elif [ -f proc/cmdline ]; then cat proc/cmdline fi -
Check systemd configuration:
# Look for systemd configuration overrides if [ -d etc/systemd/system ]; then find etc/systemd/system -name "*.conf" 2>/dev/null fi
Step 6: Generate System Configuration Summary
Create a structured summary with the following sections:
-
System Information:
- OS name and version
- Kernel version
- Architecture
- System uptime
- Last boot time
-
Package Summary:
- Total packages installed
- Key package versions (kernel, systemd, glibc, openssl, openssh)
- Known problematic packages (if any)
- Package manager issues
-
Service Status:
- Total services
- Running services count
- Failed services count
- List of failed services with reasons
- Critical service status
-
SELinux/AppArmor:
- SELinux status (enabled/disabled)
- SELinux mode (enforcing/permissive)
- Denial count
- Top denied operations
- Policy recommendations
-
Configuration Issues:
- Kernel parameter anomalies
- Resource limit issues
- Boot parameter problems
- Configuration file errors
Error Handling
-
Missing configuration files:
- Different distributions have different file locations
- Some files may not be collected based on sosreport options
- Document missing data in summary
-
Package manager variations:
- Handle both RPM and DEB systems
- Account for different package naming conventions
- Support multiple package managers (yum, dnf, apt)
-
SELinux vs AppArmor:
- Check which MAC system is in use
- Analyze accordingly
- Note if both or neither are present
-
Systemd vs init:
- Older systems may use init instead of systemd
- Check for both service management systems
- Adapt analysis based on what's present
Output Format
The system configuration analysis should produce:
SYSTEM CONFIGURATION SUMMARY
============================
SYSTEM INFORMATION
------------------
OS: {os_name} {os_version}
Kernel: {kernel_version}
Architecture: {arch}
Uptime: {uptime_days} days ({last_boot_time})
Status: {OK|WARNING|CRITICAL}
Notes:
- {system_info_note}
INSTALLED PACKAGES
------------------
Total Packages: {count}
Key Package Versions:
kernel: {version}
systemd: {version}
glibc: {version}
openssl: {version}
openssh-server: {version}
Status: {OK|WARNING|CRITICAL}
Issues:
- {package_issue_description}
SYSTEMD SERVICES
----------------
Total Units: {total}
Active: {active_count}
Failed: {failed_count}
Inactive: {inactive_count}
Failed Services:
● {service_name}.service - {description}
Reason: {failure_reason}
Last Failed: {timestamp}
● {service_name}.service - {description}
Reason: {failure_reason}
Last Failed: {timestamp}
Status: {OK|WARNING|CRITICAL}
Recommendations:
- {service_recommendation}
SELINUX
-------
Status: {enabled|disabled}
Mode: {enforcing|permissive|disabled}
Policy: {policy_name}
AVC Denials: {count} denials found
Top Denied Operations:
[{count}x] {operation} on {target} by {source}
[{count}x] {operation} on {target} by {source}
SELinux Booleans: {count} custom settings
Status: {OK|WARNING|CRITICAL}
Issues:
- {selinux_issue_description}
Recommendations:
- {selinux_recommendation}
KERNEL PARAMETERS
-----------------
Key sysctl Settings:
vm.swappiness: {value}
net.ipv4.ip_forward: {value}
kernel.panic: {value}
Custom Parameters: {count} custom settings found
Status: {OK|WARNING|CRITICAL}
Notes:
- {kernel_param_note}
RESOURCE LIMITS
---------------
Custom Limits Found: {count}
{user_or_group} {type} {item} {value}
Status: {OK|WARNING}
Notes:
- {limits_note}
CRITICAL CONFIGURATION ISSUES
-----------------------------
{severity}: {issue_description}
Evidence: {file_path}
Impact: {impact_description}
Recommendation: {remediation_action}
RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}
DATA SOURCES
------------
- OS Info: {sosreport_path}/etc/os-release
- Kernel: {sosreport_path}/uname
- Packages: {sosreport_path}/installed-rpms
- Services: {sosreport_path}/sos_commands/systemd/systemctl_list-units
- SELinux: {sosreport_path}/sos_commands/selinux/sestatus
- Audit Log: {sosreport_path}/var/log/audit/audit.log
Examples
Example 1: Failed Service Analysis
# List failed services
$ cat sos_commands/systemd/systemctl_list-units_--failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● httpd.service loaded failed failed Apache Web Server
● postgresql.service loaded failed failed PostgreSQL database
# Find failure reason in logs
$ grep "httpd.service" sos_commands/logs/journalctl_--no-pager | grep -i "failed\|error"
Jan 15 10:23:45 server systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 10:23:45 server systemd[1]: httpd.service: Failed with result 'exit-code'
Jan 15 10:23:45 server httpd[12345]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
# Interpretation: httpd failed because port 80 is already in use
Example 2: SELinux Denial Analysis
# Check for AVC denials
$ grep "avc.*denied" var/log/audit/audit.log | head -5
type=AVC msg=audit(1705320245.123:456): avc: denied { write } for pid=1234 comm="httpd" name="index.html" dev="sda1" ino=789012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0
# Interpretation:
# - httpd (web server) was denied write access
# - Target file: index.html with context user_home_t
# - Issue: Web server trying to write to user home directory
# - Solution: Fix file context or move file to proper location
Example 3: Package Version Check
# Check for specific package versions
$ grep "^openssl" installed-rpms
openssl-1.1.1k-7.el8_6.x86_64
openssl-libs-1.1.1k-7.el8_6.x86_64
$ grep "^kernel" installed-rpms
kernel-4.18.0-425.el8.x86_64
kernel-4.18.0-477.el8.x86_64
kernel-core-4.18.0-425.el8.x86_64
kernel-core-4.18.0-477.el8.x86_64
# Interpretation:
# - OpenSSL version 1.1.1k (check for known CVEs)
# - Multiple kernels installed (good for rollback)
# - Current kernel is 4.18.0-477 (from uname)
Tips for Effective Analysis
- Check service dependencies: Failed service may be due to dependency failure
- Correlate with logs: Service failures often have detailed errors in logs
- Verify configurations: Check service config files for syntax errors
- Consider timing: When did service fail? Correlate with system events
- SELinux context matters: File contexts must match policy expectations
- Package versions: Compare with known good/bad versions
- Uptime significance: Very long uptime may mean missed security updates
Common Configuration Patterns and Issues
- Service dependency failure: ServiceB fails because ServiceA is not running
- Port conflict: Service fails to bind - port already in use
- Permission denied: Service can't access required files/directories
- SELinux blocking: Service denied access by SELinux policy
- Missing dependencies: Required package not installed
- Configuration error: Syntax error in config file
- Resource limits: Service hits ulimit (open files, processes, etc.)
- Outdated kernel: Running kernel doesn't match installed packages
Configuration Issue Severity Classification
| Issue Type | Severity | Impact |
|---|---|---|
| Critical service failed | High | Core functionality unavailable |
| Optional service failed | Low | Non-essential feature unavailable |
| SELinux in permissive | Warning | Reduced security, hiding issues |
| SELinux disabled | Critical | No mandatory access control |
| Kernel very outdated | High | Missing security fixes |
| EOL OS version | Critical | No security updates |
| Many AVC denials | Warning | Policy may need tuning |
See Also
- Logs Analysis Skill: For detailed service failure log analysis
- Resource Analysis Skill: For resource limit issues
- Network Analysis Skill: For network service configuration
Commands 2
Name
sosreport:analyze
Synopsis
/sosreport:analyze <path-to-sosreport> [--only <areas>] [--skip <areas>]
Analysis Areas:
-
logs: Analyze system and application logs (journald, syslog, dmesg, application logs)- Identifies errors, warnings, critical messages
- Detects OOM killer events, kernel panics, segfaults
- Counts and categorizes errors by severity
- Provides timeline of critical events
-
resources: Analyze system resource usage (memory, CPU, disk, processes)- Memory usage, swap, and pressure indicators
- CPU information and load averages
- Disk usage and filesystem capacity
- Top resource consumers and zombie processes
-
network: Analyze network configuration and connectivity- Network interface status and IP addresses
- Routing table and default gateway
- Active connections and listening services
- Firewall rules (firewalld/iptables)
- DNS configuration and hostname resolution
-
system-config: Analyze system configuration (packages, services, security)- OS version and kernel information
- Installed package versions
- Systemd service status and failures
- SELinux/AppArmor configuration and denials
- Kernel parameters and resource limits
Description
The sosreport:analyze command performs comprehensive analysis of a sosreport archive (from https://github.com/sosreport/sos) to identify system issues, configuration problems, and potential causes of failures. It examines system logs, resource usage, network configuration, installed packages, and other diagnostic data collected by sosreport.
By default, all analysis areas are executed. Use --only to run specific areas or --skip to exclude areas from analysis.
Arguments
$1(required): Path to the sosreport archive file (.tar.gzor.tar.xz) or extracted directory--only <areas>(optional): Comma-separated list of analysis areas to run. Valid areas:logs,resources,network,system-config. If not specified, all areas are analyzed.--skip <areas>(optional): Comma-separated list of analysis areas to skip. Valid areas:logs,resources,network,system-config. Cannot be used with--only.
Implementation
The sosreport analysis is organized into several specialized phases, each with detailed implementation guidance in separate skill documents. The command supports selective analysis through optional arguments.
1. Parse Arguments and Determine Analysis Scope
-
Parse command-line arguments
- Extract the sosreport path (required first argument)
- Check for
--onlyflag and parse comma-separated areas - Check for
--skipflag and parse comma-separated areas - Validate that
--onlyand--skipare not used together
-
Validate analysis areas
- Valid areas:
logs,resources,network,system-config - If invalid area specified, return error with list of valid areas
- Normalize area names (case-insensitive, accept variations like
systemforsystem-config)
- Valid areas:
-
Determine which skills to run
- If no flags specified: Run all skills (default comprehensive analysis)
- If
--onlyspecified: Run only the specified skills - If
--skipspecified: Run all skills except the specified ones - Store the list of skills to execute for later phases
-
Example argument parsing:
# Parse: /sosreport:analyze /path/sos.tar.gz --only logs,network # Result: Run only logs-analysis and network-analysis skills # Parse: /sosreport:analyze /path/sos.tar.gz --skip resources # Result: Run logs, network, and system-config (skip resources) # Parse: /sosreport:analyze /path/sos.tar.gz # Result: Run all skills (comprehensive analysis)
2. Extract and Validate Sosreport
-
Check if path exists
- Verify the provided path points to a valid file or directory
- If file doesn't exist, return error with helpful message
-
Extract archive if needed
- If path is a
.tar.gzor.tar.xzfile:- Create extraction directory:
.work/sosreport-analyze/{timestamp}/ - Extract archive:
tar -xf <path> -C .work/sosreport-analyze/{timestamp}/ - Store extracted directory path for analysis
- Create extraction directory:
- If path is already a directory:
- Verify it's a valid sosreport directory (check for
sos_commands/,sos_logs/, etc.) - Use the directory directly
- Verify it's a valid sosreport directory (check for
- If path is a
-
Identify sosreport structure
- Locate the root directory (usually has format
sosreport-{hostname}-{date}/) - Verify expected directories exist:
sos_commands/,sos_logs/,sos_reports/
- Locate the root directory (usually has format
3. Analyze System Logs
Run condition: Only if logs area is selected (or no filters specified)
Detailed implementation: See plugins/sosreport/skills/logs-analysis/SKILL.md
Perform comprehensive log analysis including:
- Journald logs (journalctl output)
- System logs (messages, dmesg, secure)
- Application-specific logs
- Error counting and categorization
- Timeline of critical events
- OOM killer events, kernel panics, segfaults
Key outputs:
- Error statistics by severity
- Top error messages by frequency
- Critical findings with timestamps
- Log file locations for investigation
4. Analyze Resource Usage
Run condition: Only if resources area is selected (or no filters specified)
Detailed implementation: See plugins/sosreport/skills/resource-analysis/SKILL.md
Perform resource analysis including:
- Memory usage and pressure indicators
- CPU information and load averages
- Disk usage and I/O errors
- Process analysis (top consumers, zombies)
- Resource exhaustion patterns
Key outputs:
- Memory usage metrics and swap status
- CPU count and load per CPU
- Filesystems near capacity
- Top CPU and memory-consuming processes
- Resource-related issues and recommendations
5. Analyze Network Configuration
Run condition: Only if network area is selected (or no filters specified)
Detailed implementation: See plugins/sosreport/skills/network-analysis/SKILL.md
Perform network analysis including:
- Network interface configuration and status
- Routing table and default gateway
- Active connections and listening services
- Firewall rules (firewalld/iptables)
- DNS configuration and hostname resolution
- Network errors from logs
Key outputs:
- Interface status with IP addresses
- Routing configuration
- Connection statistics by state
- Firewall configuration summary
- DNS and hostname settings
- Network-related errors and issues
6. Analyze Installed Packages and System Configuration
Run condition: Only if system-config area is selected (or no filters specified)
Detailed implementation: See plugins/sosreport/skills/system-config-analysis/SKILL.md
Perform system configuration analysis including:
- OS version and kernel information
- Installed package versions
- Systemd service status
- Failed services with reasons
- SELinux/AppArmor configuration and denials
- Kernel parameters and resource limits
Key outputs:
- System information summary
- Key package versions
- Failed services with failure reasons
- SELinux status and denial count
- Configuration issues and recommendations
7. Generate Interactive Summary
-
Create findings structure
- Organize findings by category (Critical, High, Medium, Low, Info)
- Include only findings from the selected analysis areas
- For each finding, include:
- Severity level
- Category (logs, resources, network, packages, config)
- Description of the issue
- Evidence (file paths, log snippets, metrics)
- Recommended actions
-
Display summary in terminal
- Show executive summary with key statistics
- List critical and high-severity findings
- Provide file paths for detailed investigation
- Include timeline of significant events
- Suggest next steps for troubleshooting
-
Format output
SOSREPORT ANALYSIS SUMMARY ========================== System: {hostname} Report Date: {date} OS: {os_version} Kernel: {kernel_version} CRITICAL ISSUES (count) ----------------------- - [Issue description with file reference] HIGH PRIORITY (count) --------------------- - [Issue description with file reference] MEDIUM PRIORITY (count) ----------------------- - [Issue description with file reference] RESOURCE SUMMARY ---------------- - Memory: X GB used / Y GB total (Z% used) - Disk: Most full filesystem at X% - Load Average: X.XX, X.XX, X.XX TOP ERRORS IN LOGS ------------------ 1. [Error message] (count occurrences) 2. [Error message] (count occurrences) FAILED SERVICES --------------- - [service name]: [reason] RECOMMENDATIONS --------------- 1. [Actionable recommendation] 2. [Actionable recommendation] ANALYSIS LOCATION ----------------- Extracted to: {extraction_path} -
Interactive drill-down
- Offer to explore specific areas in more detail
- Allow user to ask follow-up questions about findings
- Provide file paths for manual investigation
Return Value
- Format: Interactive summary displayed in terminal with categorized findings
- Exit code:
- 0 if analysis completes successfully
- 1 if sosreport path is invalid
- 2 if sosreport structure is malformed
Examples
-
Comprehensive analysis (default):
/sosreport:analyze /tmp/sosreport-server01-2024-01-15.tar.xzExtracts archive to
.work/sosreport-analyze/{timestamp}/and performs comprehensive analysis using all skills (logs, resources, network, system-config). -
Analyze only logs and network:
/sosreport:analyze /tmp/sosreport-server01-2024-01-15.tar.xz --only logs,networkPerforms only log analysis and network analysis. Useful when investigating connectivity or service issues without needing full resource analysis.
-
Skip resource analysis:
/sosreport:analyze /tmp/sosreport.tar.gz --skip resourcesPerforms all analysis except resource analysis. Useful when you already know resource metrics and want to focus on configuration and logs.
-
Quick log-only analysis:
/sosreport:analyze /tmp/sosreport.tar.xz --only logsPerforms only log analysis. Fastest option for quickly identifying errors and critical events without analyzing configuration or resources.
-
Analyze extracted sosreport directory:
/sosreport:analyze /tmp/sosreport-server01-2024-01-15/Analyzes an already extracted sosreport directory with comprehensive analysis.
-
Selective analysis on extracted directory:
/sosreport:analyze /tmp/sosreport-server01-2024-01-15/ --only system-config,networkAnalyzes only system configuration and network from an already extracted directory.
-
Follow-up investigation:
User: /sosreport:analyze /tmp/sosreport.tar.gz --only logs Agent: [Shows log analysis summary] User: Can you now analyze the resources as well? Agent: /sosreport:analyze /tmp/sosreport.tar.gz --only resources Agent: [Shows resource analysis]
Notes
- Sosreport structure varies by OS version and sosreport version
- Command handles both compressed archives and extracted directories
- Analysis focuses on common issues but can be extended for specific use cases
- For OpenShift/Kubernetes sosreports, additional pod/container analysis may be relevant
- Large sosreports (>1GB) may take several minutes to analyze
- Selective analysis: Use
--onlyor--skipto run specific analysis areas for faster results - Performance: Running only needed analysis areas reduces analysis time significantly
- Valid areas:
logs,resources,network,system-config
Prerequisites
-
tar utility: Required for extracting compressed sosreports
- Check:
which tar - Usually pre-installed on Linux/macOS
- Check:
-
Sufficient disk space: Extracted sosreports can be large
- Check available space:
df -h .work/ - Recommend at least 2x the compressed archive size
- Check available space:
See Also
Analysis Skills
- Logs Analysis:
plugins/sosreport/skills/logs-analysis/SKILL.md- Detailed guidance for analyzing system and application logs - Resource Analysis:
plugins/sosreport/skills/resource-analysis/SKILL.md- Detailed guidance for analyzing memory, CPU, disk, and processes - Network Analysis:
plugins/sosreport/skills/network-analysis/SKILL.md- Detailed guidance for analyzing network configuration and connectivity - System Configuration Analysis:
plugins/sosreport/skills/system-config-analysis/SKILL.md- Detailed guidance for analyzing packages, services, and security settings
External Resources
- Sosreport documentation: https://github.com/sosreport/sos
- Red Hat sosreport guide: https://access.redhat.com/solutions/3592
Name
sosreport:ovs-db
Synopsis
/sosreport:ovs-db [sosreport-path] [--db] [--flows-only] [--query <json>]
Description
The ovs-db command analyzes Open vSwitch data collected in sosreports. It operates in four modes:
- Default mode: Full analysis - conf.db + all text files (requires ovsdb-tool)
- Database mode (
--db): Database only - analyze conf.db (requires ovsdb-tool) - Text files mode (
--flows-only): Text files only - no ovsdb-tool needed! - Query mode (
--query): Run custom OVSDB JSON queries (requires ovsdb-tool)
What it analyzes:
From Database (Default or --db Mode)
- System Information: OVS version, DPDK settings, external IDs (from conf.db)
- Bridge Details: Datapath type (kernel/userspace), fail mode, ports (from conf.db)
- Interface Inventory: By type, with pod-to-interface mapping (from conf.db)
From Text Files (Default or --flows-only Mode)
- System Information: OVS version, DPDK settings, external IDs (from
ovs-vsctl list Open_vSwitch) - Topology: Bridges with ports grouped by type (from
ovs-vsctl show) - Bridge Details: Datapath type, fail mode, CT zones (from
ovs-vsctl list bridge) - Interface Inventory: By type, with pod-to-interface mapping (from
ovs-vsctl list interface) - OpenFlow Flows: Flow counts, drop detection, top flows (from
ovs-ofctl dump-flows) - Port Statistics: RX/TX drops and errors (from
ovs-ofctl dump-ports) - Tunnel Ports: Configured tunnels (from
ovs-appctl tnl.ports.show) - Datapath Health: Flow table usage vs limit (from
ovs-appctl upcall.show) - OVS Internal Stats: Netlink, OpenFlow, OVSDB counters (from
ovs-appctl coverage.show)
Custom Queries (--query Mode)
- Direct OVSDB table queries (requires ovsdb-tool)
Modes of operation:
- Default: Full analysis - conf.db + all text files (requires ovsdb-tool, falls back to text files if not available)
- Database (
--db): Database only - analyze conf.db (requires ovsdb-tool) - Text Files (
--flows-only): Text files only - no ovsdb-tool needed - Query Mode (
--query): Run custom OVSDB JSON queries (requires ovsdb-tool)
Prerequisites
Default mode (full analysis):
ovsdb-toolmust be installed (from openvswitch package)- Falls back to
--flows-onlyif ovsdb-tool not found - Sosreport with
sos_commands/openvswitch/directory andconf.db
Database mode (--db):
ovsdb-toolmust be installed (from openvswitch package)- Check:
which ovsdb-tool - Fedora/RHEL:
sudo dnf install openvswitch - Ubuntu/Debian:
sudo apt install openvswitch-common
- Check:
- Sosreport with
conf.dbfile
Text files mode (--flows-only):
- No special tools needed - works out of the box!
- Sosreport with
sos_commands/openvswitch/directory
Query mode (--query):
ovsdb-toolmust be installed- Incompatible with
--flows-only
Sosreport Data:
The sosreport should contain:
sosreport-hostname-date/
├── etc/openvswitch/conf.db (for --db mode)
│ OR var/lib/openvswitch/conf.db
└── sos_commands/openvswitch/ (default mode)
├── ovs-vsctl_-t_5_show (topology)
├── ovs-vsctl_-t_5_list_* (tables)
├── ovs-ofctl_dump-flows_<bridge> (flows)
├── ovs-ofctl_dump-ports_<bridge> (port stats)
├── ovs-appctl_coverage.show (internal stats)
├── ovs-appctl_upcall.show (datapath health)
└── ...
Analysis Script:
The script is bundled with this plugin:
<plugin-root>/skills/ovs-db-analysis/scripts/analyze_ovs_db.py
Implementation
The command performs the following steps:
-
Locate Analysis Script:
SCRIPT_PATH=$(find ~ -name "analyze_ovs_db.py" -path "*/sosreport/skills/ovs-db-analysis/scripts/*" 2>/dev/null | head -1) if [ -z "$SCRIPT_PATH" ]; then echo "ERROR: analyze_ovs_db.py script not found." exit 1 fi -
Handle Input Path:
- If sosreport archive (
.tar.gz,.tar.xz): Extract to temporary directory - If directory: Use directly
- If
conf.dbfile: Use database mode automatically
- If sosreport archive (
-
Run Analysis:
- Default mode: Parse text files in
sos_commands/openvswitch/ - --db mode: Query
conf.dbusingovsdb-tool - --query mode: Execute custom OVSDB query
- Default mode: Parse text files in
-
Analyze Data:
- Parse topology and system info
- Parse bridge and interface details
- Analyze OpenFlow flows (in default and
--flows-onlymodes) - Report drops, errors, and health indicators
Return Value
The command outputs structured analysis:
================================================================================
OVS ANALYSIS - sosreport-hostname-2024-01-15
================================================================================
Mode: Text file analysis (no ovsdb-tool required)
================================================================================
OVS SYSTEM INFORMATION
================================================================================
Field Value
------------------------- --------------------------------------------------
OVS Version "3.3.4-62.el9fdp"
DB Version "8.5.0"
System Type rhcos
System Version "4.16"
DPDK Initialized false
Datapath Types [netdev, system]
External IDs:
hostname: master2.example.com
ovn-encap-ip: 10.32.110.5
ovn-encap-type: geneve
ovn-bridge-mappings: physnet:br-ex
================================================================================
OVS TOPOLOGY
================================================================================
System UUID: 7e9a3f70-86fa-4578-a849-4fd807a64a10
Total Bridges: 2
Bridge: br-ex
ports: 3
internal: br-ex
patch: patch-br-ex-to-br-int
system: nm-bond
Bridge: br-int
fail_mode: secure
datapath_type: system
ports: 12
geneve: 9 ports
internal: ovn-k8s-mp0, br-int
patch: patch-br-int-to-br-ex
================================================================================
BRIDGE DETAILS
================================================================================
Bridge: br-int
------------------------------------------------------------
Datapath: system (kernelspace)
Fail Mode: secure
Datapath ID: "00005ac5dfc26094"
Port Count: 12
CT Zones: 19
================================================================================
INTERFACE ANALYSIS
================================================================================
Total Interfaces: 15
By Type:
geneve: 9 interfaces
internal: br-int, ovn-k8s-mp0, br-ex
patch: patch-br-ex-to-br-int, patch-br-int-to-br-ex
system: nm-bond
Pod Interfaces: 0
----------------------------------------------------------------------
(none on this node)
================================================================================
OPENFLOW ANALYSIS
================================================================================
Bridge: br-int
----------------------------------------------------------------------
Total flows: 2,017
Flows with hits: 318
Drop flows: 150 (9 actively dropping)
Tables used: 53 (0-79)
Top tables by flow count:
Table 21: 200 flows
Table 13: 163 flows
⚠️ ACTIVE DROP FLOWS (9):
table=40, priority=0, packets=8,105
table=79, priority=100, packets=1,356
match: ip,reg14=0x2,metadata=0x5,dl_src=00:62:0b:ea:b5:e0
--------------------------------------------------------------------------------
PORT STATISTICS
--------------------------------------------------------------------------------
Bridge: br-int
Total ports: 12
⚠️ Ports with drops/errors:
Port 1: drops=11, errors=0
RX: 852 pkts, 23,856 bytes
TX: 7 pkts, 826 bytes
--------------------------------------------------------------------------------
DATAPATH FLOW TABLE HEALTH
--------------------------------------------------------------------------------
Current flows: 155 / 200,000 (0.1% used)
Average: 156, Max seen: 215
✓ Flow table healthy
--------------------------------------------------------------------------------
OVS INTERNAL STATISTICS
--------------------------------------------------------------------------------
METRIC DESCRIPTION TOTAL RATE/s
------------------------- ----------------------------------- --------------- ----------
netlink_sent Netlink messages sent 46,153 12.8
netlink_received Netlink messages received 56,078 15.5
txn_success OVSDB transactions (success) 471 0.1
Examples
Default Mode (Full Analysis)
-
Full analysis (requires ovsdb-tool):
/sosreport:ovs-db /tmp/sosreport-server01-2024-01-15/Analyzes conf.db + all text files. Falls back to text files if ovsdb-tool not installed.
-
Analyze from archive:
/sosreport:ovs-db /tmp/sosreport-server01-2024-01-15.tar.xzExtracts and runs full analysis.
Database Mode (--db)
-
Database only (requires ovsdb-tool):
/sosreport:ovs-db /tmp/sosreport/ --dbQueries
conf.dbonly - no flow analysis. -
Analyze conf.db directly:
/sosreport:ovs-db /var/lib/openvswitch/conf.dbAutomatically uses database mode.
Text Files Mode (--flows-only)
- Text files only (no ovsdb-tool needed):
Parses all text files in/sosreport:ovs-db /tmp/sosreport/ --flows-onlysos_commands/openvswitch/.
Query Mode (--query)
-
Query all bridges:
/sosreport:ovs-db /tmp/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Bridge", "where":[], "columns":["name","datapath_type"]}]' -
Query VXLAN tunnels:
/sosreport:ovs-db /tmp/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[["type","==","vxlan"]], "columns":["name","options"]}]' -
Check interface errors:
/sosreport:ovs-db /tmp/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Interface", "where":[], "columns":["name","error","link_state"]}]' -
Check DPDK configuration:
/sosreport:ovs-db /tmp/sosreport/ --query '["Open_vSwitch", {"op":"select", "table":"Open_vSwitch", "where":[], "columns":["dpdk_initialized","other_config"]}]'
Error Handling
Missing ovsdb-tool (only for --db mode):
Error: ovsdb-tool not found. Please install openvswitch package.
Fedora/RHEL: sudo dnf install openvswitch
Ubuntu/Debian: sudo apt install openvswitch-common
Solution: Either install ovsdb-tool or use default mode (without --db).
sos_commands/openvswitch not found:
Error: sos_commands/openvswitch not found in /path/to/sosreport
Solution: Ensure the sosreport has OVS data collected.
conf.db not found (for --db mode):
Error: conf.db not found in /path/to/sosreport
Looked for conf.db in:
- etc/openvswitch/conf.db
- var/lib/openvswitch/conf.db
Tip: Run without --db to analyze text files only
Notes
- Default mode runs full analysis (conf.db + text files), requires ovsdb-tool, falls back to text files if not available
- Database mode (
--db) analyzes conf.db only, requiresovsdb-tool - Text files mode (
--flows-only) parses text files only - no special tools needed - Query mode (
--query) runs raw OVSDB JSON queries, requires ovsdb-tool, incompatible with--flows-only - Pod-to-Interface Mapping: Uses
external_idsand interface naming conventions - Drop Detection: Identifies flows with
actions=dropthat have packet hits - Datapath Health: Checks flow table usage vs limit from upcall stats
- Query Format: Accepts raw OVSDB JSON queries in format
["Open_vSwitch", {"op":"select", ...}]
Use Cases
-
Troubleshoot Packet Drops:
- Run default analysis to see active drop flows
- Check port statistics for RX/TX drops
-
Check Datapath Health:
- Review flow table usage vs limit
- If usage > 90%, flows are being evicted too aggressively
-
Map Pods to Interfaces:
- See pod-to-OVS interface mapping
- Find which ofport a pod uses
-
Investigate DPDK Configuration:
- Check DPDK initialization status
- Review datapath types (kernelspace vs userspace)
-
Debug OVN Connectivity:
- Check external_ids for OVN configuration
- Review tunnel endpoints (geneve ports)
- Verify patch port connections between bridges
-
Audit Configuration:
- Review all bridges and their ports
- Check fail modes and CT zones
- Review OVS version and system info
Arguments
- $1 (sosreport-path): Required. Path to sosreport archive (
.tar.gz,.tar.xz), extracted directory, or directconf.dbfile. - --db: Optional. Database only mode - analyze conf.db only (requires ovsdb-tool).
- --flows-only: Optional. Text files only mode - no ovsdb-tool needed.
- --query, -q (json-query): Optional. Run a raw OVSDB JSON query (requires ovsdb-tool, incompatible with
--flows-only).
See Also
- OVN Database Analysis:
/must-gather:ovn-dbs- For analyzing OVN Northbound/Southbound databases - Sosreport Analysis:
/sosreport:analyze- Comprehensive sosreport analysis - Network Analysis:
/sosreport:analyze --only network- Network-focused analysis
Sponsors
Details
- Marketplace
- openshift-eng/ai-helpers
- Source
- plugins/sosreport
- Category
- Debugging
- Keywords
- sosreport, diagnostics, troubleshooting, rhel
More from openshift-eng/ai-helpers
All 56- teamsv0.0.18
- test-coveragev0.0.2
- testingv0.1.0
- utilsv0.0.13
- workspacesv1.0.1
- yamlv0.0.2
- agendasv0.0.3
- agentic-docsv2.2.12
- ai-sbomv0.0.1
- bigqueryv0.0.2
- bigquery-ci-datav0.0.3
- bigquery-cost-analysisv0.0.7
- civ0.0.111
- ci-extrasv0.3.0
- code-reviewv0.0.18
- compliancev0.4.1
- consolev0.1.0
- container-imagev0.0.2
- docv0.0.2
- etcdv0.0.2
- gitv0.0.7
- githubv0.3.0
- golangv0.7.0
- gopls-lspv1.0.0