name	System Configuration Analysis
description	Analyze system configuration data from sosreport archives, extracting OS details, installed packages, systemd service status, SELinux/AppArmor policies, and kernel parameters from the sosreport directory structure to diagnose configuration-related system issues

System Configuration Analysis Skill

This skill provides detailed guidance for analyzing system configuration from sosreport archives, including OS information, installed packages, systemd services, and SELinux/AppArmor settings.

When to Use This Skill

Use this skill when:

Analyzing the /sosreport:analyze command's system configuration phase
Investigating service failures or misconfigurations
Verifying package versions and updates
Checking security policy settings (SELinux/AppArmor)
Understanding system state and configuration

Prerequisites

Sosreport archive must be extracted to a working directory
Path to the sosreport root directory must be known
Understanding of Linux system administration

Key Configuration Data Locations in Sosreport

System Information:
- uname - Kernel version
- etc/os-release - OS distribution and version
- uptime - System uptime
- proc/uptime - Uptime in seconds
- sos_commands/release/ - Release information
Package Information:
- installed-rpms - RPM packages (RHEL/Fedora/CentOS)
- installed-debs - DEB packages (Debian/Ubuntu)
- sos_commands/yum/ - Yum/DNF information
- sos_commands/rpm/ - RPM database queries
Service Status:
- sos_commands/systemd/systemctl_list-units - All units
- sos_commands/systemd/systemctl_list-units_--failed - Failed units
- sos_commands/systemd/systemctl_status_--all - Detailed service status
- sos_commands/systemd/systemctl_list-unit-files - Unit files
SELinux:
- sos_commands/selinux/sestatus - SELinux status
- sos_commands/selinux/getenforce - Current enforcement mode
- sos_commands/selinux/selinux-policy - Policy information
- var/log/audit/audit.log - SELinux denials
AppArmor (if applicable):
- sos_commands/apparmor/ - AppArmor configuration
- etc/apparmor.d/ - AppArmor profiles
System Configuration Files:
- etc/ - System-wide configuration
- etc/sysctl.conf or etc/sysctl.d/ - Kernel parameters
- etc/security/limits.conf - Resource limits

Implementation Steps

Step 1: Analyze System Information

Check OS version and distribution:

if [ -f etc/os-release ]; then
  cat etc/os-release
fi

Get kernel version:

if [ -f uname ]; then
  cat uname
elif [ -f proc/version ]; then
  cat proc/version
fi

Check system uptime:

if [ -f uptime ]; then
  cat uptime
elif [ -f proc/uptime ]; then
  # Parse uptime from proc/uptime (seconds)
  awk '{printf "%.2f days\n", $1/86400}' proc/uptime
fi

Extract key system details:
- OS name and version
- Kernel version
- System architecture (x86_64, aarch64, etc.)
- Uptime (days)
Check for outdated kernel or OS:
- Compare kernel version with current stable
- Note if system hasn't been rebooted in a very long time (>365 days)
- Identify if OS version is EOL

Step 2: Analyze Installed Packages

List installed packages:

# For RPM-based systems
if [ -f installed-rpms ]; then
  cat installed-rpms
fi

# For DEB-based systems
if [ -f installed-debs ]; then
  cat installed-debs
fi

Extract key package versions:

# Important system packages
grep -E "^(kernel|systemd|glibc|openssh|openssl)" installed-rpms 2>/dev/null

# Or use awk to parse package name and version
awk '{print $1}' installed-rpms | head -20

Check for known problematic versions:
- Security vulnerabilities (if known CVEs)
- Buggy package versions
- Compatibility issues

Identify package manager issues:

# Check yum/dnf logs for errors
if [ -d sos_commands/yum ]; then
  grep -i "error\|fail" sos_commands/yum/* 2>/dev/null
fi

Count packages and categorize:
- Total packages installed
- Key package versions (kernel, systemd, glibc, etc.)
- Recently updated packages (if timestamps available)

Step 3: Analyze Service Status

List all systemd units:

if [ -f sos_commands/systemd/systemctl_list-units ]; then
  cat sos_commands/systemd/systemctl_list-units
fi

Identify failed services:

if [ -f sos_commands/systemd/systemctl_list-units_--failed ]; then
  cat sos_commands/systemd/systemctl_list-units_--failed
elif [ -f sos_commands/systemd/systemctl_list-units ]; then
  grep "failed" sos_commands/systemd/systemctl_list-units
fi

Check service details:

# Parse detailed status for failed services
if [ -f sos_commands/systemd/systemctl_status_--all ]; then
  # Extract service names and their status
  grep -E "●|Active:" sos_commands/systemd/systemctl_status_--all | head -50
fi

Count services by state:

# Count running, failed, inactive services
if [ -f sos_commands/systemd/systemctl_list-units ]; then
  awk '{print $4}' sos_commands/systemd/systemctl_list-units | sort | uniq -c
fi

Identify critical service failures:
- System services (systemd-*, dbus, NetworkManager)
- Application services (httpd, nginx, postgresql, etc.)
- Custom services

Extract failure reasons from logs:

# For each failed service, find related log entries
grep -i "failed to start\|service.*failed" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

Step 4: Analyze SELinux Configuration

Check SELinux status:

if [ -f sos_commands/selinux/sestatus ]; then
  cat sos_commands/selinux/sestatus
fi

Get SELinux mode:

if [ -f sos_commands/selinux/getenforce ]; then
  cat sos_commands/selinux/getenforce
fi

Check for SELinux denials:

# Look for AVC denials in audit log
if [ -f var/log/audit/audit.log ]; then
  grep "avc.*denied" var/log/audit/audit.log | head -50
fi

# Or in journald logs
grep -i "selinux.*denied\|avc.*denied" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

Parse denial information:
- Extract denied operations (read, write, execute, etc.)
- Identify source and target contexts
- Note which services are affected

Check for SELinux booleans:

if [ -f sos_commands/selinux/getsebool_-a ]; then
  cat sos_commands/selinux/getsebool_-a
fi

Identify SELinux issues:
- SELinux in permissive mode (may hide errors)
- SELinux disabled (security concern)
- Frequent AVC denials (policy may need adjustment)
- Context mismatches

Step 5: Check System Configuration

Review kernel parameters:

# Check sysctl settings
if [ -f sos_commands/kernel/sysctl_-a ]; then
  cat sos_commands/kernel/sysctl_-a
elif [ -d etc/sysctl.d ]; then
  cat etc/sysctl.d/*.conf 2>/dev/null
fi

Check resource limits:

if [ -f etc/security/limits.conf ]; then
  grep -v "^#\|^$" etc/security/limits.conf
fi

# Check limits.d directory
if [ -d etc/security/limits.d ]; then
  cat etc/security/limits.d/*.conf 2>/dev/null
fi

Review boot parameters:

if [ -f sos_commands/boot/grub2-editenv_list ]; then
  cat sos_commands/boot/grub2-editenv_list
elif [ -f proc/cmdline ]; then
  cat proc/cmdline
fi

Check systemd configuration:

# Look for systemd configuration overrides
if [ -d etc/systemd/system ]; then
  find etc/systemd/system -name "*.conf" 2>/dev/null
fi

Step 6: Generate System Configuration Summary

Create a structured summary with the following sections:

System Information:
- OS name and version
- Kernel version
- Architecture
- System uptime
- Last boot time
Package Summary:
- Total packages installed
- Key package versions (kernel, systemd, glibc, openssl, openssh)
- Known problematic packages (if any)
- Package manager issues
Service Status:
- Total services
- Running services count
- Failed services count
- List of failed services with reasons
- Critical service status
SELinux/AppArmor:
- SELinux status (enabled/disabled)
- SELinux mode (enforcing/permissive)
- Denial count
- Top denied operations
- Policy recommendations
Configuration Issues:
- Kernel parameter anomalies
- Resource limit issues
- Boot parameter problems
- Configuration file errors

Error Handling

Missing configuration files:
- Different distributions have different file locations
- Some files may not be collected based on sosreport options
- Document missing data in summary
Package manager variations:
- Handle both RPM and DEB systems
- Account for different package naming conventions
- Support multiple package managers (yum, dnf, apt)
SELinux vs AppArmor:
- Check which MAC system is in use
- Analyze accordingly
- Note if both or neither are present
Systemd vs init:
- Older systems may use init instead of systemd
- Check for both service management systems
- Adapt analysis based on what's present

Output Format

The system configuration analysis should produce:

SYSTEM CONFIGURATION SUMMARY
============================

SYSTEM INFORMATION
------------------
OS: {os_name} {os_version}
Kernel: {kernel_version}
Architecture: {arch}
Uptime: {uptime_days} days ({last_boot_time})

Status: {OK|WARNING|CRITICAL}
Notes:
  - {system_info_note}

INSTALLED PACKAGES
------------------
Total Packages: {count}

Key Package Versions:
  kernel: {version}
  systemd: {version}
  glibc: {version}
  openssl: {version}
  openssh-server: {version}

Status: {OK|WARNING|CRITICAL}
Issues:
  - {package_issue_description}

SYSTEMD SERVICES
----------------
Total Units: {total}
Active: {active_count}
Failed: {failed_count}
Inactive: {inactive_count}

Failed Services:
  ● {service_name}.service - {description}
    Reason: {failure_reason}
    Last Failed: {timestamp}

  ● {service_name}.service - {description}
    Reason: {failure_reason}
    Last Failed: {timestamp}

Status: {OK|WARNING|CRITICAL}
Recommendations:
  - {service_recommendation}

SELINUX
-------
Status: {enabled|disabled}
Mode: {enforcing|permissive|disabled}
Policy: {policy_name}

AVC Denials: {count} denials found

Top Denied Operations:
  [{count}x] {operation} on {target} by {source}
  [{count}x] {operation} on {target} by {source}

SELinux Booleans: {count} custom settings

Status: {OK|WARNING|CRITICAL}
Issues:
  - {selinux_issue_description}

Recommendations:
  - {selinux_recommendation}

KERNEL PARAMETERS
-----------------
Key sysctl Settings:
  vm.swappiness: {value}
  net.ipv4.ip_forward: {value}
  kernel.panic: {value}

Custom Parameters: {count} custom settings found

Status: {OK|WARNING|CRITICAL}
Notes:
  - {kernel_param_note}

RESOURCE LIMITS
---------------
Custom Limits Found: {count}

{user_or_group}  {type}  {item}  {value}

Status: {OK|WARNING}
Notes:
  - {limits_note}

CRITICAL CONFIGURATION ISSUES
-----------------------------
{severity}: {issue_description}
  Evidence: {file_path}
  Impact: {impact_description}
  Recommendation: {remediation_action}

RECOMMENDATIONS
---------------
1. {actionable_recommendation}
2. {actionable_recommendation}

DATA SOURCES
------------
- OS Info: {sosreport_path}/etc/os-release
- Kernel: {sosreport_path}/uname
- Packages: {sosreport_path}/installed-rpms
- Services: {sosreport_path}/sos_commands/systemd/systemctl_list-units
- SELinux: {sosreport_path}/sos_commands/selinux/sestatus
- Audit Log: {sosreport_path}/var/log/audit/audit.log

Examples

Example 1: Failed Service Analysis

# List failed services
$ cat sos_commands/systemd/systemctl_list-units_--failed
  UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
● httpd.service          loaded failed failed Apache Web Server
● postgresql.service     loaded failed failed PostgreSQL database

# Find failure reason in logs
$ grep "httpd.service" sos_commands/logs/journalctl_--no-pager | grep -i "failed\|error"
Jan 15 10:23:45 server systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 10:23:45 server systemd[1]: httpd.service: Failed with result 'exit-code'
Jan 15 10:23:45 server httpd[12345]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80

# Interpretation: httpd failed because port 80 is already in use

Example 2: SELinux Denial Analysis

# Check for AVC denials
$ grep "avc.*denied" var/log/audit/audit.log | head -5
type=AVC msg=audit(1705320245.123:456): avc: denied { write } for pid=1234 comm="httpd" name="index.html" dev="sda1" ino=789012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=file permissive=0

# Interpretation:
# - httpd (web server) was denied write access
# - Target file: index.html with context user_home_t
# - Issue: Web server trying to write to user home directory
# - Solution: Fix file context or move file to proper location

Example 3: Package Version Check

# Check for specific package versions
$ grep "^openssl" installed-rpms
openssl-1.1.1k-7.el8_6.x86_64
openssl-libs-1.1.1k-7.el8_6.x86_64

$ grep "^kernel" installed-rpms
kernel-4.18.0-425.el8.x86_64
kernel-4.18.0-477.el8.x86_64
kernel-core-4.18.0-425.el8.x86_64
kernel-core-4.18.0-477.el8.x86_64

# Interpretation:
# - OpenSSL version 1.1.1k (check for known CVEs)
# - Multiple kernels installed (good for rollback)
# - Current kernel is 4.18.0-477 (from uname)

Tips for Effective Analysis

Check service dependencies: Failed service may be due to dependency failure
Correlate with logs: Service failures often have detailed errors in logs
Verify configurations: Check service config files for syntax errors
Consider timing: When did service fail? Correlate with system events
SELinux context matters: File contexts must match policy expectations
Package versions: Compare with known good/bad versions
Uptime significance: Very long uptime may mean missed security updates

Common Configuration Patterns and Issues

Service dependency failure: ServiceB fails because ServiceA is not running
Port conflict: Service fails to bind - port already in use
Permission denied: Service can't access required files/directories
SELinux blocking: Service denied access by SELinux policy
Missing dependencies: Required package not installed
Configuration error: Syntax error in config file
Resource limits: Service hits ulimit (open files, processes, etc.)
Outdated kernel: Running kernel doesn't match installed packages

Configuration Issue Severity Classification

Issue Type	Severity	Impact
Critical service failed	High	Core functionality unavailable
Optional service failed	Low	Non-essential feature unavailable
SELinux in permissive	Warning	Reduced security, hiding issues
SELinux disabled	Critical	No mandatory access control
Kernel very outdated	High	Missing security fixes
EOL OS version	Critical	No security updates
Many AVC denials	Warning	Policy may need tuning

System Configuration Analysis

Install Skill

SKILL.md