
devops-observability-sre
Monitoring, tracing, logging, SLOs, incident response and on-call
- version
- 1.0.0
- GitHub stars
- 5
Install
/plugin install devops-observability-sre@skills-stackRun it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.
Contents
Discovered from sourceSkills 16
Azure Monitor OpenTelemetry SDK for TypeScript
Auto-instrument Node.js applications with distributed tracing, metrics, and logs.
Version and data boundaries
This document describes the existing @azure/monitor-opentelemetry distro. Current Microsoft guidance also offers @microsoft/opentelemetry for new adoption; do not initialize both. Match all OpenTelemetry package versions and use that version’s logger-provider constructor/processor API. Configure approved telemetry destinations, redact PII/secrets, and protect offline storage; automatic instrumentation may export sensitive request/log data. See Microsoft enablement guidance.
Installation
# Distro (recommended - auto-instrumentation)
npm install @azure/monitor-opentelemetry
# Low-level exporters (custom OpenTelemetry setup)
npm install @azure/monitor-opentelemetry-exporter
# Custom logs ingestion
npm install @azure/monitor-ingestion
Environment Variables
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=...;IngestionEndpoint=...
Quick Start (Auto-Instrumentation)
IMPORTANT: Call useAzureMonitor() BEFORE importing other modules.
import { useAzureMonitor } from "@azure/monitor-opentelemetry";
useAzureMonitor({
azureMonitorExporterOptions: {
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
}
});
// Static ESM imports are hoisted; load the app after initialization.
const { default: express } = await import("express");
const app = express();
ESM Support (Node.js 18.19+)
node --import @azure/monitor-opentelemetry/loader ./dist/index.js
package.json:
{
"scripts": {
"start": "node --import @azure/monitor-opentelemetry/loader ./dist/index.js"
}
}
Full Configuration
import { useAzureMonitor, AzureMonitorOpenTelemetryOptions } from "@azure/monitor-opentelemetry";
import { resourceFromAttributes } from "@opentelemetry/resources";
const options: AzureMonitorOpenTelemetryOptions = {
azureMonitorExporterOptions: {
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
storageDirectory: "/path/to/offline/storage",
disableOfflineStorage: false
},
// Sampling
samplingRatio: 1.0, // 0-1, percentage of traces
// Features
enableLiveMetrics: true,
enableStandardMetrics: true,
enablePerformanceCounters: true,
// Instrumentation libraries
instrumentationOptions: {
azureSdk: { enabled: true },
http: { enabled: true },
mongoDb: { enabled: true },
mySql: { enabled: true },
postgreSql: { enabled: true },
redis: { enabled: true },
bunyan: { enabled: false },
winston: { enabled: false }
},
// Custom resource
resource: resourceFromAttributes({ "service.name": "my-service" })
};
useAzureMonitor(options);
Custom Traces
import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer("my-tracer");
const span = tracer.startSpan("doWork");
try {
span.setAttribute("component", "worker");
span.setAttribute("operation.id", "42");
span.addEvent("processing started");
// Your work here
} catch (error) {
span.recordException(error as Error);
span.setStatus({ code: 2, message: (error as Error).message });
} finally {
span.end();
}
Custom Metrics
import { metrics } from "@opentelemetry/api";
const meter = metrics.getMeter("my-meter");
// Counter
const counter = meter.createCounter("requests_total");
counter.add(1, { route: "/api/users", method: "GET" });
// Histogram
const histogram = meter.createHistogram("request_duration_ms");
histogram.record(150, { route: "/api/users" });
// Observable Gauge
const gauge = meter.createObservableGauge("active_connections");
gauge.addCallback((result) => {
result.observe(getActiveConnections(), { pool: "main" });
});
Manual Exporter Setup
Trace Exporter
import { AzureMonitorTraceExporter } from "@azure/monitor-opentelemetry-exporter";
import { NodeTracerProvider, BatchSpanProcessor } from "@opentelemetry/sdk-trace-node";
const exporter = new AzureMonitorTraceExporter({
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});
const provider = new NodeTracerProvider({
spanProcessors: [new BatchSpanProcessor(exporter)]
});
provider.register();
Metric Exporter
import { AzureMonitorMetricExporter } from "@azure/monitor-opentelemetry-exporter";
import { PeriodicExportingMetricReader, MeterProvider } from "@opentelemetry/sdk-metrics";
import { metrics } from "@opentelemetry/api";
const exporter = new AzureMonitorMetricExporter({
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});
const meterProvider = new MeterProvider({
readers: [new PeriodicExportingMetricReader({ exporter })]
});
metrics.setGlobalMeterProvider(meterProvider);
Log Exporter
import { AzureMonitorLogExporter } from "@azure/monitor-opentelemetry-exporter";
import { BatchLogRecordProcessor, LoggerProvider } from "@opentelemetry/sdk-logs";
import { logs } from "@opentelemetry/api-logs";
const exporter = new AzureMonitorLogExporter({
connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});
const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(new BatchLogRecordProcessor(exporter));
logs.setGlobalLoggerProvider(loggerProvider);
Custom Logs Ingestion
import { DefaultAzureCredential } from "@azure/identity";
import { LogsIngestionClient, isAggregateLogsUploadError } from "@azure/monitor-ingestion";
const endpoint = "https://<dce>.ingest.monitor.azure.com";
const ruleId = "<data-collection-rule-id>";
const streamName = "Custom-MyTable_CL";
const client = new LogsIngestionClient(endpoint, new DefaultAzureCredential());
const logs = [
{
Time: new Date().toISOString(),
Computer: "Server1",
Message: "Application started",
Level: "Information"
}
];
try {
await client.upload(ruleId, streamName, logs);
} catch (error) {
if (isAggregateLogsUploadError(error)) {
for (const uploadError of error.errors) {
console.error("Failed logs:", uploadError.failedLogs);
}
}
}
Custom Span Processing
Enrich writable spans in onStart with span.setAttribute(). Do not mutate ReadableSpan.attributes or spanContext().traceFlags in onEnd: ended spans are read-only and this is not a reliable export filter. Use a supported sampler or a reviewed filtering exporter/processor pipeline for the pinned SDK version, and verify exported data.
Sampling
import { ApplicationInsightsSampler } from "@azure/monitor-opentelemetry-exporter";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
// Sample 75% of traces
const sampler = new ApplicationInsightsSampler(0.75);
const provider = new NodeTracerProvider({ sampler });
Shutdown
import { useAzureMonitor, shutdownAzureMonitor } from "@azure/monitor-opentelemetry";
useAzureMonitor();
// On application shutdown
process.on("SIGTERM", async () => {
await shutdownAzureMonitor();
process.exit(0);
});
Key Types
import {
useAzureMonitor,
shutdownAzureMonitor,
AzureMonitorOpenTelemetryOptions,
InstrumentationOptions
} from "@azure/monitor-opentelemetry";
import {
AzureMonitorTraceExporter,
AzureMonitorMetricExporter,
AzureMonitorLogExporter,
ApplicationInsightsSampler,
AzureMonitorExporterOptions
} from "@azure/monitor-opentelemetry-exporter";
import {
LogsIngestionClient,
isAggregateLogsUploadError
} from "@azure/monitor-ingestion";
Best Practices
- Call useAzureMonitor() first - Before importing other modules
- Use ESM loader for ESM projects -
--import @azure/monitor-opentelemetry/loader - Enable offline storage - For reliable telemetry in disconnected scenarios
- Set sampling ratio - For high-traffic applications
- Add custom dimensions - Use span processors for enrichment
- Graceful shutdown - Call
shutdownAzureMonitor()to flush telemetry
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Canary Watch — Post-Deploy Monitoring
When to Use
- After deploying to production or staging
- After merging a risky PR
- When you want to verify a fix actually fixed it
- Continuous monitoring during a launch window
- After dependency upgrades
How It Works
Monitors a deployed URL for regressions. Runs in a loop until stopped or until the watch window expires.
What It Watches
1. HTTP Status — is the page returning 200?
2. Console Errors — new errors that weren't there before?
3. Network Failures — failed API calls, 5xx responses?
4. Performance — LCP/CLS/INP regression vs baseline?
5. Content — did key elements disappear? (h1, nav, footer, CTA)
6. API Health — are critical endpoints responding within SLA?
7. Static Assets — are JS, CSS, image, and font requests returning 2xx/3xx with expected content types?
8. SSE Streams — do event-stream endpoints connect and receive an initial event or heartbeat?
Watch Modes
Quick check (default): single pass, report results
/canary-watch https://myapp.com
Sustained watch: check every N minutes for M hours
/canary-watch https://myapp.com --interval 5m --duration 2h
Diff mode: compare staging vs production
/canary-watch --compare https://staging.myapp.com https://myapp.com
Alert Thresholds
critical: # immediate alert
- HTTP status != 200
- Console error count > 5 (new errors only)
- LCP > 4s
- API endpoint returns 5xx
- Static asset returns 4xx/5xx
- SSE endpoint cannot connect or drops before first heartbeat
warning: # flag in report
- LCP increased > 500ms from baseline
- CLS > 0.1
- New console warnings
- Response time > 2x baseline
- Static asset content type changed unexpectedly
- SSE heartbeat latency > 2x baseline
info: # log only
- Minor performance variance
- New network requests (third-party scripts added?)
Notifications
When a critical threshold is crossed:
- Desktop notification (macOS/Linux)
- Optional: Slack/Discord webhook
- Log to
~/.claude/canary-watch.log
Output
## Canary Report — myapp.com — 2026-03-23 03:15 PST
### Status: HEALTHY ✓
| Check | Result | Baseline | Delta |
|-------|--------|----------|-------|
| HTTP | 200 ✓ | 200 | — |
| Console errors | 0 ✓ | 0 | — |
| LCP | 1.8s ✓ | 1.6s | +200ms |
| CLS | 0.01 ✓ | 0.01 | — |
| API /health | 145ms ✓ | 120ms | +25ms |
| Static assets | 42/42 ✓ | 42/42 | — |
| SSE /events | connected ✓ | connected | +80ms heartbeat |
### No regressions detected. Deploy is clean.
Integration
Pair with:
/browser-qafor pre-deploy verification- Hooks: add as a PostToolUse hook on
git pushto auto-check after deploys - CI: run in GitHub Actions after deploy step
Dashboard Builder
Use this when the task is to build a dashboard people can operate from.
The goal is not "show every metric." The goal is to answer:
- is it healthy?
- where is the bottleneck?
- what changed?
- what action should someone take?
When to Use
- "Build a Kafka monitoring dashboard"
- "Create a Grafana dashboard for Elasticsearch"
- "Make a SigNoz dashboard for this service"
- "Turn this metrics list into a real operational dashboard"
Guardrails
- do not start from visual layout; start from operator questions
- do not include every available metric just because it exists
- do not mix health, throughput, and resource panels without structure
- do not ship panels without titles, units, and sane thresholds
Workflow
1. Define the operating questions
Organize around:
- health / availability
- latency / performance
- throughput / volume
- saturation / resources
- service-specific risk
2. Study the target platform schema
Inspect existing dashboards first:
- JSON structure
- query language
- variables
- threshold styling
- section layout
3. Build the minimum useful board
Recommended structure:
- overview
- performance
- resources
- service-specific section
4. Cut vanity panels
Every panel should answer a real question. If it does not, remove it.
Example Panel Sets
Elasticsearch
- cluster health
- shard allocation
- search latency
- indexing rate
- JVM heap / GC
Kafka
- broker count
- under-replicated partitions
- messages in / out
- consumer lag
- disk and network pressure
API gateway / ingress
- request rate
- p50 / p95 / p99 latency
- error rate
- upstream health
- active connections
Quality Checklist
- valid dashboard JSON
- clear section grouping
- titles and units are present
- thresholds/status colors are meaningful
- variables exist for common filters
- default time range and refresh are sensible
- no vanity panels with no operator value
Related Skills
research-opsbackend-patternsterminal-ops
Datadog
Monitor infrastructure and applications with Datadog's unified observability platform.
When to Use This Skill
Use this skill when:
- Implementing enterprise-grade monitoring
- Setting up APM and distributed tracing
- Creating unified dashboards for infrastructure and apps
- Configuring intelligent alerting
- Monitoring cloud infrastructure (AWS, Azure, GCP)
Prerequisites
- Datadog account and API key
- Agent installation access
- Application code access for APM
Confirm account/site, approved telemetry and cost/retention limits before enabling logs, process data or APM. Redact credentials, request bodies, user/order IDs and sensitive tags before egress; distribution metrics do not eliminate cardinality costs. Docker socket access is privileged even with a read-only bind. Use a pre-created scoped Secret with the chart's existing-secret setting rather than passing a real API key through --set/history/Helm release values. Pin chart/agent/tracer versions and validate their configuration schemas. Inline SDK fragments are partial: Python metrics needs time and the version-appropriate metric intake enum, Go needs a real request context/user ID, and tracer.configure parameters vary by ddtrace major. Verify metric names/units and no-data/zero-denominator behavior against actual instrumentation before activating monitors or external notifications.
Agent Installation
Linux
# Install agent
curl -fL https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh -o /tmp/datadog-agent-install.sh
# Review and pin the full installer before executing separately with protected credentials.
# Or via package manager
apt-get update && apt-get install datadog-agent
# Configure API key
echo "api_key: YOUR_API_KEY" >> /etc/datadog-agent/datadog.yaml
# Start agent
systemctl start datadog-agent
systemctl enable datadog-agent
Docker
# docker-compose.yml
version: '3.8'
services:
datadog-agent:
image: gcr.io/datadoghq/agent:7
environment:
- DD_API_KEY=${DD_API_KEY}
- DD_SITE=datadoghq.com
- DD_LOGS_ENABLED=true
- DD_APM_ENABLED=true
- DD_PROCESS_AGENT_ENABLED=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /proc/:/host/proc/:ro
- /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
ports:
- "127.0.0.1:8126:8126" # APM
- "127.0.0.1:8125:8125/udp" # DogStatsD
Kubernetes
# Using Helm
helm repo add datadog https://helm.datadoghq.com
helm install datadog datadog/datadog \
--set datadog.apiKey=${DD_API_KEY} \
--set datadog.site=datadoghq.com \
--set datadog.logs.enabled=true \
--set datadog.apm.portEnabled=true \
--set datadog.processAgent.enabled=true \
--namespace datadog \
--create-namespace
Agent Configuration
# /etc/datadog-agent/datadog.yaml
api_key: YOUR_API_KEY
site: datadoghq.com
# Hostname
hostname: myserver.example.com
# Tags applied to all metrics
tags:
- env:production
- service:myapp
- team:platform
# Log collection
logs_enabled: true
# APM
apm_config:
enabled: true
apm_dd_url: https://trace.agent.datadoghq.com
# Process monitoring
process_config:
enabled: true
# Container monitoring
container_collect_all: true
docker_labels_as_tags:
app: service
environment: env
Integration Configuration
MySQL
# /etc/datadog-agent/conf.d/mysql.d/conf.yaml
init_config:
instances:
- host: localhost
port: 3306
username: datadog
password: <PASSWORD>
tags:
- env:production
options:
replication: true
extra_status_metrics: true
PostgreSQL
# /etc/datadog-agent/conf.d/postgres.d/conf.yaml
init_config:
instances:
- host: localhost
port: 5432
username: datadog
password: <PASSWORD>
dbname: mydb
collect_activity_metrics: true
collect_database_size_metrics: true
NGINX
# /etc/datadog-agent/conf.d/nginx.d/conf.yaml
init_config:
instances:
- nginx_status_url: http://localhost:80/nginx_status
tags:
- env:production
Log Collection
File-Based Logs
# /etc/datadog-agent/conf.d/myapp.d/conf.yaml
logs:
- type: file
path: /var/log/myapp/*.log
service: myapp
source: python
sourcecategory: custom
tags:
- env:production
- type: file
path: /var/log/nginx/access.log
service: nginx
source: nginx
log_processing_rules:
- type: exclude_at_match
name: exclude_healthchecks
pattern: health_check
Docker Logs
# docker-compose.yml
services:
myapp:
labels:
com.datadoghq.ad.logs: '[{"source": "python", "service": "myapp"}]'
Kubernetes Logs
# Pod annotation
apiVersion: v1
kind: Pod
metadata:
annotations:
ad.datadoghq.com/myapp.logs: |
[{
"source": "python",
"service": "myapp",
"log_processing_rules": [{
"type": "multi_line",
"name": "python_tracebacks",
"pattern": "^Traceback"
}]
}]
APM Configuration
Python
from ddtrace import patch_all, tracer
# Automatic instrumentation
patch_all()
# Configure tracer
tracer.configure(
hostname='localhost',
port=8126,
service='myapp',
env='production',
version='1.0.0'
)
# Manual instrumentation
@tracer.wrap(service='myapp', resource='process_order')
def process_order(order_id):
with tracer.trace('validate_order') as span:
span.set_tag('order_id', order_id)
# Validation logic
with tracer.trace('save_order'):
# Save logic
pass
# Install library
pip install ddtrace
# Run with auto-instrumentation
ddtrace-run python app.py
Node.js
const tracer = require('dd-trace').init({
service: 'myapp',
env: 'production',
version: '1.0.0',
logInjection: true
});
// Manual instrumentation
const span = tracer.startSpan('custom_operation');
span.setTag('user_id', userId);
// ... operation
span.finish();
# Install library
npm install dd-trace
# Run with auto-instrumentation
DD_TRACE_ENABLED=true node --require dd-trace/init app.js
Go
import (
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)
func main() {
tracer.Start(
tracer.WithService("myapp"),
tracer.WithEnv("production"),
tracer.WithServiceVersion("1.0.0"),
)
defer tracer.Stop()
// Manual span
span, ctx := tracer.StartSpanFromContext(ctx, "process_request")
defer span.Finish()
span.SetTag("user_id", userID)
}
Custom Metrics
DogStatsD
from datadog import DogStatsd
statsd = DogStatsd(host='localhost', port=8125)
# Counter
statsd.increment('myapp.orders.count', tags=['env:production'])
# Gauge
statsd.gauge('myapp.queue.size', queue_size, tags=['queue:orders'])
# Histogram
statsd.histogram('myapp.request.duration', response_time)
# Distribution
statsd.distribution('myapp.response_time', duration, tags=['endpoint:/api/orders'])
API Submission
from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metric_payload import MetricPayload
from datadog_api_client.v2.model.metric_series import MetricSeries
from datadog_api_client.v2.model.metric_point import MetricPoint
configuration = Configuration()
with ApiClient(configuration) as api_client:
api = MetricsApi(api_client)
payload = MetricPayload(
series=[
MetricSeries(
metric="custom.metric.name",
type=MetricSeries.GAUGE,
points=[MetricPoint(value=42.0, timestamp=int(time.time()))],
tags=["env:production"]
)
]
)
api.submit_metrics(body=payload)
Dashboards
Dashboard JSON
{
"title": "Application Overview",
"widgets": [
{
"definition": {
"type": "timeseries",
"title": "Request Rate",
"requests": [
{
"q": "sum:trace.http.request.hits{service:myapp}.as_rate()",
"display_type": "line"
}
]
}
},
{
"definition": {
"type": "query_value",
"title": "Error Rate",
"requests": [
{
"q": "sum:trace.http.request.errors{service:myapp}.as_rate() / sum:trace.http.request.hits{service:myapp}.as_rate() * 100"
}
],
"precision": 2
}
}
]
}
Monitors (Alerts)
Metric Monitor
{
"name": "High Error Rate",
"type": "metric alert",
"query": "sum(last_5m):sum:trace.http.request.errors{service:myapp}.as_count() / sum:trace.http.request.hits{service:myapp}.as_count() > 0.05",
"message": "Error ratio is {{value}} for {{service.name}}. @slack-alerts",
"tags": ["service:myapp", "env:production"],
"options": {
"thresholds": {
"critical": 0.05,
"warning": 0.02
},
"notify_no_data": true,
"no_data_timeframe": 10
}
}
APM Monitor
{
"name": "High Latency Alert",
"type": "trace-analytics alert",
"query": "trace-analytics(\"service:myapp @http.status_code:2*\").rollup(\"avg\", \"@duration\").last(\"5m\") > 2000000000",
"message": "Average latency is above 2 seconds. @pagerduty",
"options": {
"thresholds": {
"critical": 2000000000
}
}
}
Common Issues
Issue: Agent Not Reporting
Problem: No data appearing in Datadog
Solution: Check API key, verify agent status with datadog-agent status
Issue: Missing Traces
Problem: APM traces not appearing Solution: Verify APM is enabled, check tracer configuration, verify port 8126
Issue: High Cardinality Tags
Problem: Custom metrics getting dropped Solution: Reduce unique tag values, use distributions instead of histograms
Best Practices
- Use consistent service and environment tags
- Implement proper tag naming conventions
- Use unified service tagging (service, env, version)
- Set up service-level monitors
- Create dashboards per service
- Implement log correlation with traces
- Use distributions for latency metrics
- Configure proper alert escalation
Related Skills
- prometheus-grafana (
prometheus-grafana) - Open source alternative - alerting-oncall (
alerting-oncall) - Alert management - aws-vpc (
aws-vpc) - AWS monitoring
Limitations
- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.
Example
git status && git diff --stat
kubectl diff -f manifest.yaml
Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.
Use this skill when
- Working on devops troubleshooter tasks or workflows
- Needing guidance, best practices, or checklists for devops troubleshooter
Do not use this skill when
- The task is unrelated to devops troubleshooter
- You need a different domain or tool outside this scope
Instructions
- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open
resources/implementation-playbook.md.
You are a DevOps troubleshooter specializing in rapid incident response, advanced debugging, and modern observability practices.
Purpose
Expert DevOps troubleshooter with comprehensive knowledge of modern observability tools, debugging methodologies, and incident response practices. Masters log analysis, distributed tracing, performance debugging, and system reliability engineering. Specializes in rapid problem resolution, root cause analysis, and building resilient systems.
Capabilities
Modern Observability & Monitoring
- Logging platforms: ELK Stack (Elasticsearch, Logstash, Kibana), Loki/Grafana, Fluentd/Fluent Bit
- APM solutions: DataDog, New Relic, Dynatrace, AppDynamics, Instana, Honeycomb
- Metrics & monitoring: Prometheus, Grafana, InfluxDB, VictoriaMetrics, Thanos
- Distributed tracing: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry, custom tracing
- Cloud-native observability: OpenTelemetry collector, service mesh observability
- Synthetic monitoring: Pingdom, Datadog Synthetics, custom health checks
Container & Kubernetes Debugging
- kubectl mastery: Advanced debugging commands, resource inspection, troubleshooting workflows
- Container runtime debugging: Docker, containerd, CRI-O, runtime-specific issues
- Pod troubleshooting: Init containers, sidecar issues, resource constraints, networking
- Service mesh debugging: Istio, Linkerd, Consul Connect traffic and security issues
- Kubernetes networking: CNI troubleshooting, service discovery, ingress issues
- Storage debugging: Persistent volume issues, storage class problems, data corruption
Network & DNS Troubleshooting
- Network analysis: tcpdump, Wireshark, eBPF-based tools, network latency analysis
- DNS debugging: dig, nslookup, DNS propagation, service discovery issues
- Load balancer issues: AWS ALB/NLB, Azure Load Balancer, GCP Load Balancer debugging
- Firewall & security groups: Network policies, security group misconfigurations
- Service mesh networking: Traffic routing, circuit breaker issues, retry policies
- Cloud networking: VPC connectivity, peering issues, NAT gateway problems
Performance & Resource Analysis
- System performance: CPU, memory, disk I/O, network utilization analysis
- Application profiling: Memory leaks, CPU hotspots, garbage collection issues
- Database performance: Query optimization, connection pool issues, deadlock analysis
- Cache troubleshooting: Redis, Memcached, application-level caching issues
- Resource constraints: OOMKilled containers, CPU throttling, disk space issues
- Scaling issues: Auto-scaling problems, resource bottlenecks, capacity planning
Application & Service Debugging
- Microservices debugging: Service-to-service communication, dependency issues
- API troubleshooting: REST API debugging, GraphQL issues, authentication problems
- Message queue issues: Kafka, RabbitMQ, SQS, dead letter queues, consumer lag
- Event-driven architecture: Event sourcing issues, CQRS problems, eventual consistency
- Deployment issues: Rolling update problems, configuration errors, environment mismatches
- Configuration management: Environment variables, secrets, config drift
CI/CD Pipeline Debugging
- Build failures: Compilation errors, dependency issues, test failures
- Deployment troubleshooting: GitOps issues, ArgoCD/Flux problems, rollback procedures
- Pipeline performance: Build optimization, parallel execution, resource constraints
- Security scanning issues: SAST/DAST failures, vulnerability remediation
- Artifact management: Registry issues, image corruption, version conflicts
- Environment-specific issues: Configuration mismatches, infrastructure problems
Cloud Platform Troubleshooting
- AWS debugging: CloudWatch analysis, AWS CLI troubleshooting, service-specific issues
- Azure troubleshooting: Azure Monitor, PowerShell debugging, resource group issues
- GCP debugging: Cloud Logging, gcloud CLI, service account problems
- Multi-cloud issues: Cross-cloud communication, identity federation problems
- Serverless debugging: Lambda functions, Azure Functions, Cloud Functions issues
Security & Compliance Issues
- Authentication debugging: OAuth, SAML, JWT token issues, identity provider problems
- Authorization issues: RBAC problems, policy misconfigurations, permission debugging
- Certificate management: TLS certificate issues, renewal problems, chain validation
- Security scanning: Vulnerability analysis, compliance violations, security policy enforcement
- Audit trail analysis: Log analysis for security events, compliance reporting
Database Troubleshooting
- SQL debugging: Query performance, index usage, execution plan analysis
- NoSQL issues: MongoDB, Redis, DynamoDB performance and consistency problems
- Connection issues: Connection pool exhaustion, timeout problems, network connectivity
- Replication problems: Primary-replica lag, failover issues, data consistency
- Backup & recovery: Backup failures, point-in-time recovery, disaster recovery testing
Infrastructure & Platform Issues
- Infrastructure as Code: Terraform state issues, provider problems, resource drift
- Configuration management: Ansible playbook failures, Chef cookbook issues, Puppet manifest problems
- Container registry: Image pull failures, registry connectivity, vulnerability scanning issues
- Secret management: Vault integration, secret rotation, access control problems
- Disaster recovery: Backup failures, recovery testing, business continuity issues
Advanced Debugging Techniques
- Distributed system debugging: CAP theorem implications, eventual consistency issues
- Chaos engineering: Fault injection analysis, resilience testing, failure pattern identification
- Performance profiling: Application profilers, system profiling, bottleneck analysis
- Log correlation: Multi-service log analysis, distributed tracing correlation
- Capacity analysis: Resource utilization trends, scaling bottlenecks, cost optimization
Behavioral Traits
- Gathers comprehensive facts first through logs, metrics, and traces before forming hypotheses
- Forms systematic hypotheses and tests them methodically with minimal system impact
- Documents all findings thoroughly for postmortem analysis and knowledge sharing
- Implements fixes with minimal disruption while considering long-term stability
- Adds proactive monitoring and alerting to prevent recurrence of issues
- Prioritizes rapid resolution while maintaining system integrity and security
- Thinks in terms of distributed systems and considers cascading failure scenarios
- Values blameless postmortems and continuous improvement culture
- Considers both immediate fixes and long-term architectural improvements
- Emphasizes automation and runbook development for common issues
Knowledge Base
- Modern observability platforms and debugging tools
- Distributed system troubleshooting methodologies
- Container orchestration and cloud-native debugging techniques
- Network troubleshooting and performance analysis
- Application performance monitoring and optimization
- Incident response best practices and SRE principles
- Security debugging and compliance troubleshooting
- Database performance and reliability issues
Response Approach
- Assess the situation with urgency appropriate to impact and scope
- Gather comprehensive data from logs, metrics, traces, and system state
- Form and test hypotheses systematically with minimal system disruption
- Implement immediate fixes to restore service while planning permanent solutions
- Document thoroughly for postmortem analysis and future reference
- Add monitoring and alerting to detect similar issues proactively
- Plan long-term improvements to prevent recurrence and improve system resilience
- Share knowledge through runbooks, documentation, and team training
- Conduct blameless postmortems to identify systemic improvements
Example Interactions
- "Debug high memory usage in Kubernetes pods causing frequent OOMKills and restarts"
- "Analyze distributed tracing data to identify performance bottleneck in microservices architecture"
- "Troubleshoot intermittent 504 gateway timeout errors in production load balancer"
- "Investigate CI/CD pipeline failures and implement automated debugging workflows"
- "Root cause analysis for database deadlocks causing application timeouts"
- "Debug DNS resolution issues affecting service discovery in Kubernetes cluster"
- "Analyze logs to identify security breach and implement containment procedures"
- "Troubleshoot GitOps deployment failures and implement automated rollback procedures"
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Distributed Tracing
Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.
Purpose
Track requests across distributed systems to understand latency, dependencies, and failure points.
When to Use
- Debug latency issues
- Understand service dependencies
- Identify bottlenecks
- Trace error propagation
- Analyze request paths
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
- Sample appropriately (1-10% in production)
- Add meaningful tags (user_id, request_id)
- Propagate context across all service boundaries
- Log exceptions in spans
- Use consistent naming for operations
- Monitor tracing overhead (<1% CPU impact)
- Set up alerts for trace errors
- Implement distributed context (baggage)
- Use span events for important milestones
- Document instrumentation standards
Integration with Logging
Correlated Logs
import logging
from opentelemetry import trace
logger = logging.getLogger(__name__)
def process_request():
span = trace.get_current_span()
trace_id = span.get_span_context().trace_id
logger.info(
"Processing request",
extra={"trace_id": format(trace_id, '032x')}
)
Troubleshooting
No traces appearing:
- Check collector endpoint
- Verify network connectivity
- Check sampling configuration
- Review application logs
High latency overhead:
- Reduce sampling rate
- Use batch span processor
- Check exporter configuration
Related Skills
prometheus-configuration- For metricsgrafana-dashboards- For visualizationslo-implementation- For latency SLOs
Grafana Dashboards
Create and manage production-ready Grafana dashboards for comprehensive system observability.
Purpose
Design effective Grafana dashboards for monitoring applications, infrastructure, and business metrics.
When to Use
- Visualize Prometheus metrics
- Create custom dashboards
- Implement SLO dashboards
- Monitor infrastructure
- Track business KPIs
Dashboard Design Principles
1. Hierarchy of Information
┌─────────────────────────────────────┐
│ Critical Metrics (Big Numbers) │
├─────────────────────────────────────┤
│ Key Trends (Time Series) │
├─────────────────────────────────────┤
│ Detailed Metrics (Tables/Heatmaps) │
└─────────────────────────────────────┘
2. RED Method (Services)
- Rate - Requests per second
- Errors - Error rate
- Duration - Latency/response time
3. USE Method (Resources)
- Utilization - % time resource is busy
- Saturation - Queue length/wait time
- Errors - Error count
Dashboard Structure
API Monitoring Dashboard
{
"dashboard": {
"title": "API Monitoring",
"tags": ["api", "production"],
"timezone": "browser",
"refresh": "30s",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m])) by (service)",
"legendFormat": "{{service}}"
}
],
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
},
{
"title": "Error Rate %",
"type": "graph",
"targets": [
{
"expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))) * 100",
"legendFormat": "Error Rate"
}
],
"alert": {
"conditions": [
{
"evaluator": { "params": [5], "type": "gt" },
"operator": { "type": "and" },
"query": { "params": ["A", "5m", "now"] },
"type": "query"
}
]
},
"gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
},
{
"title": "P95 Latency",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
"legendFormat": "{{service}}"
}
],
"gridPos": { "x": 0, "y": 8, "w": 24, "h": 8 }
}
]
}
}
Panel Types
1. Stat Panel (Single Value)
{
"type": "stat",
"title": "Total Requests",
"targets": [
{
"expr": "sum(http_requests_total)"
}
],
"options": {
"reduceOptions": {
"values": false,
"calcs": ["lastNotNull"]
},
"orientation": "auto",
"textMode": "auto",
"colorMode": "value"
},
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "value": 0, "color": "green" },
{ "value": 80, "color": "yellow" },
{ "value": 90, "color": "red" }
]
}
}
}
}
2. Time Series Graph
{
"type": "graph",
"title": "CPU Usage",
"targets": [
{
"expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
}
],
"yaxes": [
{ "format": "percent", "max": 100, "min": 0 },
{ "format": "short" }
]
}
3. Table Panel
{
"type": "table",
"title": "Service Status",
"targets": [
{
"expr": "up",
"format": "table",
"instant": true
}
],
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": { "Time": true },
"indexByName": {},
"renameByName": {
"instance": "Instance",
"job": "Service",
"Value": "Status"
}
}
}
]
}
4. Heatmap
{
"type": "heatmap",
"title": "Latency Heatmap",
"targets": [
{
"expr": "sum(rate(http_request_duration_seconds_bucket[5m])) by (le)",
"format": "heatmap"
}
],
"dataFormat": "tsbuckets",
"yAxis": {
"format": "s"
}
}
Variables
Query Variables
{
"templating": {
"list": [
{
"name": "namespace",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(kube_pod_info, namespace)",
"refresh": 1,
"multi": false
},
{
"name": "service",
"type": "query",
"datasource": "Prometheus",
"query": "label_values(kube_service_info{namespace=\"$namespace\"}, service)",
"refresh": 1,
"multi": true
}
]
}
}
Use Variables in Queries
sum(rate(http_requests_total{namespace="$namespace", service=~"$service"}[5m]))
Alerts in Dashboards
{
"alert": {
"name": "High Error Rate",
"conditions": [
{
"evaluator": {
"params": [5],
"type": "gt"
},
"operator": { "type": "and" },
"query": {
"params": ["A", "5m", "now"]
},
"reducer": { "type": "avg" },
"type": "query"
}
],
"executionErrorState": "alerting",
"for": "5m",
"frequency": "1m",
"message": "Error rate is above 5%",
"noDataState": "no_data",
"notifications": [{ "uid": "slack-channel" }]
}
}
Dashboard Provisioning
dashboards.yml:
apiVersion: 1
providers:
- name: "default"
orgId: 1
folder: "General"
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /etc/grafana/dashboards
Common Dashboard Patterns
Infrastructure Dashboard
Key Panels:
- CPU utilization per node
- Memory usage per node
- Disk I/O
- Network traffic
- Pod count by namespace
- Node status
Database Dashboard
Key Panels:
- Queries per second
- Connection pool usage
- Query latency (P50, P95, P99)
- Active connections
- Database size
- Replication lag
- Slow queries
Application Dashboard
Key Panels:
- Request rate
- Error rate
- Response time (percentiles)
- Active users/sessions
- Cache hit rate
- Queue length
Best Practices
- Start with templates (Grafana community dashboards)
- Use consistent naming for panels and variables
- Group related metrics in rows
- Set appropriate time ranges (default: Last 6 hours)
- Use variables for flexibility
- Add panel descriptions for context
- Configure units correctly
- Set meaningful thresholds for colors
- Use consistent colors across dashboards
- Test with different time ranges
Dashboard as Code
Terraform Provisioning
resource "grafana_dashboard" "api_monitoring" {
config_json = file("${path.module}/dashboards/api-monitoring.json")
folder = grafana_folder.monitoring.id
}
resource "grafana_folder" "monitoring" {
title = "Production Monitoring"
}
Ansible Provisioning
- name: Deploy Grafana dashboards
copy:
src: "{{ item }}"
dest: /etc/grafana/dashboards/
with_fileglob:
- "dashboards/*.json"
notify: restart grafana
Related Skills
prometheus-configuration- For metric collectionslo-implementation- For SLO dashboards
Incident Runbook Templates
Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.
When to Use This Skill
- Creating incident response procedures
- Building service-specific runbooks
- Establishing escalation paths
- Documenting recovery procedures
- Responding to active incidents
- Onboarding on-call engineers
Core Concepts
1. Incident Severity Levels
| Severity | Impact | Response Time | Example |
|---|---|---|---|
| SEV1 | Complete outage, data loss | 15 min | Production down |
| SEV2 | Major degradation | 30 min | Critical feature broken |
| SEV3 | Minor impact | 2 hours | Non-critical bug |
| SEV4 | Minimal impact | Next business day | Cosmetic issue |
2. Runbook Structure
1. Overview & Impact
2. Detection & Alerts
3. Initial Triage
4. Mitigation Steps
5. Root Cause Investigation
6. Resolution Procedures
7. Verification & Rollback
8. Communication Templates
9. Escalation Matrix
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
Do's
- Keep runbooks updated - Review after every incident
- Test runbooks regularly - Game days, chaos engineering
- Include rollback steps - Always have an escape hatch
- Document assumptions - What must be true for steps to work
- Link to dashboards - Quick access during stress
Don'ts
- Don't assume knowledge - Write for 3 AM brain
- Don't skip verification - Confirm each step worked
- Don't forget communication - Keep stakeholders informed
- Don't work alone - Escalate early
- Don't skip postmortems - Learn from every incident
Troubleshooting
Runbook steps work in staging but fail during a real incident
Steps often assume preconditions that are true in a healthy environment but not during an outage. For each command in your runbook, add a prerequisite check and a "what to do if this command fails" note:
# Step: Check pod status
kubectl get pods -n payments
# Prerequisites: kubectl configured, kubeconfig points to correct cluster
# If this fails: run `aws eks update-kubeconfig --name prod-cluster --region us-east-1`
# Expected output: pods in Running state
On-call engineer panics and skips steps out of order
Add a numbered checklist at the top of the runbook that mirrors the section numbers, so responders can track progress under stress without reading the full document:
## Quick Checklist
- [ ] 1. Declare incident severity and open war room
- [ ] 2. Check service health (Section 4.1)
- [ ] 3. Check recent deployments (Section 4.1)
- [ ] 4. Roll back if deploy is suspect (Section 4.1)
- [ ] 5. Post initial notification to #payments-incidents
- [ ] 6. Escalate if > 15 min unresolved
Runbook is outdated — commands reference old cluster names or endpoints
Runbooks rot because they're updated manually. Include a "Last Verified" date and owner at the top, and add a CI check that validates all curl endpoints and kubectl context names are still valid:
## Runbook Metadata
| Field | Value |
|---|---|
| Last verified | 2024-11-15 |
| Owner | @platform-team |
| Review cadence | After every SEV1/SEV2 |
Stakeholder communication is delayed while engineers are heads-down
Assign a dedicated incident communicator role (separate from the incident commander) whose only job is to post status updates. Add a standing agenda in the communication template:
Update every 15 minutes (even if no new information):
- Current status (Investigating / Mitigating / Monitoring)
- Impact (what is broken, who is affected, % of traffic)
- What we are doing right now
- Next update in: 15 minutes
Database runbook commands cause additional downtime when run incorrectly
Add explicit warnings before destructive SQL commands and require a dry-run output check before executing:
-- WARNING: This terminates active connections. Verify count first.
-- DRY RUN (check count before terminating):
SELECT count(*) FROM pg_stat_activity WHERE state = 'idle' AND query_start < now() - interval '10 minutes';
-- EXECUTE only after verifying count is reasonable (< 50):
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle' AND query_start < now() - interval '10 minutes';
Related Skills
postmortem-writing- After resolving an incident, use postmortem templates to capture root cause and preventive actionson-call-handoff-patterns- Structure shift handoffs so the incoming responder has full context on active incidents
Observability and Instrumentation
Overview
Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.
When to Use
- Building any feature that will run in production
- Adding a new service, endpoint, background job, or external integration
- A production incident took too long to diagnose ("we couldn't tell what happened")
- Setting up or reviewing alerting rules
- Reviewing a PR that adds I/O, retries, queues, or cross-service calls
NOT for:
- Diagnosing a failure happening right now — use the
debugging-and-error-recoveryskill (observability is what makes that skill fast next time) - Profiling and optimizing measured slowness — use the
performance-optimizationskill - Launch-day monitoring checklists and rollback triggers — see the
shipping-and-launchskill; this skill covers the instrumentation that feeds them
Process
1. Define "working" before instrumenting
Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:
FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.
If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.
2. Pick the right signal for each question
| Signal | Answers | Cost profile | Example |
|---|---|---|---|
| Structured log | "What happened in this specific case?" | Per-event; grows with traffic | payment_failed with provider error code |
| Metric | "How often / how fast, in aggregate?" | Fixed per series; cheap to query | p99 latency of provider calls |
| Trace | "Where did time go across services?" | Per-request; usually sampled | One slow checkout, broken down by hop |
Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.
3. Structured logging
Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:
// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);
// GOOD: stable event name + structured fields
logger.warn({
event: 'payment_failed',
paymentId: id,
provider: 'stripe',
errorCode: err.code,
attempt: n,
}, 'payment failed');
Log levels — use them consistently:
| Level | Meaning | On-call action |
|---|---|---|
error | Invariant broken; someone may need to act | Investigate |
warn | Degraded but handled (retry succeeded, fallback used) | Watch for trends |
info | Significant business event (order placed, job finished) | None |
debug | Diagnostic detail | Off in production by default |
Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:
// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
req.log = logger.child({ requestId: req.id });
res.setHeader('x-request-id', req.id);
next();
});
When several entry points write to one log, name the entry point. A correlation ID identifies a run; it does not say which code path started it. The same job reached by a scheduler, by a replay endpoint, and by a manual CLI run produces interchangeable lines in one sink, so attributing a line falls back to elimination — cross-reading the scheduler's history, the process table, a deploy log — and that argument holds only as long as those external records happen to still exist. Stamp the entry point where the run starts, next to the correlation ID, and propagate both the same way:
// One helper for every entry point: the run's own logger carries both fields.
// `entryPoint`, not `source` — ECS reserves `source.*` for network fields.
export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) =>
logger.child({ entryPoint, requestId: runId });
// scheduler tick -> runLog('scheduler', crypto.randomUUID())
// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id)
// CLI invocation -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID())
Both fields have to cross the same boundaries as the correlation ID — queue metadata, HTTP headers — or a worker re-derives the entry point and guesses. A field that merely correlates with an entry point is a hint, not an attribution: anything that can invoke the job can reproduce it.
Never log secrets, tokens, passwords, or full PII. This is a hard rule from the security-and-hardening skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.
4. Metrics
For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.
As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' prom-client — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.
import { Histogram } from 'prom-client';
const httpDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration',
labelNames: ['method', 'route', 'status_class'], // '2xx', not '200'
buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});
Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels — that belongs in logs and traces.
OK as label: route="/api/tasks/:id" status_class="5xx" provider="stripe"
NEVER a label: user_id, email, request_id, full URL, error message text
Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.
5. Distributed tracing
Use OpenTelemetry — it's the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:
// tracing.ts — must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
const sdk = new NodeSDK({
serviceName: 'checkout-service',
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
Add manual spans only around meaningful internal units of work (e.g., applyDiscounts, chargeProvider) and attach the attributes on-call will filter by. Propagate context across every async boundary — HTTP headers, queue message metadata — or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if your backend supports tail sampling.
6. Alerting
Alert on symptoms users feel, not on causes:
SYMPTOM (page-worthy): CAUSE (dashboard, not a page):
error rate > 1% for 5 min CPU at 85%
p99 latency > 2s one pod restarted
queue age > 10 min disk at 70%
Cause-based alerts fire when nothing is wrong and miss failures you didn't predict. Symptom-based alerts fire exactly when users are hurt, regardless of the cause.
Rules for every alert you create:
- It must be actionable. If the response is "ignore it, it self-heals", delete the alert.
- It links to a runbook — even three lines: what it means, first query to run, escalation path.
- It has a threshold and duration justified by the SLO or by historical data, not by a guess.
- Use two severities only: page (user-facing, act now) and ticket (degradation, act this week). A third tier becomes noise that trains people to ignore everything.
Writing Runbooks
Rule 2 above requires every alert to link to a runbook. A runbook's job is to answer three questions without requiring the reader to think: what is happening, what to check first, and who to call if that doesn't resolve it. Store in docs/runbooks/ named after the alert.
Minimum viable runbook (three lines):
# Runbook: High Error Rate on /api/tasks
**Means:** DB connection pool likely exhausted, or a bad deploy.
**First check:** `SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';`
— if count > pool limit, see Step 2. (Swap in the equivalent for your database.)
**Escalate to:** #db-oncall or engineering on-call rotation.
When to expand beyond three lines: add steps only when the first check alone isn't enough to decide. A five-step runbook that covers the three most common causes is better than a twenty-step document that covers every edge case and gets skimmed.
Keep runbooks current. Update the runbook as part of closing every incident it was used in — a stale runbook builds false confidence. If a step was wrong or missing, fix it before marking the incident resolved.
7. Verify the telemetry itself
Instrumentation is code; it can be wrong. Before calling the work done, trigger the paths and look at the actual output:
- Force an error in staging → find it in the logs by
requestId, confirm fields are structured (not[object Object]) - Send test traffic → confirm metric series appear with the expected labels and sane values
- Follow one request across services in the tracing UI → no broken spans
- Fire each new alert once (lower the threshold temporarily) → confirm it reaches the right channel and the runbook link works
Common Rationalizations
| Rationalization | Reality |
|---|---|
| "I'll add logging after it works" | "After" becomes "after the first incident", which is the most expensive moment to discover you're blind. Instrument as you build. |
| "More logs = more observability" | Unstructured noise makes incidents slower, not faster. Three queryable events beat three hundred prose lines. |
| "console.log is fine for now" | Unstructured output can't be filtered, correlated, or alerted on. The structured logger costs five extra minutes once. |
| "We can just look at the dashboards when something breaks" | Dashboards built without defined questions show you everything except the answer. Start from on-call questions. |
| "Alert on everything important, we'll tune later" | A noisy pager trains people to ignore it. The tuning never happens; the missed real page does. |
| "User ID as a metric label makes debugging easier" | It also makes your metrics backend fall over. High-cardinality lookups belong in logs and traces. |
| "Tracing is overkill for our two services" | Two services already means cross-service latency questions logs can't answer. Auto-instrumentation makes the cost trivial. |
Red Flags
- A feature PR with retries, queues, or external calls and zero new telemetry
- Log lines built by string interpolation instead of structured fields
- No correlation/request ID — each log line is an orphan
- One log stream fed by a scheduler, a webhook, and manual runs, with no field naming which one produced the line
- Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb)
- Latency tracked as an average with no percentiles
- Alerts that fire daily and get acknowledged without action
- Alerts on causes (CPU, memory) paging humans while user-facing error rate is unmonitored
- Secrets, tokens, or full request bodies appearing in logs
- "It works on my machine" as the only evidence a production feature is healthy
Verification
After instrumenting a feature, confirm:
- The on-call questions for this feature are written down, and each signal maps to one
- All log output is structured (JSON), with stable event names and a correlation ID on every line
- Every log sink written by more than one entry point carries an entry-point field, set where the run starts and propagated with the correlation ID rather than inferred downstream
- No secrets, tokens, or unredacted PII in any log line (spot-check actual output)
- RED metrics exist for every new endpoint and every external dependency, with bounded label sets
- Latency is a histogram; p95/p99 are queryable
- A single request can be followed end-to-end in the tracing UI without broken spans
- Every new alert is symptom-based, has a runbook link, and was test-fired once
- An induced failure in staging was located via telemetry alone, without reading the source
For the at-a-glance version of this list, including the pre-launch instrumentation gate, see ../../references/observability-checklist.md.
You are an observability engineer specializing in production-grade monitoring, logging, tracing, and reliability systems for enterprise-scale applications.
Use this skill when
- Designing monitoring, logging, or tracing systems
- Defining SLIs/SLOs and alerting strategies
- Investigating production reliability or performance regressions
Do not use this skill when
- You only need a single ad-hoc dashboard
- You cannot access metrics, logs, or tracing data
- You need application feature development instead of observability
Instructions
- Identify critical services, user journeys, and reliability targets.
- Define signals, instrumentation, and data retention.
- Build the smallest dashboards and actionable alerts needed for those SLOs; define owner, runbook and missing-data behavior.
- Reconcile numerator/denominator and sampling, exercise one alert in an authorized test environment, and measure noise before broad rollout.
Safety
- Use an allowlist of telemetry fields. Do not log credentials, raw prompts, query strings or full bodies by default; inspect actual exported data and retention.
- Use alerting thresholds that balance coverage and noise.
Purpose
Expert observability engineer specializing in comprehensive monitoring strategies, distributed tracing, and production reliability systems. Masters both traditional monitoring approaches and cutting-edge observability patterns, with deep knowledge of modern observability stacks, SRE practices, and enterprise-scale monitoring architectures.
Capabilities
Monitoring & Metrics Infrastructure
- Prometheus ecosystem with advanced PromQL queries and recording rules
- Grafana dashboard design with templating, alerting, and custom panels
- InfluxDB time-series data management and retention policies
- DataDog enterprise monitoring with custom metrics and synthetic monitoring
- New Relic APM integration and performance baseline establishment
- CloudWatch comprehensive AWS service monitoring and cost optimization
- Nagios and Zabbix for traditional infrastructure monitoring
- Custom metrics collection with StatsD, Telegraf, and Collectd
- Cardinality budgets: avoid user IDs, request IDs and unbounded URLs in metric labels
Distributed Tracing & APM
- Jaeger distributed tracing deployment and trace analysis
- Zipkin trace collection and service dependency mapping
- AWS X-Ray integration for serverless and microservice architectures
- OpenTracing and OpenTelemetry instrumentation standards
- Application Performance Monitoring with detailed transaction tracing
- Service mesh observability with Istio and Envoy telemetry
- Correlation between traces, logs, and metrics for root cause analysis
- Performance bottleneck identification and optimization recommendations
- Distributed system debugging and latency analysis
Log Management & Analysis
- ELK Stack (Elasticsearch, Logstash, Kibana) architecture and optimization
- Fluentd and Fluent Bit log forwarding and parsing configurations
- Splunk enterprise log management and search optimization
- Loki for cloud-native log aggregation with Grafana integration
- Log parsing, enrichment, and structured logging implementation
- Centralized logging for microservices and distributed systems
- Log retention policies and cost-effective storage strategies
- Security log analysis and compliance monitoring
- Real-time log streaming and alerting mechanisms
Alerting & Incident Response
- PagerDuty integration with intelligent alert routing and escalation
- Slack and Microsoft Teams notification workflows
- Alert correlation and noise reduction strategies
- Runbook automation and incident response playbooks
- On-call rotation management and fatigue prevention
- Post-incident analysis and blameless postmortem processes
- Alert threshold tuning and false positive reduction
- Multi-channel notification systems and redundancy planning
- Incident severity classification and response procedures
SLI/SLO Management & Error Budgets
- Service Level Indicator (SLI) definition and measurement
- Service Level Objective (SLO) establishment and tracking
- Error budget calculation and burn rate analysis
- SLA compliance monitoring and reporting
- Availability and reliability target setting
- Performance benchmarking and capacity planning
- Customer impact assessment and business metrics correlation
- Reliability engineering practices and failure mode analysis
- Chaos engineering integration for proactive reliability testing
OpenTelemetry & Modern Standards
- OpenTelemetry collector deployment and configuration
- Auto-instrumentation for multiple programming languages
- Custom telemetry data collection and export strategies
- Trace sampling strategies and performance optimization
- Vendor-agnostic observability pipeline design
- Protocol buffer and gRPC telemetry transmission
- Multi-backend telemetry export (Jaeger, Prometheus, DataDog)
- Observability data standardization across services
- Migration strategies from proprietary to open standards
Infrastructure & Platform Monitoring
- Kubernetes cluster monitoring with Prometheus Operator
- Docker container metrics and resource utilization tracking
- Cloud provider monitoring across AWS, Azure, and GCP
- Database performance monitoring for SQL and NoSQL systems
- Network monitoring and traffic analysis with SNMP and flow data
- Server hardware monitoring and predictive maintenance
- CDN performance monitoring and edge location analysis
- Load balancer and reverse proxy monitoring
- Storage system monitoring and capacity forecasting
Chaos Engineering & Reliability Testing
- Chaos Monkey and Gremlin fault injection strategies
- Failure mode identification and resilience testing
- Circuit breaker pattern implementation and monitoring
- Disaster recovery testing and validation procedures
- Load testing integration with monitoring systems
- Dependency failure simulation and cascading failure prevention
- Recovery time objective (RTO) and recovery point objective (RPO) validation
- System resilience scoring and improvement recommendations
- Automated chaos experiments and safety controls
Custom Dashboards & Visualization
- Executive dashboard creation for business stakeholders
- Real-time operational dashboards for engineering teams
- Custom Grafana plugins and panel development
- Multi-tenant dashboard design and access control
- Mobile-responsive monitoring interfaces
- Embedded analytics and white-label monitoring solutions
- Data visualization best practices and user experience design
- Interactive dashboard development with drill-down capabilities
- Automated report generation and scheduled delivery
Observability as Code & Automation
- Infrastructure as Code for monitoring stack deployment
- Terraform modules for observability infrastructure
- Ansible playbooks for monitoring agent deployment
- GitOps workflows for dashboard and alert management
- Configuration management and version control strategies
- Automated monitoring setup for new services
- CI/CD integration for observability pipeline testing
- Policy as Code for compliance and governance
- Self-healing monitoring infrastructure design
Cost Optimization & Resource Management
- Monitoring cost analysis and optimization strategies
- Data retention policy optimization for storage costs
- Sampling rate tuning for high-volume telemetry data
- Multi-tier storage strategies for historical data
- Resource allocation optimization for monitoring infrastructure
- Vendor cost comparison and migration planning
- Open source vs commercial tool evaluation
- ROI analysis for observability investments
- Budget forecasting and capacity planning
Enterprise Integration & Compliance
- SOC2, PCI DSS, and HIPAA compliance monitoring requirements
- Active Directory and SAML integration for monitoring access
- Multi-tenant monitoring architectures and data isolation
- Audit trail generation and compliance reporting automation
- Data residency and sovereignty requirements for global deployments
- Integration with enterprise ITSM tools (ServiceNow, Jira Service Management)
- Corporate firewall and network security policy compliance
- Backup and disaster recovery for monitoring infrastructure
- Change management processes for monitoring configurations
AI & Machine Learning Integration
- Anomaly detection using statistical models and machine learning algorithms
- Predictive analytics for capacity planning and resource forecasting
- Root cause analysis automation using correlation analysis and pattern recognition
- Intelligent alert clustering and noise reduction using unsupervised learning
- Time series forecasting for proactive scaling and maintenance scheduling
- Natural language processing for log analysis and error categorization
- Automated baseline establishment and drift detection for system behavior
- Performance regression detection using statistical change point analysis
- Integration with MLOps pipelines for model monitoring and observability
Behavioral Traits
- Prioritizes production reliability and system stability over feature velocity
- Implements comprehensive monitoring before issues occur, not after
- Focuses on actionable alerts and meaningful metrics over vanity metrics
- Emphasizes correlation between business impact and technical metrics
- Considers cost implications of monitoring and observability solutions
- Uses data-driven approaches for capacity planning and optimization
- Implements gradual rollouts and canary monitoring for changes
- Documents monitoring rationale and maintains runbooks religiously
- Stays current with emerging observability tools and practices
- Balances monitoring coverage with system performance impact
Knowledge Base
- Installed collector/SDK versions and current primary documentation
- Modern SRE practices and reliability engineering patterns with Google SRE methodology
- Enterprise monitoring architectures and scalability considerations for Fortune 500 companies
- Cloud-native observability patterns and Kubernetes monitoring with service mesh integration
- Security monitoring and compliance requirements (SOC2, PCI DSS, HIPAA, GDPR)
- Machine learning applications in anomaly detection, forecasting, and automated root cause analysis
- Multi-cloud and hybrid monitoring strategies across AWS, Azure, GCP, and on-premises
- Developer experience optimization for observability tooling and shift-left monitoring
- Incident response best practices, post-incident analysis, and blameless postmortem culture
- Cost-effective monitoring strategies scaling from startups to enterprises with budget optimization
- OpenTelemetry ecosystem and vendor-neutral observability standards
- Edge computing and IoT device monitoring at scale
- Serverless and event-driven architecture observability patterns
- Container security monitoring and runtime threat detection
- Business intelligence integration with technical monitoring for executive reporting
Response Approach
- Analyze monitoring requirements for comprehensive coverage and business alignment
- Design observability architecture with appropriate tools and data flow
- Implement production-ready monitoring with proper alerting and dashboards
- Include cost optimization and resource efficiency considerations
- Consider compliance and security implications of monitoring data
- Document monitoring strategy and provide operational runbooks
- Implement gradual rollout with monitoring validation at each stage
- Provide incident response procedures and escalation workflows
Example Interactions
- "Design a comprehensive monitoring strategy for a microservices architecture with 50+ services"
- "Implement distributed tracing for a complex e-commerce platform handling 1M+ daily transactions"
- "Set up cost-effective log management for a high-traffic application generating 10TB+ daily logs"
- "Create SLI/SLO framework with error budget tracking for API services with 99.9% availability target"
- "Build real-time alerting system with intelligent noise reduction for 24/7 operations team"
- "Implement chaos engineering with monitoring validation for Netflix-scale resilience testing"
- "Design executive dashboard showing business impact of system reliability and revenue correlation"
- "Set up compliance monitoring for SOC2 and PCI requirements with automated evidence collection"
- "Optimize monitoring costs while maintaining comprehensive coverage for startup scaling to enterprise"
- "Create automated incident response workflows with runbook integration and Slack/PagerDuty escalation"
- "Build multi-region observability architecture with data sovereignty compliance"
- "Implement machine learning-based anomaly detection for proactive issue identification"
- "Design observability strategy for serverless architecture with AWS Lambda and API Gateway"
- "Create custom metrics pipeline for business KPIs integrated with technical monitoring"
Worked example and prerequisites
Input: a checkout API has 100,000 eligible requests in a defined window, with 120 failures, and an agreed 99.9% success SLO. Observed success is 99.88%; the 0.12% error ratio consumes budget at 1.2 times the permitted 0.1% ratio for that window. Record which requests count, how retries are handled and whether failures are measured at the user or server boundary.
Before adding an alert, verify that both counters cover the same population, test no-traffic and missing-series behavior, and attach a runbook and owner. Expected: an operator can identify the affected journey and next check without exposing request contents. This arithmetic example is not a prescribed paging threshold or a claim about a live service.
Limitations
- Missing telemetry is unknown health, not automatically zero errors.
- Sampled traces cannot directly supply an unbiased total request/error denominator without a justified estimator.
- A dashboard or vendor integration does not establish compliance; access, retention and actual exported payloads still need review.
- Alert delivery, production instrumentation, chaos experiments and incident messages require authorization for the specific environment and action.
On-Call Handoff Patterns
Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts.
When to Use This Skill
- Transitioning on-call responsibilities
- Writing shift handoff summaries
- Documenting ongoing investigations
- Establishing on-call rotation procedures
- Improving handoff quality
- Onboarding new on-call engineers
Core Concepts
1. Handoff Components
| Component | Purpose |
|---|---|
| Active Incidents | What's currently broken |
| Ongoing Investigations | Issues being debugged |
| Recent Changes | Deployments, configs |
| Known Issues | Workarounds in place |
| Upcoming Events | Maintenance, releases |
2. Handoff Timing
Recommended: 30 min overlap between shifts
Outgoing:
├── 15 min: Write handoff document
└── 15 min: Sync call with incoming
Incoming:
├── 15 min: Review handoff document
├── 15 min: Sync call with outgoing
└── 5 min: Verify alerting setup
Templates and detailed worked examples
Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.
Troubleshooting
Incoming engineer misses a critical issue because the handoff document was incomplete. Use the outgoing checklist as a gate: do not mark handoff complete until every section has at least one entry (or an explicit "none"). Make incomplete handoffs a blameless postmortem action item.
A 30-minute sync call is not possible due to timezone gaps. Fall back to the async quick handoff template (Template 2). Supplement with a short Loom or voice memo walking through the watch list. Ensure the incoming engineer has a direct contact method if they have follow-up questions.
The incoming engineer inherits a mid-incident and is immediately overwhelmed. Use the incident handoff template (Template 3) specifically. The outgoing engineer should remain available on Slack for 15 minutes after handoff, even if off-call, to answer clarifying questions.
On-call handoff documents are inconsistently formatted across teams. Adopt the shift handoff template organization-wide and store completed handoffs in a shared location (wiki, Notion, Confluence). Link each handoff from the on-call schedule entry in PagerDuty.
Incoming engineer cannot verify their alerting is working before the outgoing engineer logs off. Add a standard step: outgoing engineer fires a test alert and confirms incoming engineer receives it in PagerDuty and Slack before ending the overlap window.
Related Skills
- incident-runbook-templates — Runbooks for the incidents that are handed over mid-response
- postmortem-writing — Turn resolved incidents from the shift into structured postmortems
OpenTelemetry
Adopt vendor-neutral telemetry with consistent instrumentation across services.
Prerequisites
- Application services running in containers or on VMs
- Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver)
- Backend for metrics (Prometheus, Mimir, or OTLP receiver)
- Kubernetes cluster (for collector deployment) or VM with systemd
- Network access from services to collector, and collector to backends
Core Workflow
- Define semantic conventions for services, environments, and versions.
- Add SDK or auto-instrumentation in each service.
- Run an OpenTelemetry Collector to receive, transform, and export telemetry.
- Validate cardinality and sampling to control cost.
- Create golden signals dashboards and alerting from collected data.
Collector Configuration Example
This legacy 0.98-style configuration is an adaptation starting point, not a current production deployment. Pin a supported Collector distribution and matching SDK/operator versions, list available components, and validate the final config with that binary before rollout. New versions change internal metrics schemas and exporter names. Restrict receiver/health/metrics network access; cross-host production traffic needs authenticated TLS, not the plaintext examples. Scraping Kubernetes needs a scoped service account/RBAC and correct namespace; hostmetrics inside a container measures its available view unless host mounts/permissions are deliberately configured. Redact sensitive attributes, baggage, URLs and bodies before export; sampling is not a privacy control.
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
# Scrape Prometheus endpoints
prometheus:
config:
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+);([0-9]+)
replacement: $$1:$$2 # IPv4 example; adapt/test IPv6 bracket formatting
# Host metrics for infrastructure monitoring
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
memory: {}
disk: {}
network: {}
load: {}
processors:
batch:
send_batch_size: 1024
timeout: 5s
memory_limiter:
check_interval: 1s
limit_mib: 384
spike_limit_mib: 96 # Leave process/runtime headroom below the 512Mi container limit
attributes:
actions:
- key: deployment.environment
value: production
action: upsert
# Drop high-cardinality attributes to control cost
filter/drop-debug:
traces:
span:
- 'attributes["http.request.header.x-debug"] == "true"'
# Reduce cardinality on URL paths
transform/normalize-routes:
trace_statements:
- context: span
statements:
- replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
- replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")
# Resource detection for cloud environments
resourcedetection:
detectors: [env, system, gcp, aws, azure]
timeout: 5s
exporters:
# Send traces to Tempo/Jaeger
otlp/traces:
endpoint: tempo:4317
tls:
insecure: true
# Send metrics to Prometheus via remote write
prometheusremotewrite:
endpoint: http://mimir:9009/api/v1/push
tls:
insecure: true
# Send logs to Loki
otlp/logs:
endpoint: loki:4317
tls:
insecure: true
# Debug exporter for development
debug:
verbosity: basic
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
telemetry:
logs:
level: info
metrics:
address: 0.0.0.0:8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, filter/drop-debug, transform/normalize-routes, attributes, batch]
exporters: [otlp/traces]
metrics:
receivers: [otlp, prometheus, hostmetrics]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [otlp/logs]
Collector Kubernetes Deployment
# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
namespace: observability
spec:
replicas: 2
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
containers:
- name: collector
image: otel/opentelemetry-collector-contrib:0.98.0
args: ["--config=/etc/otel/config.yaml"]
ports:
- containerPort: 4317
name: otlp-grpc
- containerPort: 4318
name: otlp-http
- containerPort: 8888
name: metrics
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
volumeMounts:
- name: config
mountPath: /etc/otel
livenessProbe:
httpGet:
path: /
port: 13133
readinessProbe:
httpGet:
path: /
port: 13133
volumes:
- name: config
configMap:
name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: observability
spec:
selector:
app: otel-collector
ports:
- name: otlp-grpc
port: 4317
targetPort: 4317
- name: otlp-http
port: 4318
targetPort: 4318
- name: metrics
port: 8888
targetPort: 8888
Python SDK Instrumentation
# tracing_setup.py
"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os
def init_telemetry(service_name: str, service_version: str):
"""Initialize OTel SDK with traces and metrics."""
resource = Resource.create({
"service.name": service_name,
"service.version": service_version,
"deployment.environment": os.getenv("DEPLOY_ENV", "development"),
})
# Traces
trace_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
trace.set_tracer_provider(tracer_provider)
# Metrics
metric_exporter = OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=15000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
# Auto-instrument common libraries
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()
return trace.get_tracer(service_name), metrics.get_meter(service_name)
# Usage example
tracer, meter = init_telemetry("order-service", "1.2.0")
# Custom span
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("order.total", total)
# ... business logic ...
# Custom metric
request_counter = meter.create_counter(
"app.requests",
description="Total application requests",
)
request_counter.add(1, {"route": "/api/orders", "method": "POST"})
Node.js SDK Instrumentation
// tracing.cjs
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.cjs app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");
const resource = new Resource({
[ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
[ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
"deployment.environment": process.env.DEPLOY_ENV || "development",
});
const sdk = new NodeSDK({
resource,
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
exportIntervalMillis: 15000,
}),
instrumentations: [
getNodeAutoInstrumentations({
"@opentelemetry/instrumentation-http": {
ignoreIncomingRequestHook: (request) => ["/health", "/ready"].includes((request.url || "").split("?")[0]),
},
"@opentelemetry/instrumentation-express": { enabled: true },
"@opentelemetry/instrumentation-pg": { enabled: true },
"@opentelemetry/instrumentation-redis": { enabled: true },
}),
],
});
sdk.start();
process.on("SIGTERM", async () => {
try { await sdk.shutdown(); } catch { process.exitCode = 1; }
});
Auto-Instrumentation with Kubernetes Operator
# otel-auto-instrumentation.yaml
# Install the OTel Operator first:
# helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
# --namespace observability --create-namespace
# Define instrumentation for Python services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: python-instrumentation
namespace: default
spec:
exporter:
endpoint: http://otel-collector.observability:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.25"
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
env:
- name: OTEL_PYTHON_LOG_CORRELATION
value: "true"
---
# Define instrumentation for Node.js services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nodejs-instrumentation
namespace: default
spec:
exporter:
endpoint: http://otel-collector.observability:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.25"
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.49.1
To instrument a pod, add the annotation:
# For Python:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
# For Node.js:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "true"
Contents
When to Use This Skill
- Debugging latency across microservices
- Standardizing observability data model and naming
- Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends
- Building SLO dashboards with trace-to-log correlation
- Instrumenting Python or Node.js applications with tracing and metrics
- Setting up auto-instrumentation for existing services without code changes
Limitations
- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.
Example
git status && git diff --stat
kubectl diff -f manifest.yaml
Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.
Postmortem Writing
Comprehensive guide to writing effective, blameless postmortems that drive organizational learning and prevent incident recurrence.
When to Use This Skill
- Conducting post-incident reviews
- Writing postmortem documents
- Facilitating blameless postmortem meetings
- Identifying root causes and contributing factors
- Creating actionable follow-up items
- Building organizational learning culture
Core Concepts
1. Blameless Culture
| Blame-Focused | Blameless |
|---|---|
| "Who caused this?" | "What conditions allowed this?" |
| "Someone made a mistake" | "The system allowed this mistake" |
| Punish individuals | Improve systems |
| Hide information | Share learnings |
| Fear of speaking up | Psychological safety |
2. Postmortem Triggers
- SEV1 or SEV2 incidents
- Customer-facing outages > 15 minutes
- Data loss or security incidents
- Near-misses that could have been severe
- Novel failure modes
- Incidents requiring unusual intervention
Quick Start
Postmortem Timeline
Day 0: Incident occurs
Day 1-2: Draft postmortem document
Day 3-5: Postmortem meeting
Day 5-7: Finalize document, create tickets
Week 2+: Action item completion
Quarterly: Review patterns across incidents
Templates and detailed worked examples
Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.
Template 2: 5 Whys Analysis
# 5 Whys Analysis: [Incident]
## Problem Statement
Payment service experienced 47-minute outage due to database connection exhaustion.
## Analysis
### Why #1: Why did the service fail?
**Answer**: Database connections were exhausted, causing all new requests to fail.
**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.
---
### Why #2: Why were database connections exhausted?
**Answer**: Each incoming request opened a new database connection instead of using the connection pool.
**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.
---
### Why #3: Why did the code bypass the connection pool?
**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.
**Evidence**: PR #1234 shows the change, made while fixing a different bug.
---
### Why #4: Why wasn't this caught in code review?
**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.
**Evidence**: Review comments only discuss business logic.
---
### Why #5: Why isn't there a safety net for this type of change?
**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.
**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.
## Root Causes Identified
1. **Primary**: Missing automated tests for infrastructure behavior
2. **Secondary**: Insufficient documentation of architectural patterns
3. **Tertiary**: Code review checklist doesn't include infrastructure considerations
## Systemic Improvements
| Root Cause | Improvement | Type |
| ------------- | --------------------------------- | ---------- |
| Missing tests | Add infrastructure behavior tests | Prevention |
| Missing docs | Document connection patterns | Prevention |
| Review gaps | Update review checklist | Detection |
| No canary | Implement canary deployments | Mitigation |
Template 3: Quick Postmortem (Minor Incidents)
# Quick Postmortem: [Brief Title]
**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3
## What Happened
API latency spiked to 5s due to cache miss storm after cache flush.
## Timeline
- 10:00 - Cache flush initiated for config update
- 10:02 - Latency alerts fire
- 10:05 - Identified as cache miss storm
- 10:08 - Enabled cache warming
- 10:12 - Latency normalized
## Root Cause
Full cache flush for minor config update caused thundering herd.
## Fix
- Immediate: Enabled cache warming
- Long-term: Implement partial cache invalidation (ENG-999)
## Lessons
Don't full-flush cache in production; use targeted invalidation.
Facilitation Guide
Running a Postmortem Meeting
## Meeting Structure (60 minutes)
### 1. Opening (5 min)
- Remind everyone of blameless culture
- "We're here to learn, not to blame"
- Review meeting norms
### 2. Timeline Review (15 min)
- Walk through events chronologically
- Ask clarifying questions
- Identify gaps in timeline
### 3. Analysis Discussion (20 min)
- What failed?
- Why did it fail?
- What conditions allowed this?
- What would have prevented it?
### 4. Action Items (15 min)
- Brainstorm improvements
- Prioritize by impact and effort
- Assign owners and due dates
### 5. Closing (5 min)
- Summarize key learnings
- Confirm action item owners
- Schedule follow-up if needed
## Facilitation Tips
- Keep discussion on track
- Redirect blame to systems
- Encourage quiet participants
- Document dissenting views
- Time-box tangents
Anti-Patterns to Avoid
| Anti-Pattern | Problem | Better Approach |
|---|---|---|
| Blame game | Shuts down learning | Focus on systems |
| Shallow analysis | Doesn't prevent recurrence | Ask "why" 5 times |
| No action items | Waste of time | Always have concrete next steps |
| Unrealistic actions | Never completed | Scope to achievable tasks |
| No follow-up | Actions forgotten | Track in ticketing system |
Best Practices
Do's
- Start immediately - Memory fades fast
- Be specific - Exact times, exact errors
- Include graphs - Visual evidence
- Assign owners - No orphan action items
- Share widely - Organizational learning
Don'ts
- Don't name and shame - Ever
- Don't skip small incidents - They reveal patterns
- Don't make it a blame doc - That kills learning
- Don't create busywork - Actions should be meaningful
- Don't skip follow-up - Verify actions completed
Prometheus Configuration
Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.
Purpose
Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.
When to Use
- Set up Prometheus monitoring
- Configure metric scraping
- Create recording rules
- Design alert rules
- Implement service discovery
Detailed patterns and worked examples
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
Best Practices
- Use consistent naming for metrics (prefix_name_unit)
- Set appropriate scrape intervals (15-60s typical)
- Use recording rules for expensive queries
- Implement high availability (multiple Prometheus instances)
- Configure retention based on storage capacity
- Use relabeling for metric cleanup
- Monitor Prometheus itself
- Implement federation for large deployments
- Use Thanos/Cortex for long-term storage
- Document custom metrics
Troubleshooting
Check scrape targets:
curl http://localhost:9090/api/v1/targets
Check configuration:
curl http://localhost:9090/api/v1/status/config
Test query:
curl 'http://localhost:9090/api/v1/query?query=up'
Related Skills
grafana-dashboards- For visualizationslo-implementation- For SLO monitoringdistributed-tracing- For request tracing
Service Mesh Observability
Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.
When to Use This Skill
- Setting up distributed tracing across services
- Implementing service mesh metrics and dashboards
- Debugging latency and error issues
- Defining SLOs for service communication
- Visualizing service dependencies
- Troubleshooting mesh connectivity
Core Concepts
1. Three Pillars of Observability
┌─────────────────────────────────────────────────────┐
│ Observability │
├─────────────────┬─────────────────┬─────────────────┤
│ Metrics │ Traces │ Logs │
│ │ │ │
│ • Request rate │ • Span context │ • Access logs │
│ • Error rate │ • Latency │ • Error details │
│ • Latency P50 │ • Dependencies │ • Debug info │
│ • Saturation │ • Bottlenecks │ • Audit trail │
└─────────────────┴─────────────────┴─────────────────┘
2. Golden Signals for Mesh
| Signal | Description | Alert Threshold |
|---|---|---|
| Latency | Request duration P50, P99 | P99 > 500ms |
| Traffic | Requests per second | Anomaly detection |
| Errors | 5xx error rate | > 1% |
| Saturation | Resource utilization | > 80% |
Templates and detailed worked examples
Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.
Best Practices
Do's
- Sample appropriately - 100% in dev, 1-10% in prod
- Use trace context - Propagate headers consistently
- Set up alerts - For golden signals
- Correlate metrics/traces - Use exemplars
- Retain strategically - Hot/cold storage tiers
Don'ts
- Don't over-sample - Storage costs add up
- Don't ignore cardinality - Limit label values
- Don't skip dashboards - Visualize dependencies
- Don't forget costs - Monitor observability costs
SLO Implementation
Framework for defining and implementing Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets.
Purpose
Implement measurable reliability targets using SLIs, SLOs, and error budgets to balance reliability with innovation velocity.
When to Use
- Define service reliability targets
- Measure user-perceived reliability
- Implement error budgets
- Create SLO-based alerts
- Track reliability goals
SLI/SLO/SLA Hierarchy
SLA (Service Level Agreement)
↓ Contract with customers
SLO (Service Level Objective)
↓ Internal reliability target
SLI (Service Level Indicator)
↓ Actual measurement
Defining SLIs
Common SLI Types
1. Availability SLI
# Successful requests / Total requests
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
2. Latency SLI
# Requests below latency threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
3. Durability SLI
# Successful writes / Total writes
sum(storage_writes_successful_total)
/
sum(storage_writes_total)
Setting SLO Targets
Availability SLO Examples
| SLO % | Downtime/Month | Downtime/Year |
|---|---|---|
| 99% | 7.2 hours | 3.65 days |
| 99.9% | 43.2 minutes | 8.76 hours |
| 99.95% | 21.6 minutes | 4.38 hours |
| 99.99% | 4.32 minutes | 52.56 minutes |
Choose Appropriate SLOs
Consider:
- User expectations
- Business requirements
- Current performance
- Cost of reliability
- Competitor benchmarks
Example SLOs:
slos:
- name: api_availability
target: 99.9
window: 28d
sli: |
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
- name: api_latency_p95
target: 99
window: 28d
sli: |
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
Error Budget Calculation
Error Budget Formula
Error Budget = 1 - SLO Target
Example:
- SLO: 99.9% availability
- Error Budget: 0.1% = 43.2 minutes/month
- Current Error: 0.05% = 21.6 minutes/month
- Remaining Budget: 50%
Error Budget Policy
error_budget_policy:
- remaining_budget: 100%
action: Normal development velocity
- remaining_budget: 50%
action: Consider postponing risky changes
- remaining_budget: 10%
action: Freeze non-critical changes
- remaining_budget: 0%
action: Feature freeze, focus on reliability
SLO Implementation
Prometheus Recording Rules
# SLI Recording Rules
groups:
- name: sli_rules
interval: 30s
rules:
# Availability SLI
- record: sli:http_availability:ratio
expr: |
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
# Latency SLI (requests < 500ms)
- record: sli:http_latency:ratio
expr: |
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
- name: slo_rules
interval: 5m
rules:
# SLO compliance (1 = meeting SLO, 0 = violating)
- record: slo:http_availability:compliance
expr: sli:http_availability:ratio >= bool 0.999
- record: slo:http_latency:compliance
expr: sli:http_latency:ratio >= bool 0.99
# Error budget remaining (percentage)
- record: slo:http_availability:error_budget_remaining
expr: |
(sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100
# Error budget burn rate
- record: slo:http_availability:burn_rate_5m
expr: |
(1 - (
sum(rate(http_requests_total{status!~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
)) / (1 - 0.999)
SLO Alerting Rules
groups:
- name: slo_alerts
interval: 1m
rules:
# Fast burn: 14.4x rate, 1 hour window
# Consumes 2% error budget in 1 hour
- alert: SLOErrorBudgetBurnFast
expr: |
slo:http_availability:burn_rate_1h > 14.4
and
slo:http_availability:burn_rate_5m > 14.4
for: 2m
labels:
severity: critical
annotations:
summary: "Fast error budget burn detected"
description: "Error budget burning at {{ $value }}x rate"
# Slow burn: 6x rate, 6 hour window
# Consumes 5% error budget in 6 hours
- alert: SLOErrorBudgetBurnSlow
expr: |
slo:http_availability:burn_rate_6h > 6
and
slo:http_availability:burn_rate_30m > 6
for: 15m
labels:
severity: warning
annotations:
summary: "Slow error budget burn detected"
description: "Error budget burning at {{ $value }}x rate"
# Error budget exhausted
- alert: SLOErrorBudgetExhausted
expr: slo:http_availability:error_budget_remaining < 0
for: 5m
labels:
severity: critical
annotations:
summary: "SLO error budget exhausted"
description: "Error budget remaining: {{ $value }}%"
SLO Dashboard
Grafana Dashboard Structure:
┌────────────────────────────────────┐
│ SLO Compliance (Current) │
│ ✓ 99.95% (Target: 99.9%) │
├────────────────────────────────────┤
│ Error Budget Remaining: 65% │
│ ████████░░ 65% │
├────────────────────────────────────┤
│ SLI Trend (28 days) │
│ [Time series graph] │
├────────────────────────────────────┤
│ Burn Rate Analysis │
│ [Burn rate by time window] │
└────────────────────────────────────┘
Example Queries:
# Current SLO compliance
sli:http_availability:ratio * 100
# Error budget remaining
slo:http_availability:error_budget_remaining
# Days until error budget exhausted (at current burn rate)
(slo:http_availability:error_budget_remaining / 100)
*
28
/
(1 - sli:http_availability:ratio) * (1 - 0.999)
Additional patterns and templates
More detailed templates and worked examples live in references/details.md. Read that file for the full pattern library.
Sponsors
Details
- Marketplace
- marslanalvi/SkillsStack
- Source
- plugins/devops/observability-sre
- Category
- Devops
- Keywords
- devops, observability-sre
More from marslanalvi/SkillsStack
All 45- engineering-architecture-apiv1.0.0
- engineering-code-review-qualityv1.0.0
- engineering-databasesv1.0.0
- engineering-debuggingv1.0.0
- engineering-dev-workflowv1.0.0
- engineering-frontendv1.0.0
- engineering-game-developmentv1.0.0
- engineering-git-githubv1.0.0
- engineering-javascript-typescriptv1.0.0
- engineering-jvmv1.0.0
- engineering-mobilev1.0.0
- engineering-other-languagesv1.0.0
- engineering-pythonv1.0.0
- engineering-systems-languagesv1.0.0
- engineering-testingv1.0.0
- marketing-content-socialv1.0.0
- marketing-growth-adsv1.0.0
- marketing-seov1.0.0
- productivity-automation-personalv1.0.0
- productivity-documents-officev1.0.0
- productivity-thinking-planningv1.0.0
- productivity-writing-communicationv1.0.0
- security-appsec-reviewv1.0.0
- security-blockchain-web3v1.0.0