marslanalvi avatar

devops-observability-sre

Monitoring, tracing, logging, SLOs, incident response and on-call

version
1.0.0
GitHub stars
5

Install

/plugin install devops-observability-sre@skills-stack

Run it inside Claude Code after adding the marketplace with /plugin marketplace add marslanalvi/SkillsStack.

Contents

Discovered from source

Skills 16

Azure Monitor OpenTelemetry SDK for TypeScript

Auto-instrument Node.js applications with distributed tracing, metrics, and logs.

Version and data boundaries

This document describes the existing @azure/monitor-opentelemetry distro. Current Microsoft guidance also offers @microsoft/opentelemetry for new adoption; do not initialize both. Match all OpenTelemetry package versions and use that version’s logger-provider constructor/processor API. Configure approved telemetry destinations, redact PII/secrets, and protect offline storage; automatic instrumentation may export sensitive request/log data. See Microsoft enablement guidance.

Installation

# Distro (recommended - auto-instrumentation)
npm install @azure/monitor-opentelemetry

# Low-level exporters (custom OpenTelemetry setup)
npm install @azure/monitor-opentelemetry-exporter

# Custom logs ingestion
npm install @azure/monitor-ingestion

Environment Variables

APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=...;IngestionEndpoint=...

Quick Start (Auto-Instrumentation)

IMPORTANT: Call useAzureMonitor() BEFORE importing other modules.

import { useAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
  }
});

// Static ESM imports are hoisted; load the app after initialization.
const { default: express } = await import("express");
const app = express();

ESM Support (Node.js 18.19+)

node --import @azure/monitor-opentelemetry/loader ./dist/index.js

package.json:

{
  "scripts": {
    "start": "node --import @azure/monitor-opentelemetry/loader ./dist/index.js"
  }
}

Full Configuration

import { useAzureMonitor, AzureMonitorOpenTelemetryOptions } from "@azure/monitor-opentelemetry";
import { resourceFromAttributes } from "@opentelemetry/resources";

const options: AzureMonitorOpenTelemetryOptions = {
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
    storageDirectory: "/path/to/offline/storage",
    disableOfflineStorage: false
  },
  
  // Sampling
  samplingRatio: 1.0,  // 0-1, percentage of traces
  
  // Features
  enableLiveMetrics: true,
  enableStandardMetrics: true,
  enablePerformanceCounters: true,
  
  // Instrumentation libraries
  instrumentationOptions: {
    azureSdk: { enabled: true },
    http: { enabled: true },
    mongoDb: { enabled: true },
    mySql: { enabled: true },
    postgreSql: { enabled: true },
    redis: { enabled: true },
    bunyan: { enabled: false },
    winston: { enabled: false }
  },
  
  // Custom resource
  resource: resourceFromAttributes({ "service.name": "my-service" })
};

useAzureMonitor(options);

Custom Traces

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("my-tracer");

const span = tracer.startSpan("doWork");
try {
  span.setAttribute("component", "worker");
  span.setAttribute("operation.id", "42");
  span.addEvent("processing started");
  
  // Your work here
  
} catch (error) {
  span.recordException(error as Error);
  span.setStatus({ code: 2, message: (error as Error).message });
} finally {
  span.end();
}

Custom Metrics

import { metrics } from "@opentelemetry/api";

const meter = metrics.getMeter("my-meter");

// Counter
const counter = meter.createCounter("requests_total");
counter.add(1, { route: "/api/users", method: "GET" });

// Histogram
const histogram = meter.createHistogram("request_duration_ms");
histogram.record(150, { route: "/api/users" });

// Observable Gauge
const gauge = meter.createObservableGauge("active_connections");
gauge.addCallback((result) => {
  result.observe(getActiveConnections(), { pool: "main" });
});

Manual Exporter Setup

Trace Exporter

import { AzureMonitorTraceExporter } from "@azure/monitor-opentelemetry-exporter";
import { NodeTracerProvider, BatchSpanProcessor } from "@opentelemetry/sdk-trace-node";

const exporter = new AzureMonitorTraceExporter({
  connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});

const provider = new NodeTracerProvider({
  spanProcessors: [new BatchSpanProcessor(exporter)]
});

provider.register();

Metric Exporter

import { AzureMonitorMetricExporter } from "@azure/monitor-opentelemetry-exporter";
import { PeriodicExportingMetricReader, MeterProvider } from "@opentelemetry/sdk-metrics";
import { metrics } from "@opentelemetry/api";

const exporter = new AzureMonitorMetricExporter({
  connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});

const meterProvider = new MeterProvider({
  readers: [new PeriodicExportingMetricReader({ exporter })]
});

metrics.setGlobalMeterProvider(meterProvider);

Log Exporter

import { AzureMonitorLogExporter } from "@azure/monitor-opentelemetry-exporter";
import { BatchLogRecordProcessor, LoggerProvider } from "@opentelemetry/sdk-logs";
import { logs } from "@opentelemetry/api-logs";

const exporter = new AzureMonitorLogExporter({
  connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
});

const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(new BatchLogRecordProcessor(exporter));

logs.setGlobalLoggerProvider(loggerProvider);

Custom Logs Ingestion

import { DefaultAzureCredential } from "@azure/identity";
import { LogsIngestionClient, isAggregateLogsUploadError } from "@azure/monitor-ingestion";

const endpoint = "https://<dce>.ingest.monitor.azure.com";
const ruleId = "<data-collection-rule-id>";
const streamName = "Custom-MyTable_CL";

const client = new LogsIngestionClient(endpoint, new DefaultAzureCredential());

const logs = [
  {
    Time: new Date().toISOString(),
    Computer: "Server1",
    Message: "Application started",
    Level: "Information"
  }
];

try {
  await client.upload(ruleId, streamName, logs);
} catch (error) {
  if (isAggregateLogsUploadError(error)) {
    for (const uploadError of error.errors) {
      console.error("Failed logs:", uploadError.failedLogs);
    }
  }
}

Custom Span Processing

Enrich writable spans in onStart with span.setAttribute(). Do not mutate ReadableSpan.attributes or spanContext().traceFlags in onEnd: ended spans are read-only and this is not a reliable export filter. Use a supported sampler or a reviewed filtering exporter/processor pipeline for the pinned SDK version, and verify exported data.

Sampling

import { ApplicationInsightsSampler } from "@azure/monitor-opentelemetry-exporter";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

// Sample 75% of traces
const sampler = new ApplicationInsightsSampler(0.75);

const provider = new NodeTracerProvider({ sampler });

Shutdown

import { useAzureMonitor, shutdownAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor();

// On application shutdown
process.on("SIGTERM", async () => {
  await shutdownAzureMonitor();
  process.exit(0);
});

Key Types

import {
  useAzureMonitor,
  shutdownAzureMonitor,
  AzureMonitorOpenTelemetryOptions,
  InstrumentationOptions
} from "@azure/monitor-opentelemetry";

import {
  AzureMonitorTraceExporter,
  AzureMonitorMetricExporter,
  AzureMonitorLogExporter,
  ApplicationInsightsSampler,
  AzureMonitorExporterOptions
} from "@azure/monitor-opentelemetry-exporter";

import {
  LogsIngestionClient,
  isAggregateLogsUploadError
} from "@azure/monitor-ingestion";

Best Practices

  1. Call useAzureMonitor() first - Before importing other modules
  2. Use ESM loader for ESM projects - --import @azure/monitor-opentelemetry/loader
  3. Enable offline storage - For reliable telemetry in disconnected scenarios
  4. Set sampling ratio - For high-traffic applications
  5. Add custom dimensions - Use span processors for enrichment
  6. Graceful shutdown - Call shutdownAzureMonitor() to flush telemetry

When to Use

This skill is applicable to execute the workflow or actions described in the overview.

Limitations

  • Use this skill only when the task clearly matches the scope described above.
  • Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
  • Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.

Canary Watch — Post-Deploy Monitoring

When to Use

  • After deploying to production or staging
  • After merging a risky PR
  • When you want to verify a fix actually fixed it
  • Continuous monitoring during a launch window
  • After dependency upgrades

How It Works

Monitors a deployed URL for regressions. Runs in a loop until stopped or until the watch window expires.

What It Watches

1. HTTP Status — is the page returning 200?
2. Console Errors — new errors that weren't there before?
3. Network Failures — failed API calls, 5xx responses?
4. Performance — LCP/CLS/INP regression vs baseline?
5. Content — did key elements disappear? (h1, nav, footer, CTA)
6. API Health — are critical endpoints responding within SLA?
7. Static Assets — are JS, CSS, image, and font requests returning 2xx/3xx with expected content types?
8. SSE Streams — do event-stream endpoints connect and receive an initial event or heartbeat?

Watch Modes

Quick check (default): single pass, report results

/canary-watch https://myapp.com

Sustained watch: check every N minutes for M hours

/canary-watch https://myapp.com --interval 5m --duration 2h

Diff mode: compare staging vs production

/canary-watch --compare https://staging.myapp.com https://myapp.com

Alert Thresholds

critical:  # immediate alert
  - HTTP status != 200
  - Console error count > 5 (new errors only)
  - LCP > 4s
  - API endpoint returns 5xx
  - Static asset returns 4xx/5xx
  - SSE endpoint cannot connect or drops before first heartbeat

warning:   # flag in report
  - LCP increased > 500ms from baseline
  - CLS > 0.1
  - New console warnings
  - Response time > 2x baseline
  - Static asset content type changed unexpectedly
  - SSE heartbeat latency > 2x baseline

info:      # log only
  - Minor performance variance
  - New network requests (third-party scripts added?)

Notifications

When a critical threshold is crossed:

  • Desktop notification (macOS/Linux)
  • Optional: Slack/Discord webhook
  • Log to ~/.claude/canary-watch.log

Output

## Canary Report — myapp.com — 2026-03-23 03:15 PST

### Status: HEALTHY ✓

| Check | Result | Baseline | Delta |
|-------|--------|----------|-------|
| HTTP | 200 ✓ | 200 | — |
| Console errors | 0 ✓ | 0 | — |
| LCP | 1.8s ✓ | 1.6s | +200ms |
| CLS | 0.01 ✓ | 0.01 | — |
| API /health | 145ms ✓ | 120ms | +25ms |
| Static assets | 42/42 ✓ | 42/42 | — |
| SSE /events | connected ✓ | connected | +80ms heartbeat |

### No regressions detected. Deploy is clean.

Integration

Pair with:

  • /browser-qa for pre-deploy verification
  • Hooks: add as a PostToolUse hook on git push to auto-check after deploys
  • CI: run in GitHub Actions after deploy step

Dashboard Builder

Use this when the task is to build a dashboard people can operate from.

The goal is not "show every metric." The goal is to answer:

  • is it healthy?
  • where is the bottleneck?
  • what changed?
  • what action should someone take?

When to Use

  • "Build a Kafka monitoring dashboard"
  • "Create a Grafana dashboard for Elasticsearch"
  • "Make a SigNoz dashboard for this service"
  • "Turn this metrics list into a real operational dashboard"

Guardrails

  • do not start from visual layout; start from operator questions
  • do not include every available metric just because it exists
  • do not mix health, throughput, and resource panels without structure
  • do not ship panels without titles, units, and sane thresholds

Workflow

1. Define the operating questions

Organize around:

  • health / availability
  • latency / performance
  • throughput / volume
  • saturation / resources
  • service-specific risk

2. Study the target platform schema

Inspect existing dashboards first:

  • JSON structure
  • query language
  • variables
  • threshold styling
  • section layout

3. Build the minimum useful board

Recommended structure:

  1. overview
  2. performance
  3. resources
  4. service-specific section

4. Cut vanity panels

Every panel should answer a real question. If it does not, remove it.

Example Panel Sets

Elasticsearch

  • cluster health
  • shard allocation
  • search latency
  • indexing rate
  • JVM heap / GC

Kafka

  • broker count
  • under-replicated partitions
  • messages in / out
  • consumer lag
  • disk and network pressure

API gateway / ingress

  • request rate
  • p50 / p95 / p99 latency
  • error rate
  • upstream health
  • active connections

Quality Checklist

  • valid dashboard JSON
  • clear section grouping
  • titles and units are present
  • thresholds/status colors are meaningful
  • variables exist for common filters
  • default time range and refresh are sensible
  • no vanity panels with no operator value

Related Skills

  • research-ops
  • backend-patterns
  • terminal-ops

Datadog

Monitor infrastructure and applications with Datadog's unified observability platform.

When to Use This Skill

Use this skill when:

  • Implementing enterprise-grade monitoring
  • Setting up APM and distributed tracing
  • Creating unified dashboards for infrastructure and apps
  • Configuring intelligent alerting
  • Monitoring cloud infrastructure (AWS, Azure, GCP)

Prerequisites

  • Datadog account and API key
  • Agent installation access
  • Application code access for APM

Confirm account/site, approved telemetry and cost/retention limits before enabling logs, process data or APM. Redact credentials, request bodies, user/order IDs and sensitive tags before egress; distribution metrics do not eliminate cardinality costs. Docker socket access is privileged even with a read-only bind. Use a pre-created scoped Secret with the chart's existing-secret setting rather than passing a real API key through --set/history/Helm release values. Pin chart/agent/tracer versions and validate their configuration schemas. Inline SDK fragments are partial: Python metrics needs time and the version-appropriate metric intake enum, Go needs a real request context/user ID, and tracer.configure parameters vary by ddtrace major. Verify metric names/units and no-data/zero-denominator behavior against actual instrumentation before activating monitors or external notifications.

Agent Installation

Linux

# Install agent
curl -fL https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh -o /tmp/datadog-agent-install.sh
# Review and pin the full installer before executing separately with protected credentials.

# Or via package manager
apt-get update && apt-get install datadog-agent

# Configure API key
echo "api_key: YOUR_API_KEY" >> /etc/datadog-agent/datadog.yaml

# Start agent
systemctl start datadog-agent
systemctl enable datadog-agent

Docker

# docker-compose.yml
version: '3.8'

services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:7
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=datadoghq.com
      - DD_LOGS_ENABLED=true
      - DD_APM_ENABLED=true
      - DD_PROCESS_AGENT_ENABLED=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    ports:
      - "127.0.0.1:8126:8126"  # APM
      - "127.0.0.1:8125:8125/udp"  # DogStatsD

Kubernetes

# Using Helm
helm repo add datadog https://helm.datadoghq.com

helm install datadog datadog/datadog \
  --set datadog.apiKey=${DD_API_KEY} \
  --set datadog.site=datadoghq.com \
  --set datadog.logs.enabled=true \
  --set datadog.apm.portEnabled=true \
  --set datadog.processAgent.enabled=true \
  --namespace datadog \
  --create-namespace

Agent Configuration

# /etc/datadog-agent/datadog.yaml
api_key: YOUR_API_KEY
site: datadoghq.com

# Hostname
hostname: myserver.example.com

# Tags applied to all metrics
tags:
  - env:production
  - service:myapp
  - team:platform

# Log collection
logs_enabled: true

# APM
apm_config:
  enabled: true
  apm_dd_url: https://trace.agent.datadoghq.com

# Process monitoring
process_config:
  enabled: true

# Container monitoring
container_collect_all: true
docker_labels_as_tags:
  app: service
  environment: env

Integration Configuration

MySQL

# /etc/datadog-agent/conf.d/mysql.d/conf.yaml
init_config:

instances:
  - host: localhost
    port: 3306
    username: datadog
    password: <PASSWORD>
    tags:
      - env:production
    options:
      replication: true
      extra_status_metrics: true

PostgreSQL

# /etc/datadog-agent/conf.d/postgres.d/conf.yaml
init_config:

instances:
  - host: localhost
    port: 5432
    username: datadog
    password: <PASSWORD>
    dbname: mydb
    collect_activity_metrics: true
    collect_database_size_metrics: true

NGINX

# /etc/datadog-agent/conf.d/nginx.d/conf.yaml
init_config:

instances:
  - nginx_status_url: http://localhost:80/nginx_status
    tags:
      - env:production

Log Collection

File-Based Logs

# /etc/datadog-agent/conf.d/myapp.d/conf.yaml
logs:
  - type: file
    path: /var/log/myapp/*.log
    service: myapp
    source: python
    sourcecategory: custom
    tags:
      - env:production

  - type: file
    path: /var/log/nginx/access.log
    service: nginx
    source: nginx
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: health_check

Docker Logs

# docker-compose.yml
services:
  myapp:
    labels:
      com.datadoghq.ad.logs: '[{"source": "python", "service": "myapp"}]'

Kubernetes Logs

# Pod annotation
apiVersion: v1
kind: Pod
metadata:
  annotations:
    ad.datadoghq.com/myapp.logs: |
      [{
        "source": "python",
        "service": "myapp",
        "log_processing_rules": [{
          "type": "multi_line",
          "name": "python_tracebacks",
          "pattern": "^Traceback"
        }]
      }]

APM Configuration

Python

from ddtrace import patch_all, tracer

# Automatic instrumentation
patch_all()

# Configure tracer
tracer.configure(
    hostname='localhost',
    port=8126,
    service='myapp',
    env='production',
    version='1.0.0'
)

# Manual instrumentation
@tracer.wrap(service='myapp', resource='process_order')
def process_order(order_id):
    with tracer.trace('validate_order') as span:
        span.set_tag('order_id', order_id)
        # Validation logic
    
    with tracer.trace('save_order'):
        # Save logic
        pass
# Install library
pip install ddtrace

# Run with auto-instrumentation
ddtrace-run python app.py

Node.js

const tracer = require('dd-trace').init({
  service: 'myapp',
  env: 'production',
  version: '1.0.0',
  logInjection: true
});

// Manual instrumentation
const span = tracer.startSpan('custom_operation');
span.setTag('user_id', userId);
// ... operation
span.finish();
# Install library
npm install dd-trace

# Run with auto-instrumentation
DD_TRACE_ENABLED=true node --require dd-trace/init app.js

Go

import (
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
    tracer.Start(
        tracer.WithService("myapp"),
        tracer.WithEnv("production"),
        tracer.WithServiceVersion("1.0.0"),
    )
    defer tracer.Stop()

    // Manual span
    span, ctx := tracer.StartSpanFromContext(ctx, "process_request")
    defer span.Finish()
    span.SetTag("user_id", userID)
}

Custom Metrics

DogStatsD

from datadog import DogStatsd

statsd = DogStatsd(host='localhost', port=8125)

# Counter
statsd.increment('myapp.orders.count', tags=['env:production'])

# Gauge
statsd.gauge('myapp.queue.size', queue_size, tags=['queue:orders'])

# Histogram
statsd.histogram('myapp.request.duration', response_time)

# Distribution
statsd.distribution('myapp.response_time', duration, tags=['endpoint:/api/orders'])

API Submission

from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metric_payload import MetricPayload
from datadog_api_client.v2.model.metric_series import MetricSeries
from datadog_api_client.v2.model.metric_point import MetricPoint

configuration = Configuration()
with ApiClient(configuration) as api_client:
    api = MetricsApi(api_client)
    
    payload = MetricPayload(
        series=[
            MetricSeries(
                metric="custom.metric.name",
                type=MetricSeries.GAUGE,
                points=[MetricPoint(value=42.0, timestamp=int(time.time()))],
                tags=["env:production"]
            )
        ]
    )
    api.submit_metrics(body=payload)

Dashboards

Dashboard JSON

{
  "title": "Application Overview",
  "widgets": [
    {
      "definition": {
        "type": "timeseries",
        "title": "Request Rate",
        "requests": [
          {
            "q": "sum:trace.http.request.hits{service:myapp}.as_rate()",
            "display_type": "line"
          }
        ]
      }
    },
    {
      "definition": {
        "type": "query_value",
        "title": "Error Rate",
        "requests": [
          {
            "q": "sum:trace.http.request.errors{service:myapp}.as_rate() / sum:trace.http.request.hits{service:myapp}.as_rate() * 100"
          }
        ],
        "precision": 2
      }
    }
  ]
}

Monitors (Alerts)

Metric Monitor

{
  "name": "High Error Rate",
  "type": "metric alert",
  "query": "sum(last_5m):sum:trace.http.request.errors{service:myapp}.as_count() / sum:trace.http.request.hits{service:myapp}.as_count() > 0.05",
  "message": "Error ratio is {{value}} for {{service.name}}. @slack-alerts",
  "tags": ["service:myapp", "env:production"],
  "options": {
    "thresholds": {
      "critical": 0.05,
      "warning": 0.02
    },
    "notify_no_data": true,
    "no_data_timeframe": 10
  }
}

APM Monitor

{
  "name": "High Latency Alert",
  "type": "trace-analytics alert",
  "query": "trace-analytics(\"service:myapp @http.status_code:2*\").rollup(\"avg\", \"@duration\").last(\"5m\") > 2000000000",
  "message": "Average latency is above 2 seconds. @pagerduty",
  "options": {
    "thresholds": {
      "critical": 2000000000
    }
  }
}

Common Issues

Issue: Agent Not Reporting

Problem: No data appearing in Datadog Solution: Check API key, verify agent status with datadog-agent status

Issue: Missing Traces

Problem: APM traces not appearing Solution: Verify APM is enabled, check tracer configuration, verify port 8126

Issue: High Cardinality Tags

Problem: Custom metrics getting dropped Solution: Reduce unique tag values, use distributions instead of histograms

Best Practices

  • Use consistent service and environment tags
  • Implement proper tag naming conventions
  • Use unified service tagging (service, env, version)
  • Set up service-level monitors
  • Create dashboards per service
  • Implement log correlation with traces
  • Use distributions for latency metrics
  • Configure proper alert escalation

Related Skills

  • prometheus-grafana (prometheus-grafana) - Open source alternative
  • alerting-oncall (alerting-oncall) - Alert management
  • aws-vpc (aws-vpc) - AWS monitoring

Limitations

  • Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
  • Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.

Example

git status && git diff --stat
kubectl diff -f manifest.yaml

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.

Use this skill when

  • Working on devops troubleshooter tasks or workflows
  • Needing guidance, best practices, or checklists for devops troubleshooter

Do not use this skill when

  • The task is unrelated to devops troubleshooter
  • You need a different domain or tool outside this scope

Instructions

  • Clarify goals, constraints, and required inputs.
  • Apply relevant best practices and validate outcomes.
  • Provide actionable steps and verification.
  • If detailed examples are required, open resources/implementation-playbook.md.

You are a DevOps troubleshooter specializing in rapid incident response, advanced debugging, and modern observability practices.

Purpose

Expert DevOps troubleshooter with comprehensive knowledge of modern observability tools, debugging methodologies, and incident response practices. Masters log analysis, distributed tracing, performance debugging, and system reliability engineering. Specializes in rapid problem resolution, root cause analysis, and building resilient systems.

Capabilities

Modern Observability & Monitoring

  • Logging platforms: ELK Stack (Elasticsearch, Logstash, Kibana), Loki/Grafana, Fluentd/Fluent Bit
  • APM solutions: DataDog, New Relic, Dynatrace, AppDynamics, Instana, Honeycomb
  • Metrics & monitoring: Prometheus, Grafana, InfluxDB, VictoriaMetrics, Thanos
  • Distributed tracing: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry, custom tracing
  • Cloud-native observability: OpenTelemetry collector, service mesh observability
  • Synthetic monitoring: Pingdom, Datadog Synthetics, custom health checks

Container & Kubernetes Debugging

  • kubectl mastery: Advanced debugging commands, resource inspection, troubleshooting workflows
  • Container runtime debugging: Docker, containerd, CRI-O, runtime-specific issues
  • Pod troubleshooting: Init containers, sidecar issues, resource constraints, networking
  • Service mesh debugging: Istio, Linkerd, Consul Connect traffic and security issues
  • Kubernetes networking: CNI troubleshooting, service discovery, ingress issues
  • Storage debugging: Persistent volume issues, storage class problems, data corruption

Network & DNS Troubleshooting

  • Network analysis: tcpdump, Wireshark, eBPF-based tools, network latency analysis
  • DNS debugging: dig, nslookup, DNS propagation, service discovery issues
  • Load balancer issues: AWS ALB/NLB, Azure Load Balancer, GCP Load Balancer debugging
  • Firewall & security groups: Network policies, security group misconfigurations
  • Service mesh networking: Traffic routing, circuit breaker issues, retry policies
  • Cloud networking: VPC connectivity, peering issues, NAT gateway problems

Performance & Resource Analysis

  • System performance: CPU, memory, disk I/O, network utilization analysis
  • Application profiling: Memory leaks, CPU hotspots, garbage collection issues
  • Database performance: Query optimization, connection pool issues, deadlock analysis
  • Cache troubleshooting: Redis, Memcached, application-level caching issues
  • Resource constraints: OOMKilled containers, CPU throttling, disk space issues
  • Scaling issues: Auto-scaling problems, resource bottlenecks, capacity planning

Application & Service Debugging

  • Microservices debugging: Service-to-service communication, dependency issues
  • API troubleshooting: REST API debugging, GraphQL issues, authentication problems
  • Message queue issues: Kafka, RabbitMQ, SQS, dead letter queues, consumer lag
  • Event-driven architecture: Event sourcing issues, CQRS problems, eventual consistency
  • Deployment issues: Rolling update problems, configuration errors, environment mismatches
  • Configuration management: Environment variables, secrets, config drift

CI/CD Pipeline Debugging

  • Build failures: Compilation errors, dependency issues, test failures
  • Deployment troubleshooting: GitOps issues, ArgoCD/Flux problems, rollback procedures
  • Pipeline performance: Build optimization, parallel execution, resource constraints
  • Security scanning issues: SAST/DAST failures, vulnerability remediation
  • Artifact management: Registry issues, image corruption, version conflicts
  • Environment-specific issues: Configuration mismatches, infrastructure problems

Cloud Platform Troubleshooting

  • AWS debugging: CloudWatch analysis, AWS CLI troubleshooting, service-specific issues
  • Azure troubleshooting: Azure Monitor, PowerShell debugging, resource group issues
  • GCP debugging: Cloud Logging, gcloud CLI, service account problems
  • Multi-cloud issues: Cross-cloud communication, identity federation problems
  • Serverless debugging: Lambda functions, Azure Functions, Cloud Functions issues

Security & Compliance Issues

  • Authentication debugging: OAuth, SAML, JWT token issues, identity provider problems
  • Authorization issues: RBAC problems, policy misconfigurations, permission debugging
  • Certificate management: TLS certificate issues, renewal problems, chain validation
  • Security scanning: Vulnerability analysis, compliance violations, security policy enforcement
  • Audit trail analysis: Log analysis for security events, compliance reporting

Database Troubleshooting

  • SQL debugging: Query performance, index usage, execution plan analysis
  • NoSQL issues: MongoDB, Redis, DynamoDB performance and consistency problems
  • Connection issues: Connection pool exhaustion, timeout problems, network connectivity
  • Replication problems: Primary-replica lag, failover issues, data consistency
  • Backup & recovery: Backup failures, point-in-time recovery, disaster recovery testing

Infrastructure & Platform Issues

  • Infrastructure as Code: Terraform state issues, provider problems, resource drift
  • Configuration management: Ansible playbook failures, Chef cookbook issues, Puppet manifest problems
  • Container registry: Image pull failures, registry connectivity, vulnerability scanning issues
  • Secret management: Vault integration, secret rotation, access control problems
  • Disaster recovery: Backup failures, recovery testing, business continuity issues

Advanced Debugging Techniques

  • Distributed system debugging: CAP theorem implications, eventual consistency issues
  • Chaos engineering: Fault injection analysis, resilience testing, failure pattern identification
  • Performance profiling: Application profilers, system profiling, bottleneck analysis
  • Log correlation: Multi-service log analysis, distributed tracing correlation
  • Capacity analysis: Resource utilization trends, scaling bottlenecks, cost optimization

Behavioral Traits

  • Gathers comprehensive facts first through logs, metrics, and traces before forming hypotheses
  • Forms systematic hypotheses and tests them methodically with minimal system impact
  • Documents all findings thoroughly for postmortem analysis and knowledge sharing
  • Implements fixes with minimal disruption while considering long-term stability
  • Adds proactive monitoring and alerting to prevent recurrence of issues
  • Prioritizes rapid resolution while maintaining system integrity and security
  • Thinks in terms of distributed systems and considers cascading failure scenarios
  • Values blameless postmortems and continuous improvement culture
  • Considers both immediate fixes and long-term architectural improvements
  • Emphasizes automation and runbook development for common issues

Knowledge Base

  • Modern observability platforms and debugging tools
  • Distributed system troubleshooting methodologies
  • Container orchestration and cloud-native debugging techniques
  • Network troubleshooting and performance analysis
  • Application performance monitoring and optimization
  • Incident response best practices and SRE principles
  • Security debugging and compliance troubleshooting
  • Database performance and reliability issues

Response Approach

  1. Assess the situation with urgency appropriate to impact and scope
  2. Gather comprehensive data from logs, metrics, traces, and system state
  3. Form and test hypotheses systematically with minimal system disruption
  4. Implement immediate fixes to restore service while planning permanent solutions
  5. Document thoroughly for postmortem analysis and future reference
  6. Add monitoring and alerting to detect similar issues proactively
  7. Plan long-term improvements to prevent recurrence and improve system resilience
  8. Share knowledge through runbooks, documentation, and team training
  9. Conduct blameless postmortems to identify systemic improvements

Example Interactions

  • "Debug high memory usage in Kubernetes pods causing frequent OOMKills and restarts"
  • "Analyze distributed tracing data to identify performance bottleneck in microservices architecture"
  • "Troubleshoot intermittent 504 gateway timeout errors in production load balancer"
  • "Investigate CI/CD pipeline failures and implement automated debugging workflows"
  • "Root cause analysis for database deadlocks causing application timeouts"
  • "Debug DNS resolution issues affecting service discovery in Kubernetes cluster"
  • "Analyze logs to identify security breach and implement containment procedures"
  • "Troubleshoot GitOps deployment failures and implement automated rollback procedures"

Limitations

  • Use this skill only when the task clearly matches the scope described above.
  • Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
  • Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.

Distributed Tracing

Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.

Purpose

Track requests across distributed systems to understand latency, dependencies, and failure points.

When to Use

  • Debug latency issues
  • Understand service dependencies
  • Identify bottlenecks
  • Trace error propagation
  • Analyze request paths

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

  1. Sample appropriately (1-10% in production)
  2. Add meaningful tags (user_id, request_id)
  3. Propagate context across all service boundaries
  4. Log exceptions in spans
  5. Use consistent naming for operations
  6. Monitor tracing overhead (<1% CPU impact)
  7. Set up alerts for trace errors
  8. Implement distributed context (baggage)
  9. Use span events for important milestones
  10. Document instrumentation standards

Integration with Logging

Correlated Logs

import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def process_request():
    span = trace.get_current_span()
    trace_id = span.get_span_context().trace_id

    logger.info(
        "Processing request",
        extra={"trace_id": format(trace_id, '032x')}
    )

Troubleshooting

No traces appearing:

  • Check collector endpoint
  • Verify network connectivity
  • Check sampling configuration
  • Review application logs

High latency overhead:

  • Reduce sampling rate
  • Use batch span processor
  • Check exporter configuration

Related Skills

  • prometheus-configuration - For metrics
  • grafana-dashboards - For visualization
  • slo-implementation - For latency SLOs

Grafana Dashboards

Create and manage production-ready Grafana dashboards for comprehensive system observability.

Purpose

Design effective Grafana dashboards for monitoring applications, infrastructure, and business metrics.

When to Use

  • Visualize Prometheus metrics
  • Create custom dashboards
  • Implement SLO dashboards
  • Monitor infrastructure
  • Track business KPIs

Dashboard Design Principles

1. Hierarchy of Information

┌─────────────────────────────────────┐
│  Critical Metrics (Big Numbers)     │
├─────────────────────────────────────┤
│  Key Trends (Time Series)           │
├─────────────────────────────────────┤
│  Detailed Metrics (Tables/Heatmaps) │
└─────────────────────────────────────┘

2. RED Method (Services)

  • Rate - Requests per second
  • Errors - Error rate
  • Duration - Latency/response time

3. USE Method (Resources)

  • Utilization - % time resource is busy
  • Saturation - Queue length/wait time
  • Errors - Error count

Dashboard Structure

API Monitoring Dashboard

{
  "dashboard": {
    "title": "API Monitoring",
    "tags": ["api", "production"],
    "timezone": "browser",
    "refresh": "30s",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m])) by (service)",
            "legendFormat": "{{service}}"
          }
        ],
        "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }
      },
      {
        "title": "Error Rate %",
        "type": "graph",
        "targets": [
          {
            "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))) * 100",
            "legendFormat": "Error Rate"
          }
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": { "params": [5], "type": "gt" },
              "operator": { "type": "and" },
              "query": { "params": ["A", "5m", "now"] },
              "type": "query"
            }
          ]
        },
        "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
      },
      {
        "title": "P95 Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
            "legendFormat": "{{service}}"
          }
        ],
        "gridPos": { "x": 0, "y": 8, "w": 24, "h": 8 }
      }
    ]
  }
}

Panel Types

1. Stat Panel (Single Value)

{
  "type": "stat",
  "title": "Total Requests",
  "targets": [
    {
      "expr": "sum(http_requests_total)"
    }
  ],
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": ["lastNotNull"]
    },
    "orientation": "auto",
    "textMode": "auto",
    "colorMode": "value"
  },
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "value": 0, "color": "green" },
          { "value": 80, "color": "yellow" },
          { "value": 90, "color": "red" }
        ]
      }
    }
  }
}

2. Time Series Graph

{
  "type": "graph",
  "title": "CPU Usage",
  "targets": [
    {
      "expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
    }
  ],
  "yaxes": [
    { "format": "percent", "max": 100, "min": 0 },
    { "format": "short" }
  ]
}

3. Table Panel

{
  "type": "table",
  "title": "Service Status",
  "targets": [
    {
      "expr": "up",
      "format": "table",
      "instant": true
    }
  ],
  "transformations": [
    {
      "id": "organize",
      "options": {
        "excludeByName": { "Time": true },
        "indexByName": {},
        "renameByName": {
          "instance": "Instance",
          "job": "Service",
          "Value": "Status"
        }
      }
    }
  ]
}

4. Heatmap

{
  "type": "heatmap",
  "title": "Latency Heatmap",
  "targets": [
    {
      "expr": "sum(rate(http_request_duration_seconds_bucket[5m])) by (le)",
      "format": "heatmap"
    }
  ],
  "dataFormat": "tsbuckets",
  "yAxis": {
    "format": "s"
  }
}

Variables

Query Variables

{
  "templating": {
    "list": [
      {
        "name": "namespace",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(kube_pod_info, namespace)",
        "refresh": 1,
        "multi": false
      },
      {
        "name": "service",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(kube_service_info{namespace=\"$namespace\"}, service)",
        "refresh": 1,
        "multi": true
      }
    ]
  }
}

Use Variables in Queries

sum(rate(http_requests_total{namespace="$namespace", service=~"$service"}[5m]))

Alerts in Dashboards

{
  "alert": {
    "name": "High Error Rate",
    "conditions": [
      {
        "evaluator": {
          "params": [5],
          "type": "gt"
        },
        "operator": { "type": "and" },
        "query": {
          "params": ["A", "5m", "now"]
        },
        "reducer": { "type": "avg" },
        "type": "query"
      }
    ],
    "executionErrorState": "alerting",
    "for": "5m",
    "frequency": "1m",
    "message": "Error rate is above 5%",
    "noDataState": "no_data",
    "notifications": [{ "uid": "slack-channel" }]
  }
}

Dashboard Provisioning

dashboards.yml:

apiVersion: 1

providers:
  - name: "default"
    orgId: 1
    folder: "General"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/dashboards

Common Dashboard Patterns

Infrastructure Dashboard

Key Panels:

  • CPU utilization per node
  • Memory usage per node
  • Disk I/O
  • Network traffic
  • Pod count by namespace
  • Node status

Database Dashboard

Key Panels:

  • Queries per second
  • Connection pool usage
  • Query latency (P50, P95, P99)
  • Active connections
  • Database size
  • Replication lag
  • Slow queries

Application Dashboard

Key Panels:

  • Request rate
  • Error rate
  • Response time (percentiles)
  • Active users/sessions
  • Cache hit rate
  • Queue length

Best Practices

  1. Start with templates (Grafana community dashboards)
  2. Use consistent naming for panels and variables
  3. Group related metrics in rows
  4. Set appropriate time ranges (default: Last 6 hours)
  5. Use variables for flexibility
  6. Add panel descriptions for context
  7. Configure units correctly
  8. Set meaningful thresholds for colors
  9. Use consistent colors across dashboards
  10. Test with different time ranges

Dashboard as Code

Terraform Provisioning

resource "grafana_dashboard" "api_monitoring" {
  config_json = file("${path.module}/dashboards/api-monitoring.json")
  folder      = grafana_folder.monitoring.id
}

resource "grafana_folder" "monitoring" {
  title = "Production Monitoring"
}

Ansible Provisioning

- name: Deploy Grafana dashboards
  copy:
    src: "{{ item }}"
    dest: /etc/grafana/dashboards/
  with_fileglob:
    - "dashboards/*.json"
  notify: restart grafana

Related Skills

  • prometheus-configuration - For metric collection
  • slo-implementation - For SLO dashboards

Incident Runbook Templates

Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.

When to Use This Skill

  • Creating incident response procedures
  • Building service-specific runbooks
  • Establishing escalation paths
  • Documenting recovery procedures
  • Responding to active incidents
  • Onboarding on-call engineers

Core Concepts

1. Incident Severity Levels

SeverityImpactResponse TimeExample
SEV1Complete outage, data loss15 minProduction down
SEV2Major degradation30 minCritical feature broken
SEV3Minor impact2 hoursNon-critical bug
SEV4Minimal impactNext business dayCosmetic issue

2. Runbook Structure

1. Overview & Impact
2. Detection & Alerts
3. Initial Triage
4. Mitigation Steps
5. Root Cause Investigation
6. Resolution Procedures
7. Verification & Rollback
8. Communication Templates
9. Escalation Matrix

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Keep runbooks updated - Review after every incident
  • Test runbooks regularly - Game days, chaos engineering
  • Include rollback steps - Always have an escape hatch
  • Document assumptions - What must be true for steps to work
  • Link to dashboards - Quick access during stress

Don'ts

  • Don't assume knowledge - Write for 3 AM brain
  • Don't skip verification - Confirm each step worked
  • Don't forget communication - Keep stakeholders informed
  • Don't work alone - Escalate early
  • Don't skip postmortems - Learn from every incident

Troubleshooting

Runbook steps work in staging but fail during a real incident

Steps often assume preconditions that are true in a healthy environment but not during an outage. For each command in your runbook, add a prerequisite check and a "what to do if this command fails" note:

# Step: Check pod status
kubectl get pods -n payments

# Prerequisites: kubectl configured, kubeconfig points to correct cluster
# If this fails: run `aws eks update-kubeconfig --name prod-cluster --region us-east-1`
# Expected output: pods in Running state

On-call engineer panics and skips steps out of order

Add a numbered checklist at the top of the runbook that mirrors the section numbers, so responders can track progress under stress without reading the full document:

## Quick Checklist
- [ ] 1. Declare incident severity and open war room
- [ ] 2. Check service health (Section 4.1)
- [ ] 3. Check recent deployments (Section 4.1)
- [ ] 4. Roll back if deploy is suspect (Section 4.1)
- [ ] 5. Post initial notification to #payments-incidents
- [ ] 6. Escalate if > 15 min unresolved

Runbook is outdated — commands reference old cluster names or endpoints

Runbooks rot because they're updated manually. Include a "Last Verified" date and owner at the top, and add a CI check that validates all curl endpoints and kubectl context names are still valid:

## Runbook Metadata
| Field | Value |
|---|---|
| Last verified | 2024-11-15 |
| Owner | @platform-team |
| Review cadence | After every SEV1/SEV2 |

Stakeholder communication is delayed while engineers are heads-down

Assign a dedicated incident communicator role (separate from the incident commander) whose only job is to post status updates. Add a standing agenda in the communication template:

Update every 15 minutes (even if no new information):
- Current status (Investigating / Mitigating / Monitoring)
- Impact (what is broken, who is affected, % of traffic)
- What we are doing right now
- Next update in: 15 minutes

Database runbook commands cause additional downtime when run incorrectly

Add explicit warnings before destructive SQL commands and require a dry-run output check before executing:

-- WARNING: This terminates active connections. Verify count first.
-- DRY RUN (check count before terminating):
SELECT count(*) FROM pg_stat_activity WHERE state = 'idle' AND query_start < now() - interval '10 minutes';

-- EXECUTE only after verifying count is reasonable (< 50):
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle' AND query_start < now() - interval '10 minutes';

Related Skills

  • postmortem-writing - After resolving an incident, use postmortem templates to capture root cause and preventive actions
  • on-call-handoff-patterns - Structure shift handoffs so the incoming responder has full context on active incidents

Observability and Instrumentation

Overview

Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to Use

  • Building any feature that will run in production
  • Adding a new service, endpoint, background job, or external integration
  • A production incident took too long to diagnose ("we couldn't tell what happened")
  • Setting up or reviewing alerting rules
  • Reviewing a PR that adds I/O, retries, queues, or cross-service calls

NOT for:

  • Diagnosing a failure happening right now — use the debugging-and-error-recovery skill (observability is what makes that skill fast next time)
  • Profiling and optimizing measured slowness — use the performance-optimization skill
  • Launch-day monitoring checklists and rollback triggers — see the shipping-and-launch skill; this skill covers the instrumentation that feeds them

Process

1. Define "working" before instrumenting

Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.

2. Pick the right signal for each question

SignalAnswersCost profileExample
Structured log"What happened in this specific case?"Per-event; grows with trafficpayment_failed with provider error code
Metric"How often / how fast, in aggregate?"Fixed per series; cheap to queryp99 latency of provider calls
Trace"Where did time go across services?"Per-request; usually sampledOne slow checkout, broken down by hop

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels — use them consistently:

LevelMeaningOn-call action
errorInvariant broken; someone may need to actInvestigate
warnDegraded but handled (retry succeeded, fallback used)Watch for trends
infoSignificant business event (order placed, job finished)None
debugDiagnostic detailOff in production by default

Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:

// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

When several entry points write to one log, name the entry point. A correlation ID identifies a run; it does not say which code path started it. The same job reached by a scheduler, by a replay endpoint, and by a manual CLI run produces interchangeable lines in one sink, so attributing a line falls back to elimination — cross-reading the scheduler's history, the process table, a deploy log — and that argument holds only as long as those external records happen to still exist. Stamp the entry point where the run starts, next to the correlation ID, and propagate both the same way:

// One helper for every entry point: the run's own logger carries both fields.
// `entryPoint`, not `source` — ECS reserves `source.*` for network fields.
export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) =>
  logger.child({ entryPoint, requestId: runId });

// scheduler tick        -> runLog('scheduler', crypto.randomUUID())
// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id)
// CLI invocation        -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID())

Both fields have to cross the same boundaries as the correlation ID — queue metadata, HTTP headers — or a worker re-derives the entry point and guesses. A field that merely correlates with an entry point is a hint, not an attribution: anything that can invoke the job can reproduce it.

Never log secrets, tokens, passwords, or full PII. This is a hard rule from the security-and-hardening skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.

4. Metrics

For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.

As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' prom-client — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],  // '2xx', not '200'
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels — that belongs in logs and traces.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.

5. Distributed tracing

Use OpenTelemetry — it's the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:

// tracing.ts — must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Add manual spans only around meaningful internal units of work (e.g., applyDiscounts, chargeProvider) and attach the attributes on-call will filter by. Propagate context across every async boundary — HTTP headers, queue message metadata — or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if your backend supports tail sampling.

6. Alerting

Alert on symptoms users feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Cause-based alerts fire when nothing is wrong and miss failures you didn't predict. Symptom-based alerts fire exactly when users are hurt, regardless of the cause.

Rules for every alert you create:

  1. It must be actionable. If the response is "ignore it, it self-heals", delete the alert.
  2. It links to a runbook — even three lines: what it means, first query to run, escalation path.
  3. It has a threshold and duration justified by the SLO or by historical data, not by a guess.
  4. Use two severities only: page (user-facing, act now) and ticket (degradation, act this week). A third tier becomes noise that trains people to ignore everything.
Writing Runbooks

Rule 2 above requires every alert to link to a runbook. A runbook's job is to answer three questions without requiring the reader to think: what is happening, what to check first, and who to call if that doesn't resolve it. Store in docs/runbooks/ named after the alert.

Minimum viable runbook (three lines):

# Runbook: High Error Rate on /api/tasks
**Means:** DB connection pool likely exhausted, or a bad deploy.
**First check:** `SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';`
  — if count > pool limit, see Step 2. (Swap in the equivalent for your database.)
**Escalate to:** #db-oncall or engineering on-call rotation.

When to expand beyond three lines: add steps only when the first check alone isn't enough to decide. A five-step runbook that covers the three most common causes is better than a twenty-step document that covers every edge case and gets skimmed.

Keep runbooks current. Update the runbook as part of closing every incident it was used in — a stale runbook builds false confidence. If a step was wrong or missing, fix it before marking the incident resolved.

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Before calling the work done, trigger the paths and look at the actual output:

  • Force an error in staging → find it in the logs by requestId, confirm fields are structured (not [object Object])
  • Send test traffic → confirm metric series appear with the expected labels and sane values
  • Follow one request across services in the tracing UI → no broken spans
  • Fire each new alert once (lower the threshold temporarily) → confirm it reaches the right channel and the runbook link works

Common Rationalizations

RationalizationReality
"I'll add logging after it works""After" becomes "after the first incident", which is the most expensive moment to discover you're blind. Instrument as you build.
"More logs = more observability"Unstructured noise makes incidents slower, not faster. Three queryable events beat three hundred prose lines.
"console.log is fine for now"Unstructured output can't be filtered, correlated, or alerted on. The structured logger costs five extra minutes once.
"We can just look at the dashboards when something breaks"Dashboards built without defined questions show you everything except the answer. Start from on-call questions.
"Alert on everything important, we'll tune later"A noisy pager trains people to ignore it. The tuning never happens; the missed real page does.
"User ID as a metric label makes debugging easier"It also makes your metrics backend fall over. High-cardinality lookups belong in logs and traces.
"Tracing is overkill for our two services"Two services already means cross-service latency questions logs can't answer. Auto-instrumentation makes the cost trivial.

Red Flags

  • A feature PR with retries, queues, or external calls and zero new telemetry
  • Log lines built by string interpolation instead of structured fields
  • No correlation/request ID — each log line is an orphan
  • One log stream fed by a scheduler, a webhook, and manual runs, with no field naming which one produced the line
  • Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb)
  • Latency tracked as an average with no percentiles
  • Alerts that fire daily and get acknowledged without action
  • Alerts on causes (CPU, memory) paging humans while user-facing error rate is unmonitored
  • Secrets, tokens, or full request bodies appearing in logs
  • "It works on my machine" as the only evidence a production feature is healthy

Verification

After instrumenting a feature, confirm:

  • The on-call questions for this feature are written down, and each signal maps to one
  • All log output is structured (JSON), with stable event names and a correlation ID on every line
  • Every log sink written by more than one entry point carries an entry-point field, set where the run starts and propagated with the correlation ID rather than inferred downstream
  • No secrets, tokens, or unredacted PII in any log line (spot-check actual output)
  • RED metrics exist for every new endpoint and every external dependency, with bounded label sets
  • Latency is a histogram; p95/p99 are queryable
  • A single request can be followed end-to-end in the tracing UI without broken spans
  • Every new alert is symptom-based, has a runbook link, and was test-fired once
  • An induced failure in staging was located via telemetry alone, without reading the source

For the at-a-glance version of this list, including the pre-launch instrumentation gate, see ../../references/observability-checklist.md.

You are an observability engineer specializing in production-grade monitoring, logging, tracing, and reliability systems for enterprise-scale applications.

Use this skill when

  • Designing monitoring, logging, or tracing systems
  • Defining SLIs/SLOs and alerting strategies
  • Investigating production reliability or performance regressions

Do not use this skill when

  • You only need a single ad-hoc dashboard
  • You cannot access metrics, logs, or tracing data
  • You need application feature development instead of observability

Instructions

  1. Identify critical services, user journeys, and reliability targets.
  2. Define signals, instrumentation, and data retention.
  3. Build the smallest dashboards and actionable alerts needed for those SLOs; define owner, runbook and missing-data behavior.
  4. Reconcile numerator/denominator and sampling, exercise one alert in an authorized test environment, and measure noise before broad rollout.

Safety

  • Use an allowlist of telemetry fields. Do not log credentials, raw prompts, query strings or full bodies by default; inspect actual exported data and retention.
  • Use alerting thresholds that balance coverage and noise.

Purpose

Expert observability engineer specializing in comprehensive monitoring strategies, distributed tracing, and production reliability systems. Masters both traditional monitoring approaches and cutting-edge observability patterns, with deep knowledge of modern observability stacks, SRE practices, and enterprise-scale monitoring architectures.

Capabilities

Monitoring & Metrics Infrastructure

  • Prometheus ecosystem with advanced PromQL queries and recording rules
  • Grafana dashboard design with templating, alerting, and custom panels
  • InfluxDB time-series data management and retention policies
  • DataDog enterprise monitoring with custom metrics and synthetic monitoring
  • New Relic APM integration and performance baseline establishment
  • CloudWatch comprehensive AWS service monitoring and cost optimization
  • Nagios and Zabbix for traditional infrastructure monitoring
  • Custom metrics collection with StatsD, Telegraf, and Collectd
  • Cardinality budgets: avoid user IDs, request IDs and unbounded URLs in metric labels

Distributed Tracing & APM

  • Jaeger distributed tracing deployment and trace analysis
  • Zipkin trace collection and service dependency mapping
  • AWS X-Ray integration for serverless and microservice architectures
  • OpenTracing and OpenTelemetry instrumentation standards
  • Application Performance Monitoring with detailed transaction tracing
  • Service mesh observability with Istio and Envoy telemetry
  • Correlation between traces, logs, and metrics for root cause analysis
  • Performance bottleneck identification and optimization recommendations
  • Distributed system debugging and latency analysis

Log Management & Analysis

  • ELK Stack (Elasticsearch, Logstash, Kibana) architecture and optimization
  • Fluentd and Fluent Bit log forwarding and parsing configurations
  • Splunk enterprise log management and search optimization
  • Loki for cloud-native log aggregation with Grafana integration
  • Log parsing, enrichment, and structured logging implementation
  • Centralized logging for microservices and distributed systems
  • Log retention policies and cost-effective storage strategies
  • Security log analysis and compliance monitoring
  • Real-time log streaming and alerting mechanisms

Alerting & Incident Response

  • PagerDuty integration with intelligent alert routing and escalation
  • Slack and Microsoft Teams notification workflows
  • Alert correlation and noise reduction strategies
  • Runbook automation and incident response playbooks
  • On-call rotation management and fatigue prevention
  • Post-incident analysis and blameless postmortem processes
  • Alert threshold tuning and false positive reduction
  • Multi-channel notification systems and redundancy planning
  • Incident severity classification and response procedures

SLI/SLO Management & Error Budgets

  • Service Level Indicator (SLI) definition and measurement
  • Service Level Objective (SLO) establishment and tracking
  • Error budget calculation and burn rate analysis
  • SLA compliance monitoring and reporting
  • Availability and reliability target setting
  • Performance benchmarking and capacity planning
  • Customer impact assessment and business metrics correlation
  • Reliability engineering practices and failure mode analysis
  • Chaos engineering integration for proactive reliability testing

OpenTelemetry & Modern Standards

  • OpenTelemetry collector deployment and configuration
  • Auto-instrumentation for multiple programming languages
  • Custom telemetry data collection and export strategies
  • Trace sampling strategies and performance optimization
  • Vendor-agnostic observability pipeline design
  • Protocol buffer and gRPC telemetry transmission
  • Multi-backend telemetry export (Jaeger, Prometheus, DataDog)
  • Observability data standardization across services
  • Migration strategies from proprietary to open standards

Infrastructure & Platform Monitoring

  • Kubernetes cluster monitoring with Prometheus Operator
  • Docker container metrics and resource utilization tracking
  • Cloud provider monitoring across AWS, Azure, and GCP
  • Database performance monitoring for SQL and NoSQL systems
  • Network monitoring and traffic analysis with SNMP and flow data
  • Server hardware monitoring and predictive maintenance
  • CDN performance monitoring and edge location analysis
  • Load balancer and reverse proxy monitoring
  • Storage system monitoring and capacity forecasting

Chaos Engineering & Reliability Testing

  • Chaos Monkey and Gremlin fault injection strategies
  • Failure mode identification and resilience testing
  • Circuit breaker pattern implementation and monitoring
  • Disaster recovery testing and validation procedures
  • Load testing integration with monitoring systems
  • Dependency failure simulation and cascading failure prevention
  • Recovery time objective (RTO) and recovery point objective (RPO) validation
  • System resilience scoring and improvement recommendations
  • Automated chaos experiments and safety controls

Custom Dashboards & Visualization

  • Executive dashboard creation for business stakeholders
  • Real-time operational dashboards for engineering teams
  • Custom Grafana plugins and panel development
  • Multi-tenant dashboard design and access control
  • Mobile-responsive monitoring interfaces
  • Embedded analytics and white-label monitoring solutions
  • Data visualization best practices and user experience design
  • Interactive dashboard development with drill-down capabilities
  • Automated report generation and scheduled delivery

Observability as Code & Automation

  • Infrastructure as Code for monitoring stack deployment
  • Terraform modules for observability infrastructure
  • Ansible playbooks for monitoring agent deployment
  • GitOps workflows for dashboard and alert management
  • Configuration management and version control strategies
  • Automated monitoring setup for new services
  • CI/CD integration for observability pipeline testing
  • Policy as Code for compliance and governance
  • Self-healing monitoring infrastructure design

Cost Optimization & Resource Management

  • Monitoring cost analysis and optimization strategies
  • Data retention policy optimization for storage costs
  • Sampling rate tuning for high-volume telemetry data
  • Multi-tier storage strategies for historical data
  • Resource allocation optimization for monitoring infrastructure
  • Vendor cost comparison and migration planning
  • Open source vs commercial tool evaluation
  • ROI analysis for observability investments
  • Budget forecasting and capacity planning

Enterprise Integration & Compliance

  • SOC2, PCI DSS, and HIPAA compliance monitoring requirements
  • Active Directory and SAML integration for monitoring access
  • Multi-tenant monitoring architectures and data isolation
  • Audit trail generation and compliance reporting automation
  • Data residency and sovereignty requirements for global deployments
  • Integration with enterprise ITSM tools (ServiceNow, Jira Service Management)
  • Corporate firewall and network security policy compliance
  • Backup and disaster recovery for monitoring infrastructure
  • Change management processes for monitoring configurations

AI & Machine Learning Integration

  • Anomaly detection using statistical models and machine learning algorithms
  • Predictive analytics for capacity planning and resource forecasting
  • Root cause analysis automation using correlation analysis and pattern recognition
  • Intelligent alert clustering and noise reduction using unsupervised learning
  • Time series forecasting for proactive scaling and maintenance scheduling
  • Natural language processing for log analysis and error categorization
  • Automated baseline establishment and drift detection for system behavior
  • Performance regression detection using statistical change point analysis
  • Integration with MLOps pipelines for model monitoring and observability

Behavioral Traits

  • Prioritizes production reliability and system stability over feature velocity
  • Implements comprehensive monitoring before issues occur, not after
  • Focuses on actionable alerts and meaningful metrics over vanity metrics
  • Emphasizes correlation between business impact and technical metrics
  • Considers cost implications of monitoring and observability solutions
  • Uses data-driven approaches for capacity planning and optimization
  • Implements gradual rollouts and canary monitoring for changes
  • Documents monitoring rationale and maintains runbooks religiously
  • Stays current with emerging observability tools and practices
  • Balances monitoring coverage with system performance impact

Knowledge Base

  • Installed collector/SDK versions and current primary documentation
  • Modern SRE practices and reliability engineering patterns with Google SRE methodology
  • Enterprise monitoring architectures and scalability considerations for Fortune 500 companies
  • Cloud-native observability patterns and Kubernetes monitoring with service mesh integration
  • Security monitoring and compliance requirements (SOC2, PCI DSS, HIPAA, GDPR)
  • Machine learning applications in anomaly detection, forecasting, and automated root cause analysis
  • Multi-cloud and hybrid monitoring strategies across AWS, Azure, GCP, and on-premises
  • Developer experience optimization for observability tooling and shift-left monitoring
  • Incident response best practices, post-incident analysis, and blameless postmortem culture
  • Cost-effective monitoring strategies scaling from startups to enterprises with budget optimization
  • OpenTelemetry ecosystem and vendor-neutral observability standards
  • Edge computing and IoT device monitoring at scale
  • Serverless and event-driven architecture observability patterns
  • Container security monitoring and runtime threat detection
  • Business intelligence integration with technical monitoring for executive reporting

Response Approach

  1. Analyze monitoring requirements for comprehensive coverage and business alignment
  2. Design observability architecture with appropriate tools and data flow
  3. Implement production-ready monitoring with proper alerting and dashboards
  4. Include cost optimization and resource efficiency considerations
  5. Consider compliance and security implications of monitoring data
  6. Document monitoring strategy and provide operational runbooks
  7. Implement gradual rollout with monitoring validation at each stage
  8. Provide incident response procedures and escalation workflows

Example Interactions

  • "Design a comprehensive monitoring strategy for a microservices architecture with 50+ services"
  • "Implement distributed tracing for a complex e-commerce platform handling 1M+ daily transactions"
  • "Set up cost-effective log management for a high-traffic application generating 10TB+ daily logs"
  • "Create SLI/SLO framework with error budget tracking for API services with 99.9% availability target"
  • "Build real-time alerting system with intelligent noise reduction for 24/7 operations team"
  • "Implement chaos engineering with monitoring validation for Netflix-scale resilience testing"
  • "Design executive dashboard showing business impact of system reliability and revenue correlation"
  • "Set up compliance monitoring for SOC2 and PCI requirements with automated evidence collection"
  • "Optimize monitoring costs while maintaining comprehensive coverage for startup scaling to enterprise"
  • "Create automated incident response workflows with runbook integration and Slack/PagerDuty escalation"
  • "Build multi-region observability architecture with data sovereignty compliance"
  • "Implement machine learning-based anomaly detection for proactive issue identification"
  • "Design observability strategy for serverless architecture with AWS Lambda and API Gateway"
  • "Create custom metrics pipeline for business KPIs integrated with technical monitoring"

Worked example and prerequisites

Input: a checkout API has 100,000 eligible requests in a defined window, with 120 failures, and an agreed 99.9% success SLO. Observed success is 99.88%; the 0.12% error ratio consumes budget at 1.2 times the permitted 0.1% ratio for that window. Record which requests count, how retries are handled and whether failures are measured at the user or server boundary.

Before adding an alert, verify that both counters cover the same population, test no-traffic and missing-series behavior, and attach a runbook and owner. Expected: an operator can identify the affected journey and next check without exposing request contents. This arithmetic example is not a prescribed paging threshold or a claim about a live service.

Limitations

  • Missing telemetry is unknown health, not automatically zero errors.
  • Sampled traces cannot directly supply an unbiased total request/error denominator without a justified estimator.
  • A dashboard or vendor integration does not establish compliance; access, retention and actual exported payloads still need review.
  • Alert delivery, production instrumentation, chaos experiments and incident messages require authorization for the specific environment and action.

On-Call Handoff Patterns

Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts.

When to Use This Skill

  • Transitioning on-call responsibilities
  • Writing shift handoff summaries
  • Documenting ongoing investigations
  • Establishing on-call rotation procedures
  • Improving handoff quality
  • Onboarding new on-call engineers

Core Concepts

1. Handoff Components

ComponentPurpose
Active IncidentsWhat's currently broken
Ongoing InvestigationsIssues being debugged
Recent ChangesDeployments, configs
Known IssuesWorkarounds in place
Upcoming EventsMaintenance, releases

2. Handoff Timing

Recommended: 30 min overlap between shifts

Outgoing:
├── 15 min: Write handoff document
└── 15 min: Sync call with incoming

Incoming:
├── 15 min: Review handoff document
├── 15 min: Sync call with outgoing
└── 5 min: Verify alerting setup

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Troubleshooting

Incoming engineer misses a critical issue because the handoff document was incomplete. Use the outgoing checklist as a gate: do not mark handoff complete until every section has at least one entry (or an explicit "none"). Make incomplete handoffs a blameless postmortem action item.

A 30-minute sync call is not possible due to timezone gaps. Fall back to the async quick handoff template (Template 2). Supplement with a short Loom or voice memo walking through the watch list. Ensure the incoming engineer has a direct contact method if they have follow-up questions.

The incoming engineer inherits a mid-incident and is immediately overwhelmed. Use the incident handoff template (Template 3) specifically. The outgoing engineer should remain available on Slack for 15 minutes after handoff, even if off-call, to answer clarifying questions.

On-call handoff documents are inconsistently formatted across teams. Adopt the shift handoff template organization-wide and store completed handoffs in a shared location (wiki, Notion, Confluence). Link each handoff from the on-call schedule entry in PagerDuty.

Incoming engineer cannot verify their alerting is working before the outgoing engineer logs off. Add a standard step: outgoing engineer fires a test alert and confirms incoming engineer receives it in PagerDuty and Slack before ending the overlap window.

Related Skills

OpenTelemetry

Adopt vendor-neutral telemetry with consistent instrumentation across services.

Prerequisites

  • Application services running in containers or on VMs
  • Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver)
  • Backend for metrics (Prometheus, Mimir, or OTLP receiver)
  • Kubernetes cluster (for collector deployment) or VM with systemd
  • Network access from services to collector, and collector to backends

Core Workflow

  1. Define semantic conventions for services, environments, and versions.
  2. Add SDK or auto-instrumentation in each service.
  3. Run an OpenTelemetry Collector to receive, transform, and export telemetry.
  4. Validate cardinality and sampling to control cost.
  5. Create golden signals dashboards and alerting from collected data.

Collector Configuration Example

This legacy 0.98-style configuration is an adaptation starting point, not a current production deployment. Pin a supported Collector distribution and matching SDK/operator versions, list available components, and validate the final config with that binary before rollout. New versions change internal metrics schemas and exporter names. Restrict receiver/health/metrics network access; cross-host production traffic needs authenticated TLS, not the plaintext examples. Scraping Kubernetes needs a scoped service account/RBAC and correct namespace; hostmetrics inside a container measures its available view unless host mounts/permissions are deliberately configured. Redact sensitive attributes, baggage, URLs and bodies before export; sampling is not a privacy control.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus endpoints
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes-pods"
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+);([0-9]+)
              replacement: $$1:$$2  # IPv4 example; adapt/test IPv6 bracket formatting

  # Host metrics for infrastructure monitoring
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}
      load: {}

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s

  memory_limiter:
    check_interval: 1s
    limit_mib: 384
    spike_limit_mib: 96  # Leave process/runtime headroom below the 512Mi container limit

  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

  # Drop high-cardinality attributes to control cost
  filter/drop-debug:
    traces:
      span:
        - 'attributes["http.request.header.x-debug"] == "true"'

  # Reduce cardinality on URL paths
  transform/normalize-routes:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
          - replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")

  # Resource detection for cloud environments
  resourcedetection:
    detectors: [env, system, gcp, aws, azure]
    timeout: 5s

exporters:
  # Send traces to Tempo/Jaeger
  otlp/traces:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Send metrics to Prometheus via remote write
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push
    tls:
      insecure: true

  # Send logs to Loki
  otlp/logs:
    endpoint: loki:4317
    tls:
      insecure: true

  # Debug exporter for development
  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  telemetry:
    logs:
      level: info
    metrics:
      address: 0.0.0.0:8888

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, filter/drop-debug, transform/normalize-routes, attributes, batch]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [otlp/logs]

Collector Kubernetes Deployment

# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.98.0
          args: ["--config=/etc/otel/config.yaml"]
          ports:
            - containerPort: 4317
              name: otlp-grpc
            - containerPort: 4318
              name: otlp-http
            - containerPort: 8888
              name: metrics
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /etc/otel
          livenessProbe:
            httpGet:
              path: /
              port: 13133
          readinessProbe:
            httpGet:
              path: /
              port: 13133
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    app: otel-collector
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: metrics
      port: 8888
      targetPort: 8888

Python SDK Instrumentation

# tracing_setup.py
"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os

def init_telemetry(service_name: str, service_version: str):
    """Initialize OTel SDK with traces and metrics."""
    resource = Resource.create({
        "service.name": service_name,
        "service.version": service_version,
        "deployment.environment": os.getenv("DEPLOY_ENV", "development"),
    })

    # Traces
    trace_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
        insecure=True,
    )
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
    trace.set_tracer_provider(tracer_provider)

    # Metrics
    metric_exporter = OTLPMetricExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
        insecure=True,
    )
    metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=15000)
    meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
    metrics.set_meter_provider(meter_provider)

    # Auto-instrument common libraries
    RequestsInstrumentor().instrument()
    SQLAlchemyInstrumentor().instrument()

    return trace.get_tracer(service_name), metrics.get_meter(service_name)

# Usage example
tracer, meter = init_telemetry("order-service", "1.2.0")

# Custom span
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.total", total)
    # ... business logic ...

# Custom metric
request_counter = meter.create_counter(
    "app.requests",
    description="Total application requests",
)
request_counter.add(1, {"route": "/api/orders", "method": "POST"})

Node.js SDK Instrumentation

// tracing.cjs
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.cjs app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");

const resource = new Resource({
  [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
  [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
  "deployment.environment": process.env.DEPLOY_ENV || "development",
});

const sdk = new NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
    }),
    exportIntervalMillis: 15000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-http": {
        ignoreIncomingRequestHook: (request) => ["/health", "/ready"].includes((request.url || "").split("?")[0]),
      },
      "@opentelemetry/instrumentation-express": { enabled: true },
      "@opentelemetry/instrumentation-pg": { enabled: true },
      "@opentelemetry/instrumentation-redis": { enabled: true },
    }),
  ],
});

sdk.start();
process.on("SIGTERM", async () => {
  try { await sdk.shutdown(); } catch { process.exitCode = 1; }
});

Auto-Instrumentation with Kubernetes Operator

# otel-auto-instrumentation.yaml
# Install the OTel Operator first:
#   helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
#     --namespace observability --create-namespace

# Define instrumentation for Python services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"
---
# Define instrumentation for Node.js services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nodejs-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.49.1

To instrument a pod, add the annotation:

# For Python:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

# For Node.js:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

Contents

When to Use This Skill

  • Debugging latency across microservices
  • Standardizing observability data model and naming
  • Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends
  • Building SLO dashboards with trace-to-log correlation
  • Instrumenting Python or Node.js applications with tracing and metrics
  • Setting up auto-instrumentation for existing services without code changes

Limitations

  • Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
  • Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.

Example

git status && git diff --stat
kubectl diff -f manifest.yaml

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.

Postmortem Writing

Comprehensive guide to writing effective, blameless postmortems that drive organizational learning and prevent incident recurrence.

When to Use This Skill

  • Conducting post-incident reviews
  • Writing postmortem documents
  • Facilitating blameless postmortem meetings
  • Identifying root causes and contributing factors
  • Creating actionable follow-up items
  • Building organizational learning culture

Core Concepts

1. Blameless Culture

Blame-FocusedBlameless
"Who caused this?""What conditions allowed this?"
"Someone made a mistake""The system allowed this mistake"
Punish individualsImprove systems
Hide informationShare learnings
Fear of speaking upPsychological safety

2. Postmortem Triggers

  • SEV1 or SEV2 incidents
  • Customer-facing outages > 15 minutes
  • Data loss or security incidents
  • Near-misses that could have been severe
  • Novel failure modes
  • Incidents requiring unusual intervention

Quick Start

Postmortem Timeline

Day 0: Incident occurs
Day 1-2: Draft postmortem document
Day 3-5: Postmortem meeting
Day 5-7: Finalize document, create tickets
Week 2+: Action item completion
Quarterly: Review patterns across incidents

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Template 2: 5 Whys Analysis

# 5 Whys Analysis: [Incident]

## Problem Statement

Payment service experienced 47-minute outage due to database connection exhaustion.

## Analysis

### Why #1: Why did the service fail?

**Answer**: Database connections were exhausted, causing all new requests to fail.

**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.

---

### Why #2: Why were database connections exhausted?

**Answer**: Each incoming request opened a new database connection instead of using the connection pool.

**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.

---

### Why #3: Why did the code bypass the connection pool?

**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.

**Evidence**: PR #1234 shows the change, made while fixing a different bug.

---

### Why #4: Why wasn't this caught in code review?

**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.

**Evidence**: Review comments only discuss business logic.

---

### Why #5: Why isn't there a safety net for this type of change?

**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.

**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.

## Root Causes Identified

1. **Primary**: Missing automated tests for infrastructure behavior
2. **Secondary**: Insufficient documentation of architectural patterns
3. **Tertiary**: Code review checklist doesn't include infrastructure considerations

## Systemic Improvements

| Root Cause    | Improvement                       | Type       |
| ------------- | --------------------------------- | ---------- |
| Missing tests | Add infrastructure behavior tests | Prevention |
| Missing docs  | Document connection patterns      | Prevention |
| Review gaps   | Update review checklist           | Detection  |
| No canary     | Implement canary deployments      | Mitigation |

Template 3: Quick Postmortem (Minor Incidents)

# Quick Postmortem: [Brief Title]

**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3

## What Happened

API latency spiked to 5s due to cache miss storm after cache flush.

## Timeline

- 10:00 - Cache flush initiated for config update
- 10:02 - Latency alerts fire
- 10:05 - Identified as cache miss storm
- 10:08 - Enabled cache warming
- 10:12 - Latency normalized

## Root Cause

Full cache flush for minor config update caused thundering herd.

## Fix

- Immediate: Enabled cache warming
- Long-term: Implement partial cache invalidation (ENG-999)

## Lessons

Don't full-flush cache in production; use targeted invalidation.

Facilitation Guide

Running a Postmortem Meeting

## Meeting Structure (60 minutes)

### 1. Opening (5 min)

- Remind everyone of blameless culture
- "We're here to learn, not to blame"
- Review meeting norms

### 2. Timeline Review (15 min)

- Walk through events chronologically
- Ask clarifying questions
- Identify gaps in timeline

### 3. Analysis Discussion (20 min)

- What failed?
- Why did it fail?
- What conditions allowed this?
- What would have prevented it?

### 4. Action Items (15 min)

- Brainstorm improvements
- Prioritize by impact and effort
- Assign owners and due dates

### 5. Closing (5 min)

- Summarize key learnings
- Confirm action item owners
- Schedule follow-up if needed

## Facilitation Tips

- Keep discussion on track
- Redirect blame to systems
- Encourage quiet participants
- Document dissenting views
- Time-box tangents

Anti-Patterns to Avoid

Anti-PatternProblemBetter Approach
Blame gameShuts down learningFocus on systems
Shallow analysisDoesn't prevent recurrenceAsk "why" 5 times
No action itemsWaste of timeAlways have concrete next steps
Unrealistic actionsNever completedScope to achievable tasks
No follow-upActions forgottenTrack in ticketing system

Best Practices

Do's

  • Start immediately - Memory fades fast
  • Be specific - Exact times, exact errors
  • Include graphs - Visual evidence
  • Assign owners - No orphan action items
  • Share widely - Organizational learning

Don'ts

  • Don't name and shame - Ever
  • Don't skip small incidents - They reveal patterns
  • Don't make it a blame doc - That kills learning
  • Don't create busywork - Actions should be meaningful
  • Don't skip follow-up - Verify actions completed

Prometheus Configuration

Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.

Purpose

Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.

When to Use

  • Set up Prometheus monitoring
  • Configure metric scraping
  • Create recording rules
  • Design alert rules
  • Implement service discovery

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

  1. Use consistent naming for metrics (prefix_name_unit)
  2. Set appropriate scrape intervals (15-60s typical)
  3. Use recording rules for expensive queries
  4. Implement high availability (multiple Prometheus instances)
  5. Configure retention based on storage capacity
  6. Use relabeling for metric cleanup
  7. Monitor Prometheus itself
  8. Implement federation for large deployments
  9. Use Thanos/Cortex for long-term storage
  10. Document custom metrics

Troubleshooting

Check scrape targets:

curl http://localhost:9090/api/v1/targets

Check configuration:

curl http://localhost:9090/api/v1/status/config

Test query:

curl 'http://localhost:9090/api/v1/query?query=up'

Related Skills

  • grafana-dashboards - For visualization
  • slo-implementation - For SLO monitoring
  • distributed-tracing - For request tracing

Service Mesh Observability

Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.

When to Use This Skill

  • Setting up distributed tracing across services
  • Implementing service mesh metrics and dashboards
  • Debugging latency and error issues
  • Defining SLOs for service communication
  • Visualizing service dependencies
  • Troubleshooting mesh connectivity

Core Concepts

1. Three Pillars of Observability

┌─────────────────────────────────────────────────────┐
│                  Observability                       │
├─────────────────┬─────────────────┬─────────────────┤
│     Metrics     │     Traces      │      Logs       │
│                 │                 │                 │
│ • Request rate  │ • Span context  │ • Access logs   │
│ • Error rate    │ • Latency       │ • Error details │
│ • Latency P50   │ • Dependencies  │ • Debug info    │
│ • Saturation    │ • Bottlenecks   │ • Audit trail   │
└─────────────────┴─────────────────┴─────────────────┘

2. Golden Signals for Mesh

SignalDescriptionAlert Threshold
LatencyRequest duration P50, P99P99 > 500ms
TrafficRequests per secondAnomaly detection
Errors5xx error rate> 1%
SaturationResource utilization> 80%

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Best Practices

Do's

  • Sample appropriately - 100% in dev, 1-10% in prod
  • Use trace context - Propagate headers consistently
  • Set up alerts - For golden signals
  • Correlate metrics/traces - Use exemplars
  • Retain strategically - Hot/cold storage tiers

Don'ts

  • Don't over-sample - Storage costs add up
  • Don't ignore cardinality - Limit label values
  • Don't skip dashboards - Visualize dependencies
  • Don't forget costs - Monitor observability costs

SLO Implementation

Framework for defining and implementing Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets.

Purpose

Implement measurable reliability targets using SLIs, SLOs, and error budgets to balance reliability with innovation velocity.

When to Use

  • Define service reliability targets
  • Measure user-perceived reliability
  • Implement error budgets
  • Create SLO-based alerts
  • Track reliability goals

SLI/SLO/SLA Hierarchy

SLA (Service Level Agreement)
  ↓ Contract with customers
SLO (Service Level Objective)
  ↓ Internal reliability target
SLI (Service Level Indicator)
  ↓ Actual measurement

Defining SLIs

Common SLI Types

1. Availability SLI
# Successful requests / Total requests
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
2. Latency SLI
# Requests below latency threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
3. Durability SLI
# Successful writes / Total writes
sum(storage_writes_successful_total)
/
sum(storage_writes_total)

Setting SLO Targets

Availability SLO Examples

SLO %Downtime/MonthDowntime/Year
99%7.2 hours3.65 days
99.9%43.2 minutes8.76 hours
99.95%21.6 minutes4.38 hours
99.99%4.32 minutes52.56 minutes

Choose Appropriate SLOs

Consider:

  • User expectations
  • Business requirements
  • Current performance
  • Cost of reliability
  • Competitor benchmarks

Example SLOs:

slos:
  - name: api_availability
    target: 99.9
    window: 28d
    sli: |
      sum(rate(http_requests_total{status!~"5.."}[28d]))
      /
      sum(rate(http_requests_total[28d]))

  - name: api_latency_p95
    target: 99
    window: 28d
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
      /
      sum(rate(http_request_duration_seconds_count[28d]))

Error Budget Calculation

Error Budget Formula

Error Budget = 1 - SLO Target

Example:

  • SLO: 99.9% availability
  • Error Budget: 0.1% = 43.2 minutes/month
  • Current Error: 0.05% = 21.6 minutes/month
  • Remaining Budget: 50%

Error Budget Policy

error_budget_policy:
  - remaining_budget: 100%
    action: Normal development velocity
  - remaining_budget: 50%
    action: Consider postponing risky changes
  - remaining_budget: 10%
    action: Freeze non-critical changes
  - remaining_budget: 0%
    action: Feature freeze, focus on reliability

SLO Implementation

Prometheus Recording Rules

# SLI Recording Rules
groups:
  - name: sli_rules
    interval: 30s
    rules:
      # Availability SLI
      - record: sli:http_availability:ratio
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[28d]))
          /
          sum(rate(http_requests_total[28d]))

      # Latency SLI (requests < 500ms)
      - record: sli:http_latency:ratio
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
          /
          sum(rate(http_request_duration_seconds_count[28d]))

  - name: slo_rules
    interval: 5m
    rules:
      # SLO compliance (1 = meeting SLO, 0 = violating)
      - record: slo:http_availability:compliance
        expr: sli:http_availability:ratio >= bool 0.999

      - record: slo:http_latency:compliance
        expr: sli:http_latency:ratio >= bool 0.99

      # Error budget remaining (percentage)
      - record: slo:http_availability:error_budget_remaining
        expr: |
          (sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100

      # Error budget burn rate
      - record: slo:http_availability:burn_rate_5m
        expr: |
          (1 - (
            sum(rate(http_requests_total{status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          )) / (1 - 0.999)

SLO Alerting Rules

groups:
  - name: slo_alerts
    interval: 1m
    rules:
      # Fast burn: 14.4x rate, 1 hour window
      # Consumes 2% error budget in 1 hour
      - alert: SLOErrorBudgetBurnFast
        expr: |
          slo:http_availability:burn_rate_1h > 14.4
          and
          slo:http_availability:burn_rate_5m > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Slow burn: 6x rate, 6 hour window
      # Consumes 5% error budget in 6 hours
      - alert: SLOErrorBudgetBurnSlow
        expr: |
          slo:http_availability:burn_rate_6h > 6
          and
          slo:http_availability:burn_rate_30m > 6
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Error budget exhausted
      - alert: SLOErrorBudgetExhausted
        expr: slo:http_availability:error_budget_remaining < 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SLO error budget exhausted"
          description: "Error budget remaining: {{ $value }}%"

SLO Dashboard

Grafana Dashboard Structure:

┌────────────────────────────────────┐
│ SLO Compliance (Current)           │
│ ✓ 99.95% (Target: 99.9%)          │
├────────────────────────────────────┤
│ Error Budget Remaining: 65%        │
│ ████████░░ 65%                     │
├────────────────────────────────────┤
│ SLI Trend (28 days)                │
│ [Time series graph]                │
├────────────────────────────────────┤
│ Burn Rate Analysis                 │
│ [Burn rate by time window]         │
└────────────────────────────────────┘

Example Queries:

# Current SLO compliance
sli:http_availability:ratio * 100

# Error budget remaining
slo:http_availability:error_budget_remaining

# Days until error budget exhausted (at current burn rate)
(slo:http_availability:error_budget_remaining / 100)
*
28
/
(1 - sli:http_availability:ratio) * (1 - 0.999)

Additional patterns and templates

More detailed templates and worked examples live in references/details.md. Read that file for the full pattern library.

Sponsors

Details

Marketplace
marslanalvi/SkillsStack
Source
plugins/devops/observability-sre
Category
Devops
Keywords
devops, observability-sre

More from marslanalvi/SkillsStack

All 45