Observability Logging
Use logs as a core component of comprehensive observability strategy, integrating with metrics, traces, alerts, and dashboards to achieve deep system understanding and operational excellence.
When to use me
Use this skill when:
- Building comprehensive observability platforms
- Integrating logs with metrics and tracing for full observability
- Designing alerting and monitoring systems based on log patterns
- Creating dashboards that combine log-derived insights with other telemetry
- Implementing SLO/SLA monitoring using log data
- Building incident response workflows based on log analysis
- Establishing operational excellence practices
- Designing on-call procedures and runbooks
- Implementing predictive maintenance using log patterns
- Building self-healing systems based on observability signals
What I do
1. Log-Driven Metrics
- Extract metrics from logs (error rates, latency percentiles, throughput)
- Create derived metrics from log patterns and correlations
- Implement log-based counters for business and operational events
- Calculate Service Level Indicators (SLIs) from log data
- Monitor Service Level Objectives (SLOs) using log-derived metrics
- Implement burn rate alerts for error budget consumption
- Create trend analysis from historical log patterns
2. Log-Enhanced Tracing
- Enrich traces with log context for deeper insights
- Correlate trace spans with log events for complete request understanding
- Implement log-based span creation for legacy or untraced systems
- Use logs to fill tracing gaps in distributed systems
- Create unified observability views combining logs and traces
- Implement log-to-trace linking for seamless investigation
- Use trace context in logs for correlation and analysis
3. Alerting & Monitoring
- Design log-based alerts for critical patterns and anomalies
- Implement alert deduplication and correlation across log sources
- Create escalation policies based on log pattern severity
- Design alert routing to appropriate teams and individuals
- Implement alert enrichment with log context for faster diagnosis
- Create suppression rules for known issues and maintenance windows
- Monitor alert effectiveness and adjust thresholds based on historical data
4. Dashboard & Visualization
- Create operational dashboards combining logs, metrics, and traces
- Design service health dashboards with log-derived health indicators
- Implement real-time log streaming visualizations
- Create trend dashboards showing log pattern evolution
- Design incident investigation dashboards with correlated data
- Implement customizable views for different stakeholder needs
- Create predictive dashboards using machine learning on log data
5. Incident Response
- Design log-driven runbooks for common issues
- Implement automated remediation based on log patterns
- Create incident timelines from log correlation
- Design post-mortem analysis using comprehensive log data
- Implement blameless retrospectives with observability data
- Create knowledge bases from resolved incidents and log patterns
- Design escalation procedures based on observability signals
Observability Pillars Integration
Logs + Metrics + Traces = Full Observability
Example: API Service Observability
Logs (What happened):
- "API call to /api/users failed with 500 error"
- "Database connection timeout after 5000ms"
- "Cache miss for user:123"
Metrics (How much/how often):
- Error rate: 5.2%
- P95 latency: 245ms
- Throughput: 1,234 requests/second
Traces (Where in the flow):
- Request flow: API Gateway → Auth Service → User Service → Database
- Time spent: 45ms in Auth, 120ms in User Service, 80ms in Database
- Bottleneck identified: Database query in User Service
Unified Data Model
observability_data:
logs:
source: application, infrastructure, audit
format: structured (JSON)
fields: [timestamp, level, service, message, context]
metrics:
source: logs (derived), application (direct), infrastructure
types: counter, gauge, histogram, summary
dimensions: [service, endpoint, status_code, user_type]
traces:
source: instrumentation, log-derived
context: trace_id, span_id, parent_span_id
attributes: [service.name, operation.name, duration, status]
correlations:
log_to_metric: "error logs → error rate metric"
log_to_trace: "trace_id field links logs to traces"
metric_to_trace: "high latency metric → trace analysis"
Log-Driven SLO Monitoring
Error Budget Calculation from Logs
def calculate_error_budget_from_logs(logs, slo_target, time_window):
"""
Calculate error budget consumption from log data
Args:
logs: List of log entries with timestamp and success status
slo_target: SLO target (e.g., 0.999 for 99.9%)
time_window: Time window for calculation in seconds
Returns:
error_budget_consumption: Percentage of error budget consumed
"""
total_requests = len(logs)
successful_requests = sum(1 for log in logs if log.get('status') != 'error')
success_rate = successful_requests / total_requests if total_requests > 0 else 1.0
error_rate = 1.0 - success_rate
# Calculate error budget consumption
allowed_errors = (1.0 - slo_target) * total_requests
actual_errors = total_requests - successful_requests
error_budget_consumption = actual_errors / allowed_errors if allowed_errors > 0 else float('inf')
return {
'total_requests': total_requests,
'successful_requests': successful_requests,
'success_rate': success_rate,
'error_rate': error_rate,
'slo_target': slo_target,
'allowed_errors': allowed_errors,
'actual_errors': actual_errors,
'error_budget_consumption': error_budget_consumption,
'error_budget_remaining': max(0, 1.0 - error_budget_consumption)
}
Burn Rate Alerting
alerting:
error_budget_burn_rate:
# Alert when burning error budget too quickly
- name: "high_error_budget_burn_rate"
condition: "error_budget_burn_rate > 10"
# Burning 10x faster than allowed
window: "1h"
severity: "critical"
- name: "medium_error_budget_burn_rate"
condition: "error_budget_burn_rate > 2"
# Burning 2x faster than allowed
window: "6h"
severity: "warning"
slo_violation:
- name: "slo_violation_imminent"
condition: "error_budget_remaining < 0.1"
# Less than 10% error budget remaining
window: "7d"
severity: "warning"
- name: "slo_violation_occurred"
condition: "success_rate < slo_target"
# Actual violation occurring
window: "1h"
severity: "critical"
Examples
# Extract metrics from logs for SLO monitoring
npm run observability:extract-metrics -- --slo-target 0.999 --window 7d --output slo-metrics.json
# Create unified observability dashboard
npm run observability:create-dashboard -- --services "api,auth,db" --data-sources "logs,metrics,traces"
# Design log-based alerting rules
npm run observability:design-alerts -- --patterns "error_rate > 5%,latency_p95 > 1000ms" --escalation-policy "team-rotation"
# Implement incident response workflow
npm run observability:incident-workflow -- --trigger "error_spike" --actions "page,create-incident,notify-slack"
# Correlate logs with metrics and traces
npm run observability:correlate -- --time-range "last-1h" --output correlation-analysis.json
Output format
Observability Platform Configuration:
observability:
data_sources:
logs:
collection: [filebeat, fluentd, otel-collector]
processing: [parsing, enrichment, correlation]
storage: [elasticsearch, s3]
metrics:
collection: [prometheus, otel-collector]
processing: [aggregation, derivation]
storage: [prometheus, thanos]
traces:
collection: [otel-collector, jaeger-agent]
processing: [sampling, enrichment]
storage: [jaeger, tempo]
correlation:
fields:
trace_id: ["trace_id", "trace.id", "X-Trace-Id"]
service: ["service", "service.name", "component"]
user_id: ["user_id", "user.id", "userId"]
rules:
- when: "log.level == 'ERROR'"
then: "increment_metric('errors_total', labels=log.labels)"
- when: "trace.duration > 1000"
then: "log.warning('slow_trace', trace_attributes)"
- when: "metric.name == 'error_rate' and metric.value > 0.05"
then: "create_alert('high_error_rate', severity='warning')"
dashboards:
- name: "Service Health"
panels:
- type: "timeseries"
title: "Error Rate"
query: "rate(error_logs_total[5m])"
- type: "histogram"
title: "Request Latency"
query: "histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))"
- type: "log_stream"
title: "Recent Errors"
query: "level:ERROR"
- name: "Business Metrics"
panels:
- type: "counter"
title: "User Signups"
query: "log.message:'User signed up'"
- type: "timeseries"
title: "Payment Success Rate"
query: "successful_payments / total_payments"
alerting:
rules:
- alert: "HighErrorRate"
expr: "rate(error_logs_total[5m]) > 0.05"
for: "5m"
labels:
severity: "critical"
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value }} which is above 5% threshold"
- alert: "SLOErrorBudgetBurn"
expr: "error_budget_burn_rate > 10"
for: "1h"
labels:
severity: "warning"
annotations:
summary: "Error budget burning too fast"
description: "Error budget burn rate is {{ $value }}x faster than allowed"
incident_response:
workflows:
- trigger: "alert.severity == 'critical'"
actions:
- "create_incident"
- "page_on_call"
- "notify_slack('#alerts')"
- "start_zoom_war_room"
- trigger: "incident.created"
actions:
- "gather_observability_data"
- "correlate_logs_metrics_traces"
- "suggest_runbooks"
- "update_status_page"
Observability Maturity Assessment:
Observability Maturity Assessment
────────────────────────────────
Organization: Example Corp
Assessment Date: 2026-02-26
Overall Score: 72/100
Pillar Scores:
- Logging: 85/100 (Structured, correlated, well-managed)
- Metrics: 65/100 (Basic metrics, limited derivation)
- Tracing: 56/100 (Partial implementation, gaps in coverage)
- Alerting: 70/100 (Effective but could be smarter)
- Visualization: 74/100 (Good dashboards, could be more unified)
Integration Assessment:
✅ Logs include trace context (trace_id, span_id)
✅ Metrics derived from logs (error rates, throughput)
⚠️ Traces not fully correlated with logs (60% coverage)
⚠️ Alerting not using derived SLO metrics
✅ Dashboards combine multiple data sources
Gap Analysis:
1. Missing: Unified observability data model
2. Missing: Automated correlation across pillars
3. Missing: Predictive analytics on observability data
4. Missing: Self-healing based on observability signals
5. Missing: Comprehensive SLO monitoring
Observability ROI Analysis:
- Current MTTR (Mean Time To Resolution): 45 minutes
- Target MTTR with improved observability: 15 minutes
- Estimated reduction in incident impact: $15,000/month
- Estimated improvement in developer productivity: 20%
- Estimated reduction in on-call burden: 30%
Recommendations:
1. HIGH PRIORITY: Implement unified observability data model
2. HIGH PRIORITY: Improve trace coverage and correlation
3. MEDIUM PRIORITY: Implement SLO-based alerting
4. MEDIUM PRIORITY: Add predictive analytics capabilities
5. LOW PRIORITY: Explore self-healing automation
Implementation Roadmap:
- Phase 1 (1 month): Unified data model and correlation
- Phase 2 (2 months): SLO monitoring and alerting
- Phase 3 (3 months): Predictive analytics
- Phase 4 (6 months): Self-healing capabilities
- Ongoing: Continuous improvement and optimization
Notes
- Observability is a journey, not a destination - continuous improvement is essential
- Start with the questions you need to answer - design observability around those
- Correlation across data sources is more valuable than individual source depth
- Consider the cost of observability - balance value with expense
- Involve all stakeholders - developers, operators, business teams, executives
- Measure observability effectiveness - track MTTR, incident frequency, etc.
- Document observability practices - runbooks, dashboards, alert definitions
- Regularly review and refine - observability needs evolve with the system
- Balance automation with human insight - don't automate away necessary human judgment
- Security and compliance considerations - observability data may contain sensitive information