Kubernetes Continual Service Improvement

Continuously improve Kubernetes platforms including SLO management, cost optimization, performance tuning, FinOps, and platform maturity.

Keywords

kubernetes, slo, sli, sla, error budget, cost optimization, finops, capacity, performance, improvement, maturity, metrics, toil, feedback, defining, managing, optimizing, measuring, reducing, planning, assessing, implementing, creating

When to Use This Skill

• Defining and managing SLOs
• Optimizing cluster costs (FinOps)
• Measuring and reducing toil
• Conducting capacity planning
• Assessing platform maturity
• Implementing feedback loops
• Creating improvement roadmaps

Related Skills

• k8s-platform-operations - SLI data sources
• k8s-platform-tenancy - Cost allocation
• k8s-security-hardening - Security posture baselines
• k8s-security-redteam - Findings backlog input
• k8s-namespace-troubleshooting - Per-namespace diagnostics
• flux-gitops-patterns - Change management

Quick Reference

SLO Framework

Service Level Indicators

Availability:

- record: platform:availability:ratio_5m
  expr: |
    sum(up{job=~"kubernetes-.*"})
    / count(up{job=~"kubernetes-.*"})

Latency (p99):

- record: platform:latency:p99_5m
  expr: |
    histogram_quantile(0.99,
      sum(rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m]))
      by (le))

Error Rate:

- record: platform:error_rate:ratio_5m
  expr: |
    sum(rate(apiserver_request_total{code=~"5.."}[5m]))
    / sum(rate(apiserver_request_total[5m]))

SLO Targets

Error Budget Alerts

- alert: ErrorBudgetBurnRate
  expr: |
    (1 - platform:availability:ratio_5m) > (1 - 0.999) * 14.4
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Error budget burning fast"

Cost Optimization (FinOps)

Efficiency Metrics

Resource Analysis

# CPU efficiency
kubectl top pods -A --no-headers | awk '{
  split($3, cpu, "m"); actual+=cpu[1]
} END {print "Total CPU:", actual, "m"}'

# Find idle deployments
kubectl get deployments -A -o json | \
  jq -r '.items[] | select(.spec.replicas==0) | "\(.metadata.namespace)/\(.metadata.name)"'

# Unmounted PVCs (no temp files)
comm -23 \
  <(kubectl get pvc -A -o json | jq -r '.items[] | select(.status.phase=="Bound") | .metadata.name' | sort) \
  <(kubectl get pods -A -o json | jq -r '.items[].spec.volumes[]?.persistentVolumeClaim.claimName' | sort -u)

Cost Reduction Strategies

Cost Allocation Labels

metadata:
  labels:
    cost-center: engineering
    team: platform
    environment: production
    application: api-gateway

Toil Measurement

Toil Indicators

• Manual, repetitive tasks
• No lasting value
• Scales with service size
• Automatable

Toil Tracking

toil_tasks:
  - name: "Manual tenant onboarding"
    frequency: "5/week"
    duration: "30min"
    annual_hours: 130
    automation_effort: "M"

  - name: "Certificate rotation"
    frequency: "4/year"
    duration: "2h"
    annual_hours: 8
    automation_effort: "S"

Toil Reduction Target

• Current: X hours/week
• Target: 50% reduction in 6 months
• Method: Automation, self-service

Platform Maturity Model

Capability Assessment

capabilities:
  provisioning:
    current: 2
    target: 4
    gap: "No self-service"
  monitoring:
    current: 3
    target: 4
    gap: "Missing SLOs"
  security:
    current: 3
    target: 4
    gap: "Manual audits"

Feedback Loops

Tenant Satisfaction (NPS)

survey:
  - "How satisfied with platform stability? (1-5)"
  - "How easy to deploy applications? (1-5)"
  - "How responsive is support? (1-5)"
  - "What should we improve?"

Platform Metrics Dashboard

dashboards:
  executive:
    - Availability %
    - Cost per tenant
    - Incident count
  tenant:
    - Resource usage
    - Deploy success rate
    - Error rates
  platform_team:
    - All SLIs
    - Error budget remaining
    - Capacity utilization

Improvement Cadence

Reporting Format

Monthly platform reports should cover: Availability (SLO vs actual, error budget remaining), Incidents (count by severity, MTTR), Cost (total, per-tenant, month-over-month change), Capacity (CPU%, memory%), Improvements delivered, Next month plan

Common Mistakes

Improvement Backlog Format

For each item: Title, Category (Performance/Reliability/Security/Cost/UX), Priority (P1-P3), Effort (S/M/L/XL), Current state, Target state, Metrics (before → target), Dependencies