ACCESSING CLUSTERS

CRITICAL: Always prefix kubectl/flux commands with inline KUBECONFIG assignment. Do NOT use export or &&

• the variable must be set in the same command:

CORRECT - inline assignment

KUBECONFIG=~/.kube/<cluster>.yaml kubectl get pods

WRONG - export with && breaks in some shell contexts

export KUBECONFIG=~/.kube/<cluster>.yaml && kubectl get pods

Cluster Context

CRITICAL: Always confirm cluster before running commands.

Cluster Purpose Kubeconfig

dev

Manual testing ~/.kube/dev.yaml

integration

Automated testing ~/.kube/integration.yaml

live

Production ~/.kube/live.yaml

KUBECONFIG=~/.kube/<cluster>.yaml kubectl <command>

Accessing Internal Services

Platform services are exposed through the internal ingress gateway over HTTPS. DNS URLs are useful for browser-based access (Grafana, Hubble UI, Longhorn UI).

OAuth2 Proxy caveat: Prometheus, Alertmanager, and some other services are behind OAuth2 Proxy. DNS URLs redirect to an OAuth login page and cannot be used for API queries via curl. Use kubectl exec or port-forward instead for programmatic access.

Service Live Auth API Access

Prometheus https://prometheus.internal.tomnowak.work

OAuth2 Proxy kubectl exec or port-forward

Alertmanager https://alertmanager.internal.tomnowak.work

OAuth2 Proxy kubectl exec or port-forward

Grafana https://grafana.internal.tomnowak.work

Built-in auth Browser only

Hubble UI https://hubble.internal.tomnowak.work

None Browser

Longhorn UI https://longhorn.internal.tomnowak.work

None Browser

Garage Admin https://garage.internal.tomnowak.work

None Browser

Domain pattern: <service>.internal.<cluster-suffix>.tomnowak.work

• live: internal.tomnowak.work

• integration: internal.integration.tomnowak.work

• dev: internal.dev.tomnowak.work

Querying Prometheus/Alertmanager API

Option 1: kubectl exec (quick, no setup)

KUBECONFIG=~/.kube/<cluster>.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus --
wget -qO- 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result'

KUBECONFIG=~/.kube/<cluster>.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus --
wget -qO- 'http://localhost:9090/api/v1/alerts' | jq '.data.alerts[] | select(.state == "firing")'

KUBECONFIG=~/.kube/<cluster>.yaml kubectl exec -n monitoring alertmanager-kube-prometheus-stack-0 -c alertmanager --
wget -qO- 'http://localhost:9093/api/v2/alerts' | jq .

Option 2: Port-forward (for scripts and repeated queries)

KUBECONFIG=~/.kube/<cluster>.yaml kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 & curl -s "http://localhost:9090/api/v1/query?query=up" | jq '.data.result'

Using the helper scripts:

Prometheus (start port-forward first; script defaults to http://localhost:9090)

KUBECONFIG=~/.kube/<cluster>.yaml kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 & .claude/skills/prometheus/scripts/promql.sh alerts --firing

Loki (no HTTPRoute — always requires port-forward)

KUBECONFIG=~/.kube/<cluster>.yaml kubectl port-forward -n monitoring svc/loki-headless 3100:3100 & export LOKI_URL=http://localhost:3100 .claude/skills/loki/scripts/logql.sh tail '{namespace="monitoring"}' --since 15m

Common kubectl Patterns

Read-only commands used during daily operations and investigations:

Command Purpose

kubectl get pods -n <ns>

List pods in a namespace

kubectl get pods -A

List pods across all namespaces

kubectl describe pod <pod> -n <ns>

Detailed pod info with events

kubectl logs <pod> -n <ns> --tail=100

Recent logs from a pod

kubectl logs <pod> -n <ns> --previous

Logs from previous container instance

kubectl get events -n <ns> --sort-by='.lastTimestamp'

Recent events timeline

kubectl top pods -n <ns>

CPU/memory usage per pod

kubectl top nodes

CPU/memory usage per node

kubectl get ns <ns> --show-labels

Namespace labels (network policy profiles)

kubectl explain <resource>

API schema reference for a resource type

Flux GitOps Commands

Status and Reconciliation

Check status

KUBECONFIG=/.kube/<cluster>.yaml flux get all KUBECONFIG=/.kube/<cluster>.yaml flux get kustomizations KUBECONFIG=~/.kube/<cluster>.yaml flux get helmreleases -A

Trigger reconciliation

KUBECONFIG=/.kube/<cluster>.yaml flux reconcile source git flux-system KUBECONFIG=/.kube/<cluster>.yaml flux reconcile kustomization <name> KUBECONFIG=~/.kube/<cluster>.yaml flux reconcile helmrelease <name> -n <namespace>

Flux Status Interpretation

Status Meaning Action

Ready: True

Resource is reconciled and healthy None - operating normally

Ready: False

Resource failed to reconcile Check the message/reason for details

Stalled: True

Resource has stopped retrying after repeated failures Suspend/resume to reset (see sre skill)

Suspended: True

Resource is intentionally paused Resume when ready: flux resume <type> <name>

Reconciling

Resource is actively being applied Wait for completion

Researching Unfamiliar Services

When investigating unknown services, spawn a haiku agent to research documentation:

Task tool:

• subagent_type: "general-purpose"
• model: "haiku"
• prompt: "Research [service] troubleshooting docs. Focus on:
  1. Common failure modes
  2. Health indicators
  3. Configuration gotchas Start with: [docs-url]"

Chart URL to Docs mapping:

Chart Source Documentation

charts.jetstack.io

cert-manager.io/docs

charts.longhorn.io

longhorn.io/docs

grafana.github.io

grafana.com/docs

prometheus-community.github.io

prometheus.io/docs

Common Confusions

BAD: Use helm list to check Helm release status GOOD: Use kubectl get helmrelease -A

• Flux manages releases via CRDs, not Helm CLI

Keywords

kubernetes, kubectl, kubeconfig, flux, flux status, cluster access, internal URL, service URL, port-forward, helm release, gitops, reconciliation