Groq Observability

Overview

Monitor Groq LPU inference API for latency, token throughput, and cost. Groq's defining characteristic is extreme speed -- responses arrive in 50-200ms for small completions, with token generation rates of 500-800 tokens/second.

Prerequisites

• Groq API integration at api.groq.com

• Metrics backend (Prometheus or similar)

• Understanding of Groq's rate limit structure (per-key RPM and TPM)

Instructions

Step 1: Instrument the Groq Client

import Groq from 'groq-sdk';

async function trackedCompletion(groq: Groq, model: string, messages: any[]) { const start = performance.now(); const res = await groq.chat.completions.create({ model, messages }); const duration = performance.now() - start; const tps = (res.usage?.completion_tokens || 0) / (duration / 1000); # 1000: 1 second in ms

emitHistogram('groq_latency_ms', duration, { model }); emitGauge('groq_tokens_per_second', tps, { model }); emitCounter('groq_tokens_total', res.usage?.total_tokens || 0, { model, direction: 'total' }); emitCounter('groq_tokens_total', res.usage?.prompt_tokens || 0, { model, direction: 'input' }); emitCounter('groq_tokens_total', res.usage?.completion_tokens || 0, { model, direction: 'output' });

// Groq pricing is very low -- track for volume visibility const pricing: Record<string, number> = { 'llama-3.3-70b-versatile': 0.59, 'llama-3.1-8b-instant': 0.05, 'mixtral-8x7b-32768': 0.24 }; # 32768 = configured value const costPer1M = pricing[model] || 0.10; emitCounter('groq_cost_usd', (res.usage?.total_tokens || 0) / 1e6 * costPer1M, { model });

return res; }

Step 2: Track Rate Limit Headers

// Groq returns rate limit info in response headers function parseRateLimitHeaders(headers: Headers) { emitGauge('groq_rate_limit_remaining_requests', parseInt(headers.get('x-ratelimit-remaining-requests') || '0')); emitGauge('groq_rate_limit_remaining_tokens', parseInt(headers.get('x-ratelimit-remaining-tokens') || '0')); const resetMs = parseInt(headers.get('x-ratelimit-reset-requests') || '0'); emitGauge('groq_rate_limit_reset_ms', resetMs); }

Step 3: Alert on Performance Degradation

groups:

• name: groq rules:
  • alert: GroqLatencyHigh expr: histogram_quantile(0.95, rate(groq_latency_ms_bucket[5m])) > 1000 # 1000: 1 second in ms annotations: { summary: "Groq P95 latency exceeds 1 second (normally <200ms)" }
  • alert: GroqRateLimitNear expr: groq_rate_limit_remaining_requests < 10 annotations: { summary: "Groq rate limit nearly exhausted -- requests may be throttled" }
  • alert: GroqThroughputDrop expr: groq_tokens_per_second < 100 for: 5m annotations: { summary: "Groq tokens/sec dropped below 100 (expected 500+)" } # HTTP 500 Internal Server Error
  • alert: GroqErrors expr: rate(groq_errors_total[5m]) > 0.05 annotations: { summary: "Groq API error rate elevated" }

Step 4: Build a Speed-Focused Dashboard

Key panels: time-to-first-token distribution (Groq's USP), tokens/second by model, rate limit utilization gauge, request volume by model, cumulative cost (low but trackable), and latency comparison across models. Since Groq's main value is speed, latency degradation is the highest-priority alert.

Step 5: Log Request Metadata

{"ts":"2026-03-10T14:30:00Z","model":"llama-3.3-70b-versatile","latency_ms":142,"ttft_ms":48,"tokens_per_sec":623,"prompt_tokens":256,"completion_tokens":89,"cost_usd":0.00002,"rate_limit_remaining":45} # 2026: 256: 623 = configured value

Error Handling

Issue Cause Solution

429 with long retry-after RPM or TPM limit hit Implement token-aware request queuing

Latency spike to 2s+ Model overloaded or large prompt Reduce prompt size or switch to lighter model

503 Service Unavailable

Groq capacity issue Implement fallback to alternative provider

Tokens/sec drop Streaming disabled Enable streaming for better perceived performance

Examples

Basic usage: Apply groq observability to a standard project setup with default configuration options.

Advanced scenario: Customize groq observability for production environments with multiple constraints and team-specific requirements.

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale

Resources

• Official monitoring documentation

• Community best practices and patterns

• Related skills in this plugin pack