Troubleshoot Olakai Agent Monitoring

This skill helps diagnose and fix common issues with Olakai AI agent monitoring, KPI calculations, and SDK integration.

For full documentation, see: https://app.olakai.ai/llms.txt

The Golden Rule: Test → Fetch → Validate

Always diagnose by generating a real event and inspecting it. Don't guess - look at the actual data.

# 1. Trigger your agent/app to generate an event
# 2. Fetch the event
olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json

# 3. Inspect it completely
olakai activity get EVENT_ID --json | jq '{customData, kpiData}'

This reveals exactly what's happening:

• No event? → SDK/API key problem
• customData missing fields? → SDK code problem
• kpiData shows strings? → Formula storage problem
• kpiData shows null? → CustomDataConfig or field name problem

Quick Diagnosis Commands

Run these first to understand the current state:

# Check CLI authentication
olakai whoami

# List recent events (are any coming through?)
olakai activity list --limit 10

# List agents (is your agent registered?)
olakai agents list

# List custom data configs (are they set up?)
olakai custom-data list

# List KPIs for an agent
olakai kpis list --agent-id YOUR_AGENT_ID

# Check session/chat decoration status (for CHAT-scope KPIs)
olakai activity sessions --agent-id YOUR_AGENT_ID

Issue: No Events Appearing

Symptom

Events from your agent aren't showing up in olakai activity list or the dashboard.

Diagnostic Steps

1. Verify API Key

# Check environment variable is set
echo $OLAKAI_API_KEY

# Should start with "sk_" and be ~40+ characters
# If empty or wrong, set it:
export OLAKAI_API_KEY="sk_your_key_here"

2. Check SDK Initialization

TypeScript - ensure init() is awaited:

const olakai = new OlakaiSDK({ apiKey: process.env.OLAKAI_API_KEY! });
await olakai.init();  // <-- Must await this!

Python - ensure config is called before instrumentation:

olakai_config(os.getenv("OLAKAI_API_KEY"))  # <-- Must be first
instrument_openai()  # <-- Then instrument

3. Enable Debug Mode

TypeScript:

const olakai = new OlakaiSDK({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: true,  // <-- Add this
});

Python:

olakai_config(api_key, debug=True)

Look for error messages in console output.

4. Test Direct API Call

curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $OLAKAI_API_KEY" \
  -d '{
    "prompt": "Test prompt",
    "response": "Test response",
    "app": "test-agent"
  }'

Expected: 200 OK with event ID If 401: Invalid API key If 400: Check request format

5. Check Network/Firewall

Ensure your environment can reach https://app.olakai.ai

curl -I https://app.olakai.ai/api/health

Common Fixes

Issue: KPIs Showing String Values Instead of Numbers

Symptom

KPI values display as "VariableName" instead of numeric values like 42.

Example:

"kpiData": {
  "Tools Discovered": "ToolsDiscovered",  // Wrong: string
  "New Tools Found": "NewToolsFound"       // Wrong: string
}

Should be:

"kpiData": {
  "Tools Discovered": 20,  // Correct: number
  "New Tools Found": 3     // Correct: number
}

Root Cause

KPI formulas are stored as raw strings instead of parsed AST objects.

Diagnostic Steps

1. Check KPI Formula Storage

olakai kpis list --agent-id YOUR_AGENT_ID --json | jq '.[] | {name, calculatorParams}'

Wrong (raw string):

{
  "name": "Tools Discovered",
  "calculatorParams": {
    "formula": "ToolsDiscovered"  // <-- String, not object
  }
}

Correct (AST object):

{
  "name": "Tools Discovered",
  "calculatorParams": {
    "formula": {
      "type": "variable",
      "name": "ToolsDiscovered"
    }
  }
}

2. Check CustomDataConfig Exists

olakai custom-data list --json | jq '.[].name'

Ensure every variable referenced in KPI formulas has a corresponding CustomDataConfig.

Fix

Option A: Update via CLI (Recommended)

The CLI now validates and parses formulas automatically:

# This will parse "ToolsDiscovered" into proper AST
olakai kpis update KPI_ID --formula "ToolsDiscovered"

Option B: Validate First, Then Check

# Validate the formula
olakai kpis validate --formula "ToolsDiscovered" --agent-id YOUR_AGENT_ID

# Should return:
# {
#   "valid": true,
#   "type": "number",
#   "parsedFormula": { "type": "variable", "name": "ToolsDiscovered" }
# }

Option C: Recreate the KPI

# Delete the broken KPI
olakai kpis delete KPI_ID --force

# Create with proper formula (CLI now parses automatically)
olakai kpis create \
  --name "Tools Discovered" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "ToolsDiscovered" \
  --aggregation SUM

Post-Fix Verification

After fixing, trigger a new event and check:

olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json | jq '.prompts[0].kpiData'

Should now show numeric values.

Issue: CustomData Not Appearing in Events

Symptom

You're sending customData but it's not showing in event details.

Diagnostic Steps

1. Check Event Details

olakai activity get EVENT_ID --json | jq '.customData'

If null or missing fields, the SDK isn't sending them.

2. Verify SDK Code

TypeScript - customData in call options:

const response = await openai.chat.completions.create(
  { model: "gpt-4o", messages },
  {
    customData: {
      myField: "value",      // <-- Check this is present
      myNumber: 42,
    },
  }
);

TypeScript - customData in manual event:

olakai.event({
  prompt,
  response,
  customData: {
    myField: "value",
    myNumber: 42,
  },
});

3. Check Field Names Match CustomDataConfig

Field names are case-sensitive. Ensure exact match:

# What you configured
olakai custom-data list --json | jq '.[].name'
# Output: "ToolsDiscovered", "NewToolsFound"

# What you're sending (must match exactly)
customData: {
  ToolsDiscovered: 20,    // Correct
  toolsDiscovered: 20,    // WRONG - case mismatch
  tools_discovered: 20,   // WRONG - different format
}

Common Fixes

Issue: customData Fields Not Available in KPIs

Symptom

You're sending fields in customData but they can't be used in KPI formulas - they don't appear as available variables.

Root Cause

The SDK accepts any JSON in customData, but only fields with CustomDataConfigs become KPI variables.

SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula
                        ↑
                  REQUIRED for field
                  to be usable in KPIs

Fields without CustomDataConfigs:

• ✅ Are stored in the event record
• ❌ Cannot be referenced in KPI formulas
• ❌ Don't appear in formula variable suggestions
• ❌ Effectively wasted for analytics purposes

Diagnostic

1. List what CustomDataConfigs exist:

olakai custom-data list --json | jq '.[].name'

2. Compare to what you're sending in SDK:

// If you're sending these fields:
customData: {
  ItemsProcessed: 10,   // Is there a CustomDataConfig for this?
  SuccessRate: 0.95,    // Is there a CustomDataConfig for this?
  RandomField: "xyz",   // Is there a CustomDataConfig for this?
}

3. Check for mismatches:

• Field sent but no config → Cannot use in KPIs
• Config exists but field not sent → KPI shows null

Fix

Create CustomDataConfigs for every field you want to use in KPIs:

# For each field you need in KPIs, create a config (replace YOUR_AGENT_ID)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "RandomField" --type STRING

# Verify for this agent
olakai custom-data list --agent-id YOUR_AGENT_ID

Best Practice: Design your CustomDataConfigs FIRST, then write SDK code that sends only those fields.

Common Mistakes

Issue: KPIs from Another Agent Not Working

Symptom

You created KPIs for one agent and expected them to apply to a different agent. The second agent's events show no kpiData, or olakai kpis list --agent-id SECOND_AGENT_ID returns an empty list.

Root Cause

KPIs are unique per agent. Each KPI definition is bound to exactly one agent by its agentId. KPIs cannot be shared, inherited, or reused across agents — even within the same workflow or account.

Diagnostic

# Check KPIs on the FIRST agent (where they were originally created)
olakai kpis list --agent-id AGENT_A_ID --json | jq '.[].name'
# Output: "Items Processed", "Success Rate"

# Check KPIs on the SECOND agent (where you expected them to work)
olakai kpis list --agent-id AGENT_B_ID --json | jq '.[].name'
# Output: (empty) ← KPIs don't carry over

Fix

Create the KPIs separately for the second agent:

# Recreate each KPI for the new agent
olakai kpis create \
  --name "Items Processed" \
  --agent-id AGENT_B_ID \
  --calculator-id formula \
  --formula "ItemsProcessed" \
  --aggregation SUM

olakai kpis create \
  --name "Success Rate" \
  --agent-id AGENT_B_ID \
  --calculator-id formula \
  --formula "SuccessRate * 100" \
  --aggregation AVERAGE

Prevention

When creating a new agent, always:

1. Check if KPIs are needed: "Does this agent need performance metrics?"
2. Create KPIs explicitly for the new agent — don't assume they exist from another agent
3. Verify with olakai kpis list --agent-id NEW_AGENT_ID

Note: CustomDataConfigs do NOT need to be recreated — they are account-level and shared. Only KPIs are agent-specific.

Issue: Redundant customData Fields

Symptom

CustomDataConfigs exist for fields that are already tracked by the platform (sessionId, agentId, timestamps, etc.), cluttering the configuration.

Why This Happens

The SDK accepts any JSON in customData, so agents sometimes send "helpful" extra data that's already tracked elsewhere.

What's Already Tracked (Don't Duplicate)

Fix

1. Identify redundant configs:

olakai custom-data list --json | jq '.[].name'

Look for names like: sessionId, agentId, timestamp, model, provider, tokenCount

2. Check if they're used in KPIs:

olakai kpis list --agent-id YOUR_AGENT_ID --json | jq '.[].calculatorParams.formula'

3. If not used in KPIs, remove from SDK code: Don't delete the configs (they may have historical data), but stop sending these fields.

4. Update SDK code to only send KPI-relevant fields:

customData: {
  // ✅ Keep: Used in KPI formulas
  ItemsProcessed: 10,
  SuccessRate: 1.0,

  // ❌ Remove: Already tracked by platform
  // sessionId: session.id,      // Already tracked
  // agentId: agentConfig.id,    // Already tracked
  // timestamp: Date.now(),      // Already tracked
}

Best Practice

Before creating a CustomDataConfig, ask:

• "Will I use this in a KPI formula?" → Yes = Create
• "Will I filter events by this?" → Yes = Create
• "Is this already tracked by the platform?" → Yes = Don't create

Issue: KPIs Show null Values

Symptom

KPI values are null instead of numbers.

"kpiData": {
  "Success Rate": null,
  "Items Processed": null
}

Root Cause Options

1. CustomData field not sent - The event doesn't include that field
2. CustomDataConfig doesn't exist - Platform can't resolve the variable
3. Type mismatch - Sending string when NUMBER expected

Diagnostic Steps

1. Check if CustomData is Present

olakai activity get EVENT_ID --json | jq '.customData'

If the field is missing from customData, the KPI can't calculate.

2. Check CustomDataConfig Exists

olakai custom-data list --json | jq '.[].name'

Every field referenced in KPI formulas needs a config.

3. Check Formula Validity

olakai kpis validate --formula "YourVariable" --agent-id YOUR_AGENT_ID

If invalid, you'll see the error.

Fix

1. Ensure CustomDataConfig Exists

olakai custom-data create --agent-id YOUR_AGENT_ID --name "YourVariable" --type NUMBER

2. Ensure SDK Sends the Field

customData: {
  YourVariable: 42,  // Must be present and correct type
}

3. Trigger New Event

Old events won't recalculate. Generate a new event to verify the fix.

Issue: Classifier KPI Shows Null or Not Updating

Symptom

A classifier KPI (created from a template like sentiment_scorer or time_saved_estimator) shows null values or does not update after events are sent.

Root Cause Options

1. Chat has too few turns — CHAT-scoped classifier KPIs need the conversation (session) to have enough turns before the classifier can produce a meaningful result.
2. Chat decoration has not run yet — Classifier KPIs run during chat decoration, which happens after the conversation is completed or after a delay (not on every individual event).
3. Missing sessionId — Events are not grouped into a conversation because sessionId is not being passed, so there is no chat to decorate.

Diagnostic Steps

1. Check the KPI scope

olakai kpis list --agent-id YOUR_AGENT_ID --json | jq '.[] | {name, scope, calculatorId}'

Classifier KPIs should have "scope": "CHAT" and "calculatorId": "classifier".

2. Verify events have a sessionId

olakai activity list --agent-id YOUR_AGENT_ID --limit 5 --json | jq '.prompts[] | {id, sessionId}'

If sessionId is null or different for each event, the platform cannot group them into a conversation for decoration.

3. Wait for chat decoration

Chat decoration (which triggers classifier evaluation) runs after a conversation is considered complete or after a processing delay. If you just sent events, wait a few minutes and check again.

4. Inspect session decoration status

olakai activity sessions --agent-id YOUR_AGENT_ID

This shows a summary of how many sessions (chats) have been decorated vs. still pending. If most sessions show NEW status, the decoration pipeline hasn't processed them yet. If sessions show DECORATION_FAILED, check the error column for details.

# Get full diagnostic details as JSON
olakai activity sessions --agent-id YOUR_AGENT_ID --json

Fix

1. Ensure SDK sends a consistent sessionId:

// All turns in the same conversation must share a sessionId
olakai.event({
  prompt: userMessage,
  response: aiResponse,
  sessionId: conversationId,  // Same ID for all turns in this conversation
  userEmail: user.email,
});

2. Ensure the conversation has enough data:

Classifier KPIs analyze the full conversation context. A single-turn chat may not produce useful results for sentiment analysis. Send at least 2-3 turns before expecting a value.

3. Check that the template exists:

olakai kpis templates

Verify the template-id you used when creating the KPI is a valid template.

Issue: CLI Authentication Failed

Symptom

Error: Authentication required
Error: Token expired
Error: Unauthorized

Fix

1. Re-authenticate

olakai logout
olakai login

2. Verify Authentication

olakai whoami

3. Check Credentials File

ls -la ~/.config/olakai/
cat ~/.config/olakai/credentials.json

Should contain accessToken and refreshToken.

4. Check Environment

# Default is production
olakai whoami

# For staging
OLAKAI_ENV=staging olakai whoami

Issue: Events Not Associated with Agent

Symptom

Events appear in general activity but not under your agent.

Root Cause

The app field in events doesn't match the agent name.

Diagnostic

# Check agent name
olakai agents list --json | jq '.[] | {id, name}'

# Check event app field
olakai activity get EVENT_ID --json | jq '.app'

Fix

Option A: Match App Name to Agent

In your SDK code, ensure the app name matches:

olakai.event({
  prompt,
  response,
  app: "Your Agent Name",  // Must match agent name exactly
});

Or with wrapped client:

const openai = olakai.wrap(new OpenAI({ apiKey }), {
  provider: "openai",
  defaultContext: {
    app: "Your Agent Name",
  },
});

Option B: Update Agent Name

olakai agents update AGENT_ID --name "App Name From Events"

Issue: High Latency / Slow Events

Symptom

LLM calls take much longer when monitoring is enabled.

Diagnostic

Check if it's the SDK or the LLM:

const startSDK = Date.now();
const response = await openai.chat.completions.create(...);
console.log(`Total time: ${Date.now() - startSDK}ms`);

Fixes

1. SDK Uses Fire-and-Forget

The SDK should NOT block your application. If it is:

// Ensure you're not awaiting event()
olakai.event(params);  // No await - fire and forget

2. Check Network Latency

time curl -I https://app.olakai.ai/api/health

If >500ms, consider:

• Running in a region closer to US-East
• Checking for proxy/VPN overhead

3. Disable Debug Mode in Production

const olakai = new OlakaiSDK({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: false,  // Disable in production
});

Issue: Duplicate Events

Symptom

Each LLM call creates multiple events.

Root Cause

Usually double-initialization or multiple wrapping.

Diagnostic

Search your code for:

• Multiple olakai.init() calls
• Multiple olakai.wrap() calls
• Multiple instrument_openai() calls

Fix

Use Singleton Pattern

// lib/olakai.ts
let instance: OlakaiSDK | null = null;

export async function getOlakai(): Promise<OlakaiSDK> {
  if (!instance) {
    instance = new OlakaiSDK({ apiKey: process.env.OLAKAI_API_KEY! });
    await instance.init();
  }
  return instance;
}

Diagnostic Commands Reference

# Authentication
olakai whoami                          # Check current user
olakai login                           # Re-authenticate

# Events/Activity
olakai activity list --limit N         # Recent events
olakai activity list --agent-id ID     # Events for specific agent
olakai activity get EVENT_ID --json    # Full event details

# Agents
olakai agents list                     # All agents
olakai agents list --json | jq '.[].name'  # Just names

# KPIs
olakai kpis list --agent-id ID         # KPIs for agent
olakai kpis list --agent-id ID --json | jq '.[] | {name, calculatorParams}'
olakai kpis validate --formula "X"     # Test formula

# Custom Data
olakai custom-data list                # All configs
olakai custom-data list --json | jq '.[].name'  # Just names

# Quick Health Check
olakai whoami && olakai activity list --limit 1 && echo "OK"

Troubleshooting Decision Tree

Events not appearing?
├── Check API key is set → export OLAKAI_API_KEY=sk_...
├── Check SDK initialized → await olakai.init()
├── Enable debug mode → debug: true
└── Test direct API → curl POST /api/monitoring/prompt

KPIs showing strings?
├── Check formula storage → olakai kpis list --json
├── Formula is raw string → olakai kpis update ID --formula "X"
└── Missing CustomDataConfig → olakai custom-data create --agent-id ID

KPIs showing null?
├── Check customData sent → olakai activity get ID --json
├── Field missing → Add to customData in SDK
├── CustomDataConfig missing → olakai custom-data create --agent-id ID
└── Type mismatch → NUMBER needs number, STRING needs string

customData field not usable in KPIs?
├── Check CustomDataConfig exists → olakai custom-data list --agent-id ID
├── Config missing → olakai custom-data create --agent-id ID --name "Field" --type NUMBER
└── Case mismatch → Ensure exact case match between SDK and config

Events not under agent?
├── Check app name matches → Compare event.app to agent.name
├── Mismatch → Update agent name or SDK app field

KPIs not appearing on new agent?
├── KPIs are agent-specific, NOT shared across agents
├── Check KPIs exist for THIS agent → olakai kpis list --agent-id THIS_AGENT_ID
├── If empty → Create KPIs for this agent (can't reuse from other agents)
└── CustomDataConfigs ARE shared → no need to recreate those

Classifier KPI showing null or not updating?
├── Check KPI scope is CHAT → olakai kpis list --agent-id ID --json
├── Check events have sessionId → olakai activity list --agent-id ID --json
├── sessionId missing → Add sessionId to SDK event calls
├── Too few turns → Send 2-3+ conversation turns before expecting a value
└── Just sent events → Wait for chat decoration (runs after delay/completion)

Key Insight: The customData → KPI Pipeline

Only fields registered as CustomDataConfigs become available in KPI formulas:

SDK customData → CustomDataConfig → Context Variable → KPI Formula → kpiData
       ↓                ↓                  ↓               ↓            ↓
  Any JSON         Schema definition   Available var   Expression   Computed value
                   (REQUIRED)

Common pitfall: Sending extra fields in customData without CustomDataConfigs - they're stored but unusable for KPIs.