Searching MLflow Traces

Trace Data Structure

• TraceInfo: trace_id, status (OK/ERROR), timestamp_ms, execution_time_ms, tags, metadata, assessments (human feedback, evaluation results)
• Spans: Tree of operations with name, type, attributes, start_time, end_time

Workflow

1. Check CLI usage (required): mlflow traces search --help
2. Build filter query using syntax below
3. Execute search with appropriate flags
4. Retrieve details for specific traces if needed

Step 1: Check CLI Usage

mlflow traces search --help

Always run this first to get accurate flags for the installed MLflow version.

Step 2-3: Search Examples

# By status
mlflow traces search --experiment-id 1 --filter-string-string "trace.status = 'ERROR'"

# Output format (table or json)
mlflow traces search --experiment-id 1 --output json

# Include span details
mlflow traces search --experiment-id 1 --include-spans

# Order results
mlflow traces search --experiment-id 1 --order-by "timestamp_ms DESC"

# Pagination
mlflow traces search --experiment-id 1 --max-results 50 --page-token <token>

# Time range filter (timestamps in milliseconds since epoch)
# Get current time in ms: $(date +%s)000
# Last hour: $(( $(date +%s)000 - 3600000 ))
mlflow traces search --experiment-id 1 --filter-string "trace.timestamp_ms > $(( $(date +%s)000 - 3600000 ))"

# By execution time (slow traces > 1 second)
mlflow traces search --experiment-id 1 --filter-string "trace.execution_time_ms > 1000"

# By tag
mlflow traces search --experiment-id 1 --filter-string "tag.environment = 'production'"

# Escape special characters in tag/metadata names with backticks
mlflow traces search --experiment-id 1 --filter-string "tag.\`model-name\` = 'gpt-4'"
mlflow traces search --experiment-id 1 --filter-string "metadata.\`user.id\` = 'abc'"

# By metadata
mlflow traces search --experiment-id 1 --filter-string "metadata.user_id = 'user_123'"

# By assessment
mlflow traces search --experiment-id 1 --filter-string "feedback.rating = 'positive'"

# Combine conditions (AND only, no OR)
mlflow traces search --experiment-id 1 --filter-string "trace.status = 'ERROR' AND trace.execution_time_ms > 500"

# Full text search
mlflow traces search --experiment-id 1 --filter-string "trace.text LIKE '%error%'"

# Limit results
mlflow traces search --experiment-id 1 --filter-string "trace.status = 'OK'" --max-results 10

Step 4: Retrieve Single Trace

mlflow traces get --trace-id <trace_id>

Filter Syntax

For detailed syntax, fetch from documentation:

WebFetch(
  url: "https://mlflow.org/docs/latest/genai/tracing/search-traces.md",
  prompt: "Extract the filter syntax table showing supported fields, operators, and examples."
)

Common filters:

• trace.status: OK, ERROR, IN_PROGRESS
• trace.execution_time_ms, trace.timestamp_ms: numeric comparison
• tag.<key>, metadata.<key>: exact match or pattern
• span.name, span.type: exact match or pattern
• feedback.<name>, expectation.<name>: assessments

Pattern operators: LIKE, ILIKE (case-insensitive), RLIKE (regex)

Python API

For mlflow.search_traces(), see: https://mlflow.org/docs/latest/genai/tracing/search-traces.md