Clay Reliability Patterns

Overview

Production reliability patterns for Clay data enrichment pipelines. Clay's async enrichment model, credit-based billing, and multi-provider waterfall architecture require specific resilience strategies.

Prerequisites

• Clay API access configured

• Understanding of Clay's async enrichment model

• Queue infrastructure (Redis) for retry handling

Instructions

Step 1: Implement Enrichment Job Tracking

Track submitted jobs in Redis with table ID, row count, submission time, and status. Detect stuck enrichments by checking elapsed time against a 30-minute timeout.

Step 2: Add Credit Budget Circuit Breaker

Track daily credit usage in Redis. Before each batch, check if estimated credits would exceed the daily limit. Stop processing when budget is exhausted to prevent runaway costs.

Step 3: Monitor Provider Health

Track success/failure rates per enrichment provider over a sliding window of 100 requests. Filter out providers below 50% success rate from waterfall configurations.

Step 4: Use Dead Letter Queue for Failed Batches

Process rows in batches of 50 with 2-second delays. On failure, push the batch to a DLQ for manual review instead of blocking the pipeline.

For complete Python implementations with Redis-backed tracking, load the reference guide: Read(${CLAUDE_SKILL_DIR}/references/implementation-guide.md)

Error Handling

Issue Cause Solution

Runaway credit spend No budget limit Implement credit circuit breaker

Stuck enrichments Async job not completing Track jobs with timeout detection

Provider degradation Upstream data source down Monitor provider success rates

Batch failures API errors mid-batch DLQ for failed rows, retry later

Resources

• Clay API Reference

• Enrichment Best Practices

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale

See Clay Reliability Patterns implementation details for output format specifications.

Examples

Basic usage: Apply clay reliability patterns to a standard project setup with default configuration options.

Advanced scenario: Customize clay reliability patterns for production environments with multiple constraints and team-specific requirements.