Broker Configurations

Purpose: Comprehensive message broker configuration for Celery with production-ready patterns for Redis, RabbitMQ, and Amazon SQS.

Activation Triggers:

• Broker connection errors

• Setting up new Celery project

• Implementing high availability

• SSL/TLS configuration needed

• Performance tuning required

• Multi-broker comparison

• Cloud deployment (AWS/GCP/Azure)

• Sentinel or cluster setup

Key Resources:

• templates/redis-config.py

• Production Redis configuration

• templates/rabbitmq-config.py

• RabbitMQ with quorum queues

• templates/sqs-config.py

• AWS SQS with IAM roles

• templates/connection-strings.env

• Connection string formats

• templates/ssl-config.py

• SSL/TLS configuration

• scripts/test-broker-connection.sh

• Connection testing

• scripts/setup-redis.sh

• Redis installation and configuration

• scripts/setup-rabbitmq.sh

• RabbitMQ cluster setup

• examples/redis-sentinel.md

• High availability with Sentinel

• examples/rabbitmq-ha.md

• RabbitMQ clustering and quorum queues

• examples/sqs-setup.md

• Complete AWS SQS integration

Broker Selection Guide

Quick Comparison

Feature Redis RabbitMQ SQS

Performance Excellent (small msgs) Very Good Good

Reliability Good (with Sentinel) Excellent (quorum) Excellent

Monitoring Yes Yes Limited

Remote Control Yes Yes No

Management Manual/Cloud Manual Fully Managed

Cost Server/Cloud Server/Cloud Pay-per-use

Best For Speed, simple setup Reliability, features AWS, serverless

Decision Matrix

Choose Redis when:

• Need maximum speed for small messages

• Want simple setup and maintenance

• Already using Redis for caching/results

• Budget for managed Redis (Upstash, ElastiCache)

• Can accept brief downtime during failover

Choose RabbitMQ when:

• Need guaranteed message delivery

• Require complex routing patterns

• Need worker remote control features

• Want built-in monitoring and management UI

• Have dedicated ops team for management

Choose SQS when:

• Running on AWS infrastructure

• Want zero operational overhead

• Need unlimited automatic scaling

• Prefer pay-per-use pricing

• Don't need worker remote control

Redis Broker Setup

1. Basic Configuration

Install dependencies

pip install "celery[redis]"

Use template

cp templates/redis-config.py celeryconfig.py

Configure environment

cp templates/connection-strings.env .env

Edit .env with actual values

Key settings:

templates/redis-config.py

broker_url = 'redis://:password@localhost:6379/0'

broker_transport_options = { 'visibility_timeout': 3600, 'retry_on_timeout': True, 'max_connections': 50, }

2. Production Settings

CRITICAL: Set maxmemory-policy noeviction in Redis config:

Run setup script

./scripts/setup-redis.sh --install --configure

Or manually

redis-cli CONFIG SET maxmemory-policy noeviction

Why: Prevents Redis from evicting task data, which would cause task loss.

3. High Availability with Sentinel

See comprehensive guide

cat examples/redis-sentinel.md

Quick setup

docker-compose -f examples/docker-compose.sentinel.yml up -d

Configure Celery for Sentinel

CELERY_BROKER_URL='sentinel://host1:26379;host2:26379;host3:26379/0'

Connection string format:

Sentinel URL

broker_url = 'sentinel://sentinel1:26379;sentinel2:26379/0'

broker_transport_options = { 'master_name': 'mymaster', 'sentinel_kwargs': {'password': 'sentinel_password'}, }

RabbitMQ Broker Setup

1. Basic Configuration

Install dependencies

pip install "celery[amqp]"

Use template

cp templates/rabbitmq-config.py celeryconfig.py

Run setup script

./scripts/setup-rabbitmq.sh --install --configure

Key settings:

templates/rabbitmq-config.py

broker_url = 'amqp://user:password@localhost:5672/vhost'

CRITICAL for quorum queues

broker_transport_options = { 'confirm_publish': True, # Required! }

2. Quorum Queues (Recommended)

Provides:

• Automatic replication

• Leader election on failure

• No message loss

from kombu import Queue

task_queues = ( Queue( 'default', queue_arguments={ 'x-queue-type': 'quorum', 'x-delivery-limit': 3, } ), )

3. High Availability Cluster

See comprehensive guide

cat examples/rabbitmq-ha.md

Docker cluster setup

docker-compose -f examples/docker-compose.rabbitmq-cluster.yml up -d

Verify cluster

docker exec rabbitmq-1 rabbitmqctl cluster_status

HAProxy load balancing:

Distribute connections across cluster

broker_url = 'amqp://user:password@haproxy:5670/vhost'

AWS SQS Broker Setup

1. IAM Configuration

Create IAM policy and user

aws iam create-policy
--policy-name CelerySQSPolicy
--policy-document file://examples/celery-sqs-policy.json

Or use IAM role (recommended for EC2/ECS)

See examples/sqs-setup.md for complete guide

2. Basic Configuration

Install dependencies

pip install "celery[sqs]"

Use template

cp templates/sqs-config.py celeryconfig.py

With IAM role (recommended):

No credentials needed

broker_url = 'sqs://'

broker_transport_options = { 'region': 'us-east-1', 'visibility_timeout': 3600, 'polling_interval': 1, 'wait_time_seconds': 10, # Long polling }

With explicit credentials:

broker_url = 'sqs://access_key:secret_key@'

3. FIFO Queues

For ordered task processing:

task_queues = ( Queue( 'celery-default.fifo', queue_arguments={ 'FifoQueue': 'true', 'ContentBasedDeduplication': 'true', } ), )

Send with message group ID

task.apply_async( args=[data], properties={'MessageGroupId': 'user-123'} )

4. Result Backend

SQS doesn't support results - use S3 or DynamoDB:

Option 1: S3

result_backend = 's3://my-bucket/celery-results/'

Option 2: DynamoDB

result_backend = 'dynamodb://' result_backend_transport_options = { 'table_name': 'celery-results', }

Option 3: Redis (hybrid)

result_backend = 'redis://redis.example.com:6379/0'

SSL/TLS Configuration

1. Redis with SSL

Use templates/ssl-config.py

from templates.ssl_config import app

Or manually

broker_url = 'rediss://password@host:6380/0' # Note: rediss://

broker_use_ssl = { 'ssl_cert_reqs': ssl.CERT_REQUIRED, 'ssl_ca_certs': '/path/to/ca.pem', 'ssl_certfile': '/path/to/client-cert.pem', 'ssl_keyfile': '/path/to/client-key.pem', }

2. RabbitMQ with SSL

broker_url = 'amqps://user:password@host:5671/vhost' # Note: amqps://

broker_use_ssl = { 'ssl_cert_reqs': ssl.CERT_REQUIRED, 'ssl_ca_certs': '/path/to/ca.pem', 'ssl_certfile': '/path/to/client-cert.pem', 'ssl_keyfile': '/path/to/client-key.pem', }

3. Environment Variables

.env

BROKER_SSL_ENABLED=true BROKER_SSL_CERT=/path/to/client-cert.pem BROKER_SSL_KEY=/path/to/client-key.pem BROKER_SSL_CA=/path/to/ca-cert.pem BROKER_SSL_VERIFY_MODE=CERT_REQUIRED

Testing and Validation

1. Test Connection

Test any broker type

./scripts/test-broker-connection.sh redis ./scripts/test-broker-connection.sh rabbitmq ./scripts/test-broker-connection.sh sqs

Check specific features

./scripts/test-broker-connection.sh redis --ssl

Script checks:

• Broker connectivity

• Authentication

• SSL/TLS validation

• Configuration correctness

• Performance baseline

2. Python Test

from celery import Celery

app = Celery(broker='redis://localhost:6379/0')

Test connection

try: with app.connection() as conn: conn.ensure_connection(max_retries=3, timeout=5) print("✅ Connection successful") except Exception as e: print(f"❌ Connection failed: {e}")

Common Issues and Solutions

Redis: "maxmemory-policy not noeviction"

Problem: Redis evicting task data Solution:

./scripts/setup-redis.sh --configure

Or manually:

redis-cli CONFIG SET maxmemory-policy noeviction

RabbitMQ: "Basic.publish: NOT_FOUND"

Problem: Queue doesn't exist or wrong vhost Solution:

Check vhost

rabbitmqctl list_vhosts

Check permissions

rabbitmqctl list_permissions -p /celery_vhost

SQS: "Access Denied"

Problem: IAM permissions insufficient Solution:

Verify IAM policy includes all required actions

See examples/sqs-setup.md for complete policy

Test credentials

aws sts get-caller-identity aws sqs list-queues --queue-name-prefix celery-

Connection Timeout

Problem: Network/firewall blocking connection Solution:

Test network connectivity

telnet broker-host 6379 # Redis telnet broker-host 5672 # RabbitMQ

Check firewall rules

Verify security groups (AWS)

Check iptables rules

Performance Tuning

Redis Optimization

broker_transport_options = { 'visibility_timeout': 3600, 'max_connections': 100, # Increase for high concurrency 'socket_timeout': 5.0, 'socket_keepalive': True, 'health_check_interval': 30, }

Worker settings

worker_prefetch_multiplier = 4 # Prefetch 4x concurrency worker_max_tasks_per_child = 1000

RabbitMQ Optimization

broker_transport_options = { 'confirm_publish': True, 'max_retries': 3, }

Use quorum queues for reliability

Adjust prefetch for throughput

worker_prefetch_multiplier = 4

Disable QoS for maximum speed (classic queues only)

Note: Not compatible with worker autoscaling

SQS Optimization

Reduce API calls (costs)

broker_transport_options = { 'polling_interval': 5, # Poll less frequently 'wait_time_seconds': 20, # Max long polling }

CRITICAL: Prevent visibility timeout

worker_prefetch_multiplier = 1 # Must be 1 for SQS

Monitoring

Key Metrics

Redis:

• Connection count: INFO clients

• Memory usage: INFO memory

• Key count: DBSIZE

• Commands/sec: INFO stats

RabbitMQ:

• Queue length: rabbitmqctl list_queues

• Consumer count: Check management UI

• Memory usage: rabbitmqctl status

• Message rate: Check management UI

SQS:

• ApproximateNumberOfMessagesVisible : Queue backlog

• NumberOfMessagesSent : Task creation rate

• NumberOfMessagesReceived : Task completion rate

• NumberOfEmptyReceives : Polling efficiency

Health Check Script

#!/bin/bash

health-check.sh

BROKER_TYPE="${1:-redis}"

case "$BROKER_TYPE" in redis) redis-cli PING || exit 1 ;; rabbitmq) rabbitmqctl status || exit 1 ;; sqs) aws sqs list-queues --queue-name-prefix celery- || exit 1 ;; esac

echo "✅ Broker healthy"

Security Best Practices

1. Authentication

• Redis: Always set requirepass

• RabbitMQ: Use strong passwords, dedicated vhosts

• SQS: Use IAM roles, not access keys

2. Network Security

• Use SSL/TLS for production

• Configure firewalls/security groups

• Use VPC/private networks when possible

• Restrict broker access to worker nodes only

3. Access Control

• Principle of least privilege

• Separate credentials per environment

• Rotate passwords regularly

• Use secret management (Vault, Doppler, AWS Secrets Manager)

4. Monitoring

• Enable audit logging

• Monitor authentication failures

• Alert on configuration changes

• Track connection patterns

Resources

Templates:

• redis-config.py

• Production Redis configuration

• rabbitmq-config.py

• RabbitMQ with quorum queues

• sqs-config.py

• AWS SQS with IAM

• connection-strings.env

• All connection formats

• ssl-config.py

• SSL/TLS configuration

Scripts:

• test-broker-connection.sh

• Test any broker

• setup-redis.sh

• Redis installation and config

• setup-rabbitmq.sh

• RabbitMQ cluster setup

Examples:

• redis-sentinel.md

• High availability setup

• rabbitmq-ha.md

• Clustering and quorum queues

• sqs-setup.md

• Complete AWS integration

Documentation:

• Redis: https://redis.io/docs/

• RabbitMQ: https://www.rabbitmq.com/documentation.html

• SQS: https://docs.aws.amazon.com/sqs/

• Celery Brokers: https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/

Version: 1.0.0 Celery Compatibility: 5.0+ Supported Brokers: Redis 6+, RabbitMQ 3.8+, Amazon SQS

Security Compliance

This skill follows strict security rules:

• All code examples use placeholder values only

• No real API keys, passwords, or secrets

• Environment variable references in all code

• .gitignore protection documented