Google Cloud Pub/Sub Core Knowledge

Full Reference: See advanced.md for Java/Python/Go producer patterns, Spring Cloud GCP consumer, push subscription handlers, and Terraform for DLT, IAM, VPC Service Controls, and monitoring alerts.

Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.

Quick Start (Emulator)

docker-compose.yml

services: pubsub: image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085 ports: - "8085:8085"

docker-compose up -d

Set emulator env

export PUBSUB_EMULATOR_HOST=localhost:8085

Create topic and subscription

gcloud pubsub topics create orders gcloud pubsub subscriptions create order-processor --topic=orders

Test

gcloud pubsub topics publish orders --message='{"id":"123"}' gcloud pubsub subscriptions pull order-processor --auto-ack

Core Concepts

Concept Description

Topic Named resource for publishing messages

Subscription Named resource for receiving messages

Publisher Sends messages to topic

Subscriber Receives messages from subscription

Ack Deadline Time to acknowledge before redelivery

Message Retention How long unacked messages are kept

Delivery Types

Type Description Use Case

Pull Subscriber requests messages Batch processing, variable load

Push Pub/Sub sends to endpoint Serverless, webhooks

BigQuery Direct export to BigQuery Analytics pipelines

Cloud Storage Direct export to GCS Data archival

Architecture

Publisher ──▶ Topic ──▶ Subscription 1 (Pull) ──▶ Subscriber ──▶ Subscription 2 (Push) ──▶ Cloud Run ──▶ Subscription 3 ──▶ BigQuery

Node.js Producer (@google-cloud/pubsub)

import { PubSub } from '@google-cloud/pubsub';

const pubsub = new PubSub({ projectId: 'my-project', });

const topic = pubsub.topic('orders');

// Publish single message const messageId = await topic.publishMessage({ data: Buffer.from(JSON.stringify(order)), attributes: { 'correlation-id': correlationId, 'order-type': order.type, }, orderingKey: order.customerId, // For ordered delivery });

// Batch publishing (automatic batching) const publishOptions = { batching: { maxMessages: 100, maxMilliseconds: 10, }, }; const batchTopic = pubsub.topic('orders', publishOptions);

Node.js Consumer (Pull Subscription)

const subscription = pubsub.subscription('order-processor', { flowControl: { maxMessages: 100, maxExtensionMinutes: 10, }, ackDeadline: 30, });

const messageHandler = async (message) => { try { const order = JSON.parse(message.data.toString()); const correlationId = message.attributes['correlation-id'];

await processOrder(order);
message.ack();

} catch (error) { console.error('Processing failed:', error); message.nack(); // Will be redelivered } };

subscription.on('message', messageHandler); subscription.on('error', (error) => console.error('Subscription error:', error));

// Graceful shutdown process.on('SIGTERM', async () => { await subscription.close(); });

Dead Letter Topics

Create DLT

gcloud pubsub topics create orders-dlq gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq

Create subscription with DLT

gcloud pubsub subscriptions create order-processor
--topic=orders
--dead-letter-topic=orders-dlq
--max-delivery-attempts=5

When NOT to Use This Skill

Use alternative messaging solutions when:

• AWS-native architecture - SQS has better AWS integration

• Azure-native architecture - Use Azure Service Bus

• Event streaming with replay - Use Dataflow or Kafka

• On-premise deployment - Use RabbitMQ or ActiveMQ

• Multi-cloud portability - Use Kafka or RabbitMQ

• Complex routing patterns - RabbitMQ provides more flexibility

• JMS compliance required - Use ActiveMQ

Anti-Patterns

Anti-Pattern Why It's Bad Solution

No dead letter topic Failed messages lost Configure DLT for all subscriptions

Short ack deadline Duplicate processing Set deadline > max processing time

No retry policy Immediate redelivery on failure Configure exponential backoff

Synchronous publish Poor throughput Use batching and async publish

Pull without flow control Consumer overwhelmed Set max_messages limit

No message ordering when needed Out of order processing Use ordering keys

Large message payloads Higher costs, poor performance Use Cloud Storage with reference

No IAM least privilege Security risk Use service accounts with minimal roles

Quick Troubleshooting

Issue Likely Cause Fix

Messages not received No subscription or wrong topic Create subscription, verify topic

Duplicate messages Ack deadline expired Increase ack deadline or process faster

Messages in DLT Max delivery attempts exceeded Check processing logic, review DLT

Permission denied Missing IAM roles Grant Publisher/Subscriber roles

Ordering not working No ordering key or wrong subscription Set ordering key, enable message ordering

High latency Batching delay or network Reduce batch delay, check network

Push subscription failing Endpoint down or auth failure Check endpoint health, verify auth

Backlog growing Slow consumers Add consumers or optimize processing

Production Readiness

Monitoring Metrics

Metric Alert Threshold

num_undelivered_messages

10000

oldest_unacked_message_age

3600s

num_outstanding_messages

10000

dead_letter_message_count

0

publish_latencies p99 > 1s

Checklist

• IAM roles with least privilege

• Service account per component

• Dead letter topic configured

• Retry policy configured

• Ack deadline appropriate

• Message retention set

• Monitoring alerts configured

• Schema validation (if needed)

• Message ordering (if needed)

• VPC Service Controls (if needed)

Reference Documentation

Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.

Available topics: basics , producers , consumers , production