Expert-Level GCP Pub/Sub Setup for Dependable Messaging

resource "google_pubsub_topic" "orders" {
  project = var.project_id
  name = "orders"
  message_retention_duration = "604800s"
  schema_settings {
    schema = google_pubsub_schema.order.id
    encoding = "JSON"
  }
}

resource "google_pubsub_schema" "order" {
  project = var.project_id
  name = "order-schema"
  type = "AVRO"
  definition = file("schemas/order.avsc")
}

resource "google_pubsub_subscription" "orders_processor" {
  project = var.project_id
  name = "orders-processor"
  topic = google_pubsub_topic.orders.id
  ack_deadline_seconds = 60
  message_retention_duration = "604800s"
  retain_acked_messages = false
  expiration_policy {
    ttl = ""
  }
  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "600s"
  }
  dead_letter_policy {
    dead_letter_topic = google_pubsub_topic.orders_dlq.id
    max_delivery_attempts = 5
  }
}

resource "google_pubsub_topic" "orders_dlq" {
  project = var.project_id
  name = "orders-dlq"
  message_retention_duration = "2592000s"
}

Dead Letter Policy is critical. Without it, a "poison" message (malformed payload that always crashes the handler) will be retried indefinitely, blocking other messages. DLQ intercepts it after 5 attempts—you get an alert and can fix the payload.

Processing Messages with Python

from google.cloud import pubsub_v1
import json

def process_message(message: pubsub_v1.subscriber.message.Message):
    try:
        data = json.loads(message.data.decode("utf-8"))
        order_id = data.get("order_id")
        if not is_already_processed(order_id):
            process_order(data)
            mark_as_processed(order_id)
        message.ack()
    except Exception as e:
        print(f"Error: {e}")
        message.nack()

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(PROJECT_ID, "orders-processor")
flow_control = pubsub_v1.types.FlowControl(max_messages=100)
streaming_pull_future = subscriber.subscribe(
    subscription_path,
    callback=process_message,
    flow_control=flow_control,
)

FlowControl(max_messages=100) limits the number of concurrently processed messages. Without it, a fast producer can exhaust the consumer's memory.

Security: IAM and Service Accounts

gcloud pubsub topics add-iam-policy-binding orders \
  --member=serviceAccount:[email protected] \
  --role=roles/pubsub.publisher

gcloud pubsub subscriptions add-iam-policy-binding orders-processor \
  --member=serviceAccount:[email protected] \
  --role=roles/pubsub.subscriber

Monitoring and Alerting

Key metrics in Cloud Monitoring:

• pubsub.googleapis.com/subscription/num_undelivered_messages — growing? Consumer is lagging.
• pubsub.googleapis.com/subscription/oldest_unacked_message_age — if > ack_deadline, messages are being redelivered.
• pubsub.googleapis.com/subscription/dead_letter_message_count — alert on non-zero value.

At a load of 200K messages/sec, num_undelivered_messages should be near zero. Proper alert configuration reduces MTTR by 60% (according to Google Cloud Pub/Sub documentation).

Comparison: Pub/Sub vs Cloud Tasks vs RabbitMQ

Pub/Sub wins in speed and managed convenience; Cloud Tasks when deduplication and ordering are needed; RabbitMQ for self-managed solutions under strict control. For example, Pub/Sub offers 10x the throughput of RabbitMQ at scale.

Implementation Process and Timeline

We work in stages:

1. Analyze architecture and event model requirements.
2. Design topics, subscriptions, schemas.
3. Implement Terraform modules with parameters (ack deadline, retry, DLQ).
4. Integrate with consumers (pull/push).
5. Set up monitoring and alerts.
6. Document runbooks and train the team.

What's Included in the Work

When you order turnkey Pub/Sub configuration, you get:

• Topics and subscriptions with correct parameters (ack deadline, retry, DLQ).
• Schema validation to protect against invalid data.
• IAM roles and service accounts.
• Terraform code (modules) in your repository.
• Monitoring dashboards in Cloud Monitoring and alerts.
• Architecture documentation and runbook.
• Team training on idempotency and error handling.
• Launch support (1 week).

Timeline: topology with one topic and subscriptions — 1 day. Complex event-driven schemes with multiple topics, schema validation, and push integration with Cloud Run/Functions — 2–3 days. Pricing is calculated individually. Typical project cost: $2,000–$5,000, with ongoing savings of $5,000/month in infrastructure.

Ready for a reliable event-driven architecture? Contact us for a preliminary assessment of your project. Order turnkey Pub/Sub configuration—we guarantee stable operation under any load.

We recommend reading the official Google Pub/Sub documentation for further details.