Event-driven systems are often explained as a way to decouple services, but the real magic is how they let you replay history to debug, recover, or even build entirely new features without touching the original producer.
Let’s see this in action. Imagine a simple e-commerce order system. When an order is placed, we want to notify several downstream services: inventory, shipping, and notifications.
First, we set up a Pub/Sub topic. This is the central hub for our events.
gcloud pubsub topics create order-events
Now, our order service, when it successfully creates an order, publishes a message to this topic. The message payload is just a JSON string containing the order details.
from google.cloud import pubsub_v1
import json
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path('your-gcp-project-id', 'order-events')
def publish_order_event(order_data):
message_data = json.dumps(order_data).encode('utf-8')
future = publisher.publish(topic_path, message_data)
print(f"Published message ID: {future.result()}")
# Example usage:
order_details = {
"order_id": "ORD12345",
"customer_id": "CUST987",
"items": [{"product_id": "PROD001", "quantity": 2}],
"timestamp": "2023-10-27T10:00:00Z"
}
publish_order_event(order_details)
Next, we create subscriptions for each service that needs to react to these order events. These subscriptions act like mailboxes, holding messages until a subscriber can process them.
gcloud pubsub subscriptions create inventory-sub --topic order-events
gcloud pubsub subscriptions create shipping-sub --topic order-events
gcloud pubsub subscriptions create notification-sub --topic order-events
Each of these downstream services will have a subscriber client that pulls messages from its dedicated subscription.
from google.cloud import pubsub_v1
import json
import time
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path('your-gcp-project-id', 'inventory-sub')
def callback(message):
print(f"Received order event: {message.data.decode('utf-8')}")
# In a real scenario, you'd process the order here (e.g., update inventory)
message.ack() # Acknowledge the message so it's removed from the subscription
streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}")
# Keep the main thread alive to process messages
try:
streaming_pull_future.result()
except KeyboardInterrupt:
streaming_pull_future.cancel()
streaming_pull_future.join()
This setup decouples the order service from the inventory, shipping, and notification services. If the notification service is temporarily down, orders can still be placed, and their events will be held in the notification-sub subscription until the service recovers. The order service doesn’t need to know or care if any downstream services are available.
The true power emerges when you consider Pub/Sub’s durable nature and replay capabilities. By default, Pub/Sub retains messages for 7 days. This means if your inventory-sub goes offline for an hour, it will simply pick up where it left off, processing all the messages that arrived while it was unavailable, without the order service having to re-publish anything.
But you can extend this retention. For order-events topic, you can set message retention to up to 7 days.
gcloud pubsub topics update order-events --message-retention-duration=7d
This allows for more extended downtimes or, more powerfully, for rebuilding systems. Suppose you want to add a new fraud-detection service. You don’t need to reprocess all historical orders from your database. You can simply create a new subscription to the order-events topic, perhaps with a longer retention period if needed, and start processing events from the moment the subscription is created. If you need to go back further, you can configure the topic with a longer retention (up to 7 days) and then create a subscription and backfill.
Pub/Sub’s internal architecture, using a distributed log-like structure, ensures that messages are durably stored and can be delivered to multiple subscribers independently. Each subscriber maintains its own cursor into this log. When a subscriber acknowledges a message, it’s simply advancing its cursor. This independence is key to the replayability. You can even create a subscription after an event has occurred, and as long as the topic’s message retention period hasn’t expired, that new subscription will receive the message.
The most counterintuitive aspect of Pub/Sub is realizing that a "message" isn’t just sent and forgotten; it’s a durable record that can be read by any number of entities, at any time, as long as it’s within the retention window, and each reader can track its own progress independently.
The next evolution in event-driven architectures often involves managing the schema of these events to ensure compatibility between producers and consumers over time.