Professional Cloud Developer

Architecting Messaging Solutions with Cloud Pub/Sub

Cloud Pub/Sub is an asynchronous messaging service that lets different parts of an application communicate without being directly connected. This is called a decoupled architecture, which makes systems more scalable and reliable. For example, one service can fail or restart without messages being lost for other services.

A key decision is choosing a subscription model. A push subscription sends messages directly to a web endpoint, which is good for triggering serverless functions. A pull subscription requires the receiving service to ask for messages, giving it more control over how fast it processes high-volume traffic. To prevent bad messages from blocking the system, you can use a dead-letter topic to capture messages that repeatedly fail delivery. Other important features include message ordering to keep events in the right sequence and exactly-once delivery to avoid processing the same data twice.

To handle sudden surges in traffic, you can use flow control. This lets you set limits, like the maximum number of unacknowledged messages a subscriber can hold at once. This protects your services from being overwhelmed. You can also integrate Pub/Sub with other services. For instance, an export subscription can send data directly to BigQuery for analysis, and you can replay past messages to recover from failures or test new features with real data.

Ensuring Reliability and Idempotency in Asynchronous Systems

Building reliable asynchronous systems means designing them to handle failures gracefully. A core principle is idempotent processing, which ensures that if the same action (like processing a message) happens more than once, the end result is the same as if it happened only once. This is crucial when retries occur.

When a failure happens, a good strategy is Exponential Backoff for retries. This means waiting a short time after a failure, then increasing the wait time for each subsequent attempt. Adding a random element to the delay helps prevent many retries from happening at the same instant, which is important during traffic spikes. You must also be aware of system limits, like the 10MB size limit for integration variables, as exceeding them can disable logging and make troubleshooting harder.

Data consistency is another important consideration. In services like Datastore, you can choose between strong consistency (getting the latest data every time) and eventual consistency (faster reads, but data might be slightly older). For managing resources, using a connection pool helps efficiently handle database connections by reusing them and setting a maximum lifetime to avoid unexpected closures. To accurately test performance, you should run benchmarks for at least 20 minutes to see how the system behaves under a steady load.

Orchestrating Event-Driven Workflows with Eventarc

Eventarc is a service that routes events from various sources to specific destinations, enabling event-driven architectures. This creates loosely connected applications that can react to changes in real time. You set up this routing by creating an Eventarc trigger.

There are three main types of triggers based on the event source: events from Google Cloud services (like Cloud Storage), events from your own custom applications, and events from external third-party SaaS providers. Each trigger must specify exactly what kind of event it is listening for.

Choosing the right destination for the event is important for performance. Common destinations include Cloud Run services, GKE clusters, and Workflows. To keep things fast, it's best to place the trigger, the event source, and the destination all in the same Google Cloud region. Security is managed through service accounts and IAM roles. The trigger needs a service account with the Eventarc Event Receiver role to get events, and if the destination is private, that same account needs the Invoker role to run the code.

You can configure triggers visually in the Application Integration editor. After you set the rules, you must publish the integration to activate the event flow. You can also enable Retry on failure so Eventarc will try again if the destination is temporarily unavailable, ensuring events are not lost.