Prepare and test your skills
Prepare and test your skills
Message reliability and lifecycle in Pub/Sub (opens in a new tab) is the set of controls that decide whether a message is redelivered, discarded, or saved for later. A publisher sends a message to a topic without waiting for processing, and Pub/Sub leases each message to a subscriber client and tracks whether the message is acknowledged. Reliability depends on how the subscription handles acknowledgment, redelivery, retries, dead-lettering, ordering, and duplicate suppression.
Message delivery semantics in Pub/Sub define whether a message can be delivered more than once or lost during a failure. Pub/Sub provides at-least-once delivery by default, which means every message is delivered one or more times but duplicates are possible even after acknowledgment. The choice of semantic controls the trade-off between losing data and handling duplicates.
| Semantic | Delivery guarantee | Duplicate and loss behavior |
|---|---|---|
| At-least-once | Message is always delivered one or more times | Duplicates possible, loss avoided through redelivery |
| At-most-once | Message is delivered zero or one time | Duplicates prevented, loss possible if a failure occurs |
| Exactly-once | Message is processed one time only | Neither lost nor duplicated when correctly configured |
At-least-once delivery keeps data from being lost because unacknowledged messages are redelivered until the acknowledgment deadline expires or a retry limit is reached. At-most-once and exactly-once delivery reduce or remove duplicates, but they require careful acknowledgment handling and, for exactly-once, acceptance of higher latency.
The acknowledgment lifecycle in Pub/Sub controls when a message is discarded, retained, or redelivered. By default Pub/Sub discards a message from a subscription after the subscriber acknowledges it, so subscribers must process a message before acknowledging it or the message is lost on processing failure. Retained acknowledged messages are an exception because a subscription can retain acknowledged messages for up to 7 days and replay them with a seek operation.
A message moves through an ordered lifecycle between Pub/Sub and the subscriber:
Subscriber flow control protects this lifecycle during traffic spikes by limiting maximum outstanding messages and total outstanding message bytes. When either limit is crossed, the client stops pulling more messages until outstanding messages are acknowledged, which trades throughput against the cost of running more subscribers.
A retry policy in Pub/Sub determines how many delivery attempts Pub/Sub makes before treating a message as unprocessable. If a subscriber cannot process a message before the deadline, Pub/Sub redelivers the message and counts each delivery as an attempt. For pull subscriptions the lease can be extended with lease management, while for push subscriptions the acknowledgment deadline setting may need to be increased to give the endpoint more time.
A dead-letter topic is a separate topic that stores messages that repeatedly fail processing. After the configured number of delivery attempts is exhausted, Pub/Sub automatically forwards the message to the dead-letter topic for later inspection instead of redelivering it forever. This separation keeps a few bad messages from blocking healthy messages and preserves failed payloads for debugging, reprocessing, or manual recovery.
Message ordering in Pub/Sub preserves the publish order for messages that share the same ordering key. Ordering must be enabled on the subscription, while publishers attach the same ordering key to related messages and publish them in the same region. To keep messages in the correct region, publishers and subscribers use locational endpoints that route traffic to that region.
Ordered delivery changes acknowledgment behavior because acknowledgments for later messages are not processed until earlier messages with the same ordering key are acknowledged. For example, if messages 1, 2, and 3 share a key and only message 3 is acknowledged, Pub/Sub treats all three as unacknowledged until messages 1 and 2 are also acknowledged and then redelivers all three. Push subscriptions are not recommended for frequently used ordering keys because Pub/Sub allows only one outstanding message per ordering key at a time, so StreamingPull or Pull subscriptions provide better parallelism and lower latency.
Duplicate handling in Pub/Sub starts with idempotent subscribers that produce the same result when the same message is processed more than once. If subscribers cannot be made idempotent, Dataflow can be incorporated to deduplicate messages before processing. A high rate of duplicates often means subscribers are not acknowledging messages or the acknowledgment deadline is too short.
Common redelivery patterns have different causes and remedies:
Exam tip: Exactly-once delivery is available only for pull subscriptions and increases publish-to-subscribe latency, so choose it only when the application cannot tolerate any duplicates.
Orchestrating workflows with Cloud Tasks means managing asynchronous execution outside the user request path. An application creates a task and adds it to a queue, and Cloud Tasks delivers that task to a target handler at a controlled rate. This pattern keeps long-running work from blocking HTTP responses while protecting backends from overload and extra cost.
Cloud Tasks handles background work that should not run after an HTTP response is delivered. Background work is any work an app performs for a request after the HTTP response has been sent, and apps should finish all asynchronous operations before delivering the response. For long-running jobs, the application enqueues the work with Cloud Tasks, where HTTP requests are long-lived and return a response only after the asynchronous work ends.
Performing background work inline instead can increase billing because App Engine (opens in a new tab) may scale up additional instances due to high CPU load even with no active requests, and users may see higher latency while requests wait in the pending queue for available instances. Enqueuing the work moves that load to a task handler that runs at a controlled rate. The result is lower user-facing latency and fewer unnecessary instances.
Push queues (opens in a new tab) in App Engine are the queues that hold tasks until Cloud Tasks or the Task queue service delivers them to targets. App Engine provides a default push queue that is configured and ready to use, so an application can start by adding all tasks to the default queue without creating other queues. To add queues or change the default configuration, the team edits the queue configuration file and uploads it with gcloud app deploy queue.yaml.
Queue configuration controls routing, throughput, and lifecycle limits:
target, such as v2.task-module for queue-blue, so tasks flow from the queue to that target module.rate, such as 5/s or 1/s, so tasks are dequeued and sent to their targets at that rate.Deploying the queue configuration with a service account requires the serviceusage.services.list permission, granted through the serviceusage.serviceUsageViewer role or a custom role. For transactional push-queue tasks that fail to enqueue during a datastore commit, the application relies on an internal cron job that sweeps for missed tasks. A cron.yaml entry calls /_ah/cloudtask/sweep on a schedule such as every 30 minutes, so failed enqueues are eventually processed.
Rate control in push queues uses a token bucket for each named queue. The bucket holds tokens up to bucket_size, or up to 5 tokens when no bucket size is specified, and App Engine refills the bucket continuously based on the queue rate. The rate, bucket_size, and max_concurrent_requests directives together control how quickly tasks flow from the queue to the target handlers.
Task execution follows an ordered cycle:
rate, which allows more tasks to run.If the queue holds tasks and the bucket holds tokens, App Engine processes as many tasks simultaneously as there are tokens. That burst can consume system resources, compete with user-serving requests, and cause datastore contention. Use max_concurrent_requests when the workflow must limit how many tasks run at once or protect a shared datastore.
Request priority in App Engine determines how Cloud Tasks traffic waits during heavy traffic. During heavy traffic, App Engine may place requests in a pending queue while waiting for an available instance. Legacy Task queue requests sit at lower priority than other queued requests, and App Engine Cloud Tasks requests share that same lower priority for compatibility reasons. Requests from HTTP-target Cloud Tasks are treated as regular HTTP traffic and do not sit at that lower priority.
The difference affects latency when user traffic and task traffic share the same service:
| Task traffic type | Pending-queue priority | Design implication |
|---|---|---|
| Legacy Task queue | Lower than other queued requests | Task traffic can wait longer when user traffic is high |
| App Engine Cloud Tasks | Same lower priority as Task queue | Same queuing delay risk as legacy Task queue traffic |
| HTTP-target Cloud Tasks | Same as regular HTTP traffic | Avoids the low-priority queuing penalty |
When a service receives high-volume standard HTTP traffic plus much lower-volume Task queue or Cloud Tasks traffic, the task traffic sees a disproportionate latency impact. To avoid priority queuing, split the traffic types to separate versions or use HTTP-target tasks. For latency-sensitive work served from Cloud Tasks, use a dedicated major version or service.
Rate limiting protects task workflows and backends from rapid or sustained load increases. The surge may come from malicious requests such as network-based distributed denial-of-service (DDoS) attacks or from configuration errors in client software. Without rate limiting, the application or the resources it uses can fail and become unavailable. Suitable controls include quotas on the number of API requests a Google Cloud service can receive and project-level quotas on specific resources, which also limit the billing a project can incur for those resources.
Graceful degradation keeps the system available under high load by reducing performance or accuracy instead of failing completely, then resuming full functionality when load returns to a manageable level. For task-driven systems, the key protections are independent throttling at each replica to prevent cascading failures when excess traffic shifts between zones, and early dropping of excess requests at the frontend layer to protect backend components. Tools such as Apigee can control API request rates during high-traffic periods through policy rules that define how requests scale back. Dropping some frontend requests may cause errors for some users, but it prevents global failure and allows more graceful recovery than circuit-breaking, where all traffic is dropped during overload.
Exam tip: App Engine Cloud Tasks share the lower pending-queue priority of legacy Task queue traffic, while HTTP-target Cloud Tasks are treated as regular HTTP traffic.
Pub/Sub messaging patterns are the ways publishers, topics, subscriptions, and subscribers connect to move data asynchronously. Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that decouples services that produce messages from services that process those messages. A publisher application sends events to a topic without waiting for processing, and Pub/Sub delivers those events to subscriber applications attached through subscriptions, typically with latency around 100 milliseconds.
A Pub/Sub data flow moves data from publishers to a topic and then from the topic to one or more subscriptions that feed subscribers. The publisher creates a message containing data, attributes, or an ordering key and sends a publish request to the Pub/Sub server for a named topic. Each subscription attached to that topic receives its own copy of each message, so publishers and subscribers remain independent and can scale separately. Pub/Sub provides at-least-once delivery with best-effort ordering, and it uses per-message parallelism by leasing individual messages to subscriber clients and tracking whether each message is processed successfully.
A Pub/Sub subscription defines how messages travel from Pub/Sub to subscriber code, using push, pull, or export delivery. In push delivery, Pub/Sub sends each message as an HTTP POST request to a webhook endpoint such as Cloud Functions, so the subscriber does not need to call Pub/Sub. In pull delivery, the subscriber client uses a client library to request messages from Pub/Sub, by default over a StreamingPull connection that keeps messages flowing. An export subscription stores messages directly to a Google Cloud resource such as a BigQuery table, without requiring Dataflow as an intermediary.
Push subscriptions fit specific constraints where pull is difficult to use:
For general cases, the high-level client library with pull is recommended because it handles authentication, throughput optimization, and message formatting with minimal operational work. This choice reduces code the team must maintain while keeping throughput high.
Publishing messages with client libraries is how a publisher client creates messages and sends them to a Pub/Sub topic. Each published message must contain message data, attributes with additional metadata, or an ordering key, and the Pub/Sub service adds a message ID unique to the topic plus a receive timestamp. Messages are limited to 10 MB, so larger payloads are handled by storing the object in Cloud Storage and publishing a message that contains the URL, which the subscriber then downloads. A topic without an attached subscription or topic retention discards published messages, so a subscription must be attached or retention enabled before publishing begins.
Publisher applications can be configured with four main options:
The high-level client library publishes asynchronously and batches multiple messages into one publish request based on request size, message count, and time, which improves throughput and lowers cost but adds publish latency. Publisher-side flow control prevents overload during traffic spikes by limiting outstanding requests, and the publish call returns a future that should be handled asynchronously with retries or persistence on failure.
Consuming messages and the acknowledgment lifecycle describes how a subscriber receives messages and confirms processing within its acknowledgment deadline, known as ackDeadline. If the deadline passes without an acknowledgment, the message stops being outstanding and becomes available for redelivery, and it continues to be redelivered until the message retention period expires. Subscriber flow control limits how many unacknowledged messages the client holds by setting maximum outstanding messages and total outstanding message bytes, which prevents overload and lets autoscaling catch up or spreads work over time.
A message in a subscription moves through three lifecycle states:
Exam tip: Process a message before acknowledging it, because Pub/Sub discards an acknowledged message from the subscription and will not redeliver it unless acknowledged-message retention with a seek operation applies.
A Pub/Sub publish-subscribe pattern describes how many publishers, topics, subscriptions, and subscribers connect to share work. The same topic can feed one subscription for shared processing or multiple subscriptions when each downstream system needs every message. Multiple subscribers can also share one subscription to divide the work and increase throughput.
| Pattern | Publisher to topic to subscription shape | How messages are distributed |
|---|---|---|
| Fan-in | Many publishers to one topic to one subscription to one subscriber | The single subscriber receives all published messages from the topic |
| Load-balanced | One or many publishers to one topic to one subscription to many subscribers | Each subscriber receives a different subset of messages, so adding subscribers increases processing scale |
| Fan-out | One or many publishers to one topic to many subscriptions, each with its own subscriber | Every subscription receives the same set of messages, which supports different data operations on identical input |
Fan-out is the right choice when different applications must independently process the same events, while load-balanced sharing of one subscription is the right choice when the goal is parallel task processing such as compressing files, sending notifications, or reformatting images. The shape of the topic-to-subscription connection therefore decides whether work is duplicated or divided.
Google Cloud Pub/Sub provides at-least-once delivery by default, meaning every message is delivered one or more times with duplicates possible, while at-most-once delivers zero or one time with possible loss and exactly-once processes one time only when correctly configured. At-least-once avoids loss by redelivering unacknowledged messages until the acknowledgment deadline expires or a retry limit is reached, while exactly-once is available only for pull subscriptions and increases publish-to-subscribe latency.
In Google Cloud Pub/Sub, Pub/Sub leases an individual message to a subscriber client and starts the acknowledgment deadline, the subscriber processes it and can extend the lease, then sends an acknowledgment on success or a negative acknowledgment for immediate redelivery. Pub/Sub discards the message after acknowledgment and redelivers it when the deadline expires without acknowledgment, so subscribers must process a message before acknowledging it, except that a subscription can retain acknowledged messages for up to 7 days and replay them with a seek operation.
A dead-letter topic in Google Cloud Pub/Sub is a separate topic that stores messages that repeatedly fail processing after the configured number of delivery attempts is exhausted. Pub/Sub automatically forwards the exhausted message there for later inspection instead of redelivering it forever, which keeps a few bad messages from blocking healthy messages and preserves failed payloads for debugging, reprocessing, or manual recovery.
Message ordering in Google Cloud Pub/Sub preserves publish order for messages that share the same ordering key when ordering is enabled on the subscription. Publishers attach the same ordering key to related messages and publish them in the same region using locational endpoints, and acknowledgments for later messages are not processed until earlier messages with the same key are acknowledged, so if messages 1, 2 and 3 share a key and only 3 is acknowledged, all three are treated as unacknowledged and redelivered.
Last updated