Connecting Apps to Cloud SQL, Firestore and Pub/Sub

Secure and performant datastore connections match each workload to the right storage service and protect access with least privilege. Traffic flows from the application to the datastore through an authenticated and encrypted path, while caching and replication reduce latency and protect availability. In Google Cloud, Cloud SQL (opens in a new tab) provides managed relational databases, Firestore provides NoSQL document storage, Cloud Storage provides object storage, Spanner provides globally distributed relational storage, and Memorystore for Redis provides managed in-memory caching.

1.1Secure Access to Cloud Storage

Cloud Storage is low-cost object storage that holds any amount of data as buckets and objects accessible from inside and outside Google Cloud. Access to buckets and objects is controlled by Identity and Access Management (IAM) and access control lists (ACLs), and IAM is recommended in most cases because it grants permissions at the bucket and project levels. Granular control uses IAM roles assigned to a user, group, or service account, IAM policies with conditions, and principal access boundary policies that limit what a principal can reach, including managed folders. Data is encrypted by default with Google-owned and Google-managed keys, with customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) or customer-supplied encryption keys (CSEKs) for more control, while Sensitive Data Protection can discover, classify, and de-identify sensitive data loaded through buckets.

Data residency is controlled by bucket location because data is stored or replicated only within the specified regions. Regional buckets replicate data synchronously across zones in one region, while dual-region and multi-region buckets replicate asynchronously across regions for higher availability.

1.2Choosing Relational, Non-Relational, and Globally Distributed Databases

Database choice constrains consistency, availability, and scalability, so different application services often use different database types. Relational databases provide transactions, strong consistency, referential integrity, and sophisticated queries across tables, but they are typically hard to scale and need careful high-availability management. Non-relational (NoSQL) databases typically sacrifice some relational features for higher availability and easier scalability, often choosing availability over consistency.

Spanner is a strongly consistent, highly available, and globally distributed database with SQL support that combines relational querying with global scale:

Choose a relational database when transactions and strong consistency are required, choose NoSQL when the model can be eventually consistent and needs scale, and use both strategically when services have different needs.

1.3Caching, Replication, and Resilient Connections

Memorystore for Redis is a fully managed service powered by the Redis in-memory datastore that provides low-latency access and high throughput for heavily accessed data. The cache sits between the application and slower disk-based storage, so requests flow from the application to the cache first and reach the downstream database only on a cache miss. This placement reduces storage latency, reduces load on downstream databases, and helps the application and other database consumers scale further. Memorystore for Redis can use a high-availability configuration with cross-zone replication and automatic failover, and cached data can continue to serve traffic during overload or outage of the storage layer even if results are incomplete or stale.

High availability also depends on where data is replicated. AlloyDB for PostgreSQL clusters provide high availability by default with automatic failover, where the primary instance uses redundant nodes in two zones in a region, plus cross-region replication for recovery from regional outages. BigQuery datasets store data synchronously in two zones in the specified region. Place compute and data in the same region when latency matters, and choose dual-region, multi-region, or cross-region replication when the workload must survive a regional outage.

Resilient connections keep datastore links reliable by limiting traffic and handling errors gracefully:

• Circuit breaker: Stops sending traffic to an overloaded or failing service to give it time to recover.
• Exponential backoffs: Retries failed requests with increasing delays to avoid overwhelming the service.
• Graceful degradation: Returns limited or cached results when the full datastore response is unavailable.

A service mesh links distributed services and applies retries, failovers, and circuit breakers as traffic flows from the application through the mesh to each downstream service. For message ingestion, Pub/Sub publisher flow control limits publish rates during transient spikes so publishers do not overwhelm subscribers and downstream datastores.

1.4Secure Connectivity and Authentication for Relational Databases

Secure connectivity for relational databases means applications reach Cloud SQL and AlloyDB for PostgreSQL through verified identities and encrypted paths instead of open addresses and static passwords. Traffic flows from the application to a local proxy or connector, and from that proxy to the database instance over an authorized encrypted tunnel. The Cloud SQL Auth Proxy is a client-side proxy that must authenticate as a Cloud SQL IAM identity, and a dedicated service account is recommended for production because its credential file can be created specifically for the proxy and duplicated into system images for multiple machines. Creating that service account requires the resourcemanager.projects.setIamPolicy permission.

IAM permissions determine whether the proxy identity can open connections. The proxy service account must have a role that includes cloudsql.instances.connect, using one of these predefined roles:

• Cloud SQL Client, which provides connectivity permission for applications
• Cloud SQL Editor, which adds broader management permissions
• Cloud SQL Admin, which provides full administrative permissions

If legacy project roles are used, the service account must have at least the Editor role. The proxy must run continuously with the same uptime as the application, because if the proxy stops all existing connections drop and no new connections can be created. The pattern is to run one proxy process per workstation or virtual machine under systemd, upstart, supervisor, or as a Windows Service with a wrapper, and to include the proxy in the virtual machine image for auto-scaling. The proxy depends on the Cloud SQL Admin API and issues 2 API calls per hour per connected instance while running, with highest usage at startup, and version 1.12 or later is required for private IP.

Private paths and encryption protect data in transit. AlloyDB supports TLS 1.3 with an automatically generated server certificate, while AlloyDB Language Connectors and the AlloyDB Auth Proxy use IAM for authorization and mutual TLS (mTLS) with client certificates for protection. The Cloud SQL Auth Proxy transparently upgrades non-TLS connections to encrypted connections. For Cloud SQL, SSL enforcement uses ALLOW_UNENCRYPTED_AND_ENCRYPTED, the default that allows both types, and ENCRYPTED_ONLY, which allows only SSL/TLS connections, set through the console Connections and Security tab, gcloud, or Terraform ssl_mode. AlloyDB private connectivity keeps traffic on private networks through the AlloyDB Auth Proxy or Language Connector secure TCP tunnel, Serverless VPC Access for Cloud Run, and Private Service Connect or private services access for another VPC, with Private Service Connect configured at cluster creation when needed. Additional controls include VPC Service Controls perimeters, Shared VPC centralized networking, and Cloud VPN or Cloud Interconnect links from corporate networks.

Database authentication determines whether clients use stored passwords or Google identities:

1.5Programmatic Access and Identity for Firestore and Cloud Storage

Programmatic access for cloud services is how an application connects to Firestore in Datastore mode and Cloud Storage with the correct identity and least privilege. Day-to-day data access uses Datastore mode client libraries and the Datastore Data API over REST and RPC with GQL for queries, while managed export and import is available through the Google Cloud console, the Google Cloud CLI, and the Datastore Admin API over REST and RPC. Export output flows from Firestore in Datastore mode to a Cloud Storage bucket in LevelDB log format, and import reverses that path from bucket objects back into the database. Exports use eventually consistent reads, do not contain indexes, may include entities written after export start while missing some written before it, and imports reuse exported entity IDs, overwrite matching entities, reserve those IDs during import, and leave unaffected entities in place.

Export and import authorization is split between two identities on different sides of the trust boundary:

The Firestore service agent uses the name service-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com and performs the bucket work on behalf of Firestore, so the caller never writes bucket files directly during a managed transfer. Older databases may still use the App Engine default service account, and migration to the service-specific agent is recommended, especially when VPC Service Controls must protect transfers. Other callers should receive Datastore Import Export Admin, Datastore Owner, or a custom role with only the needed export or import permission.

Reliable setup follows an ordered sequence where each step depends on the previous one:

1. Enable billing: Enable billing on the project because managed export and import require it.
2. Create the bucket: Create the Cloud Storage bucket in the same location as the database; Requester Pays and Rapid buckets cannot be used.
3. Grant caller IAM: Assign export permission, import permission, or both to the account that starts the operation.
4. Grant service-agent access: Grant the Firestore service agent access to the bucket before starting, including cross-project buckets.

Operational limits allow 50 concurrent exports and imports per project, 20 requests per minute, and 100 entity-filter combinations per request. Export metadata must keep the path gs://BUCKET_NAME/OPTIONAL_NAMESPACE_PATH/PARENT_FOLDER_NAME/PARENT_FOLDER_NAME.overall_export_metadata, and moving output must preserve folder names and metadata filenames. Protection combines automatic AES-256 server-side encryption, usually in Galois/Counter Mode (GCM), with Transport Layer Security (TLS) in transit and optional client-side encryption that encrypts data twice, where the application must manage client-side keys because Firestore cannot detect that encryption.

Optimizing data access performance and security means choosing transfer paths, locations, and storage classes that lower latency and cost while allowing only trusted identities to reach data. Performance depends on the connection path, bucket location, and storage class, while security depends on Identity and Access Management (IAM), service accounts, and network perimeters. The handover pattern connects private environments to Google Cloud projects through shared storage protected by those controls.

2.1Handover Pattern for Reading and Writing Data

The handover pattern is an architecture where workloads in a private computing environment or another cloud upload data to shared storage for Google Cloud workloads to consume. Data flows from the private environment to Cloud Storage (opens in a new tab) buckets or Pub/Sub topics, and then from that shared storage to workloads in an application Virtual Private Cloud (VPC) for processing, analytics, and frontend use. Frontend applications in the VPC are exposed through Cloud Load Balancing or API Gateway, so virtual machine (VM) instances are not directly reachable from the internet. Communication with Cloud Storage or Pub/Sub travels over public networks or over private connectivity with VPN, Cloud Interconnect, or Cross-Cloud Interconnect.

The connection choice depends on expected traffic volume, whether the setup is temporary or permanent, and security and compliance requirements. Private connectivity is preferred for sustained or sensitive transfers to reduce latency and avoid high volumes over the public internet, while public paths may fit temporary or low-volume use. The gated ingress variation uses Private Service Connect endpoints for Google APIs, which provide private IP addressing to Cloud Storage, BigQuery, and other Google service APIs over VPN or Interconnect. When evaluating transfers, assess cost, expected transfer time, and security to balance speed against expense and compliance.

2.2Optimizing Performance, Location, and Transfer Cost

Performance optimization for data access starts with selecting the transfer path and storage location that match latency, availability, and cost goals. Cloud Interconnect or Cross-Cloud Interconnect reduces latency for large transfers and for access to Private Service Connect endpoints inside the VPC. For the ingestion bucket, choose a storage class based on retention and access frequency so frequently read data stays performant and infrequently read data stays low-cost.

Cloud Storage bucket location types control replication for resilience and speed:

For block and shared-file needs, Google Cloud Hyperdisk allows separate and dynamic provisioning of IOPS and throughput with Hyperdisk Balanced, while Filestore regional instances replicate data synchronously across three zones for configuration, tools, and logs mounted on multiple VMs. Regional Persistent Disk volumes provide synchronous replication across two zones in a region for durable VM-attached storage.

2.3Enforcing Fine-Grained Access with IAM and Service Accounts

Fine-grained access control is enforced with IAM roles that grant permissions at the bucket, project, folder, or organization level. Cloud Storage supports IAM and access control lists (ACLs), and IAM is recommended in most cases because it centralizes access across buckets and projects. IAM policies can restrict Cloud Storage buckets and Pub/Sub topics to trusted workloads, and Cloud SQL IAM database authentication lets administrators configure the instance and add IAM users as database users so login is monitored through one interface.

Applications authenticate with service accounts rather than shared user credentials, and privileged access is managed by placing users in Google Groups and granting roles to the group. Group-based access stays correct during join, move, and leave events because membership changes update access without rewriting policies. Least privilege is maintained with the IAM recommender, which uses machine learning to recommend removal of unneeded roles and to identify unused service accounts and permissions. Time-bound access uses conditional role bindings and group membership expiration, while organization policy constraints limit where resources can be created to prevent storing data in the wrong region.

2.4Preventing Exfiltration and Protecting Sensitive Data

Data protection for reading and writing combines network perimeters, residency controls, and encryption to keep data inside selected boundaries. VPC Service Controls restricts access and reduces exfiltration risk by creating a service perimeter at the project or VPC network level that defines the virtual boundaries from which a service can be accessed. The perimeter can extend to a hybrid environment over an authorized VPN or Cloud Interconnect, and it can restrict Cloud SQL import and export through the Cloud SQL Admin API or Cloud Storage API so data stays in chosen network locations.

Message and object residency is controlled with Pub/Sub message storage policies and by storing or replicating Cloud Storage data within the specified region. Encryption is automatic for customer content at rest and in transit, with Pub/Sub encrypting messages by default using Google-owned and Google-managed keys. For more control, customer-managed encryption keys (CMEK) add control where Cloud SQL uses customer keys on API calls and Pub/Sub supports CMEK for message encryption. Keys can live in Cloud Key Management Service (Cloud KMS) in a chosen location, in Cloud HSM inside a hardware security module in a chosen region, or outside Google infrastructure with Cloud External Key Manager (Cloud EKM). Data loaded through Cloud Storage that may contain sensitive fields can be discovered, classified, and de-identified with Sensitive Data Protection before further processing.

2.5Managing Data Operations in Relational and NoSQL Databases

Managing data operations in relational and NoSQL databases means choosing the right access pattern for each workload and keeping reads, writes, and recovery consistent and scalable. Cloud SQL is a fully managed relational service offering MySQL, PostgreSQL, and SQL Server engines for online transaction processing (OLTP) workloads. Cloud SQL Enterprise Plus edition adds a 99.99% availability SLA with near-zero downtime maintenance, optimized hardware and software, intelligent data caching for read-intensive workloads, 35 days of log retention, and advanced disaster recovery for MySQL with orchestrated failover and switchback and zero data loss after cross-regional failover.

Spanner is a globally distributed relational database that uses GoogleSQL, an ANSI-compliant dialect formerly called Google Standard SQL. GoogleSQL in Spanner divides operations into statement types that flow from the application through the Spanner client to tables, views, and roles:

• Query statements, also called Data Query Language (DQL), scan tables or expressions and return result rows
• Data Definition Language (DDL) statements create and modify tables, views, and database roles
• Data Manipulation Language (DML) statements insert, update, and delete rows
• Data Access Control statements grant and revoke privileges at the table and column level
• Transaction Control statements manage transactions for data modifications

Spanner applies strong consistency across distributed clusters, so a read after a committed write sees current data. The Spanner commit timestamp records when each row was committed, which lets applications select only rows added or modified since the last full backup for incremental backup.

2.6NoSQL Operations, Consistency, and Recovery

NoSQL operations with Firestore in Datastore mode, often called Datastore, use a schemaless document model built for automatic scaling and high performance. Datastore organizes data as kinds containing entities, where each entity has properties and a unique key, which maps naturally to objects in object-oriented and scripting languages. Applications access Datastore through multiple clients and a SQL-like query language, and all queries are served by previously built indexes so query performance scales with result-set size rather than dataset size. Datastore is schemaless and does not enforce uniform properties across entities of the same kind, so scaling is achieved by distributing writes automatically and restricting queries to index-supported patterns.

The object mapping across models is:

Datastore does not support join operations, inequality filtering on multiple properties, or filtering based on results of a subquery. Applications that need joins and subqueries should use a Firestore Native or MongoDB-compatible API instead. Datastore provides atomic transactions where operations either all succeed or none occur, strong consistency for all queries, high availability through redundancy across Google data centers, and automatic encryption at rest.

Transaction consistency centers on atomicity, strong consistency, and controlled recovery from bad writes. Spanner adds point-in-time recovery (PITR) that can restore data from up to 7 days in the past, and the same 7-day window applies to BigQuery snapshot decorators for corrupted tables. Availability depends on replication: Cloud SQL supports high-availability configuration within a region and cross-region replicas, while Spanner throughput and recovery time depend on configured compute capacity because node count affects recovery time objective (RTO). For small RTO values, a warm standby Spanner instance can be kept at minimum nodes for storage and throughput.

Backup and recovery move data from the live database to Cloud Storage and back to a new table, dataset, or instance during restore. Managed Airflow, a managed version of Apache Airflow, can schedule these operations with a directed acyclic graph (DAG) using operators such as BigQueryToCloudStorageOperator, DatastoreExportOperator, MySqlToGoogleCloudStorageOperator, and PostgresToGoogleCloudStorageOperator:

Spanner Backup and Restore has lower RTO than Spanner export because restore directly mounts the backup without copying data, while export must copy Avro files through Cloud Storage. BigQuery follows the same staged pattern with long-term storage that drops storage price by 50 percent after a table is unedited for 90 consecutive days, replication to two zones in a single region, and recovery by querying a past snapshot, exporting uncorrupted data, keeping time-partitioned tables, copying datasets across regions, or reloading original data kept in Cloud Storage.

Choosing the right service depends on structure, transaction needs, and query patterns:

• Choose Cloud SQL when the system needs OLTP with full SQL support and relational joins
• Choose Spanner when the system needs globally distributed relational data with strong consistency and cross-region recovery
• Choose Firestore in Datastore mode when the system needs highly available structured data at scale with automatic write distribution
• Choose BigQuery when the workload is interactive querying for online analytical processing (OLAP)
• Choose Cloud Storage when the data consists of large immutable blobs such as images and movies

2.7Integrating Object Storage and Analytical Warehouses

Integrating object storage and analytical warehouses follows a staged flow where data moves from a source system to Cloud Storage and then from Cloud Storage to BigQuery. Cloud Storage is the object storage service that holds staging files as objects in buckets, and BigQuery is the analytical warehouse that stores those files as queryable tables in datasets. Cloud Storage acts as the durable landing zone while BigQuery provides high-performance ingestion and SQL analytics, so choosing the right transfer and load pattern controls latency, cost, and throughput.

Uploading source files starts by extracting data to a vendor-neutral format, with Apache Avro, ORC, or Parquet preferred because schema is embedded with the data. If CSV or another delimited format is used, the schema must be supplied separately at load time or through schema auto-detection. The upload method depends on data location, size, and bandwidth:

The Google Cloud CLI supports multi-threaded parallel uploads, resumes after errors, and encrypts traffic in transit. Latency is minimized by creating the bucket closest to the data center, and when possible the bucket location should match the BigQuery dataset location.

Loading Cloud Storage data into BigQuery is the batch ingestion step that reads bucket objects into destination tables. The BigQuery Data Transfer Service for Cloud Storage supports CSV, newline-delimited JSON, Avro, Parquet, and ORC, including supported compressed data. A recurring transfer requires the destination table and schema to exist before creation, all files matching the source URI must share that schema, only live objects can be transferred, and archived versioned objects are not supported. A controlled transfer moves through an ordered sequence:

1. Extract: Export source data to files in staging format and copy the files to staging storage.
2. Copy: Copy data from the source, such as Amazon S3, to a Cloud Storage bucket using Storage Transfer Service, the Google Cloud CLI, or client libraries.
3. Load: Load bucket data into BigQuery using BigQuery Data Transfer Service, the bq tool, or client libraries.

For one-time loads, batch load from Cloud Storage is used, while scheduled loads use Cloud Storage transfers in BigQuery Data Transfer Service with parameterized source paths and destination tables for date-organized buckets. After loading, the open source Data Validation Tool can compare source and target data with aggregations such as count, sum, and average.

Recurring ingestion with write preferences controls whether each run adds or replaces rows. A transfer with APPEND or WRITE_APPEND performs an incremental transfer that appends only data changed since the previous successful run, detected by comparing each file's updated timestamp against the last successful transfer time. For example, a file created at 00:00Z is loaded by the 03:00Z run, the next day's run loads nothing when no updated time is newer, and a later file is picked up by the following run. A truncated-transfer preference replaces data instead. If files change while a transfer runs, exactly-once transfer is not guaranteed, and schema changes between runs cause failure.

Transforming data with ETL and ELT patterns determines whether logic runs before BigQuery or inside BigQuery. Transformation in the source environment can suffer from limited compute and extra transfer time for enriched data. Transformation on Google Cloud uses a managed service that reads from the bucket and writes directly to BigQuery, while ELT loads raw files into a staging table first and then runs SQL into the final production table. Managed transformation services divide work by complexity:

• Cloud Data Fusion graphically builds fully managed ETL and ELT pipelines with preconfigured connectors
• Dataflow defines and runs complex transformation graphs with the Apache Beam model on serverless capacity
• Managed Service for Apache Spark runs Spark and Hadoop clusters as a fully managed service with Cloud Storage and BigQuery connectors
• Third-party partner tools externalize pipeline building when a managed Google service is not used

Pub/Sub messaging patterns are the ways publishers, topics, subscriptions, and subscribers connect to move data asynchronously. Pub/Sub (opens in a new tab) is an asynchronous and scalable messaging service that decouples services that produce messages from services that process those messages. A publisher application sends events to a topic without waiting for processing, and Pub/Sub delivers those events to subscriber applications attached through subscriptions, typically with latency around 100 milliseconds.

3.1Topics, Subscriptions, and Asynchronous Data Flow

A Pub/Sub data flow moves data from publishers to a topic and then from the topic to one or more subscriptions that feed subscribers. The publisher creates a message containing data, attributes, or an ordering key and sends a publish request to the Pub/Sub server for a named topic. Each subscription attached to that topic receives its own copy of each message, so publishers and subscribers remain independent and can scale separately. Pub/Sub provides at-least-once delivery with best-effort ordering, and it uses per-message parallelism by leasing individual messages to subscriber clients and tracking whether each message is processed successfully.

A Pub/Sub subscription defines how messages travel from Pub/Sub to subscriber code, using push, pull, or export delivery. In push delivery, Pub/Sub sends each message as an HTTP POST request to a webhook endpoint such as Cloud Functions, so the subscriber does not need to call Pub/Sub. In pull delivery, the subscriber client uses a client library to request messages, by default over a StreamingPull connection that keeps messages flowing. An export subscription stores messages directly to a Google Cloud resource such as a BigQuery table, without requiring Dataflow as an intermediary.

Push subscriptions fit specific constraints where pull is difficult to use:

• The subscriber application cannot include the Pub/Sub client library as a dependency.
• The subscriber client cannot make outgoing requests to Pub/Sub.
• The same endpoint must process messages from different topics and subscriptions without knowing the subscription list in advance.

For general cases, the high-level client library with pull is recommended because it handles authentication, throughput optimization, and message formatting with minimal operational work.

3.2Publishing Messages with Client Libraries

Publishing messages with client libraries is how a publisher client creates messages and sends them to a Pub/Sub topic. Each published message must contain message data, attributes with additional metadata, or an ordering key, and the Pub/Sub service adds a message ID unique to the topic plus a receive timestamp. Messages are limited to 10 MB, so larger payloads are handled by storing the object in Cloud Storage and publishing a message that contains the URL, which the subscriber then downloads. A topic without an attached subscription or topic retention discards published messages, so a subscription must be attached or retention enabled before publishing begins.

Publisher applications can be configured with the Google Cloud console for testing and manual publishing, the Google Cloud CLI for command-line publishing, high-level Cloud Client libraries for high throughput and low latency, and low-level REST and RPC APIs built on auto-generated gRPC libraries. The high-level client library publishes asynchronously and batches multiple messages into one publish request based on request size, message count, and time, which improves throughput and lowers cost but adds publish latency. Publisher-side flow control prevents overload during traffic spikes by limiting outstanding requests, and the publish call returns a future that should be handled asynchronously with retries or persistence on failure.

3.3Message Reliability, Lifecycle, and Delivery Semantics

Message reliability and lifecycle in Pub/Sub is the set of controls that decide whether a message is redelivered, discarded, or saved for later. A publisher sends a message to a topic without waiting for processing, and Pub/Sub leases each message to a subscriber client and tracks whether the message is acknowledged. Reliability depends on acknowledgment, redelivery, retries, dead-lettering, ordering, and duplicate suppression.

Message delivery semantics define whether a message can be delivered more than once or lost during failure:

At-least-once delivery is the default and keeps data from being lost because unacknowledged messages are redelivered until the acknowledgment deadline expires or a retry limit is reached. At-most-once and exactly-once reduce or remove duplicates, but they require careful acknowledgment handling and, for exactly-once, acceptance of higher latency.

The acknowledgment lifecycle controls when a message is discarded, retained, or redelivered. By default Pub/Sub discards a message from a subscription after the subscriber acknowledges it, so subscribers must process a message before acknowledging it or the message is lost on processing failure. Retained acknowledged messages are an exception because a subscription can retain acknowledged messages for up to 7 days and replay them with a seek operation. A message moves through an ordered lifecycle:

1. Lease: Pub/Sub leases an individual message to a subscriber client and starts the acknowledgment deadline.
2. Process: The subscriber processes the message while holding the lease and can extend the lease when processing takes longer.
3. Acknowledge: The subscriber sends an acknowledgment on success or a negative acknowledgment to request immediate redelivery.
4. Redeliver or discard: Pub/Sub discards the message after acknowledgment or redelivers it when the deadline expires without acknowledgment.

Subscriber flow control protects this lifecycle during spikes by limiting maximum outstanding messages and total outstanding message bytes. When either limit is crossed, the client stops pulling more messages until outstanding messages are acknowledged, which trades throughput against the cost of running more subscribers. Consuming messages also uses three lifecycle states: acknowledged (acked) where the processed message is deleted once all subscriptions acknowledge it, unacknowledged (unacked) where no acknowledgment arrived before the deadline so redelivery may occur, and negatively acknowledged (nacked) where the subscriber explicitly rejects the message, for example with modifyAckDeadline set to 0, and Pub/Sub redelivers it immediately under the retry policy.

3.4Retry Policies, Ordering, and Duplicate Handling

A retry policy in Pub/Sub determines how many delivery attempts Pub/Sub makes before treating a message as unprocessable. If a subscriber cannot process a message before the deadline, Pub/Sub redelivers the message and counts each delivery as an attempt. For pull subscriptions the lease can be extended with lease management, while for push subscriptions the acknowledgment deadline setting may need to be increased to give the endpoint more time. A dead-letter topic is a separate topic that stores messages that repeatedly fail processing. After the configured number of delivery attempts is exhausted, Pub/Sub automatically forwards the message to the dead-letter topic for later inspection instead of redelivering it forever. This separation keeps a few bad messages from blocking healthy messages and preserves failed payloads for debugging and reprocessing.

Message ordering in Pub/Sub preserves publish order for messages that share the same ordering key. Ordering must be enabled on the subscription, while publishers attach the same ordering key to related messages and publish them in the same region. To keep messages in the correct region, publishers and subscribers use locational endpoints that route traffic to that region. Ordered delivery changes acknowledgment behavior because acknowledgments for later messages are not processed until earlier messages with the same ordering key are acknowledged. For example, if messages 1, 2, and 3 share a key and only message 3 is acknowledged, Pub/Sub treats all three as unacknowledged until messages 1 and 2 are also acknowledged and then redelivers all three. Push subscriptions are not recommended for frequently used ordering keys because Pub/Sub allows only one outstanding message per ordering key at a time, so StreamingPull or Pull subscriptions provide better parallelism and lower latency.

Duplicate handling starts with idempotent subscribers that produce the same result when the same message is processed more than once. If subscribers cannot be made idempotent, Dataflow can be incorporated to deduplicate messages before processing. A high rate of duplicates often means subscribers are not acknowledging messages or the acknowledgment deadline is too short. Common patterns have different remedies: consistent redelivery of many messages means subscribers are overloaded or missing deadlines, so adjust flow control, extend leases, increase the push acknowledgment deadline, or check subscription health; occasional redelivery before the deadline or shortly after acknowledgment is expected behavior affecting several messages at once, so the system must tolerate infrequent duplicates; repeated redelivery of a few messages means the subscriber is not handling those messages correctly or Pub/Sub is recovering from disruption, so verify acknowledgment logic and configure a dead-letter topic. Exactly-once delivery is available only for pull subscriptions and increases publish-to-subscribe latency, so it should be chosen only when the application cannot tolerate duplicates.

3.5Publish-Subscribe Shapes for Scale

A Pub/Sub publish-subscribe pattern describes how many publishers, topics, subscriptions, and subscribers connect to share work. The same topic can feed one subscription for shared processing or multiple subscriptions when each downstream system needs every message. Multiple subscribers can also share one subscription to divide the work and increase throughput.

Fan-out is the right choice when different applications must independently process the same events, while load-balanced sharing of one subscription is the right choice when the goal is parallel task processing such as compressing files, sending notifications, or reformatting images.

3.6Orchestrating Workflows with Cloud Tasks

Orchestrating workflows with Cloud Tasks means managing asynchronous execution outside the user request path. An application creates a task and adds it to a queue, and Cloud Tasks delivers that task to a target handler at a controlled rate. This pattern keeps long-running work from blocking HTTP responses while protecting backends from overload and extra cost. Cloud Tasks handles background work that should not run after an HTTP response is delivered. Background work is any work an app performs for a request after the HTTP response has been sent, and apps should finish all asynchronous operations before delivering the response.

Performing background work inline instead can increase billing because App Engine may scale up additional instances due to high CPU load even with no active requests, and users may see higher latency while requests wait in the pending queue for available instances. Enqueuing the work moves that load to a task handler that runs at a controlled rate. The result is lower user-facing latency and fewer unnecessary instances.

Push queues in App Engine are the queues that hold tasks until Cloud Tasks or the Task queue service delivers them to targets. App Engine provides a default push queue that is configured and ready to use, so an application can start by adding all tasks to the default queue without creating other queues. To add queues or change the default configuration, the team edits the queue configuration file and uploads it with gcloud app deploy queue.yaml. Queue configuration controls routing, throughput, and lifecycle limits:

• An application can create up to 100 queues, and queues cannot be created dynamically.
• Each queue can define a target, such as v2.task-module for queue-blue, so tasks flow from the queue to that target module.
• Each queue can define a rate, such as 5/s or 1/s, so tasks are dequeued and sent to their targets at that rate.
• An application must not assume tasks run in enqueue order.
• After a queue is deleted, the team must wait approximately 7 days before creating a new queue with the same name.

Deploying the queue configuration with a service account requires the serviceusage.services.list permission, granted through the serviceusage.serviceUsageViewer role or a custom role. For transactional push-queue tasks that fail to enqueue during a datastore commit, the application relies on an internal cron job that sweeps for missed tasks. A cron.yaml entry calls /_ah/cloudtask/sweep on a schedule such as every 30 minutes, so failed enqueues are eventually processed.

Rate control in push queues uses a token bucket for each named queue. The bucket holds tokens up to bucket_size, or up to 5 tokens when no bucket size is specified, and App Engine refills the bucket continuously based on the queue rate. The rate, bucket_size, and max_concurrent_requests directives together control how quickly tasks flow from the queue to target handlers. Task execution follows an ordered cycle:

1. Consume token: Each time the application executes a task, one token is removed from that queue's bucket.
2. Process while tokens remain: The app continues processing tasks from the queue until the bucket runs out of tokens.
3. Refill over time: App Engine adds new tokens continuously at the configured rate, which allows more tasks to run.

If the queue holds tasks and the bucket holds tokens, App Engine processes as many tasks simultaneously as there are tokens. That burst can consume system resources, compete with user-serving requests, and cause datastore contention. Use max_concurrent_requests when the workflow must limit how many tasks run at once or protect a shared datastore.

Request priority in App Engine determines how Cloud Tasks traffic waits during heavy traffic. During heavy traffic, App Engine may place requests in a pending queue while waiting for an available instance. Legacy Task queue requests sit at lower priority than other queued requests, and App Engine Cloud Tasks requests share that same lower priority for compatibility reasons. Requests from HTTP-target Cloud Tasks are treated as regular HTTP traffic and do not sit at that lower priority:

When a service receives high-volume standard HTTP traffic plus much lower-volume Task queue or Cloud Tasks traffic, the task traffic sees a disproportionate latency impact. To avoid priority queuing, split the traffic types to separate versions or use HTTP-target tasks. For latency-sensitive work served from Cloud Tasks, use a dedicated major version or service. Rate limiting protects workflows from rapid load increases from malicious distributed denial-of-service (DDoS) attacks or client configuration errors, using quotas on API requests and project-level resource quotas that also limit billing. Graceful degradation keeps the system available under high load by reducing performance or accuracy instead of failing completely, using independent throttling at each replica and early dropping of excess frontend requests to protect backends, with tools such as Apigee controlling API request rates through policy rules.

Key Facts

• Cloud Storage bucket and project access should use IAM rather than ACLs, and VPC Service Controls perimeters can block data movement even when IAM would allow it.
• The Cloud SQL Auth Proxy requires the cloudsql.instances.connect permission, must run continuously as a system service, and issues 2 API calls per hour per connected instance while running.
• Firestore in Datastore mode exports and imports require both caller permissions and Firestore service agent bucket access, with limits of 50 concurrent operations and 20 requests per minute per project.
• A recurring Cloud Storage transfer to BigQuery defaults to APPEND and detects new files by comparing each file's updated timestamp to the last successful run, and only live objects can be transferred.
• Pub/Sub provides at-least-once delivery by default, discards messages after acknowledgment unless 7-day retained-message replay with seek applies, and limits messages to 10 MB so larger payloads must be stored in Cloud Storage with only the URL published.