Building Scalable Secure Reliable Cloud Native Apps

Designing high-performing applications and APIs means choosing the right compute platform, routing traffic efficiently, and scaling resources to demand. Traffic flows from users through load balancers to backends on Compute Engine, Google Kubernetes Engine (GKE), or Cloud Run, with caching, APIs, and event routing shaping performance along the path.

1.1Choosing Compute Platforms and Shipping Containers

Choosing a compute platform means matching control and scaling needs to Compute Engine (opens in a new tab), Google Kubernetes Engine (GKE), or Cloud Run. Compute Engine is Infrastructure as a Service where you manage the guest operating system, kernel, machine type, and network rules, with support for all NVIDIA GPUs and Google TPUs. GKE manages the Kubernetes control plane, with Autopilot mode where Google manages nodes and Standard mode where you manage node pools, while Cloud Run fully manages infrastructure and runs stateless request-driven services and jobs that exit when done.

Container delivery promotes the same immutable image from development to quality assurance to production. Source changes trigger Cloud Build to compile code, test the image, and push passing images to Artifact Registry with semantic versions or commit hashes. On Cloud Run, a revision is an immutable snapshot of image and settings that scales to zero and supports traffic splitting for canary and blue-green releases. On GKE, a Deployment declares replicas, a Service exposes pods at a stable endpoint, and autoscalers change pod and node counts.

Traffic splitting divides requests by explicit weight rather than health. On Cloud Run, percentages split the main service URL across revisions while tagged revision URLs bypass the split for testing. On GKE, weights in routing resources split across Services or clusters, with safe ramps following the 500/50/5 pattern of starting at or below 500 transactions per second, increasing by no more than 50%, and waiting at least 5 minutes.

1.2Routing, Affinity, and Caching for Fast Delivery

Load balancer selection depends on protocol and geography in Cloud Load Balancing. Application Load Balancers terminate client connections at Google Front Ends or Envoy proxies and route HTTP and HTTPS by host, path, headers, and parameters to backends on Compute Engine, GKE, Cloud Run, on-premises, or other clouds. Passthrough Network Load Balancers preserve source and destination addresses, let backends terminate TCP, UDP, ESP, GRE, ICMP, and ICMPv6 connections, and return responses directly through direct server return.

Scope and affinity control where traffic goes and whether repeat clients stick to one backend. Global load balancers use a single anycast IP with backends in multiple regions, while regional load balancers keep backends and TLS termination in one region for compliance. Session affinity sends repeat requests to the same backend to reuse cached state, with Generated Cookie and header affinity for HTTP(S) on Application Load Balancers and Client IP affinity for TCP and UDP on Network Load Balancers. Affinity breaks when the backend fails health checks and moves from healthy to unhealthy, when backends are added or removed, or when UTILIZATION mode reshuffles groups, so RATE or CONNECTION modes are more stable.

Managed caching with Memorystore for Redis serves hot data from memory in sub-millisecond time. Application traffic flows from clients in an authorized Virtual Private Cloud (VPC) to the cache private endpoint, which has no public IP. Basic Tier is a standalone instance, while Standard Tier provides cross-zone replicas, automatic failover, up to 300 GB, up to five read replicas, and a 99.9% availability SLA. Read-through loads on miss, write-through writes to cache and storage together, and write-back writes to cache first for write-heavy work.

1.3Building APIs and Controlling Request Rates

API management gives apps one stable entry point while backends change. Traffic flows from clients to the gateway or proxy and then to backends on Cloud Run, GKE, Compute Engine, or App Engine. API Gateway fronts Cloud Run and App Engine with logging, monitoring, and authorization, Cloud Endpoints uses the Extensible Service Proxy (ESP) or ESPv2 to proxy REST and gRPC, and Apigee uses API proxies as a facade with rate limiting, quotas, and analytics.

REST and gRPC differ in contract and interaction style. HTTP REST uses resource URLs and JSON described by a Discovery document with resources, schemas, and OAuth scopes. gRPC uses defined RPC methods with typed messages, supporting unary calls such as ProcessOrder and bidirectional streaming such as BidiProcessOrder for concurrent audio, text, and updates. Pairing gateways with an Application Load Balancer adds multi-region balancing, Cloud CDN caching, and Cloud Armor protection.

Rate controls sit between consumers and backends so excess requests are rejected before services are exhausted. Quotas limit sustained volume over time and cap billing, while the SpikeArrest policy blocks sudden bursts and returns HTTP 429 when exceeded. An API product bundles proxies, paths, methods, and quotas, and developer apps receive API keys tied to those products. API Gateway backends allow only authenticated access, where the gateway service account needs roles/run.invoker for Cloud Run.

1.4Connecting Services with Events and Scheduled Work

Asynchronous integration with Cloud Pub/Sub decouples publishers from subscribers so services scale independently. A publisher sends each message to a topic, and Pub/Sub delivers a copy to every attached subscription. Eventarc routes events from more than 130 Google providers to destinations such as Cloud Run functions or Workflows without the producer naming the consumer.

Delivery modes differ in who initiates transfer:

• Pull delivery lets subscriber clients request messages with credentials, control rate, and process in parallel with batched throughput.
• Push delivery lets the Pub/Sub server send one message per request to a public HTTPS endpoint, where the client signals overload with HTTP errors.
• Export subscriptions write directly to an existing BigQuery table, Cloud Storage bucket, or Bigtable table with no subscriber client.

Reliability requires acknowledging only after successful processing, using ordering keys and DeadLetterPolicy, and writing idempotent handlers because duplicates can occur. An Eventarc trigger stays inactive until its integration is published and works best when trigger, resource, and destination are in the same region. Workflows runs ordered YAML or JSON sequences across HTTP APIs, Cloud Tasks queues asynchronous work with rate limiting, and Cloud Scheduler triggers recurring work on schedule.

1.5Placing Workloads, Sizing Resources, and Managing Cost

Geographic placement controls latency and failure isolation from universe to region to zone. Placing endpoints in the region closest to users shortens round-trip time, because traffic flows from users to the regional endpoint and then to dependent data services. Zonal resources such as Compute Engine virtual machines and Persistent Disk volumes need cross-zone redundancy, regional resources such as regional managed instance groups (MIGs) and Cloud Run services survive a single-zone failure, and multi-region and global resources such as VPC networks and Pub/Sub topics resist regional failures.

Resource sizing matches compute, storage, IOPS, throughput, and latency to goals. Persistent Disk regional volumes replicate synchronously across two zones, Hyperdisk Balanced provisions IOPS and throughput separately, Cloud Storage regional buckets replicate across zones, and dual-region buckets with turbo replication provide a 15-minute recovery point objective. Compute Engine custom machine types size vCPUs and memory independently, Spot VMs offer the same performance at lower cost but can be preempted, MIGs scale on CPU thresholds, GKE uses Cluster Autoscaler plus Horizontal and Vertical Pod Autoscalers, and Cloud Run scales to zero and bills pay-per-use rounded to the nearest 100 milliseconds.

Cost control moves data and compute to the cheapest tier that meets access needs. Standard and Nearline storage fit frequent access, while Coldline and Archive fit infrequent access with lower storage cost but retrieval fees and minimum durations. Object Lifecycle Management moves or deletes objects when all rule conditions on age and version state are true, while Autoclass adjusts classes automatically. Recommender suggests idle VM and machine-type changes, while committed use discounts require one-year or three-year commitments and sustained use discounts apply automatically. Synchronous replication gives zero recovery point objective within a region but adds commit latency, while only asynchronous replication fits remote-region disaster recovery because synchronous cross-region writes add too much latency.

Designing secure applications means applying retention, scanning, identity, and isolation controls so only trusted code and identities reach data. Policies flow from organization to project to bucket or resource, findings flow from scanners to developers, and requests flow from users through authentication checks to protected backends.

2.1Retaining and Organizing Data with Cloud Storage Policies

Data retention with Cloud Storage starts with buckets as policy boundaries. Every bucket belongs to a project that belongs to an organization, and a bucket sets location, storage class, access, and protection for all objects inside it. Regional buckets replicate synchronously across zones in one region, while dual-region and multi-region buckets replicate asynchronously across regions. Separate buckets for workloads such as photos and videos let each have its own location and retention rule, while prefix naming such as logs/2026/ and labels such as team: analytics make lifecycle rules and cost reporting selective.

Object Lifecycle Management moves or deletes objects when metadata matches bucket rules. Cloud Storage checks age, creation date, and version state, then runs the transition or deletion only when every condition in the rule is true. Storage class trades storage cost for retrieval cost, with Standard for high-frequency access, Nearline for monthly access such as backups, Coldline for less frequent access, and Archive for lowest-frequency access.

Versioning, holds, and soft delete decide whether deletion can proceed. Object Versioning preserves an overwritten object as a noncurrent version, soft delete retains deleted objects for recovery, and object holds block deletion while the flag is present. Bucket-level retention enforces Write Once, Read Many protection from creation time, and hierarchical namespace and object locking can only be enabled at bucket creation.

2.2Scanning Code and Enforcing Zero-Trust Access

Vulnerability response finds flaws in images and configurations, ranks them by risk, and blocks risky deploys. Artifact Analysis scans images in Artifact Registry on push and monitors them for up to 30 days, so a clean image can become flagged later without a new push. A note describes a vulnerability, while an occurrence records each instance of that note on an image. Security Command Center (SCC) centralizes findings across virtual machines, containers, and Cloud Run, with Standard for basic scanning, Premium for posture and attack paths, and Enterprise for multi-cloud protection.

Deploy-time enforcement ensures only approved images run. Cloud Build builds in isolated environments, Artifact Analysis scans stored images, and Binary Authorization allows only images with required attestations to deploy to Google Kubernetes Engine (GKE) (opens in a new tab) and Cloud Run. The chain runs in order of build, scan, attest by a trusted authority, then enforce at deployment. Because attestation binds to the immutable digest, changing image contents requires a new attestation signed with the private key and verified with the public key in the attestor.

Zero-trust access assumes no user is trusted by network location. Web Security Scanner tests public URLs on App Engine, Compute Engine, and GKE for cross-site scripting and misconfigurations, with code deployed first, scan triggered next, and findings flowing back for triage. Identity-Aware Proxy (IAP) forwards traffic to the app or virtual machine (VM) only after verifying Google identity and the IAP-Secured Web App User role, enabling SSH without an external IP. VPC Service Controls blocks access to Google services outside a service perimeter for all identities, while Chrome Enterprise Premium evaluates request context for user identities in the organization.

2.3Storing Secrets and Authenticating Workloads and Users

Central secrets management keeps certificates and credentials in one governed store. Secret Manager stores versions with API access, and every change in the central store must synchronize to dependent stores such as MQTT broker certificate stores. Distribution uses direct push during provisioning and pull during rotation, and the old certificate must be invalidated only after the device reconnects with the new certificate and confirms completion.

Keyless authentication with Workload Identity Federation replaces long-lived keys with short-lived tokens. The workload exchanges its ambient credential for a Google token through Security Token Service (STS), which verifies the issuer, applies attribute mappings, and returns a federated token. The ordered exchange is obtain external credential, send credential to STS, verify and map identity, receive federated token, then access Google Cloud directly or by impersonating a service account.

Envelope encryption with Cloud Key Management Service (Cloud KMS) keeps the key encryption key (KEK) in KMS while data lives in the integrated service. With customer-managed encryption keys (CMEK), you create the key ring in the same location as data, grant the service account CryptoKey Encrypter/Decrypter, and audit Encrypt and Decrypt calls. Key states move in order from enabled to disabled to scheduled for destruction to destroyed, where disabling is reversible and destroying is permanent.

Programmatic authentication with Application Default Credentials (ADC) finds credentials automatically in fixed order:

1. GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to a JSON file.
2. Local ADC file created by gcloud auth application-default login.
3. Attached service account from the metadata server.

The attached service account is preferred for production even though it is checked last, because the platform manages short-lived credentials. OAuth 2.0 access tokens carry authorization with scopes, while OpenID Connect ID tokens prove authentication to a specific audience. The Cloud SQL Auth Proxy and AlloyDB Auth Proxy build secure TCP tunnels where the app connects to localhost and the proxy upgrades to TLS with IAM authorization, and Identity Platform isolates customer sign-in with tenants as separate silos in one project.

2.4Applying Least Privilege with IAM and Private Service Paths

IAM security grants only needed permissions at the narrowest hierarchy level. The hierarchy flows from organization to folder to project to service-specific resource, and the effective allow policy is the union of a resource policy plus all ancestor policies. A policy attached to a project reaches all resources inside it, while a policy attached to a specific bucket or topic reaches only that resource and prevents lateral movement.

Role choice trades precision for management effort. Basic Owner, Editor, and Viewer roles grant broad project access and should never be used for production service accounts. Predefined roles bundle job-function permissions, while custom roles bundle only listed permissions. Requests from a Compute Engine VM must pass both IAM and access scope checks, so a narrow scope can block an action the role would permit, which is why cloud-platform scope with least-privilege IAM roles is recommended.

Service-to-service isolation keeps traffic private and identity-checked. Direct VPC egress lets Cloud Run and Cloud Run functions send directly to internal VPC addresses without a connector, while Private Service Connect (PSC) exposes a producer service to a consumer VPC through a private endpoint without exposing the producer network. Cloud Service Mesh enforces mutual TLS (mTLS) where client and server verify each other, with auto mTLS accepting plain text by default until strict mTLS is enforced. A NetworkPolicy acts as a pod-level firewall using podSelector and policyTypes for Ingress and Egress, and creating the first policy in a namespace denies all traffic of that type that is not explicitly allowed. Single-purpose service accounts with short-lived tokens from the metadata server replace broad defaults and long-lived JSON keys.

Storing and accessing data means matching volume, latency, and consistency needs to the right storage system and schema. Writes flow from applications to the chosen database or object store, replicas copy data for availability, and readers see either the latest write or a briefly older copy depending on consistency.

3.1Selecting Storage for Volume, Latency, and Access Pattern

High-throughput and low-latency stores serve different working sets. Bigtable is a fully managed multidimensional key-value store that supports tens of thousands of queries per second with consistent single-digit millisecond latency and petabyte-scale storage with failover between replicated clusters. Memorystore is fully managed in-memory storage for caching in front of a primary database, where Redis Standard tiers replicate automatically across zones while Memcached does not replicate across zones or regions.

Block, object, and file systems differ in mountability and request limits. Persistent Disk (PD) provides up to 64 TB per disk and up to 128 disks per Compute Engine instance, attaches read-write to only one instance but read-only to thousands, and can be accessed only in the same region. Cloud Storage provides virtually unlimited object storage accessible worldwide but is not POSIX-compliant, starting around 1000 write and 5000 read requests per second per bucket and autoscales over minutes, so sudden spikes cause errors without ramp-up. Filestore supports thousands of simultaneous NFS connections, NetApp Volumes supports NFS and SMB with snapshots, and Cloud Storage FUSE mounts buckets cheaply without POSIX compliance.

Relational and non-relational choices differ in schema and scale. Cloud SQL offers MySQL, PostgreSQL, and SQL Server for transaction processing with automated backups and cross-region replicas, with Enterprise Plus adding a 99.99% availability SLA. Cloud Spanner (opens in a new tab) is a distributed SQL database with global strong consistency, while Firestore in Datastore mode is a schemaless NoSQL document database with automatic scaling and strong consistency. Throughput measures data moved per second, IOPS measures small operations per second, and storing data in the same zone as compute reduces time and transfer cost.

3.2Designing Schemas for Bigtable, Firestore, Spanner, and AlloyDB

Row key design in Cloud Bigtable controls sorting, splitting, and access speed. Each row has one row key, rows store in sorted key order so nearby keys support fast range scans, and requests flow from applications through an app profile to clusters. Structured row keys join fields such as user ID, purchase date, and order number, where field order sets sort order. Cluster sizing takes the larger of storage needs, dividing unreplicated size by 3.5 TB per SSD node, and compute needs, dividing sustained operations per second by 10,000.

Document modeling in Firestore in Native mode starts from queries rather than normalized tables. Documents hold fields the application reads together so one read returns a complete unit. Indexing locates matching documents directly instead of scanning unrelated data, with Query Insights identifying queries that need an index. Time-to-live policies expire unneeded documents, while backups and point-in-time recovery preserve data for restore.

Relational schema design for Cloud Spanner and AlloyDB for PostgreSQL distributes load by key choice. Spanner sorts rows by primary key across servers, so sequential keys concentrate writes and create hotspots. High-write tables should use UUID Version 4 with GENERATE_UUID() stored as STRING or bit-reversed positive sequences that spread inserts across the 64-bit space. Secondary indexes on frequently filtered columns avoid full-table scans, and tuning follows load sample data, connect application, verify consistency, load test, then modify and retest with Key Visualizer revealing hotspots.

3.3Understanding Consistency, Replication, and Failover

Consistency trades freshness for availability during partitions. With eventual consistency, data flows from originating Node A to replicas B and C without blocking reads, so replicas stay available but may briefly show stale data. With strong consistency, the system locks data during update, so replicas stay consistent but are unreadable until replication finishes. Eventual consistency fits approximate views such as like counts, while strong consistency fits billing completion or inventory decrement.

Bigtable replication copies each write from the receiving cluster to every other cluster in the same instance. Single-cluster routing sends all requests to one cluster and can provide read-after-write on that cluster, while multi-cluster routing sends each request to the closest cluster for lowest latency with eventual consistency across clusters. Adding a second cluster improves availability and read locality but does not double write throughput, because every write is stored twice and each cluster spends CPU pulling changes. Failover reroutes from an unresponsive cluster to another with last write wins conflict resolution.

Cloud Spanner provides global strong consistency using TrueTime timestamps applied in timestamp order, where blind writes can proceed in parallel and single-split transactions bypass two-phase commit for speed. Cloud SQL and AlloyDB use primary and secondary topology where the primary accepts writes and replicas copy asynchronously and can lag, so correctness-critical reads should go to the primary while tolerant reads such as dashboards can use replicas. Synchronous replication waits for standby confirmation for zero recovery point objective within a region, while only asynchronous replication fits remote-region disaster recovery.

3.4Granting Temporary Access and Writing Analytics Data

A V4 signed URL for Cloud Storage embeds authentication in query parameters for one action on one object. The URL is created by a service account that already has permission, letting a person without Google credentials read or write for a limited time. The resource URL identifies the bucket and object, while X-Goog-Algorithm, X-Goog-Credential, X-Goog-Date, X-Goog-Expires, X-Goog-SignedHeaders, and X-Goog-Signature define who signed it and how long it lasts. The HTTP method binds the operation, with GET for downloads and PUT for uploads, and the longest X-Goog-Expires is 604800 seconds (7 days).

Delegated access requires two permissions working together. The signer needs Service Account Token Creator to impersonate the service account, and the service account needs the storage operation such as storage.objects.get for reads or storage.objects.create for uploads. Programmatic generation with client libraries creates per-user URLs inside code using Application Default Credentials, while gcloud storage commands create one-off URLs for testing. Uniform bucket-level access is preferred over access control lists, which are limited to 100 entries.

Writing to BigQuery for analytics favors columnar formats and partitioned tables so queries scan less data. Apache Parquet and ORC perform better than JSON or CSV, Hive layouts prune partitions, and clustering sorts by filter columns to skip blocks. Rows streamed with the Storage Write API first land in the __UNPARTITIONED__ partition before repartitioning. Batch loading uses fewer large jobs through Cloud Storage staging files, while streaming sends events through Pub/Sub and Dataflow for near-real-time dashboards. The Storage Write API default stream gives at-least-once delivery, pending streams buffer until commit for atomic availability, and committed streams with client-supplied offsets give exactly-once delivery.

Key Facts

• Compute Engine provides self-managed virtual machines, Google Kubernetes Engine provides managed Kubernetes orchestration with Autopilot and Standard modes, and Cloud Run provides serverless containers that scale to zero and bill pay-per-use to the nearest 100 milliseconds.
• Application Load Balancers route HTTP and HTTPS by host, path, and headers with proxy termination, while passthrough Network Load Balancers preserve client addresses and support TCP, UDP, ESP, GRE, ICMP, and ICMPv6.
• Artifact Analysis scans container images on push and monitors them for 30 days, while Binary Authorization blocks deployment when required attestations are missing.
• Application Default Credentials searches in order of GOOGLE_APPLICATION_CREDENTIALS file, local ADC file, then attached service account, with the attached account preferred for production.
• A Cloud Storage V4 signed URL binds method, resource, signed headers, and expiration into X-Goog-Signature, with a maximum X-Goog-Expires of 604800 seconds.