Prepare and test your skills
Prepare and test your skills
Advanced access patterns and resource optimization is the practice of matching each workload to the right datastore, securing access with least privilege, and designing connections for low latency and high availability. In Google Cloud, Cloud Storage (opens in a new tab) provides object storage for any amount of data, relational and non-relational databases provide complementary tradeoffs for consistency and scale, and Memorystore for Redis provides managed in-memory caching in front of slower storage. Secure, temporary access methods such as Signed URLs for Cloud Storage and low-latency communication such as gRPC for Firestore extend these patterns for distributed applications.
Cloud Storage is low-cost object storage where data can be accessed from inside and outside Google Cloud and is replicated across locations for redundancy. Access to buckets and objects is granted through Identity and Access Management (IAM) and access control lists (ACLs), and in most cases IAM is recommended because it grants permissions at the bucket and project levels. Granular IAM control extends to buckets and managed folders, and can be refined with IAM roles mapped to a user, group, or service account, IAM policies with conditions, and principal access boundary (PAB) policies that limit what a principal can access.
Data in Cloud Storage is encrypted by default with Google-owned and Google-managed encryption keys, with options for customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) or customer-supplied encryption keys (CSEKs) when more control is required. Sensitive data loaded through Cloud Storage can be discovered, classified, and de-identified with Sensitive Data Protection. Data residency is controlled by region selection, because data is stored or replicated only within the regions specified for the bucket.
Exam tip: Use IAM, not ACLs, for bucket and project level access in Cloud Storage; regional buckets replicate synchronously across zones, while dual-region and multi-region buckets replicate asynchronously across regions.
Database choice constrains availability and scalability, so different parts of an application often use heterogeneous storage with different database types. Relational databases offer transactions, strong consistency, referential integrity, and sophisticated querying across tables, but they are typically hard to scale and require careful management in high-availability configuration. Non-relational (NoSQL) databases typically sacrifice some relational features for increased availability and easier scalability, often choosing availability over consistency under the CAP theorem.
Spanner (opens in a new tab) is a strongly consistent, highly available, and globally distributed database with support for SQL, combining relational querying with global scale:
| Option | Consistency model | Availability and scale tradeoff |
|---|---|---|
| Relational database | Strong consistency with transactions and referential integrity | Hard to scale; needs careful high-availability management |
| NoSQL database | Often eventually consistent | Higher availability and easier scalability |
| Spanner | Strongly consistent with SQL support | Highly available and globally distributed |
Choose a relational database when transactions and strong consistency are required, choose NoSQL when the data model can be eventually consistent and needs higher availability and scale, and use both strategically when different services have different needs.
Memorystore for Redis is a fully managed service powered by the Redis in-memory datastore that provides low-latency access and high throughput for heavily accessed data. A cache sits between the application and the slower disk-based storage layer, so requests flow from the application to the cache first and only reach the downstream database when the data is not cached. This placement reduces request latency to storage, reduces load on downstream databases, and lets both the application and other components that share the database scale further.
Caching also increases resiliency because the cache can continue to handle requests when the underlying storage layer is overloaded or unavailable, even if the returned data is incomplete or not up to date:
Memorystore for Redis can be deployed in a high-availability configuration that provides cross-zone replication and automatic failover, so cached data survives a zone failure.
High availability in Google Cloud depends on where data is replicated and how traffic fails over when a zone or region fails. AlloyDB for PostgreSQL (opens in a new tab) clusters provide high availability (HA) by default with automatic failover, where the primary instance uses redundant nodes in two different zones within a region to stay robust against zone outages. For recovery from region outages, AlloyDB supports cross-region replication. Place compute and data in the same region when latency matters, and choose dual-region, multi-region, or cross-region replication when the workload must survive a regional outage or serve users across regions.
Replication and failover behavior differs by storage location choice:
Resilient connections protect an overloaded or failing service by limiting traffic and handling errors gracefully, rather than retrying until the failure cascades. A service mesh (opens in a new tab) links distributed services and decouples business logic from networking, providing resiliency features such as request retries, failovers, and circuit breakers. Traffic flows from the application through the mesh to each downstream service, and the mesh applies the resiliency policy before the request reaches the overloaded service.
Three patterns work together to keep datastore connections reliable under load:
For message-based ingestion, Pub/Sub publisher flow control limits the rate of publish requests during transient spikes in message traffic, so publishers do not overwhelm subscribers and downstream datastores.
Secure connectivity and authentication for relational databases means applications reach Cloud SQL and AlloyDB for PostgreSQL through verified identities and encrypted paths instead of open network addresses and static passwords. Traffic flows from the application to a local proxy or connector, and from that proxy to the database instance over an authorized, encrypted tunnel. Identity and Access Management (IAM) controls who can connect, while private IP addresses, virtual private cloud (VPC) controls, and Transport Layer Security (TLS) protect data in transit.
The Cloud SQL Auth Proxy (opens in a new tab) is a client-side proxy that must authenticate as a Cloud SQL IAM identity before it can authorize connections to a Cloud SQL instance. Using a dedicated service account is the recommended method for production instances not running on a Compute Engine instance, because a credential file can be created specifically for the proxy and stays linked to the proxy as long as it is running. The same credential file can be duplicated into a system image when multiple machines need to invoke the proxy. Creating that service account requires the resourcemanager.projects.setIamPolicy permission, so a project owner or another user with that permission must create it if the developer does not have it.
IAM permissions for the Cloud SQL Auth Proxy determine whether the proxy identity is allowed to open connections to the instance. When finer-grained IAM roles are used, the proxy service account must have a role that includes the cloudsql.instances.connect permission. The predefined Cloud SQL roles that include this permission are:
If legacy project roles are used instead of Cloud SQL roles, the service account must have at least the Editor role.
The Cloud SQL Auth Proxy in production is a persistent service that must have the same uptime requirements as the application process. If the proxy process stops, all existing connections through it are dropped and the application cannot create new connections to the Cloud SQL instance. The proxy should run under systemd, upstart, or supervisor on Linux, and as a Windows Service on Windows using a wrapper tool because the proxy binary alone is not a Windows Service. Many application processes can share a single proxy process, so the pattern is to run one proxy client process per workstation or virtual machine and to include the proxy in the virtual machine image when using auto-scaling.
The proxy also depends on the Cloud SQL Admin API, and its API calls count against project quota. Usage is highest when the proxy starts, and while the proxy is running it issues 2 API calls per hour per connected instance. Google occasionally releases new proxy versions, and version 1.12 or later must be running to connect using private IP.
Private network paths and SSL/TLS encryption (opens in a new tab) protect database traffic from exposure to public networks. AlloyDB supports TLS 1.3 between the database instance and clients, with the server certificate generated automatically, and the Cloud SQL Auth Proxy transparently upgrades non-TLS connections to encrypted connections. AlloyDB Language Connectors use mutual TLS (mTLS) with client certificates, and both the AlloyDB Auth Proxy and Language Connectors use IAM to authorize the connection and mTLS to protect it. For Cloud SQL, SSL enforcement is controlled by an SSL mode that allows either mixed traffic or encrypted-only traffic.
To require SSL/TLS on a Cloud SQL instance, change the instance security setting through the console, gcloud, or Terraform:
The equivalent gcloud values are ALLOW_UNENCRYPTED_AND_ENCRYPTED, which is the default that allows both types, and ENCRYPTED_ONLY, which only allows SSL/TLS connections. Terraform uses the same idea with ssl_mode = "ENCRYPTED_ONLY" inside the ip_configuration block.
Exam tip: Enforcing SSL/TLS at the same time as using the Cloud SQL Auth Proxy or Cloud SQL Connectors with SQL Server can cause double encryption, which can negatively impact performance.
Database authentication for AlloyDB determines whether a client proves its identity with a stored password or with a Google identity. Built-in database authentication uses a username and password with authorization controlled by GRANT and REVOKE statements. IAM database authentication uses IAM principals such as users and service accounts, and AlloyDB Language Connectors can automate IAM authentication with short-lived access tokens.
| Method | Credential used | Management tradeoff |
|---|---|---|
| Built-in authentication | Database username and password | Requires separate password storage, rotation, and GRANT or REVOKE management |
| IAM authentication | Existing Google Account or service account with short-lived tokens | Centralizes access control in IAM and removes separate database passwords |
IAM authentication provides unified access control across Google Cloud resources, reduced credential management because users reuse Google credentials, and short-lived tokens that reduce the risk from leaked passwords.
Private connectivity for AlloyDB keeps traffic from the application VPC to the database off public networks. Applications can connect through the AlloyDB Auth Proxy or an AlloyDB Language Connector to create a secure TCP tunnel to the instance. Serverless VPC Access connects AlloyDB directly with Cloud Run, while Private Service Connect or private services access connects an application in another VPC using the AlloyDB private IP address. Private Service Connect must be configured when the database cluster is created if that path will be used. Additional perimeter controls include VPC Service Controls for context-based perimeter security, Shared VPC for centralized network management, and Cloud VPN or Cloud Interconnect for secure links between a corporate network and Google Cloud.
Programmatic access and identity management for cloud services is how an application connects to Firestore in Datastore mode and Cloud Storage with the correct identity and only the permissions it needs. Access is initiated through client libraries with Application Default Credentials (ADC), the Google Cloud console, the Google Cloud CLI, or the Datastore Admin API, while Identity and Access Management (IAM) controls which user account, service account, or service agent can perform each action. Large-scale transfers between Firestore in Datastore mode and Cloud Storage depend on both identities working together: the caller starts the operation and the Firestore service agent reads or writes the bucket objects.
Access interfaces for Firestore in Datastore mode are the programmatic paths an application or operator uses to read, write, export, and import data. The managed export and import service (opens in a new tab) is available through the Google Cloud console, the Google Cloud CLI, and the Datastore Admin API over REST and RPC, so the same operation can be started interactively or from code and automation. Day-to-day data access uses Datastore mode client libraries and the Datastore Data API over REST and RPC, with GQL for queries.
gcloud): scripted exports, imports, and project configuration from Cloud Shell or an installed CLI.The export output flows from Firestore in Datastore mode (opens in a new tab) to a Cloud Storage bucket in LevelDB log format, and an import flow reverses that path from bucket objects back into the database. Exports use eventually consistent reads, do not contain indexes, and may include entities written after the export starts while missing some written before it starts. Imports reuse the entity IDs from the export, overwrite any existing entity with the same ID, reserve those IDs while the import runs to prevent collisions, and leave unaffected entities in place.
Identities that authorize datastore operations are split between the caller and the Firestore service agent, and both must have permission for the transfer to succeed. The caller is the user account or service account that starts the export or import, and it must hold datastore.databases.export to export or datastore.databases.import to import. The Firestore service agent performs the Cloud Storage work on behalf of Firestore in Datastore mode and uses the name service-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com.
| Identity | What it authorizes | Access it needs |
|---|---|---|
| Caller user account or service account | Starting an export or import through the console, gcloud, or Admin API | datastore.databases.export for export, datastore.databases.import for import |
| Firestore service agent | Reading and writing export files in the Cloud Storage bucket during the operation | Access to the destination bucket, including cross-project buckets |
The Firestore service agent sits on the trust boundary between the database control plane and object storage: the caller never writes bucket files directly during a managed transfer, the service agent does. Older databases may still use the App Engine default service account for this boundary, and migration to the service-specific Firestore service agent is recommended.
Exam tip: Use the service-specific Firestore service agent, not the App Engine default service account, when VPC Service Controls must protect import and export operations.
Least-privilege roles and setup order enforce that each identity receives only the permissions needed for its part of the connection. A project owner already has the required export and import permissions, while other callers should receive Datastore Import Export Admin, Datastore Owner, or a custom role containing only datastore.databases.export or only datastore.databases.import. For Firestore with MongoDB compatibility operations (opens in a new tab), the caller also needs Cloud Storage access such as Owner or Storage Admin, which keeps database permissions separate from bucket permissions.
Preparing a reliable connection follows an ordered setup where each step depends on the previous one:
Operational limits shape large-scale transfers: a project allows a maximum of 50 concurrent exports and imports, 20 export and import requests per minute, and 100 entity-filter combinations per request. The export metadata file must keep its parent-folder name in the path gs://BUCKET_NAME/OPTIONAL_NAMESPACE_PATH/PARENT_FOLDER_NAME/PARENT_FOLDER_NAME.overall_export_metadata, and moving or copying output must preserve the folder name, subfolder contents, and metadata filename.
Protection of data during access and transfer combines automatic server-side encryption with Transport Layer Security (TLS) on the network and optional client-side encryption. Cloud Storage and Firestore in Datastore mode automatically encrypt data with AES-256, in most cases using Galois/Counter Mode (GCM), before it is written to disk, with no setup and no change to how the service is accessed. Data is automatically decrypted when read by an authorized user, and TLS protects reads and writes as they travel over the Internet.
Key ownership determines who controls the at-rest keys on each side of the boundary. By default Google manages the keys with hardened key management, strict access controls, auditing, and regular rotation of keystore master keys. Customer-managed encryption keys (CMEK) (opens in a new tab) let the customer manage Datastore keys instead, while client-side encryption encrypts data before writing so it is encrypted twice — once with customer keys and once with server-side keys. Firestore in Datastore mode cannot detect prior client-side encryption and has no knowledge of client-side keys, so the application must securely manage those keys itself.
cloudsql.instances.connect permission, must run continuously as a system service, and issues 2 API calls per hour per connected instance while running.ENCRYPTED_ONLY mode allows only SSL/TLS connections, while ALLOW_UNENCRYPTED_AND_ENCRYPTED allows both encrypted and unencrypted traffic.Use Identity and Access Management for bucket and project level access in Cloud Storage because it is recommended in most cases for granting permissions at the bucket and project levels. Granular control extends to buckets and managed folders through IAM roles mapped to a user, group, or service account, IAM policies with conditions, and principal access boundary policies that limit what a principal can access.
Choose a relational database when transactions and strong consistency are required, choose a non-relational database when the data model can be eventually consistent and needs higher availability and easier scalability, and use both strategically when different services have different needs. Spanner is a strongly consistent, highly available, and globally distributed database with support for SQL that combines relational querying with global scale.
The proxy service account must have a role that includes the cloudsql.instances.connect permission to open connections to the instance. The predefined Cloud SQL Client, Cloud SQL Editor, and Cloud SQL Admin roles include this permission, while legacy project roles require at least the Editor role.
Both the caller and the Firestore service agent must have permission for the transfer to succeed. The caller user account or service account that starts the operation must hold datastore.databases.export to export or datastore.databases.import to import, while the Firestore service agent performs the Cloud Storage work and must have access to the destination bucket, including cross-project buckets.
Last updated