Prepare and test your skills
Prepare and test your skills
Selecting the appropriate storage system means matching data volume, access pattern, and throughput and latency targets to the service that can sustain them. Bigtable handles massive durable datasets at high throughput, Memorystore provides low-latency in-memory caching, block and file systems serve attached and shared files, and Cloud Storage (opens in a new tab) provides virtually unlimited object storage with worldwide access. The choice also depends on relational needs, request-rate limits, and how data must scale from gigabytes to petabytes.
Bigtable is a fully managed, high-performance database service that implements a multidimensional key-value store for large analytical and operational workloads. A Bigtable instance can support tens of thousands of queries per second with consistent single-digit millisecond latency for reads and writes, and storage scales to petabytes while tolerating node failure. Bigtable instances are zonal resources, and replication can be configured across zones in a region or across regions, with each cluster holding an independent copy and automatic failover if one zone fails.
Memorystore is the fully managed in-memory storage service for low-latency caching and scratch space in front of a primary database. An application sends frequent reads to Memorystore instead of the persistent store, so repeated reads are served from memory and load on the database drops. Memorystore supports Redis, Memcached, and Valkey backends with different availability behavior: Redis Standard tiers replicate automatically across zones, Valkey offers zonal and multi-zonal configurations chosen at creation, and Memcached does not replicate across zones or regions so nodes should be auto-distributed across zones.
| Option | Best fit | Volume and performance |
|---|---|---|
| Bigtable | Durable store for massive data with frequent writes and reads | Petabytes, tens of thousands of queries per second, single-digit millisecond latency |
| Memorystore | In-memory cache or scratch space for frequent reads | Memory-sized working set, low-latency reads, availability depends on backend configuration |
Exam tip: Memorystore for Redis in Standard tiers automatically replicates across zones, while Memorystore for Memcached does not replicate across zones or regions.
Persistent Disk (PD) is POSIX-compliant block storage for Compute Engine with a maximum of 64 TB per disk and up to 128 disks attached to one instance. A Persistent Disk can attach in read-write mode to only one instance at a time, but the same disk can attach in read-only mode to hundreds or thousands of instances such as render workers. Persistent Disk can be accessed only by resources in the same region, so compute and disks must stay together, with Standard drives for sequential work and solid-state drives (SSDs) for higher random performance.
Cloud Storage is highly redundant object storage that keeps files in buckets and is accessible worldwide through the gcloud CLI or API. Cloud Storage has virtually unlimited capacity and one API across storage classes, but it is not POSIX-compliant and cannot be mounted as a logical volume, so pipeline data must be copied locally before use. Each bucket starts with about 1000 object write requests per second and about 5000 object read requests per second, and Cloud Storage autoscales by redistributing load across servers over minutes, so spikes above those rates cause higher latency and errors without gradual ramp-up.
Filestore, PixStor, Google Cloud NetApp Volumes, and Cloud Storage FUSE bridge block and object storage when thousands of clients need shared files. Filestore is a clustered file system that supports thousands of simultaneous Network File System (NFS) connections and can sync with on-premises network-attached storage (NAS) without selective or bidirectional sync. PixStor is a scale-out file system that supports thousands of NFS or POSIX clients with on-demand caching and automatic write-back, NetApp Volumes supports NFS, Server Message Block (SMB), and multiprotocol access with snapshots but is not in every region, and Cloud Storage FUSE mounts buckets cheaply but is not POSIX-compliant.
Relational database services store data in tables with rows, columns, and a fixed schema, while non-relational services store flexible documents or entities. Cloud SQL is a fully managed relational service offering MySQL, PostgreSQL, and SQL Server engines for online transaction processing (OLTP), business intelligence, and consistent relationships without duplicated data. Cloud SQL protects data with automated backups and binary logging for point-in-time recovery, supports high-availability (HA) configuration and cross-region replicas, and its Enterprise Plus edition adds a 99.99% availability SLA, optimized hardware and caching, and 35 days of log retention.
Cloud Spanner is a SQL-based relational service using GoogleSQL with Data Definition Language (DDL), Data Query Language (DQL), Data Manipulation Language (DML), Data Access Control, and Transaction Control statements. Spanner exports full databases to Avro files in Cloud Storage with Dataflow templates and re-imports them, or moves data selectively with the Dataflow connector. Firestore in Datastore mode (Datastore) is a schemaless NoSQL document database where entities of the same kind can have different properties, with atomic transactions, high availability, automatic scaling, strong consistency, index-based queries without joins, and encryption at rest.
Throughput, input/output operations per second (IOPS), and latency together decide whether storage can keep up with a workload. IOPS matters most for many small requests, throughput matters most when large objects saturate bandwidth, and distance matters because storing data in the same zone as compute reduces time, transfer cost, and I/O between layers. For Compute Engine, zonal Persistent Disk keeps data in one zone while regional Persistent Disk synchronously replicates writes across two zones for failover, and Google Cloud Hyperdisk improves on Persistent Disk with Hyperdisk Balanced provisioning IOPS and throughput separately.
Cloud Storage offers Standard for high-frequency access, Nearline for access at most once a month such as backups, Coldline for lower-frequency access, and Archive for lowest-frequency access. Nearline, Coldline, and Archive lower storage cost but add retrieval costs and minimum-duration charges, so active data placed there costs more. Location also controls survival: regional buckets replicate synchronously across zones, dual-region and multi-region buckets store data redundantly in at least two locations with synchronous metadata and asynchronous data replication, turbo replication gives dual-region buckets a 15-minute recovery point objective (RPO), and Filestore regional instances replicate synchronously across three zones and can replicate to another region.
Performance characteristics of object storage describe how storage class, request rate and distribution, and data volume and location together shape throughput and latency. Standard and Nearline fit frequent access while Coldline and Archive fit infrequent access with lower storage cost but higher operation cost, so frequent Class A or B operations on Coldline or Archive data mean the data belongs in Standard or Nearline. Storage Intelligence advisor tracks Cumulative object count and Storage volume over time in Key charts, flags Total bytes stored increased above trend when versions accumulate, and flags Spike in cross-region egress when compute in one region such as us-central1 repeatedly reads a bucket in another region such as us-east1.
Request distribution matters as much as total rate because Cloud Storage maintains a lexicographically ordered key index per bucket that is updated on every write or delete. Sequential names or many writes under one prefix concentrate load on a narrow index range and cause hotspotting, while non-sequential names and spreading bursts across buckets distribute load. Between Dataflow and Cloud Storage, Apache Beam TextIO or AvroIO connectors read gs:// files in parallel across files and across sub-ranges of uncompressed files, writes gain parallelism from sharding, and gRPC with Apache Beam SDK 2.55.0 or later plus --additional-experiments=use_grpc_for_gcs speeds transfers.
Designing appropriate schemas means choosing keys, document layouts, and indexes that distribute load and match query patterns. Cloud Bigtable sorts rows by row key across nodes, Firestore in Native mode stores documents for direct lookup, and Cloud Spanner and AlloyDB for PostgreSQL use relational keys and indexes across distributed servers. Good schema design spreads writes, avoids hotspots, and lets indexes serve queries without scanning unrelated data.
Row key design in Cloud Bigtable controls how rows are sorted, split across nodes, and accessed. Each row has one row key, Bigtable stores rows in sorted key order so nearby keys stay together for fast range scans, and requests flow from applications through an app profile to one or more clusters. Single-cluster routing sends all requests to one cluster, multi-cluster routing sends each request to the nearest available cluster, and nodes can be added or removed without downtime with improvement visible within minutes.
Cluster sizing in Bigtable must cover both storage and compute for the chosen keys. For SSD storage, divide total unreplicated data size by 3.5 TB, which is 5 TB per node at 0.70 utilization, to estimate storage nodes. For compute, divide sustained reads and writes per second by 10,000 to estimate minimum SSD nodes, with more nodes for low-latency queries. The cluster needs the larger of the two estimates, and each cluster in a multi-cluster instance must store all instance data and handle the total insert rate.
Structured row keys in Bigtable are multi-column keys defined by an optional row key schema, such as user_id, purchase_date, and order_number joined with delimitedBytes encoding. Data is still stored as bytes, but GoogleSQL for Bigtable applies schema types during SQL queries, and field order sets sort order and scan adjacency. The schema can be viewed in Bigtable Studio in the Explorer pane or via gcloud CLI and AdminClient, and it is supplied as YAML or JSON at creation. Because the ReadRows API ignores the row key schema, structured row-key fields must be queried with SQL on the _key column plus MAP-type family columns, using temporal filters such as with_history for versions and CAST functions when types are unknown.
Firestore (opens in a new tab) in Native mode is a flexible, scalable NoSQL document database for mobile, web, and server development. Documents hold the fields an application reads together, so one read returns a complete unit without assembling many tables. Modeling starts from the queries the application will run, not from a normalized table layout, which lets the same database serve small interactive lookups and larger operational workloads.
Indexing in Firestore makes filtered or sorted queries fast by locating matching documents directly instead of scanning unrelated data. Query Insights identifies queries that can be optimized with an index, and Firestore Enterprise edition adds guidance for read and write scaling. Document lifecycle controls keep the active set lean: time-to-live (TTL) policies automatically expire unneeded documents, backups preserve contents for recovery, and point-in-time recovery (PITR) restores data to a prior time.
Exam tip: Use Query Insights to identify queries you can optimize with an index.
Relational schema optimization for Cloud Spanner and AlloyDB for PostgreSQL chooses keys, indexes, and table structures that distribute load and speed queries. Cloud Spanner sorts and distributes rows by primary key across many servers, so key choice controls write throughput and locality. AlloyDB for PostgreSQL is a fully managed PostgreSQL-compatible service for demanding hybrid transactional and analytical processing (HTAP) workloads, where live data can be queried directly through BigQuery federation without change data capture pipelines.
Primary key design in Cloud Spanner requires every multi-row table to have a one- or multi-column primary key, and sequential keys concentrate new writes on one server range and create hotspots. A primary key cannot be changed without deleting and recreating the table, so high-write tables should use non-sequential keys from the start. The default recommendation is UUID Version 4 with GENERATE_UUID() stored as STRING, while integer keys should use bit-reversed positive sequences that spread inserts across the 64-bit space.
| Key option | Value type | Distribution behavior |
|---|---|---|
UUID Version 4 with GENERATE_UUID() | STRING | Random values prevent concentration of new writes |
| Bit-reversed positive sequence | Integer | Reversed bits spread sequential inserts across the 64-bit space |
Secondary indexes in Cloud Spanner index frequently filtered columns to avoid full-table scans, similar to PostgreSQL b-tree indexes, and they replace PostgreSQL UNIQUE constraints such as a unique email column. Tuning follows an ordered workflow: load sample data, connect the application so traffic flows to Spanner, verify consistency, load test and use query insights to find expensive queries, then modify and retest. Suboptimal performance comes from inefficient queries, sustained high CPU, transaction locking found with transaction insights, inefficient schema design, and hotspots visible in Key Visualizer statistics.
Understanding replication consistency means knowing whether readers always see the latest write or may briefly see older data. Strong consistency locks data during update so all observers agree immediately, while eventual consistency keeps replicas available and converges to the last value if writes stop. The choice trades freshness and correctness against availability, latency, and throughput across zones and regions.
Applying Consistency, Availability, and Partition Tolerance (CAP) principles means selecting storage based on which guarantee matters most when the network separates locations. Consistency means every reader sees the latest write immediately, availability means every replica stays readable before replication finishes, and partition tolerance means the system keeps working across zones and regions despite delays. A distributed system cannot provide all three at once during a partition, so service selection trades fresh data against speed and reachability.
With eventual consistency, data flows from originating Node A to replica Nodes B and C without blocking reads, so replicas stay available but readers of B or C may briefly see stale data. With strong consistency, the system locks data during update and replication, so replicas stay consistent but are not accessible until the update finishes. The Internet Domain Name System (DNS) shows eventual consistency at scale, staying highly available for more than a hundred million lookups while changes take time to reach every client. Eventual consistency fits approximate views such as buddy-list presence, like counts, or very large result sets, while strong consistency fits billing completion, game points earned, or small result sets where users know exactly what should appear.
Bigtable (opens in a new tab) replication copies each write from the receiving cluster to every other cluster in the same instance. With multi-cluster routing the replication is eventually consistent, so reads on another cluster may return older data until replication completes, with longer lag across greater distance. Each cluster also spends CPU pulling changes from other clusters, so a cluster in a multi-cluster instance often needs more nodes than a single-cluster instance with similar traffic.
Routing is controlled by an app profile and determines read-your-writes behavior. Single-cluster routing sends all requests to one configured cluster, which can provide read-after-write semantics on that cluster and isolates workloads but does not reduce distance latency. Multi-cluster routing sends each request to the closest available cluster and then replicates writes, which minimizes latency but gives eventual consistency across clusters. Replication improves read throughput and geographic availability but does not increase write throughput, because every piece of data is written twice: a 6-node single cluster doubles write throughput over 3 nodes, while two 3-node clusters in two zones may handle only half the writes but survive zonal failure.
| Routing option | Where requests go | Consistency and latency effect |
|---|---|---|
| Single-cluster routing | All requests go to one configured cluster | Reads can see their own writes on that cluster, without closest-cluster latency reduction |
| Multi-cluster routing | Each request goes to closest cluster, writes replicate | Lowest latency from shortest distance, with eventual consistency across clusters |
Failover in Bigtable reroutes traffic from an unresponsive cluster to another cluster and reverts to eventual consistency with an internal last write wins conflict rule. Manual failover applies to single-cluster routing when a cluster shows many transient errors, timeouts, or unacceptable latency. Automatic failover applies to multi-cluster routing where Bigtable retries on the next closest cluster, with shorter deadlines triggering earlier rerouting, so clients should sit in or near more than one region for faster failover.
Exam tip: Adding a second Bigtable cluster improves availability and read locality, but it does not double write throughput the way adding the same nodes to a single cluster does.
Cloud Spanner is a distributed relational database that provides global strong transactional consistency using TrueTime timestamps applied in timestamp order. Blind writes without prior reads in the same transaction can proceed in parallel because TrueTime values differ, avoiding race conditions. A transaction spanning splits uses two-phase commit (2PC) while a single-split transaction bypasses 2PC with a faster protocol, and a stale read avoids the replica-freshness check that a strong read pays for. Range sharding speeds range scans, while a hash value in the primary key spreads writes when range scans are not needed.
Cloud SQL and AlloyDB (opens in a new tab) use a primary and secondary topology where the primary cluster accepts writes and secondary clusters and read replicas copy from the primary. AlloyDB cross-data-center replication places secondaries in different data centers so traffic flows from primary to each copy and survives primary outages. Asynchronous replicas behave like eventual consistency and can lag, so a write to the primary followed by an immediate replica read may show stale data, while synchronous replication waits for the replica and behaves more like strong consistency at higher latency.
Designing for lag means routing correctness-critical operations such as money movement, inventory decrement, and billing completion to the primary or a strong read, and routing tolerant high-volume reads such as dashboards and listings to replicas. Locking and waiting keeps every observer consistent but limits throughput on locked data, while serving reads during replication keeps the system available and scalable but requires handling stale reads, retries, and user experience when the latest write is not yet visible.
A V4 signed URL for Cloud Storage (opens in a new tab) is a complete XML API URL that embeds authentication in its query string. The URL is created by a service account that already has permission for the action, and it lets a person without Google credentials perform one specific action on one object for a limited time. Anyone holding the URL can use it while active, and Cloud Storage verifies the signature before serving the request.
A V4 signed URL consists of a resource URL plus six X-Goog-* query parameters that define who signed it, what must match, and how long it lasts. The resource part identifies the bucket and object, such as https://storage.googleapis.com/example-bucket/cat.jpeg, while the query string carries authorization and every parameter included at signing is covered by the signature. The parameters are X-Goog-Algorithm such as GOOG4-RSA-SHA256, X-Goog-Credential with client email, date, location, service, and request type, X-Goog-Date in YYYYMMDD'T'HHMMSS'Z' form such as 20181026T181309Z, X-Goog-Expires in seconds such as 900 for 15 minutes, X-Goog-SignedHeaders such as host or content-type;host, and X-Goog-Signature as the hex-encoded result.
The canonical request is the standardized description that gets hashed and signed, capturing the HTTP method, canonical URI from the escaped object name, ordered canonical headers with lowercased names, sorted order, and normalized whitespace, plus the canonical query string. Its SHA-256 hash goes into a four-line UTF-8 string-to-sign: signing algorithm, active datetime matching X-Goog-Date, credential scope as DATE/LOCATION/storage/goog4_request, and hashed canonical request. RSA signing signs directly with the service account private key, HMAC signing first derives the key through chained HMAC-SHA256 over date, location, service, and request type, and the final URL appends X-Goog-Signature with a query string matching the canonical request exactly.
The HTTP method binds the URL to one operation, so the sent method must match the signed method. GET downloads data such as cat.jpeg typically with host signed, while PUT uploads data such as cat.png typically with content-type;host signed. Parameters such as response-content-disposition can be included at signing to override metadata for an authenticated GET, and for resumable uploads a signed URL is generally needed only for the initial POST because later PUTs use the returned session URI over HTTPS.
The service account is the signing identity for a signed URL because its email appears in the credential and its key creates the signature. Cloud Storage verifies the signature against that account when the URL is used, so the URL carries that account's authority for one operation only. This delegation lets external users, mobile clients, or browsers without Google credentials read or write directly, and it ends at expiration without granting broader access.
A valid signed URL needs two Identity and Access Management (IAM) permissions working together. The caller creating the signature must be allowed to impersonate the service account, and the service account itself must be allowed to perform the Cloud Storage operation. Preparation follows an ordered sequence: create the service account if needed, grant it data access such as reading object data for a download URL, and provide a signing key in JSON or PKCS12 format when the algorithm is not built into Google Cloud. The human or workload signer needs the Service Account Token Creator role, while the service account needs the operation permission such as storage.objects.get for reads or storage.objects.create for uploads.
The Service Account Credentials API signs on behalf of a service account without exposing its private key. The caller invokes signBlob for blobs or signJwt for JSON Web Tokens via REST, gcloud beta iam service-accounts sign-jwt, or clients such as IamCredentialsClient.SignBlob(), and must hold iam.serviceAccounts.signBlob from the Token Creator role. A signBlob response returns signedBlob usable as a bearer token until the system-managed key expires plus the keyId used, and client library names must use projects/-/serviceAccounts/my-service-account@my-project.iam.gserviceaccount.com with - as the project wildcard.
Programmatic generation with client libraries is the application path for per-user, per-object URLs inside code. The application authenticates with Application Default Credentials from the environment or default App Engine service account, then signs a URL for a specific bucket and object. Data flows from the signing service account to the generated URL to the end user, who presents the URL directly to Cloud Storage without Google authentication. Libraries span C++, C#, Go, Java, Node.js, PHP, Python, Ruby, and Rust, such as Python storage.Client() and Ruby Google::Cloud::Storage.new, and code controls object name, method, expiration, and signed headers for least privilege.
Command-line generation with gcloud storage cp and gsutil-style workflows is the operator path for testing, debugging, and one-off sharing without code. Traffic flows from the operator workstation to Cloud Storage, and the URL or object can be handed to another person or piped to a process. Code gives repeatable policy enforcement for many URLs, while the command line gives speed for administration. Expiration is controlled by X-Goog-Date as start plus X-Goog-Expires as lifetime, with common values 600 seconds, 900 seconds, and 3600 seconds, usable from 15 minutes before the active time until expiration, and access ends at expiration or key rotation.
Access control boundaries mean the URL does not replace bucket policy: the signer must already be authorized. Reading needs storage.objects.get from Storage Object Viewer (roles/storage.objectViewer), console browsing also needs storage.buckets.list and storage.objects.list found in Storage Admin (roles/storage.admin), and persistent keys should be avoided for Application Default Credentials, impersonation, or workload identity federation. Uniform bucket-level access is preferred over access control lists (ACLs) because ACLs apply per bucket or object, are limited to 100 entries, and can bypass domain restrictions.
Exam tip: The longest X-Goog-Expires value for a Cloud Storage V4 signed URL is 604800 seconds (7 days); longer requested lifetimes are rejected.
Continue with this study guide: Creating signed URLs to grant access to Cloud Storage objects
Writing data to BigQuery (opens in a new tab) for analytics and AI/ML means shaping files, tables, and ingestion paths so queries scan less data and results stay fresh. BigQuery charges for bytes scanned and uses columnar storage with persisted statistics, so well-structured writes lower cost and speed training and dashboards. Batch loading favors fewer large jobs, streaming favors continuous availability, and the BigQuery Storage Write API (gRPC) unifies both with high throughput and exactly-once options.
Source file format and layout control how fast Cloud Storage data can be loaded and queried. Columnar formats such as Apache Parquet and Apache ORC generally perform much better for analytic queries than row-based JSON or CSV, and splittable formats such as Avro are usually faster than CSV with embedded newlines or unsplittable compressed files. A Hive partitioned layout organizes files by partitioning column values so BigQuery prunes partitions, while too many tiny files or very large files hurt performance.
Partitioning in BigQuery divides a table by date, ingestion time, or time-unit column so filtered queries read only relevant segments. Rows streamed with the Storage Write API first land in the __UNPARTITIONED__ partition and are later repartitioned with no service level agreement (SLA) on timing, so queries needing only partitioned data should filter with _PARTITIONTIME IS NOT NULL or _PARTITIONDATE IS NOT NULL. For ingestion-time tables the destination partition is inferred from current system UTC time but can be overridden with a decorator such as table1$20250601, and partitioned-table modification limits replace standard daily limits for append or overwrite jobs.
BigQuery-native storage enables optimizations not possible with BigLake tables that reference external files. If BigLake queries miss SLA and cannot be tuned, converting to BigQuery storage and repointing authorized views generally causes no disruption. Consumers can then use materialized views with different partitioning and clustering and BigQuery BI Engine for dashboards. Freshness can be tuned with max_staleness on the table or @@max_staleness_override such as INTERVAL 10 MINUTE to serve recent snapshots when real-time freshness is not required.
Managing schema design and evolution in BigQuery means shaping tables for fast analytics while staying compatible as sources change. Migrating tables as-is works for early iterations without rebuilding indexes or downtime, but keeping the old warehouse model leaves technical debt and misses nested fields, partitioning, and clustering. Partitioning divides data into partitions by TIMESTAMP, DATE, or ingestion pseudocolumns without changing table structure, so smaller partitions mean fewer bytes read.
Clustering in BigQuery automatically sorts data by a few specified columns and colocates related rows in optimally sized blocks. BigQuery uses no indexes and relies on massively parallel architecture that adds machines to scan and aggregate concurrently, so clustered columns let it skip blocks, estimate cost better, and aggregate faster. Columns frequently used for filtering should become clustering columns. Denormalization with nested and repeated fields is the native BigQuery representation using the Dremel model and Capacitor columnar format, reducing joins compared to star schemas with denormalized facts and dimensions or snowflake schemas with normalized dimensions.
| Schema model | Structure | Relationship to BigQuery |
|---|---|---|
| Star schema | Fact table with metrics plus keys to dimension tables | Supported, but not native |
| Snowflake schema | Like star, with normalized dimension tables | Supported, but not native |
| Nested and repeated fields | Related records stored inside parent rows | Native representation, fewer joins required |
Schema evolution is managed by exposing data through authorized views, functions, and datasets instead of direct base-table access, so producers can optimize tables without breaking consumers. Dataset versioning keeps view names and versions the dataset such as sales_v1 to sales_v2, view versioning keeps the dataset and versions views such as catalog_v1 to catalog_v2, and hybrid versioning allows compatible changes in one dataset but requires a new dataset for incompatible changes. For machine learning, Agent Platform Feature Store reuses tabular features, unstructured data stays in Cloud Storage governed by object tables, and curated datasets are shared with BigQuery sharing.
Selecting ingestion means matching arrival pattern to queryability needs, with Dataflow (opens in a new tab) Managed I/O choosing automatically or BigQueryIO allowing tuning. Batch loading accumulates data and loads it with fewer larger jobs, commonly writing staging files to Cloud Storage then running a load job, waiting for thousands of rows and using wildcards to load many files as one table operation and avoid quotaExceeded errors. Streaming ingestion writes records directly as events arrive, commonly sending endpoint logs to Pub/Sub where a streaming Dataflow pipeline streams to BigQuery for ad hoc queries and near-real-time dashboards, with Pub/Sub retention buffering outages for low recovery point objective.
| Method | How data flows to BigQuery | Delivery and use |
|---|---|---|
| STORAGE_WRITE_API | Direct from Dataflow workers to BigQuery storage over gRPC | Exactly-once; choose when duplicates are not acceptable |
| STORAGE_API_AT_LEAST_ONCE | Direct from Dataflow workers to BigQuery storage over gRPC | At-least-once with lower latency; duplicates possible |
| FILE_LOADS | Workers to Cloud Storage staging files, then load job to table | Batch default for large bounded loads |
| STREAMING_INSERTS | Workers through legacy streaming API to table | Legacy default for unbounded data; avoid for new projects |
The Storage Write API organizes writes around a stream where multiple streams can write concurrently to one table. The default stream needs no creation, serves data immediately, and gives at-least-once semantics. Application-created streams add control: pending type buffers records until commit for atomic batch availability via CreateWriteStream, AppendRows in a loop, FinalizeWriteStream, then BatchCommitWriteStreams, while committed type makes records readable immediately for streaming. Exactly-once delivery requires a committed-type stream where the client supplies a record offset with each append, so BigQuery writes only if the offset matches the next offset; omitting the offset appends to the end and retries can duplicate. The gRPC form is recommended with binary protocol buffer or Arrow formats, lower cost with up to 2 TiB free per month, stream-level and cross-stream transactions, and schema-change notification, requiring bigquery.tables.updateData such as bigquery.dataEditor.
Exam tip: Exactly-once delivery requires a committed-type stream with client-supplied offsets; omitting the offset appends to the current end of the stream, so retrying a failed append can create a duplicate.
Continue with this study guide: Writing data to BigQuery for analytics and AI/ML workloads
Bigtable is the durable store for massive data with frequent writes and reads at petabyte scale, tens of thousands of queries per second and single-digit millisecond latency, while Memorystore is an in-memory cache or scratch space in front of a primary database for frequent reads from a memory-sized working set. Repeated reads are sent to Memorystore instead of the persistent store so load on the database drops.
Persistent Disk provides a maximum of 64 TB per disk and up to 128 disks attached to one Compute Engine instance. A Persistent Disk can attach in read-write mode to only one instance at a time but in read-only mode to hundreds or thousands of instances, and it can be accessed only by resources in the same region.
High-write tables should use UUID Version 4 with GENERATE_UUID() stored as STRING or bit-reversed positive sequences instead of sequential keys because sequential keys concentrate new writes on one server range. A primary key cannot be changed without deleting and recreating the table, so non-sequential keys should be chosen from the start.
A V4 signed URL binds the HTTP method, resource, signed headers, credential scope and expiration into X-Goog-Signature and lets a person without Google credentials perform one specific action on one object for a limited time. The maximum X-Goog-Expires value is 604800 seconds (7 days) and longer requested lifetimes are rejected.
Last updated