Prepare and test your skills
Prepare and test your skills
Cloud Storage (opens in a new tab) lifecycle management rules are automatic bucket rules that move or delete objects after a set period. BigQuery (opens in a new tab) uses Cloud Storage as its staging and external storage layer, so files left in buckets keep adding storage cost. A lifecycle rule applies to the whole bucket and acts on current and future objects without manual cleanup. This automation lowers cost and enforces retention because temporary files, exports, and old staging data are removed on schedule.
Lifecycle rules in Cloud Storage control how long objects remain in a bucket and which storage class they use over time. The Cloud Storage service watches object age and other object properties, then moves the object to a lower-cost class or deletes the object when the condition is met. Because the service evaluates the rule continuously, cost control and compliance do not depend on a person remembering to delete files. Lifecycle rules are best when data has a known useful lifetime, such as temporary load files or backups kept for a fixed period.
Cloud Storage is the staging layer that feeds BigQuery, and data flows from the source environment to a Cloud Storage bucket and then from the bucket into BigQuery:
A process flow showing staged files uploaded to a Cloud Storage bucket, loaded into BigQuery, then automatically deleted by the bucket's Object Lifecycle Management rule once the age condition is met, reducing storage cost.
Cloud Storage commonly holds three kinds of data in these workflows:
Because external tables and exports depend on the bucket, the bucket lifecycle rule directly affects whether BigQuery can still read that data.
Safe automatic cleanup in Cloud Storage depends on bucket location, permissions, and the separation between storage and query access. The bucket should be created in the location closest to the data source, and when possible in the same location as the BigQuery dataset, so latency is low and cross-region reading costs are avoided. The identity that runs transfers and cleanup needs storage.objects.get to read objects, storage.objects.list to list objects with wildcards, and storage.objects.delete to delete source files after a successful transfer. The predefined storage.objectAdmin role includes all of these permissions.
Data lake administrators create connections and control bucket access, data warehouse administrators create tables and access controls, and data analysts query through BigQuery without direct read access to bucket objects. Analysts therefore cannot bypass table controls by reading bucket files directly.
Exam tip: A transfer can delete source files from Cloud Storage after success only when the identity also has storage.objects.delete; without that permission, files remain and continue to incur storage costs.
Granular expiration and versioning rules in BigQuery (opens in a new tab) and Cloud Storage (opens in a new tab) remove targeted data subsets instead of deleting whole tables or buckets. BigQuery provides granularity through table expiration times and partition-level deletion, with the dataset controlling retention. Cloud Storage provides granularity by distinguishing live object versions from archived versions based on age and version history.
BigQuery table expiration is metadata stored with each table and configurable when the table is created. That metadata also includes the table schema and the partitioning and clustering specifications, so expiration can align with how data is divided. The data retention policy for a table is determined by the configuration of the dataset that contains the table. Partition-level expiration removes only the targeted subset, which lowers storage cost without affecting recent partitions.
When a specific partition is deleted from a partitioned table, data from that partition is retained for recovery while other partitions are not affected. When a table is deleted or fully replaced, such as with WRITE_TRUNCATE or CREATE OR REPLACE TABLE, the previous contents are retained in the same way. This behavior allows recovery of recently replaced data without keeping the entire table history.
Cloud Storage object versioning keeps a live version and older archived versions of the same object. BigQuery load jobs and the BigQuery Data Transfer Service work only with live objects, so version history directly affects what BigQuery can read. Archived objects are not supported for transfers, and including a generation number in the Cloud Storage URI causes the load job to fail.
Metadata caching for BigLake and object tables also follows the live version boundary. When the cache is populated or refreshed, BigQuery captures metadata based on the live version of the objects at that time. Queries that use the cache then read data for that cached version, even if a newer version becomes live afterward, and accessing the newer version requires a metadata cache refresh.
Exam tip: BigQuery reads only the live Cloud Storage version; use an archived version or a generation number in the URI and the transfer or load fails.
Time travel and fail-safe provide two ordered retention stages after granular deletion in BigQuery. Data that is replaced or removed first enters the time travel window, where it remains queryable and recoverable, and then enters the fail-safe period for emergency recovery only. The time travel window can be configured to balance storage cost with retention needs, while the fail-safe period is a fixed seven days and cannot be configured or extended.
Data in fail-safe storage cannot be queried or recovered directly. Recovery from fail-safe storage requires contacting Cloud Customer Care, and recovery is performed at the table level from the point in time when the table was deleted. Once the fail-safe period has passed, deleted data cannot be recovered.
Billing for these stages depends on the dataset storage billing model. The dataset defaults to logical billing, and changing the model takes 24 hours to take effect:
Under logical storage billing, active and long-term storage are billed in the base rate while time travel and fail-safe storage are included and not billed separately. When a table is created there is no time travel or fail-safe cost, but changed or deleted data saved by those windows is billed if storage fees do not apply elsewhere.
Lifecycle management policies are automated rules that delete data after a set period to lower storage costs and enforce retention requirements. Cloud Storage (opens in a new tab) uses Object Lifecycle Management to delete objects based on conditions, while BigQuery (opens in a new tab) uses table and partition expiration to remove row-level data that is no longer needed. The policy applies to the container, either the bucket or the table, and the service performs deletion without manual cleanup jobs.
Deletion methods in Cloud Storage fall into two categories chosen by object count and workflow need. Single-object deletion deletes individual objects for targeted, immediate updates, while bulk object deletion deletes many objects at once or on an automated schedule for cost and hygiene at scale.
| Method | Scope | Choose when |
|---|---|---|
| Single-object deletion | One object per DELETE request | Correcting an error, cleaning up a temporary file, handling an application event such as a deleted profile photo, or removing sensitive data |
| Bulk object deletion | Many objects with higher throughput | Removing old logs, enforcing a 30-day retention period, clearing a bucket for migration, or cleaning up pipeline intermediates |
Traffic flows from the chosen tool to the bucket, and the bucket depends on the deletion request to change object state from stored to deleted. Manual tools require a person or application to start each deletion, while lifecycle policies let the Cloud Storage service evaluate and delete without further action.
Manual deletion tools in Cloud Storage give direct control over which objects are removed and when deletion happens. Single-object tools each send an individual DELETE request per object, which is precise but inefficient for large datasets.
Single-object deletion can be performed with:
gcloud storage rm for command-line deletionManual bulk deletion is designed for higher throughput with reduced overhead. The Google Cloud console deletes up to one million objects interactively and processes the deletion in the background, with status shown under the Notifications button in the console header. The gcloud storage rm --recursive command deletes all objects that share a common prefix and suits small-to-medium bulk deletions. The multi-object delete XML API deletes up to 1,000 objects programmatically in a single POST request and is interoperable with Amazon S3 tools such as the S3 CLI and Boto3. Storage batch operations deletes millions or billions of objects asynchronously with built-in tracking for migration or decommissioning.
Exam tip: Choose Object Lifecycle Management when objects must be deleted automatically based on age or storage class, not the console or gcloud storage rm which require a person to start each deletion.
Object Lifecycle Management is the Cloud Storage feature that automatically deletes objects when they match configured conditions. The policy applies to the bucket, and the Cloud Storage service continuously evaluates objects in that bucket against conditions such as age or storage class. When an object meets a condition, the service performs the configured deletion without a manual request. Because the rule lives on the bucket, all current and future objects in the bucket are subject to the same evaluation.
This automation supports cost optimization and compliance across buckets. Common uses are removing temporary build artifacts and old logs that are no longer needed, automatically enforcing a retention period such as 30 days, and cleaning up intermediate processing objects after a pipeline produces a final report. Data hygiene stays consistent as new data arrives because every new object is evaluated under the same bucket rule.
Object Versioning is the Cloud Storage feature that retains multiple versions of an object in the same bucket instead of overwriting history. When versioning is enabled, a bucket contains live versions that are current and noncurrent versions that have been replaced but retained. Lifecycle conditions must distinguish between these states so the policy does not delete active data by mistake.
For versioned buckets, configure deletion rules with the isLive condition set to False so the rule applies only to noncurrent versions. This boundary keeps the live object on one side and expired history on the other, allowing teams to keep current data while automatically removing old versions to control cost. Related protections sit outside lifecycle management but affect whether deletion can proceed: soft delete protects against accidental deletion, while Object Retention Lock and Bucket Lock enforce retention policies that block deletion until retention requirements are met.
Table and partition expiration in BigQuery is the automated deletion mechanism for structured data that loses value over time. Expiration is set on a table or on individual partitions, and BigQuery removes the data after the specified period without manual cleanup jobs. Data therefore flows from active to expired to deleted based on time. This approach removes row-level data that is no longer needed while keeping the table structure and recent partitions intact.
This approach is often paired with aggregation to preserve long-term value at lower cost. When detailed rows are referenced less over the long term, teams build aggregation queries to compute and store metrics from that data, then let table or partition expiration systematically remove the row-level data. Storage charges fall because only the smaller aggregated results remain, while metrics stay available for long-term consumption.
storage.objects.delete, which is included in the storage.objectAdmin role.isLive condition to False so only noncurrent versions are deleted and live data is preserved.Cloud Storage lifecycle management rules apply to the whole bucket and automatically move objects to a lower-cost storage class or delete them when age or storage-class conditions are met. Because BigQuery uses Cloud Storage as its staging and external storage layer, the rules remove temporary load files, exports, and old staging data on schedule without manual cleanup.
Choose Object Lifecycle Management in Cloud Storage when objects must be deleted automatically based on age or storage class. Manual tools such as the Google Cloud console require a person to start each deletion, while the Cloud Storage service continuously evaluates bucket objects against lifecycle conditions and deletes matching objects without further action.
BigQuery table expiration and partition expiration automatically remove data after a specified period without manual cleanup jobs. Partition-level expiration removes only the targeted subset, which lowers storage cost without affecting recent partitions while keeping the table structure intact.
Replaced or removed BigQuery data first enters the time travel window where it remains queryable and recoverable, then enters the fail-safe period for emergency recovery only. Fail-safe retention is a fixed seven days that cannot be configured or extended, cannot be queried or recovered directly, and requires contacting Cloud Customer Care for table-level recovery from the point in time when the table was deleted.
A media analytics organization stores daily application log files in a Google Cloud Storage bucket. Due to data compliance and cost-optimization policies, the organization requires log files to be automatically deleted once they reach 30 days of age, without requiring manual intervention or custom maintenance scripts.
Which Cloud Storage feature should you configure to satisfy this requirement?
Last updated