Data Lifecycle Management with OLM and BigQuery

Multi-storage tiering and cost optimization is the practice of placing each dataset in the lowest-cost storage option that still meets its needs for access speed, latency, residency, and durability. Hot data that is read often stays in high-performance storage, while cooler data moves to cheaper infrequent-access classes. Total cost includes storage price plus retrieval fees and minimum-duration charges, so frequent reads favor expensive storage and rare reads favor cheap storage with retrieval penalties.

1.1Storage Classes for Access Frequency and Retrieval Cost

Cloud Storage (opens in a new tab) classes separate objects by how often they are accessed and how long they must be kept. Standard storage is for high-frequency, latency-sensitive reads with no infrequent-access penalty. Nearline, Coldline, and Archive storage lower the per-byte storage price but add fees for retrieving data or metadata and charge for minimum storage durations even if an object is deleted early. Data flows from the application to the bucket in the selected class, and every later read or early deletion triggers an additional fee from the infrequent-access class back to the requester.

The four classes trade storage price against retrieval cost in a fixed order:

Choose Standard storage when frequent reads and operational latency dominate cost, and choose Nearline, Coldline, or Archive storage only when storage-price savings outweigh higher retrieval costs and early-delete risk.

1.2Bucket Location Types, Replication, and Residency

Bucket location type controls where Cloud Storage stores and replicates object data and how much availability that replication provides. Regional buckets replicate data synchronously across multiple zones within one region, so a single zone outage does not lose data. Dual-region and multi-region buckets store data redundantly in at least two separate geographic locations, where metadata is written synchronously across regions and object data is replicated asynchronously. Dual-region buckets can use turbo replication, which replicates objects across the region pair with a recovery point objective (RPO) of 15 minutes. Cloud Storage stores or replicates data within the specified bucket region to support data residency, so choose a regional bucket when single-region residency and lower cost matter most and choose a dual-region or multi-region bucket when higher cross-location availability justifies higher cost.

1.3Automated Lifecycle Management for Tier Transitions

Object Lifecycle Management is the bucket-level policy engine that automates tier transitions and deletion based on retention and access-frequency requirements. A lifecycle configuration on the bucket evaluates objects and applies the SetStorageClass action to move aging objects from Standard storage to Nearline, Coldline, or Archive storage without rewriting the application. The same configuration can later apply a deletion rule to remove objects whose business value has expired, so storage cost falls as access frequency falls. The lifecycle policy depends on the bucket, and the bucket depends on correct Identity and Access Management (IAM) permissions at the bucket and project levels. Lifecycle deletion rules should use an isLive condition of False so the rule applies only to noncurrent versions and does not delete live production objects by mistake.

1.4Retention and Deletion Protection for Tiered Data

Retention and deletion protection are the controls that keep automated tiering from violating compliance or destroying data that must be preserved. Tiered storage still carries the duty to retain personal information only as long as necessary and to prevent erasure during a required retention period. Locked retention policies, holds, and soft deletion sit between the lifecycle engine and the stored object to block premature deletes.

• Retention policy with Bucket Lock: The retention policy on the bucket prevents deletion or replacement until the retention period ends, and Bucket Lock makes that policy permanent for compliance.
• Soft Delete: Soft Delete on the bucket preserves deleted and overwritten objects for a specified period so accidental or malicious deletion can be recovered.
• Object holds: Object holds on specific objects prevent their deletion or update until the hold is explicitly released.
• Object versioning: Object versioning on the bucket preserves prior versions as noncurrent objects so system changes remain trackable and recoverable.

Use locked retention policies when law requires provable immutability, and use Soft Delete and holds when operational safety requires recovery from human error. Storage Transfer Service (opens in a new tab) moves data from Amazon S3, Azure Blob Storage, or on-premises sources into Cloud Storage, after which the destination bucket lifecycle, location, and retention controls determine long-term tiering cost and compliance.

BigQuery (opens in a new tab) expiration and tiered storage policies are automatic lifecycle controls that delete or move data when it is no longer needed for active use. Expiration can be set at the dataset, table, and partition levels, so governance teams can control costs and storage limits without manually dropping objects. For data that must be kept longer, the same lifecycle approach uses BigQuery snapshots inside BigQuery or data exports to Cloud Storage as lower-cost, longer-term tiers.

2.1Dataset-Level Default Expiration

Dataset-level default expiration is the fallback rule that applies to new tables created in a BigQuery dataset. The dataset setting defines default table expiration times, so every table created without its own expiration inherits the dataset lifetime and is automatically removed when that time passes. This level is owned by data governance and is useful for sandbox, staging, or temporary analysis datasets where inactive data should not accumulate active storage costs. Existing tables with their own explicit expiration keep their own setting rather than adopting a later change to the dataset default.

2.2Table-Level Expiration

Table-level expiration is an explicit lifetime set on an individual BigQuery table that controls when that table is automatically deleted. A table expiration is defined by the data owner and overrides the dataset default, which allows critical tables to be retained longer and temporary tables to be removed sooner. Updating a table expiration changes only that table and does not change the dataset or other tables in the dataset. Use table-level expiration when business requirements, regulatory retention, or access frequency differ from table to table within the same dataset.

2.3Partition-Level Expiration

Partition-level expiration is the default lifetime for partitions inside partitioned tables, managed as the default dataset partition expiration. The partition expiration setting controls how long individual partitions remain in active storage before they are automatically dropped, while the table definition itself remains. This level is designed for time-series and event data where recent partitions are queried often and older partitions become inactive. Use partition expiration to keep a table continuously available for new data while automatically trimming old partitions to control costs and stay within storage limits.

2.4Automated Backup and Lifecycle Enforcement at Scale

Scalable backup automation is the Cloud Run (opens in a new tab)-based framework that enforces table-level and fallback lifecycle policies across folders, projects, datasets, and tables. The scope of a run explicitly includes and excludes folders, projects, datasets, and tables, so one schedule can cover an organization while skipping exempt data. The framework supports four related objectives:

• disaster recovery objectives are met by recurrent automated copies of in-scope tables
• human errors are recoverable by restoring from snapshots or exported copies
• regulatory compliance is supported by consistent retention and auditable backup state
• operational efficiency is improved by replacing manual per-table backup work with scheduled automation

The automated run moves data through an ordered workflow where each service depends on the previous service:

1. Schedule trigger: Cloud Scheduler sends a Pub/Sub message to the dispatcher service on a cron schedule, and the message carries the included and excluded scope.
2. Discovery: The dispatcher service, built on Cloud Run, calls the BigQuery API to list the tables that fall within the scope.
3. Fan-out: The dispatcher service submits one Pub/Sub message per table to the configurator service.
4. Policy resolution: The Cloud Run configurator service computes the effective backup policy for each table from the table-level policy defined by data owners, or from the fallback policy defined by the data governance officer when no table-level policy exists.
5. Routing: The configurator service submits one request per table to the next service based on the computed backup policy.
6. Backup execution: Depending on the policy, a custom Cloud Run service calls the BigQuery API to create a BigQuery snapshot of the table or to run a data export of the table to Cloud Storage.
7. Completion detection: When the backup method is a data export, a Cloud Logging log sink listens for export job completion events to enable asynchronous execution of the next step.
8. Result handling: After backup services complete, Pub/Sub triggers the tagger service.
9. State recording: For each table, the tagger service logs the backup result and updates the backup state in the Cloud Storage metadata layer, which depends on Cloud Storage and Datastore for durable state.

2.5Backup Methods for Tiered Retention

Backup methods for tiered retention determine where an inactive copy lives and how it is restored. A BigQuery snapshot keeps the backup as a snapshot inside BigQuery, so the data remains in the BigQuery storage boundary and can be restored directly to BigQuery. A data export to Cloud Storage moves table data from BigQuery to a Cloud Storage bucket, so the data leaves the BigQuery active-storage boundary and lives in the Cloud Storage object-store boundary for long-term, low-cost retention and cross-regional recovery. The export path then depends on Cloud Logging completion events before the tagger service records success, while the snapshot path completes directly through the BigQuery API.

Cloud Storage Object Lifecycle Management (OLM) is bucket-level automation that transitions objects between storage classes or deletes them when defined conditions are met. A bucket holds a lifecycle configuration made of rules, and each rule pairs a condition with an action, so data moves through its life cycle without manual intervention. This automation aligns storage cost with access frequency while enforcing retention and deletion consistently for cost, availability, and compliance objectives.

3.1Object Lifecycle Management Configuration

Cloud Storage lifecycle configuration is the set of rules attached to a bucket that Cloud Storage evaluates against objects over time. Each rule contains a condition, such as object age or current storage class, and an action, such as changing storage class or deleting the object. When an object metadata satisfies the condition, Cloud Storage automatically applies the action to the object in that bucket. Best practice is to enable lifecycle management for Cloud Storage buckets rather than relying on manual cleanup, and to track changes with Cloud Asset Inventory and object versioning. Lifecycle rules implement two families of actions, where the SetStorageClass action changes the storage class of less-often accessed objects to reduce cost while keeping the data, and the Delete action removes objects automatically for cost optimization, data hygiene, and retention limits.

3.2Storage Class Transitions and Automated Deletion

Storage class transitions match storage cost to data-retention and access-frequency requirements through either manual selection or Autoclass. With a default storage class, the bucket owner chooses the initial class for new objects, and OLM rules then transition objects with a SetStorageClass action as they age. With Autoclass enabled, Cloud Storage automatically manages storage class transitions for the bucket data. Set permitted regions for storage classes and verify regional, dual-region, or multi-region deployment so the cost choice still meets residency and compliance requirements.

Automated deletion through OLM is bulk object deletion based on conditions, distinct from single-object deletion or interactive bulk deletion in the console. Choose the deletion tool by scale and selectivity:

• OLM deletion: Use when objects should expire by age or storage class without manual work.
• gcloud storage rm --recursive: Use for prefix-based cleanup of a group of objects under a path.
• Multi-object delete: Use for up to 1,000 objects in one programmatic request.
• storage batch operations (opens in a new tab): Use for millions or billions of objects with tracking at scale.

For versioned buckets, deletion rules should use the isLive condition set to False so the rule targets noncurrent versions, preserving the live version while expiring old generations.

3.3Retention Policies and Bucket Lock for Compliance

Bucket Lock is the Cloud Storage feature that enforces a bucket-level retention policy for regulatory compliance. The policy defines a retention period as a length of time, and every object in the bucket is protected from deletion or overwrite until that period expires. Once the retention policy is locked, it cannot be removed or shortened, which protects personal information from erasure during the specified retention period. Object Retention Lock applies retention at the individual object level within buckets that have object retention enabled, where an object retention configuration sets a mode and a retainUntilTime. A bucket retention policy must be unlocked to be removed, while a locked policy is permanent for the retention duration, and object-level retention allows different expiration times for different regulated records.

3.4Holds, Versioning, and Soft Delete

Object holds are per-object protections that prevent deletion independently of retention periods or lifecycle rules. A temporary hold blocks deletion until explicitly released and must be reviewed on active objects so data does not accumulate beyond its purpose. An event-based hold blocks deletion until an event releases it, and a bucket can set a default event-based hold so every new object starts on hold. Storage batch operations can set temporary or event-based holds at scale, and releasing the hold is what allows lifecycle deletion or application deletion to proceed. Object Versioning and soft delete provide recovery rather than compliance immutability, where Object Versioning retains multiple generations as live and noncurrent versions so overwrites create history, and soft delete retains deleted objects for a configured number of days for recovery from accidents. Enable versioning to track system changes, enable soft delete to recover from accidental deletion, and use Bucket Lock or holds when the requirement is to prevent deletion rather than to recover after it.

Key Facts

• Cloud Storage Standard is for high-frequency access, while Nearline, Coldline, and Archive lower storage price but add retrieval fees and minimum-duration charges.
• Dual-region buckets can use turbo replication with a recovery point objective (RPO) of 15 minutes across the region pair.
• BigQuery expiration can be set at dataset, table, and partition levels, where table expiration overrides the dataset default and partition expiration drops old partitions while keeping the table.
• A BigQuery snapshot keeps the backup inside BigQuery for direct restore, while a data export moves table data to Cloud Storage for long-term, low-cost retention.
• A lifecycle deletion rule with isLive = False targets only noncurrent object versions and preserves the live version.