StudyCerts
StudyCertsProfessional Cloud Data Engineer
professional-cloud-data-engineer
Advanced
Gauge your current knowledge
Advanced
Gauge your current knowledge
Lifecycle management of data
...
professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
Object Lifecycle Management (OLM) in Cloud Storage automates data management by moving objects between storage classes or deleting them based on rules you set. You create these rules at the bucket level to transition data from expensive classes like Standard Storage to cheaper ones like Nearline Storage or Coldline Storage as the data ages. This automation reduces storage costs without manual effort, and you can configure, view, or disable these rules using the Google Cloud console, gcloud command-line tool, or client libraries.
Retention policies enforce that objects must be kept in a bucket for a minimum period of time, which is crucial for regulatory compliance. You set a retention period on the bucket, and all objects inside are protected from deletion until that time passes. For immutable compliance, you can lock this policy using Bucket Lock, which prevents anyone from shortening the retention period or removing the policy, though the duration can still be increased. A locked policy remains in effect until the entire bucket is deleted.
Object holds provide a finer level of control by preventing specific objects from being deleted or overwritten. A temporary hold can be manually applied and released, while an event-based hold automatically releases when a predefined condition is met. You can also set a default event-based hold on a bucket so all new objects are automatically placed under hold. These holds are essential for legal and regulatory scenarios where data must be preserved intact for an investigation or audit.
Soft delete protects against accidental deletion by keeping deleted objects in a recoverable state for a configurable period, which defaults to seven days. You can adjust or disable this based on your needs. Object versioning maintains multiple generations of an object, and you can set lifecycle policies to automatically expire old, non-current versions. Together, these features give you control over data recovery and the storage costs associated with keeping historical versions.
Storage batch operations let you perform large-scale actions on billions of objects efficiently with a single job. You can delete objects, update metadata, apply or remove holds, or change encryption keys across massive datasets. These serverless jobs can process up to a billion objects within a few hours and support dry runs for validation, automatic retries, and detailed monitoring. Jobs are defined using manifest files or object prefixes to select the target objects and have a maximum runtime of 14 days.
The final data deletion in Cloud Storage is a secure, multi-stage process. After the soft delete recovery window, data undergoes logical deletion from active systems. This is achieved either through traditional overwriting or cryptographic erasure, which destroys the encryption keys needed to read the data, rendering it permanently unreadable. Understanding this process is important for compliance with data protection regulations that require proof that deleted data is truly irrecoverable.
BigQuery uses two main storage billing models: logical storage billing and physical storage billing. Logical billing charges for the uncompressed size of your data, while physical billing charges for the compressed size and includes fees for extended data retention features like time travel. You choose the model at the dataset level, and selecting the right one is a key decision for controlling costs and aligning with your data access patterns.
You can automate data deletion to control storage costs by setting expiration rules at multiple levels. At the dataset level, you set a default expiration time for all new tables created within it. Individual tables can override this dataset default with their own custom retention period. For partitioned tables, you can set partition-level expiration, which automatically drops individual time-based data chunks (like daily partitions) once their age exceeds the set limit. This layered approach ensures stale data is purged systematically.
For partitioned tables, partition expiration is calculated from the UTC time boundary of the partition (e.g., midnight for a daily partition). Once a partition expires, BigQuery automatically deletes it. Under the physical billing model, you can also reduce the default time travel window from seven days to a shorter period. Time travel allows you to query data as it existed at a point in the past, and shortening this window reduces the amount of historical data retained, further lowering storage costs.
BigQuery automatically provides a long-term storage discount. If a table or partition is not modified for 90 consecutive days, its storage cost is automatically reduced by 50%. The data's performance, availability, and durability do not change; it simply becomes cheaper to store. Any edit to the data, such as adding new rows, resets the 90-day counter. This feature makes it cost-effective to retain large volumes of historical analytical data for long-term trends and reporting.
Multi-storage tiering involves moving data between different storage classes based on how often it is accessed. Cloud Storage classes range from Standard (for hot, frequently accessed data) to Nearline, Coldline, and Archive (for progressively colder, less-accessed data). Each class has a different price for storing data and for retrieving it. You use lifecycle management policies to automate transitions between these classes, such as moving objects from Standard to Coldline 30 days after creation, or deleting them after a final retention period ends.
Designing an effective strategy requires analyzing your data access patterns. You must balance the cheaper monthly storage cost of colder classes against their higher data retrieval fees and slower access times. For example, you might keep current month's application logs in Standard storage for quick analysis, move logs from the previous quarter to Nearline, and archive logs older than a year to Coldline or Archive. The strategy should be automated with lifecycle policies so data flows to the appropriate cost tier without manual intervention.
To implement tiering, you architect your platform with a separate data storage layer (like Cloud Storage buckets) and a data computation layer (like Dataflow). You then configure buckets and their lifecycle rules using the Google Cloud console, gcloud CLI, or infrastructure-as-code tools like Terraform. Automation is critical; policies can be triggered by object age, current storage class, or other metadata, ensuring continuous cost optimization as data naturally cools over time.
For advanced optimization, consider data location to colocate storage with compute resources and reduce network transfer costs. Be aware of the minimum storage duration charges for colder classes like Nearline (30 days) and Coldline (90 days), which penalize early deletion. Incorporate object versioning and soft delete into your lifecycle planning for data protection. Regularly monitor storage usage and costs with Cloud Monitoring and refine your policies as business needs and access patterns change to maintain the optimal balance between cost, performance, and compliance.