Data Access Control, Lifecycle and Encryption

1.1IAM Permissions, Roles, and Least Privilege

Identity and Access Management (IAM) (opens in a new tab) is the Google Cloud system that controls which principal can act on which resource. A principal such as a user, Google group, or service account receives a role on a resource, and that role contains permissions. Least privilege means granting only the narrowest role on the narrowest resource that still allows the work.

An IAM permission is the smallest unit of access and allows one specific action. Permissions use the format SERVICE.RESOURCE.VERB, such as compute.instances.list, bigtable.instances.list, or pubsub.topics.publish. Permissions are never assigned directly to a principal in BigQuery. When a principal calls an API, IAM checks whether any granted role contains the required permission, and one data operation can require several permissions.

An IAM role is a named collection of permissions granted to a principal on a resource. When a principal receives multiple roles, effective access is the union of all permissions, so access only grows. A role granted on a parent flows to descendants through policy inheritance. A role granted on a project therefore applies to all BigQuery resources in that project, while a role granted on a dataset or on a single table applies only to that dataset or table and its children.

1.2Basic, Predefined, and Custom IAM Roles

Basic, predefined, and custom roles differ in breadth, management, and fit for least-privilege data access. Basic roles give broad access across many services, predefined roles give granular access for one service, and custom roles give an exact customer-chosen list:

The legacy basic roles, formerly called primitive roles, build on each other. Viewer allows read-only viewing across services, Editor includes all Viewer access plus creating, changing, and deleting most resources, and Owner includes all Editor access plus administration such as managing roles and billing. Basic roles contain thousands of permissions, conditions cannot be added to their bindings, and Owner does not contain every permission. Predefined roles are scoped to one service, so cross-service work needs one role per service, and predefined roles can gain permissions over time while custom roles stay fixed.

1.3Resource-Level Control in BigQuery

Resource level in BigQuery controls whether access is broad or limited to one dataset or table. Administrators choose the narrowest resource level that still lets the principal do its work. Access granted at the organization, folder, or project level applies to all BigQuery resources in that container and fits principals that need many datasets. Access granted at the dataset level applies to one dataset, while access granted at the table, view, or routine level applies to one object without full dataset access. Grantable objects include datasets, tables, views, routines, connections, saved queries, data canvases, data preparations, pipelines, and repositories.

1.4Private and Public Access for Cloud Storage

Cloud Storage (opens in a new tab) buckets are private by default and become shared only through explicit grants. Private access uses IAM policies where each role bundles permissions such as storage.objects.get or storage.objects.create. An IAM policy set on a project applies to all buckets and objects in that project, while a policy set on a bucket applies only to that bucket and its contents. Permissions from different levels add together.

Managed folders refine private control without forming a true directory tree. A managed folder is a resource overlay used only for permission checking and can grant additional access beyond the bucket policy. An IAM allow policy cannot be set directly on an individual object, and an IAM deny policy cannot be set on a bucket, managed folder, or object. A deny policy attached higher in the hierarchy still overrides any allow below it.

Public access defines how internet clients reach buckets and objects. Buckets stay private until a read role is granted to a public principal. allUsers means anyone on the internet with no authentication and fits only truly public content, while allAuthenticatedUsers means most authenticated Google accounts and still requires sign-in. A bucket-level public policy applies to every current and future object, so sensitive data should stay private with least-privilege roles granted to groups.

1.5Uniform Bucket-Level Access and Object ACLs

Uniform bucket-level access is a bucket setting that makes IAM the only access control system for that bucket. When uniform access is enabled, only bucket-level and higher IAM roles grant access, and bucket ACLs and object ACLs are disabled. New objects automatically follow the bucket IAM policy instead of receiving a separate default object ACL.

Without uniform access, IAM and ACLs work in tandem and access is granted when either system allows it. An ACL entry pairs a scope such as a user, group, domain, allUsers, or allAuthenticatedUsers with OWNER, READER, or WRITER permission. Storage Legacy Bucket roles stay synchronized with bucket ACLs, so changing one updates the other, while other IAM roles leave ACLs unchanged. Uniform access is managed by principals with the Storage Admin role, which contains storage.buckets.get, storage.buckets.list, and storage.buckets.update.

1.6Data Sharing with Analytics Hub

Analytics Hub is the BigQuery capability for sharing data across organizational boundaries without moving it to a separate system. A data exchange is the provider-side resource that defines where data is offered, and a listing inside the exchange describes one shared asset and controls discovery and requests. Data flows from the provider source dataset through the listing to each subscriber project, while the provider keeps ownership and revocation control. Subscribers discover the listing and then query the shared asset in BigQuery alongside internal data.

Zero-copy sharing lets subscribers query data where it lives without creating a copy. Because no data is duplicated, updates appear immediately and the provider can revoke access in one place. Support for lakehouse tables depends on catalog management. Apache Iceberg tables managed by BigQuery support Analytics Hub with column-level security, masking, and policy tags, while Apache Iceberg tables managed by the Lakehouse runtime catalog do not. Before sharing, check sensitivity, regulatory rules, audience, and business value, and use de-identification or curated views for sensitive data.

2.1Storage Classes from Standard to Archive

Cloud Storage (opens in a new tab) provides Standard, Nearline, Coldline, and Archive storage ordered from warmest to coldest. All new objects start in Standard storage, which is optimized for repeated reads. Nearline fits infrequently accessed data, while Coldline and Archive fit rarely accessed and long-term retention data at lower storage cost with higher operation and retrieval cost.

A mismatch between access frequency and class increases cost. The Storage Intelligence advisor reports a spike in Class A or B operations on Coldline or Archive data when cold data is read frequently, which means the data belongs in Standard or Nearline.

2.2Autoclass, Retention Limits, and Access Monitoring

Autoclass automatically manages the storage class of each object based on access patterns. All new objects begin in Standard storage, unaccessed objects transition to progressively colder classes ending at the terminal class, and an object whose data is read moves back to Standard storage. By default the terminal class is Nearline storage, with an option to configure Archive storage for longer retention. Reading or editing only metadata does not promote the object.

Retention and size rules set boundaries for tiering. Objects smaller than 128 KiB remain permanently in Standard storage, with only object data counted toward that limit. Soft-deleted objects keep their existing class until retention ends, while a restored object begins again in Standard storage. Retaining noncurrent versions indefinitely increases stored bytes, so lifecycle rules should delete noncurrent versions or move them colder. Monitoring uses Cloud Monitoring resource usage, Recommender right-sizing suggestions, and tiering statistics for hot-tier and cold-tier bytes.

2.3Deletion Rules for Cloud Storage and BigQuery

Object Lifecycle Management (opens in a new tab) uses automatic bucket rules that move or delete objects when age or other conditions are met. The Cloud Storage service evaluates the rules continuously, so cost control does not depend on manual cleanup. Lifecycle rules fit data with a known useful lifetime such as temporary load files or fixed-period backups.

Cloud Storage is the staging layer that feeds BigQuery. Data flows from the source environment to a staging bucket and then from the bucket into a BigQuery dataset:

1. Upload: Extracted files are copied from on-premises systems, another cloud, or application output into a staging bucket.
2. Load: Data flows from the bucket into BigQuery through batch load, BigQuery Data Transfer Service, Dataflow, or another transfer process.
3. Cleanup or retention: Source objects are kept, moved to a lower-cost class, or deleted according to the bucket lifecycle rule.

Because external tables and exports depend on the bucket, the bucket rule affects whether BigQuery can still read that data. A transfer can delete source files after success only when the identity also has storage.objects.delete.

Granular expiration in BigQuery removes targeted subsets instead of whole tables. Table expiration is metadata set at creation, partition expiration removes only targeted partitions, and teams often pair expiration with aggregation into smaller metrics tables. After deletion, replaced data first enters the time travel window where it remains queryable, then enters the fail-safe period for emergency recovery only. Fail-safe is a fixed seven days, cannot be queried directly, and requires Cloud Customer Care for table-level recovery.

2.4Archiving, Durability, and Location Controls

Evaluating archiving means matching cost, access frequency, and compliance to the right class and lifecycle policy. A policy can move 90-day-old Standard objects to Nearline and later to Archive, then delete them after retention ends. Nearline fits backups read at most about once a month. Soft Delete preserves deleted objects for recovery, Object holds prevent deletion while a hold is in place, and Bucket Lock and Object Retention Lock enforce minimum retention.

Redundant infrastructure keeps archives safe through backup capacity, data copies, alternate network paths, and geographically distributed data centers. Customers select regions such as us-east1 or multi-regions such as asia and europe, and the resource locations constraint can allow or deny locations. VPC Service Controls creates a service perimeter that restricts storage API import and export so data stays in selected networks. Google Cloud encrypts content at rest and in transit automatically, with options for customer-managed keys, Access Approval, and Access Transparency audit logs.

3.1Cloud SQL Managed Backups and Backup Options

Cloud Storage (opens in a new tab) acts as the durable destination for exports and backup files, while Cloud SQL provides incremental backups of an entire database instance. Data flows from the live instance to a backup copy stored separately, so deletion, corruption, or regional outage does not remove the only copy. Only the first backup is full-sized, later backups contain only changed data, and all backups are encrypted by default.

Cloud SQL backups restore an instance to a previous state, create a new instance in a different region or zone for disaster recovery, and create copies for development, testing, and migration.

3.2Cloud SQL Backup Types, Restore, and Retention

Cloud SQL backup types define when a copy is taken. On-demand backups can be created at any time even without automatic backups, automated backups run on a scheduled cadence during a backup window, and final backups are taken at instance deletion. The restore workflow returns data to a new or existing instance, not to individual tables. A vaulted restore can create a new instance with a new ID, overwrite a selected existing instance, or overwrite the source instance, which replaces all current data including point-in-time logs. Point-in-time recovery always creates a new instance.

Retention controls how long a backup remains available. Automated backup retention can range from 1 day to 10 years depending on option and policy. Standard on-demand backups remain until manually deleted, while enhanced on-demand backups use custom retention up to 10 years. When the oldest backup in an incremental chain is deleted, the next oldest grows to become the new full backup. Retained backups remain at the project level after instance deletion and can restore to a new or existing instance, while replicas have no separate backups and a promoted replica needs its own configuration.

3.3Cloud Storage Protection, Exports, and Centralized Services

Cloud Storage protection keeps objects recoverable from deletion, overwrites, and ransomware. A common pattern is a scheduled transfer of data to Cloud Storage or a database export uploaded to Cloud Storage where you manage the lifecycle. Older backup copies can move to Nearline, Coldline, and Archive classes for lower storage cost with retrieval costs and minimum durations. Choose a Cloud SQL backup to restore a whole instance disk or clone for disaster recovery, and choose a Cloud Storage export to keep a copy after deletion, move a single database or table, or retain data in a chosen class.

Protection uses layered controls:

• Object holds place metadata flags on objects to prevent deletion or overwriting.
• Soft delete retains recently deleted objects and buckets for a set period, enabled by default with seven day retention.
• Object Versioning archives an overwritten live object as a noncurrent version until explicitly removed.
• Bucket Lock defines a per-bucket retention policy that cannot be reduced or removed.
• Object Retention Lock defines per-object retain-until dates that block deletion until expiry.

Backup and DR Service centralizes protection by storing Persistent Disk snapshots in Cloud Storage buckets with monitoring and auditing. Storage Transfer Service migrates online data between Cloud Storage, Amazon S3, and on-premises storage, while Transfer Appliance ships very large offline datasets to Cloud Storage.

3.4Replication, Location Types, and Failure Domains

Disaster recovery scenarios for data (opens in a new tab) use replication to copy data so one zone or region failure does not cause loss. Cloud Storage achieves durability through erasure coding, redundant storage across devices, and regularly rechecked checksums, with design for at least 99.999999999% annual durability. Availability is the ability to access data immediately and depends on storage class and location type.

Cloud Storage uses an active-active design with no primary and secondary bucket and no manual failover. A write is confirmed after redundant storage in one initial region, then flows asynchronously to the other region. Default replication targets 99.9% of new objects within one hour and 100% within 12 hours. Turbo replication is a premium dual-region option targeting 100% within a 15-minute recovery point objective, regardless of object size.

3.5Cloud SQL Replicas and Primary Location Choices

Cloud SQL replication separates performance work from reliability work. A read replica offloads read queries to reduce load and latency but does not provide automatic failover. A high-availability standby maintains an in-sync copy in another zone that becomes available after about 60 seconds of primary unresponsiveness. A cross-region replica maintains a copy in another region to survive regional failure. For MySQL, the primary records changes in its binary log, the replica receives them into its relay log, then applies them.

Primary and secondary locations work together for failover and restore. The primary serves reads and writes, while the secondary holds a standby or backup in another zone or region. The live Cloud SQL instance always uses a regional location that cannot be changed after creation, while multi-regional locations us, eu, and asia hold only backups. A zonal primary has low resilience, a regional primary with cross-zone standby survives zone failure, and dual-region or multi-region storage or backups survive region failure.

4.1Encryption Key Options for Control and Simplicity

Encryption key choice in Google Cloud is decided by who owns the key encryption key (KEK) that protects data. Google-managed encryption keys (GMEK), also called Google default encryption, are owned and managed by Google with no customer action. Customer-managed encryption keys (CMEK) are created and managed by the customer in Cloud Key Management Service (Cloud KMS), while customer-supplied encryption keys (CSEK) are raw key material supplied directly by the customer.

GMEK is the automatic baseline where Google owns symmetric KEKs that protect data at rest. Data flows from the service to Google-managed keys for encrypt and decrypt operations, so ownership and rotation stay on the Google side. CMEK uses envelope encryption where the service generates a data encryption key (DEK) to encrypt data and the customer KEK wraps that DEK. Because the service depends on Cloud KMS to unwrap the DEK on every access, revoking or disabling the CMEK blocks reads, writes, and processing. CSEK keeps key supply on the customer side but increases operational burden, and Cloud SQL supports CMEK but does not support CSEK.

4.2Cloud KMS Hierarchy and CMEK Integration

Cloud KMS lets customers own and control keys instead of relying on automatic Google-owned encryption. Default encryption uses AES-256 with keys held in Keystore and validated to FIPS 140-3 Level 1, where Google manages rotation and customers cannot view keys or logs. CMEK moves ownership to the customer, supports unlimited key versions, and adds control over location, destruction, IAM access, rotation, external key import, policy enforcement, logging, and crypto-shredding by destroying the key.

A key hierarchy moves from location to operation: a key ring holds one or more keys, and each key works through its key versions. A key ring groups keys under one location and project, and its location must correspond to the protected resource. A key is the named resource the service uses, and a key version holds the cryptographic material with a primary version used for new encryption. Manually enabling CMEK follows an ordered dependency:

1. Create key ring and key: Create a Cloud KMS key ring and symmetric key in the same location as the future resource.
2. Grant access: Grant the service agent the CryptoKey Encrypter/Decrypter role on the CMEK.
3. Create new resource with the key: Select the key when creating a new instance, since CMEK cannot be added to an existing instance.
4. Operate and rotate: Set a rotation schedule and monitor logs for unavailable keys.

Cloud KMS Autokey generates key rings, keys, and service-account grants on demand during resource creation. Organizations can enforce CMEK with the constraint constraints/gcp.restrictNonCmekServices and restrict key projects with constraints/gcp.restrictCmekCryptoKeyProjects.

4.3Key Rotation, Replacement, and Destruction

Key lifecycle management covers creating, rotating, replacing, disabling, and destroying keys over time. Rotating a key creates a new primary version for new encryption, but previous versions remain enabled so existing data can still be decrypted. Rotation alone does not re-encrypt existing data and does not disable older versions. Replacing a key moves protection to a new key by updating bucket defaults, updating code that names the old key, re-encrypting objects with Rewrite Object, then disabling the old key only after re-encryption.

This control creates responsibility. Disabling a key in active use can cause data loss, so wait two weeks after rotation before disabling the previous version. If access is lost, data is deleted after 30 days while full recovery after re-enabling can take up to two weeks.

4.4Encryption in Transit and Encryption at Rest

Encryption in transit protects data while it moves across networks, while encryption at rest protects data stored on disks, snapshots, and images. Google Cloud encrypts both states by default. In transit, Transport Layer Security (TLS) keeps moving data unreadable, such as traffic flowing from Dataflow workers to Cloud Storage, BigQuery, or Pub/Sub. Administrators keep transit inside the private network with Private Google Access for workers without public IPs and VPC Service Controls perimeters that prevent copying data outside even with valid credentials.

Encryption at rest uses Advanced Encryption Standard (AES) for boot disks, data disks, clones, images, and snapshots. Protection flows from a source disk to its clones and snapshots, so snapshots must use the same key as the source.

Persistent Disk offers the most key choices, while Hyperdisk and Local SSD are more limited. Choose Google-owned keys for simplicity, CMEKs for customer control, and CSEKs only where supported. Compute Engine rotates Google-owned keys yearly, CMEK rotation should be automatic, and CSEK is deprecated for Compute Engine.

Key Facts

• IAM permissions allow one action using SERVICE.RESOURCE.VERB format, roles bundle permissions with union access and inheritance from parent to child, and least privilege uses the narrowest role on the narrowest resource.
• Cloud Storage buckets are private by default, public access uses allUsers without authentication or allAuthenticatedUsers with sign-in, and uniform bucket-level access makes IAM the only control by disabling ACLs.
• Cloud Storage classes run Standard, Nearline, Coldline, Archive from warmest to coldest, Autoclass starts in Standard and cools to Nearline by default, and objects under 128 KiB stay in Standard.
• Cloud SQL backups are incremental and encrypted by default, point-in-time recovery always creates a new instance, and high availability replicates synchronously to a standby in another zone for automatic failover.
• Envelope encryption wraps a data encryption key with a key encryption key, rotating a key creates a new primary version without re-encrypting old data, and destroying a version permanently prevents decryption.