BigQuery Project Dataset and Table Governance

Designing table architecture for compliance and lifecycle management means separating storage, access, and key controls so confidential data stays isolated and auditable. BigQuery (opens in a new tab) datasets are child resources of projects in the Identity and Access Management (IAM) hierarchy, and a table inherits permissions from the dataset that contains it. Table-level access control breaks that inheritance by granting a user, group, or service account access to specific tables or views without granting access to the complete dataset.

1.1Separation of Confidential and Non-Confidential Storage

Separation of confidential and non-confidential storage starts with projects and perimeters, not just tables. When data is imported from Google Cloud, BigQuery stores confidential data in the confidential data perimeter protected by VPC Service Controls (opens in a new tab), while non-confidential data remains in a separate non-confidential data project. When data is imported from an external source such as on-premises, BigQuery stores the encrypted data and the wrapped encryption key in separate tables, so compromise of the data table alone does not expose key material. Each warehouse uses its own customer-managed encryption key (CMEK), with one CMEK for the non-confidential project and another for the confidential project, and each CMEK must be in the same location as its resources.

Data flows from the source into the ingestion subsystem through Cloud Storage (opens in a new tab) or Pub/Sub, then through a Dataflow pipeline into the BigQuery warehouse inside the perimeter. An encrypted connection with Cloud VPN or Cloud Interconnect protects movement from external networks, with Dedicated Interconnect recommended for direct, high-throughput streaming. Allowlisted IP addresses in access-level policy rules permit only the expected external environment to reach Google Cloud.

1.2Column-Level Governance With Policy Tags

Column-level governance with policy tags restricts confidential columns even when a user can read the rest of the table. Access is defined with policy tags bound to columns, and a data analyst must have the Fine-Grained Reader role plus permission on the tag to read tagged data. Anything that is not tagged remains available to all users who have access to the data warehouse, which enforces data minimization by leaving only truly non-sensitive columns broadly readable.

The reference taxonomy uses three hierarchical tags that build on each other:

These controls persist after re-identification, so re-identified data still cannot be read until access is explicitly granted to the user. For additional protection without blocking column access, column-level dynamic data masking selectively obscures column data for groups of users, while row-level security filters access to specific rows based on qualifying user conditions.

1.3De-Identification Pipeline and Table Lifecycle

The de-identification pipeline reduces what is stored in governed tables to the minimum necessary for business goals. Raw data lands in Cloud Storage, then a Dataflow pipeline calls Sensitive Data Protection to find and tokenize confidential fields before loading the result into BigQuery. For structured data the pipeline uses record transformations based on fields, such as checks for credit card numbers and card PINs, while for unstructured data it uses information types (infoTypes) to detect sensitive values.

The ordered lifecycle for internal sources is:

1. Ingest: Source data arrives in Cloud Storage or Pub/Sub under CMEK protection.
2. Detect and transform: Sensitive Data Protection scans the payload and tokenizes fields tagged as confidential.
3. Load to separated tables: De-identified data lands in the non-confidential warehouse and confidential values land in the confidential warehouse with policy tags applied.
4. Re-identify on demand: Authorized analysis unwraps values only through approved decryption, still gated by column-level access controls.

This sequence enforces data minimization because data that is not strictly necessary is never collected, kept, or used in cleartext. Where possible, synthetic or fully anonymized data replaces raw PII, and transformation methods such as pseudonymization, de-identification, and generalization like bucketing preserve utility while reducing exposure.

1.4Encryption and Key Lifecycle for Tables

Encryption and key lifecycle for tables operates at two layers: platform encryption of stored data and column-level protection of sensitive values. BigQuery encrypts all data at rest and in transit by default, and customers who need control over keys use customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) (opens in a new tab) with Cloud Hardware Security Module (Cloud HSM). By default the CMEK is rotated every 30 days, and organizations that must manage keys outside Google Cloud can use Cloud External Key Manager (Cloud EKM) but then assume responsibility for rotation and other key management activities.

Client-side column encryption adds a separate lifecycle for external ingestion. The Tink encryption library, which is compatible with BigQuery Authenticated Encryption with Associated Data (AEAD) functions, encrypts sensitive columns before they leave the external network. Tink uses a data encryption key (DEK) generated locally or from Cloud HSM to encrypt the column, and a key encryption key (KEK) generated as a symmetric CMEK in Cloud HSM to wrap the DEK. Both the wrapped DEK table and the encrypted-data table are stored in BigQuery, and BigQuery uses AEAD decryption to unwrap the DEK with the KEK and decrypt the protected column only when an authorized analyst queries it.

1.5Centralized Governance, Lineage, and Auditability

Centralized governance, lineage, and auditability ties projects, datasets, and tables together for compliance. Knowledge Catalog provides the organization-wide governance layer that discovers, manages, monitors, and governs data and artificial intelligence (AI) artifacts across data platforms and grants access only to trusted users. Its governed-table responsibilities include managing data lineage including column-level lineage from BigQuery, managing data quality checks and data profiles and feature metadata, managing discovery and exploration across different data marts, creating a business glossary with aspects and aspect types, unifying governance across BigLake and open-format tables like Iceberg and Delta Lake, and supporting a data mesh that decentralizes ownership to domain teams while preserving central policy.

Lifecycle governance also depends on audit trails and residency controls. Admin Activity audit logs are enabled by default for all services and record configuration changes, while Data Access audit logs are enabled by default for BigQuery and track reads, creations, and modifications of user-provided data. BigQuery, Cloud Storage, Pub/Sub, and Cloud Logging all help meet data residency requirements by storing or replicating data only within the regions specified, and Cloud Storage lifecycle controls limit prolonged retention or manipulation of sensitive objects.

Establishing data governance through project and resource hierarchy means using the Google Cloud resource hierarchy to isolate data by trust level and then enforcing that isolation with identity, policy, and network controls. Google Cloud Organization (opens in a new tab) sits at the top, folders group environments and business units in the middle, and projects hold datasets and tables at the bottom. This structure lets governance, ingestion, confidential, and non-confidential workloads live in different projects with different controls, so a compromise or misconfiguration in one area does not expose the rest.

2.1Multi-Tiered Hierarchy with Organizations and Folders

The multi-tiered hierarchy with organizations and folders is the skeleton that separates testing from production and governance from data. Google Cloud Resource Manager logically groups every resource under an organization, then into folders, then into projects, and access controls follow the principle of least privilege down that chain. Folders typically represent environments such as bootstrap, common, production, non-production or staging, and development. Most data warehouse projects deploy into the production folder, while the data governance project deploys into the common folder, so key management, catalog management, and logging stay separate from the data itself.

2.2Project Segregation by Sensitivity and Environment

Project segregation by sensitivity and environment places datasets with different sensitivity in different projects so that Identity and Access Management (IAM), encryption, and perimeter rules can differ by project. The confidential data project holds re-identified and restricted BigQuery datasets, while the non-confidential data project holds de-identified or sanitized datasets, and separate ingestion and governance projects hold pipelines and keys. The internal project owns the service account that updates shared data, and the shared data project holds only the sanitized copy that external groups can view. First data lands and is de-identified in the ingestion landing zone, then it flows to the non-confidential or confidential projects, and only after explicit grants can analysts read the restricted copy.

2.3Enforcing Boundaries with IAM Policies

Enforcing boundaries with IAM policies determines who can act on which project, dataset, and table under least privilege. Identity and Access Management (IAM) (opens in a new tab) policies are attached at the organization, folder, and project levels, so a grant at a higher level flows down to everything below it unless a more restrictive control blocks it. Dedicated, least-privilege service accounts isolate pipelines from human users, so a training or ingestion service account can read from only one Cloud Storage bucket and write to only one registry or dataset.

• IAM roles on projects and datasets give users and groups only no-access, read-only, or write access needed for their role, mapped by user, group, service account, or tags.
• IAM Conditions add attribute-based limits, such as allowing a service account to trigger a pipeline only when the request comes from a trusted build trigger.
• Principal access boundary (PAB) policies limit what resources a principal can access even when other policies would allow more.
• Privileged Access Manager controls just-in-time, temporary elevation for select principals, with audit logs recording the elevated activity.
• Groups for external sharing give outside accounts viewer permission on the shared data project only, while multifactor authentication (MFA) is required for all users who access sensitive data.

2.4Enforcing Guardrails with Organization Policies

Enforcing guardrails with organization policies constrains what resources can do rather than granting access. The Organization Policy Service (opens in a new tab) sets centralized guardrails applied globally at the organization or selectively at a folder or project. Because they inherit downward, a guardrail set on the production folder automatically covers every confidential and non-confidential project inside it.

• Disable service account creation with iam.disableServiceAccountCreation set to true to prevent uncontrolled identity sprawl.
• Disable service account key creation and disable automatic role grants to default service accounts to force use of managed, least-privilege identities.
• Require OS Login for VMs with compute.requireOsLogin set to true and disable serial port logging to Cloud Logging with compute.disableSerialPortLogging set to true.
• Restrict protocol forwarding creation to internal only with compute.restrictProtocolForwardingCreationForTypes set to INTERNAL and constrain Shared VPC subnetworks with compute.restrictSharedVpcSubnetworks to a specific private subnet.
• Require CMEK protection with gcp.restrictNonCmekServices set to bigquery.googleapis.com so BigQuery datasets cannot be created without customer-managed keys.
• Restrict Cloud Run functions ingress with cloudfunctions.allowedIngressSettings set to ALLOW_INTERNAL_AND_GCLB so functions accept only internal and load-balancer traffic.

2.5Preventing Exfiltration with VPC Service Controls Perimeters

Preventing exfiltration with VPC Service Controls (opens in a new tab) perimeters creates a security boundary around Google-managed services that controls data movement across the perimeter, even when the caller has valid IAM credentials. Each perimeter contains specific projects and services, and traffic flows from inside the perimeter to outside only when an explicit ingress or egress rule allows it. A perimeter bridge then allows controlled egress from a more protected perimeter to a less protected one, such as from internal data projects to external share projects, without merging the two perimeters.

• Data ingestion perimeter accepts incoming batch or streaming data and de-identifies it, so the landing zone protects the rest of the workloads from untrusted content.
• Confidential data perimeter stores re-identified data in a restricted area with additional access controls and monitoring for analysis.
• Data perimeter for external ingestion isolates encrypted data from other workloads before decryption and governance checks.
• Governance perimeter stores encryption keys, defines what is considered confidential, and holds catalog management and logging separate from the warehouse.

Implementing dataset and table-level security and classification in BigQuery is a layered governance model that combines resource hierarchy, Identity and Access Management (IAM), data classification, and fine-grained filters. BigQuery (opens in a new tab) resources are ordered as organization, folder, project, dataset, and then table, column, and view, and access flows down this hierarchy unless a lower-level grant overrides it. Classification with policy tags determines which columns are sensitive, while table access control, dynamic data masking, row-level security, and authorized views determine who can see what.

3.1IAM Hierarchy and Dataset-Level Permissions

IAM hierarchy and dataset-level permissions grant access by assigning roles to a user, group, or service account on a specific resource in the hierarchy. Datasets are child resources of projects, and a table inherits permissions from the dataset that contains it unless a more specific grant is applied. Organization and project roles affect the ability to run jobs or manage the project, whereas dataset roles affect the ability to access or modify the data inside the project.

Least privilege is enforced by choosing narrow roles and adding conditions that limit when the grant applies. BigQuery supports three role families that differ in scope and control:

• Basic roles such as Owner, Editor, and Viewer apply broadly at the project level and should be avoided for sensitive data because they grant more than data access.
• Predefined roles support common data patterns, such as data viewer, data editor, job user, and Fine-Grained Reader for protected columns.
• Custom roles combine a user-specified list of permissions for granular access when no predefined role matches the job function.

Access can be further constrained with IAM policies with conditions, principal access boundary (PAB) policies that limit what a principal can reach, and Privileged Access Manager for just-in-time temporary elevation that is recorded in audit logs.

3.2Table-Level Access Control

BigQuery table-level access control breaks dataset inheritance by granting access to a specific table or view without granting access to the complete dataset. The table policy names the users, groups, and service accounts that can read the table, so a pipeline service account can write to one ingestion table while analysts can read only curated marts. This design supports project, dataset, and table separation where a storage project holds datasets, a separate project runs query jobs, and each sensitive table carries its own grant. Choose table-level grants when datasets mix sensitivity levels and creating a separate dataset per table would fragment management.

3.3Column-Level Security and Classification with Policy Tags

BigQuery column-level security restricts sensitive columns using policy tags from a classification taxonomy, or type-based classification of data. A taxonomy defines tags such as sensitivity levels, the tags are attached to columns, and only principals with the Fine-Grained Reader role on the tag plus dataset or table read access can read the tagged values. Discovery with Sensitive Data Protection and governance with Knowledge Catalog feed this process by finding personally identifiable information (PII), storing data profiles in BigQuery, and enriching catalog entries with aspects, lineage, and business glossary terms.

A common three-tier taxonomy shows how tag hierarchy controls inheritance from higher to lower sensitivity. A 3_Confidential tag protects very sensitive data such as credit card numbers and grants access to 2_Private and 1_Sensitive columns. A 2_Private tag protects sensitive PII such as first name and grants access to 1_Sensitive columns but not to 3_Confidential columns. A 1_Sensitive tag protects non-public business data such as credit limit and grants no access to 2_Private or 3_Confidential columns. Columns with no tag remain available to all users who have access to the warehouse, so untagged columns must be an explicit decision rather than a default.

3.4Dynamic Data Masking and Column Encryption

BigQuery column-level dynamic data masking lets authorized users query a column while the service obscures values for groups of users based on masking rules. Data flows from the stored column through the masking policy to the query result, so support teams can join and aggregate without seeing raw PII while data stewards with the masking exception still see plaintext. Masking preserves utility for analytics and troubleshooting where full denial would block the query, while policy tags remain the control when the value must be completely hidden.

BigQuery column-level encryption with Authenticated Encryption with Associated Data (AEAD) functions protects values even from operators who can read the table storage. The sender encrypts sensitive fields with a data encryption key (DEK) from the Tink library, wraps the DEK with a key encryption key (KEK) stored in Cloud Key Management Service (Cloud KMS) with Cloud HSM, and stores the encrypted data and wrapped DEK in separate BigQuery tables. At query time BigQuery unwraps the DEK with the KEK and decrypts the column, so access depends on both IAM on the table and IAM on the KMS key. BigQuery also encrypts all data at rest and in transit by default with no opt-out, with customer-managed encryption keys (CMEK) available when compliance requires control of dataset default keys.

3.5Row-Level Security and Authorized Views

BigQuery row-level security filters rows at query time based on a qualifying condition tied to the requester, such as SESSION_USER() matching a tenant ID or region attribute. The filter sits between the stored table and the query engine, so two users who query the same table receive different row subsets without separate tables. Row filters are ideal for multi-tenant fact tables where tenant segmentation can share partitioning and clustering, and clustering the parent table on tenant ID improves pruning for these filtered scans.

BigQuery authorized views (opens in a new tab) enforce row, column, and business-logic boundaries by separating query permission from storage permission. The view is created in a separate dataset, the view is authorized against the source dataset that holds the base tables, and the consumer is granted read access only to the view dataset with no read access to the base tables. When the user queries the view, BigQuery checks the user against the view and checks the view against the source tables, so the view identity bridges the trust boundary between the consumer project and the protected storage project. The same pattern applies to authorized datasets and authorized routines, including authorized user-defined functions (UDFs), which must be explicitly authorized before they can read protected tables on behalf of the caller.

Tenant isolation can use either tenant-specific authorized views or tenant-specific subset tables, and the choice trades governance overhead against performance and placement control. Authorized resources, which include authorized views, authorized datasets, and authorized functions, have a hard limit of 2500 per dataset, while there is no limit on the number of datasets in a project or tables in a dataset.

Choose authorized views when tenant logic changes frequently and instant redesign of the exposed object matters, because changing the view SQL immediately changes what tenants access. Choose subset tables when tenants need different regions, independent partitioning, or isolated logging and transformation, accepting the cost of schema changes and policy reapplication for each copy.

Key Facts

• BigQuery tables inherit permissions from their containing dataset unless table-level access control grants specific users, groups, or service accounts access to only that table or view.
• A 3_Confidential policy tag grants access to 2_Private and 1_Sensitive columns, while 1_Sensitive grants no access to higher tags, and untagged columns remain readable to anyone with warehouse access.
• Customer-managed encryption keys (CMEKs) for BigQuery rotate every 30 days by default, and each confidential and non-confidential warehouse must use a separate CMEK in the same location as its resources.
• VPC Service Controls blocks data from leaving a perimeter to outside resources even when the caller has valid IAM credentials, with movement allowed only by explicit ingress, egress, or perimeter bridge rules.
• Authorized resources including authorized views, authorized datasets, and authorized functions have a hard limit of 2500 per dataset, while there is no limit on datasets per project or tables per dataset.