Secure Data Sharing Rules in BigQuery

Implement Secure Data Sharing Mechanisms and Auditing is the practice of letting authorized collaborators query data without copying it into uncontrolled environments while recording every access. BigQuery sharing (Analytics Hub) and BigQuery data clean rooms provide the controlled framework for cross-organization and privacy-sensitive collaboration, while Cloud Storage (opens in a new tab) and Looker Action Hub extend sharing to objects and dashboards. Cloud Audit Logs, Cloud Logging, and Cloud Monitoring provide the audit trail that proves who accessed data, what changed, and whether sharing rules were followed.

1.1Choose a Sharing Method That Keeps Control With the Provider

A secure sharing method defines where data lives, who can query it, and what is allowed to leave the boundary. BigQuery sharing lets a provider publish datasets once and lets subscribers query them across organizations without moving copies, while BigQuery data clean rooms add a privacy-enforced environment where multiple parties can join and analyze sensitive data under strict rules. Looker Action Hub shares results directly from business intelligence dashboards into built-in destinations, so dashboard data flows from Looker to the destination without manual export.

Choose BigQuery sharing when consumers need governed read access to curated datasets, choose data clean rooms when two parties must combine sensitive data without exposing raw rows to each other, and choose Cloud Storage IAM when the unit of sharing is files that require distinct upload, view, and management rights.

1.2Enforce Least-Privilege Access on Shared Data

Least-privilege access means every identity receives only the minimum permission needed for its role, such as no-access, read-only, or write. Identity and Access Management (IAM) enforces this by mapping predefined or custom roles to a user, group, or service account, refining grants with IAM conditions and resource tags, and adding row-level security and column-level security in BigQuery to restrict rows and columns within a dataset. Cloud Storage applies the same principle at the bucket level, where a policy can separate management from data visibility.

The finance-marketing pattern shows how separation works in practice. IT staff receive Storage Legacy Bucket Owner on the bucket, which allows bucket management tasks such as deleting objects and changing the IAM policy plus listing bucket contents, but not viewing or downloading object data. The marketing analyst receives Storage Object User on the same bucket, which allows uploading, viewing, updating, and deleting objects. Temporary elevation is handled by Privileged Access Manager, which grants just-in-time privilege elevation with audit logging, while principal access boundary (PAB) policies cap what a principal can reach and Identity-Aware Proxy (IAP) with context-aware access restricts access by IP address and device attributes.

1.3Contain Movement With Perimeters and Encryption

A sharing rule must also control exfiltration, which is data leaving the trusted boundary through APIs, networks, or copies. VPC Service Controls creates a service perimeter around AI, ML, and data resources and sensitive data, and ingress and egress rules on that perimeter control which data flows enter and leave. Firewall rules then restrict inbound and outbound traffic to those resources under a deny-all-by-default policy that explicitly allows only traffic meeting defined criteria.

Encryption and sharing guardrails protect data at rest, in transit, and from public exposure. Customer-managed encryption keys (CMEKs) in Cloud KMS give the customer control of keys while CMEK-integrated services including Vertex AI Agent Platform, Cloud Storage, Cloud SQL, Spanner, and BigQuery encrypt data at rest. HTTPS protects API calls to AI and ML services, with HTTPS load balancers used to enforce it, while organization controls such as domain restricted sharing, uniform bucket-level access, public access prevention, and restrictions on public access to BigQuery datasets and Cloud Storage buckets keep sharing inside authorized domains and IAM-only paths.

1.4Build Governance, Logging, and Audit Trails

A data governance layer monitors data flows, transformations, and access across teams and preserves logs of access and manipulation for audit. Knowledge Catalog provides the organization-wide governance platform to centrally discover, manage, monitor, and govern data and AI artifacts and to grant access to trusted users. Knowledge Catalog responsibilities include:

• Managing data lineage, including column-level lineage from BigQuery, to trace data from source tables to shared outputs.
• Managing data quality checks, data profiles, and automatic data quality validation before training or sharing.
• Managing discovery, exploration, and processing across data marts, plus feature metadata and model artifacts.
• Creating a business glossary, enriching metadata with aspects and aspect types, and unifying governance across BigLake and open formats such as Iceberg and Delta.
• Building a data mesh that decentralizes ownership to domain teams and sending Sensitive Data Protection inspection results from BigQuery into the catalog.

Operational audit trails depend on complementary log sources that together record API activity, pipeline behavior, and network movement. Cloud Audit Logs collects audit trails of API calls, Cloud Logging captures real-time pipeline logs with live tailing, and Cloud Monitoring collects metrics and alerting on logs to detect anomalies, unauthorized attempts, and pipeline tampering. VPC Flow Logs for Compute Engine, load balancer logging, centralized security monitoring, retained audit records, and on-demand audit log access extend that trail to network and compliance review. Do not log personally identifiable information (PII) or confidential data in experiments or log servers, because logs themselves become a sharing surface that must stay compliant.

Establish Data Governance and Classification Frameworks is the foundation for sharing rules built on a federated catalog, a strict data hierarchy, lineage tracking, and centralized policy enforcement. The catalog provides a current inventory of what can be shared, the hierarchy defines ownership and access boundaries, lineage records where data came from and how it was transformed, and organization policies enforce which operations are allowed. Together these controls let authorized projects query shared data products without copying data or metadata.

2.1Federated Catalog and Connection for Shared Data

The Federated Catalog (opens in a new tab) is a BigQuery Lakehouse mechanism that continuously queries the remote SAP Business Data Cloud (SAP BDC) catalog for the current list of shares, schemas, and tables. The Connection is the separate Lakehouse resource that links and authenticates BigQuery to SAP BDC. Because metadata flows from the SAP BDC tenant to BigQuery on every discovery, consumers never rely on stale or manually copied inventories.

Federated querying means data products published in SAP BDC are exposed through secure Delta Sharing endpoints, and BigQuery federates queries from the Google Cloud project to those endpoints with zero copy overhead. The trust boundary sits between the Google Cloud project and the SAP BDC tenant, and the SAP BDC Connector is the SAP-side configuration that authorizes access from a specific Google Cloud project. The workflow depends on the Connector: without an authorized project-to-tenant pairing, the Connection cannot authenticate and the Federated Catalog cannot list shares.

2.2Data Hierarchy as Ownership and Sharing Boundary

The SAP BDC data hierarchy shared in BigQuery follows a five-level address: PROJECT_ID.CATALOG_ID.SHARE_NAME.SCHEMA_NAME.TABLE_NAME. This structure maps directly to the Delta Sharing protocol used by SAP BDC and makes every shared object addressable from project down to table.

The Catalog ID establishes ownership because one Catalog contains only the shares from a single Connector authorized for one project. The Share is the stewardship unit because tables are grouped, published, and consumed together as a data product. Schema and Table then provide progressively narrower scopes where classification and access rules can be applied without breaking the parent grouping.

2.3Governed Reuse Through Catalog Registration

The Cortex Framework enforces governed reuse by registering the Lakehouse catalog as a module of type lakehouse_delta_share under data.modules.catalogs in config/config.yaml. Registration declares the catalogId, projectId, location, and the allowed shareId values, such as customer_v1_he2_100_p8123. Only shares listed in that registration can be bound by downstream products.

Downstream custom data products reference sources through dependencyBindings in the form {catalog_id}.{share_id}.{table_name}, where the catalog ID matches the catalog id, the share ID matches a declared share, and the table name matches the physical table inside that share. During compilation, Cortex Framework reads the catalog metadata and translates each binding into a fully qualified BigQuery federated table name. The downstream product therefore depends on the catalog for both discovery and lineage, and no table can be consumed unless its ownership chain is explicitly declared.

2.4Lineage Tracking and Custom Constraints

The Knowledge Catalog (opens in a new tab), formerly Dataplex Universal Catalog, is the system that records data lineage as processes, runs, and events. A lineage Process describes the transformation or movement, while runs and events record individual executions. Governance of lineage focuses on the Process because it is the stable definition of who created derived data and from which sources.

Custom Organization Policy constraints allow or deny creation of lineage Process resources based on supported attributes:

• process name
• source type
• origin

Conditions are written in Common Expression Language (CEL) and evaluated when a Process creation is requested. Enforcement is inherited from the organization to folders to projects, so a rule set at a higher level applies to all descendants unless inheritance is explicitly changed. Management requires the Organization Policy Administrator role (roles/orgpolicy.policyAdmin), which includes permissions for listing constraints, creating and updating custom constraints, and creating and setting policies.

2.5Centralized Policy Enforcement for Sharing Rules

Centralized policy enforcement for sharing rules is provided by Organization Policy, which gives programmatic control over Google Cloud resources across the resource hierarchy. A policy is a set of restrictions called constraints that can be enforced at the organization, folder, or project level. Predefined constraints cover common service controls, while custom constraints define granular control over specific fields for CREATE and UPDATE methods.

A custom constraint is defined in YAML by its target resource types, method types, CEL condition, action type, display name, and description. This structure lets governance teams enforce requirements such as customer-managed encryption keys or repository field restrictions before a resource is created. The Organization Policy Service, including predefined and custom policies, is offered at no charge, so enforcement can be applied broadly as the foundation on which all data sharing rules depend.

Configure Granular Access Controls and Entitlements is the practice of granting only the minimum data access a consumer needs to perform an authorized task on a shared dataset. In Google Cloud, Identity and Access Management (IAM) provides the coarse role foundation, while BigQuery column-level security, row-level security, policy tags, and masking or tokenization enforce fine-grained entitlements inside the data itself. Knowledge Catalog acts as the central control plane that holds this security context so the same policy applies consistently across engines such as BigQuery SQL and Apache Spark.

3.1Role-Based Access Control with Least Privilege

Role-based access control (RBAC) with least privilege assigns different levels of access based on user roles so each identity receives only no-access, read-only, or write access required for its work. The assignment maps IAM roles for different products to a user, group, or service account based on project needs, access patterns, or resource tags. This separation prevents data theft, preserves model and data integrity, and limits the surface area for accidents or attacks when sensitive data resides in lakes, warehouses, or feature stores.

Granularity is applied at both the project and application level, with access flowing from the IAM policy to the specific resource that holds the data. BigQuery (opens in a new tab) enforces IAM roles and permissions for datasets and resources within datasets, including tables and views, while Cloud Storage enforces IAM policies on buckets and managed folders that separate bucket management from object viewing. Dedicated, least-privilege service accounts enforce workload isolation, so a training or sharing pipeline service account can read from only one Cloud Storage bucket and write to only one registry or dataset. Because storage permissions are decoupled from query permissions, a principal can be allowed to list a bucket or dataset without being allowed to read its objects or columns.

3.2Conditional and Boundary Controls for Shared Data

Conditional and boundary controls refine IAM roles by limiting when, where, and how far a granted role can reach. IAM Conditions manage granular, attribute-based access to data, models, and configurations such as code, resource settings, and hyperparameters. A condition allows access only when its attributes match, for example allowing a service account to trigger a pipeline only if the request originates from a trusted build trigger.

Additional boundaries control privilege scope and network context. Principal access boundary (PAB) policies limit the resources that a selected principal can access even if other policies would grant broader access, while Privileged Access Manager controls just-in-time temporary elevation with later review through audit logs. Identity-Aware Proxy (IAP) with context-aware access and Access Context Manager restrict access based on IP address and end-user device attributes, with access re-evaluated at each authentication. Choose IAM Conditions when the rule depends on request attributes, PAB policies when the goal is to cap maximum reach, and temporary elevation when standing administrative access would violate least privilege.

3.3Column-Level Security with Policy Tags

Column-level security in BigQuery restricts who can see content in specific columns, so analysts with dataset access still cannot read tagged sensitive columns without an explicit entitlement. To define access, an administrator creates policy tags in a taxonomy and attaches those tags to table schema fields. A data analyst must then hold the Fine-Grained Reader role on the relevant tag to read that column, and anything without a tag remains available to all users who have warehouse access.

A common three-tier taxonomy shows how entitlements nest from low to high sensitivity:

The policy travels with the base table. Column policies applied to a base table are applied to all authorized views on that table regardless of permissions on those views, while each physical subset table must reapply the column policy for it to take effect. This inheritance makes views safer for sharing slices of one governed table.

3.4Row-Level Security and Controlled Sharing Views

Row-level security in BigQuery restricts which rows a principal can read from a table, so two consumers can query the same shared table and each receive only their authorized subset. Row filters evaluate the viewer identity at query time and return only matching rows, which enforces least privilege without copying the table per tenant. This row control composes with column controls, so a shared dataset can hide both unauthorized rows and unauthorized columns in one query path.

For multi-tenant sharing, tenant access to subsets of large fact tables can use tenant-specific authorized views or tenant-specific subset tables. Authorized views share the partitioning, clustering, and region of the base table and are limited to 2,500 authorized resources per dataset, but they allow instant redesign of what tenants access and log access against the base table. Subset tables can be partitioned, clustered, and regionalized for each tenant and log access separately, but they require schema changes and separate policy enforcement. Choose authorized views when tenant segmentation, instant policy change, and centralized logging matter most, and choose subset tables when per-tenant performance tuning or regional placement outweighs duplication cost.

3.5Dynamic Data Masking and Tokenization with Sensitive Data Protection

Dynamic data masking and tokenization preserve data utility while preventing unauthorized disclosure of confidential values in shared datasets. Sensitive Data Protection discovers and classifies confidential data and then de-identifies it during ingestion, using record transformations based on fields for structured data and information types (infoTypes) for unstructured data. A masking policy then reveals only masked values to unentitled readers at query time, while tokenization replaces the original value with a reversible token that still cannot be read until explicit column access is granted.

Tokenization for a confidential warehouse follows an ordered pipeline:

1. Ingest: Source data flows from Cloud Storage or Pub/Sub into a Dataflow pipeline protected by customer-managed encryption keys (CMEKs) managed in Cloud HSM.
2. De-identify: Data flows from Dataflow through Sensitive Data Protection, which tokenizes fields tagged as confidential using a cryptographic key wrapped by Cloud HSM.
3. Store: The de-identified output flows from Dataflow into the BigQuery warehouse, with separate keys for the non-confidential and confidential projects.

Even after re-identification is possible, column-level access controls still block reads until access is explicitly granted. This layering ensures that possession of the pipeline or storage permission does not by itself grant visibility into PII such as credit card numbers or card PINs.

3.6Centralized Enforcement with Knowledge Catalog and Secure Connections

Knowledge Catalog (opens in a new tab) provides policy-compliant lakehouse access by functioning as a universal context engine across open formats and engines. Data stays in its open Apache Iceberg (Parquet) format inside Cloud Storage while Lakehouse manages the table metadata, and Knowledge Catalog dynamically enforces the attached security tags and masking policies whether the query arrives from BigQuery SQL or Apache Spark. Verification therefore tests the same identity against both engines to confirm identical enforcement.

This unified architecture depends on a logical security boundary that separates storage from querying. Instead of letting users or query engines directly access Cloud Storage buckets, queries flow through a secure Cloud resource connection that decouples storage permissions from data permissions. VPC Service Controls reinforces the boundary by defining a service perimeter around internal data assets, with an explicit perimeter bridge allowing only controlled egress from internal data projects into externally shared data projects. The connection depends on Knowledge Catalog metadata, the perimeter depends on ingress and egress rules, and the shared dataset depends on all three to prevent direct bucket access from bypassing fine-grained entitlements.

Key Facts

• BigQuery sharing lets subscribers query provider datasets without copying data, while BigQuery data clean rooms let multiple parties join sensitive data inside a governed environment with restricted outputs.
• Storage Legacy Bucket Owner can manage a Cloud Storage bucket and list its contents but cannot view object data, while Storage Object User can upload, view, update, and delete objects.
• The SAP BDC sharing hierarchy is PROJECT_ID.CATALOG_ID.SHARE_NAME.SCHEMA_NAME.TABLE_NAME, where the Catalog maps to the Lakehouse connection and the Share is the data product unit.
• An authorized view in BigQuery automatically inherits column-level policy tags from its base table, while a separate subset table must have each policy reapplied.
• VPC Service Controls perimeters with ingress and egress rules plus Cloud Audit Logs, Cloud Logging, and Cloud Monitoring together control exfiltration and prove compliance for shared data.