Publishing datasets

Establish Secure Shared Datasets via BigQuery Analytics Hub

BigQuery Analytics Hub is a platform for securely sharing datasets across different teams or organizations. Its core design separates storage from compute. This means the data publisher maintains control over the original data, while consumers use their own BigQuery projects and resources to run queries, eliminating the need to copy the data.

To share data without exposing the underlying tables, publishers use authorized views and authorized datasets. These act as secure interfaces, allowing consumers to query data through SQL while the publisher retains full control over the source schema and access rules. When the underlying data structure changes, publishers can manage the transition smoothly using one of two versioning strategies:

• Dataset versioning: The entire dataset name is updated (e.g., from sales_v1 to sales_v2), while the view names inside remain the same.
• View versioning: The dataset name stays constant, but version identifiers are appended to the view names (e.g., catalog_v1 and catalog_v2).

Security is enforced through layered controls. Identity and Access Management (IAM) provides the foundation for user permissions. Row-level security and column-level security add fine-grained filters, restricting which rows or sensitive columns a user can see based on their identity or group membership. For high-security collaboration, VPC Service Controls create a secure perimeter to prevent data from being moved outside authorized boundaries. Data masking can also be applied to dynamically hide sensitive values at query time.

For software-as-a-service (SaaS) scenarios, a dedicated tenant model can be used, where each customer's data and compute resources are isolated in separate projects. This simplifies billing and performance management. To control compute costs in multi-tenant setups, administrators use BigQuery reservations to allocate query processing capacity (slots) fairly among different projects.

Optimize Discoverability, Metadata, and Cost-Sharing Models

Making shared data easy to find and understand is critical for self-service analytics. Dataplex Universal Catalog (formerly Data Catalog) is the managed service for data discovery and metadata management. It automatically registers technical metadata—like table names and schemas—for BigQuery tables, authorized views, and BigLake views. Publishers should enrich these assets with business-friendly descriptions and tags to improve searchability.

BigQuery's sharing features (including Analytics Hub) use a publish-subscribe model that enables zero-copy sharing. Subscribers access data directly in the publisher's project without creating a physical copy, which simplifies management and ensures data freshness. Publishers can share datasets, tables, views, machine learning models, and even Pub/Sub topics through curated listings in data exchanges.

Managing costs for shared data involves specific models:

• In Cloud Storage, the Requester Pays model allows publishers to shift data access and egress costs to the consumers.
• In BigQuery, the publisher is always responsible for storage costs. Subscribers pay for the queries they run, either through on-demand pricing or their own capacity reservations.

The Data Catalog also tracks cost-related metrics, such as query and storage costs derived from Cloud Billing data, which helps in monitoring spending and detecting anomalies. Metadata tagging is used to attach governance information to data assets, including encryption status, data classification, and retention policies. These tags can trigger automated policy enforcement and compliance checks.

Access is governed by IAM bindings and formalized data sharing agreements. These agreements, stored in a dedicated warehouse, are linked to metadata tags. When consumers query sensitive data, they may be required to specify a valid "purpose," which is checked against their entitlements. All data access is logged to an audit trail in BigQuery for compliance monitoring.

Implement Robust Access Governance and Row-Level Security

Access governance is the framework of controls that determines who can see and use what data within published datasets. The goal is to enforce the principle of least privilege, granting users the minimum access necessary for their work, especially when sharing data via BigQuery sharing or Analytics Hub.

The first layer of control is Identity and Access Management (IAM), using predefined or custom roles to grant permissions at the project, dataset, or table level. For secure sharing, authorized views are essential. They allow a publisher to let specific users or groups query a view without granting them any access to the underlying source tables.

For granular control within a single table, two key features are used:

• Row-level security (RLS): Filters the rows a user can see based on their identity (e.g., a user only sees data for their department).
• Column-level security (CLS): Restricts access to specific sensitive columns. This can be combined with dynamic data masking, which obscures column values (e.g., showing only the last four digits of a number) for unauthorized users at query time.

Advanced security is achieved with VPC Service Controls, which create a logical security perimeter around BigQuery resources to block data movement to unauthorized locations. For datasets shared via an Analytics Hub listing, publishers can enable data egress restrictions. This setting prevents subscribers from copying, exporting, or creating new tables from the shared data, keeping it contained within the secure query environment.

All access and query activity is recorded in Cloud Audit Logs. This audit trail is vital for investigating access patterns, proving compliance with data sharing agreements, and detecting potential security incidents.

Key Facts

• BigQuery Analytics Hub enables secure, zero-copy data sharing by separating storage (controlled by the publisher) from compute (managed by the consumer).
• Authorized views and authorized datasets allow publishers to share queryable data interfaces without granting direct access to underlying source tables.
• Row-level security and column-level security enforce granular access controls within a table, filtering data based on user identity or masking sensitive values.
• Cost-sharing models like Requester Pays (Cloud Storage) and on-demand query billing (BigQuery) allocate data consumption costs to the consumer.
• VPC Service Controls establish a security perimeter to prevent data exfiltration, and data egress restrictions in Analytics Hub block subscribers from copying shared data out of the platform.