Analytics Hub Exchanges, Listings and Shared Datasets

BigQuery (opens in a new tab) data sharing organizes collaboration into exchanges that hold listings, and listings that provision linked datasets for consumers. A data exchange is the organizational sharing boundary for private or public use, a listing is the structured published entry that describes and exposes a source dataset or table, and a linked dataset is the read-only consumer-side reference that queries the source. The provider project continues to host storage and governance while consumer projects attach through their own query execution, billing, and slot quotas. Regional availability, replication scope, and colocation rules determine where exchanges, listings, and linked datasets can operate and stay synchronized.

1.1Sharing Requirements and Exchange Boundaries

Sharing requirements determine whether an exchange must be private to specific projects, users, and groups or public for broader discovery. BigQuery supports cross-project sharing where different projects, users, and groups can query datasets hosted in other projects. Separation of query execution lets autonomous teams work within their own projects without affecting other users and projects by separating slot quotas and query billing from the projects that host the datasets.

Provider governance stays centralized while consumption scales out across many consumer projects. Storage resources are allocated as data is consumed and deallocated as data is removed or tables are dropped, so an exchange does not require pre-provisioned storage for each consumer. Use a private exchange when locality, compliance, or commercial controls require restricted membership, and use a public exchange pattern when the goal is broad discovery with centralized source control.

Location policy forms an additional exchange boundary. The Organization Policy Service can restrict the locations in which datasets can be created, which constrains where exchanges and source datasets may exist. BigQuery determines the location to run load, query, or extract jobs based on the datasets referenced in the request, and a query that accesses data stored in more than one location can be treated as a global query. Develop a data management plan for geographically managing data when regional storage resources such as BigQuery datasets or Cloud Storage buckets are used.

1.2Publishing Listings and Provisioning Linked Datasets

A listing publishes a structured reference to source data, and provisioning creates a linked dataset in the consumer project that resolves back to that source. Consumers query through the linked dataset from their own project context, so their jobs, quotas, and costs remain isolated from the provider. Colocate analysis workloads with the source dataset to preserve performance, because network distance and bandwidth directly affect federated and connector-based access.

Colocation constraints apply to both external sources exposed through listings and tools that consume linked datasets:

• Cloud SQL (opens in a new tab) federated queries require the Cloud SQL instance to be in the same location as the BigQuery dataset, with multi-region datasets paired to a single region in the same geographic area.
• Spanner federated queries require the Spanner instance to be in the same location as the BigQuery dataset, with multi-region datasets paired to a single region in the same geographic area.
• Bigtable external tables require the Bigtable instance to be in the same location as the BigQuery dataset, and multi-region dataset locations are not recommended because external query performance depends on minimal latency and optimal network bandwidth.
• Managed Service for Apache Spark clusters querying through the BigQuery connector should be colocated with the BigQuery dataset.
• Vertex AI Workbench instances querying through Jupyter notebooks should be colocated with the BigQuery dataset.

1.3Regional Availability, Replication, and Durability

Regional availability for shared data is controlled by dataset location and automatic replication across Google data centers. BigQuery replicates data depending on dataset location to maximize reliability and availability without requiring server, storage, or physical corruption management by the operator. Single-region and multi-region locations differ in replication scope and failure protection.

If multi-region storage is not available in a required region but a different region can be used, the BigQuery Data Transfer Service can copy a dataset to a different region. If only one region can be used, back up tables with BigQuery exports scheduled through Cloud Scheduler or Managed Airflow, using formats such as Avro with SNAPPY compression or JSON with GZIP compression while observing export limits. To reduce backup cost, store export files in Cloud Storage Nearline Storage or Coldline Storage and define a lifecycle policy to delete files after the retention period.

1.4Automatic Synchronization Boundaries and Recovery

Automatic synchronization boundaries define what BigQuery keeps current without action and what requires explicit backup or restore. Replication, column statistics gathered during load, and automatic storage optimization operate continuously without consuming consumer query slots. Point-in-time recovery through history is bounded: historical snapshots of tables can be queried for up to seven days and deleted tables can be restored within two days by using time travel with snapshot syntax.

Batch operations and snapshotting allow different backup strategies, so unchanged tables and partitions do not need frequent re-export. One export backup of the partition or table after the load or extract, transform, load operation finishes is enough. For append-only warehouses partitioned by date, create a copy of the required partitions in a new table before running an export, then transfer updated files with tools such as the gcloud storage command, accounting for egress charges and for late updates such as a 7-day return window that can change recent partitions.

Exporting BigQuery data to columnar storage in an on-premises disaster recovery location follows an ordered flow:

1. Send source data to processing: BigQuery data flows from the source tables to an Apache Spark job on Managed Service for Apache Spark, which permits schema evolution.
2. Write processed files: The processed files flow from the Spark job to Cloud Storage in high-compression columnar formats such as Parquet and ORC for interoperability with engines such as Hive and Presto.
3. Connect networks: Traffic flows from the Virtual Private Cloud network to the on-premises network over Cloud Interconnect.
4. Transfer to recovery storage: The files flow from Cloud Storage through the Spark job path to the on-premises disaster recovery location.

Enforcing security on shared assets means combining project separation, granular Identity and Access Management (IAM), network perimeters, and fine-grained data controls. The shared data project holds sanitized information copied from the internal project, and the internal project owns the service account that updates the shared data. External access is managed through groups with viewer permission, while VPC Service Controls, authorized views, and row-level and column-level security enforce governance on the shared datasets.

2.1Identity and Access Management for Shared Data Projects

Identity and Access Management (IAM) (opens in a new tab) in BigQuery controls who can publish, administer, and consume shared datasets through the resource hierarchy of organization, folder, project, and resource. Datasets are child resources of projects, and tables inherit permissions from the dataset that contains them, so organization and project roles govern the ability to run jobs or manage projects while dataset roles govern the ability to access or modify data. To grant access, assign roles to a user, group, or service account, and manage external parties through external user groups where group membership is the administrative control and members receive viewer permission to the shared dataset.

IAM provides three role types that support least-privilege sharing:

• Basic roles: Owner, Editor, and Viewer roles applied at the project or service resource level.
• Predefined roles: Granular roles for common patterns, such as BigQuery Data Editor and Analytics Hub Admin for sharing administration.
• Custom roles: User-specified collections of permissions for specialized publisher or subscriber needs.

For stronger governance, combine dataset and table-level access control with IAM policies with conditions, principal access boundary (PAB) policies to limit what a principal can reach, and Privileged Access Manager for just-in-time temporary elevation with audit logs. Service accounts are limited to a soft quota of 100 service accounts per project, and BigQuery concurrency defaults to 100 queries per project that issues queries, so large multi-tenant sharing designs must plan quota and project scale.

2.2VPC Service Controls Perimeters and Bridges for External Sharing

VPC Service Controls creates a security boundary around sharing projects that prevents accidental exposure of BigQuery datasets outside internal projects and controls data exfiltration. The recommended design uses two perimeters with different enforcement: the internal data perimeter protects core data assets and enforces access to BigQuery, while the externally shared data perimeter hosts datasets that can be shared with outside organizations and disables enforcement of access to BigQuery. Data flows from the protected internal data projects across a perimeter bridge into the external data share projects, and only after the bridge allows that egress.

The recommended bridge is the internal-to-external data bridge, which allows the more protected internal projects to egress data into external share projects. VPC Service Controls is limited to 100 perimeters including bridges per organization and 10,000 projects within service perimeters organization-wide. When a project-per-tenant sharing design would exceed that scale, choose a dataset-per-tenant design instead.

2.3Authorized Views Compared With Subset Tables for Tenant Isolation

Authorized views and tenant-specific subset tables are the two mechanisms for giving each tenant access to only a subset of a large fact table without granting access to the base table. An authorized view is a view authorized on the base dataset so specified users can query the view without having read access to the underlying tables, while a subset table is a separate physical copy filtered for one tenant. Choose authorized views when instant redesign and centralized logging matter, and choose subset tables when per-tenant performance tuning or regional placement matters.

2.4Row-Level and Column-Level Security on Shared Datasets

Row-level and column-level security restricts which rows and columns a subscriber can see inside a dataset they are otherwise allowed to query. Column-level access control provides fine-grained access to sensitive columns using policy tags or type-based classification, column-level dynamic data masking selectively obscures column data for groups of users while still allowing access to the column, and row-level security filters data so access to specific rows depends on qualifying user conditions. Table-level access control determines which users, groups, and service accounts can access a specific table or view without granting access to the complete dataset, which is essential when only some tables in a shared dataset are approved for external use.

Policy tags implement a hierarchy for confidential columns in the warehouse. For example, a 3_Confidential tag covers very sensitive information such as credit card numbers, a 2_Private tag covers sensitive personally identifiable information such as first name, and a 1_Sensitive tag covers non-public data such as credit limit. Users with access to 3_Confidential also gain access to 2_Private and 1_Sensitive columns, users with 2_Private also gain 1_Sensitive but not 3_Confidential, and users with only 1_Sensitive gain neither higher tier. Anything that is not tagged is available to all users who have access to the warehouse, and a data analyst must have the Fine-Grained Reader role to read tagged columns, so data remains unreadable until access is explicitly granted even after re-identification.

2.5Encryption and Egress Controls for Shared Assets

Encryption and egress controls protect shared data at rest, in transit, and at query time. BigQuery encrypts all data at rest and in transit by default, and governance designs layer customer-managed encryption keys (CMEK) in Cloud Key Management Service (Cloud KMS) with Cloud HSM when the organization must control key location, rotation, and use. Column-level encryption for tenant-sensitive fields uses AEAD encryption keys from the open source Tink library, which are compatible with BigQuery AEAD functions, so the hosting organization can store and process data without reading the sensitive contents.

Data egress controls automatically prevent data clean room (opens in a new tab) subscribers from copying and exporting raw data, and contributors can add controls that prevent copying and exporting query results. Analysis rules can be set only on views, not on tables or materialized views, so sharing a table directly gives subscribers raw data access. Query templates built on table-valued functions (TVFs) let owners share predefined queries without sharing the underlying tables and views, which helps block unauthorized attempts to extract raw data.

BigQuery sharing (opens in a new tab) (formerly Analytics Hub) is a zero-copy data exchange platform where a publisher shares a dataset in place and a subscriber queries it through a live pointer. The separation of compute and storage in the BigQuery architecture makes this split possible without replicating data. The publisher project retains the stored bytes while each subscriber project supplies the compute for the queries it runs against its linked dataset.

3.1Billing Split Between Publishers and Subscribers

The billing split for BigQuery sharing follows storage versus compute, with no additional charge for managing data exchanges or listings. The publisher pays for storing the shared BigQuery dataset in its project, while the subscriber pays for the queries it runs against the shared data. For streaming sharing, the publisher of a shared Pub/Sub (opens in a new tab) topic pays for data written plus network egress, while the subscriber pays for data read plus network egress.

This split means a popular shared dataset does not increase the publisher query bill when many vendors or partners query it. The publisher cost grows with stored bytes and Pub/Sub publish volume, while each subscriber cost grows with bytes scanned and subscribe volume in its own project. Commercial terms for paid data products are handled separately through Cloud Marketplace orders.

3.2Query Pricing and Cost Controls for Subscribers

Subscriber query cost for shared BigQuery datasets uses standard BigQuery analytics pricing in the subscriber project. With on-demand pricing, the subscriber pays for data scanned by its queries and operates within a fixed per-project query-processing capacity. With capacity-based pricing, the subscriber purchases dedicated query-processing capacity through reservations and runs queries against that capacity.

BigQuery enforces project-level quotas on running queries, so subscriber concurrency is governed in the subscriber project rather than the publisher project. As queries start and finish, BigQuery redistributes resources fairly between new and running queries, which keeps performance dependent on the number of concurrent queries rather than submission order. Subscribers can control spend with custom quotas, custom cost controls, and billing alerts that limit or warn on query usage.

3.3Optimizing Query Performance on Shared Data

Query performance on a linked dataset is the performance of a query that reads from the publisher shared dataset but executes in the subscriber project. Subscribers can view the query plan in the Google Cloud console and can request execution details with INFORMATION_SCHEMA.JOBS views or the jobs.get REST API method. The query plan describes query stages and steps, which helps locate expensive behavior such as a stage that writes much more output than other stages and signals that filtering should happen earlier in the query.

Storage format and layout have a large effect on bytes scanned and speed when shared tables use BigLake tables over Cloud Storage. Columnar formats such as Parquet or ORC generally perform much better for analytic queries than JSON or CSV formats. A Hive partitioned layout lets queries prune partitions and speeds up queries that filter on partitioning columns, while the number of files and file size must also be balanced during design.

If queries on BigLake tables cannot meet service-level agreement requirements and cannot be tuned, the documented remedy is to convert the exposed data to BigQuery storage and redefine the authorized views to use the BigQuery tables. This change generally does not disrupt consumers or require query rewrites because the linked dataset still points to the same logical view. Queries in BigQuery storage can then use optimizations that are not possible with BigLake tables, such as materialized views with different partitioning and clustering than the base tables and use of the BigQuery BI Engine.

3.4Auditing and Monitoring Subscriber Access

Publisher visibility into shared use comes from provider usage metrics that track activity against each shared dataset. Publishers can track jobs that run against the shared dataset, consumption details by subscriber projects and organizations, and total rows and bytes processed. These metrics show the number of jobs executed, total bytes scanned, and subscribers for each organization, which supports chargeback analysis, popularity tracking, and governance review.

Subscribers and publishers can audit detailed behavior with Cloud Logging, BigQuery audit logs, and INFORMATION_SCHEMA views. Cloud Logging provides a detailed record of user activity and system events that helps enforce data governance policies and identify potential security risks. BigQuery audit logs and INFORMATION_SCHEMA.JOBS views support analysis of query behavior, including who ran which job, how much data each job scanned, and how execution was distributed across stages.

Monitoring for shared Pub/Sub topics follows a different path than BigQuery datasets. Usage metrics for shared Pub/Sub topics and subscriptions appear in the Metrics dashboard and in Cloud Monitoring, but they are not available in INFORMATION_SCHEMA.SHARED_DATASET_USAGE. Billing usage metrics for Cloud Marketplace-integrated listings are also not captured in provider usage metrics or in INFORMATION_SCHEMA views.

Publishers monitoring streaming sharing in Cloud Monitoring can review signals such as:

• Publish message count from the shared topic publisher project to subscribers
• Publish requests against the shared topic
• Publish throughput in bytes from publisher to subscriber delivery
• Top 5 subscriptions consuming the shared topic
• Ingested bytes into the shared topic

These signals connect publisher write activity to subscriber read activity without duplicating topic or message data, since subscribing creates a linked Pub/Sub subscription in the subscriber project that reads from the source topic.

Key Facts

• The publisher of a shared BigQuery dataset pays for data storage while the subscriber pays for queries run against the shared data in the subscriber project.
• An authorized view lets users query a view without read access to underlying tables and is limited to 2500 authorized resources per dataset, while subset tables are separate filtered copies with no dataset or table count limits.
• VPC Service Controls sharing is limited to 100 perimeters including bridges per organization and 10,000 projects within service perimeters organization-wide.
• Historical snapshots of BigQuery tables can be queried for up to seven days and deleted tables can be restored within two days using time travel.
• Single-region BigQuery storage replicates across zones in the same region, while multi-region storage replicates across different regions to protect against zone or region unavailability.