An enterprise maintains hundreds of terabytes of clickstream logs stored in Amazon S3 within the us-east-1 region. The data analytics team needs to combine and analyze this remote data alongside core customer demographic tables residing natively in BigQuery in the US multi-region. The solution must satisfy the following constraints: Minimize cross-cloud egress charges and network bandwidth consumption. Eliminate the operational overhead of deploying, managing, or scheduling ETL/ELT pipelines. Enforce unified, fine-grained access governance across cloud providers using standard identity delegation. Which architecture should the team implement?

Create a BigQuery Omni external connection using an AWS IAM role for access delegation, define BigLake external tables referencing the S3 datasets, and execute cross-cloud queries that filter and aggregate data in the AWS region before transferring the result set to Google Cloud.: What BigQuery Omni and BigLake are BigQuery Omni is an extension of Google Cloud's serverless analytics engine that runs the BigQuery query engine directly inside other cloud environments, such as Amazon Web Services (AWS) and Microsoft Azure. Combined with BigLake, it allows users to create external tables over structured and semi-structured data stored in Amazon S3 or Azure Blob Storage. How it meets the requirements Minimizes egress costs: BigQuery Omni executes computation locally where the data resides in AWS (us-east-1). When performing cross-cloud queries, filtering and initial aggregations are executed within the BigQuery Omni AWS compute layer using local slots, transferring only the necessary intermediate or final result rows/columns (via a temporary table mechanism) over the network to the BigQuery region in Google Cloud. Eliminates ETL/ELT operational overhead: There is no need to stand up data migration infrastructure, replication tasks, or intermediate pipeline schedulers. Analytics can be executed directly across multi-cloud storage using standard GoogleSQL syntax. Enforces secure access delegation: BigQuery Omni leverages native AWS IAM roles with trust policies tied to Google Cloud service accounts. This allows administrators to manage access delegation without sharing permanent AWS access keys or managing complex third-party credentials. Key advantages and capabilities Decoupled compute and storage: Maintains BigQuery's core architectural separation while executing compute in external clouds. Fine-grained access controls: Enables centralized table-level, column-level, and row-level security through BigLake without moving raw data files. Pushdown optimization: Filters and projections are applied locally before cross-cloud data movement occurs. Standard SQL tooling: Analysts can use familiar BigQuery interfaces (Google Cloud console, bq CLI, APIs, BI Engine) seamlessly. Why this is optimal This approach avoids raw bulk data replication across cloud boundaries, dramatically lowering network transit fees and latency while keeping governance and security centralized in Google Cloud.

StudyCerts

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Bigquery Omni Federated Cross Cloud Analytics

Section: Designing for data and application portability (e.g., multi-cloud and data residency requirements)

An enterprise maintains hundreds of terabytes of clickstream logs stored in Amazon S3 within the us-east-1 region. The data analytics team needs to combine and analyze this remote data alongside core customer demographic tables residing natively in BigQuery in the US multi-region.

The solution must satisfy the following constraints:

Minimize cross-cloud egress charges and network bandwidth consumption.
Eliminate the operational overhead of deploying, managing, or scheduling ETL/ELT pipelines.
Enforce unified, fine-grained access governance across cloud providers using standard identity delegation.

Which architecture should the team implement?

Execute BigQuery federated queries using the EXTERNAL_QUERY function over a JDBC connection pointing directly to the Amazon S3 bucket to process records on the fly.

Configure Storage Transfer Service to execute hourly scheduled batch syncs of raw S3 clickstream files into a Cloud Storage staging bucket, and trigger a Dataflow pipeline to load the data into BigQuery.

Create a BigQuery Omni external connection using an AWS IAM role for access delegation, define BigLake external tables referencing the S3 datasets, and execute cross-cloud queries that filter and aggregate data in the AWS region before transferring the result set to Google Cloud.

Deploy a Google Cloud Dataproc cluster in Google Cloud configured with the Hadoop S3 connector to read the remote S3 data directly into Spark memory and join it with BigQuery.

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Bigquery Omni Federated Cross Cloud Analytics

Section: Designing for data and application portability (e.g., multi-cloud and data residency requirements)

The solution must satisfy the following constraints:

Minimize cross-cloud egress charges and network bandwidth consumption.
Eliminate the operational overhead of deploying, managing, or scheduling ETL/ELT pipelines.
Enforce unified, fine-grained access governance across cloud providers using standard identity delegation.

Which architecture should the team implement?

Execute BigQuery federated queries using the EXTERNAL_QUERY function over a JDBC connection pointing directly to the Amazon S3 bucket to process records on the fly.

Deploy a Google Cloud Dataproc cluster in Google Cloud configured with the Hadoop S3 connector to read the remote S3 data directly into Spark memory and join it with BigQuery.

Explanations

What BigQuery Omni and BigLake are

BigQuery Omni is an extension of Google Cloud's serverless analytics engine that runs the BigQuery query engine directly inside other cloud environments, such as Amazon Web Services (AWS) and Microsoft Azure. Combined with BigLake, it allows users to create external tables over structured and semi-structured data stored in Amazon S3 or Azure Blob Storage.

How it meets the requirements

Minimizes egress costs: BigQuery Omni executes computation locally where the data resides in AWS (us-east-1). When performing cross-cloud queries, filtering and initial aggregations are executed within the BigQuery Omni AWS compute layer using local slots, transferring only the necessary intermediate or final result rows/columns (via a temporary table mechanism) over the network to the BigQuery region in Google Cloud.
Eliminates ETL/ELT operational overhead: There is no need to stand up data migration infrastructure, replication tasks, or intermediate pipeline schedulers. Analytics can be executed directly across multi-cloud storage using standard GoogleSQL syntax.
Enforces secure access delegation: BigQuery Omni leverages native AWS IAM roles with trust policies tied to Google Cloud service accounts. This allows administrators to manage access delegation without sharing permanent AWS access keys or managing complex third-party credentials.

Key advantages and capabilities

Decoupled compute and storage: Maintains BigQuery's core architectural separation while executing compute in external clouds.
Fine-grained access controls: Enables centralized table-level, column-level, and row-level security through BigLake without moving raw data files.
Pushdown optimization: Filters and projections are applied locally before cross-cloud data movement occurs.
Standard SQL tooling: Analysts can use familiar BigQuery interfaces (Google Cloud console, bq CLI, APIs, BI Engine) seamlessly.

Why this is optimal

This approach avoids raw bulk data replication across cloud boundaries, dramatically lowering network transit fees and latency while keeping governance and security centralized in Google Cloud.

View source documentation

Professional Cloud Data Engineer

Bigquery Omni Federated Cross Cloud Analytics

Professional Cloud Data Engineer

Bigquery Omni Federated Cross Cloud Analytics

What BigQuery Omni and BigLake are

How it meets the requirements

Key advantages and capabilities

Why this is optimal

Related questions