professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
An enterprise maintains hundreds of terabytes of clickstream logs stored in Amazon S3 within the us-east-1 region. The data analytics team needs to combine and analyze this remote data alongside core customer demographic tables residing natively in BigQuery in the US multi-region.
The solution must satisfy the following constraints:
Which architecture should the team implement?
Configure Storage Transfer Service to execute hourly scheduled batch syncs of raw S3 clickstream files into a Cloud Storage staging bucket, and trigger a Dataflow pipeline to load the data into BigQuery.
Deploy a Google Cloud Dataproc cluster in Google Cloud configured with the Hadoop S3 connector to read the remote S3 data directly into Spark memory and join it with BigQuery.
Execute BigQuery federated queries using the EXTERNAL_QUERY function over a JDBC connection pointing directly to the Amazon S3 bucket to process records on the fly.
Create a BigQuery Omni external connection using an AWS IAM role for access delegation, define BigLake external tables referencing the S3 datasets, and execute cross-cloud queries that filter and aggregate data in the AWS region before transferring the result set to Google Cloud.
Configure Storage Transfer Service to execute hourly scheduled batch syncs of raw S3 clickstream files into a Cloud Storage staging bucket, and trigger a Dataflow pipeline to load the data into BigQuery.
Deploy a Google Cloud Dataproc cluster in Google Cloud configured with the Hadoop S3 connector to read the remote S3 data directly into Spark memory and join it with BigQuery.
Execute BigQuery federated queries using the EXTERNAL_QUERY function over a JDBC connection pointing directly to the Amazon S3 bucket to process records on the fly.
Create a BigQuery Omni external connection using an AWS IAM role for access delegation, define BigLake external tables referencing the S3 datasets, and execute cross-cloud queries that filter and aggregate data in the AWS region before transferring the result set to Google Cloud.
BigQuery Omni is an extension of Google Cloud's serverless analytics engine that runs the BigQuery query engine directly inside other cloud environments, such as Amazon Web Services (AWS) and Microsoft Azure. Combined with BigLake, it allows users to create external tables over structured and semi-structured data stored in Amazon S3 or Azure Blob Storage.
StudyCerts
StudyCertsus-east-1). When performing cross-cloud queries, filtering and initial aggregations are executed within the BigQuery Omni AWS compute layer using local slots, transferring only the necessary intermediate or final result rows/columns (via a temporary table mechanism) over the network to the BigQuery region in Google Cloud.bq CLI, APIs, BI Engine) seamlessly.This approach avoids raw bulk data replication across cloud boundaries, dramatically lowering network transit fees and latency while keeping governance and security centralized in Google Cloud.