A financial analytics company is migrating its daily batch data pipeline to Google Cloud. The workload has the following characteristics and constraints: The current pipeline consists of extensive data transformations written in Apache Spark (PySpark and Scala). The batch process runs once per day during an off-peak two-hour window to ingest transaction files from Cloud Storage and write structured results to BigQuery. The engineering team wants to minimize code refactoring effort and avoid the maintenance and cost of idle infrastructure during non-processing hours. Which execution framework and architecture should the company select?

Execute the existing Spark jobs on ephemeral Cloud Dataproc clusters orchestrated by Cloud Composer, creating clusters on demand and deleting them immediately after job completion: What Ephemeral Cloud Dataproc Clusters Are Cloud Dataproc is a fully managed service for executing open-source data processing engines, such as Apache Spark, Apache Hadoop, and Hive. An ephemeral cluster pattern creates dedicated compute resources specifically for the duration of a discrete workflow and terminates the cluster immediately upon job completion, persisting data externally in Cloud Storage and BigQuery. How It Meets the Requirements Preserves existing Apache Spark codebase: Dataproc natively supports PySpark, Scala, and Java Spark jobs without requiring complex code rewrites or logic translation. Eliminates idle infrastructure costs: By creating the cluster at the start of the daily batch window and deleting it right after the Spark job finishes, the organization only pays for compute resources consumed during the active two-hour execution window. Seamless orchestration: Cloud Composer (managed Apache Airflow) natively supports operators to provision Dataproc clusters, submit batch jobs, and tear down clusters automatically in a DAG workflow. Key Advantages and Capabilities Native ecosystem compatibility: Runs existing Spark code, libraries, and dependencies out of the box. Rapid provisioning: Dataproc clusters spin up and down in approximately 90 seconds, making ephemeral architectures efficient for short batch jobs. Decoupled compute and storage: Utilizes the Google Cloud Storage connector and BigQuery connector, allowing compute nodes to be entirely stateless. Cost governance: Leverages per-second billing, custom VM machine types, and optional preemptible or Spot VMs to reduce total batch run costs. Why This Is Optimal Compared to refactoring pipelines into Apache Beam or BigQuery SQL, ephemeral Dataproc allows the team to migrate legacy Spark applications with minimal operational friction while completely eliminating the continuous cost overhead of persistent servers.

Ephemeral Dataproc Spark Batch Pipeline | Professional Cloud Data Engineer

AdvancedProfessional Cloud Data Engineer

A data engineering team manages a daily batch pipeline orchestrated by Cloud Composer that processes terabytes of transactional log files from Cloud Storage into partitioned BigQuery tables. Recently, transient worker pod evictions and network timeouts have triggered task retries, resulting in duplicate data rows in BigQuery and failed task executions caused by exhausted worker disk space. You need to re-architect the workflow orchestration to ensure fault tolerance, prevent duplicate records during task retries (idempotency), and prevent worker resource exhaustion. Which architectural strategy should you implement?

Batch

AdvancedProfessional Cloud Data Engineer

A data engineering team maintains a nightly batch Apache Beam pipeline running on Cloud Dataflow that aggregates multi-terabyte transactional logs. The pipeline groups data by merchant ID using a GroupByKey transform and writes aggregated metrics into BigQuery. Monitoring shows the following bottlenecks during job execution: The GroupByKey transform suffers from severe stragglers due to extreme key skew caused by a few high-volume merchants, leaving most worker vCPUs idle while a single worker runs out of memory. Worker nodes encounter severe local Persistent Disk I/O bottlenecks and disk thrashing during intermediate shuffle phases. Compute costs for the nightly batch window are exceeding the allocated budget. Which combination of optimization strategies should the team implement to resolve the performance bottlenecks and minimize execution costs?

Batch

AdvancedProfessional Cloud Data Engineer

An enterprise data engineering team is designing a cost-optimized storage architecture on Cloud Storage for large-scale analytical training datasets and compliance logs. The data lifecycle and operational requirements are as follows: Ingestion & Active Processing (Days 1–30): Newly ingested datasets are actively read, transformed, and queried multiple times per day by data pipelines. Periodic Validation (Days 31–90): Datasets are accessed at most once per month for model validation and intermittent quarterly reporting. Long-term Compliance (Day 91 to 7 Years): Data must be retained for regulatory compliance for seven years, accessed less than once a year, but any ad-hoc audit queries require sub-second (millisecond) retrieval latency. Cost Governance: The strategy must minimize total storage costs while strictly avoiding early deletion fees and premature class transition penalties. Which Object Lifecycle Management policy should the team configure to meet these requirements?

Planning for storage costs and performance

AdvancedProfessional Cloud Data Engineer

An enterprise data platform ingests streaming customer interaction records and log files continuously into Cloud Storage, while also running batch analytical transformations in BigQuery. The security and compliance team has established the following requirements for sensitive data governance: Scan incoming and existing data for standard personally identifiable information (PII) including email addresses and phone numbers. Detect internal customer account identifiers that adhere to a specific pattern (for example, ACC-[0-9]{8}). Detect sensitive keywords against a managed master list containing over 500,000 proprietary customer account aliases stored in Cloud Storage. Maintain centralized, reusable inspection definitions across all continuous ingestion pipelines, storage scanners, and BigQuery batch jobs. Store detailed finding metadata, including confidence levels and location coordinates, for automated compliance auditing. Which architecture and configuration should the data engineering team implement using Sensitive Data Protection (Cloud DLP)?

Privacy (e.g., strategies to handle personally identifiable information)

AdvancedProfessional Cloud Data Engineer

An enterprise data engineering team is configuring a centralized Data Studio (Looker Studio) reporting dashboard connected to BigQuery. The dashboard will be shared with both internal business analysts who have report edit permissions and external stakeholders who have view-only access. The solution must satisfy the following governance and operational requirements: Report editors must be prevented from modifying the underlying data source connection parameters or viewing raw connection configurations. Data access must not break if the original author of the report or data source leaves the organization. External viewers must be able to view the aggregated dashboard data without requiring direct BigQuery IAM dataset permissions granted to their individual identities. Stakeholders with view access must be restricted from downloading, printing, or copying the dashboard data. Which configuration should the data engineering team implement?

Publishing reports and visualizations

AdvancedProfessional Cloud Security Engineer

An enterprise requires on-premises workloads to privately access Google Cloud APIs and services without routing traffic over the public internet. The on-premises data center is connected to a Google Cloud Virtual Private Cloud (VPC) network via Cloud Interconnect. To comply with security policies, the API communication must be restricted to services supported by VPC Service Controls. Which combination of DNS and routing configurations should the enterprise implement?

Establishing private connectivity between VPC and Google APIs (Private Google Access, Private Google Access for on-premises hosts, restricted Google access, Private Service Connect).

Professional Cloud Data Engineer

Ephemeral Dataproc Spark Batch Pipeline

Related questions

Professional Cloud Data Engineer

Ephemeral Dataproc Spark Batch Pipeline

What Ephemeral Cloud Dataproc Clusters Are

How It Meets the Requirements

Key Advantages and Capabilities

Why This Is Optimal

Related questions

What Ephemeral Cloud Dataproc Clusters Are

How It Meets the Requirements

Key Advantages and Capabilities

Why This Is Optimal