An enterprise data engineering team is building an event-driven, automated data processing pipeline on Google Cloud. The workflow must execute sequentially whenever new files arrive in Cloud Storage: Validate incoming event metadata and write an audit payload. Execute a containerized data-cleansing task implemented as a Cloud Run job, passing dynamic input file paths and execution parameters as environment variable overrides at runtime. Launch a downstream Dataflow batch pipeline using the outputs generated by the Cloud Run job. The entire orchestration layer must be completely serverless, scale to zero when idle with no persistent compute infrastructure, and provide built-in retry and state-handling capabilities. Which orchestration approach should the team implement?

Create a Cloud Workflows pipeline triggered by an Eventarc Cloud Storage event, using native connectors to invoke the Cloud Run job with container overrides and execute the Dataflow job.: ### What Cloud Workflows is **Cloud Workflows** is a fully managed, serverless orchestration platform that coordinates Google Cloud services, external APIs, and containerized workloads into reliable, automated state machines without requiring any underlying infrastructure management. ### How it meets the requirements - **Event-driven execution**: It integrates natively with **Eventarc**, enabling instant workflow execution whenever new files or Pub/Sub messages are received. - **Dynamic parameter passing and container overrides**: Using the built-in Cloud Run connector (`googleapis.run.v1.namespaces.jobs.run`), Workflows dynamically injects parameters such as input file locations and environment variables into the container via `containerOverrides`. - **Chaining serverless services and Dataflow**: It coordinates the sequential execution of logging, containerized Cloud Run jobs, and Dataflow pipelines while managing state transitions and capturing step results. - **Serverless and pay-per-use**: Cloud Workflows requires zero provisioned VM infrastructure, scales to zero when idle, and charges only for execution steps. ### Key advantages and capabilities - **Native API Connectors**: Offers out-of-the-box connectors for **Cloud Run**, **Dataflow**, **BigQuery**, and **Cloud Storage** with automatic authentication and error handling. - **Dynamic runtime configuration**: Passes state, JSON payloads, and dynamic execution arguments between heterogeneous pipeline steps. - **Configurable resilience**: Provides native retry policies, exponential backoff, and exception handling directly within the YAML/JSON workflow definition. - **Zero operational overhead**: Eliminates cluster maintenance, worker sizing, and software patching. ### Why this is optimal Unlike heavy, dedicated orchestrators like Apache Airflow or self-managed worker pools, **Cloud Workflows** paired with **Eventarc** delivers an entirely serverless, low-latency, and cost-effective solution tailored for coordinating microservices, container jobs, and managed data pipelines.

Serverless Orchestration Cloud Run Dataflow Workflows | professional-cloud-data-engineer

Serverless Orchestration Cloud Run Dataflow Workflows

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data engineering team is building an event-driven, automated data processing pipeline on Google Cloud. The workflow must execute sequentially whenever new files arrive in Cloud Storage:

Validate incoming event metadata and write an audit payload.
Execute a containerized data-cleansing task implemented as a Cloud Run job, passing dynamic input file paths and execution parameters as environment variable overrides at runtime.
Launch a downstream Dataflow batch pipeline using the outputs generated by the Cloud Run job.

The entire orchestration layer must be completely serverless, scale to zero when idle with no persistent compute infrastructure, and provide built-in retry and state-handling capabilities.

Which orchestration approach should the team implement?

Deploy an Apache Airflow environment on a self-managed Google Kubernetes Engine (GKE) cluster, and use BashOperator tasks to run gcloud CLI commands that trigger the Cloud Run job and Dataflow pipeline.

Create a Cloud Workflows pipeline triggered by an Eventarc Cloud Storage event, using native connectors to invoke the Cloud Run job with container overrides and execute the Dataflow job.

Create a Dataproc Workflow Template with OrderedJob configurations to execute the container image and coordinate the Dataflow template submission.

Configure a Managed Instance Group (MIG) with a custom task-farming agent that polls Pub/Sub topics to execute shell commands for Cloud Run and Dataflow.

Professional Cloud Data Engineer

Serverless Orchestration Cloud Run Dataflow Workflows

Related questions