An enterprise data engineering team is building an event-driven, automated data processing pipeline on Google Cloud. The workflow must execute sequentially whenever new files arrive in Cloud Storage: Validate incoming event metadata and write an audit payload. Execute a containerized data-cleansing task implemented as a Cloud Run job, passing dynamic input file paths and execution parameters as environment variable overrides at runtime. Launch a downstream Dataflow batch pipeline using the outputs generated by the Cloud Run job. The entire orchestration layer must be completely serverless, scale to zero when idle with no persistent compute infrastructure, and provide built-in retry and state-handling capabilities. Which orchestration approach should the team implement?

Create a Cloud Workflows pipeline triggered by an Eventarc Cloud Storage event, using native connectors to invoke the Cloud Run job with container overrides and execute the Dataflow job.: What Cloud Workflows is Cloud Workflows is a fully managed, serverless orchestration platform that coordinates Google Cloud services, external APIs, and containerized workloads into reliable, automated state machines without requiring any underlying infrastructure management. How it meets the requirements Event-driven execution: It integrates natively with Eventarc, enabling instant workflow execution whenever new files or Pub/Sub messages are received. Dynamic parameter passing and container overrides: Using the built-in Cloud Run connector (googleapis.run.v1.namespaces.jobs.run), Workflows dynamically injects parameters such as input file locations and environment variables into the container via containerOverrides. Chaining serverless services and Dataflow: It coordinates the sequential execution of logging, containerized Cloud Run jobs, and Dataflow pipelines while managing state transitions and capturing step results. Serverless and pay-per-use: Cloud Workflows requires zero provisioned VM infrastructure, scales to zero when idle, and charges only for execution steps. Key advantages and capabilities Native API Connectors: Offers out-of-the-box connectors for Cloud Run, Dataflow, BigQuery, and Cloud Storage with automatic authentication and error handling. Dynamic runtime configuration: Passes state, JSON payloads, and dynamic execution arguments between heterogeneous pipeline steps. Configurable resilience: Provides native retry policies, exponential backoff, and exception handling directly within the YAML/JSON workflow definition. Zero operational overhead: Eliminates cluster maintenance, worker sizing, and software patching. Why this is optimal Unlike heavy, dedicated orchestrators like Apache Airflow or self-managed worker pools, Cloud Workflows paired with Eventarc delivers an entirely serverless, low-latency, and cost-effective solution tailored for coordinating microservices, container jobs, and managed data pipelines.

StudyCerts

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Serverless Orchestration Cloud Run Dataflow Workflows

Section: Scheduling and orchestrating jobs in a repeatable way

An enterprise data engineering team is building an event-driven, automated data processing pipeline on Google Cloud. The workflow must execute sequentially whenever new files arrive in Cloud Storage:

Validate incoming event metadata and write an audit payload.
Execute a containerized data-cleansing task implemented as a Cloud Run job, passing dynamic input file paths and execution parameters as environment variable overrides at runtime.
Launch a downstream Dataflow batch pipeline using the outputs generated by the Cloud Run job.

The entire orchestration layer must be completely serverless, scale to zero when idle with no persistent compute infrastructure, and provide built-in retry and state-handling capabilities.

Which orchestration approach should the team implement?

Create a Cloud Workflows pipeline triggered by an Eventarc Cloud Storage event, using native connectors to invoke the Cloud Run job with container overrides and execute the Dataflow job.

Configure a Managed Instance Group (MIG) with a custom task-farming agent that polls Pub/Sub topics to execute shell commands for Cloud Run and Dataflow.

Create a Dataproc Workflow Template with OrderedJob configurations to execute the container image and coordinate the Dataflow template submission.

Deploy an Apache Airflow environment on a self-managed Google Kubernetes Engine (GKE) cluster, and use BashOperator tasks to run gcloud CLI commands that trigger the Cloud Run job and Dataflow pipeline.

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Serverless Orchestration Cloud Run Dataflow Workflows

Section: Scheduling and orchestrating jobs in a repeatable way

An enterprise data engineering team is building an event-driven, automated data processing pipeline on Google Cloud. The workflow must execute sequentially whenever new files arrive in Cloud Storage:

Validate incoming event metadata and write an audit payload.
Execute a containerized data-cleansing task implemented as a Cloud Run job, passing dynamic input file paths and execution parameters as environment variable overrides at runtime.
Launch a downstream Dataflow batch pipeline using the outputs generated by the Cloud Run job.

The entire orchestration layer must be completely serverless, scale to zero when idle with no persistent compute infrastructure, and provide built-in retry and state-handling capabilities.

Which orchestration approach should the team implement?

Create a Cloud Workflows pipeline triggered by an Eventarc Cloud Storage event, using native connectors to invoke the Cloud Run job with container overrides and execute the Dataflow job.

Configure a Managed Instance Group (MIG) with a custom task-farming agent that polls Pub/Sub topics to execute shell commands for Cloud Run and Dataflow.

Create a Dataproc Workflow Template with OrderedJob configurations to execute the container image and coordinate the Dataflow template submission.

Explanations

What Cloud Workflows is

Cloud Workflows is a fully managed, serverless orchestration platform that coordinates Google Cloud services, external APIs, and containerized workloads into reliable, automated state machines without requiring any underlying infrastructure management.

How it meets the requirements

Event-driven execution: It integrates natively with Eventarc, enabling instant workflow execution whenever new files or Pub/Sub messages are received.
Dynamic parameter passing and container overrides: Using the built-in Cloud Run connector (googleapis.run.v1.namespaces.jobs.run), Workflows dynamically injects parameters such as input file locations and environment variables into the container via containerOverrides.
Chaining serverless services and Dataflow: It coordinates the sequential execution of logging, containerized Cloud Run jobs, and Dataflow pipelines while managing state transitions and capturing step results.
Serverless and pay-per-use: Cloud Workflows requires zero provisioned VM infrastructure, scales to zero when idle, and charges only for execution steps.

Key advantages and capabilities

Native API Connectors: Offers out-of-the-box connectors for Cloud Run, Dataflow, BigQuery, and Cloud Storage with automatic authentication and error handling.
Dynamic runtime configuration: Passes state, JSON payloads, and dynamic execution arguments between heterogeneous pipeline steps.
Configurable resilience: Provides native retry policies, exponential backoff, and exception handling directly within the YAML/JSON workflow definition.
Zero operational overhead: Eliminates cluster maintenance, worker sizing, and software patching.

Why this is optimal

Unlike heavy, dedicated orchestrators like Apache Airflow or self-managed worker pools, Cloud Workflows paired with Eventarc delivers an entirely serverless, low-latency, and cost-effective solution tailored for coordinating microservices, container jobs, and managed data pipelines.

View source documentation

Professional Cloud Data Engineer

Serverless Orchestration Cloud Run Dataflow Workflows

Professional Cloud Data Engineer

Serverless Orchestration Cloud Run Dataflow Workflows

What Cloud Workflows is

How it meets the requirements

Key advantages and capabilities

Why this is optimal

Related questions