Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Concept
pipeline
Topic in 2 exam domains
- pipeline · Concept
- pipeline · Feature
- Pipeline · Feature · BigQuery DataFrames pipeline class
- PIPELINE · Concept
- Pipeline · Concept · BigQuery resource
Also called: Eventarc Advanced pipeline, new pipeline, pipeline container, pipeline-level configuration, processing pipeline, sub-pipeline
A processing mechanism that configures a target destination for matched events and can transform them before delivery.
Where this appears on the exam
2 subdomains in 1 of 5 domains
Related topics
FoundationsBuilding blocksThis topicAppliedCustomer-Managed Encryption KeysFeatureApplication Default CredentialsFeatureCloud StorageServiceSecret ManagerServiceGKEServiceWorkflowsServicePub/SubServiceCloud BuildServiceCloud SQLService
pipelineConcept
Cloud RunServiceregionConceptPoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Cloud Run works with pipeline From the official documentation: “You can use pipelines to filter, transform, and deliver events to target services such as Cloud Run and Workflows.”
- Customer-Managed Encryption Keys secures pipeline From the official documentation: “When you enable CMEK for an Eventarc Advanced pipeline, all messages that pass through the pipeline are fully encrypted with that CMEK key.”
- pipeline configures region From the official documentation: “All of the core data processing for Dataflow happens in the region that is specified in the pipeline code.”
- pipeline works with Workflows From the official documentation: “You can use pipelines to filter, transform, and deliver events to target services such as Cloud Run and Workflows.”
- Secret Manager secures pipeline From the official documentation: “- **Secure your secrets:** Never hardcode passwords, API keys, or database credentials in your Apache Beam code or pipeline options. Access them dynamically during the `Setup` or `StartBundle` phases of your pipeline using Secret Manager.”
- Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
- Cloud Run depends on Pub/Sub From the official documentation: “Cloud Run function triggered by Pub/Sub to fetch HYPR MFA logs and write to GCS.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
- Cloud Build works with Cloud Storage From the official documentation: “Cloud Build then stores Terraform files and the Terraform State File in a Cloud Storage bucket.”
- Cloud Run works with Cloud SQL From the official documentation: “Connecting from Cloud Run (fully managed) to Cloud SQL”
- Cloud SQL works with GKE From the official documentation: “The app uses Cloud SQL, Vertex AI, and either Google Kubernetes Engine (GKE) or Cloud Run.”
- Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Pub/Sub works with Secret Manager From the official documentation: “Receive notifications via Pub/Sub when Secret Manager resources change.”
- Application Default Credentials works with Secret Manager From the official documentation: “To authenticate to Secret Manager, set up Application Default Credentials.”
- Cloud Build works with Cloud Run From the official documentation: “Cloud Build - create container images and deploy to Cloud Run.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Application Default Credentials works with Cloud Run From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Application Default Credentials works with Workflows From the official documentation: “To authenticate to Workflows, set up Application Default Credentials.”
- Cloud Build works with Cloud SQL From the official documentation: “This page contains information and examples for connecting to a Cloud SQL instance from a service running in Cloud Build.”
- Cloud Build works with Secret Manager From the official documentation: “The secret is configured to be accessible by Cloud Run and Cloud Build.”
- Cloud Run works with Cloud Storage From the official documentation: “Receive events from Cloud Storage in a Cloud Run service”
- Cloud Run is an alternative to GKE From the official documentation: “For your containerized websites, you can also deploy them to [Cloud Run](/run) in addition to [using GKE](#kubernetes-engine).”
- Cloud Run works with Secret Manager From the official documentation: “The secret is configured to be accessible by Cloud Run and Cloud Build.”
- Cloud Run works with Workflows From the official documentation: “Use Workflows with Cloud Run and Cloud Run functions tutorial”
- Cloud SQL configures region From the official documentation: “When you create a Cloud SQL instance, you choose a [region](/docs/geography-and-regions) where the instance and its data are stored.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- GKE works with Secret Manager From the official documentation: “To use Secret Manager with workloads running on Compute Engine or GKE, the underlying instance or node must have the `cloud-platform` OAuth scope.”
- Pub/Sub works with Workflows From the official documentation: “Pub/Sub can be integrated into multistep serverless [Workflows](/workflows) declaratively.”
- Application Default Credentials works with Cloud SQL From the official documentation: “Application Default Credentials are part of the client libraries you can use to access Cloud SQL.”
- Application Default Credentials configures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see [Set up authentication for client libraries](/storage/docs/authentication#client-libs).”
- Cloud Build works with GKE From the official documentation: “Remove the permission granted to Cloud Build to connect to GKE.”
- Cloud Build depends on Secret Manager From the official documentation: “Previously, for GitLab Enterprise and Bitbucket Data Center connections, when Secret Manager secrets were retrieved, permissions were checked against the Cloud Build service agent (P4SA) credentials only.”
- Cloud Run depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Cloud Run works with GKE From the official documentation: “Leverage Google Kubernetes Engine (GKE) and Cloud Run to deploy applications to the Google Cloud, optimizing costs, performance, and management overhead.”
- Cloud Run works with Pub/Sub From the official documentation: “A Cloud Run app receives the Pub/Sub message.”
- Cloud SQL works with Secret Manager From the official documentation: “This page describes four use cases for using Secret Manager to manage secrets with Cloud SQL:”
- Customer-Managed Encryption Keys depends on Cloud Storage From the official documentation: “If used for storing Customer-managed Encryption Keys (CMEK): Cloud Storage (See [Cloud Storage pricing](/storage/pricing).)”
- Customer-Managed Encryption Keys secures GKE From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Customer-Managed Encryption Keys works with Secret Manager From the official documentation: “In Secret Manager, you can use a CMEK to protect data at rest.”
- Customer-Managed Encryption Keys secures Secret Manager From the official documentation: “You can either create CMEKs manually to protect your Secret Manager resources or use Cloud KMS Autokey.”
- GKE works with Pub/Sub From the official documentation: “GKE publishes upgrade notifications to [Pub/Sub](/pubsub/docs/overview), providing you with a channel to receive information from GKE about your clusters.”
- Secret Manager depends on Application Default Credentials From the official documentation: “To authenticate to Secret Manager, set up Application Default Credentials.”
- Secret Manager secures Cloud SQL From the official documentation: “Secret Manager provides centralized and secure storage of sensitive data such as Cloud SQL configuration.”
- Secret Manager works with Workflows From the official documentation: “You can use a Workflows connector to access Secret Manager within a workflow.”
- Workflows depends on Application Default Credentials From the official documentation: “To authenticate to Workflows, set up Application Default Credentials.”
- Application Default Credentials works with Cloud Build From the official documentation: “To authenticate to Cloud Build, set up Application Default Credentials.”
- Application Default Credentials configures Cloud Run From the official documentation: “To authenticate to Cloud Run, set up Application Default Credentials.”
- Application Default Credentials works with GKE From the official documentation: “To authenticate to GKE, set up Application Default Credentials.”
- Application Default Credentials secures Secret Manager From the official documentation: “To authenticate to Secret Manager, set up Application Default Credentials.”
- Application Default Credentials secures Workflows From the official documentation: “To authenticate to Workflows, set up Application Default Credentials.”
- Cloud Build depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Build, set up Application Default Credentials.”
- Cloud Build depends on Cloud Storage From the official documentation: “Cloud Build uses Cloud Storage to store build resources.”
How learners score on this topic
Last updated