Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Apache Beam
Topic in 16 exam domains
- Apache Beam · Service
- Apache Beam · Concept
Also called: Apache Beam SDK for Java, beam
An open-source unified model for defining parallel data-processing pipelines, providing DoFn abstractions for processing batch and streaming data.
Where this appears on the exam
14 objectives in 3 of 5 domains
~25% of exam
- Data cleansing
- Identifying the services (e.g., Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Apache Spark, Hadoop ecosystem, and Apache Kafka)
- Transformations
- Batch
- Streaming (e.g., windowing, late arriving data)
- Processing logic
- AI data enrichment
- Integrating with new data sources
Domain 3
Storing the data~20% of exam
~18% of exam
Related topics
This topicBuilds on theseApplied
Apache BeamConcept
Application Default CredentialsFeatureVPC Service ControlsServiceBigtableServiceCloud MonitoringServiceCloud StorageServicePub/SubServiceCloud LoggingServiceDataflowServiceBigQueryServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Apache Beam works with Dataflow From the official documentation: “This page explains how to run an Apache Beam pipeline on Dataflow with GPUs.”
- Dataflow depends on Apache Beam From the official documentation: “Dataflow is based on the open-source Apache Beam project.”
- Apache Beam works with Bigtable From the official documentation: “Write data to Cloud Bigtable with Apache Beam.”
- Apache Beam works with BigQuery From the official documentation: “You can also use data pipeline sinks provided in [Apache Spark](/dataproc/docs/concepts/connectors/cloud-storage#other_sparkhadoop_clusters) or [Apache Beam](https://beam.apache.org/documentation/io/built-in/google-bigquery/#writing-to-bigquery) to write data into BigQuery.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials works with Bigtable From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- Application Default Credentials secures Bigtable From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Bigtable depends on Application Default Credentials From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery works with Bigtable From the official documentation: “External BigQuery data source backed by Bigtable”
- BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
- BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud Monitoring works with Pub/Sub From the official documentation: “Use Cloud Monitoring and Pub/Sub to set up custom monitoring and alerting policies for exported logs.”
- Application Default Credentials secures Cloud Monitoring From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Bigtable is an alternative to Cloud Storage From the official documentation: “If the joining data is in the gigabyte or terabyte range, use either Bigtable or Cloud Storage to merge the joining data to the driving data, depending on the nature of the data.”
- Bigtable works with Dataflow From the official documentation: “The Dataflow job then writes the transformed data to your Bigtable table.”
- Bigtable works with Pub/Sub From the official documentation: “Bigtable supports ingestion from Pub/Sub subscriptions. You can configure a Pub/Sub subscription to send messages to a Bigtable table.”
- Cloud Logging works with Cloud Monitoring From the official documentation: “If a log entry matches the filter of the alerting policy, then Cloud Logging extracts labels and forwards the data to Cloud Monitoring.”
- Cloud Logging works with Dataflow From the official documentation: “Information stored in [Cloud Logging](/logging/docs/region-support) is primarily generated by the code in your Dataflow program.”
- Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
- Application Default Credentials configures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Application Default Credentials works with Cloud Logging From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Application Default Credentials works with Cloud Monitoring From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Application Default Credentials configures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see [Set up authentication for client libraries](/storage/docs/authentication#client-libs).”
- BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
- BigQuery is an alternative to Dataflow From the official documentation: “To automate the EDA process, use BigQuery or a managed data processing service like Dataflow.”
- BigQuery works with VPC Service Controls From the official documentation: “BigQuery integrates with VPC Service Controls to restrict access to your BigQuery resources from specific networks.”
- Bigtable works with Cloud Logging From the official documentation: “You can audit the requests that Bigtable sends to Cloud KMS on your behalf in [Cloud Logging](/kms/docs/audit-logging), if you have [enabled Cloud KMS audit logs](/logging/docs/audit/configure-data-access#config-console-enable) for the Cloud KMS API in your project.”
- Bigtable works with Cloud Monitoring From the official documentation: “The following Bigtable metrics are provided to Cloud Monitoring.”
- Bigtable works with Cloud Storage From the official documentation: “Another way you can back up Bigtable is to use a managed data export to Cloud Storage.”
- Cloud Logging is an alternative to Cloud Monitoring From the official documentation: “To see the status of your invalidation request or find a definitive list of recently submitted requests, you can use Cloud Logging or Cloud Monitoring.”
- Cloud Logging is an alternative to Cloud Storage From the official documentation: “When you specify your own service account for builds, you must store your build logs either in Cloud Logging or in a user-created Cloud Storage bucket.”
- Cloud Logging works with VPC Service Controls From the official documentation: “This page describes how you can find VPC Service Controls errors using Cloud Logging.”
How learners score on this topic
Last updated