Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud service
Datastream
Topic in 4 exam domains
- Datastream · Service
- DataStream · Concept · Google SecOps
Also called: Datastream service
A serverless change data capture and replication service that supports replicating change events from sources like Microsoft Dataverse.
Where this appears on the exam
3 objectives in 2 of 5 domains
~22% of exam
~25% of exam
Related topics
FoundationsThis topicWorks alongsideAppliedIdentity and Access ManagementServicecustomer-managed encryption keysFeatureApplication Default CredentialsFeature
DatastreamService
Cloud StorageServiceKnowledge CatalogServiceCompute EngineServiceBigQueryServiceSpannerServiceDataflowServiceAlloyDBServicePub/SubServiceAlloyDB for PostgreSQLServiceApache IcebergConceptCloud SQLServiceGoogle Cloud StorageServiceCloud LoggingServiceCloud Data FusionServiceCloud RunServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- BigQuery works with Datastream From the official documentation: “Permissions needed for datastream to write to BigQuery.”
- Cloud Storage works with Datastream From the official documentation: “When you stream to Cloud Storage, Datastream organizes data by object name and timestamp.”
- AlloyDB for PostgreSQL works with Datastream From the official documentation: “Datastream works with the following types of PostgreSQL databases: - AlloyDB for PostgreSQL”
- Cloud SQL works with Datastream From the official documentation: “This page describes an example NAT VM configuration that lets Datastream privately connect to a Cloud SQL instance.”
- Datastream works with Spanner From the official documentation: “Learn how Datastream ingests data from [Spanner](/datastream/docs/sources-spanner).”
- Dataflow works with Datastream From the official documentation: “In addition, Datastream supports writing the change event stream into Cloud Storage, and offers streamlined integration with Dataflow templates to build custom workflows for loading data into a wide range of destinations, such as Cloud SQL and Spanner.”
- Datastream works with Google Cloud Storage From the official documentation: “| How are files created in Cloud Storage? | Datastream creates a folder for each table.”
- Datastream works with Knowledge Catalog From the official documentation: “You can search for and manage your Datastream resources using Knowledge Catalog, which is a platform for storing, managing, and accessing your metadata.”
- AlloyDB works with Datastream From the official documentation: “To enable Datastream to connect to the AlloyDB instance, you need to set up a TCP proxy in the consumer project.”
- Apache Iceberg works with Datastream From the official documentation: “This section describes the destinations to which you can stream data using Datastream. Supported destinations are: - BigQuery - Cloud Storage - Apache Iceberg tables.”
- Cloud Data Fusion depends on Datastream From the official documentation: “When you run a replication job with an Oracle Database, Cloud Data Fusion uses the Datastream service in the backend.”
- Cloud Data Fusion is an alternative to Datastream From the official documentation: “Alternatively, to replicate data into BigQuery, you can also use Cloud Data Fusion or [Datastream](/datastream/docs/overview).”
- customer-managed encryption keys works with Datastream From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Datastream.”
- customer-managed encryption keys secures Datastream From the official documentation: “The CMEK feature lets you use your own cryptographic keys for data at rest in Datastream.”
- Identity and Access Management secures Datastream From the official documentation: “You can get and set permissions for Datastream using the Google Cloud Console, the IAM methods of the API, or the Google Cloud SDK.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- AlloyDB depends on Application Default Credentials From the official documentation: “To authenticate to AlloyDB, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- AlloyDB works with Application Default Credentials From the official documentation: “To authenticate to AlloyDB, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Cloud Run works with Cloud SQL From the official documentation: “Learn how to connect to Cloud SQL from Cloud Run”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures Spanner From the official documentation: “To authenticate to Spanner, set up Application Default Credentials.”
- Knowledge Catalog depends on Application Default Credentials From the official documentation: “To authenticate to Knowledge Catalog, set up Application Default Credentials.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials works with Google Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials works with Spanner From the official documentation: “To authenticate to Spanner, set up Application Default Credentials.”
- BigQuery works with Knowledge Catalog From the official documentation: “Some Knowledge Catalog functionalities trigger job execution using BigQuery.”
- Cloud Logging works with customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
- Cloud SQL works with Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- Spanner depends on Application Default Credentials From the official documentation: “To authenticate to Spanner, set up Application Default Credentials.”
- Cloud Run works with Pub/Sub From the official documentation: “Route Cloud Pub/Sub events to Cloud Run”
- Cloud SQL works with Compute Engine From the official documentation: “Create a Compute Engine VM Instance to host a sample web app that connects to Cloud SQL.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Cloud Run works with Cloud Storage From the official documentation: “Route Cloud Storage events to Cloud Run”
- Application Default Credentials secures Compute Engine From the official documentation: “To authenticate to Compute Engine, set up Application Default Credentials.”
- BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
- BigQuery works with Spanner From the official documentation: “Reverse ETL (BigQuery to Spanner)”
- Cloud Run depends on Pub/Sub From the official documentation: “Create a Pub/Sub topic that Cloud Scheduler will publish to and the Cloud Run function will subscribe to.”
- Cloud SQL works with Identity and Access Management From the official documentation: “This page describes how Cloud SQL is integrated with IAM .”
- Cloud SQL works with Knowledge Catalog From the official documentation: “You can use Knowledge Catalog to organize and enrich the metadata of Cloud SQL resources.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- customer-managed encryption keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
- Dataflow works with Spanner From the official documentation: “Use Dataflow to load data into Spanner.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Cloud SQL From the official documentation: “Cloud SQL uses Cloud Logging.”
How learners score on this topic
Last updated