Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Concept
datasets
Topic in 2 exam domains
- datasets · Concept
- datasets · Feature · Customer Experience Insights
- Datasets · Concept
Top-level containers in BigQuery used to organize and control access to tables and views.
Where this appears on the exam
2 subdomains in 1 of 5 domains
Related topics
FoundationsBuilding blocksBuilding blocksThis topicWorks alongsideApplication Default CredentialsFeatureCloud StorageServiceCloud LoggingServicePub/SubServiceCloud SQLServiceDataflowServiceBigQueryService
datasetsConcept
DatastreamServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- Dataflow works with datasets From the official documentation: “BigQuery uses datasets to contain the data that it receives from Dataflow.”
- datasets is part of BigQuery From the official documentation: “To organize and control access to data in BigQuery, you provision top-level containers called [datasets](/bigquery/docs/datasets-intro).”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- BigQuery works with Datastream From the official documentation: “Datastream supports three types of partitioning for BigQuery tables:”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
- Cloud Storage works with Datastream From the official documentation: “Because Cloud Storage is in the same network as Datastream, no special connectivity is required.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Cloud Logging works with Cloud SQL From the official documentation: “Database Migration Service and Cloud SQL use Cloud Logging.”
- Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials configures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials works with Cloud SQL From the official documentation: “Application Default Credentials are part of the client libraries you can use to access Cloud SQL.”
- Application Default Credentials configures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see [Set up authentication for client libraries](/storage/docs/authentication#client-libs).”
- BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Cloud Logging is an alternative to Cloud Storage From the official documentation: “Instead, they are routed to either Cloud Storage (recommended) or Cloud Logging, depending on your organization's configuration.”
- Cloud SQL works with Datastream From the official documentation: “The change data capture pipeline uses VPC peering to establish a connection between Datastream and private IP Cloud SQL.”
- Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
- Application Default Credentials works with Cloud Logging From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
- Cloud Logging works with Dataflow From the official documentation: “You can control the volume of Dataflow logs ingested by Cloud Logging by following the [Filter logs documentation](/dataflow/docs/guides/filter-logs).”
- Cloud SQL depends on Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
- Cloud SQL works with Dataflow From the official documentation: “Use Datastream to replicate changes from your MySQL instance to either BigQuery or Cloud Storage, and then use Dataflow or Managed Service for Apache Spark to bring the data into your Cloud SQL instance.”
- Dataflow works with Datastream From the official documentation: “Integrating Datastream with Dataflow job templates lets you customize your stream processing logic when you replicate data into BigQuery for near real-time analytics.”
How learners score on this topic
Last updated