Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud feature
scheduled query
Topic in 3 exam domains
Also called: BigQuery scheduled queries, BigQuery scheduled query, BigQuery transfer jobs, scheduled queries, Scheduling queries, scheduling recurring queries
A BigQuery recurring job mechanism used to periodically run queries, such as exporting logs to Google Cloud Storage.
Where this appears on the exam
2 objectives in 1 of 5 domains
Related topics
FoundationsBuilding blocksBuilding blocksThis topicApplication Default CredentialsFeatureBigQuery Data Transfer ServiceServiceVPC Service ControlsServicecustomer-managed encryption keysFeatureCloud MonitoringServiceGoogle Cloud StorageServiceBigtableServicePub/SubServiceCloud LoggingServiceCloud SQLServiceDataflowServiceBigQueryService
scheduled queryFeature
Points to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- scheduled query is part of BigQuery From the official documentation: “The simplest form of workload scheduling is [scheduling recurring queries](/bigquery/docs/scheduling-queries) directly in BigQuery.”
- BigQuery Data Transfer Service works with scheduled query From the official documentation: “Although scheduled queries use features of the [BigQuery Data Transfer Service](/bigquery/docs/dts-introduction), scheduled queries are not transfers, and are not subject to [load job limits](/bigquery/quotas#load_jobs).”
- Bigtable works with scheduled query From the official documentation: “Scheduled queries are useful when you want to import Bigtable data into BigQuery on a recurring basis.”
- Cloud Monitoring works with scheduled query From the official documentation: “Scheduled queries automatically create a metric in Cloud Monitoring.”
- Dataflow is an alternative to scheduled query From the official documentation: “for recurring exports, use [BigQuery scheduled queries](/bigquery/docs/scheduling-queries) or [Dataflow](/dataflow/docs/overview).”
- Google Cloud Storage works with scheduled query From the official documentation: “To automatically export Gmail logs from BigQuery to Cloud Storage on a recurring schedule, create a scheduled query using the EXPORT DATA statement.”
- scheduled query depends on BigQuery Data Transfer Service From the official documentation: “Although scheduled queries use features of [BigQuery Data Transfer Service](/bigquery/docs/dts-introduction), they are not transfers and are not subject to the load jobs quota.”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials works with Bigtable From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- Application Default Credentials secures Bigtable From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- BigQuery works with BigQuery Data Transfer Service From the official documentation: “The BigQuery Data Transfer Service uses load jobs to load Amazon S3 data into BigQuery.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Bigtable depends on Application Default Credentials From the official documentation: “To authenticate to Bigtable, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials works with Google Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Logging works with customer-managed encryption keys From the official documentation: “To use Cloud Logging with Customer-managed encryption keys (CMEK), you must complete the steps in the [Enable CMEK for an organization](/logging/docs/routing/managed-encryption#enable) page in the Cloud Logging documentation.”
- BigQuery works with Bigtable From the official documentation: “External BigQuery data source backed by Bigtable”
- BigQuery works with Cloud Monitoring From the official documentation: “You can use Monitoring to export your metrics to BigQuery.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
- Cloud Monitoring depends on Application Default Credentials From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Cloud Monitoring works with Cloud SQL From the official documentation: “You can also get more details by viewing the Cloud Monitoring logs for your Cloud SQL destination instance.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Cloud SQL From the official documentation: “Cloud SQL uses Cloud Logging.”
- customer-managed encryption keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
- customer-managed encryption keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
- customer-managed encryption keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Cloud Logging depends on Application Default Credentials From the official documentation: “To authenticate to Logging, set up Application Default Credentials.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Monitoring works with Dataflow From the official documentation: “Dataflow is fully integrated with Cloud Monitoring.”
- BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- BigQuery Data Transfer Service works with customer-managed encryption keys From the official documentation: “When you specify a CMEK with a transfer, the BigQuery Data Transfer Service applies the CMEK to any intermediate on-disk cache of ingested data so that the entire data transfer workflow is CMEK compliant.”
- BigQuery Data Transfer Service works with Pub/Sub From the official documentation: “If you are using the Google Cloud console, you can schedule recurring copies, and configure an email or Pub/Sub notifications with the BigQuery Data Transfer Service.”
- Cloud Monitoring works with Pub/Sub From the official documentation: “Use Cloud Monitoring and Pub/Sub to set up custom monitoring and alerting policies for exported logs.”
- Cloud SQL works with Google Cloud Storage From the official documentation: “Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.”
- customer-managed encryption keys secures Bigtable From the official documentation: “In a CMEK-protected instance, Bigtable uses your CMEK key to protect your data at rest.”
- Google Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- VPC Service Controls secures Cloud SQL From the official documentation: “VPC Service Controls perimeters protect the Cloud SQL Admin API.”
- Application Default Credentials secures Cloud Monitoring From the official documentation: “To authenticate to Monitoring, set up Application Default Credentials.”
- Bigtable works with Dataflow From the official documentation: “The Dataflow job then writes the transformed data to your Bigtable table.”
- Bigtable works with Pub/Sub From the official documentation: “Bigtable supports ingestion from Pub/Sub subscriptions. You can configure a Pub/Sub subscription to send messages to a Bigtable table.”
- Cloud Logging works with Cloud Monitoring From the official documentation: “If a log entry matches the filter of the alerting policy, then Cloud Logging extracts labels and forwards the data to Cloud Monitoring.”
- Cloud Logging works with Dataflow From the official documentation: “Information stored in [Cloud Logging](/logging/docs/region-support) is primarily generated by the code in your Dataflow program.”
- Cloud Logging is an alternative to Google Cloud Storage From the official documentation: “When you specify your own service account for builds, you must store your build logs in a Cloud Logging bucket, in a user-created Cloud Storage bucket, or in a Google Cloud-created and user-owned Cloud Storage bucket.”
- customer-managed encryption keys secures Cloud Logging From the official documentation: “Cloud Logging supports encrypting Log storage with CMEK keys.”
- Dataflow depends on Google Cloud Storage From the official documentation: “- A gcs bucket for staging DataFlow files. Replace `<BUCKET_NAME>`: the name of a valid Google Cloud Storage bucket.”
- Dataflow works with Google Cloud Storage From the official documentation: “Data is transferred between Spanner and Cloud Storage using Dataflow.”
- VPC Service Controls secures Google Cloud Storage From the official documentation: “Storage Transfer Service supports on-premises transfers to Cloud Storage buckets protected by VPC Service Controls, under the following conditions:”
How learners score on this topic
Last updated