Apache Beam SDK

Related topics

• Apache Beam SDK works with Dataflow From the official documentation: “You can use the Apache Beam SDK to build pipelines for Dataflow.”
• Apache Beam SDK works with Cloud Storage From the official documentation: “The Apache Beam SDK stages files in Cloud Storage, creates a job request file, and submits the file to Dataflow.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
• Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
• BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
• BigQuery works with Datastream From the official documentation: “Datastream supports three types of partitioning for BigQuery tables:”
• Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
• Cloud Storage works with Datastream From the official documentation: “Because Cloud Storage is in the same network as Datastream, no special connectivity is required.”
• Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
• Customer-Managed Encryption Keys secures Cloud Storage From the official documentation: “CMEKs encrypt data at rest in Cloud SQL, Cloud Storage, and within GKE clusters.”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
• Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
• Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
• Customer-Managed Encryption Keys secures Cloud SQL From the official documentation: “Enable CMEK for Cloud SQL Databases”
• BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
• Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Storage works with Customer-Managed Encryption Keys From the official documentation: “Cloud Storage supports CMEK for routing logs.”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Customer-Managed Encryption Keys secures BigQuery From the official documentation: “You can either create CMEKs manually to protect your BigQuery resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys secures Pub/Sub From the official documentation: “Encrypt Pub/Sub topic with CMEK”
• Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
• Application Default Credentials configures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
• BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
• Application Default Credentials works with Cloud SQL From the official documentation: “Application Default Credentials are part of the client libraries you can use to access Cloud SQL.”
• Application Default Credentials configures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see [Set up authentication for client libraries](/storage/docs/authentication#client-libs).”
• Cloud SQL works with Datastream From the official documentation: “The change data capture pipeline uses VPC peering to establish a connection between Datastream and private IP Cloud SQL.”
• Customer-Managed Encryption Keys depends on Cloud Storage From the official documentation: “If used for storing Customer-managed Encryption Keys (CMEK): Cloud Storage (See [Cloud Storage pricing](/storage/pricing).)”
• Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
• BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
• Cloud SQL depends on Cloud Storage From the official documentation: “When ingesting data from Cloud SQL, data is first staged to a Cloud Storage bucket.”
• Cloud SQL works with Dataflow From the official documentation: “Use Datastream to replicate changes from your MySQL instance to either BigQuery or Cloud Storage, and then use Dataflow or Managed Service for Apache Spark to bring the data into your Cloud SQL instance.”
• Customer-Managed Encryption Keys is part of BigQuery From the official documentation: “If you require more control over encryption keys, BigQuery supports [customer-managed encryption keys](/bigquery/docs/customer-managed-encryption) in the [Cloud Key Management Service](/kms/docs).”
• Customer-Managed Encryption Keys is part of Cloud SQL From the official documentation: “Cloud SQL also lets you add another layer of encryption to data using [customer-managed encryption keys (CMEK)](/sql/docs/postgres/cmek).”
• Customer-Managed Encryption Keys is part of Cloud Storage From the official documentation: “[Cloud Storage](/storage/docs/introduction) provides features such as customer-managed encryption, lifecycle management, and aging and retention policies.”
• Customer-Managed Encryption Keys works with Dataflow From the official documentation: “This sample shows how to use encryption keys managed by the customer, with a Dataflow pipeline.”
• Customer-Managed Encryption Keys secures Dataflow From the official documentation: “You can either create CMEKs manually to protect your Dataflow resources or use Cloud KMS Autokey.”
• Customer-Managed Encryption Keys works with Datastream From the official documentation: “If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in [Cloud KMS](/kms/docs) with CMEK-integrated services including Datastream.”
• Customer-Managed Encryption Keys secures Datastream From the official documentation: “The CMEK feature lets you use your own cryptographic keys for data at rest in Datastream.”
• Dataflow works with Datastream From the official documentation: “Integrating Datastream with Dataflow job templates lets you customize your stream processing logic when you replicate data into BigQuery for near real-time analytics.”