Apache Beam

Related topics

• Apache Beam works with Dataflow From the official documentation: “This page explains how to run an Apache Beam pipeline on Dataflow with GPUs.”
• Dataflow depends on Apache Beam From the official documentation: “Dataflow is based on the open-source Apache Beam project.”
• Apache Beam works with BigQuery From the official documentation: “You can also use data pipeline sinks provided in [Apache Spark](/dataproc/docs/concepts/connectors/cloud-storage#other_sparkhadoop_clusters) or [Apache Beam](https://beam.apache.org/documentation/io/built-in/google-bigquery/#writing-to-bigquery) to write data into BigQuery.”
• MLTransform is part of Apache Beam From the official documentation: “Apache Beam provides a set of transforms for preprocessing data for training and inference. The `MLTransform` class wraps various transforms in one `PTransform`, simplifying your workflow.”
• BigQuery works with Sensitive Data Protection From the official documentation: “Uses the Data Loss Prevention API to analyze BigQuery data.”
• BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
• BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• BigQuery works with Cloud Storage From the official documentation: “External BigQuery data source backed by Cloud Storage”
• BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
• VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
• Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
• Pub/Sub works with Sensitive Data Protection From the official documentation: “Watch the Pub/Sub topic until the DLP job finishes”
• BigQuery is an alternative to Google Cloud Storage From the official documentation: “The training data can be either a CSV file in Cloud Storage or a table in BigQuery.”
• BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• BigQuery works with Google Cloud Storage From the official documentation: “Load Cloud Storage data into BigQuery”
• Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
• Cloud Storage works with Sensitive Data Protection From the official documentation: “Uses the Data Loss Prevention API to analyze a file on GCS.”
• BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
• Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
• Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
• Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
• Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
• BigQuery depends on Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
• Cloud Logging works with Dataflow From the official documentation: “Information stored in [Cloud Logging](/logging/docs/region-support) is primarily generated by the code in your Dataflow program.”
• Cloud Logging is an alternative to Google Cloud Storage From the official documentation: “When you specify your own service account for builds, you must store your build logs in a Cloud Logging bucket, in a user-created Cloud Storage bucket, or in a Google Cloud-created and user-owned Cloud Storage bucket.”
• Dataflow depends on Cloud Storage From the official documentation: “Begin by creating a Cloud Storage bucket using the Google Cloud console or Google Cloud CLI. The Dataflow pipeline uses this bucket as a temporary storage location.”
• Dataflow depends on Google Cloud Storage From the official documentation: “- A gcs bucket for staging DataFlow files. Replace `<BUCKET_NAME>`: the name of a valid Google Cloud Storage bucket.”
• Dataflow works with Google Cloud Storage From the official documentation: “Data is transferred between Spanner and Cloud Storage using Dataflow.”
• Dataflow works with Sensitive Data Protection From the official documentation: “Use Dataflow and Sensitive Data Protection to securely tokenize and import data from a relational database to BigQuery.”
• Sensitive Data Protection works with VPC Service Controls From the official documentation: “For information about VPC Service Controls support for Sensitive Data Protection, see the [Supported products](/vpc-service-controls/docs/supported-products#table_dlp) table in the VPC Service Controls documentation.”
• VPC Service Controls secures Google Cloud Storage From the official documentation: “Storage Transfer Service supports on-premises transfers to Cloud Storage buckets protected by VPC Service Controls, under the following conditions:”
• BigQuery depends on Cloud Storage From the official documentation: “BigQuery relies on Cloud Storage for staging file data and supports reading from and writing to several other sources and Google Cloud services.”
• BigQuery is an alternative to Dataflow From the official documentation: “To automate the EDA process, use BigQuery or a managed data processing service like Dataflow.”
• BigQuery works with VPC Service Controls From the official documentation: “BigQuery integrates with VPC Service Controls to restrict access to your BigQuery resources from specific networks.”
• Cloud Logging is an alternative to Cloud Storage From the official documentation: “When you specify your own service account for builds, you must store your build logs either in Cloud Logging or in a user-created Cloud Storage bucket.”
• Cloud Logging works with Google Cloud Storage From the official documentation: “See [Routing and storage overview](/logging/docs/routing/overview) to route logs from Logging to Cloud Storage, BigQuery, or Pub/Sub.”
• Cloud Logging works with VPC Service Controls From the official documentation: “This page describes how you can find VPC Service Controls errors using Cloud Logging.”
• Dataflow works with VPC Service Controls From the official documentation: “To customize DNS resolution when using Dataflow with VPC Service Controls, use Cloud DNS [private zones](/dns/docs/overview#dns-private-zones) instead of using custom BIND servers.”
• Google Cloud Storage works with Pub/Sub From the official documentation: “Both Google Managed Service for Apache Kafka and Pub/Sub integrates with various Google Cloud services like Dataflow, BigQuery, Cloud Storage, and more.”
• Google Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
• Google Cloud Storage works with Sensitive Data Protection From the official documentation: “This page describes how Sensitive Data Protection can create de-identified copies of data stored in Cloud Storage.”
• Sensitive Data Protection secures BigQuery From the official documentation: “To help meet compliance requirements, use Sensitive Data Protection to inspect your BigQuery tables and to help protect your sensitive data.”
• VPC Service Controls secures Cloud Logging From the official documentation: “The API for Cloud Logging can be protected by VPC Service Controls and the product can be used normally inside service perimeters.”
• VPC Service Controls secures Dataflow From the official documentation: “Place your Dataflow pipelines inside a security perimeter using VPC Service Controls.”
• VPC Service Controls secures Sensitive Data Protection From the official documentation: “VPC Service Controls enables you to define a security perimeter around your projects and resources, including Sensitive Data Protection resources.”