Build and operate the data processing systems behind analytics and ML on Google Cloud, including BigQuery, Dataflow, and Pub/Sub, and track your readiness for the Professional Data Engineer exam.
Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Google Cloud feature
data lineage
Topic in 5 exam domains
- data lineage · Feature
- Data Lineage · Service
- data lineage · Concept
- Data lineage · Feature
- Data Lineage · Feature
- Data Lineage · Concept · Metadata type
- Data lineage · Concept · Google Cloud data lineage
Also called: Automated data lineage, column-level data lineage, data lineage graphs, Data lineage integration, Knowledge Catalog data lineage, Knowledge Catalog Data Lineage integration, Knowledge Catalog Lineage, Lineage, lineage information, Lineage service, Spark data lineage
A Knowledge Catalog capability that tracks data movements and transformations across domains and projects.
Where this appears on the exam
5 subdomains in 4 of 5 domains
~25% of exam
Domain 3
Storing the data~20% of exam
~15% of exam
~18% of exam
Related topics
FoundationsBuilding blocksThis topicWorks alongsideVPC Service ControlsServiceApplication Default CredentialsFeatureservice accountConceptCloud StorageServiceManaged Service for Apache AirflowServiceBigQueryServiceKnowledge CatalogServiceDataflowServiceVertex AIServicePub/SubServiceManaged Service for Apache SparkService
data lineageFeature
Cloud Data FusionServiceBigQuery OmniFeatureVertex AI PipelinesFeatureCloud SQLServiceCloud LoggingServicePoints to what builds on itWorks with or alternative toHover or tap a line to see what connects the two
- BigQuery works with data lineage From the official documentation: “Data lineage in Knowledge Catalog lets you track downstream dependencies for tables and views in your BigQuery project.”
- data lineage is part of Knowledge Catalog From the official documentation: “Data lineage is a Knowledge Catalog feature that lets you track data movement.”
- BigQuery Omni works with data lineage From the official documentation: “Data lineage is available in the following [BigQuery Omni regions](/bigquery/docs/omni-introduction):”
- Cloud Data Fusion works with data lineage From the official documentation: “Data lineage supports integration with the following Google Cloud services:”
- Cloud Storage works with data lineage From the official documentation: “Data lineage is available for all Managed Service for Apache Spark jobs except SparkR and Spark streaming jobs, and supports BigQuery and Cloud Storage data sources.”
- data lineage is part of Dataflow From the official documentation: “Data lineage is a Dataflow feature that lets you track how data moves through your systems: where it comes from, where it is passed to, and what transformations are applied to it.”
- data lineage works with Knowledge Catalog From the official documentation: “<li><a href="/knowledge-catalog/docs/use-lineage#view-lineage">Data lineage</a> viewing</li>”
- data lineage works with Managed Service for Apache Airflow From the official documentation: “Managed Airflow uses environment-level data lineage integration control.”
- data lineage works with Managed Service for Apache Spark From the official documentation: “You can capture lineage events with Managed Service for Apache Spark Hive jobs and publish them to the Data Lineage API.”
- data lineage works with Vertex AI From the official documentation: “Data lineage in Knowledge Catalog (formerly Dataplex Universal Catalog) lets you track where your data originates, how it transforms, and where it moves across systems such as BigQuery and Vertex AI.”
- data lineage works with Vertex AI Pipelines From the official documentation: “Data lineage is automatically enabled for Vertex AI Pipelines pipelines, tracking input artifacts and execution parameters (such as models, datasets, and components), as well as downstream derived assets.”
- BigQuery is an alternative to Cloud Storage From the official documentation: “The `import_result_sink` must be a Cloud Storage path or a BigQuery table.”
- BigQuery works with Cloud Storage From the official documentation: “Sample to load Avro data from Cloud Storage into a new BigQuery table”
- Application Default Credentials secures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery works with Cloud SQL From the official documentation: “Add credentials to connect BigQuery to Cloud SQL.”
- BigQuery works with Cloud Logging From the official documentation: “BigQuery utilizes Cloud Logging for some of your log data.”
- Cloud SQL works with Cloud Storage From the official documentation: “Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.”
- Application Default Credentials works with Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- Application Default Credentials secures Vertex AI From the official documentation: “To authenticate to Vertex AI, set up Application Default Credentials.”
- Dataflow works with Pub/Sub From the official documentation: “Pub/Sub notifies Dataflow about the upload.”
- Application Default Credentials works with Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Application Default Credentials secures Pub/Sub From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- BigQuery works with Dataflow From the official documentation: “Read from BigQuery to Dataflow”
- Application Default Credentials works with BigQuery From the official documentation: “Create a BigQuery client using application default credentials.”
- Cloud Storage works with Dataflow From the official documentation: “Write from Dataflow to Cloud Storage”
- Pub/Sub depends on Application Default Credentials From the official documentation: “To authenticate to Pub/Sub, set up Application Default Credentials.”
- VPC Service Controls secures BigQuery From the official documentation: “BigQuery data is protected with VPC Service Controls”
- Application Default Credentials secures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Cloud Data Fusion works with Managed Service for Apache Spark From the official documentation: “Deployed Cloud Data Fusion pipelines run on Managed Service for Apache Spark clusters by default.”
- Dataflow works with Managed Service for Apache Airflow From the official documentation: “Launch Dataflow pipelines with Cloud Composer”
- BigQuery depends on Application Default Credentials From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- BigQuery works with Knowledge Catalog From the official documentation: “Some Knowledge Catalog functionalities trigger job execution using BigQuery.”
- service account works with Vertex AI From the official documentation: “The service account can interact with Vertex AI services.”
- Application Default Credentials secures Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- BigQuery works with Pub/Sub From the official documentation: “createBigQuerySubscription creates a Pub/Sub subscription that exports messages to BigQuery.”
- Cloud Logging works with Pub/Sub From the official documentation: “Pub/Sub writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud Storage is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- Vertex AI depends on Application Default Credentials From the official documentation: “To authenticate to Vertex AI, set up Application Default Credentials.”
- Application Default Credentials works with Dataflow From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials secures Managed Service for Apache Spark From the official documentation: “To authenticate to Managed Service for Apache Spark, set up Application Default Credentials.”
- Cloud Logging works with Cloud SQL From the official documentation: “Database Migration Service and Cloud SQL use Cloud Logging.”
- Cloud Logging works with Cloud Storage From the official documentation: “Cloud Storage writes various log entries to Cloud Logging in the Google Cloud console.”
- Cloud SQL works with Vertex AI From the official documentation: “This page describes how to integrate Cloud SQL with [Vertex AI](/vertex-ai/docs).”
- Cloud Storage works with Managed Service for Apache Spark From the official documentation: “Managed Service for Apache Spark accesses these files on Cloud Storage.”
- Cloud Storage works with Pub/Sub From the official documentation: “| **Google Cloud** | Cloud Storage | Pub/Sub |”
- Dataflow depends on Application Default Credentials From the official documentation: “To authenticate to Dataflow, set up Application Default Credentials.”
- Application Default Credentials configures BigQuery From the official documentation: “To authenticate to BigQuery, set up Application Default Credentials.”
- Application Default Credentials secures Knowledge Catalog From the official documentation: “To authenticate to Knowledge Catalog, set up Application Default Credentials.”
- BigQuery works with Managed Service for Apache Airflow From the official documentation: “You can use Managed Airflow to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.”
- BigQuery is an alternative to Pub/Sub From the official documentation: “To keep audit logs for a longer period of time or to use more powerful search capabilities, you can route copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub.”
- BigQuery works with Vertex AI From the official documentation: “The page provides guidance for getting predictions from Vertex AI, based on retail [data exported into BigQuery](/retail/docs/export-data-into-bq).”
- Cloud Data Fusion works with Knowledge Catalog From the official documentation: “Cloud Data Fusion provides an integration with Knowledge Catalog for lineage.”
- Cloud SQL works with Knowledge Catalog From the official documentation: “You can use Knowledge Catalog to organize and enrich the metadata of Cloud SQL resources.”
- Cloud Storage depends on Application Default Credentials From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials.”
- Managed Service for Apache Spark depends on Cloud Storage From the official documentation: “Managed Service for Apache Spark uses Cloud Storage as its default distributed file system.”
- Vertex AI depends on service account From the official documentation: “To use the Google Vertex AI model endpoints, you must add Vertex AI permissions to the service account that you used while installing AlloyDB Omni.”
- VPC Service Controls secures Cloud Storage From the official documentation: “Storage Transfer Service supports transfers to Cloud Storage buckets protected by VPC Service Controls.”
- VPC Service Controls secures Managed Service for Apache Airflow From the official documentation: “Managed Airflow can be selected as a secured service inside a VPC Service Controls perimeter.”
- Application Default Credentials works with Cloud SQL From the official documentation: “Application Default Credentials are part of the client libraries you can use to access Cloud SQL.”
- Application Default Credentials configures Cloud Storage From the official documentation: “To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see [Set up authentication for client libraries](/storage/docs/authentication#client-libs).”
How learners score on this topic
Last updated