An enterprise data platform spans multiple Google Cloud projects containing BigQuery tables and multi-terabyte Cloud Storage buckets containing semi-structured JSON and Parquet datasets. Your data governance team requires a centralized metadata architecture that fulfills the following criteria: Automatically discover schema definitions and partition metadata from Cloud Storage files and publish them for BigQuery SQL querying without manual table definition. Standardize business metadata (such as data classification and retention periods) using reusable schemas. Ensure users can discover assets via semantic search only if they have IAM permissions to view the underlying resource, while restricting sensitive compliance tags to authorized compliance officers. Which architecture should you implement?

Organize Cloud Storage assets into Dataplex zones with Discovery enabled to register metadata in BigQuery publishing datasets. Create schematized Tag Templates in Dataplex/Data Catalog, using private tag templates for sensitive compliance fields and public tag templates for general metadata, and control discovery through IAM roles.: ### What Dataplex and Data Catalog Provide **Dataplex Universal Catalog** and **Data Catalog** provide an automated, policy-driven metadata management and discovery service. Dataplex manages distributed data lakes and data zones, automatically extracting technical metadata through **Discovery jobs**, while Data Catalog provides schematized **Tag Templates** to capture business and operational metadata across resources like BigQuery, Cloud Storage, and Pub/Sub. ### How It Meets the Requirements - **Automated Cloud Storage Schema Discovery:** Enabling **Dataplex Discovery** on Cloud Storage assets automatically scans structured and semi-structured files (such as Parquet, Avro, and newline-delimited JSON), groups them into tables, extracts partition definitions, and creates external tables in a **BigQuery publishing dataset** within the zone's region. - **Standardized Business Metadata:** **Tag Templates** define structured key-value schemas (strings, enums, booleans, dates) for data classification, retention policies, and data owners, ensuring uniform tagging across the entire data estate. - **Policy-Based Semantic Search and Access Control:** Data Catalog search queries automatically respect the user's underlying **IAM permissions** (e.g., `bigquery.tables.get` or `roles/bigquery.metadataViewer`). A user cannot view search results for assets they lack permissions to access. Furthermore, configuring **private tag templates** restricts the visibility and searchability of sensitive compliance metadata exclusively to principals granted explicit IAM permissions on the template. ### Key Advantages - **Automated Metadata Publishing:** Eliminates manual DDL scripting for newly ingested files in Cloud Storage data lakes. - **Fine-Grained Metadata Security:** Separates public metadata visibility from restricted governance metadata using public vs. private Tag Templates. - **Centralized Enterprise Search:** Delivers faceted and natural language/semantic search across BigQuery datasets, Cloud Storage filesets, and Pub/Sub topics. - **Regional Compatibility:** Automatically creates publishing datasets aligned with the storage region of the lake and asset. ### Why This Is Optimal This architecture leverages fully managed Google Cloud governance services without requiring custom ETL pipelines, third-party metastores, or bespoke metadata synchronization scripts.

Dataplex Catalog Tag Template Governance Architecture | professional-cloud-data-engineer

Dataplex Catalog Tag Template Governance Architecture

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data platform spans multiple Google Cloud projects containing BigQuery tables and multi-terabyte Cloud Storage buckets containing semi-structured JSON and Parquet datasets.

Your data governance team requires a centralized metadata architecture that fulfills the following criteria:

Automatically discover schema definitions and partition metadata from Cloud Storage files and publish them for BigQuery SQL querying without manual table definition.
Standardize business metadata (such as data classification and retention periods) using reusable schemas.
Ensure users can discover assets via semantic search only if they have IAM permissions to view the underlying resource, while restricting sensitive compliance tags to authorized compliance officers.

Which architecture should you implement?

Deploy Cloud Functions triggered by Cloud Storage object finalization to parse file headers, execute BigQuery DDL to generate external tables, and write metadata records into a centralized BigQuery governance dataset secured with authorized views.

Store metadata in JSON Lines sidecar files alongside raw data in Cloud Storage, ingest these files into a Vertex AI Search data store, and query the data store using natural language document search.

Organize Cloud Storage assets into Dataplex zones with Discovery enabled to register metadata in BigQuery publishing datasets. Create schematized Tag Templates in Dataplex/Data Catalog, using private tag templates for sensitive compliance fields and public tag templates for general metadata, and control discovery through IAM roles.

Build a Cloud Data Fusion batch pipeline on an hourly schedule to ingest Cloud Storage metadata, load technical schemas into Dataproc Metastore, and apply Google Cloud resource labels to underlying storage buckets.

Professional Cloud Data Engineer

Dataplex Catalog Tag Template Governance Architecture

Related questions