Governed Staging Catalog Profiling and Discovery

Enterprise metadata cataloging and searchable discovery is the practice of centrally ingesting technical metadata, enriching it with business context, and exposing it through governed search. In Google Cloud, Knowledge Catalog (opens in a new tab) in Dataplex provides that foundation as a fully managed and scalable metadata management service. Metadata flows from source systems into catalog entries in Knowledge Catalog, and trusted users discover those entries through search that enforces permissions before returning results.

1.1Centralized Metadata Ingestion and Organization

Centralized metadata ingestion and organization is how Knowledge Catalog automatically collects, organizes, and indexes metadata from supported Google Cloud sources. Metadata flows from source systems such as BigQuery, Cloud Storage, and Pub/Sub into catalog entries in Knowledge Catalog, so engineers do not manually register each asset. For sources outside Google Cloud, stewards create custom entries to bring external metadata into the same index. The catalog also unifies governance across BigLake and open-format tables like Iceberg and Delta, and it supports a data mesh where decentralized domain teams own their data products while discovery remains centralized.

The metadata lifecycle moves in a fixed order:

1. Ingest: Source systems publish technical metadata to Knowledge Catalog, and Knowledge Catalog stores the entry with its lineage and schema.
2. Enrich: Data stewards and governance teams attach business context to the entry through tags, aspects, and glossary terms.
3. Discover: Trusted users issue semantic search against Knowledge Catalog, and Knowledge Catalog returns only the entries and metadata the caller is permitted to see.

1.2Business and Technical Enrichment with Tags and Glossaries

Business and technical enrichment in Knowledge Catalog adds business meaning to raw technical metadata to improve discoverability and management. Data Catalog tag templates provide schematized tags that stewards attach to entries, while aspects and aspect types provide structured enrichment that travels with the entry. A business glossary establishes a standardized vocabulary so different teams use the same term for the same domain concept. Sensitive-data inspection results from BigQuery (opens in a new tab) can also be sent to Knowledge Catalog as aspects, linking classification directly to the discoverable entry.

1.3Secure Policy-Based Search and Discovery

Secure policy-based search and discovery is search in Knowledge Catalog that enforces Identity and Access Management (IAM) before returning results. Users call the search function in Knowledge Catalog, Knowledge Catalog validates the caller against entry-level and tag-level IAM permissions, and only then returns matching metadata. Data scientists and machine learning practitioners therefore need sufficient permissions to access Knowledge Catalog and to use search, separate from permissions to read the underlying BigQuery, Cloud Storage, or Pub/Sub data.

This design separates finding data from reading data. The control plane in Knowledge Catalog manages the index, lineage, data-quality checks, data profiles, feature metadata, and model artifacts, while the underlying storage systems in BigQuery, Cloud Storage, and Pub/Sub enforce data-plane reads. Domain teams publish reusable data products that appear as listings for discovery, and consumers discover through Knowledge Catalog and request access through governed sharing.

Governed data staging and multi-tiered lakehouse architecture is a design that moves data through progressive, policy-controlled stages from raw ingestion to business-ready consumption. The staging topology combines Cloud Storage for durable object storage, Knowledge Catalog for centralized governance, and the Lakehouse runtime catalog for open-table metadata, with BigQuery as the high-performance consumption layer. Data flows from external and operational sources into lower stages, then is cleansed, conformed, and aggregated as it advances, with authorization, lineage, and lifecycle controls applied at each transition.

2.1Progressive Staging with Medallion Layers and Perimeters

Progressive staging organizes the lakehouse into bronze, silver, and gold logical layers that control data quality and access as data matures. The bronze layer holds raw ingestion from sources such as Amazon S3 Parquet files, Cloud Storage event logs, and live AlloyDB for PostgreSQL data accessed through BigQuery connections. The silver layer holds cleansed and conformed data after large-scale joins, windowing, and behavioral aggregations, often executed by Managed Service for Apache Spark with Lightning Engine. The gold layer holds curated business-level aggregations for analytics and AI, and the documented best practice is to use BigQuery for this gold layer to maximize query performance and concurrency.

This medallion pattern maps directly to governed lake and zone concepts where raw, curated, and refined zones isolate untrusted landing data from trusted products. A separate landing zone and data ingestion perimeter protect the rest of the workloads from incoming batch or streaming data while it is categorized and de-identified. A confidential or data perimeter then isolates sensitive or encrypted content, while a governance perimeter separates key management, catalog management, and logging from the warehouse data itself. Movement between stages is therefore both a data-quality transition and a security-boundary crossing.

2.2Lakehouse Runtime Catalog Hierarchy

The Lakehouse runtime catalog (opens in a new tab) is the top-level regional metadata service that centralizes discovery across Spark, Trino, and BigQuery. The Lakehouse runtime catalog hosts individual catalog instances and implements the open-source Apache Iceberg REST Catalog API to manage namespaces and tables, with extensions for catalog management. Within the resource hierarchy, the Lakehouse runtime catalog acts as the root container that connects open-table metadata in Cloud Storage to compatible query engines.

Organization inside the Lakehouse runtime catalog follows the Project.Catalog.Namespace.Table (P.C.N.T) naming structure. A catalog is a logical metastore container backed by Cloud Storage warehouse locations, and its region automatically matches the underlying bucket's region. A namespace is a logical grouping of tables within a catalog, similar to a database, schema, or BigQuery dataset, and it automatically uses the same region as its parent catalog. Key governance properties of this hierarchy include:

• Access delegation through credential vending: administrators can enable credential vending on the catalog so the catalog autogenerates short-lived credentials with reduced scope instead of granting users direct bucket permissions.
• Namespace security boundaries: IAM roles can be granted at the namespace level to manage access to all tables contained within it.
• Location and nesting rules: options for custom namespace locations depend on the catalog's warehouse type, nested namespaces are not supported, and custom table paths must be nested under the parent namespace path.
• Conflict prevention: resulting table locations automatically receive a random string suffix so no two tables occupy the same location.

2.3Table Types and Open-Format Governance

Table choice in the lakehouse determines where metadata lives, where data lives, and which engines can interoperate. The Lakehouse runtime catalog recommends Apache Iceberg tables created from open-source engines and stored in Cloud Storage (opens in a new tab) , managed through the REST endpoint for open compatibility. BigQuery-managed options keep metadata in the BigQuery catalog while storing data and physical metadata in Cloud Storage, and external tables remain self-managed outside the Lakehouse runtime catalog.

Only Apache Iceberg V2 tables, which are generally available, and V3 tables, which are in Preview, are supported by the Lakehouse runtime catalog. Iceberg V1 tables are not supported and must be upgraded to V2 before use with Lakehouse. An open lakehouse uses these open table formats so the same Cloud Storage files remain compatible with different processing engines, while Dataproc Metastore and the Lakehouse runtime catalog provide managed metastore services that unify governance across BigLake and open-format tables like Iceberg and Delta.

2.4Governed Query and Data Retrieval Flow

The governed query flow is the ordered path a request follows from engine submission through policy check to direct storage read. When a query targets an Apache Iceberg table managed by Lakehouse, the engine first resolves metadata and authorization before touching data files in Cloud Storage. The sequence preserves governance because the catalog validates identity and fine-grained policies before returning locations or vending credentials.

1. Submission: user submits a SQL query to a compatible engine such as Apache Spark, Trino, or BigQuery.
2. Metadata request: engine requests table metadata from the Lakehouse runtime catalog to identify the table and its metadata location.
3. Authorization: catalog validates the request against Identity and Access Management (IAM) and fine-grained security policies, when supported by the endpoint in use.
4. Metadata response: catalog returns the metadata, and if credential vending is enabled, also provides a short-lived token for secure storage access.
5. Data retrieval: engine uses the metadata and optional token to read data files directly from Cloud Storage.
6. Execution: engine processes the data and returns the results.

Because retrieval goes directly from the engine to Cloud Storage after the catalog check, storage permissions, catalog IAM, and vended-token scope must align. If vending is disabled, the caller needs direct bucket access; if vending is enabled, the short-lived token carries reduced-scope access for that read.

2.5Integrity, Lifecycle, and Operational Controls Across Stages

Integrity, lifecycle, and operational controls keep staged data accurate, minimal, and observable as it moves from raw to refined. Knowledge Catalog provides the organization-wide governance plane for lineage, data quality checks, data profiles, discovery across data marts, feature and model metadata, business glossaries, and metadata enrichment through aspects and aspect types. BigQuery adds column-level lineage, while Sensitive Data Protection discovers and classifies sensitive data in ingestion paths including Cloud Storage and BigQuery, and de-identifies structured data with record transformations and unstructured data with infoTypes.

Protection across stages depends on encryption, perimeters, and audit logging. Customer-managed encryption keys (CMEKs) in Cloud KMS with Cloud HSM control encryption at rest for ingestion, Dataflow, Pub/Sub, and separate BigQuery warehouses for non-confidential and confidential data, with keys kept in the same location as resources and rotated by default every 30 days. VPC Service Controls defines the ingestion, confidential or data, and governance perimeters with ingress and egress rules, firewall rules deny by default and allow only defined traffic, and Admin Activity audit logs plus Data Access audit logs track configuration changes and data reads. Cloud Storage lifecycle controls and data-minimization practices limit prolonged retention of unnecessary or sensitive data, including pseudonymization, de-identification, and use of synthetic or anonymized data where possible.

For reliable operation of staging pipelines, standardize interactive work with session templates so notebook environments share configuration, assign custom batch and job names to serverless Spark batches so executions can be filtered in Cloud Logging and the console, and enable diagnostic bundles with retained driver and executor logs for complex pipelines. These practices preserve processing efficiency by making stage transitions repeatable, traceable, and faster to troubleshoot without duplicating data through fragile extract, transform, and load pipelines.

Automated data profiling, classification, and sensitive data protection is the continuous practice of scanning lakes and warehouses to measure quality, locate sensitive data such as personally identifiable information (PII), and enforce access based on classification. On Google Cloud this work is split between Sensitive Data Protection (which includes Cloud Data Loss Prevention API (DLP API)) for discovery, inspection, and de-identification, and Knowledge Catalog for centralized governance, metadata, data quality, and policy enforcement. Together they let raw data flow from ingestion to BigQuery and Cloud Storage while profiles, tags, and masking travel with it to control who can see what.

3.1Continuous Discovery and Data Profiles

Sensitive Data Protection (opens in a new tab) discovery is a continuous scanning service configured at the organization, folder, or project level that automatically finds, classifies, and reports on new data assets as they appear. When someone creates a new project and uploads a large dataset, the discovery service generates data profiles without manual inspection jobs. Each profile contains metrics and insights including sensitivity level, data risk level, infoType coverage, and location, with a premade report that breaks exposure down by risk, by infoType, and by location.

Discovery acts as a source of truth for audit and security operations because profiles can be exported automatically to Security Command Center, Google Security Operations, BigQuery for historical analysis, and Knowledge Catalog as aspects. The export path matters for compliance: if a column is later deleted, no new column data profile is generated, so teams that need audit history must configure the profiler to save profiles to BigQuery before deletion. Before restricting profiled tables with column controls, the Sensitive Data Protection service agent must already hold permission to read them, otherwise profiling returns an error.

3.2Knowledge Catalog Governance and Data Quality

Knowledge Catalog governance and data quality is the organization-wide control plane that centrally discovers, manages, monitors, and governs data and AI artifacts across data platforms. Knowledge Catalog manages column-level lineage, automatic data quality checks and data profiles, discovery and exploration across data marts, feature and model metadata, and a business glossary for standardized vocabulary. Metadata is enriched through aspects and aspect types, which allows inspection results from Sensitive Data Protection to be attached directly to catalog entries.

Knowledge Catalog unifies governance across BigLake and open-format tables such as Iceberg and Delta Lake, and it supports a data mesh where ownership is decentralized to domain teams while controls remain central. In that model, the catalog depends on discovery and quality signals from below, while access policy defined in the catalog applies downward to lakes and warehouses. Teams use automatic data quality capabilities in Knowledge Catalog to validate formats, ranges, and distributions before training, which also helps detect anomalies that can indicate data poisoning.

3.3Classification with InfoTypes and Free-Text Detection

Classification in Sensitive Data Protection is driven by infoTypes and record transformations that identify sensitive patterns in structured and unstructured data. For structured data, record transformations match specific fields, for example credit card numbers and card PINs in a de-identification template applied by a Dataflow pipeline. For unstructured data, infoType detectors such as PHONE_NUMBER, US_SOCIAL_SECURITY_NUMBER, and DATE_OF_BIRTH scan free-form content.

A column with a high free-text score and evidence of multiple infoTypes is a strong signal of hidden PII in notes or comment fields, such as Customer was born on January 1, 1985. Data profiles alone do not locate the exact values, so the remediation path flows from the profile to an on-demand inspection job on the BigQuery table or Cloud Storage bucket to pinpoint findings, then to de-identification. This distinction matters because a structured column can be tokenized by field, while a free-text column must first be inspected to understand where PII occurs.

3.4Taxonomy-Based Policy Tags for Column-Level Security

A BigQuery taxonomy is a hierarchical classification scheme whose policy tags attach to columns to enforce column-level security. Access is granted through the Fine-Grained Reader role on the tag, and anything untagged remains available to anyone with warehouse access. The hierarchy is inclusive downward, which controls how confidential, private, and sensitive data compose:

Policy tags continue to block reads even after data is re-identified, so re-identification in the confidential perimeter does not bypass authorization. Choose automatic tagging from discovery when new tables must inherit sensitivity without manual labeling, and choose manually curated taxonomies when regulatory boundaries require explicit ownership of what counts as confidential.

3.5De-Identification, Tokenization, and Key Management

De-identification is the transformation applied during ingestion that preserves analytical utility while hiding raw identifiers through masking, tokenization with re-identifiable surrogates, pseudonymization, generalization such as bucketing, and character masking. The pipeline flows from Cloud Storage or Pub/Sub through Dataflow, where Sensitive Data Protection tokenizes data tagged as confidential, then into separate BigQuery warehouses for non-confidential and confidential data. For external sources, data is encrypted before upload with the Tink library and stored with the encrypted payload and wrapped key in separate tables, while internal sources are de-identified inline during ingestion.

Cryptographic control depends on Cloud Key Management Service (Cloud KMS) with Cloud HSM managing customer-managed encryption keys (CMEKs). Ingestion uses a CMEK for the Dataflow and Pub/Sub path, a Cloud HSM-wrapped key for Sensitive Data Protection de-identification, and separate dataset default CMEKs for the non-confidential and confidential BigQuery (opens in a new tab) warehouses. Each CMEK must reside in the same location as its resource and rotates by default every 30 days, with Cloud External Key Manager (Cloud EKM) available when compliance requires keys managed outside Google Cloud. At query time, BigQuery remote functions and GoogleSQL AEAD functions extend the same model by de-identifying, re-identifying, or encrypting individual values without copying full tables.

3.6Remediation Based on Data Risk

Remediation based on data risk starts from the data risk and sensitivity calculated in each profile and follows three distinct outcomes. Expected and properly protected data is published for monitoring, unexpected or unprotected data is locked down or de-identified, and unclear findings trigger deeper inspection. For assets already protected, profiles flow into audit reports and into Security Command Center, Google SecOps, and Knowledge Catalog to track severity and business metadata over time.

For BigQuery assets flagged with high data risk, meaning evidence of sensitive data without additional protections, consider the following controls:

• Apply a BigQuery policy tag to sensitive columns to restrict reads to specific principals.
• Create a de-identified or masked copy so analysts and engineers work without raw PII, using built-in de-identified copies for Cloud Storage data.
• Enable automatic tagging and set the profiled asset data risk to Low only after controls are verified.
• Remove the high-risk column or table if the data is not needed for business goals.

After controls are established, monitoring must continue for drift that returns the asset to high risk, including new columns, schema changes, and perimeter or IAM modifications isolated by VPC Service Controls and least-privilege roles.

Key Facts

• Knowledge Catalog automatically ingests metadata from BigQuery, Cloud Storage, and Pub/Sub and enforces entry-level and tag-level IAM checks before returning search results.
• Tags and aspects make assets findable by business meaning, while BigQuery policy tags restrict who can read confidential columns.
• The Lakehouse runtime catalog uses Project.Catalog.Namespace.Table naming and supports only Apache Iceberg V2 and V3 tables, so V1 tables must be upgraded to V2.
• A governed Iceberg query flows from engine to Lakehouse runtime catalog for metadata and authorization, then directly from the engine to Cloud Storage for data files.
• Sensitive Data Protection discovery generates profiles with sensitivity, data risk, infoType coverage, and location, and policy tag 3_Confidential grants access to 2_Private and 1_Sensitive while the reverse is not true.