Column-level security and dynamic data masking in BigQuery control access to sensitive data fields by evaluating user roles against data classifications at query time. Administrators define a taxonomy of policy tags in Data Catalog using hierarchical sensitivity tiers, such as 3_Confidential, 2_Private, and 1_Sensitive. Access to a higher-level tag automatically grants access to lower-level tags within the hierarchy, but lower-level tags do not grant access upward. A user must have both standard table read permissions and the Fine-Grained Reader role on the specific policy tag to view raw data. When users lack the Fine-Grained Reader role, dynamic data masking intercepts the query output and returns transformed values, such as hashes or partial masks, without altering the underlying stored records.
Column-level encryption protects individual data values within BigQuery tables using Authenticated Encryption with Associated Data (AEAD) functions. Administrators store encryption keys inside keysets and apply AEAD functions to encrypt specific table columns while leaving non-sensitive columns in plaintext. Reading cleartext data requires dual-access control: a querying user must have standard BigQuery table permissions and explicit read access to the specific cryptographic keys. Primary key management relies on Cloud Key Management Service (Cloud KMS) and Cloud Hardware Security Module (Cloud HSM) to manage customer-managed encryption keys (CMEK). Cloud KMS rotates CMEKs automatically every 30 days by default, and keys must reside in the same geographic region as the datasets they protect.
Service account impersonation grants temporary, credential-free access to sensitive machine learning environments by generating short-lived security tokens. Instead of downloading long-lived JSON service account keys, data practitioners authenticate with their user identity and request a temporary access token for an authorized service account. Administrators configure this workflow by granting the user the Service Account Token Creator and Service Account User roles on the target service account. The target service account holds only the minimal permissions required for the workload, such as reading a specific BigQuery feature store. This pattern prevents credential leakage, centralizes access management, and produces clear identity audit trails in Cloud Audit Logs.
Cloud Sensitive Data Protection (formerly Cloud DLP) automatically discovers, classifies, and assesses sensitive data across storage repositories and analytical warehouses. The built-in data profiler continuously scans BigQuery tables across an entire Google Cloud organization to calculate data risk scores and identify sensitive fields. For deeper examinations, on-demand inspection jobs scan individual table cells using more than 200 built-in pattern classifiers called infoTypes. These infoTypes detect specific personally identifiable information (PII), including national identification numbers, credit card numbers, and email addresses. Security teams use these profiling metrics to identify unprotected fields and direct encryption or masking policies to the appropriate columns.
Batch and streaming de-identification pipelines use Cloud Dataflow and Cloud Sensitive Data Protection to sanitize datasets during ingestion before writing to analytical storage. In this architecture, Cloud Dataflow reads raw files from Cloud Storage or streaming events from Cloud Pub/Sub and batches the records. Cloud Dataflow transmits these batches to the Cloud Sensitive Data Protection API, which applies configured de-identification templates to redact or tokenize sensitive elements. The sanitized records flow from Cloud Dataflow into destination BigQuery tables, ensuring raw PII never enters analytical datasets. To avoid request failures and maintain throughput, pipelines must adhere to specific operational constraints:
Cloud Sensitive Data Protection provides multiple transformation techniques to remove identifying information while preserving statistical utility for machine learning models:
Cryptographic methods rely on a token encryption key wrapped by Cloud KMS and Cloud HSM within a de-identification template. Rotating the wrapping key preserves data accessibility, but rotating the root token encryption key invalidates existing tokens and requires re-tokenizing the source data.
BigQuery remote functions execute real-time de-identification and re-identification directly inside SQL queries without requiring persistent streaming pipelines. When a user runs a query containing a remote function, BigQuery sends the selected rows through a BigQuery Connection to an external endpoint running on Cloud Run. The Cloud Run service acts as a translation layer, passing incoming records to the Cloud Sensitive Data Protection API to execute the assigned de-identification template. The Cloud Run service then returns the transformed values to BigQuery to complete the query execution. Queries using remote functions should operate on filtered subsets of 10,000 items or fewer to prevent API timeouts and optimize execution costs.
Differential privacy injects calibrated statistical noise into query results or training datasets to prevent reconstruction and membership inference attacks against machine learning models. Membership inference attacks occur when an attacker analyzes model behavior to deduce whether a specific individual's record was part of the original training data. By adding mathematical noise, differential privacy masks the presence or absence of any single data record while maintaining aggregate trends across the whole dataset. BigQuery natively supports differential privacy by providing specialized aggregation functions directly inside standard SQL queries. Applying differential privacy algorithms during feature generation ensures that exported training data preserves user privacy before model optimization begins.
Client-side encryption with the open-source Tink cryptographic library protects sensitive features before records are uploaded to Google Cloud infrastructure. An application uses Tink locally to generate a Data Encryption Key (DEK) that encrypts sensitive column values prior to network transmission. The application then uses a Key Encryption Key (KEK) managed in Cloud HSM to wrap and protect the DEK. Both the ciphertext and the wrapped DEK are loaded into BigQuery, ensuring unencrypted cleartext never touches cloud storage. When authorized users query the table, BigQuery AEAD functions unwrap the DEK using Cloud KMS and decrypt the field values on the fly.
VPC Service Controls establish isolated security perimeters around Google Cloud resources to prevent sensitive machine learning data from being exfiltrated. A service perimeter isolates services like BigQuery, Cloud Storage, and Vertex AI inside a boundary that blocks unauthorized network ingress and data egress. When teams need to share machine learning features across different organizational projects, administrators configure perimeter bridges or fine-grained egress rules. These network configurations restrict data movement to authorized source and destination projects without opening the data to the public internet. Inside BigQuery, teams share curated data subsets across projects using authorized views, which inherit column-level policy tags from base tables automatically.
Data lineage and centralized cataloging ensure complete traceability of sensitive features from source ingestion through model deployment. Dataplex Universal Catalog tracks data lineage by mapping how raw datasets move through transformations and pipeline tasks. Model Registry in Vertex AI links trained model versions to their specific training code, performance metrics, and source dataset versions tracked by Dataplex. For auditing and governance, Cloud Audit Logs records all environment interactions, dividing events into default Admin Activity logs and configurable Data Access logs. Security teams monitor Data Access logs alongside Security Command Center to track who viewed sensitive features and identify potential policy violations across machine learning workflows.
StudyCerts
StudyCertsroles/datacatalog.categoryFineGrainedReader) on a policy tag to view protected data in plaintext.Format-Preserving Encryption (FPE) encrypts data while keeping the original character set and length, making it suitable for fields that must adhere to strict legacy schemas like phone numbers. Deterministic Encryption (DE) produces distinct ciphertext strings that allow machine learning systems to join records across tables without revealing the underlying plaintext.
Dynamic data masking transforms sensitive column values at query runtime based on the user's IAM roles, leaving the underlying stored table data intact. Static de-identification irreversibly modifies or tokenizes the data during pipeline ingestion before writing the sanitized records to persistent storage.
BigQuery remote functions should be used for ad-hoc, in-flight SQL transformations on smaller, filtered subsets of 10,000 items or fewer. Cloud Dataflow should be selected for high-throughput, large-scale batch and streaming pipelines that ingest raw data continuously into the analytical warehouse.
VPC Service Controls create a secure boundary around Google Cloud resources, such as BigQuery and Cloud Storage, blocking unauthorized network egress to the public internet or unapproved projects. They allow controlled data sharing between projects through perimeter bridges while preventing data exfiltration.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills