Prepare and test your skills
Prepare and test your skills
Fine-grained data access governance is the enforcement of least-privilege access to sensitive machine learning features and datasets through layered access controls, column and row filters, network perimeters, and encryption keys. BigQuery (opens in a new tab) sits at the center of this model, where Identity and Access Management (IAM) roles grant coarse access while policy tags, masking rules, and row filters control what a query can actually return. VPC Service Controls (opens in a new tab) and Customer-Managed Encryption Keys (CMEK) (opens in a new tab) add isolation and key ownership around that access layer so data cannot leave the perimeter or be decrypted without explicit grants.
Least-privilege role-based access control means every user, group, and service account receives only the minimum permissions needed for its role, such as no-access, read-only, or write. Least-privilege access protects data lakes, feature stores, and training hyperparameters from data theft, model tampering, and accidental change by shrinking what any identity can affect. For ML pipelines this means dedicated service accounts per stage, where a training pipeline service account can read from only one Cloud Storage bucket and write model artifacts only to Model Registry. Granularity is implemented by mapping product IAM roles to identities based on project needs, access patterns, or resource tags. Vertex AI access control combines predefined and custom IAM roles to create different access patterns for data scientists, annotators, and MLOps operators. Because tables inherit permissions from the dataset that contains them, table-level IAM is used to grant access to specific tables or views without granting access to the entire dataset.
BigQuery column-level security restricts access to specific columns through a taxonomy of policy tags managed in Data Catalog. A column tagged with a policy tag is invisible to a query unless the caller holds both BigQuery data access and the Fine-Grained Reader role on that tag. Anything without a tag remains available to anyone with dataset access, so sensitive fields stay unreadable until access is explicitly granted. A production taxonomy uses a nested hierarchy to model sensitivity levels:
3_Confidential for very sensitive data such as credit card numbers; holders also gain access to 2_Private and 1_Sensitive columns.2_Private for sensitive personally identifiable information (PII) such as first name; holders also gain access to 1_Sensitive columns but not 3_Confidential columns.1_Sensitive for non-public data such as credit limit; holders gain no access to 2_Private or 3_Confidential columns.The hierarchy flows from higher sensitivity downward, so granting 3_Confidential implies the lower tiers but granting 1_Sensitive grants nothing higher. Sensitive Data Protection data profiler supports this design by scanning all BigQuery tables across organization, folders, and projects and reporting predicted infoTypes, data risk, and sensitivity levels for each column.
BigQuery row-level security filters which rows a user can see based on a qualifying condition evaluated at query time. Row-level access policies attach to a table and compare the current user against row values, which supports patterns such as restricting a salesperson to only their assigned accounts. Authorized views provide the complementary pattern, where a view in one dataset queries base tables in another dataset and users query only the view without direct access to the base tables. Row controls differ from column controls in what they hide and how they evaluate, and the choice depends on whether the risk is columns, rows, displayed values, or cryptographic access:
| Control | Hides | Grants access through |
|---|---|---|
| BigQuery column-level security | Entire sensitive columns | Policy tag plus Fine-Grained Reader role |
| BigQuery row-level security | Non-qualifying rows in a table | Row access policy condition on the user |
| BigQuery dynamic data masking | Real column values at query runtime | Masking rule plus role check at query time |
| BigQuery column-level encryption | Cleartext values without the key | Table permission plus key permission |
Dynamic data masking leaves the column queryable but automatically masks values unless the caller holds the masking exemption, typically the Fine-Grained Reader role. Choose policy tags when a group must never see a column, row policies when a group must see only a subset of rows, masking when most users need shape but not values, and encryption when even privileged table readers must not see cleartext without key access.
Exam tip: Column-level security blocks the column, dynamic data masking obscures values at query runtime, and column-level encryption requires both table permission and key permission to read cleartext.
BigQuery column-level encryption encrypts selected sensitive columns with Authenticated Encryption with Associated Data (AEAD) functions instead of encrypting the whole table. BigQuery creates keysets that hold the keys for encryption, decryption, and rotation within the keyset, and those keys encrypt and decrypt individual values in the table. Reading cleartext enforces dual-access control because the user must have permission to both the BigQuery table and the encryption key. Client-side encryption extends this pattern before data enters Google Cloud. The Tink encryption library, which is compatible with BigQuery AEAD, encrypts sensitive columns locally using a data encryption key (DEK) generated locally or from Cloud Hardware Security Module (Cloud HSM). A key encryption key (KEK), stored as a symmetric CMEK keyset in Cloud HSM and governed by IAM, wraps the DEK, and both the wrapped DEK and the encrypted data land in BigQuery in separate tables. At query time, BigQuery AEAD decryption first unwraps the DEK with the KEK and then decrypts the protected column, so key revocation alone blocks cleartext access.
Identity and Access Management (IAM) provides predefined roles for common BigQuery patterns and custom roles for a user-specified list of permissions. Organization and project roles control the ability to run jobs or manage the project, while dataset roles control the ability to access or modify data inside the project. Basic roles such as Owner, Editor, and Viewer predate IAM and should be minimized in production; when predefined and basic roles combine, the effective permissions are the union of both. Fine-grained decisions are enforced with attribute-based controls layered on roles:
Service accounts for ingestion and training follow the same least-privilege rule, where only authorized identities can access the confidential data project and each pipeline identity holds limited roles scoped to its buckets, topics, and datasets.
VPC Service Controls creates a service perimeter that acts as a security boundary around AI and ML resources, including BigQuery datasets, storage buckets, and training services. Data flows only inside the perimeter by default, and cross-boundary movement requires explicit ingress and egress rules that name the protected services, the allowed sources, and the allowed destinations. Placing confidential and non-confidential warehouses in separate projects inside separate perimeters enforces separation of duties and blocks exfiltration from the Virtual Private Cloud (VPC). Network enforcement inside and at the edge of the perimeter denies by default and allows by exception. Firewall policies deny all inbound and outbound traffic and then explicitly allow only traffic that meets port, protocol, and source criteria. For external ingestion, Dedicated Interconnect or Cloud VPN protects data flowing between the on-premises environment and Google Cloud, and allowlisted IP addresses in access levels control which external networks can reach the perimeter. For controlled sharing across privacy boundaries, Analytics Hub and BigQuery data clean rooms provide governed sharing without copying raw confidential tables.
Customer-Managed Encryption Keys (CMEK) in Cloud Key Management Service (Cloud KMS) give the data owner control over the key encryption keys that protect data at rest. BigQuery encrypts all data at rest and in transit by default and this default cannot be disabled, but CMEK replaces Google-managed keys with customer-controlled keys for the ingestion pipeline, Dataflow workers, Pub/Sub topics, and BigQuery warehouses. CMEK-integrated services in this architecture include Agent Platform, Cloud Storage, Cloud SQL, Spanner, and BigQuery. Key lifecycle depends on location alignment and rotation. The CMEK location determines where the key is stored and served, and the key must reside in the same location as the protected resource or requests fail. By default the CMEK rotates every 30 days, and separate CMEK keys isolate the non-confidential warehouse from the confidential warehouse so compromise or rotation in one project does not affect the other. If compliance requires keys outside Google Cloud, Cloud External Key Manager (Cloud EKM) holds the external key, but the customer then assumes responsibility for rotation and availability.
Cloud Sensitive Data Protection is the Google Cloud service for automatically discovering, classifying, and transforming PII before model training. Cloud Sensitive Data Protection (opens in a new tab) inspects datasets with more than 200 built-in classifiers called infoTypes, then applies de-identification to reduce handling risk while preserving data utility for joins and analytics. BigQuery (opens in a new tab) stores the resulting warehouse data, Dataflow (opens in a new tab) executes tokenization during ingestion, and Cloud Run hosts centralized functions for query-time transformation.
Automated discovery with data profiler finds where sensitive and high-risk data lives without manual sampling. Data profiler automatically scans all BigQuery tables and columns across the organization, including all folders and projects, and outputs predicted infoTypes, assessed data risk and sensitivity levels, and table metadata. The profiles are stored in BigQuery for analysis and can be sent to Knowledge Catalog as aspects for lineage, data quality checks, discovery, and governance. Discovery also extends to Vertex AI platform datasets at organization, folder, or project level, while pipeline logs must exclude PII and confidential data to avoid creating a second exposure path.
De-identification transformations replace raw identifiers with privacy-safe values that remain useful for preprocessing and training. The approach follows data minimization: do not collect, keep, or use data beyond business goals, and use synthetic or fully anonymized data where possible. For structured data, Cloud Sensitive Data Protection uses record transformations based on fields, such as checking defined columns for credit card numbers and card PINs. For unstructured data, Cloud Sensitive Data Protection uses infoType detectors to find and transform free-text PII. Transformation choice depends on whether reversibility and analytics utility are required:
The ingestion-time tokenization pipeline de-identifies confidential data before it lands in the training warehouse. Data flows from the source system to the de-identification service to separated warehouse projects inside a VPC Service Controls perimeter. The pipeline depends on Cloud Sensitive Data Protection detectors configured for the confidential fields and on distinct keys for ingestion and storage. The pipeline runs in a fixed order:
Query-time de-identification in BigQuery sanitizes results during the query instead of maintaining a separate de-identified copy. A central service hosted on Cloud Run exposes Sensitive Data Protection operations as BigQuery remote functions, such as fns.dlp_freetext_encrypt, so analysts call de-identification and re-identification through SQL. The BigQuery connection invokes the Cloud Run endpoint, the Cloud Run service calls the Cloud Data Loss Prevention API to de-identify or re-identify each item, and BigQuery returns the transformed values in the query result. This pattern avoids operating a Dataflow cluster for small or on-demand requests.
Cost and quota control require passing only final, filtered results to the function and keeping the BigQuery dataset, Cloud Run service, and Sensitive Data Protection templates in the same cloud region. The solution batches requests to handle Sensitive Data Protection limits, including a maximum of 50,000 table values and a default request size limit of 0.5 MB. To control total quota consumption, limit items passed through the remote function to 10,000 or less, where each value in a pii_column counts as one item.
Key management for de-identification pipelines separates ingestion protection from token transformation and warehouse storage. Cloud Key Management Service (Cloud KMS) with Cloud Hardware Security Module (Cloud HSM) manages customer-managed encryption keys, and the key location must match the location of the resources that use it. By default each CMEK rotates every 30 days, and organizations that must hold keys outside Google Cloud can use Cloud External Key Manager (Cloud EKM) but then own rotation and management. The internal-source ingestion model uses three key scopes:
For external sources, client-side encryption with Tink adds protection before data enters Google Cloud. Tink uses locally generated data encryption keys wrapped by a symmetric CMEK key-encryption key held in Cloud HSM, and BigQuery stores the encrypted data and the wrapped DEK in separate tables. BigQuery then uses AEAD decryption functions to unwrap the DEK with the KEK and decrypt the protected column only for authorized reads.
Column-level enforcement in BigQuery controls who can read re-identified or still-sensitive columns after transformation completes. Policy tags attach a sensitivity taxonomy to columns, dynamic data masking applies masking rules at query runtime based on the caller's roles, and column-level encryption encrypts only selected columns with AEAD keysets. Column-level encryption enforces dual-access control because the reader must have permission to both the table and the encryption key to see cleartext. The secured-warehouse taxonomy defines three hierarchical levels, and anything untagged remains available to all users with warehouse access:
| Policy tag | Example data | Access result |
|---|---|---|
| 3_Confidential | Credit card numbers | Grants access to 3_Confidential, 2_Private, and 1_Sensitive columns |
| 2_Private | First name and other sensitive PII | Grants access to 2_Private and 1_Sensitive columns, denies 3_Confidential columns |
| 1_Sensitive | Credit limit and other non-public data | Grants access to only 1_Sensitive columns, denies 2_Private and 3_Confidential columns |
Access to masked or tagged columns requires the Fine-Grained Reader role, and the default masking rule hides values such as credit limit until explicit access is granted. These controls ensure that even after data is re-identified by token reversal or AEAD decryption, the sensitive fields still cannot be read until access is explicitly granted to the user.
Exam tip: Use record transformations for structured field-based de-identification and infoTypes for unstructured free-text de-identification; in both cases re-identified BigQuery columns still require the Fine-Grained Reader role plus the matching policy tag.
Privacy-preserving machine learning on Google Cloud is the combination of data transforms, encryption, perimeters, access controls, and governance that lets teams preprocess, train, and share models without exposing PII. Sensitive Data Protection, BigQuery (opens in a new tab), Knowledge Catalog, VPC Service Controls (opens in a new tab), and Identity and Access Management (IAM) divide the work: transforms reduce identifiability, encryption and perimeters block exfiltration, IAM limits who can see what, and catalogs plus logs prove what happened. Together these controls reduce membership inference, model inversion, reconstruction of training inputs, and poisoning of training data.
Data minimization is the principle of not collecting, keeping, or using data that is not strictly necessary for the business goal, preferring synthetic or fully anonymized data where possible. Minimization matters because collecting PII or sensitive attributes increases bias risk, leakage risk during training, and compliance exposure across teams. Sensitive Data Protection implements utility-preserving transforms before data reaches training or sharing. Data flows from raw ingestion through inspection to a de-identified warehouse copy. Inspection uses infoTypes and data profiles to discover PII across BigQuery, Cloud Storage, and Vertex AI datasets, then Sensitive Data Protection applies record transformations for structured data and infoType detectors for unstructured data. Tokenization via a Dataflow pipeline moves confidential values from Cloud Storage into BigQuery while replacing raw values, and Cloud Storage lifecycle rules then limit prolonged retention and manipulation of the raw copy.
BigQuery differential privacy is a privacy-enhancing technology that adds noise to query results so individual rows cannot be identified while aggregate patterns remain learnable. The mechanism applies differentially private aggregations in SQL, transforming the output of the query rather than storing a masked copy of the table. Because noise is added at query time, repeated inference queries reveal less about whether a specific person was in the training set, which directly counters membership inference and reconstruction from outputs. De-identification and differential privacy depend on each other but operate at different layers. Pseudonymization, generalization such as bucketing, and tokenization reduce identifiability of stored values shared with preprocessors and trainers, while differential privacy protects the answers returned to analysts and models that learn from aggregates. Choose stored-value transforms when teams need to share usable datasets, and choose differentially private aggregations when teams need to publish statistics or train on counts and sums without exposing contributors.
Controlled collaboration lets separate teams analyze shared data without copying raw PII across trust boundaries. BigQuery sharing through Analytics Hub and BigQuery data clean rooms provide a security and privacy framework where a provider publishes data once and consumers query it inside governed boundaries. Data flows from the provider project to the authorized consumer query, and the perimeter plus IAM policy controls what leaves the boundary. Looker Action Hub extends the same pattern from dashboards to destinations by sending results directly into built-in destinations from a secure cloud environment. Use Analytics Hub when the goal is governed dataset sharing at scale, use data clean rooms when multiple parties must join and analyze sensitive data under restricted disclosure, and use Action Hub when the handoff starts from business intelligence dashboards.
Encryption on Google Cloud protects data at rest by default, in transit, and optionally while in use, with customer control through keys. Customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) let the customer control key generation, storage, and rotation while Google-managed services encrypt at rest. The CMEK location determines where the key is stored and must be in the same location as the protected resource, with rotation every 30 days by default and Cloud External Key Manager (Cloud EKM) available when keys must stay outside Google Cloud. The following CMEK-integrated services encrypt AI data at rest when configured with Cloud KMS:
Server-side encryption protects CMEKs held on customer servers, HTTPS protects all API calls to AI and ML services with HTTPS load balancers to enforce it, and Cloud HSM wraps the data-encryption keys used for ingestion and de-identification. For column granularity, BigQuery AEAD encryption functions encrypt only selected columns with keysets that support rotation, creating dual-access control where the reader needs both table permission and key permission to see cleartext. Client-side encryption with Tink extends this boundary to before upload, storing both the encrypted column and the wrapped data-encryption key in BigQuery so unwrapping with the key-encryption key and AEAD decryption happens only for authorized analysts. For data in use, Confidential Computing and Shielded VMs provide a trusted execution environment so sensitive data and code are protected in memory during training, ideally inside dedicated projects or VPCs that isolate training from other workloads.
Network perimeters create a security boundary around AI resources so training data and model artifacts cannot be exfiltrated from the Virtual Private Cloud (VPC). VPC Service Controls defines a service perimeter that includes AI and ML resources and sensitive data, with ingress rules controlling data flow in and egress rules controlling data flow out. Traffic that violates the perimeter policy is denied even when the caller has IAM permission, so the perimeter depends on IAM but enforces a separate boundary. Firewall rules restrict inbound and outbound traffic to AI resources under a deny-all-by-default policy that explicitly allows only matching traffic. Training and serving further reduce exposure by using Private Service Connect interfaces and private endpoints so custom jobs communicate privately with authorized Google Cloud services, and by using private IP addresses for training jobs instead of public IPs. Choose VPC Service Controls when the risk is copying data between Google-managed services, and choose firewall plus private connectivity when the risk is network ingress or egress to the training and prediction endpoints.
Role-based access control with least privilege assigns each user, group, or service account only the minimum permissions needed for its role, such as no-access, read-only, or write. Least privilege limits data theft from data lakes, feature stores, and hyperparameters, preserves model integrity, and shrinks the surface for accidents and attacks. Access is evaluated per request against IAM roles, IAM Conditions, and context, with audit logs recording who accessed or modified data and models. Granularity spans project, resource, and data layers. IAM roles map to users, groups, or service accounts by project need, access pattern, or tag, while IAM Conditions enforce attribute-based rules such as allowing a pipeline trigger only from a trusted Cloud Build trigger. Application-level controls then narrow inside the resource: Cloud Storage enforces IAM on buckets and managed folders, BigQuery enforces IAM on datasets plus row-level and column-level security, and Vertex AI access control combines predefined or custom roles for models, endpoints, pipelines, and Feature Store resources. Principal access boundary policies cap what a principal can reach, Privileged Access Manager grants just-in-time temporary elevation with later audit, Identity-Aware Proxy adds IP and device context, and Access Context Manager re-evaluates Workbench access at each authentication. Pipelines must use dedicated least-privilege service accounts and must never log PII or confidential data in experiments, Cloud Logging, or experiment servers. Column protections differ in where they enforce and what the reader needs:
| Protection | Enforcement point | Reader needs for cleartext |
|---|---|---|
| BigQuery column-level security with policy tags | Access check on the tagged column | Explicit grant to the tagged column |
| BigQuery dynamic data masking | Masking rule applied at query runtime | Fine-Grained Reader role, otherwise masked value returns |
| BigQuery column-level encryption with AEAD keysets | Cryptographic decrypt of the value | Permission on the table plus permission on the encryption key |
Exam tip: Column-level encryption requires both table access and key access to read cleartext, while masking and column-level security rely on IAM grants alone.
Governance for PII is a centralized layer that discovers data, tracks its transformations, and monitors access across teams. Knowledge Catalog provides organization-wide discovery, management, monitoring, and governance of data and AI artifacts, including data lineage with BigQuery column-level lineage, data quality checks and profiles, business glossary vocabulary, metadata enrichment with aspects, unified governance across BigLake and open formats like Iceberg and Delta Lake, and data mesh ownership across domains. Sensitive inspection results flow from BigQuery into Knowledge Catalog, and Vertex AI dataset profiles stored in BigQuery reveal where PII concentrates. Operational monitoring depends on that catalog plus telemetry. Vertex AI Feature Store schedules monitoring of features and feature groups for drift, Cloud Logging captures real-time pipeline logs, Cloud Monitoring collects metrics and alerts, and Cloud Audit Logs capture API calls and administrative actions for audit and unauthorized-access detection. Vertex AI ML Metadata logs sources, transformations, parameters, and experiment results with automatic encryption at rest, Model Registry versions each model and links it to its dataset version, code version, and evaluation metrics, and Knowledge Catalog tracks pipeline artifact lineage for debugging and root-cause analysis. Export audit logs to BigQuery for long-term analysis and stream them to Cloud Logging or a SIEM for real-time response.
AI-specific attack vectors include data poisoning, where injected training rows manipulate behavior or degrade performance, model inversion, where outputs are used to reconstruct sensitive inputs, and adversarial attacks that exploit model sensitivity. The attack surface changes as data, code, endpoints, and integrations change, so mitigation combines secure by design posture management with pipeline validation and robust training. Security Command Center provides cloud posture management with attack exposure scores and paths, Google Threat Intelligence tracks emerging AI techniques, and logging, monitoring, and vulnerability management close the loop on detection and remediation. Poisoning mitigation validates integrity before training and hardens learning against residual bad rows. Automatic data quality checks in Knowledge Catalog verify formats, ranges, and distributions, Sensitive Data Protection with Model Armor discovers, classifies, and protects sensitive and intellectual-property data including in large language model interactions, and BigQuery anomaly detection flags statistical outliers with statistical or ML models. Robust training then trains multiple models on different data subsets with hyperparameter tuning under ensemble methods so poisoned points affect fewer models, while data augmentation rebalances distributions and can add adversarial examples. Human review through Vertex AI model evaluation and Gen AI evaluation detects bias, anomalies, and suspicious patterns that automated checks miss, and comprehensive monitoring of data and ML pipelines tracks inconsistent behavior, unexpected transformation, or unauthorized access that signals tampering.
BigQuery column-level security hides entire sensitive columns and requires both BigQuery data access and the Fine-Grained Reader role on the policy tag, while BigQuery row-level security hides non-qualifying rows based on a row access policy condition evaluated at query time. BigQuery dynamic data masking leaves the column queryable but automatically masks values at query runtime unless the caller holds the masking exemption, typically the Fine-Grained Reader role. BigQuery column-level encryption encrypts selected columns with Authenticated Encryption with Associated Data (AEAD) functions and enforces dual-access control so the reader needs permission to both the BigQuery table and the encryption key to see cleartext.
Access flows from higher sensitivity downward so granting 3_Confidential grants access to 2_Private and 1_Sensitive columns, granting 2_Private grants access to 1_Sensitive but not 3_Confidential, and granting 1_Sensitive grants nothing higher. 3_Confidential covers very sensitive data such as credit card numbers, 2_Private covers sensitive personally identifiable information (PII) such as first name, and 1_Sensitive covers non-public data such as credit limit, while anything without a tag remains available to anyone with dataset access.
Use BigQuery remote functions hosted on Cloud Run for query-time de-identification when sanitizing results during the query for small or on-demand requests to avoid operating a Dataflow cluster. The BigQuery connection invokes the Cloud Run endpoint which calls the Cloud Data Loss Prevention API and returns transformed values, with the dataset, service, and templates kept in the same cloud region and only final filtered results passed to the function. To handle limits and control quota, batch requests within a maximum of 50,000 table values and a 0.5 MB default request size and limit items to 10,000 or less per call.
Last updated