professional-cloud-data-engineer
Gauge your current knowledge
Gauge your current knowledge
Dynamic data masking automatically hides sensitive information in query results based on who is accessing the data, without changing the underlying data or creating separate copies. The system examines the querying identity's access levels and applies appropriate masking rules before returning results, ensuring users only see what they are authorized to view.
Dataplex policy tags classify and protect sensitive columns in BigQuery at the column level. These tags are managed through Data Catalog and allow administrators to define fine-grained access controls that restrict who can see unmasked data. When a user without sufficient permissions queries a table containing sensitive columns, the data is automatically obfuscated according to the masking rules associated with the policy tags.
The configuration process involves several key steps. First, administrators must create a taxonomy of policy tags in Dataplex that reflects their organization's data classification scheme, such as categorizing data as personally identifiable information (PII), financial data, or health information. Next, these policy tags are applied to specific columns in BigQuery tables using Data Catalog, establishing the security boundaries for each data element.
Sensitive Data Protection (formerly Cloud DLP) plays a crucial role in identifying what data needs protection. The service can automatically scan data assets to discover sensitive information like credit card numbers, social security numbers, and other PII. This discovery process helps administrators understand where sensitive data resides and which columns require policy tag protection and dynamic masking.
The enforcement of dynamic data masking happens at query time through BigQuery's integration with IAM. When a query requests data from a column with policy tags, BigQuery evaluates the user's permissions against the tag's access requirements. Users with viewer roles might see partially masked data like "XXX-XX-1234" for social security numbers, while users with owner or editor roles see the complete unmasked data. This approach follows the principle of least privilege, ensuring that sensitive information is only accessible to those who genuinely need it for their work.
Security in BigQuery is intrinsically linked to Google Cloud's Identity and Access Management (IAM) system, which provides centralized, resource-level permissions. You assign predefined or custom IAM roles to users, groups, or service accounts to grant permissions on resources like projects, datasets, and tables. For more granular control beyond dataset- or table-level access, BigQuery offers several key features. Authorized views allow you to share a subset of data from a base table without granting direct access to the underlying table itself. For row-level security, you can create policies that filter which rows a user can see based on their identity or other conditions, such as their email domain. Column-level security is achieved using policy tags and data classification, restricting access to sensitive columns to only users granted a specific Fine-Grained Reader role.
Column-level dynamic data masking automatically obscures column data, such as showing only the last four digits of a credit card number, at query runtime based on the user's roles and predefined masking rules. For stronger protection, column-level encryption allows you to encrypt individual values within a column using cryptographic keys, providing dual-access control where a user must have both table permissions and access to the encryption key to read the plaintext data. All data in BigQuery is encrypted at rest and in transit by default, with the option to use customer-managed encryption keys (CMEK) in Cloud KMS for greater control.
To discover and classify sensitive data at scale, you can use Sensitive Data Protection to scan BigQuery tables and identify sensitive information types like credit card numbers or personally identifiable information. This scanning supports both broad data profiling across an organization and targeted on-demand inspection of specific tables. Findings from Sensitive Data Protection can inform your policy tag taxonomy and guide data de-identification processes, such as tokenization, which can be applied before data is loaded into BigQuery or via Dataflow pipelines.
When connecting visualization tools like Looker Studio or Looker to BigQuery, it is critical to manage access securely. Use service accounts with limited IAM roles for these tools instead of broad user accounts. Enforce service account impersonation where users must explicitly assume a secure service account role to run queries, avoiding the use of long-lived service account keys. Ensure that any queries run through these tools leverage the underlying row- and column-level security policies, as a user's permissions are enforced regardless of the tool used to access the data, maintaining consistent security.
Creating an automated data transformation pipeline is crucial for securing personally identifiable information (PII) before it reaches analysis or visualization stages. Utilizing Sensitive Data Protection services, organizations can build a data de-identification streaming pipeline using Dataflow to ingest raw CSV files from Cloud Storage and output protected records to BigQuery. This pipeline leverages over 100 built-in classifiers to inspect uncharacterized data and apply targeted de-identification methods. By processing data in transit, teams reduce the risk of exposing sensitive identifiers while preserving the utility of the dataset for downstream analysis.
Choosing the correct de-identification method depends on formatting requirements and whether the process must be reversible. For legacy compatibility, Format-Preserving Encryption (FPE) preserves the exact length and character set of the input, whereas Deterministic Encryption (DE) offers stronger authentication. Both methods allow for reversing tokens, but other techniques do not support de-tokenization. Cryptographic hashing provides data integrity but cannot be reversed. Masking and redaction overwrites or hides original values using characters like #. Bucketing and date-shifting reduces the uniqueness of data, such as grouping ages into ranges. Careful method selection allows administrators to balance legacy application support with robust data protection.
Securely managing configurations and cryptographic keys is essential to prevent unauthorized re-identification of sensitive data. Security administrators should deploy Sensitive Data Protection templates to decouple configuration from implementation, allowing the reuse of rules across multiple datasets. Cryptographic transformations require a token encryption key, which must be protected using Cloud KMS to generate a wrapped key. Rotating this token encryption key breaks the integrity of the tokenization, requiring a complete re-tokenization of the entire dataset.
Once de-identified data is stored in BigQuery, multiple layers of access control can be enforced to maintain data security. Administrators can use dynamic data masking to automatically hide column values at query runtime based on user roles and specific policy tags. Additionally, column-level security restricts access to highly sensitive fields by requiring specialized roles, like the Fine-Grained Reader role. The data profiler tool automatically scans tables to identify where high-risk data exists, generating metrics and risk levels to inform security strategies. These post-ingestion controls guarantee that sensitive fields remain protected even after the data is loaded into the warehouse.
Protecting data during transit into Google Cloud requires a combination of network perimeters and encryption mechanisms. Organizations can utilize VPC Service Controls to establish security perimeters around managed services, mitigating the risk of unauthorized data exfiltration. When importing data from external networks, secure connections should be established using either Cloud VPN or Cloud Interconnect. To secure data before ingestion, client-side encryption can be performed using the Tink library, which secures columns at rest using data encryption keys (DEKs). Enforcing these perimeter and transport security controls prevents bad actors from intercepting sensitive datasets before de-identification occurs.
StudyCerts
StudyCerts