Identity and Access Management for Data Workloads

Implementing fine-grained access and context-aware security means every request to data is checked against identity, data sensitivity, and request context before access is allowed. In Google Cloud, Identity and Access Management (IAM) (opens in a new tab) provides role-based permissions, BigQuery (opens in a new tab) enforces table-, column-, and row-level rules, IAM Conditions and Access Context Manager add device, network, and time context, and VPC Service Controls blocks data from leaving a defined network perimeter. Traffic flows from principals through IAM grants to datasets, then through data-level policies and perimeter checks before rows or columns are returned.

1.1Least-Privilege IAM Foundation

The least-privilege IAM foundation grants users, groups, and service accounts only the no-access, read-only, or write permissions required for their job function. Organization and project roles control the ability to run jobs or manage projects, while dataset roles control the ability to read or modify data inside a project, and tables inherit permissions from the containing dataset unless table-level access overrides that inheritance. Predefined roles cover common patterns and custom roles cover a user-specified list of permissions, with dedicated service accounts preferred for pipelines so a training pipeline can read only one Cloud Storage bucket and write only to Model Registry.

Grants can also be bounded after they are assigned. Principal access boundary (PAB) policies define the maximum resources a principal can reach even when other policies grant more, and Privileged Access Manager provides just-in-time temporary elevation whose use appears in audit logs. Separation of duties splits administration across groups, bindings are reviewed regularly with policy analysis, and designs must respect the soft quota of 100 service accounts per project, which limits project-per-tenant patterns that create one account per tenant.

1.2BigQuery Fine-Grained Data Security

BigQuery fine-grained data security controls which tables, columns, and rows a principal can read after dataset access is granted. BigQuery table-level access control grants a user, group, or service account access to a specific table or view without granting access to the whole dataset. BigQuery column-level security restricts sensitive columns through policy tags in a taxonomy, BigQuery column-level dynamic data masking obscures values for some groups while keeping column access, and BigQuery row-level security filters rows based on a qualifying condition on the user.

Policy tags form a hierarchy where higher sensitivity includes lower levels. A 3_Confidential tag for credit card numbers also grants 2_Private and 1_Sensitive, a 2_Private tag for first names grants 1_Sensitive but not 3_Confidential, and a 1_Sensitive tag for credit limit grants neither higher level. Columns without tags remain available to anyone with dataset access, access to a protected column requires the Fine-Grained Reader role on the tag, and the column policy still blocks reads after tokenized data is re-identified until access is explicitly granted.

Tenant isolation for large fact tables can use authorized views or tenant-specific subset tables, which differ in scale, performance, and governance:

1.3Conditional and Context-Aware Access for Sensitive Data

Conditional and context-aware access adds where, when, and from what device a request is allowed to the question of who the principal is. IAM Conditions enforce attribute-based access control on a grant, such as allowing a service account to trigger a Vertex AI pipeline only when the request originates from a trusted Cloud Build trigger. Identity-Aware Proxy (IAP) policies can add IP address and end-user device attributes, while Access Context Manager with Chrome Enterprise Premium evaluates access on every authentication to a Workbench instance and on every API request to the Google Cloud console and APIs.

Chrome Enterprise Premium is user-centric and rejects user API traffic that fails conditions such as IP-based access or certificate-based access for mutual Transport Layer Security (mTLS), while VPC Service Controls is resource-centric and applies to both user and service account identities. Session length for Google Cloud services must be set between 1 and 24 hours, with a 16-hour default automatically enforced for new customers, which forces regular reauthentication with password or security key. For Compute Engine Secure Shell (SSH) access, OS Login with 2-Step Verification adds a password or security-key checkpoint that blocks an attacker holding only stolen OAuth tokens.

1.4VPC Service Controls and Exfiltration Defense

VPC Service Controls mitigates data exfiltration by placing Google-managed resources inside a security perimeter and controlling movement across that perimeter. Sensitive data is placed in its own perimeter, IAM and multifactor authentication (MFA) protect that data, and ingress rules control what can enter while egress rules control what can leave. Traffic flows from inside the perimeter to Google APIs only when origin and target satisfy perimeter rules, so requests with compromised credentials from attacker-controlled endpoints outside the perimeter are blocked even when IAM would otherwise allow them.

A common pattern uses an internal-data perimeter with BigQuery enforcement for core assets, an externally-shared-data perimeter for shareable datasets, and a perimeter bridge that allows egress only from internal-data projects into external-share projects. Firewall policies deny all traffic by default and allow only matching criteria for AI and data resources, while organization policies add guardrails such as disabling external IP allocation, restricting resource creation to specific locations, and disabling service account or key creation. VPC Service Controls is limited to 10,000 projects inside perimeters organization-wide and 100 perimeters including bridges per organization, so high-scale project-per-tenant designs should prefer dataset-per-tenant designs instead.

Designing least-privilege IAM frameworks for data workloads means binding every human, group, and service account to only the minimum role needed on a specific resource. In Google Cloud, Cloud IAM enforces this by binding a principal to a role on a resource, so access flows from the principal through the role to the data. The framework spans lakes, warehouses, feature stores, and pipelines, where organization and project roles control job execution and project management while dataset and bucket roles control data reads and writes, and permissions inherit downward so a BigQuery table inherits from its dataset unless narrowed.

2.1Role Design with Predefined and Custom Roles

Role design with predefined and custom roles matches Cloud IAM roles to job functions instead of granting broad ownership. Predefined roles support common patterns such as BigQuery data viewer or editor workflows, while custom roles provide granular access according to a user-specified list of permissions. Choose predefined roles when a standard pattern fits, and choose custom roles when the workload needs a smaller set than any predefined role allows, such as a pipeline that can read one dataset but cannot list others.

Least-privilege assignment prefers groups because membership can change while the role binding stays stable, and IAM Recommender analyzes actual use to suggest reductions over time. Separation of duties follows an ordered review cycle:

1. Understand roles: inventory predefined roles and permissions for BigQuery, Cloud Storage, and related services.
2. Define policy: state which duties must not combine in one principal, such as data engineering and security administration.
3. Create groups: create Cloud IAM groups for analysts, engineers, encrypted-data viewers, plaintext readers, and administrators.
4. Grant roles to groups: find least-privilege roles and grant them to groups rather than individual users.
5. Analyze and adjust: regularly analyze bindings with policy analysis and replace broad grants with custom roles where necessary.

2.2Fine-Grained Access for BigQuery and Cloud Storage

Fine-grained access for BigQuery and Cloud Storage applies Cloud IAM at the lowest resource level that still allows the job to run. In Cloud Storage, IAM policies apply to buckets and managed folders, so a training service account can be limited to reading only one specific bucket. In BigQuery, roles apply to organizations, projects, datasets, and resources within datasets including tables, columns, and views, so a user can receive access to specific tables or views without receiving access to the complete dataset.

Table-level permissions determine who can reach a table or view, and column and row controls then narrow what those principals see inside it:

A common taxonomy uses 3_Confidential for credit card numbers, 2_Private for personally identifiable information (PII) like first name, and 1_Sensitive for non-public data like credit limit. Access flows downward in sensitivity, so 3_Confidential access includes 2_Private and 1_Sensitive while 1_Sensitive alone cannot read higher levels. Untagged columns remain available to dataset readers, and controls still apply after re-identification until tag access is explicitly granted.

2.3Service Accounts and Secure Impersonation for Automated Pipelines

Service accounts and secure impersonation provide machine identity for automated pipelines without distributing long-lived keys. A dedicated least-privilege service account owns exactly one pipeline stage, so a training account holds permission to read data from only a specific Cloud Storage bucket and to write model artifacts to Model Registry and nothing else. The pipeline runtime authenticates as that account when it calls BigQuery, Cloud Storage, or Vertex AI APIs, which keeps the trust boundary between human users and automation explicit.

By default, Dataflow (opens in a new tab) attempts to use the Compute Engine default service account, which often has broad Editor permissions across the project. A secure design creates a dedicated non-privileged worker service account and passes it at submission with --serviceAccount=YOUR_SERVICE_ACCOUNT@YOUR_PROJECT_ID.iam.gserviceaccount.com, so workers receive only pipeline permissions and route to Google APIs through Private Google Access when public IPs are disabled. Secure impersonation removes key files because a human or automation identity holding Service Account User (roles/iam.serviceAccountUser) and Service Account Token Creator (roles/iam.serviceAccountTokenCreator) on the target account requests a short-lived token to act as that account, while Workload Identity Federation extends the same pattern to external identities without external credentials.

2.4Conditional and Just-in-Time Controls to Enforce Boundaries

Conditional and just-in-time controls enforce least privilege by adding attributes, scope limits, and expiration to an otherwise valid role grant. IAM Conditions enforce conditional attribute-based access control, for example allowing a service account to trigger a Vertex AI pipeline only if the request originates from a trusted Cloud Build trigger, or granting access based on resource tags, IP address, and device attributes. The condition is evaluated on every request, so traffic from outside the trusted source, network, or device posture is denied even when the principal holds the role.

Three complementary controls bound what an elevated principal can do and for how long:

• Principal access boundary (PAB) policies limit access to certain resources by defining maximum usable permissions, which blocks template or cross-project access even when a role would otherwise allow it.
• Privileged Access Manager controls just-in-time temporary elevation for select principals, with later review through audit logs of the elevation activity.
• Identity-Aware Proxy (IAP) with Access Context Manager and Chrome Enterprise Premium restricts access based on IP address and device attributes, with access re-evaluated each time a user authenticates to resources such as Vertex AI Workbench instances.

Together these controls separate durable eligibility from exercisable access. Durable bindings define who is eligible, IAM Conditions and context-aware levels define where and from what device the request must originate, principal access boundaries define which resources remain reachable, and Privileged Access Manager defines when elevated access becomes active and when it expires back to eligible.

Enforcing organization policies and resource hierarchy constraints means setting centralized guardrails that apply automatically to every child resource without per-project configuration. The Organization Policy Service (opens in a new tab) evaluates constraints against the Google Cloud resource hierarchy, which flows from the Organization resource at the apex to folders to projects to service resources. For data workloads, these guardrails control where data can reside, whether long-lived credentials can be created, and whether data can be exposed publicly.

3.1Resource Hierarchy and Policy Inheritance

The Google Cloud resource hierarchy is the ownership and inheritance chain that determines billing, visibility, and policy enforcement. An organization contains one or more folders, each folder contains one or more projects, and each project contains service resources such as BigQuery datasets and Cloud Storage buckets. Resource Manager manages this structure, while the Organization Policy Service evaluates constraints against it.

A constraint set on an organization, folder, or project applies to that node and to all descendant folders and projects. For example, if gcp.restrictTLSVersion denies TLS_VERSION_1 at the organization level, clients using TLS 1.0 are denied handshakes for every child folder and project. A child can override inherited policy by setting its own organization policy, and that child policy then flows to its own descendants through the same inheritance. Tags extend this model by annotating resources so IAM and organization policies can conditionally allow or deny actions based on tag attachment.

3.2Disabling Long-Lived Service Account Credentials

Service-account credential controls are organization policy constraints that remove unmanaged long-term keys and force deliberate least-privilege grants. By default, some Google Cloud services create default service accounts and automatically grant them the Editor role (roles/editor) on the project, so compromise of one workload can lead to control of the entire project. The boolean constraint constraints/iam.automaticIamGrantsForDefaultServiceAccounts set to False disables those automatic grants and requires explicit role assignment.

The constraint constraints/iam.disableServiceAccountKeyCreation set to True blocks creation of user-managed keys in affected projects, and constraints/iam.disableServiceAccountKeyUpload blocks upload of external public keys to service accounts. Together these constraints keep credentials inside Google-managed short-lived issuance instead of downloadable files that can leak from workstations, logs, or CI/CD pipelines. When keys cannot be eliminated, constraints/iam.serviceAccountKeyExpiryHours set to <= 2160 limits key lifetime to 2160 hours, or 90 days, so a stolen key stops working after the interval.

The preferred alternative is Workload Identity Federation, which lets external workloads authenticate without a service account key. The external system presents a short-lived OpenID Connect token, the token is exchanged dynamically through the Security Token Service for a temporary Google Cloud access token that expires in minutes, and IAM grants access directly to the federated principalSet:// identifier filtered by attribute conditions.

3.3Controlling Resource Location and Allowed Services

Resource location and service-usage controls are list constraints that enforce data residency and limit the attack surface across the hierarchy. The resource location constraint restricts which regions can store and process data, so a pipeline can only create BigQuery datasets, Cloud Storage buckets, and encryption keys in pre-approved regions. This control addresses regulatory residency needs and accounts for regionality differences between global, multi-regional, and regional services.

The resource service usage constraint restricts which Google Cloud APIs can be enabled, which prevents teams from activating unapproved services in production folders. The Restrict TLS Versions constraint gcp.restrictTLSVersion uses a deny list that denies explicit values and allows all others; a baseline denies TLS_VERSION_1 and TLS_VERSION_1.1, and restricting to TLS 1.3 only also denies TLS_VERSION_1.2. An error occurs if an allow list is attempted. For requirements without a predefined constraint, custom constraints enforce fine-grained rules such as permitting only approved Compute Engine machine families, limiting persistent disk sizes, or mandating specific firewall tag configurations.

The table summarizes the enterprise baseline for data workloads: location and service controls bound where code runs and what it can use, while credential and public-access controls bound how it authenticates and shares data. Because all four are set above individual projects, a new pipeline project inherits the same boundaries without per-project configuration.

3.4Preventing Public and External Access to Data

Public and external-access controls are organization policies that override resource-level sharing decisions and confine IAM grants to managed identities. The constraint constraints/storage.publicAccessPrevention overrides bucket-level IAM policies or access control lists that attempt to grant access to allUsers or allAuthenticatedUsers. Traffic from the public internet to the bucket is therefore denied even if a data engineer misconfigures a bucket policy during pipeline deployment.

Domain and member controls apply the same boundary to identity. The constraints constraints/iam.allowedPolicyMemberDomains for Domain restricted sharing and constraints/iam.managed.allowedPolicyMembers define the Cloud Identity or Google Workspace customer IDs whose principals can be added to IAM policies. When enforced on the organization, IAM denies bindings that would grant roles to external accounts outside organizational control for multifactor authentication and password policy. This prevents exfiltration paths where an external principal with weak authentication gains read access to analytical datasets, and it treats unmanaged consumer accounts as external for sharing decisions.

Key Facts

• BigQuery column policies use hierarchical tags where 3_Confidential includes 2_Private and 1_Sensitive, and protected columns require the Fine-Grained Reader role on the tag.
• Column policies on a BigQuery base table apply automatically to all authorized views, while each subset table must have the policy applied separately.
• Short-lived impersonation requires roles/iam.serviceAccountUser plus roles/iam.serviceAccountTokenCreator on the target service account, with Workload Identity Federation used to avoid long-lived keys.
• VPC Service Controls blocks data movement outside a service perimeter even when IAM allows it, and is limited to 10,000 projects inside perimeters and 100 perimeters including bridges per organization.
• Organization policy constraints/iam.disableServiceAccountKeyCreation set to True blocks user-managed keys, while constraints/iam.automaticIamGrantsForDefaultServiceAccounts set to False blocks automatic Editor grants to default service accounts.