Securing Credentials with Secret Manager

Encryption, replication, and audit logging in Secret Manager (opens in a new tab) work together to protect secret payloads, control where copies reside, and detect misuse. Cloud Key Management Service (Cloud KMS) (opens in a new tab) supplies customer-controlled keys for encryption at rest, the replication policy controls data residency, and Cloud Audit Logs (opens in a new tab) record use of secrets and keys. Loss of access to the protecting key makes secret data inaccessible, so key location, permissions, and lifecycle must be designed before the secret is created.

1.1Customer-Managed Encryption with Cloud KMS

Customer-managed encryption keys (CMEK) in Secret Manager replace Google default encryption when compliance requires customer control over data at rest. With Google default encryption, Google owns and manages the symmetric key encryption keys (KEKs) that wrap data under envelope encryption with no customer action. With CMEK, the customer creates a key ring and symmetric key in Cloud KMS and controls protection level, location, rotation schedule, access permissions, cryptographic boundaries, audit visibility, and lifecycle.

Access from Secret Manager to the key flows from Secret Manager through a Google-managed service agent to Cloud KMS. The service agent must hold the roles/cloudkms.cryptoKeyEncrypterDecrypter role on the key, and the key location must match the location of the secret it will protect. For multi-region deployments, regional colocation is required; for example, a us or eu selection can require a matching key in us-central1 or europe-west1, and provisioning fails if the matching key does not exist. CMEK must be selected when the protected resource is created and cannot be added to an existing unprotected resource.

1.2Replication Policies and Data Residency

A replication policy in Secret Manager defines where secret payload replicas are stored and therefore determines data residency compliance. An automatic replication policy lets Google replicate the secret within its managed footprint, while a user-managed replication policy pins replicas to specific regions selected by the customer. Choose automatic replication when operational simplicity matters more than residency pinning, and choose user-managed replication when regulation or policy requires secrets to stay in named regions.

Replication and CMEK location depend on each other because the Cloud KMS key ring, key, and secret replicas must share the same effective location. A user-managed replica in us-west1 requires a key ring and key in us-west1, and multi-region footprints map to matching Cloud KMS locations, such as a nam5-style footprint mapping to the Cloud KMS us multi-region and a eur3-style footprint mapping to europe. Because encryption is configured at creation time, replication locations and matching keys must be prepared before the secret is created.

1.3Audit Logging for Secret and Key Access

Audit logging for Secret Manager answers who accessed which secret and whether the protecting key remains usable. Cloud Audit Logs record secret administration and access events, while Cloud KMS Data Access audit logs record when a service makes Encrypt and Decrypt calls to Cloud KMS. Secret Manager does not call Cloud KMS on every data request; it maintains a poller that checks the key periodically, so a healthy key produces a steady pattern of about two entries every five minutes per protected resource.

To investigate key use, enable Data Access logging for the Cloud KMS API and query in Cloud Logging filtered to that key:

1. Enable logging: Turn on Data Access audit logs for the Cloud KMS API in the key project.
2. Filter to the key: Query with resource.type="cloudkms_cryptokey" plus resource.labels.key_ring_id, resource.labels.crypto_key_id, and resource.labels.location.
3. Interpret principals: Check method (Encrypt versus Decrypt) and principal_email to verify that only the expected service agent identity is using the key.

A break in the polling pattern, a surge of Decrypt calls, use from an unexpected principal_email, or FAILED_PRECONDITION errors with cloudkms.cryptoKeyVersions.useToEncrypt permission denied all indicate revoked access, a disabled key, or potentially anomalous use.

1.4Organization Policy Enforcement and Key Lifecycle

Organization policy constraints enforce CMEK and key governance across an organization, folder, or project. When constraints/gcp.restrictNonCmekServices includes the protected service in its deny list, creation of new unprotected resources is refused and a valid Cloud KMS key must be selected. When constraints/gcp.restrictCmekCryptoKeyProjects is configured, only CryptoKeys from allowed projects, folders, or organizations can protect new resources.

Removing key access follows an ordered lifecycle from reversible to irreversible loss:

1. Revoke the role: Remove roles/cloudkms.cryptoKeyEncrypterDecrypter from the service agent to block use immediately.
2. Disable the key version: Temporarily disable the key version to make data inaccessible but recoverable.
3. Destroy the key version: Permanently destroy the key version to make loss irreversible.

Revoke the role first because permission changes are consistent within seconds, which lets the impact be observed before a destructive step. Re-enabling a key can take minutes to up to 3 hours, and full service recovery can take up to two weeks.

Access governance and network security controls for Google Secret Manager (opens in a new tab) combine identity permissions with network boundaries to ensure only authorized workloads can read secrets. Identity and Access Management (IAM) (opens in a new tab) decides which service account can access which secret, OAuth scopes decide whether the underlying Compute Engine or GKE host can call the API, and VPC Service Controls (opens in a new tab) decides whether that API call can cross a security perimeter. Together they enforce least privilege and prevent exfiltration across enterprise workloads.

2.1Granular IAM for Secret Access

Granular IAM for Secret Manager means granting access per secret to a specific service account rather than granting broad project access. A binding attaches the role roles/secretmanager.secretAccessor on a single google_secret_manager_secret resource to one member, such as serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com, so only that workload identity can read that secret version. The Cloud Run pattern shows the same principle: the service references secrets by secret name and version through value_source.secret_key_ref, but the reference only succeeds when the runtime service account holds the accessor binding on each secret.

This model depends on blocking overly broad defaults and keeping duties separate. Best practices include blocking automatic IAM grants to default service accounts, creating Google Kubernetes Engine (GKE) (opens in a new tab) clusters with limited service-account privileges, and granting required permissions to the App Engine service account when workloads run on App Engine. When access must change for movers, leavers, or rotated workloads, the per-secret binding is removed or replaced, so no other secret or workload is affected.

2.2Authentication Scopes and Workload Prerequisites

Authentication scopes are a second authorization gate for workloads on Compute Engine and GKE that use Secret Manager. The instance, instance group, or node pool must be provisioned with the cloud-platform OAuth scope, https://www.googleapis.com/auth/cloud-platform, or API calls fail with Request had insufficient authentication scopes even when IAM is correct. Traffic therefore flows from application code through the client library to the Secret Manager API only after both checks pass: the attached service account holds IAM permission and the host holds the scope.

Enabling access follows an ordered dependency:

1. Enable the Secret Manager API in the project and establish authentication for API requests.
2. Grant secret-level IAM to the workload service account that needs to read the secret.
3. Provision the host scope with cloud-platform at creation time, for example with gcloud compute instances create, or update an existing host with gcloud compute instances set-service-account to attach the service account and scope.

App Engine follows the same dependency without host scopes: the App Engine service identity must be granted the required Secret Manager permissions before the application can access secrets.

2.3VPC Service Controls Perimeters Against Exfiltration

VPC Service Controls is the network-layer boundary that prevents data exfiltration of Secret Manager API calls across projects and networks. A service perimeter encloses selected projects and explicitly defines allowed services for the perimeter, so a request from an authorized service account inside the perimeter to Secret Manager inside the same perimeter is permitted, while a copy or read directed to a project or service outside the perimeter is denied. Restricting APIs within VPC Service Controls service perimeters and enabling the service scope restriction in Access Context Manager access policies tighten this boundary so only intended Google Cloud APIs are usable inside the perimeter.

Access Context Manager policies add context to the perimeter decision by evaluating attributes such as network origin and identity before allowing cross-boundary access. Defining service perimeters, defining allowed services for each perimeter, and retrieving VPC peering configurations ensure there is no unintended bridge that lets secret data flow from a protected project to an external resource. Changes to perimeters are high-risk, so monitoring for modifications to VPC Service Controls perimeters is treated as a security detection case.

2.4Private Connectivity and Egress Guardrails

Private connectivity and egress guardrails keep secret traffic on private paths and block internet escape routes for enterprise workloads. Private Google Access for VPC subnets lets instances without external IP addresses reach Google APIs including Secret Manager, so the request travels from the VPC subnet to Google APIs without traversing the public internet. Complementary controls include blocking external IP address access on Compute Engine VM instances, restricting default network creation, enabling private GKE clusters, and using Cloud VPN or Cloud Interconnect for hybrid connections instead of public endpoints.

Egress and ingress rules provide defense in depth around the perimeter:

• Enforce a deny-all egress firewall rule and restrict Cloud Run egress settings by organization policy.
• Restrict outbound traffic and limit inbound access to SSH port 22 and RDP port 3389.
• Configure VPC firewall rules, subnets, and VPN gateways to implement traffic flow controls using logical networking.
• Disable external IPv6 usage and block generic access to sensitive ports unless explicitly required.

Lifecycle management in Google Cloud Secret Manager (opens in a new tab) is the continuous control of a secret from creation through version replacement, scheduled rotation, and eventual expiration. A secret is the named container, such as projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_ID, and each secret version is an immutable copy of the credential value stored inside it. Continuous credential hygiene depends on three linked controls: managing version states, running rotation on a schedule, and enforcing expiration so stale credentials cannot live forever.

3.1Secret Versions and Lifecycle States

A secret version in Secret Manager is a single immutable value with its own resource name, creation time, and state. Versions appear in the ENABLED state after creation or rotation, for example projects/PROJECT_NUMBER/locations/LOCATION/secrets/SECRET_ID/versions/1 with "state": "ENABLED", and in the DISABLED state after a disable operation. A disabled version cannot be used by future access requests until it is re-enabled, while other versions of the same secret are unaffected. This state model allows safe rotation because the workflow depends on the new version becoming ENABLED while the prior version is retained in DISABLED form for rollback, rather than being deleted immediately.

Secret values can be in any format but must not be larger than 64 KiB, and the first value uploaded at creation automatically becomes the first version. For specialized use, a regional secret can be created with a fixed secretType such as CLOUD_SQL_DB_CREDENTIALS, which tells Secret Manager that rotation must also update the linked Cloud SQL (opens in a new tab) instance. The secret type cannot be added or changed on an existing secret, so the type decision must be made when the secret is created.

3.2Rotation Schedules and the rotateSecret Operation

A rotation schedule in Secret Manager is a start time plus a recurring period that defines when the secret value must be replaced. In the console the schedule is configured with a Starting on date and time, after which the automatic rotation timer runs and triggers replacement at each interval. For regional secrets the manual trigger is the rotateSecret method on the regional endpoint secretmanager.LOCATION.rep.googleapis.com, called as POST /v1/projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_ID:rotateSecret.

When rotation runs for a Cloud SQL credential secret, Secret Manager first creates a new secret version and then attempts to update the credentials in the Cloud SQL instance from that new version. If that update succeeds, the Rotation status changes to Enabled and the scheduled automatic rotation timer begins for the next interval. If the update fails, rotation does not reach the Enabled status, which preserves the signal that the live database password and the stored version are not yet in sync. An event-driven design extends this flow by having the rotation schedule notify downstream automation, where a Cloud Pub/Sub (Pub/Sub) notification carries the rotation event to a Cloud Functions handler that generates the new credential, adds it as a new version, and propagates it to the target system before the old version is disabled.

3.3Expiration Policies for Time-Bound Secrets

An expiration policy (opens in a new tab) in Secret Manager is a time limit after which the entire secret is no longer usable, which enforces hygiene even when rotation is missed. Expiration can be set in the console by selecting Set expiration date and entering a month, day, year, and time, or programmatically at creation with one of two equivalent controls. Both controls cause automatic handling at the deadline, so applications that depend on the secret must refresh or replace it before expiration to avoid outages.

The gcloud secrets create commands reflect this choice directly through --expire-time "TIMESTAMP" versus --ttl "DURATION", both used with a replication policy such as --replication-policy "automatic". Client libraries express the same distinction, with TTL built as a duration object attached to the secret at creation. Because expiration applies to the secret container and rotation creates new versions inside it, the two controls compose: rotation maintains freshness during the lifetime, while expiration guarantees termination at the end of it.

Key Facts

• Secret Manager CMEK requires granting the Google-managed service agent the roles/cloudkms.cryptoKeyEncrypterDecrypter role, and the Cloud KMS key location must match the secret location.
• A healthy CMEK-protected resource produces about two Cloud KMS Data Access entries every five minutes from the key poller.
• Compute Engine and GKE workloads need both secret-level IAM and the cloud-platform OAuth scope, or Secret Manager calls fail with insufficient authentication scopes.
• Secret versions are immutable, limited to 64 KiB, and move between ENABLED and DISABLED states to support safe rotation and rollback.
• Secret expiration uses expire-time for an absolute RFC 3339 deadline and time-to-live (TTL) for a relative lifetime from creation.