Schedule database exports

Operational monitoring and error handling for scheduled exports is the practice of tracking each export job through its lifecycle and catching failures before a recovery point is missed. A scheduled pipeline typically flows from Cloud Scheduler (opens in a new tab) to Pub/Sub (opens in a new tab) to an export function or API, then to a Google Cloud Storage (GCS) (opens in a new tab) bucket. Because export services often have no built-in daily scheduler, reliable operation depends on combining external scheduling with monitoring, retry logic, and audit trails.

1.1Monitoring Export Job Status and Progress

Export job monitoring centers on an export job, a single asynchronous operation that exports a specific time range and is tracked by a unique dataExportId. Each export job moves through lifecycle states in order: first IN_QUEUE while waiting for capacity, then PROCESSING while data is transferred, then either FINISHED_SUCCESS or FINISHED_FAILURE. Operators monitor this lifecycle by calling GetDataExport for one job ID or ListDataExport with a filter for type, time, or status.

For timer-driven database exports, monitoring must follow the full chain from scheduler to storage. The Cloud Scheduler result confirms only that a message was sent to the Pub/Sub topic, so success there does not prove the export succeeded. Function logs in Logs Explorer report whether the export started or returned an error, the long-running operations list shows running and completed operations, and timestamp-organized files in the GCS bucket confirm completion. A practical verification sequence is:

1. Check scheduler execution: Run the Cloud Scheduler job manually, click Refresh, and confirm the result shows Success and Last run shows the current time.
2. Check function logs: Open Logs Explorer for the export function and look for successful initiation versus error entries.
3. Check operation progress: List long-running export operations to see if the operation is still running, succeeded, or failed.
4. Check destination data: Open the GCS bucket and confirm new timestamped files were written for the expected window.

1.2Automated Retries and Fault Tolerance

Automated retry logic lets a managed export pipeline absorb temporary problems without operator action. The Data Export API uses exponential backoff that automatically retries a job several times for transient issues such as temporary network problems. Its job management system also uses queuing and prioritization to keep the platform stable, prevent one tenant from monopolizing resources, and support point-in-time exports of up to 100 TB.

Streaming pipelines add parallel fault tolerance around the same idea. Log data flows from Cloud Logging to a Pub/Sub topic and subscription, then through a primary Dataflow (opens in a new tab) pipeline to the destination, with a secondary pipeline that replays messages if delivery fails. Dataflow autoscales workers as volume and backlog change, automatically resends data after transient server or network issues, and routes undeliverable messages to dead-letter topics to avoid loss. If all retries are exhausted, the system marks the job FINISHED_FAILURE with a detailed error message, while an immutable audit trail records creation, start, success, or failure with user, timestamp, and parameters.

1.3Diagnosing Failures and Protecting Recovery Objectives

Failure diagnosis for scheduled exports starts from detailed API errors that pair a canonical code with a specific cause and fix. Common failure patterns for export jobs include:

• INVALID_ARGUMENT with INVALID_REQUEST means request parameters are invalid and must be corrected before retrying.
• PERMISSION_DENIED with INSUFFICIENT_PERMISSIONS for the customer-managed key means the key cannot be validated and must be fixed.
• NOT_FOUND with BUCKET_NOT_FOUND means the destination bucket does not exist, while REQUEST_NOT_FOUND means the dataExportId does not exist.
• FAILED_PRECONDITION with BUCKET_INVALID_REGION means the bucket region does not match the tenant region, INSUFFICIENT_PERMISSIONS means the service account lacks object create, get, and bucket get permissions, and INVALID_CANCELLATION means cancellation was attempted after the job left IN_QUEUE.
• RESOURCE_EXHAUSTED with CONCURRENT_REQUEST_LIMIT_EXCEEDED means too many concurrent requests are running, while REQUEST_SIZE_LIMIT_EXCEEDED means estimated volume exceeds the maximum per request.
• INTERNAL with INTERNAL_ERROR means a platform-side error occurred and the request can be tried again.

A separate scheduling failure is easy to confuse with an export failure. A NOT_FOUND status from Cloud Scheduler with log entry type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished often indicates missing permissions on the Cloud Scheduler or Dataflow service accounts, not a missing export job. Because recurring exports depend on customer-built automation, any missed schedule or permanent failure affects the recovery point, so operators must also check the dataRbacFiltered flag on FINISHED_SUCCESS jobs to see if filtering reduced exported volume.

Orchestration and triggering for scheduled database exports is the pattern of using Cloud Scheduler (opens in a new tab) to start work on a timetable, Pub/Sub to deliver that trigger, and Cloud Run functions (formerly Cloud Functions) to run the export. In this pattern, data flows from Cloud Scheduler to a Pub/Sub topic to a function, and then from the database to a Cloud Storage bucket. The same building blocks are recommended for Firestore (opens in a new tab) in Datastore mode exports and suggested for Cloud SQL (opens in a new tab), which does not provide built-in automated exports.

2.1Scheduled Export Architecture

The scheduled export architecture connects four systems in a fixed order. First, a Cloud Scheduler job publishes a message on a schedule. Then Pub/Sub delivers that message to a function subscribed to the topic. Then the function calls the database export API, and the managed export service writes output files to Cloud Storage. The workflow depends on two prerequisites: the Google Cloud project must have billing enabled, and a Cloud Storage destination bucket must exist near the database location. A Requester Pays bucket cannot be used for export operations.

The end-to-end flow preserves efficiency because no server runs between exports:

1. Schedule fires: Cloud Scheduler publishes the configured payload to the Pub/Sub topic at the cron time.
2. Trigger delivers: Pub/Sub delivers the message to the datastore_export function.
3. Export starts: The function decodes the payload and calls the export operation with the bucket and optional filters.
4. Data lands: The database service writes timestamp-organized output files to the Cloud Storage bucket.

2.2Configuring Cloud Scheduler with Cron Schedules

Cloud Scheduler is the time-based trigger that starts each export without manual action. A Cloud Scheduler job defines a name such as scheduledDatastoreExport, a frequency in unix-cron format, and a timezone such as UTC. The job target is Pub/Sub, set to the same topic the function listens to, such as startDatastoreExport. Cloud Scheduler encodes the payload in base-64 by default, and the function decodes it before starting the export.

The payload controls what gets exported. The payload is a JSON object that must include a bucket value and can include entity filters:

• bucket: destination such as gs://bucket_name, required for every export.
• kinds: optional list such as ["User", "Task"] to export only those entity kinds.
• namespaceIds: optional list such as ["", "Testers"] to limit namespaces, where an empty string means the default namespace.

2.3Triggering Exports with Cloud Functions and Pub/Sub

Cloud Run functions provide the export logic that runs only when the scheduled message arrives. The function is created with a Cloud Pub/Sub trigger on the scheduler topic, inline Python source code, and an entry point named datastore_export. The sample runtime uses Python with the dependency google-cloud-datastore==2.23.0 and a client for datastore_admin_v1.DatastoreAdminClient. When invoked, the function reads the bucket, builds an entity filter from kinds and namespaceIds when present, and sends an ExportEntitiesRequest for the project.

The Pub/Sub trigger decouples timing from execution. Cloud Scheduler does not call the database directly, and the function does not keep a schedule itself. This separation means the schedule can change in Cloud Scheduler without redeploying code, and the same topic can trigger retries or manual runs. For Cloud SQL, the same idea applies: build a custom system with Cloud Scheduler, Pub/Sub, and Cloud Run functions because automated exports are not built into the service.

2.4Managing Service-to-Service Authentication with IAM

Service-to-service authentication for scheduled exports relies on Identity and Access Management (IAM) roles granted to the function identity. The function uses a default service account to authorize both the export call and the write to Cloud Storage. The exact service account depends on the function generation, and second-generation functions need an extra invoker role.

The service account must also hold database and storage permissions. The export permission is Cloud Datastore Import Export Admin (roles/datastore.importExportAdmin), which also covers Firestore permissions. The storage permission is granted on the destination bucket as Storage Object User (roles/storage.objectUser) or Storage Admin for broader management. Without these bindings, exports fail with permission errors even when schedule and trigger succeed.

2.5Verifying Scheduled Exports

Verification of a scheduled export checks each stage separately because success at one stage does not prove success at the next. Testing starts with Run now on the Cloud Scheduler job, then Refresh until the result column shows Success and Last run shows the current time. That result proves delivery to Pub/Sub, not completion of the database export. The function logs in Logs Explorer show whether the export started or returned an error, and long-running operation commands plus the Cloud Storage browser show progress and final output files.

Storage configuration and data security control where scheduled database exports can be written and who can write them. A scheduled export writes database files to a Cloud Storage (opens in a new tab) bucket using a Google-managed service account, and that bucket must meet location and type rules while the identity holds the correct Identity and Access Management (IAM) permissions. Encryption with customer-managed keys adds a second authorization boundary between the export identity, the bucket, and the key.

3.1Cloud Storage Destination Bucket Requirements

Cloud Storage destination bucket requirements define the bucket that a scheduled export job is allowed to use. The project must have billing enabled, and the bucket should be created in a location near the database location to keep transfer efficient and consistent. For Firestore exports, the bucket cannot be a Requester Pays bucket or a Rapid bucket.

• Billing must be enabled on the Google Cloud project before export and import operations can run.
• The export bucket should be in a location near the Firestore database location.
• The export bucket cannot be a Requester Pays bucket or a Rapid bucket.
• The export path can include a folder prefix such as gs://BUCKET_NAME/my-exports-folder/export-name, or the service creates a timestamp-based prefix.

Repeated scheduled exports create new objects on every run, so the bucket fills over time unless old export folders are removed or transitioned. That growth is why bucket choice and later lifecycle controls are part of export planning rather than an afterthought.

3.2IAM Permissions for Export Identities

IAM permissions for export identities separate what the human operator needs from what the automated service account needs. The operator who starts or schedules an export needs database and storage roles, while the database service account that writes the files needs write access to the destination bucket. If the bucket is missing or the service account lacks the bucket role, the export fails with an HTTP Error 403 for missing bucket permissions.

The three export identities use different service accounts but follow the same pattern where the writer needs object creation rights on the destination bucket:

The Firestore service agent name can be viewed in the console next to Import/Export jobs run as, and cross-project access is granted with gsutil iam ch or through bucket IAM policy binding in the console. Permission changes can take a few minutes to propagate, so a newly granted role may not work immediately.

3.3Customer-Managed Encryption Keys for Exports

Customer-managed encryption keys (CMEK) provide encryption for export artifacts using a key the customer controls in Cloud Key Management Service (Cloud KMS). A Looker (Google Cloud core) (opens in a new tab) scheduled export requires a CMEK key even when the instance itself does not use CMEK, and the key must be in a different region from the Looker instance or be a global key. The export identity must be able to both encrypt with the key version and write the encrypted objects, which links key permissions to bucket permissions.

The Looker instance service account needs the Cloud KMS CryptoKey Encrypter/Decrypter role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the CMEK key, or a custom role with cloudkms.cryptoKeyVersions.useToEncrypt and cloudkms.cryptoKeyVersions.useToDecrypt. A one-time Looker export uses a separate export-specific CMEK that is different from the CMEK used to encrypt instance data, and the service account also needs storage.objects.create plus bucket IAM policy permissions to complete the console flow. Without both the bucket role and the key role, the scheduled job cannot create the encrypted export artifact in Cloud Storage.

3.4Cross-Project Exports and Service Agent Migration

Cross-project exports move data from a database in one project to a Cloud Storage bucket in another project. A Firestore service agent can access buckets in the same project as the Firestore database by default, but traffic from the export job to a bucket in another project is denied unless the destination bucket explicitly grants access to that source-project service agent. The migration tool lists recently used buckets that lack read and write permission for the Firestore service agent so missing cross-project grants can be found before migration.

Older Firestore databases may still authorize exports with the App Engine default service account (PROJECT_ID@appspot.gserviceaccount.com) instead of the Firestore-specific service agent. The Firestore service agent is preferred because it is specific to Firestore, while the App Engine account is shared by more than one service. If Virtual Private Cloud (VPC) Service Controls are used, the database must use the Firestore service agent because VPC Service Controls are not compatible with the App Engine service account.

3.5Bucket Lifecycle Management and Protective Controls

Bucket lifecycle management and protective controls limit cost, exposure, and accidental loss for buckets that accumulate scheduled exports. Lifecycle and retention rules act on objects after the export job writes them to Cloud Storage, while access controls act before the write by deciding which identities can reach the bucket. Together they keep daily exports available for recovery without leaving every historical export exposed forever. A one-time export (opens in a new tab) follows the same storage protections once its objects land in the bucket.

• Block public access to Cloud Storage buckets that hold database exports.
• Use uniform bucket-level access for consistent IAM control.
• Enable lifecycle management, including deletion rules and SetStorageClass transitions, to clean up or cool down old scheduled exports.
• Enforce retention policies with Bucket Lock and review temporary holds on active objects where regulatory retention applies.
• Enforce versioning, enforce bucket owners, enable logging of key Cloud Storage activity, and use log buckets for audit trails.

Key Facts

• Scheduled database exports flow in order from Cloud Scheduler to Pub/Sub to a Cloud Run function, which calls the database export API that writes timestamp-organized files to Cloud Storage.
• A Cloud Scheduler result of Success only proves a message was delivered to Pub/Sub; export completion must be confirmed in function logs, long-running operations, and destination bucket files.
• Export jobs move through lifecycle states in order: IN_QUEUE, then PROCESSING, then FINISHED_SUCCESS or FINISHED_FAILURE.
• Firestore, Cloud SQL, and Looker export identities must hold object-create roles on the destination Cloud Storage bucket, and a cross-project bucket requires an explicit grant to the source project's service agent.
• Looker scheduled exports require a customer-managed encryption key in a different region or a global key, plus Cloud KMS CryptoKey Encrypter/Decrypter on the service account.