Leverage database technologies to support generative AI and LLM use cases

Optimize and Secure AI-Integrated Database Workloads is the practice of sizing database compute for expensive vector operations while locking down how embeddings and source data are accessed during retrieval. In Google Cloud, AlloyDB for PostgreSQL (opens in a new tab) stores pgvector embeddings for transactional retrieval, Vertex AI Vector Search (opens in a new tab) serves low-latency large-scale similarity search, and Gemini Enterprise Agent Platform (opens in a new tab) runs embedding models and large language model (LLM) inference. Scaling, monitoring, identity controls, and network perimeters work together so retrieval stays fast without exposing sensitive data to the model or to users.

1.1Scaling Resources for Vector Operations

Scaling for vector workloads means matching compute and availability to query load and to the cost of similarity search. AlloyDB for PostgreSQL clusters provide high availability by default with an active node and a standby node in two different zones in a region, with automatic failover if the active node fails. A basic primary instance costs less because it has no standby node, but it is not robust against zone outages and has longer downtime during maintenance, so choose it only when high availability is not required.

Vertex AI Vector Search handles query scaling differently from AlloyDB for PostgreSQL. The Vector Search index endpoint supports autoscaling that adds nodes automatically as queries per second increase, up to a maximum that you specify. For predictable AlloyDB for PostgreSQL CPU and memory needs, committed use discounts reduce cost, while batching high queries-per-second Vector Search requests reduces node usage and cost. To absorb transient spikes before they reach the database, configure flow control in Pub/Sub publisher settings and adjust retry-request variables for failed publishes.

1.2Monitoring Retrieval and Similarity Search Health

Monitoring for AI-integrated databases tracks both the retrieval path and the quality of grounded answers. The serving subsystem sends a user request from the frontend to an embedding model, performs semantic search for that embedding in the AlloyDB for PostgreSQL vector store, combines the original request with retrieved raw data into a contextualized prompt, and sends that prompt to the LLM inference stack on Agent Platform. The application stores request-response activity in Cloud Logging (opens in a new tab) for viewing in Cloud Monitoring, loads responses into BigQuery for offline analytics, screens responses with responsible AI filters, and then returns the screened response to the frontend.

The quality evaluation subsystem provides specialized feedback on retrieval relevance:

1. Trigger: Pub/Sub triggers a Cloud Run function for evaluation.
2. Configure: The function starts by using configuration data stored in an AlloyDB for PostgreSQL database.
3. Collect prompts: The function pulls evaluation prompts previously loaded by the data ingestion subsystem from AlloyDB for PostgreSQL.
4. Score: The function uses the prompts to assess serving responses and produces evaluation scores for metrics like factual accuracy and relevance.
5. Analyze: The function loads scores plus the evaluated prompts and responses to BigQuery for future analysis.

Audit visibility comes from Admin Activity audit logs, which are enabled by default for services in the architecture and record configuration changes, and Data Access audit logs, which are enabled by default for BigQuery and can be enabled for other services to track reads of configuration, metadata, and user-provided data.

1.3Governing Retrieval Access with Identity Controls

Identity control for AI-integrated databases applies role-based access control (RBAC) with least privilege so each user, group, or service account has only no-access, read-only, or write access needed for its role. Identity and Access Management (IAM) (opens in a new tab) is the primary control plane, and access is refined with IAM conditions on data, models, and configurations, principal access boundary policies to limit which resources a principal can reach, and Privileged Access Manager for just-in-time temporary elevation that is later auditable. For Cloud Storage, IAM at bucket and project levels is recommended in most cases over access control lists (ACLs), while BigQuery adds dataset and resource-level IAM roles plus row-level and column-level security for sensitive fields.

Connection-level authorization protects the retrieval path into AlloyDB for PostgreSQL. By default an AlloyDB for PostgreSQL instance accepts only connections that use SSL, and the AlloyDB Auth Proxy connector adds IAM-based connection authorization with a Transport Layer Security (TLS) 1.3 connection using a 256-bit Advanced Encryption Standard (AES) cipher to verify client and server identities and encrypt traffic. Java, Python, and Go clients use the corresponding Language Connector instead of the Auth Proxy connector. Agent Platform access combines predefined or custom IAM roles, and private prediction endpoints rely on the target service IAM for model access even though IAM is not directly applied to the private endpoint itself.

1.4Protecting Data with Perimeters, Encryption, and Residency

Perimeter and encryption controls keep embeddings and source documents inside a trusted boundary during ingestion and retrieval. Virtual Private Cloud (VPC) Service Controls (VPC-SC) creates a service perimeter around AI and machine learning resources and sensitive data to mitigate exfiltration from AlloyDB for PostgreSQL and to protect private prediction endpoints, with ingress and egress rules controlling data flow across the perimeter. Private Service Connect interfaces and private IP addresses let custom training and prediction traffic communicate with authorized Google Cloud services without public exposure, while firewall policies deny all traffic by default and explicitly allow only required traffic.

Encryption and data handling differ by product but share common defaults. AlloyDB for PostgreSQL, Cloud Storage, and Pub/Sub encrypt stored data by default with Google-owned and Google-managed keys, and each supports customer-managed encryption keys (CMEKs) in Cloud Key Management Service (KMS) when you must control the keys. Sensitive data loaded through Cloud Storage can be discovered, classified, and de-identified with Sensitive Data Protection, and data residency is met by storing or replicating AlloyDB for PostgreSQL, BigQuery, Cloud Storage, Pub/Sub, and logging data only in specified regions.

The available platform-level controls vary across AI services:

Retrieval-Augmented Generation (RAG) is the practice of grounding a large language model (LLM) with real-time data retrieved from an operational dataset before the model generates a response. Grounding constrains the model to current, domain-specific facts instead of only its training knowledge, which improves accuracy for questions about private or fast-changing data. On Google Cloud, a RAG architecture connects a database that holds embeddings and source content to Vertex AI (opens in a new tab) models that create embeddings and generate text.

2.1Choosing a Grounding and Vector Search Path

A grounding path is the search layer that supplies context to the LLM prompt, and Google Cloud offers different paths for different starting points. Embeddings stored in a vector database are the optimal format for similarity search, but grounding can also use a standard query against an existing database such as Cloud SQL or Firestore and place the query result in the prompt. Gemini models can also decide on their own whether grounding with Google Search is needed for up-to-date internet information.

Choose Vertex AI Search when the requirement is a fully managed, optimized solution over most data sources. Choose RAG Engine when custom orchestration is needed, Vector Search when the workload demands large-scale low-latency vector serving, and in-database grounding when the grounding data already lives in an operational SQL database.

2.2Integrating Cloud SQL with Vertex AI for In-Database AI

Cloud SQL (opens in a new tab) integration with Vertex AI lets applications apply LLMs hosted in Vertex AI directly to rows in Cloud SQL. The integration is supported on Cloud SQL for PostgreSQL version 12 and later, Cloud SQL for MySQL (opens in a new tab) version 8.0.36 and later, and Cloud SQL for SQL Server (opens in a new tab) 2025 in Enterprise and Enterprise Plus editions. For SQL Server the integration is in Preview and supports model endpoints from Vertex AI, Hugging Face, and OpenAI. The integration extends SQL so model calls run inside a transaction and return enriched data without moving rows to an external service.

The integration adds two SQL-level capabilities for PostgreSQL and MySQL, and an embedding capability for SQL Server. Invoke predictions calls a model from SQL to produce a real-time prediction for each row. Generate embeddings sends text to an embedding model that translates the prompt into numerical vectors, and those vectors become inputs to vector functions that compare and sort text by relative semantic distance. Because prediction and embedding happen in the SQL layer, operational workflows gain real-time enrichment and semantic search directly where the data lives.

2.3Data Ingestion and Embedding Storage with AlloyDB

The data ingestion subsystem is the pipeline that prepares domain data for retrieval by chunking source content, creating embeddings, and storing them for search. A Cloud Run function receives new data, formats it as required, and divides it into chunks suitable for embedding. The function then calls an embedding model on Gemini Enterprise Agent Platform to create vectorized embeddings of each chunk. The function stores those embeddings in an AlloyDB for PostgreSQL (opens in a new tab) database with the pgvector extension enabled, so the serving subsystem can later retrieve relevant domain data by semantic similarity.

2.4Serving Subsystem for Retrieval and Generation

The serving subsystem is the request-response flow that turns a user question into a contextualized prompt and then into a screened answer. Data flows from the user frontend to the embedding model, to the AlloyDB vector store, to the LLM inference stack on Agent Platform, and back to the user.

1. User request: Users submit natural-language requests through a frontend such as a chatbot or mobile app to the generative AI application.
2. Query embedding: The application converts the request to embeddings using the same model and parameters used during ingestion.
3. Semantic retrieval: The application performs semantic search for the query embedding in the AlloyDB vector store, which finds matches by intent rather than exact text, then combines the original request with the retrieved raw data to create a contextualized prompt.
4. LLM generation: The application sends the contextualized prompt to an LLM inference stack on Agent Platform, where a foundation LLM or custom LLM generates a response constrained to the provided context.
5. Logging and analytics: The application can store request-response logs in Cloud Logging for monitoring with Cloud Monitoring and load responses to BigQuery for offline analytics.
6. Responsible filtering and delivery: The application screens the response with responsible AI filters and then sends the screened response to the user through the frontend.

2.5Quality Evaluation and Scalable Vector Options

The quality evaluation subsystem is the loop that measures how well serving responses meet factual accuracy and relevance goals. Pub/Sub triggers a Cloud Run function that starts from configuration data stored in AlloyDB for PostgreSQL. The function pulls evaluation prompts previously loaded by the ingestion subsystem, uses those prompts to score responses generated by the serving subsystem, and loads the evaluation scores with the evaluated prompts and responses to BigQuery for future analysis.

Vertex AI Feature Store provides an additional scalable vector database option for RAG Engine when large data volumes and low latency are required. Feature Store is a managed service that manages feature data in a BigQuery table or view and enables low-latency online serving. A FeatureOnlineStore instance created with optimized online serving supports vector similarity search to return approximate nearest neighbors, which are the semantically similar entities used as grounding context for RAG applications.

Vector storage and indexing in AlloyDB lets you keep operational data and high-dimensional embeddings in one PostgreSQL-compatible database and run fast semantic search without a separate vector database. AlloyDB AI (opens in a new tab) provides extensions to store embeddings in a vector column, build approximate nearest-neighbor indexes, and accelerate filtered search for retrieval-augmented generation (RAG), semantic search, and recommendation engines. Choosing an index means trading query latency, memory use, recall accuracy, and build time.

3.1Storing High-Dimensional Vectors in AlloyDB

Vector storage in AlloyDB uses the customized vector (opens in a new tab) extension, which is AlloyDB's version of the open-source pgvector extension. The extension adds a vector column type that holds generated embeddings, so text, images, or other content converted to numbers by an embedding model can sit in the same table as the business data. Keeping embeddings and operational rows together lets a query combine semantic similarity with SQL filtering, keyword search, and reranking in one engine. Batch embedding functions can then generate embeddings for large tables from a single, secure source.

3.2Index Options for Similarity Search

Vector indexes in AlloyDB avoid scanning every embedding by organizing vectors for approximate nearest-neighbor search. The vector extension supports IVF indexes with scalar quantization for smaller storage, plus the stock IVFFlat and Hierarchical Navigable Small World (HNSW) indexes from pgvector. The separate alloydb_scann extension, formerly postgres_ann, implements an index powered by the Scalable Nearest Neighbors (ScaNN) algorithm and works with PostgreSQL 14, 15, 16, and 17. The google_columnar_engine extension can load a ScaNN index into the columnar engine to further accelerate vector search.

3.3Comparing HNSW and ScaNN Performance

HNSW and ScaNN both speed semantic search, but they differ in speed, memory, and scale. HNSW is usually well suited for higher-dimensional data that fits largely within the in-memory cache and scales well to 10-20 million vectors. ScaNN is Google's optimized index for large-scale production search, including filtered search where vector similarity is combined with SQL conditions.

Both index types can be accelerated by the AlloyDB columnar engine, which is especially helpful for filtered k-nearest-neighbor search.

3.4Tuning and Accelerating Vector Search

Tuning a vector index balances queries per second (QPS) against recall, which is how often the true nearest neighbors are returned. You tune index parameters to favor faster responses with slightly lower recall or slower, more thorough search with higher recall. First data flows from the embedding model into the vector column, then the index organizes those vectors, and only after the index is built can queries use it for fast similarity ranking. Google recommends creating a vector index for similarity queries, using the columnar engine for filtered search, and using Auto mode to automatically configure and maintain ScaNN indexes as data grows.

Key Facts

• Retrieval-Augmented Generation (RAG) grounds a large language model with real-time data retrieved from an operational dataset, and the serving flow runs from frontend to embedding model to AlloyDB vector store to LLM inference with responsible AI filtering before delivery.
• Cloud SQL for PostgreSQL version 12 and later and Cloud SQL for MySQL version 8.0.36 and later can invoke Vertex AI predictions and generate embeddings directly inside SQL transactions.
• AlloyDB for PostgreSQL stores embeddings in a vector column using its customized vector extension and supports IVF, IVFFlat, HNSW, and ScaNN indexes for approximate nearest-neighbor search.
• ScaNN uses 3x less memory than HNSW and provides up to 10x faster index creation, up to 4x faster queries, and up to 10x faster filtered search than HNSW.
• VPC Service Controls creates a service perimeter with ingress and egress rules to prevent exfiltration of embeddings and source data, while AlloyDB for PostgreSQL accepts only SSL connections by default and supports IAM-based authorization through the AlloyDB Auth Proxy with TLS 1.3.