Prepare and test your skills
Prepare and test your skills
Foundation models (opens in a new tab) are large models trained on vast amounts of data that serve as a base for generating new content. They learn patterns from training data and use statistical modeling to predict likely responses to prompts. Traditional AI (opens in a new tab) mainly classifies or predicts from historical data, while generative AI (opens in a new tab) uses foundation models to create new text, code, and images. Google offers these models as managed APIs that teams can discover and use through Vertex AI.
A modality describes the kind of data a foundation model can accept, process, or generate. Foundation models are often grouped by text, code, image, and multimodal capabilities, but a single model can span more than one modality. This grouping helps teams match a model to the input and output their task needs.
| Modality | Core characteristic | Example tasks or access |
|---|---|---|
| Text | Large language models (LLMs) use embeddings to understand context and meaning and generate coherent responses. | Gemini and PaLM 2 models can be prompted in Vertex AI Studio; Hugging Face text models can deploy to Vertex AI or Google Kubernetes Engine (GKE). |
| Code | Natural language prompts create code assets and explanations. | Generating API specs, documentation, functions, command-line commands, Terraform scripts, tests, and code explanations. |
| Image | Image generation learns patterns through diffusion techniques, and embeddings capture visual features. | Imagen is listed among Google models on Vertex AI. |
| Multimodal | A model processes or generates more than one modality, such as text, images, video, or audio. | Gemini Quickstart is multimodal; many foundation models span multiple modalities. |
Exam tip: Modality describes what a foundation model accepts or generates; a multimodal model can handle several modalities, so it is not limited to a single category.
A pre-trained foundation model is a broad base that teams adapt to narrower downstream tasks. The work flows from choosing a base model to prompting, tuning, and deploying it, with evaluation looping back among selection, prompting, and tuning. If no training dataset exists or available data is too small for custom training, teams can start with prompt engineering and then tune with data examples.
Access and hosting options determine where a foundation model runs and how it is customized. Google Cloud provides Google-built models through Vertex AI, while other models can run on Vertex AI or on customer-managed infrastructure. The choice controls who manages servers, accelerators, and deployment settings.
Embeddings are a shared mechanism that lets foundation models handle different modalities. They convert text, image, and video into arrays of floating point numbers called vectors, often reducing dimensionality. In the resulting continuous vector space, closer proximity means data points have similar meanings. Large language models use embeddings to understand text context, image generation uses embeddings to capture visual features, and retrieval-augmented generation (RAG) uses embeddings to match a user query with relevant documents in a knowledge base.
Grounding connects a model's output to verifiable sources, such as internal company documentation, project-specific data, or communication records, and it reduces hallucinations. Function calling connects LLMs to external tools like APIs and functions: the model analyzes a prompt, generates structured data that names the function and parameter values, the function runs, and results return to the model for its response. Function calling is a critical component in AI agents because it gives the model a structured way to choose tools and format inputs.
Model selection is the process of choosing the right model tier and settings to balance computational cost, response latency, and output quality. In BigQuery (opens in a new tab), natural language inputs flow from SQL functions to pre-trained Gemini Enterprise Agent Platform models and built-in BigQuery models, and generated results flow back into tables for analysis. Larger, more flexible models and longer prompts generally improve quality but increase token usage, cost, and latency, so teams must match the function tier and infrastructure to the size and importance of the task.
Control over model choice determines how much a team can tune quality versus cost and speed. General-purpose AI functions give full control and transparency on the choice of model, prompt, and parameters to use. This control is useful when the team needs to analyze structured or unstructured data, answer questions about data, or experiment with different prompts to improve output quality. The tradeoff is that more control requires more tuning and closer tracking of token usage and cost.
General-purpose functions in BigQuery support common enterprise tasks such as:
The three tiers of BigQuery AI functions represent different tradeoffs between flexibility, cost, and scale. General-purpose functions prioritize flexibility, managed functions prioritize optimized cost and quality at scale, and task-specific functions prioritize ready-made accuracy for a single task. Choosing a tier is the first performance decision because it sets the default balance of latency, cost, and output control.
| Function tier | Control over model and prompt | Choose when |
|---|---|---|
| General-purpose AI functions | Full control of model, prompt, and parameters | The work needs flexible inference, custom schemas, or embeddings |
| Managed AI functions | Streamlined syntax, optimized for cost and quality | The work runs over large tables and needs lower cost per row |
| Task-specific functions | Fixed behavior powered by Cloud AI APIs | The work is translation, vision, speech, or document processing |
General-purpose functions include AI.GENERATE for flexible inference on any data, AI.GENERATE_TEXT as a table-valued version that also supports partner and open models, and structured-output variants such as AI.GENERATE_TABLE, AI.GENERATE_BOOL, AI.GENERATE_DOUBLE, and AI.GENERATE_INT. Embedding functions include AI.EMBED for text or image data and AI.GENERATE_EMBEDDING to add a column of embedded text, image, audio, video, or document data for semantic search and clustering. Managed functions include AI.IF for natural-language filtering, AI.SCORE for rating quality or sentiment, AI.CLASSIFY for user-defined categories, and AI.AGG for summarization, with optimized mode in Preview scaling to millions or billions of rows.
Exam tip: Choose managed AI functions when the priority is optimized cost and quality over millions of rows, and choose general-purpose functions when the priority is control over the model, prompt, and parameters.
Balancing cost, latency, and quality in BigQuery depends on token use, row volume, and monitoring. The AI.COUNT_TOKENS utility estimates the number of tokens in an input prompt before a query runs, so teams can predict cost and latency before scaling up. BigQuery also provides pricing controls, token-usage tracking, cost tracking, monitoring, and error management for AI queries. For large workloads, starting with a small sample, checking quality, then switching to a managed function with optimized mode reduces the risk of high cost and slow responses.
Infrastructure behind the model also affects throughput and cost. AI Hypercomputer (opens in a new tab) is an integrated architecture that combines accelerators, open software, and flexible consumption to maximize goodput, the rate of useful data transferred, for demanding AI workloads. Accelerators connect over high-bandwidth networking to improve throughput across thousands of instances, while compute options scale differently. Batch scales automatically based on queued jobs, Cloud Run (opens in a new tab) scales automatically based on requests or CPU consumption, and Google Kubernetes Engine (GKE) adds or removes containers with autoscalers and nodes with the cluster autoscaler. A team that needs low-latency interaction favors automatic, request-driven scaling, while a team that needs bulk processing over large tables favors job-queue or optimized batch execution.
The model ecosystem in Vertex AI (opens in a new tab) is the collection of pre-trained and open models that teams can discover, deploy, and run for generative AI applications. The central hub for this ecosystem is Vertex AI Model Garden, which catalogs Google-built models, partner models, and open-source models in one place. A model is the set of parameters and structure that processes input data and returns output, and inference (opens in a new tab) is the process of running new data through a model to calculate that output. Governance controls who can use models and how deployed models are checked for quality.
Vertex AI Model Garden is the catalog where teams list available models, review verified deployment settings, and deploy open models to endpoints for prediction. Teams can list deployable models by name, such as filtering for gemma, and the list excludes tuned models so only base deployable versions are shown. For each model, Model Garden publishes verified deployment specifications including machine type, accelerator type, and container image URI. Deployment consumes quota in a designated quota project, so the project used for deployment must be set before listing and deploying.
Deploying an open model from Model Garden follows an ordered workflow:
gemma, and record the chosen model ID like google/gemma3@gemma-3-12b-it.g2-standard-12 with one NVIDIA_L4 accelerator, plus the container image to run it.First-party models are generative AI models built by Google and offered as managed services on Vertex AI. Examples include Gemini Enterprise Agent Platform models for text and multimodal tasks and Imagen, which is Google's text-to-image service on Vertex AI. These models are consumed through APIs without managing servers or accelerators, and BigQuery AI functions (opens in a new tab) can also call pre-trained Gemini models directly from SQL. Inference on managed models can run as online inference for real-time requests on individual data points or as batch inference that processes a group of requests and writes results to one file.
Imagen supports image creation, editing, and understanding from text and image inputs:
Third-party models are models built by Google partners and made available through Vertex AI. The leading example is Anthropic Claude, which teams access through the Anthropic Vertex AI SDK to send prompts and display generated text. Claude models on Vertex AI can use function calling to query external tools during generation. For example, a Claude prompt can call the Text Search Places API to retrieve place information and then return a complete grounded response.
Open-source models are community and industry models, including Hugging Face models, that teams self-deploy on their own Vertex AI endpoints. Examples include Google Gemma versions such as gemma-2-27b and gemma-3-1b-it, and Meta models such as meta-llama/Llama-3.3-70B-Instruct. Unlike managed first-party APIs, open models require the team to choose infrastructure, such as g2-standard-48 with four NVIDIA_L4 accelerators, and to accept the end-user license agreement at deploy time. This self-deployment model gives direct control over the endpoint, machine shape, and container that serves predictions.
Model choice in Vertex AI balances who maintains the model, how it is consumed, and how much operational work the team accepts. First-party models minimize infrastructure work, partner models add specialized capabilities under partner terms, and open models maximize deployment control at the cost of managing endpoints. The table below summarizes when each group fits best.
| Model group | Example models | How teams use it | Customization and operations |
|---|---|---|---|
| First-party Google | Gemini models, Imagen | Call a managed API or BigQuery AI function; choose batch or online inference | Fine-tune options such as Imagen style or subject models; Google manages infrastructure |
| Third-party partner | Anthropic Claude | Call through a partner SDK on Vertex AI; combine prompts with tool calls | Extend behavior with function calling to external APIs such as Google Places API |
| Open-source | Gemma, Llama-3.3-70B-Instruct, other Hugging Face models | List in Model Garden, review machine and accelerator specs, deploy to an endpoint | Control machine type, accelerator count, and container; must accept EULA and operate the endpoint |
Exam tip: Managed first-party and partner models are called as services, while open models such as Gemma must be self-deployed to a Vertex AI endpoint with an explicit machine type, accelerator, and EULA acceptance.
Governance in Vertex AI controls who can access models and data and how model quality is verified. Identity and Access Management (IAM) provides granular permissions that define who can do what on which Google Cloud resources. Permissions are granted to principals such as users, groups, or service accounts through roles, which enables precise control within a project or organization. Usage is also governed by quotas, because Model Garden list and deploy operations count against the quotas of the configured quota project.
Quality governance uses systematic evaluation before and after deployment. Teams define an evaluation specification, run the evaluation against the model, and retrieve metrics that describe performance. For summarization tasks, Vertex AI can compute ROUGE metrics that compare a model response against a reference ground truth. This combination of IAM access control, quota management, and metric-based evaluation keeps model selection tied to privacy, cost, and business risk.
General-purpose AI functions in BigQuery give full control over the choice of model, prompt, and parameters for flexible inference, while managed AI functions use streamlined syntax optimized for cost and quality at scale. Choose general-purpose functions when the priority is control over the model, prompt, and parameters, and choose managed functions when the priority is optimized cost and quality over millions of rows.
Use a first-party Google model such as Gemini or Imagen when you want to call a managed API or BigQuery AI function without managing servers or accelerators, and use an open-source model such as Gemma or Llama-3.3-70B-Instruct when you want direct control over deployment. Open-source models from Vertex AI Model Garden must be self-deployed to a Vertex AI endpoint with an explicit machine type, accelerator, and acceptance of the end-user license agreement, while Google manages infrastructure for first-party models.
Modality describes the kind of data a foundation model can accept, process, or generate, such as text, code, or image. A multimodal foundation model processes or generates more than one modality, such as text, images, video, or audio, so it is not limited to a single category.
Embeddings convert text, image, and video into arrays of floating point numbers called vectors where closer proximity means similar meanings, and large language models use them to understand text context while image generation uses them to capture visual features. Retrieval-augmented generation (RAG) uses embeddings to match a user query with relevant documents in a knowledge base.
An insurance company is modernizing its claims processing platform. The new application needs to support claims adjusters by processing multiple types of input within a unified workflow:
The leadership team wants to deploy a single base AI architecture capable of understanding and reasoning across these diverse data formats simultaneously rather than managing several disconnected, single-purpose pipelines.
Which foundation model classification meets these requirements?
Last updated