Prepare and test your skills
Prepare and test your skills
Model interaction and refinement describes the two main ways to improve what a generative AI model produces. Prompt engineering (opens in a new tab) changes the request sent to the model, while model tuning (opens in a new tab) changes the model itself through training. Prompt engineering is faster and lighter because the model stays untouched, while tuning is heavier and is used when a specialized task needs more than a good prompt can deliver.
Prompt engineering is the iterative process of writing a prompt, reading the model's response, and adjusting the prompt until the output matches the goal. A prompt is a natural language request sent to a generative AI model, and it can contain text, images, videos, audio, documents, or multiple modalities at once. Because the model's learned parameters stay the same, prompt engineering only changes the instructions and context the model sees at request time.
Common prompt engineering techniques differ by how many examples the prompt includes:
Examples help control formatting, phrasing, scope, and overall patterns in the response. Too few examples may not guide the model enough, while too many can cause the model to overfit to the examples and fail to generalize, so experimentation is needed to find the right balance.
Model tuning customizes the default behavior of a foundation model (opens in a new tab) so it consistently produces desired results without complex prompts. Unlike prompt engineering, tuning changes the model by training it on a dataset you build, using a method such as supervised tuning, reinforcement learning from human feedback (RLHF) tuning, or model distillation. The size of the training dataset and the choice of method depend on the model and the goal, and specialized niche tasks often need only a smaller dataset to show strong improvement.
Tuning is useful when prompt design alone cannot handle the task, such as when a language model must learn specific terminology. A tuned model can then follow simpler prompts, which can reduce the cost and latency of requests. After a tuned model is production-ready, it can be deployed to an endpoint and monitored like any standard machine learning operations (MLOps) workflow.
Prompt engineering and model tuning solve different problems with different levels of effort. Prompt engineering changes only the prompt text, while model tuning changes model behavior through training. The practical order is fixed: first try prompt engineering, then move to tuning only if the model still performs poorly, for example if it hallucinates or fails specific output requirements.
| Approach | What changes | Inputs needed | Choose when |
|---|---|---|---|
| Prompt engineering | The prompt text only; model parameters stay the same | Well-structured prompts, optionally with examples | The model responds well enough with clearer instructions or examples |
| Model tuning | The model's behavior, through training | A training dataset and a chosen tuning method | The task is specialized and prompt design alone is not enough |
Exam tip: Prompt engineering changes the request sent to the model, while prompt tuning changes the model itself by training it on a dataset.
Model evaluation measures how prompts and customizations affect performance using an evaluation dataset of prompt and ground truth pairs. Ground truth is the ideal response used for comparison, and metrics-based evaluations can be automated and scaled quickly with quantifiable results. Metrics-based checks can oversimplify results and miss the context of natural language, while automatic side-by-side evaluation uses a third, larger model to pick the higher-quality response but can carry bias, so human evaluation should still be used.
Deployment depends on whether the model was tuned. Google's foundation models in generative AI on Vertex AI (opens in a new tab) already have endpoints that are specific to your Google Cloud project and immediately available. If you tune one of those models, you must deploy it to an endpoint before use. When deploying, Vertex AI (opens in a new tab) offers a fully managed environment where it instantiates and manages resources for you, and a self-managed environment where you keep fine-grained control but manage resources yourself.
Foundation models (opens in a new tab) are large, powerful models trained on vast amounts of data that often span text, images, video, and audio. Foundation models use statistical modeling to predict likely responses to prompts and to generate new content, learning patterns such as language patterns for text and diffusion techniques for images. In Google Cloud, foundation models are accessible through a managed API, including the Gemini family of models and models in the Model Garden.
A large language model (LLM) is a text-driven foundation model trained on a vast amount of data. Large language models learn patterns and relationships within language using techniques such as reinforcement learning and instruction fine-tuning. Large language models rely on embeddings to understand the context and meaning of text, which lets the LLM generate coherent and relevant responses for natural language processing tasks with text input and text output.
Large language models support common text tasks including:
Multimodal models are foundation models that can understand and generate content from multiple modalities, including text, images, videos, and audio. Multimodal models accept prompts that can contain text, images, videos, audio, documents, or multiple modalities together. The Gemini (opens in a new tab) family of models is designed for multimodal use and can process information from images, videos, and text together.
Multimodal models support capabilities such as answering complex questions that combine text with related images. They also support creating summaries from multiple sources that can include text, image, and video or audio components. Choose a multimodal model when the prompt or the answer must combine text with images, video, or audio.
Exam tip: Large language models (LLMs) are text-based, whereas foundation models can be trained with and receive input from multiple modalities, including text, images, audio, and video.
Diffusion models provide the image-generation capability inside foundation models by learning diffusion techniques from training data. Diffusion models use embeddings that capture the visual features of images, which enables the models to create realistic and diverse outputs. A diffusion model takes a text prompt as input and generates a new image that resembles the style and patterns of the training data.
The same capability also supports editing tasks driven by text prompts, such as removing an object or changing a color scheme. It also supports generating short videos or animations from text prompts or scripts. Choose diffusion-based generation when the goal is to create or edit images from text.
Embeddings are arrays of floating point numbers called vectors that are created during training by converting text, image, and video into a format models can process. Embeddings map data into a continuous vector space where closer proximity reflects data points with similar meanings. Embeddings often reduce the dimensionality of data, which helps enhance computational efficiency and enables processing of large datasets.
Because embeddings preserve meaning and visual features in the same kind of numerical form, they connect the two model types. Embeddings enable large language models to discern language patterns and enable diffusion models to discern visual patterns. In short, data flows from raw text and images into vectors, and the models then operate on those vectors.
Large language, multimodal, and diffusion capabilities differ by the data types they process and the content they generate. The underlying structure of each capability determines whether it handles text only or multiple data types. The choice depends on whether the work is text in and text out, combined modalities, or image creation.
| Capability | Input data types | What it generates |
|---|---|---|
| Large language model | Text | Text such as generated text, translations, summaries, and answers |
| Multimodal foundation model | Multiple modalities, including text, images, videos, and audio | Responses that combine or summarize multiple modalities, including text with related images |
| Diffusion-based image model | Text prompt descriptions, plus images for editing | New realistic images, modified images, and short videos or animations |
Artificial intelligence (AI) (opens in a new tab) is the broadest field, covering systems that learn from data to classify information, predict outcomes, or generate new content. Machine learning (ML) (opens in a new tab) is a subset of AI in which models learn patterns from data rather than following only hand-written rules. Generative AI is a specific field of machine learning that develops and uses models for generating new content, so the nesting is AI contains ML and ML contains generative AI.
Artificial intelligence includes both traditional AI and generative AI approaches to working with data. Traditional AI models excel at learning from existing data to classify information or predict future outcomes based on historical patterns. Generative AI models expand those capabilities to create summaries, uncover complex hidden correlations, or generate new content like text, images, or videos. The shared idea is learning patterns from training data, while the difference is whether the output labels the future or creates something new.
Machine learning provides the models that learn from historical data sources such as tabular data and images. Traditional AI use cases built with ML typically focus on predicting a future outcome or assigning data to a category. In these tasks, data flows from historical examples into a trained model, and the model outputs a label, number, forecast, or group for new input.
Common types of traditional AI work include:
Natural language processing (NLP) is the area of AI concerned with understanding and working with human language. Large language models (LLMs) are text-driven foundation models trained on vast amounts of data that perform NLP tasks. Those tasks include text generation, machine translation, text summarization, and question answering. In this hierarchy, an LLM depends on ML training, and NLP describes the language work the LLM performs.
Generative AI is a type of AI that uses foundation models (opens in a new tab) to generate new content by learning the underlying patterns and style of the training data. A foundation model is a large, powerful model trained on vast amounts of data, often across text, images, video, and audio, that uses statistical modeling to predict likely responses and create new output. The new output resembles the data it was trained on, including text, images, audio, or video. Choose this approach when the goal is creation, summarization, or multimodal understanding rather than only classification or prediction.
| Approach | Primary goal | Typical output |
|---|---|---|
| Traditional AI | Classify information or predict future outcomes from historical patterns | Labels, numbers, forecasts, or customer segments |
| Generative AI | Generate new content that reflects style and patterns in training data | New text, images, audio, video, summaries, or code |
Exam tip: Traditional AI classifies or predicts from historical data, while generative AI creates new content that reflects patterns in its training data.
Artificial intelligence is the broadest field covering systems that learn from data to classify information, predict outcomes, or generate new content. Machine learning is a subset of artificial intelligence in which models learn patterns from data rather than following only hand-written rules, and generative AI is a specific field of machine learning that develops and uses models for generating new content.
You should first try prompt engineering, which changes only the prompt text while the model parameters stay the same, and move to model tuning only if the model still performs poorly. Model tuning changes the model itself by training it on a dataset you build and is used when a specialized task needs more than a good prompt can deliver, such as when a language model must learn specific terminology.
Large language models are text-driven foundation models that take text as input and produce text such as generated text, translations, summaries, and answers. Multimodal foundation models can understand and generate content from multiple modalities including text, images, videos, and audio, while diffusion models take a text prompt as input and generate or edit realistic images and short videos or animations.
Embeddings are arrays of floating point numbers called vectors that are created during training by converting text, image, and video into a format models can process. They map data into a continuous vector space where closer proximity reflects similar meanings, which lets large language models discern language patterns and diffusion models discern visual patterns.
An enterprise architecture team is designing an AI-powered platform to support a global marketing department. The platform must handle two main generative tasks:
Which statement accurately compares the underlying data processing capabilities and architectural roles of Large Language Models (LLMs), Multimodal Foundation Models, and Diffusion Models?
Last updated