Prepare and test your skills
Prepare and test your skills
Visual and Multimodal Analysis covers Google Cloud APIs that read existing images and video and models that create new visual content. Analytical APIs describe what is already in the pixels, while generative models create or reason about images from prompts. Cloud Vision API (opens in a new tab) handles still images, Video Intelligence API (opens in a new tab) handles video over time, and models such as Imagen (opens in a new tab) and Gemini handle generation and multimodal reasoning.
Cloud Vision API is a pretrained API that analyzes a single still image in one request and returns structured annotations. An application sends image bytes or a Cloud Storage reference with one or more feature types, and annotations flow back from the API to the application. Because each request is independent, the API has no notion of motion, duration, or change between frames. Choose Cloud Vision API when the input is a photo, scan, PDF page, or single video frame that must be understood on its own.
The API selects behavior through explicit feature types that control what is returned for that image:
Exam tip: Use TEXT_DETECTION for sparse text in a photo, but use DOCUMENT_TEXT_DETECTION for dense documents, PDF/TIFF files, or handwriting; DOCUMENT_TEXT_DETECTION takes precedence when both are present.
Video Intelligence API is a pretrained API that analyzes video as a sequence of frames over time and returns time-stamped annotations. Data flows from a video file in Cloud Storage or a local upload to an annotation operation, and segments showing what happened and when flow back to the application. Because the API tracks information across frames, it can describe motion, duration, and change that a single-image call cannot see. Choose Video Intelligence API when the question involves temporal event analysis within video streams.
Temporal analysis centers on tracking and segmentation rather than one-frame inspection. The API annotates labels across a clip, detects persons and faces as they move through scenes, detects text that appears in video frames, detects explicit content, detects shot changes where the camera cuts, and transcribes speech from the audio track. The same operation supports both stored files and streaming input, so a batch file and a live stream both produce annotations with timestamps and confidence scores.
Generative image models create new visual assets, while Cloud Vision API and Video Intelligence API only describe existing pixels. Imagen is a text-to-image generation model on Vertex AI (opens in a new tab) that produces or edits images from a text prompt, including automatic mask detection and inpainting where Imagen creates the mask area and then adds requested content inside it. Gemini models are multimodal foundation models trained on large datasets of text, images, code, and other media that can reason across text, image, video, audio, and code in a single prompt. An application can therefore send an image plus a question and receive generated text, which is generation and reasoning rather than fixed-feature detection.
A decision tree that branches on whether to describe existing pixels or create and reason, then on still image versus video over time, to pick Cloud Vision API, Video Intelligence API, Imagen, or Gemini.
The choice depends on input and primary action, as compared below:
| API or Model | Input | Primary action | Choose when |
|---|---|---|---|
| Cloud Vision API | Single image or document page | Detects objects, text, faces, labels, and properties in that frame | The task is object detection or OCR on stills and scans |
| Video Intelligence API | Video file or stream with audio and frames | Tracks labels, persons, faces, text, shots, and speech over time | The task needs temporal event analysis within video streams |
| Imagen and Gemini | Text prompt, optionally plus image, video, or audio | Generates new images, edits regions, or answers across modalities | The task is to create, complete, or reason about visual content |
Custom training follows the same split between analysis and generation. AutoML Vision (opens in a new tab) object detection can be trained to find domain-specific sections of images, and AutoML Vision classification can classify image content. AutoML Vision classification is not generally recommended for document classification because text-based methods after OCR tend to be more accurate.
Conversational and Linguistic Interfaces are Google Cloud AI APIs and platforms that let applications hear, understand, and speak human language. Single-function APIs handle one conversion step, while conversational platforms orchestrate those steps into automated customer service agents and natural language processing workflows. The core pattern is that audio becomes text, text is understood and acted on, and response text becomes audio again.
Speech-to-Text is the API that integrates speech recognition into applications by transcribing audio into written text. Audio flows from the source file to the API, and text flows back as the result, such as a voicemail recording becoming a text message. In BigQuery, audio transcription follows an ordered pattern where a remote model is created over the Speech-to-Text endpoint with CLOUD_AI_SPEECH_TO_TEXT_V2 and the ML.TRANSCRIBE function runs against that model. The function reads audio files from object tables, inference occurs in Agent Platform, and results are stored in BigQuery for further processing.
Text-to-Speech is the API that converts text into natural-sounding speech and performs the opposite direction of Speech-to-Text. Text flows from the application to the API, and synthesized audio flows back for playback to a caller or app user. In a voice agent, Text-to-Speech depends on the upstream dialog decision, because the platform must first decide what to say before the API can speak it. Choose Text-to-Speech when the output must be heard over voice channels rather than read on a screen.
Cloud Natural Language API is the pretrained API for advanced natural language understanding on written text. Text flows from a table or application to the API, and linguistic insights such as classification and sentiment analysis flow back, such as estimating whether product feedback shows customers like a product. In BigQuery, a remote model is created with CLOUD_AI_NATURAL_LANGUAGE_V1 and the ML.UNDERSTAND_TEXT function runs against that model. The function works with data in standard tables, inference occurs in Agent Platform, and results are stored in BigQuery.
Conversational AI Platform is the collection of conversational AI tools and solutions that orchestrate language APIs into working agents. Dialogflow ES designs standard conversational interfaces, while Dialogflow CX handles complex conversations using virtual agents with lifelike conversational AI. The single-function APIs convert one modality, but Dialogflow manages the multi-turn flow where user input arrives, language understanding determines the next step, and a response is returned across voice and digital channels. Choose Dialogflow ES when the conversation is standard and linear, and choose Dialogflow CX when the conversation needs complex branching, state handling, and scaling.
An automated customer service agent is a composed system where Speech-to-Text, Dialogflow, and Text-to-Speech depend on each other in order. The ordered flow carries data through three stages:
Supporting services surround this core loop and handle routing, assistance, insight, and language. These services extend the agent without changing the core order:
Exam tip: Speech-to-Text turns audio into text for input, while Text-to-Speech turns text into audio for output — Dialogflow sits between them to orchestrate what happens during the conversation.
The language options split by input, output, and role in a workflow. Each option has a distinct place in the chain from voice entry to spoken exit to offline analysis:
| Option | Input to output | Role in workflow |
|---|---|---|
| Speech-to-Text | Audio to text | Entry point that makes voice input readable by downstream processing |
| Text-to-Speech | Text to audio | Exit point that makes a decided response hearable by the caller |
| Cloud Natural Language API | Text to classification and sentiment | Analysis step that understands meaning in stored text |
| Dialogflow | Conversation turn to next action and response | Orchestrator that coordinates turns across voice and digital channels |
Use Speech-to-Text plus Dialogflow plus Text-to-Speech together for voice agents. Use the Cloud Natural Language API alone or inside BigQuery for bulk text analysis without conversation management.
Information Retrieval and Discovery is the process of finding and retrieving relevant information from a large collection of data. In Google Cloud, information retrieval (IR) (opens in a new tab) is a key component of Vertex AI Search (opens in a new tab). Vertex AI Search uses IR to retrieve documents from a corpus based on a user's query, so enterprise applications can discover answers in their own content instead of relying only on what a model already knows.
Vertex AI Search is a pre-built search solution for building a discovery engine or a Retrieval Augmented Generation (RAG) application. Vertex AI offers a suite of APIs to help builders create their own search engine or build their own RAG applications on the same retrieval foundation. Because the search capability is pre-built, a team does not have to build ranking and retrieval from scratch and can focus on connecting its corpus and queries.
Data flows from the user's query to Vertex AI Search, and Vertex AI Search retrieves matching documents from the corpus back to the application. The corpus is the large collection of data that is searched, and the query is the request that selects the relevant documents from that corpus. This relationship lets one retrieval service support discovery across diverse repositories while keeping the query, the corpus, and the returned documents clearly separated.
Grounding with Vertex AI Search is the practice of connecting Gemini (opens in a new tab) output to an organization's own data stored in a Vertex AI Search data store. The data store holds the enterprise content, Gemini generates the answer, and Vertex AI Search supplies the relevant retrieved documents that ground that answer. The RAG Engine (opens in a new tab) can use Vertex AI Search as a retrieval backend, so retrieval flows from the Vertex AI Search data store to the RAG Engine and then to the Gemini model that produces the grounded response.
This grounding pattern gives two advantages for enterprise-grade discovery. First, semantic search finds meaning-based matches in the corpus rather than only keyword matches, which improves discovery across diverse content. Second, pre-built retrieval avoids custom ranking work and provides current, source-linked context for generative answers instead of relying only on pretrained knowledge.
Exam tip: Ground Gemini output to a Vertex AI Search data store when the answer must come from your own data, and use Vertex AI Search as the retrieval backend for RAG Engine when you need IR over a corpus.
Cloud Vision API analyzes a single still image in one request and returns structured annotations, while Video Intelligence API analyzes video as a sequence of frames over time and returns time-stamped annotations. Choose Cloud Vision API when the input is a photo, scan, PDF page, or single video frame that must be understood on its own, and choose Video Intelligence API when the task needs temporal event analysis such as tracking motion, duration, and change within video streams.
Use TEXT_DETECTION for sparse text within a larger image and use DOCUMENT_TEXT_DETECTION for dense text, PDF or TIFF documents, or handwriting. DOCUMENT_TEXT_DETECTION takes precedence when both feature types are present in the same Cloud Vision API request.
In an automated customer service agent, Speech-to-Text first transcribes spoken audio from the caller into text, Dialogflow then orchestrates the turn by determining the next action and response, and Text-to-Speech finally converts the response text back to audio for the caller. Speech-to-Text is the entry point that makes voice input readable, Dialogflow is the orchestrator that coordinates turns across voice and digital channels, and Text-to-Speech is the exit point that makes the decided response hearable.
Vertex AI Search retrieves matching documents from a corpus or data store based on a user query, and those retrieved documents supply the current, source-linked context that grounds the answer generated by Gemini models. The Retrieval Augmented Generation (RAG) Engine can use Vertex AI Search as a retrieval backend, so retrieval flows from the Vertex AI Search data store to the RAG Engine and then to the Gemini model that produces the grounded response.
An enterprise organization wants to implement an internal knowledge discovery platform across disparate repositories, including unstructured PDF policy manuals in Cloud Storage, structured employee directory tables in BigQuery, and public-facing corporate websites.
The development team needs an out-of-the-box solution that supports semantic search, extracts keywords automatically, and generates Large Language Model (LLM) answers grounded in internal data with citations, without having to manually generate embeddings or manage vector index sharding.
Which Google Cloud service should the cloud architect recommend?
Vertex AI Search
Vertex AI Vector Search
Vertex AI Feature Store
Cloud Natural Language AI
Last updated