Prepare and test your skills
Prepare and test your skills
Agent reasoning paths and decision flows are the ordered sequence of model calls, tool uses, retrieved context, and reasoning steps that produce one agent response. Troubleshooting means rebuilding that sequence from user request to final output and finding which step introduced error, bias, or unexpected behavior. On Gemini Enterprise Agent Platform (opens in a new tab), this combines process-level explainability with production monitoring, where Vertex AI Model Monitoring (opens in a new tab) watches output health while trace data isolates the faulty component.
Agent execution decomposition breaks one agent response into linked model calls, tool invocations, and reasoning transitions. Data flows from the user request to the orchestrating model, then to each selected tool or retrieval system, then back to the model for synthesis, and finally to the user-facing output. Each handoff is a decision point where the agent selects the next model, the next tool with its arguments, or the next reasoning branch. When output is unsafe, biased, off-topic, or factually wrong, decomposition shows whether the failure came from model reasoning, a tool return, retrieved context, or prompt construction.
For retrieval-augmented generation (RAG) agents, retrieval needs separate checks for relevance of retrieved context and groundedness of the response to its sources. The Gen AI evaluation service (opens in a new tab) automates assessment against defined criteria, including factual accuracy, safety attributes, toxicity scores, and style or format adherence. Key observability signals include input-output token counts, hallucination rates, query resolution failures, vector database query throughput, and user-feedback ratings. Together these signals show whether the agent failed at retrieval, reasoning, generation, or task completion.
Trace logs are the primary explainability mechanism for complex large language models (LLMs) and agents because they record the process the agent followed. A complete trace names the source model call, each destination tool or subordinate model, the inputs and outputs at each component boundary, and the order in which steps executed. Prompt engineering can guide the model to expose its reasoning steps, while RAG applications should return citations that link each grounded claim to its retrieved source.
Production capture depends on Cloud Logging (opens in a new tab), Cloud Trace, and Cloud Monitoring working together across the inference chain:
Audit trails add accountability by recording who accessed specific models or tools, what data was used for fine-tuning or prompting, and what queries reached production endpoints. Vertex ML Metadata and Model Registry preserve lineage from raw data to processed datasets to trained model versions, including training parameters, evaluation metrics, dataset versions, and known limitations. That lineage lets an operator move backward from a bad agent step to the exact pipeline artifact, model version, or data source that produced it.
Vertex AI Model Monitoring is the managed service for detecting training-serving skew, data drift, and prediction drift for deployed models. The service compares live inference data against the training baseline and sends alerts when incoming data skews too far, which signals that retraining or re-evaluation may be needed. For agent endpoints, monitoring also tracks shifts in prediction distributions and spikes in low-confidence predictions that often precede degraded reasoning or tool selection.
Continuous output monitoring applies the same drift idea to generative behavior and safety. Operators watch generative outputs for unsafe, biased, off-topic, or malicious content, and Model Armor screens prompts and responses for common risks and sensitive data exposure. Granular prediction validation checks outputs against expected ranges, formats, and schemas, with alerts routed through Cloud Monitoring to notification channels and automated remediation. Monitoring results then feed back into continuous training, evaluation with Agent Platform Pipelines, and grounding checks that provide a second layer of source verification.
Exam tip: Use feature attributions for tabular models deployed to Agent Platform endpoints, and use trace logs for LLM agents where explainability means reconstructing the process the agent followed.
Fault isolation is the ordered workflow that links a production symptom to one failing step in the agent chain. The workflow depends on the trace, the monitoring alerts, and the lineage records all referencing the same request and model version. Reconstruction comes first, then testing of each component in isolation, so a retrieval error is not mistaken for a reasoning error.
Remediation such as prompt revision, tool replacement, retrieval tuning, or retraining happens only after the source step is identified. For tabular sub-models inside a larger agent, Vertex Explainable AI (opens in a new tab) adds feature-level evidence with integrated gradients for deep neural network classifiers and regressors and sampled Shapley for boosted tree and random forest models. The What-If Tool (WIT) and Language Interpretability Tool (LIT) support interactive exploration of bias and natural language behavior during that deep-dive stage.
Explanation methods divide by model type because tabular predictions and agent decisions fail in different ways. Tabular explainability attributes one prediction to its input features, while agent explainability attributes one outcome to a chain of calls and decisions.
| Option | Primary unit | Choose when |
|---|---|---|
| Vertex Explainable AI for tabular models | Feature attribution for one prediction | The question is which input features drove a classification or regression on an Agent Platform endpoint |
| Trace logs for LLM agents | Ordered process of model calls, tool uses, and reasoning steps | The question is which step in the inference chain caused error, bias, or unexpected output |
Both forms belong in monitoring dashboards and, where trust requires it, exposed directly through application interfaces. Continuous monitoring then detects shifts that indicate emerging bias or unfairness, completing the loop from production observation back to decomposed troubleshooting.
Monitoring for explanation quality and drift continuously checks that model explanations remain stable, consistent, and faithful to model behavior after deployment on Agent Platform Inference. A prediction can stay online while its explanation degrades, so drift detection, metrics, and explanation signals must share the same dashboards and retraining loop. When thresholds are breached, traffic flows from the endpoint to alerting and retraining systems that investigate, roll back, or retrain the model.
Explanation signals for Agent Platform endpoints are the outputs that show why a model acted as it did. For tabular models, feature attributions (opens in a new tab) indicate which input features contributed most to each prediction. For generative models and agents, explainability means explaining the process the agent followed, such as trace logs that record tool calls, retrieved context, and intermediate steps, plus citations for retrieved information in RAG applications. The What-If Tool, which integrates with TensorBoard, lets teams interactively explore model behavior and potential bias without changing production code.
Explanation quality degrades in three related ways that monitoring must separate. Stability means similar inputs produce similar attributions over time, consistency means the same model version produces the same reasoning pattern across data slices, and fidelity means the explanation still matches what the model actually used to decide. Documentation tracks this through shifts in top-ranked features, shifts in prediction distributions, spikes in low-confidence predictions, changes in toxicity and coherence, grounding failures against source text, and disparate performance across fairness slices. When feature importance reorders, trace patterns diverge, or a subgroup shows emerging bias, the explanation no longer aligns with domain knowledge even if aggregate accuracy looks unchanged.
Drift detection on Model Monitoring on Agent Platform (opens in a new tab) compares live production data against the training baseline to catch changes that undermine explanations. A drift detection job runs on a schedule against logged predictions and features, evaluates distance or distribution metrics, and emits results to Cloud Monitoring for dashboards and alerts. The job depends on versioned training data and on prediction logs from the endpoint, so Model Registry lineage that links each model version to its dataset version, training parameters, and evaluation metrics is required to interpret any shift correctly.
| Drift type | What is compared | What it signals for explanations |
|---|---|---|
| Data drift | Live input feature distribution compared to training distribution | Attributions may shift because the model sees unfamiliar feature values, even if outputs have not moved yet |
| Prediction drift | Live model output distribution compared to training or prior serving distribution | Decision boundaries or agent behavior have moved, so prior explanations and fairness assessments may no longer hold |
| Explanation drift | Live attribution patterns, trace patterns, or slice-level behavior compared to validated baseline | Top features, reasoning steps, or subgroup outcomes changed, indicating loss of stability, consistency, or fidelity |
Data drift is detected by monitoring feature serving for drift, validating inputs against expected ranges and formats, and tracking distributions over time with tools such as TensorFlow Data Validation (TFDV) (opens in a new tab), BigQuery (opens in a new tab), and Dataflow. Prediction drift is detected by monitoring prediction distributions for shifts, validating output schema, tracking toxicity and coherence for generative endpoints, and watching for unsafe, biased, off-topic, or non-compliant outputs. Explanation drift is inferred when those signals combine with direct checks: reordered attributions, new trace paths, failed grounding checks from the Gen AI evaluation service, and fairness measures that diverge across slices.
Exam tip: Data drift is a change in inputs versus training, while prediction drift is a change in model outputs; explanation drift is confirmed only when attributions, traces, or slice-level reasons also shift.
Automated checks turn explanation monitoring into metrics, dashboards, and notifications that trigger action. First, the team configures drift detection jobs with thresholds and alerts for data drift, prediction drift, latency, throughput, and error rates. Then results flow from Model Monitoring into Cloud Monitoring (opens in a new tab), where they join custom metrics for business KPIs and log-based metrics from Cloud Logging. Finally, alert policies route violations to notification channels such as email, Slack, or PagerDuty and to automated remediation such as scaling, rollback, or retraining triggers.
Operational checks that support explanation trustworthiness include:
Storage and real-time paths preserve evidence for diagnosis. Validation results and monitoring metrics are stored in BigQuery for retrieval, trend analysis, and historical tracking, while detailed prediction logs are kept in Cloud Logging to debug specific requests. For generative endpoints, real-time validation events flow from the endpoint check to Pub/Sub, which then fans out to alerting, logging, and incident management systems. Audit trails that record who accessed models or tools, what data was used for fine-tuning or prompting, and what queries reached production endpoints flow to the same logging path.
Monitoring output feeds back into continuous training and governance rather than ending at an alert. When data patterns change significantly or explanation fidelity drops, the alert triggers an Agent Platform Pipelines workflow that revalidates data, re-evaluates the model, and retrains or fine-tunes on updated data. Evaluation results are logged to Agent Platform ML Metadata for lineage and to Model Registry as versioned metadata, including training parameters, dataset version, framework or foundation model details, and documented limitations, intended uses, and known biases.
Controlled deployment protects explanation trust during the transition. Traffic splitting on Agent Platform endpoints supports rolling deployment, canary deployment, and A/B testing of agent versions, so a small subset of production traffic first flows from the endpoint to the new model version while performance and error rates are monitored. Automated rollback reverts endpoint traffic to the previous stable model version if monitoring alerts fire or performance thresholds are missed. Before any traffic shift, automated endpoint testing checks prediction correctness, latency, throughput, and API function, and pipeline gates re-run explainability evaluations such as feature attributions and responsible AI checks.
Select and Configure Explainability Methods for Agent Platform Models in this material means configuring AI agents to work directly with enterprise data models and document models through standard connections. The workflow centers on the Looker extension for VS Code (opens in a new tab) for LookML development and a Document AI (opens in a new tab) custom processor for field extraction. Agents inspect schemas, propose code or labels, and validate results before changes are committed.
The AI-assisted development workflow connects a local IDE, the Looker extension, a managed context server, and prebuilt instructions. Your IDE and AI Agent (opens in a new tab) is a local editor such as VS Code, Cursor, or Claude Code with an integrated agent such as Gemini CLI or Claude. The Looker extension for VS Code provides syntax highlighting, autocomplete, bidirectional file synchronization with a Looker instance, and integrated validation. The Looker-managed MCP server uses the Model Context Protocol (MCP) to connect agents directly to the enterprise database and LookML models, and a custom or self-hosted MCP server such as MCP Toolbox for Databases can also be used. Prebuilt skill files give the agent coding standards and project-specific instructions, installed and updated automatically by the extension or through the Looker: Install Skills in this Workspace and Looker: Install Skills Globally commands.
Data flows from the database and LookML models through the MCP server to the agent in the IDE. The agent reads local LookML files, inspects database schema through the MCP server, applies changes locally, and runs LookML validation to self-correct before commit. A Looker admin configures the Looker-managed MCP server on the Looker instance, and the extension connects to it automatically.
Prerequisites establish authentication, context access, project content, and permissions before prompting begins. The setup depends on all four elements being complete, because the agent cannot inspect schemas or edit models without them.
develop permission for the model to be edited.Only after these steps does natural language prompting produce usable LookML, because the agent needs the local files, live schema, and write permission together.
Prompting the agent uses natural language plus workspace skills to generate, refactor, and troubleshoot LookML. Clear prompts that name the connection, tables, files, and desired outputs yield the best results. The agent works from local files and live schema context rather than from static examples.
ecommerce_db, inspect the users and orders tables, generate users.view.lkml and orders.view.lkml with primary keys, standard dimensions, and count measures, then generate ecommerce.model.lkml that explores orders and joins users on user_id.products.view.lkml, find number dimensions representing price or cost, generate matching sum and average measures with descriptions, and match the style from the prebuilt skills.Inaccessible view: users, and asking the agent to review ecommerce.model.lkml and users.view.lkml, identify the cause, and propose a fix using MCP validation tools.In each case the level of detail in the prompt controls the quality of the generated or fixed LookML.
Review, validation, and deployment move agent-proposed edits from the local workspace to the development branch and then to production. The local workspace is the source, the development branch on the Looker server is the sync target, and Git is the deployment path.
Exam tip: Always run Looker: Validate LookML before sync and deploy, because saving automatically syncs to the development branch on the Looker server.
Auto-labeling with a Document AI custom processor uses defined field names and prior annotations to label documents at scale. The processor depends on descriptive field definitions, imported documents, and human review to produce training-ready data. Field definition starts on the Get Started tab with Create new field, where a descriptive name and property description provide context, location information, and text patterns for each entity. Good property descriptions disambiguate confusing sources and guide the model toward consistent extraction across structure variations.
Document import happens on the Build tab with Import documents, where the document path, target set, auto-labeling option, and foundation model are selected. Review happens under Build and Manage Dataset, where predictions are highlighted in purple for correction and missing fields are added. After review, select Mark as labeled and ensure the document is in either the Training or Testing set, because inaccurate fields will affect model performance.
Data drift compares the live input feature distribution to the training distribution, prediction drift compares the live model output distribution to the training or prior serving distribution, and explanation drift is confirmed only when attribution patterns, trace patterns, or slice-level behavior also shift from the validated baseline. Data drift signals that attributions may shift because the model sees unfamiliar feature values, prediction drift signals that decision boundaries or agent behavior have moved, and explanation drift signals loss of stability, consistency, or fidelity.
Use Vertex Explainable AI for tabular models deployed to Agent Platform endpoints when the question is which input features drove a classification or regression, and use trace logs for large language model agents when the question is which step in the inference chain caused error, bias, or unexpected output. Vertex Explainable AI attributes one prediction to its input features, while trace logs record the ordered process of model calls, tool uses, retrieved context, and reasoning steps.
Reconstruct the execution order by pulling trace logs and Cloud Trace spans for the failed request to list every model call, tool use, and reasoning step from user input to final output, then correlate that step with Cloud Monitoring alerts, Vertex AI Model Monitoring drift or skew findings, low-confidence spikes, toxicity scores, or log-based metrics. Then validate arguments sent to each tool, data returned by each tool or retriever, and prompt context passed to each model call, check model version, dataset version, and pipeline artifacts in Model Registry and Vertex ML Metadata, and confirm with targeted evaluation through the Gen AI evaluation service and grounding checks.
Last updated