Prototyping Model Garden Models in Notebooks

Notebook Environment Integration and SDK-Based Experimentation is the practice of running Vertex AI SDK and Google Cloud client library code in a managed notebook runtime with GPU or TPU accelerators. The runtime uses Agent Identity (opens in a new tab) to supply credentials automatically, so notebook code can query Google Cloud APIs and deploy prototypes without managing service account keys. The flow runs from the runtime to the metadata server to the client library to the Google Cloud service. The metadata server issues a short-lived token, the client library attaches the token to API calls, and Google Cloud authorizes the calls with Identity and Access Management (IAM).

1.1Granting Agent Access to Google Cloud Resources

Granting agent access means binding an IAM role to the agent identity at the scope of a single agent, a project, or an organization. The binding connects a principal identifier to a target resource defined by service and resource name, such as storage with a bucket name or bigquery with a dataset ID. The deployment project is identified by organization ID and project number, while a single-agent binding additionally requires location and reasoning engine ID.

The three principal scopes differ by identifier format and blast radius:

Choose a single-agent principal when least privilege matters, and choose a project or organization principal set only when all agents in that container require the same role. The Google Cloud CLI grant replaces SERVICE, RESOURCE_NAME, PRINCIPAL_IDENTIFIER, ORGANIZATION_ID, PROJECT_NUMBER, LOCATION, ENGINE_ID, and ROLE with the values for the target resource and identity scope.

1.2Referencing Agent Identity in SDK Code

Referencing Agent Identity in code means letting Application Default Credentials resolve to the agent token instead of hard-coding credentials. Google Cloud client libraries automatically use Agent Identity when code runs in a supported environment, such as Agent Runtime or a Cloud Run container. Manual use follows the same flow where credentials move from the metadata server to google.auth.default(), then to the service client, then to the Google Cloud API.

For example, a tool built with the Agent Development Kit (ADK) creates a Cloud Vision API client with the retrieved credentials and project ID, builds a vision.Image from a Cloud Storage URI, and calls label_detection. The API response flows back to the tool, which returns success with label descriptions or failure with response.error.message. Because the client depends on runtime-provided credentials, the same notebook or agent code works across environments without credential files.

1.3Securing the Runtime with Context-Aware Access

Securing the runtime with Context-Aware Access means applying the Google-managed security baseline that protects Agent Identity credentials beyond the Agent Gateway. The policy enforces Demonstrating Proof of Possession (DPoP) by authenticating the agent access token and enforces mutual Transport Layer Security (mTLS) to access the Agent Gateway. Certificate-bound tokens can then be used only from their intended trusted runtime environment, for example a Cloud Run container, which makes stolen credentials unreplayable and helps protect against credential theft and account takeover.

Opting out removes token binding and leaves short-lived access tokens vulnerable to theft or replay. Opt out only for rare requirements:

• specific token-sharing among agents
• direct token injection in the header
• unrecoverable 401 UNAUTHENTICATED authentication failures

To opt out, set the deployment environment variable GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES to False in config.env_vars.

1.4Deploying with Agent Identity Enabled

Deploying with Agent Identity enabled activates the identity binding for code running in Agent Runtime. The deployment depends on the Vertex AI SDK client, the agent framework wrapper, and the identity configuration working together. First the client is initialized, then the agent is wrapped, and only after that is the remote engine created with identity enforcement.

1. Initialize the Vertex AI client: Create vertexai.Client with project, location, and http_options set to api_version="v1beta1" for Agent Identity support.
2. Wrap the agent: Wrap the framework agent in AdkApp, for example AdkApp(agent=agent), so the runtime can invoke it.
3. Create with identity type: Call client.agent_engines.create with agent=app and config containing identity_type set to types.IdentityType.AGENT_IDENTITY and requirements including google-cloud-aiplatform[agent_engines,adk].

When deployment completes, calls from the remote app to storage, bigquery, vision, and other Google Cloud services authenticate as the agent principal and are authorized by the IAM roles granted to that principal or principal set.

Parameter-Efficient Tuning and Prototype Endpoint Deployment is the practice of taking a notebook-developed prototype, including parameter-efficient approaches such as LoRA and PEFT, and deploying it directly to a managed runtime for latency and performance validation. The deployment path runs from notebook code through the Vertex AI SDK to an Agent Runtime instance, where Agent Identity (opens in a new tab) provides a system-attested principal for calling Google Cloud services. Secure web access to development interfaces is provided separately by Component Gateway on the cluster first master node.

2.1Deploying Prototypes Directly From Notebook Workflows

Deploying prototypes directly from notebook workflows creates a managed Agent Runtime endpoint for the prototype without separate serving infrastructure. The notebook initializes a vertexai.Client with project, location, and http_options=dict(api_version="v1beta1") for Agent Identity support, wraps framework code such as an ADK Agent in AdkApp, and calls client.agent_engines.create() with agent=app and config={"identity_type": types.IdentityType.AGENT_IDENTITY} plus requirements and staging_bucket. The call returns a remote_app whose api_resource.spec.effective_identity is the provisioned identity for validation traffic.

An identity can be provisioned before code is ready to allow IAM policy setup before validation. The notebook calls client.agent_engines.create() with only config={"display_name": "identity-for-agent", "identity_type": types.IdentityType.AGENT_IDENTITY} to create the instance, then adds code later with agent_engine.update(...). This separation lets platform and security owners grant resource access to the future endpoint principal before the prototype is deployed. Once code is added, the same endpoint serves validation requests with the pre-authorized identity.

Deployment can also be performed through lightweight tooling that sets the same identity flag. The Agents CLI deploys with agents-cli deploy --agent-identity for rapid testing with foundational monitoring resources. ADK deployment uses a .agent_engine_config.json file containing { "identity_type": "AGENT_IDENTITY" } before running adk deploy. Both paths create an Agent Runtime instance with a read-only, system-attested identity based on SPIFFE.

2.2Agent Identity and Access Control for Deployed Prototypes

Agent Identity is the SPIFFE-based principal that a deployed prototype uses to authenticate to Google Cloud services. The identity format is principal://TRUST_DOMAIN/NAMESPACE/AGENT_NAME, for example principal://agents.global.org-ORGANIZATION_ID.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/LOCATION/reasoningEngines/ENGINE_ID. The trust domain is auto-provisioned when the Agent Platform API is enabled as agents.global.org-ORGANIZATION_ID.system.id.goog for an organization or agents.global.proj-PROJECT_NUMBER.system.id.goog for a project without an organization. The namespace is the immutable resource path and the agent name is the immutable reasoning-engine ID.

Google Cloud client libraries use Agent Identity automatically from the trusted runtime metadata server. A prototype can call google.auth.default() to retrieve Agent Identity credentials and pass them to a service client, such as vision.ImageAnnotatorClient, so traffic flows from the prototype to the target API as the agent principal. Access is granted with IAM roles on the target resource, where the principal identifier selects the scope of access:

The grant binds the selected principal to a role on the target service and resource, such as a storage bucket or BigQuery dataset. Because the binding is evaluated at request time, rotating prototype code does not require new keys as long as the same Agent Runtime identity serves the traffic. Broader principalSet bindings simplify validation across many prototypes but expand the blast radius compared to a single-agent grant.

2.3Securing Prototype Endpoints With Context-Aware Access

Securing prototype endpoints with Context-Aware Access means applying the default Google-managed policy that makes Agent Identity credentials unreplayable outside their intended runtime. The policy enforces Demonstrating Proof of Possession (DPoP) for the agent access token beyond the Agent Gateway and requires mTLS to access the Agent Gateway. The control plane therefore binds certificate-bound tokens to the trusted environment, such as a Cloud Run container, so stolen short-lived tokens cannot be replayed from elsewhere. This baseline protects prototype validation traffic against credential theft and account takeover.

Opting out removes token binding and should be reserved for rare token-sharing or header-injection requirements or an unresolvable 401 UNAUTHENTICATED error. To opt out, the deployment sets config={"env_vars": {"GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": False}} when the Agent Runtime instance is created. After opt-out, short-lived access tokens become vulnerable to theft and replay because DPoP and mTLS binding no longer constrain their use. The tradeoff is direct token portability at the cost of the default replay protection.

2.4Accessing Prototype Interfaces Through Component Gateway

Component Gateway provides secure browser and API access to web interfaces on the first master node without SSH tunnels or inbound firewall changes. When Component Gateway is enabled, the platform adds Apache Knox and Inverting Proxy to the cluster first master node to proxy a specific subset of services. Component Gateway serves default and optional component interfaces and REST APIs such as Hadoop YARN, Apache Livy, and history servers, but it does not proxy arbitrary node:port addresses, which still require an SSH SOCKS proxy. Users must hold the dataproc.clusters.use permission to open the proxied interfaces.

A Component Gateway-enabled cluster is created through the console, gcloud CLI, API, or Terraform with the same effect. The console path uses this ordered workflow:

1. Open creation: Open the Create cluster page in the Google Cloud console.
2. Expand options: Click Additional configuration, then edit Optional components.
3. Enable gateway: Select the Enable components in the UI checkbox to enable Component Gateway.
4. Save options: Click Save to apply the optional-component configuration.
5. Create: Configure remaining cluster settings and click Create cluster.

The equivalent programmatic controls are --enable-component-gateway in gcloud dataproc clusters create, EndpointConfig.enableHttpPortAccess=true in a clusters.create request, and the google_dataproc_cluster Terraform endpoint_config block. The default Knox Gateway SSL certificate is valid for 13 months from cluster creation, and all Component Gateway URLs become inactive after expiry until the certificate is regenerated. For Managed Service for Apache Spark clusters, the Compute Engine default service account must hold the Dataproc Worker role on the project, while the human creator holds Dataproc Editor and Service Account User, because the service account provisions the cluster resources.

Vertex AI Model Garden is the central catalog for discovering, evaluating, and prototyping with foundation models inside Google Cloud. Model Garden aggregates first-party Google models, third-party partner models, and open-source models in one searchable inventory, and each listing exposes its modalities, license terms, deployment options, and notebook-ready sample code. Prototyping happens by opening a model card and launching it directly into Vertex AI Workbench or Colab Enterprise, where the notebook inherits the project identity, dependencies, and Vertex AI SDK access needed to test prompts and APIs.

3.1First-Party, Third-Party, and Open-Source Model Categories

First-party, third-party, and open-source model categories organize Model Garden by publisher and operating model, which determines support, tuning, and deployment control. First-party models are Google-built models such as Gemini for text, code, reasoning, and multimodal work, plus Imagen for images and Veo for video, and they are served as fully managed APIs with enterprise support and continuous updates. Third-party models are commercial partner models such as Anthropic Claude, Meta Llama, Mistral AI, and AI21, which are curated into Model Garden as managed APIs or deployable weights with publisher-defined terms and versioning. Open-source models are community weights such as Llama, Mistral, Gemma, and Falcon variants that can be previewed in the catalog and then deployed to a dedicated Vertex AI Endpoint for full control over serving, scaling, and fine-tuning.

The three categories differ in who manages the serving stack and who owns the operational risk:

Once deployed, notebook code calls all three categories through the same Vertex AI SDK patterns, so a prototype can swap baselines without rewriting the surrounding data pipeline.

3.2Modalities and Capability Evaluation

Modality defines what a model accepts as input and produces as output, and Model Garden exposes this on every model card as text, code, image, audio, video, or multimodal combinations. A Gemini multimodal model accepts interleaved text, images, audio, and video and returns text or code, while Imagen and Veo specialize in text-to-image and text-to-video generation, and code-specialized variants optimize for completion, explanation, and translation. Evaluation starts in Model Garden by filtering the catalog by task and modality, then comparing model cards for context window length, supported languages, function calling, grounding, and safety tuning.

Effective baseline evaluation in a notebook follows an ordered comparison:

1. Define the task metric: Establish accuracy, latency, cost per 1K tokens, and safety thresholds from business requirements before testing any model.
2. Prompt the managed API: Send representative prompts from Workbench or Colab Enterprise to first-party and third-party APIs to measure zero-shot and few-shot quality without provisioning hardware.
3. Stress modality limits: Test long context, multi-image reasoning, audio length, and structured output to confirm the model sustains quality at production input sizes.
4. Compare against tunable baselines: Deploy one or two open-source candidates to endpoints only after managed APIs establish the quality bar, so tuning effort is spent where it closes a measured gap.

This order keeps prototyping fast because managed APIs validate feasibility first, and dedicated deployments are reserved for cases that need customization or data control.

3.3Licensing and Commercial Trade-Offs

Licensing in Model Garden determines whether model outputs and weights can be used commercially, redistributed, or fine-tuned. First-party and third-party managed APIs are consumed under Google Cloud terms and publisher use policies, where payment is per API call or per generated token and no weight redistribution occurs. Open-source models carry their own licenses, ranging from permissive licenses such as Apache 2.0 to community licenses such as Llama Community License and Gemma Terms of Use, which may restrict redistribution, require attribution, or limit use by very large platforms.

Licensing trade-offs directly shape baseline selection for business requirements. Choose a managed first-party or third-party API when the priority is speed to prototype, predictable pay-as-you-go cost, and Google-managed compliance controls for data handling. Choose an open-source deployable model when the priority is weight inspection, on-tenant serving, custom distillation or LoRA fine-tuning, or avoiding per-token fees at high sustained volume, accepting that the team then owns monitoring, patching, and scaling.

3.4Hardware Constraints and Serving Footprint

Hardware constraints apply differently to APIs and to self-deployed weights. Managed first-party and third-party APIs hide accelerators entirely, exposing only quotas, rate limits, regions, and context-window tiers that bound throughput in the notebook prototype. Open-source deployments expose the full serving footprint, where the model size in billions of parameters determines the minimum accelerator memory, replica count, and quantization strategy needed on the Vertex AI Endpoint.

A large dense model requires multiple high-memory GPUs such as NVIDIA A100 80GB or H100 instances, drives higher idle cost, and scales throughput by adding replicas behind the endpoint. A smaller or quantized model fits on a single L4 or T4 GPU, starts faster, costs less per hour, and suits latency-sensitive or edge-adjacent prototypes, but may sacrifice reasoning depth or multilingual coverage. The notebook prototype must therefore record not only quality scores but also tokens per second, time to first token, concurrent request behavior, and GPU utilization, because a model that wins on quality but exceeds the latency or cost budget is not a viable baseline.

3.5Selecting the Baseline for Notebook Prototyping

Baseline selection in Model Garden is the decision that balances capability, license, and hardware against technical and business constraints. The selection starts from the model card in Model Garden, flows into a managed notebook where prompts and evaluation datasets run against candidate APIs, and ends with a documented choice of one managed baseline plus one deployable challenger when customization is in scope. The notebook depends on Model Garden for discovery and sample code, depends on Vertex AI APIs for managed inference, and depends on Vertex AI Endpoints only when open-source weights require dedicated accelerators.

• Choose Gemini first-party models when the workload needs broad multimodality, long context, tool use, and native integration with Vertex AI Search, agents, and grounding services.
• Choose third-party models when evaluation shows a measurable advantage on domain language, safety posture, or instruction following that justifies separate terms and quota management.
• Choose open-source models when the solution requires VPC-contained serving, weight-level auditing, or downstream fine-tuning on proprietary data that cannot leave the project boundary.

Key Facts

• Vertex AI Model Garden aggregates first-party Google models, third-party partner models, and open-source models with exposed modalities, license terms, deployment options, and notebook-ready sample code.
• Agent Identity is a SPIFFE-based principal that lets code in Agent Runtime authenticate to Google Cloud services without service account keys and be authorized by IAM roles.
• Context-Aware Access enforces Demonstrating Proof of Possession (DPoP) and mutual Transport Layer Security (mTLS) so agent tokens are usable only from their intended trusted runtime.
• Component Gateway adds Apache Knox and Inverting Proxy to the cluster first master node to proxy selected web interfaces, requires dataproc.clusters.use permission, and uses a default certificate valid for 13 months.
• Open-source deployments require dedicated accelerators on Vertex AI Endpoints, where large models need multiple A100 80GB or H100 GPUs and smaller or quantized models can fit on a single L4 or T4 GPU.