Professional Machine Learning Engineer
Vertex AI provides two primary ways to get predictions from a trained model: online prediction and batch prediction. Online prediction sends individual requests to a deployed model and receives immediate responses, which is ideal for real-time applications like fraud detection or recommendation systems. Batch prediction processes large volumes of data stored in Cloud Storage and writes results back to Cloud Storage, which works better for periodic scoring tasks like generating daily product rankings. The choice between them depends on whether the application needs instant results or can wait for asynchronous processing of accumulated data.
A model must be deployed to a Vertex AI endpoint before it can serve online predictions. The endpoint is a URL that accepts prediction requests over HTTP, and it runs the model in a container that handles request parsing, inference, and response formatting. Models can be deployed to regional endpoints that run in a specific Google Cloud region, or to global endpoints that can route traffic to the nearest available region. Each endpoint can host multiple model versions, and traffic can be split between them to support gradual rollouts or A/B testing.
Vertex AI supports two container approaches for model serving: pre-built containers and custom containers. Pre-built containers come with popular ML frameworks like TensorFlow, PyTorch, or scikit-learn already installed, and you simply upload your saved model files. Custom containers let you bring your own runtime environment, which is necessary when using less common frameworks or when you need specific library versions. Both container types run on Vertex AI's managed infrastructure, which handles scaling, health checking, and logging without requiring you to manage underlying compute resources.
Vertex AI endpoints can automatically adjust the number of compute instances based on demand to balance cost and latency. You set a minimum and maximum number of machine instances, and the service adds or removes instances as needed. The scaling signal is typically the number of concurrent requests per instance: when the average requests per instance exceeds a threshold, more instances are added, and when it falls below the threshold, instances are removed. This allows the endpoint to handle traffic spikes without manual intervention while scaling back down during quiet periods.
The compute capacity for model serving is determined by the machine type you select when deploying the model. Machine types differ in the number of CPUs, amount of memory, and whether they include GPUs for accelerated inference. GPU acceleration significantly speeds up inference for large models like deep neural networks but costs more than CPU-only options. The choice of machine type affects both the latency of individual predictions and the maximum throughput the endpoint can handle before auto-scaling adds more instances.
When multiple model versions are deployed to the same endpoint, traffic can be split between them using percentage-based routing. This enables blue-green deployments where new model versions initially receive a small fraction of traffic, then gradually take over as confidence grows. The load balancer distributes incoming requests across all healthy instances, and if an instance fails its health check it is automatically removed from the rotation. Health checks run continuously to ensure only functioning instances receive traffic, which maintains reliability even as the fleet scales up and down.
Online prediction provides immediate results for individual requests and requires a deployed model endpoint, while batch prediction processes stored data asynchronously and is better for scoring large datasets on a schedule.
Use custom containers when your model requires a specific runtime environment, library versions, or frameworks that are not available in the pre-built container options.
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts