LLM deployment on Vertex AI: what breaks in production
LLM deployment on Vertex AI usually goes smoothly in a demo: one request, warm context, manual response validation. Vertex AI covers the baseline layer — quotas, autoscaling, model versioning — but production begins where requests run in parallel around the clock. This is exactly where model deployment in the cloud exposes what is not visible in a test run.
Peak inference load hits latency first: the request queue builds up, response time grows from fractions of a second to tens of seconds, and some calls fail by timeout. This is model failure under load — the service is formally alive, but the user does not receive a response.
Degradation is nonlinear: up to a certain threshold everything looks normal, and after that it collapses avalanche-like, and scaling cannot keep up with growth.
A separate class is memory leaks in long-lived processes around the model: context accumulation, unreleased buffers, growing caches. The instance stays up for a day or two, then restarts on a liveness check, and on cold start latency increases again. On graphs, this looks like rhythmic dips that are attributed to "cloud instability."
The third failure point is updates: changing the model version or system instruction changes the response distribution, and what yesterday strictly followed the format begins to improvise. Below we examine how these three scenarios are addressed by configuration and processes.
What managed model deployment provides
The model is described declaratively: image version, instance size with accelerator, minimum and maximum number of replicas, limit on concurrent requests. Each change is a new revision, to which its own share of traffic is bound.
This is the essence of managed deployment: configuration lives in a repository, not in the head of the engineer who manually brought up the server.
Model response time stops depending on whether someone managed to warm up a replica. Always-on instances eliminate cold start, and new replicas are added based on queue length and accelerator utilization, not on a one-minute average.
At peaks, the 95th percentile of response time remains flat — scaling triggers before the queue starts to grow.
The main gain is eliminating failures that previously came with an update. Rollout proceeds in shares: first, part of the traffic goes to the new revision, we compare response time and error rate, and only then move the rest.
If metrics drop, rollback to the previous revision takes minutes and does not require a rebuild — the class of failures "an update brought down the working environment" disappears.
Inference scaling is about multiple models coming into play. Revisions for different tasks live behind a common endpoint, each has its own quotas and limits, and the load balancer distributes the load.
Vertex AI provides this out of the box, so the team spends time on response quality and data rather than on a homemade layer around model launch.
LLM hosting options: comparison of modes
The choice of hosting mode determines three things: where model weights and data physically reside, who is responsible for scaling under load, and how quickly a new version is rolled out.
In practice, four options are compared — from a fully managed environment (for example, deployment on Vertex AI) to one isolated within the customer's perimeter. Below is what each provides and what platform limitations you will have to accept.
| Option | When it fits | Limitations |
|---|---|---|
| Managed cloud platform | Need fast start, elastic scaling for peaks, no in-house operations team | Data and logs reside with the provider, limits on model size and accelerator types, some settings unavailable |
| Self-managed cloud environment | Need control over network, keys, and logs, but deployment speed matters | You are responsible for the accelerator fleet, updates, and fault tolerance |
| Isolated environment | Personal data, regulator requirements, prohibition on external calls | Scales more slowly, hardware is purchased for peak load, manual updates |
| Hybrid scheme | Sensitive data is processed inside, heavy computation outside | More expensive to support: two environments, end-to-end tracing, unified access policy |
Hybrid provides the best balance, but works only with formalized data classification: without it, part of requests with personal data goes to the external environment unnoticed. Therefore, the hosting mode is chosen together with request routing rules, not after them.
Implementation sequence: from audit to handover
We fix the sequence before start: each stage ends with a verifiable artifact; otherwise acceptance turns into a conversation about impressions. The stages proceed sequentially, and at each one we understand exactly what is needed from your team.
- Infrastructure audit. We analyze what already exists: projects, networks, permissions, volumes and quality of data for training and fine-tuning. Separately, we record load requirements — how many concurrent requests the service handles and what response latency is acceptable. The output is a map of constraints and risks with priorities.
- Hosting scheme. We decide what goes to the managed service and what is deployed on your own capacity: models with sensitive data usually require the second option. The scheme shows failure points, network boundaries, and the release order for new versions.
- Environment setup. Separate environments for development and production, service accounts with the minimum required roles, private network access without public addresses, secrets in a secure vault. The entire description is stored as code, so the environment is reproducible rather than assembled by hand.
- Model build and connection. We prepare an image with weights and dependencies, configure autoscaling for peaks, concurrency limits, and logging without personal data. The new version is first released to a small share of traffic.
- Acceptance check. We run a profile of real requests: measure latency on long inputs, observe behavior when a node fails and quota is exhausted, check rollback to the previous version — it should take minutes, not hours.
- Handover and team training. We hand over access, schemes, the update procedure, and analysis of typical incidents, and train your engineers. For the first weeks of service operation, we stay alongside and adjust settings based on actual metrics rather than assumptions.
What remains with you: configurations and procedures
A working model is only the top layer. Below remains what allows you to maintain it on your own: a reproducible environment, clear permissions, documented procedures.
We hand over configuration source files in a repository, not a set of screenshots from the control panel — staging and production environments can be brought up from code again without contacting us.
- Environment configuration. Model parameters, request limits, instance sizes, and environment variables — in a version control system, with explicit description of differences between environments.
- Integration scheme. The request path from your application to the model: authentication, queues, cache, handling long responses, and retries on timeouts.
- Access policies. Roles of service accounts, permissions to invoke the model and read data, separation of access between test and production environments.
- Documentation. API contracts, request and response formats, error codes, behavior when quotas are exhausted and when an external source is unavailable.
- Update procedure. How a new version of the model, prompt, or dataset is rolled out: who approves it, what quality checks pass before release.
- Rollback procedure. Where we return to during degradation: previous version, checkpoints, target recovery time, and the decision owner.
- Observability. A set of indicators — failure rate, response latency, token consumption per request — and thresholds that trigger an alert.
Case study: eliminated response degradation under peak load
Peak load is the only mode that truly matters in production.
For a support request service, generation on Google Cloud Vertex AI held up to a certain flow, and beyond that degradation began: responses stretched out, some requests were cut off by timeout, the queue accumulated and accelerated latency even more. We investigated not the model, but how it was deployed.
What we changed
We separated resource pools for light and heavy requests, moved the timeout limit to the layer in front of the model, added a cache for repeated requests, and added warm-up traffic before opening the load window.
Version updates stopped being a downtime event: the new revision is brought up alongside, receives part of the traffic, and the previous one remains ready for rollback.
| Metric | Before | After | What we changed |
|---|---|---|---|
| Response under peak | 6–9 s, up to 20 s by the end of the window | 1.5–2 s, variance ±15% | Pool separation, request timeout |
| Failure rate | 7% | 0.4% | Concurrency limiting and cache |
| Recovery after failure | 40 min of manual actions | 4 min, automatic rollback | Warm-up and disabling unhealthy pool |
| Model version update | Night window, manual rollback | 10% traffic to new revision | Gradual shift and instant rollback |
The main thing is that behavior stopped depending on how well the day went. Under overload, the service returns a short response or an honest error instead of a hanging wait, and response degradation is visible in metrics before the user sees it. This is the difference between "the model works" and "the solution works."
How does a managed platform differ from a self-managed environment?
A managed platform takes on the infrastructure layer: you choose a model from the catalog, the hosting region, and the access mode, while scaling under load, updates, and service availability are handled by the provider.
Deployment takes days, not months, and the first requests go into operation almost immediately. The freed-up time is spent by the team on what actually affects the result — data preparation and quality of wording.
The flip side is the boundaries of control. Only what is exposed in the platform interface can be configured: quotas, network isolation, request logging, access policy.
You cannot change the inference engine, rewrite the queue scheduler, or keep model weights within your own perimeter here — you will have to wait until the provider offers the required capability.
A self-managed environment is the opposite set of trade-offs. Weights, library versions, fine-tuning scheme, and network isolation are fully in your hands, but capacity planning, queues, fault tolerance, and driver compatibility also transfer there.
This is not "install it and it works": you need a team that operates model inference in production and investigates degradations.
Data ownership is the second decision point. In the managed option, requests and logs remain within the customer's cloud perimeter, access is granted through isolated channels, and obligations for processing and deletion are fixed in the contract.
In a self-managed environment, data physically does not leave your site, but storage, key management, auditing, and deletion on demand are built and maintained by you.
The choice comes down to three questions: how sensitive the data is, how predictable the load is, and whether there is a team for ongoing support.
A hybrid often wins — validating the scenario on Vertex AI, then moving the sensitive part to a self-managed environment; with a unified access layer, the application does not have to be rewritten.
Technical doubts: answering point by point
Doubts at the assessment stage are almost always the same: can LLM deployment on Vertex AI fit into a closed environment, will the service survive a model update, how much has to be rewritten in the existing application, and who will get access to request logs. We answer point by point — details are fixed in the project documentation before work starts.
Will you deploy in a closed environment without internet access?
Yes. We place request processing in a private network: outbound traffic is closed by perimeter rules, and model images and dependencies are pulled from an internal registry. Only metrics and depersonalized events go outside through an approved gateway.
How is the model updated without downtime?
The new version is brought up alongside the old one and tested on shadow traffic until response quality matches the reference. Route switching is performed at the gateway, and the previous version remains running — rollback takes seconds.
How much will have to be rewritten in the existing application?
Only the model access layer changes: the application calls an internal gateway in the usual request format. Product logic, interfaces, and integrations remain; the work shifts to data preparation and quality validation on your scenarios.
Who gets access to logs and how is it segregated?
By roles through the corporate directory: a developer sees technical metrics, an analyst sees depersonalized dialogues, and support sees only their own requests. Log retention is limited, and every access to them is recorded in the audit log.
Let's discuss the task and come back with an estimate and plan
To start, it is enough to describe the task in free form: what scenarios the model covers, where the data comes from, what requirements there are for access and privacy, and whether there is a hard launch date.
If there is already a prototype on Vertex AI or work with requests and responses — attach it, this shortens the analysis.
Based on the analysis, we come back with an estimate and plan that fixes the scope of work, stage timelines, and control points — places where you see the result, not a report about the process.
- a list of inputs that are missing for the start, and who provides them: project access, datasets, subject matter expert;
- scope of work by stage: data preparation and labeling, model selection and configuration, quality evaluation loop, deployment on Vertex AI, and observability;
- timelines for each stage and dependencies between them — what runs in parallel and what blocks the next step;
- control points with a measurable acceptance criterion: share of correct responses on a holdout set, response time, behavior on failure;
- deployment scheme: which managed services are used, how environments are separated, and how permissions are restricted;
- handover order: configurations, documentation, training for your team, and a rollback plan to the previous version.
Describe the task to the extent you have it, and we will come back with an estimate and work plan and agree on the start.
