OpenAI API integration: what breaks in production
When an OpenAI prototype goes into production, the first thing that breaks is not the quality of the model's responses, but the scaffolding around it.
The request leaves synchronously from the user handler, the connection drops at peak load — and the person sees a loading indicator instead of a result, even though the model has already computed and returned the response.
Next, timeouts and retries come into play. The client applies a crude retry to any network error, without distinguishing whether the request reached the model. This creates duplicates in business logic: two identical responses, quota charged twice, an email sent twice.
An idempotency key on the integration side removes this loop — the retry is tied to the original attempt instead of spawning new ones.
Request quota accounting is usually done "by eye", with a single counter for all scenarios. When a heavy background recalculation and a live user request share one limit, both fail, and the service goes into cascading degradation. Separate pools with priorities and an overflow queue keep interactive paths alive.
The most expensive class is silent failures. The model returns a success status with a truncated or template response: application logs are clean, metrics are green, but the output is garbage that the customer notices.
This is caught by validating the structure and meaning of the response on the integration side, rather than hoping for a 200 code. Below, we break down how each of these areas is covered.
What managed language model integration provides
Managed integration is a layer between your scenario and the model provider: request routing, retries, rate limiting, and a unified call log.
The scenario no longer calls the model directly — it works through a stable interface that decides which route to send the request through and what to do on error.
Response stability rests on structure validation: a response that does not match the expected schema does not go further, but is re-requested with a refined wording.
The share of successful calls is measured in the stream, not in a demo — it shows how each prompt change affects the result.
We keep response time within the target range: streaming the first chunk, caching repeated requests, choosing the model for task complexity. Simple requests do not wait for the heavy model, so the scenario responds predictably under load.
The failure class changes: model provider unavailability stops being a product failure.
The backup route picks up the request, the queue finishes incomplete requests — scenario downtime is measured in minutes, and recovery time is visible on a graph, not discovered from complaints.
Everything that happens at this layer goes into a log with tracing for each call. This way, degradation is found before the user notices it.
Connection modes: direct call, proxy, gateway
There are three ways to organize access to models, and the difference between them shows up not on day one, but as load grows, a second team appears, or the provider changes.
The connection mode determines where keys are stored, who counts request usage, and how much code will have to be rewritten when migrating.
Direct model call is the fastest start: the service calls the model itself, and the key is stored in its configuration. While there is one product and one team, this is justified — there is no intermediate link or extra latency.
The limitations of this mode appear later: the key spreads across environments, each service handles retries and limits in its own way, and changing the model turns into edits in several places.
| Mode | When it fits | Limitations |
|---|---|---|
| Direct model call | one service, one provider, early product stage | keys in every environment, no shared accounting of usage and retries |
| Proxy layer | need to hide keys from the client and add response caching | still one provider; routing and limits are written manually |
| Request gateway | multiple teams, models, and providers, audit requirements | more complex launch: the gateway itself must be deployed and maintained |
What determines the choice
The proxy layer removes keys from client applications and provides a point for cache — a repeated request does not go to the model.
But request routing between models and distribution of limits across teams remain manual work, and the proxy gradually accumulates logic that does not belong there.
A request gateway is a managed routing layer: rules decide which model responds to which type of task, limits are counted per team or project, and the log shows the path of each request.
The cost is a separate component that must be updated and monitored; for a single service, such an add-on is excessive.
Implementation stages: from scenario audit to handover
Model integration is not just a key in environment variables, but several layers: routing between models, retries, response validation, and observability.
Below is the order of work, in which each stage leaves an artifact that moves to the next and remains with the customer's team after handover.
- Scenario audit — inventory of calls: what tasks the model solves, how many requests per hour, context size, response time requirements. We find duplicate requests, manual workarounds, and places where the model is not needed at all.
- Routing scheme — distribute tasks across models in the lineup: a fast one for parsing and classification, a stronger one for reasoning. We define escalation rules for low confidence and caching for repeated requests.
- Retry rules — timeouts, delay between attempts, limit on concurrent calls, separate handling of rate limit errors. Without this, one spike of failures turns into a storm of retries.
- Quality validation — a reference set of requests with expected responses, automatic runs on every change to instructions and rules, and the share of correct responses as the main metric.
- Load testing — behavior at peak: response time, volume of model calls, correct degradation when the provider is unavailable.
- Handover to operations — configurations and instructions in the repository, request logging, metrics dashboard, incident response procedure, team training.
What we hand over: configurations and procedures
Along with the working OpenAI API integration, the customer receives a package that the on-call team can use to maintain the solution without chatting with us.
This is not a folder of files, but a set of artifacts, each covering its own class of questions: how it connects, who is allowed, what to do on failure.
- Connection configuration — model endpoints, model selection for a specific scenario, retries on failures, key rotation procedure, and a list of parameters that can be changed on the fly.
- Access policies — roles and key scopes, environment separation, secret storage rules, procedure for granting and revoking access when an employee leaves.
- Limits and quotas — thresholds for requests per minute and context size, system behavior when exhausted, priorities between scenarios, warning points.
- Operations manual — step-by-step startup and shutdown procedure, scheduled maintenance procedure, where to look at latency, error rate, and volume of processed requests, and analysis of typical failures.
- Escalation procedure — incident levels, response time for each, who decides to switch to the backup route, and what the on-call person does before the relevant engineer arrives.
- Dependency map and decision log — services, queues, and storage in the request chain, failure points, workaround routes, and rationale for selected parameters.
Such a package reduces onboarding from weeks to days: a new engineer brings up the environment according to instructions, not from the memory of a departed developer.
We agree on the procedures with your processes at the handover stage — otherwise they will remain a document for reporting, not a working tool.
Case study: failures and response time before and after restructuring
A request processing service called the model directly from business logic — without a queue, retries, or response validation. At peak loads, the failure rate reached 12%: the user got an error, and the incident was investigated manually.
Recovery time after a failure exceeded 40 minutes, and almost every third response was edited by an operator before sending.
Key OpenAI calls were moved behind an intermediate layer: a queue, retries with increasing delay, schema-based response validation, and a repeated request on mismatch.
Lightweight operations — classification and field extraction — were moved to a fast model, while complex reasoning was left to the strong one. This removed load from the heaviest path and stabilized response time.
We added caching for repeated requests, a timeout limit, and automatic switching to a backup provider on degradation. Each call writes metrics: model, duration, failure reason. Incidents are now investigated using data, not manually from logs.
Metrics before and after
Failure rate — 12% → 0.8%. Recovery time — from 40 minutes of manual work to automatic switching in a couple of minutes. Reduction in manual edits: 30% → 6%. The 95th percentile of response time was halved. Engineering hours that went into firefighting failures were redirected to product development.
What to do if some calls already go directly?
If part of services already access models directly, you cannot turn it off all at once — you will stop what works. Therefore, the first step is not technical but accounting: inventory of calls.
You need a map — which service calls which model, at what frequency, how critical the call is for the product, and where access keys are stored.
Then a control gateway is placed between applications and OpenAI.
It accepts the same requests that services sent directly and at first simply passes them through itself without changing anything — this is how the dual circuit works: the old path remains live, while the new one already sees all traffic and counts it.
Backward compatibility is key here: the request and response format is preserved, no changes to client code are needed, and services continue to work as before.
The gradual transition proceeds one route at a time. Each service connects to the shared circuit separately and with a switch that can return to direct calling at any moment — risk is localized to one service, not the entire system.
While migration is in progress, you can see the share of traffic through the gateway and compare responses from the old and new paths on identical requests.
When all traffic passes through the circuit, direct calls are closed. After that, keys live in one place and are rotated centrally, limits and retries are configured by a common policy, and revoking access stops being a search across repositories.
If the task is familiar — describe how many services and requests you have now, and we will come back with a migration plan.
Common technical questions about integration
Doubts at the integration stage are usually the same: what goes out in a request, how to avoid hitting rate limits, how to measure response quality, and what to do when the model provider fails. Below is a brief look at how this works in our integrations.
What data of ours goes into the request?
We assemble the context on our side and send it already trimmed. Personal data is masked before the call: identifiers are replaced with pseudonyms, and the reverse mapping stays with the customer. Call logs store metadata, not the text of the conversation.
How to avoid hitting request quotas?
Keys are separated by environment and service, and we request limits based on the load profile. Peak spikes are smoothed by a queue with retries and increasing delay, and as the ceiling approaches, a notification arrives — before the user sees an error.
How to tell that response quality has not dropped?
We run a set of reference examples with expected results on every model or prompt change. We look at the match rate and, separately, at confidently incorrect responses: they are more dangerous than explicit failures. The run catches regressions before release.
What to do when the model provider fails?
Redundancy is built into the access layer: on errors or growing latency, traffic switches to a second endpoint, and some scenarios switch to a smaller local model. We describe the degradation order in advance: where the response will become slower, where a template will be used.
Where is the boundary of responsibility?
We are responsible for integration, keys, observability, and predictable behavior during failures. On the customer's side are data processing rules and the list of information allowed to be sent. We fix this boundary in writing before the start.
Let's discuss the task and come back with an estimate and plan
To come back with an estimate and work plan, rather than general words, we need at minimum from you: a description of the scenario and 2-3 examples of real requests with the expected response.
Access to a test environment or its description and a list of constraints — what data must not be sent to an external service — are useful. If an OpenAI API integration already exists, we will add failure and timeout logs to the analysis: they show where response quality is lost.
What comes back is a document you can use to make a decision, not to discuss intentions. It will include:
- scope of work: which components we cover, what we reuse from existing code, what we write from scratch;
- stage timelines: broken down into prototype, validation on real requests, and rollout to production;
- support terms: model update procedure, response to quality degradation, procedure for prompt edits;
- failure handling scheme: retries, rate limiting, backup route when the provider is unavailable;
- acceptance metrics: share of correct responses on your example set and response time on a typical request;
- phased rollout plan with the ability to roll back each part separately from the others.
Describe the task in free form — what should work, for whom, and what counts as a failure. We will come back with an estimate and a plan.
