What breaks in serverless CI/CD as the team grows
While there are a dozen functions and one person handles releases, deployment rests on their local machine and memory: build, upload, check, post in chat.
As the team grows, the same order falls apart — several developers edit the same function declaration (SAM, Serverless Framework), manual releases run in parallel, and no one sees the full picture of changes.
The first symptom is environment desynchronization. In the test environment, the environment variables, roles, and limits are different; some triggers and queues were once created manually through the console.
Deployment to prod either overwrites these resources or fails on a name conflict, and the release turns into sorting out differences between staging and prod instead of shipping changes.
There is no rollback as a standard operation. The previous version of the function remains in the cloud, but traffic is switched to it manually, and if the event schema or permissions changed along with the code, reverting becomes a separate emergency task.
At night this is felt most acutely: recovery is measured not in minutes but in hours of chat and manual commands.
Access rights are blurred: anyone with access to the infrastructure account can deploy to prod, and there is no boundary between "built and tested in staging" and "released to prod." Deployment automation for functions does not appear on its own under such conditions — it is designed separately. Below we look at how this is addressed.
Measurable effects: deployment speed and failure rate
While the build runs on an engineer's machine and deployment follows a multi-step instruction, the time from finished code to a working version is measured in hours.
A mistake in a step is not visible until production, and reverting to the previous version is done manually — during an incident, this is the most time-consuming path.
After moving to a managed pipeline, the configuration of functions, permissions, and events is described declaratively: with AWS SAM and similar tools, a single file remains the source of truth for all environments.
The build runs in an isolated environment from pinned dependencies, so the "works locally, doesn't work in the cloud" discrepancy ceases to exist as a class. Deployment time is reduced to 5-8 minutes, and deployment itself becomes routine rather than an event.
The share of successful releases grows from about 70% to 95% and above. Failures related to configuration go away: a forgotten environment variable, excessive permissions on a role, diverged library versions. Logical defects in code remain — they are caught by running tests before production.
Frequency changes as well: instead of two scheduled releases per week — several deployments per day as soon as ready. Changes become small, the failure surface becomes narrow, and rollback takes 1-2 minutes because the artifact is immutable and a reference to it is switched.
We measure the effect before work begins and after: time from code change to production, share of deployments without manual intervention, average time to return to the previous version, and number of releases per week.
Deployment modes in serverless CI/CD: comparison of approaches
The deployment mode is chosen not by team size but by how the function receives traffic.
A synchronous call through a gateway can be split by weights and the error rate observed; an asynchronous trigger from a queue or on a schedule cannot: there the new version takes all events at once, and the impact can be limited only by switching the binding or reverting the previous version.
| Option | When it is appropriate | What limits it |
|---|---|---|
| Full deployment in one run | Staging, internal services, a bundle of several functions that change together | There is no intermediate control: an error is visible only after the fact, rollback is a redeployment of the previous set |
| Gradual release by traffic shares | Prod with live request flow and inbound metrics | Automatic stop criteria are needed; asynchronous calls cannot be divided by shares |
| Environment switching | A critical service where predictable rollback in seconds matters | Double set of resources; external storages and data schemas are shared, migrations will have to be made compatible |
| New version with an alias | Instant return without rebuilding and rerunning tests is needed | Requires a version retention policy and control so the alias does not lag behind trigger bindings |
The working scheme is not one mode for everything, but its own for each group of functions: API handlers are rolled out by traffic shares, background tasks switch entirely.
In AWS Lambda and related platforms, weights and aliases are defined in the same template as the function itself, so the deployment strategy is versioned together with the code and goes through review.
Rollback of a version is planned in advance: the previous alias is retained, the data schema is checked for compatibility with it, and a return step is kept in the pipeline that triggers based on the error metric, not on a call from the on-call engineer.
How the pipeline is built: from audit to handover
Pipeline implementation begins not with writing configuration but with a build audit: we look at how artifacts get to production now, which steps are done manually, and where engineers keep the sequence of actions in their heads.
The output is a map of bottlenecks and a clear sequence of work where each stage produces a verifiable result.
- Build and release cycle audit. We review current builds, manual operations, failure points, and time from commit to deployment. We record what will be automated first and what is deliberately left as is.
- Environment scheme. We determine how many environments are needed and how they differ in parameters, permissions, and data. We agree that the same artifact is promoted between environments without rebuilding — otherwise checks lose their meaning.
- Build pipeline setup. Functions are described declaratively (SAM or Serverless Framework); the pipeline builds them from the description: runs style checks and unit tests, packages the artifact once, and places it in versioned storage.
- Environment and access setup. Parameters are moved out of code into environment configuration, secrets into secure storage, and access to cloud resources is granted through roles with the minimum necessary permissions instead of long-lived keys.
- Pre-deployment checks. Integration tests on a deployed staging environment, validation of permissions and parameters, smoke tests immediately after release. Rollback to the previous version is rehearsed in advance and takes minutes, not hours.
- Release validation and handover. We run the release together with the customer's team, measure build and recovery time, and train them: how to start a deployment, roll back, add checks, and change environments independently. The environment scheme, pipeline description, and failure procedure remain.
Infrastructure as code, secrets, and access policies
Everything that describes the environment, permissions, and deployment procedure lives in the repository next to the code — not in chat and not in the memory of the engineer who ran the project.
The environment is brought up from the description rather than assembled manually from memory, and returning to a previous state is as simple as reverting a commit. Below is the set of artifacts that remain with the customer and work without us.
- Environment descriptions as code: functions, execution roles, timeouts, bindings to events and queues — declaratively, with change history and diff review before applying.
- Secret management: keys and connection strings are moved to storage, injected at deployment time, and only references to names remain in the repository — a code leak does not give access to production.
- Access policies: separate roles for the pipeline and for the function itself, the minimum necessary set of permissions per resource, and a ban on long-lived keys and shared accounts.
- Pipeline documentation: stages, trigger conditions, which checks block deployment, how to read the build log, and where to look for the cause of a failure.
- Deployment and rollback regulations: version publication procedure, traffic switching method, time to return to the previous version, and conditions under which rollback is triggered automatically.
- Environment separation: separate sets of parameters and limits for test and production environments, and a ban on shared resources between them.
- Reproduction instructions: how to bring up the environment from scratch using the description without access to the original staging environment.
These documents remove dependence on specific people: a new engineer gets up to speed on the project in days, not weeks of questions.
Implementation case study: metrics before and after
The project is a request processing service: about fifteen functions, three environments, a release every two weeks.
Description and deployment were handled through AWS SAM, but manually from the on-call engineer's machine: environment settings lived in their personal files, and staging was rebuilt from scratch after an unsuccessful release. Rollback took hours — previous function versions and traffic routes were not retained.
What was changed in the process
- Infrastructure was moved to the repository: environments, permissions, queues, and triggers are reproduced with one command.
- Build and deployment were separated: the artifact is built once and promoted across environments without rebuilding.
- Configuration validation was added before deployment — lost permissions and incorrect variables are caught in advance, not in prod.
- A staged release was introduced: first part of the traffic, then all of it, with automatic rollback when errors increase.
- Previous function versions and routes are retained — rollback takes minutes instead of hours.
- Parallel releases were separated: each branch has its own set of resources, and teams no longer wait for each other.
Metrics before and after
Recovery time decreased from 4–6 hours to 8 minutes, and build time from 18 to 4 minutes. The share of releases that had to be redone fell from a third to about one in twenty: configuration is checked before deployment.
Three teams can now deploy in parallel instead of one, and environment overhead has stopped growing — unused staging environments are shut down on a schedule.
How gradual deployment differs from replacing the environment entirely
Replacing the environment entirely means a new stack next to the old one and switching a pointer to it: there is almost no downtime, but warmed-up instances and open connections are not transferred, and release rollback is possible only while the old stack is alive and has not diverged from the data schema.
Gradual deployment works differently: two versions live in the same environment, and traffic between them is divided by weights.
Technically it looks like this: the build publishes a new version, the alias continues to point to the previous one, then a controlled weight shift follows — 10%, 25%, 50%, 100% with pauses.
At each pause, automation checks metrics: error rate, latency, number of triggered alarms; exceeding a threshold returns the weights back without rebuilding. Traffic slices provide validation on live requests and a real load profile, which a synthetic test does not.
Serverless models have their own nuances: a new version has no warmed-up instances, the first requests go into a cold start, and growth in the number of concurrent calls runs into account limits.
That is why weights are shifted in small steps and with warm-up, not in a single switch. Compatibility is also critical: if a release changes the message format in a queue or the storage structure, two versions next to each other will not be able to work.
Hence the selection rule. Gradual deployment is the default mode when logic changes but the contract between versions remains the same: release rollback here is returning the weights, seconds and no risk to data.
Environment replacement is for when the set of integrations, data format, or runtime changes and the old version cannot be kept alongside: then rollback means switching the pointer back plus a separate plan for already written data.
Both modes coexist in one deployment loop: gradual for most releases, environment replacement for rare large changes.
Technical concerns we resolve before starting
Before starting, the customer checks not the tool itself but how it will fit the current process: what will happen to already written functions, where secrets will reside, and what will happen when several teams release at the same time.
Below are answers to the most frequently asked questions, which are also the ones that most often slow down the start.
Will existing functions have to be rewritten?
No. The pipeline and infrastructure description templates are wrappers around the code: the same handler, the same dependencies, the same invocation method. Packaging and delivery change — artifact build, checks, staged deployment.
Where are secrets and access keys stored?
In secure storage, not in the repository and not in templates. The pipeline injects them at deployment time, as a separate set for each environment, and rotation happens without editing code.
What happens with parallel releases?
Each branch gets an isolated set of resources, and infrastructure state is locked and recorded during apply. Teams do not overwrite each other's changes, and conflict surfaces at the checks stage, not in the production environment.
How is rollback to the previous version arranged?
A release leaves a versioned artifact and a reference to the previous stable build. Return is switching traffic to it, without rebuilding and manual edits in the console.
Let's discuss the task and come back with a work plan
So that the project estimate and work plan are based on facts rather than assumptions, we need only the initial inputs: how the application is built now, which environments are needed, where secrets are stored, and who is responsible for deployment to production.
Describe the task in free form — the repository, the current method of deploying functions, and requirements for pre-release checks. If some details are not yet formulated, we will work them out together.
In response, you will receive:
- pipeline scheme: build, checks, promotion order across environments, and stop conditions;
- build and deployment configurations for your stack — with separation of shared and environment-specific parameters;
- policy for handling secrets and permissions between the build environment and the cloud account;
- rollback procedure to the previous version, indicating how quickly it is returned to service;
- set of checks before promotion: tests, configuration validation, static analysis;
- project estimate: stages, their sequence, effort, and risks for each;
- work plan with acceptance criteria — what is considered done at each step;
- list of what will be required from your side: access, roles, and the person responsible for prod.
Describe the task — we will come back with a project estimate and work plan showing the stages, dependencies, and control points.