GCP Cloud Functions setup: where the production environment breaks
GCP Cloud Functions setup usually goes smoothly in a demo and breaks in the production environment.
A function is deployed under a service account with overly broad permissions — that is faster at the start — and a couple of months later no one can say which buckets and tables it has access to.
An attempt to narrow the service account's permissions before an audit stops order processing: some calls were going through it.
The second typical failure is secrets in environment variables. The database key and the external service token sit next to ordinary settings: they are visible in the console, end up in configuration dumps and logs, and rotation requires rebuilding and redeploying.
Alongside that is the function's cold start: with infrequent traffic, the environment shuts down the instance, and the first request after a pause does not fit within the client's timeout.
A separate category is event retries. A handler without idempotency charges money twice or sends a second email when a message is redelivered.
And a silent timeout of an external call often looks like successful completion: the code returns an acknowledgement without waiting for a response.
We explain how this is closed — with minimal roles, secrets in a separate vault, instance warming, and handlers that are safe to repeat.
What changes after implementation: measurable effects
A function as a delivery unit removes a layer of infrastructure work: there are no servers, OS updates and security patches are handled by the platform, and the team deals only with code and its behavior.
The effect of such a transition is measured in four quantities — function response time, execution error rate, deployment time, and the class of eliminated failures. In Google Cloud, they are visible in the service metrics within the first day after traffic switching.
Response time is determined primarily by starting new instances under load. We keep a minimal set of warmed instances, move initialization of clients to databases and external services outside the handler, and limit concurrency per instance.
Under typical load, the hot path fits within tens of milliseconds instead of seconds, and growth in the number of requests does not turn into growth in latency.
The execution error rate falls because secrets no longer live in environment variables and repository files: they are issued to the function on the fly and rotated without rebuilding.
Access is built on roles rather than static keys, so misconfiguration is caught at the deployment stage, not in production. The share of unsuccessful executions drops from percent levels to tenths of a percent.
Deployment time is reduced to a single pipeline operation, and the new revision receives traffic gradually — first a portion of requests, then the rest.
If response or error metrics exceed the threshold, returning to the previous revision takes less than a minute: traffic is simply switched, and the code does not need to be rebuilt. Service recovery time is reduced from tens of minutes of manual actions to a single operation.
A separate class of failures also goes away: those related to credential leakage through build artifacts and logs, and failures caused by manual configuration edits on the server.
Secret rotation stops being an event that halts the service, and access for contractors and new engineers is granted by role for a specific function — and revoked the same way.
Function deployment modes: comparison of approaches
The deployment mode determines who is responsible for the runtime, how permissions are granted, and how the function receives secrets. This affects cold start time, the available set of dependencies, and where to look when troubleshooting a failure.
| Option | When it fits | Limitations |
|---|---|---|
| Fully managed execution | Typical handlers: HTTP calls, queue events, object uploads to storage | Fixed set of runtime and system library versions; access to secrets only through a service account and Secret Manager; call time limit |
| Custom function image | System packages are needed, or a shared image with other services | You maintain the build and updates of the base layer; the image is heavier — cold start is longer |
| Hybrid with always-on services | Heavy warm-up, database connections, shared libraries; the function is a thin layer in front of the service | Two environments for monitoring and rollback; the network call adds latency; an internal address and token are required |
Trigger types are not rigidly tied to the mode: an HTTP endpoint, a queue subscription, a storage event, and a scheduled call work both in managed execution and in a custom image.
Only the source changes: in a hybrid setup, the function is more often called by an internal service rather than an external client.
Switching between modes almost always means repackaging the entry point, not rewriting the logic. IAM bindings and secrets are transferred as is if access to them goes through a unified configuration layer.
Work stages: from configuration audit to handover
The work proceeds in a fixed order — from inventory to handover to the operating team. The order is not accidental: the role and access policy scheme is designed before secret configuration; otherwise, service account bindings have to be redone twice. Below are six stages, each with a verifiable result.
- Inventory and configuration audit. We collect all functions, triggers, environments, and external dependencies, and note where permissions are granted directly to people and secrets are stored in environment variables. The output is a map: what we migrate, what we rewrite, and what we decommission.
- Role and access policy scheme. For each function — a separate service account with a minimal set of roles; service-to-service calls are authenticated, and only endpoints where this is a deliberate decision remain public.
- Secret configuration and rotation. Keys and tokens move to Secret Manager with versioning, and the function accesses the required version according to its account's permissions. Rotation becomes a planned procedure rather than an emergency replacement across the entire project.
- Load testing. We run scenarios with cold start, parallel calls, and external dependency failure: we look at response time, retries, and behavior when quotas are exhausted.
- Documentation. We record the access scheme, the list of secrets with owners, the rollout and rollback procedure, and known quota and timeout limitations.
- Handover. We walk through the configuration with the customer's team, hand over access and procedures, and stay in touch during the period of independent operation.
A stage is closed not when deployment happens, but when your team can add a function, grant it permissions, and update a secret without our help.
What the customer retains after environment handover
Environment handover is not a console walkthrough, but a set of files and procedures by which your team can roll out a release, roll it back, and handle an incident on an on-call shift without us.
Below is what remains in the repository and internal knowledge base; each item is checked during acceptance.
- Infrastructure configurations as code: functions, triggers, environment variables, and secret bindings — the environment is brought up from the repository, not manually.
- Role and access matrix: for each service account, it is visible which calls it initiates, who is allowed to deploy and read secrets; excessive permissions are marked for revocation.
- Secret rotation procedure: frequency, responsible person, and the order in which the new version is enabled before the old one is disabled — without downtime or code changes.
- Integration map: queues, schedules, external calls, and failure points — with a note on which component degrades first and what the user sees.
- Infrastructure documentation: projects, quotas, limits, environment variables, and service connections — on one page, which a new engineer can use to get up to speed on the environment in a day.
- On-call instructions: typical alerts, where to look at logs and metrics, how to roll back the latest release, and whom to escalate to.
- Rollback plan: retention policy for previous versions and recovery time to a working state confirmed during drills.
The items are interconnected: the role matrix references the rotation procedure, and the on-call instructions reference the integration map and rollback plan. If something cannot be reproduced without us, the acceptance item is not closed.
Case study: migrated a monolith to functions without downtime
The client came with a monolith on virtual machines: deployment once every two weeks, manually, following a wiki instruction. Deployment time was about 40 minutes, and every release came with a downtime window: some requests received errors while processes were warming up.
Secrets were stored in shared environment variables and were visible to all services at once — one typo in permissions opened access to someone else's data.
How we migrated
We split the monolith by domains and moved each into a separate function behind a shared router: the old handlers remained operational while traffic was switched in increments — 5, 25, 50, 100%.
Permissions were distributed by service account: each function has only the roles it needs to operate, and keys and tokens moved to Secret Manager with versioning and rotation.
The infrastructure was deployed on Google Cloud and described declaratively — a new environment comes up with a single command, without manual configuration.
What we achieved
Deployment time was reduced from 40 minutes to 4: rollout is automatic, without manual steps or waiting at the console.
The error rate per release dropped from 3.1% to 0.4% — most failures were caused by shared keys and discrepancies between environments, which disappeared along with manual configuration.
Service downtime during migration was zero: switching was done by traffic increments, and rollback returns the previous function version in seconds.
How do serverless functions differ from always-on services?
The key difference is when the process that handles a request appears. A serverless function is created for a specific call and terminates with it: the platform itself decides whether to keep the instance warm or release it for the next one.
An always-on service keeps the process running continuously — connections, in-memory cache, and background tasks live between requests.
Latency predictability diverges first. For functions, a call after a pause goes through a cold start: environment initialization, reading secrets from Secret Manager, permission checks via IAM.
Most responses are fast, but individual calls stand out by multiples, and for interactive scenarios this is noticeable to the user. An always-on service warms up once, and after that response time stays within a narrow corridor.
Persistent connections are the second point of divergence.
A database pool, bidirectional exchange with a client, and long-lived data streams require the process to live longer than a single call; a function does not retain such state, and it is moved to an external layer — as the number of requests grows, this adds latency and failure points.
The cost of idle time is mirrored: functions are paid for actual work, while an idle service is paid for reserved capacity. On Google Cloud, both models live in the same project, so the architecture is more often assembled as a hybrid.
| Option | When it fits | Limitations |
|---|---|---|
| Serverless functions | Event-driven tasks: file processing, notifications from external systems, infrequent requests | Cold start, no long-lived state |
| Always-on service | Interactive interfaces, persistent connections, consistent response | Paying for reserved capacity even without traffic |
| Functions with a warmed minimum of instances | Traffic spikes with a strict response requirement | Warm-up is paid for continuously |
The choice is not based on the principle of 'what is more modern,' but on the load profile: how often calls arrive, whether there are peaks, and whether long connections are needed.
Technical concerns before production rollout
Before production rollout, the customer brings a list of accumulated questions — from access permissions to platform behavior under load. Below are five that most often determine whether the release will go smoothly, along with our brief answers to each.
The general principle is the same: permissions and secrets are described declaratively and live outside the function code, so changing them does not require rewriting the logic or rebuilding the artifact.
We have accumulated outdated roles — will they have to be rewritten from scratch?
We start with an inventory: we collect which permissions are actually used and migrate service accounts to new bindings through groups. We remove excess roles in stages — first we review the access log, then revoke. This way, permission cleanup does not turn into an outage.
Why can't keys be kept in environment variables?
Environment variables end up in the revision description and build logs — anyone with view permission will see the secret. We move the values to Secret Manager and leave only the secret name and the version the function reads in the variables.
Will there be downtime when permissions change?
Role bindings are applied to running instances; there is no need to stop the service. Downtime arises elsewhere: if a function reads a secret once at startup and caches it, revoking the old key before the cache expires creates a window of failures.
We create an overlap — the new secret is available in advance, and the old one is disabled after all instances have re-read the value.
How does key rotation work without rolling out a new version?
Secrets store versions, so rotation amounts to adding a new version and switching the pointer — the code does not change. Rollback is the same: we return the pointer to the previous version, and the function works with the old value again while we investigate the cause.
Which platform limitations will show up under load?
There are limits on request execution time, the number of concurrent instances, image size, and cold start latency.
They are checked on a staging environment under a load profile close to production: then hitting a limit is visible in advance, and it is addressed by tuning concurrency or splitting the task.
Let's discuss the task: audit, design, and implementation plan
An estimate over chat almost always gives a wide range, so we start with facts: function configurations, the list of triggers, granted roles, and how secrets are currently stored and rotated.
That is enough to see bottlenecks — excessive permissions, values in environment variables, and lack of separation between environments.
- Current function and trigger configurations — we will compare them with the actual load, find duplicates and forgotten instances.
- List of accounts and roles — we will show where access is broader than necessary and reduce it to the minimum required.
- Secret storage and rotation scheme — we will check that values do not end up in logs, build code, or change history.
- Environment constraints: change windows, rollback requirements, and who is on call and how during rollout.
- Audit of the current configuration: a report listing risks sorted by likelihood and cost of failure.
- Implementation plan: stages, order of changes, checkpoints, and what can be rolled out without downtime.
- Timeline estimate: a range by stage plus the conditions under which it narrows or, conversely, grows.
- Knowledge transfer: documentation, acceptance runs, and a walkthrough with your team using its own examples.
We return not a report for the drawer, but working configurations, role schemes, and rotation instructions — so that you can support the solution further on your own. Describe the task and the current state — we will come back with an estimate and a plan.