Table of Contents
- The Objection Everyone Raises
- What a Cold Start Actually Costs
- Connection Pooling Does Not Work
- The Local Development Gap
- Cost Inversion at Sustained Load
- Where Serverless Genuinely Wins
- Edge Functions Are a Different Trade-Off
- Designing for the Execution Model
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Cold starts affect a small fraction of requests and are mitigable. The problems that actually cause serverless projects to struggle are database connection exhaustion, the difficulty of local testing, and cost that exceeds containers once load becomes steady.
The Objection Everyone Raises
Propose serverless functions and the first objection is cold starts. It is the best-known limitation and, for most workloads, among the least consequential.
The reasoning behind the concern is sound in outline. A function with no warm instance requires the platform to provision an execution environment, load the runtime, initialise your code, and only then handle the request. That adds latency.
What the concern usually omits is the frequency. Under steady traffic, most requests reach an already-warm instance. Cold starts occur on the first invocation, during scale-up beyond current capacity, and after idle periods. In a service handling continuous traffic, the affected proportion is small.
Meanwhile, the difficulties that genuinely derail serverless adoption receive less attention because they are less quotable: a function that scales to two thousand concurrent executions opens two thousand database connections and exhausts the database. Local development requires either emulating the platform imperfectly or deploying to test. And the per-invocation pricing that was inexpensive at low volume becomes more expensive than a container once load is steady.
Those three problems are architectural rather than latency-related, and they are what this article is about.
What a Cold Start Actually Costs
Approximate figures, useful for reasoning rather than as precise predictions:
| Runtime | Typical cold start | Notes |
|---|---|---|
| Compiled native binary | 50–150 ms | Minimal initialisation |
| Node.js / Python, small | 150–400 ms | Dependent on dependency count |
| Node.js / Python, large deps | 400 ms–1.5 s | Import time dominates |
| JVM / .NET | 1–4 s | Runtime and JIT initialisation |
| Any runtime, in a VPC | add 0–1 s | Historically severe, much improved |
| Container image, large | 1–5 s | Image pull and extraction |
What actually drives the number, in order: the size of your deployment bundle and the time spent importing dependencies, the runtime’s own startup cost, and any initialisation your code performs before it can handle a request.
Mitigations, ordered by effectiveness per unit of effort:
Reduce bundle size. Tree-shaking, removing unused dependencies, and bundling rather than shipping a full dependency directory. This is frequently the largest available improvement and it costs nothing at runtime.
Move initialisation out of the request path. Anything expensive should happen at module load, outside the handler, so it is paid once per instance rather than per request.
Provisioned concurrency. Keeping instances warm eliminates cold starts entirely, at the cost of paying for idle capacity — which is a partial return to the server model you were avoiding.
Choose a faster runtime for latency-sensitive paths. A compiled language starts in a fraction of the time a JVM does.
Lazy-load infrequently used dependencies. Import inside the branch that needs them rather than at module top level.
The reason cold starts are not the main problem is that this list is short, well-understood, and effective. The problems in the following sections do not have equivalently tidy solutions.
Connection Pooling Does Not Work
This is the difficulty that most frequently causes serverless projects to fail, and it follows directly from the execution model.
Traditional applications maintain a connection pool — a small number of database connections shared across many concurrent requests. Ten application instances with twenty pooled connections each is two hundred connections total, which most databases handle comfortably.
Serverless inverts this. Each concurrent execution is an isolated environment with its own connections. A function scaling to a thousand concurrent executions attempts a thousand connections. Traditional relational databases exhaust their connection limits well below that, and each connection carries meaningful memory overhead on the database server.
Worse, the failure mode is a cascade. Connection exhaustion causes errors, errors cause retries, retries cause more invocations, which attempt more connections. The database becomes the bottleneck precisely when traffic is highest.
Approaches, with honest trade-offs:
A connection proxy. A managed service maintaining a pool that functions connect through. This is the standard answer for relational databases and it adds a component, a cost, and a small latency increase.
HTTP-based data APIs. Some managed databases offer a stateless HTTP interface, which removes connection management entirely. Latency is typically higher per query.
Serverless-native databases. Data stores designed for high connection counts or stateless access. The best fit architecturally, and it constrains your database choice.
Reuse connections across invocations. Declaring the client outside the handler means a warm instance reuses its connection. This helps and does not bound total connections, since concurrency still multiplies.
Concurrency limits on the function. Capping maximum concurrent executions protects the database and means requests queue or fail under load. It is a real control and it trades availability for stability.
There is no option here that is simply free. This is the genuine architectural cost of the model.
The Local Development Gap
Serverless meaningfully degrades the development loop, and the degradation is felt continuously rather than occasionally.
The specifics: platform emulators approximate the cloud environment and differ in ways that matter — identity and permissions, event payload shapes, timeouts, and concurrency behaviour. Integration with managed services frequently has no local equivalent, so testing requires the real thing. Debugging a function that only fails under production concurrency requires production. And a deploy-to-test cycle, even a fast one, is dramatically slower than a local restart.
What teams do about it, in roughly increasing order of investment:
Structure code so business logic is independent of the platform. The handler becomes a thin adapter that extracts inputs and calls a plain function. That plain function is testable locally at speed, which recovers most of the lost velocity.
Per-developer cloud environments. Each engineer deploys to their own isolated stack, with tooling that syncs changes quickly. This is the approach most mature serverless teams converge on.
Contract tests against emulators, integration tests against real services. Accepting that emulators verify shape rather than behaviour.
Structured logs and tracing from the start. When local reproduction is difficult, production observability substitutes for a debugger, and it must be built in rather than added during an incident.
The first item is the highest-leverage and the most frequently skipped. A codebase where business logic is entangled with platform-specific handler code cannot be tested without the platform, which makes every test slow.
Cost Inversion at Sustained Load
Serverless pricing is compelling at low and variable volume and inverts at sustained high volume. Knowing roughly where the crossover sits prevents an expensive surprise.
The mechanism: you pay per invocation and per unit of execution time. That is excellent when the alternative is a container idling at three percent utilisation. It is poor when the alternative is a container running at seventy percent utilisation continuously, because the container’s fixed cost is amortised across far more work.
Rough orientation rather than precise figures:
| Traffic pattern | Typical cheaper option |
|---|---|
| Sporadic, unpredictable | Serverless, substantially |
| Business hours only | Serverless |
| Steady moderate load | Roughly comparable |
| Steady high load, always on | Containers, often by several times |
| Very spiky with high peaks | Serverless, for the peak absorption |
Secondary cost factors that surprise teams: memory allocation is billed alongside duration, and over-allocating memory to reduce duration sometimes reduces total cost and sometimes does not — it needs measuring rather than assuming. Data transfer and the cost of the connection proxy add to the total. And provisioned concurrency, if used to eliminate cold starts, reintroduces the idle cost that serverless was avoiding.
The productive framing is that serverless is a bet on variable load. Where load is genuinely variable, the bet pays. Where load is steady, you are paying a premium for elasticity you do not use.
Where Serverless Genuinely Wins
The cases where the model’s properties align with the problem:
Event-driven processing. Reacting to storage uploads, queue messages, or stream records. The execution model matches the workload shape exactly.
Scheduled tasks. Paying only for the seconds a nightly job runs, rather than for a machine that exists to run it.
Highly variable traffic. Absorbing a hundred-fold spike without capacity planning is genuinely difficult to replicate with containers, and it is where serverless is most clearly superior.
Glue between managed services. Small transformations and routing logic where operating a service would be disproportionate.
Low-volume internal tooling. Endpoints called a few times daily, where a container’s idle cost dominates.
Rapid prototyping. Deploying without infrastructure decisions is a real velocity advantage early.
Where it is a poor fit: long-running work exceeding execution limits, workloads needing persistent connections such as websockets, anything requiring substantial in-memory state between requests, steady high-throughput services, and applications with strict and consistent latency requirements where cold start variance is unacceptable.
Edge Functions Are a Different Trade-Off
Edge runtimes deserve separate treatment because they solve a different problem with different constraints.
What they provide: execution geographically near the user, which removes network round-trip latency, and startup times low enough that cold starts are largely a non-issue.
What they cost: a restricted runtime, typically a web-standard environment rather than a full server runtime, so many libraries do not work. Tight limits on execution time and memory. No persistent connections, and frequently no direct database access — which means the connection problem is worse rather than better. And a distributed data problem, because a function running in thirty locations reading from one central database has replaced network latency to your server with network latency to your database.
The workloads that fit are specific: request modification and header manipulation, authentication and authorisation checks against a token, routing and rewriting, personalisation from data cached at the edge, and lightweight response transformation.
The pattern that works well is edge functions handling the request-path concerns that benefit from proximity, delegating anything requiring data to a regional service. Treating edge functions as a general compute platform runs into the runtime and data constraints quickly.
Designing for the Execution Model
Properties to build in rather than discover:
Idempotent handlers. Platforms retry on failure, and at-least-once delivery is the norm for event sources. A handler that is not safe to run twice will eventually cause a duplicate.
Initialisation outside the handler. Clients, configuration, and connections created at module scope are reused by warm instances.
No reliance on in-memory state between requests. Instances are recycled without notice and requests are not affinity-routed.
Explicit timeouts on every outbound call, shorter than the function timeout. A hanging call that consumes the full function timeout produces a platform-level timeout with no useful error.
Idempotency keys on downstream writes. Combined with retries, this is what makes at-least-once delivery safe.
Dead letter queues on every async invocation. Without one, failed events disappear silently, which is a data loss mode rather than an availability one.
Business logic separated from the handler. For testability, and because it makes migrating off the platform possible if the cost inversion arrives.
Common Pitfalls
Optimising cold starts while ignoring connections. The connection problem causes outages; cold starts cause latency.
Direct relational database connections at scale. Exhausts the connection limit and cascades under load.
Business logic inside handlers. Makes local testing effectively impossible.
No dead letter queues. Failed async events vanish without trace.
Non-idempotent handlers. Retries are part of the model, not an exception.
Assuming serverless is always cheaper. It inverts at sustained load, sometimes dramatically.
Treating edge functions as general compute. Runtime restrictions and data locality bite quickly.
Conclusion
Cold starts are the best-known serverless limitation and among the least important. They affect a minority of requests, and reducing bundle size plus moving initialisation out of the request path addresses most of the impact.
The problems worth planning for are different. Connection management is the one that causes outages, because per-execution isolation multiplies connections in a way relational databases cannot absorb — a proxy, an HTTP data interface, or a concurrency cap is required rather than optional. Local development degrades meaningfully, and separating business logic from handlers is what recovers the lost velocity. And cost inverts once load becomes steady, which means the model is a bet on variability.
Where load is genuinely variable and the work is event-shaped, serverless is a strong fit and the operational savings are real. Where load is steady and high, you are paying a premium for elasticity you are not using.
Frequently Asked Questions
Are cold starts still a significant problem? Less than commonly assumed. Under steady traffic they affect a small share of requests, and bundle size reduction plus moving initialisation out of the handler addresses most of the impact. Provisioned concurrency eliminates them at the cost of paying for idle capacity.
How should database connections be handled? A connection proxy for relational databases, an HTTP data interface where available, or a database designed for high connection counts. Reusing the client across warm invocations helps and does not bound total concurrency.
When do containers become cheaper? Roughly when load becomes steady enough that a container would run at reasonable utilisation continuously. The exact point depends on invocation volume, duration, and memory allocation, so it is worth modelling with your actual numbers.
Can a serverless application be tested locally? Business logic, yes, if it is separated from the handler. Full platform behaviour, not reliably — emulators differ in identity, event shapes, and concurrency. Per-developer cloud environments are the common answer.
What execution duration is appropriate? Short. Functions running for minutes are usually better as containers or as a workflow orchestration of shorter steps. Long-running functions also amplify the connection and cost concerns.
Is vendor lock-in a genuine concern? Moderately. The handler signature and platform integrations are specific, and the business logic need not be. Keeping logic in plain functions makes the platform layer thin and replaceable.
Should serverless be used for a public API? It can work well, particularly with variable traffic. Budget for the connection proxy, measure the latency distribution including cold starts against your requirements, and model the cost at projected volume before committing.
