Table of Contents
- The Operator Reflex
- What an Operator Actually Is
- The Reconciliation Loop Is the Hard Part
- When an Operator Earns Its Cost
- When Templating Is Sufficient
- Failure Modes Specific to Controllers
- CRD Design Decisions You Cannot Undo
- Operating the Operator
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: An operator is a distributed system that runs continuously and modifies your cluster. That cost is justified when a system requires ongoing operational decisions — failover, rebalancing, backup coordination. It is not justified to render templates.
The Operator Reflex
A team needs to deploy their application to Kubernetes with some configuration variation across environments. Someone proposes writing an operator.
The reasoning is usually sound-sounding: an operator gives us a clean declarative interface, encodes our operational knowledge, and is the idiomatic Kubernetes approach. All three statements are defensible in isolation.
What frequently goes unexamined is what the operator will actually do. Reading the implementation months later, many operators turn out to render templates and apply the result — which is what a templating tool does without introducing a long-running controller with cluster-wide permissions that your team must now maintain.
The distinction that matters is whether there is ongoing operational logic. An operator’s value comes from continuously observing state and taking corrective action. If the logic runs once at deployment and never again, the continuous part is unused and you have paid for a control loop that does nothing after the first pass.
This article is about telling those cases apart, because the cost difference is substantial and mostly deferred — it appears as maintenance burden rather than as implementation effort.
What an Operator Actually Is
Two components, and both parts matter.
A custom resource definition extends the Kubernetes API with a new object type. After registering it, the API server accepts objects of that type, stores them, and serves them. That is all it does — a CRD alone is a typed record in the cluster’s database with no behaviour.
A controller is a process that watches those objects and acts. It observes the desired state expressed in the resource, observes the actual state of the world, and takes action to reduce the difference. Continuously, forever.
The controller is where all the value and all the complexity live. It is a long-running process, typically with broad permissions, that modifies cluster state based on its own logic. Which makes it, precisely, a distributed system component you have written and now operate.
Understanding this framing clarifies the cost. Deploying an operator means deploying software with cluster-modifying authority that must handle concurrency, partial failures, API rate limits, version skew, and its own upgrades. None of that is exotic and all of it is work that does not exist if you render a template and apply it.
The Reconciliation Loop Is the Hard Part
The reconciliation model is elegant in description and demanding in implementation, because correctness requires properties that are easy to violate.
Idempotency is mandatory. Reconcile is called repeatedly for the same object — on changes, on resyncs, on controller restarts, on unrelated events. Every invocation must be safe. A reconcile that creates something without checking whether it exists produces duplicates on the second call.
Level-triggered, not edge-triggered. The controller must act on current observed state rather than on the event that woke it. Events are lost, duplicated, and delivered out of order. Logic that depends on having seen a specific transition breaks. This is the single most common conceptual error in operator implementation.
Requeue rather than block. If reconciliation cannot complete — a dependency is not ready, an external system is unavailable — the correct response is to return and request another attempt later, with backoff. Blocking inside reconcile consumes a worker and stalls other objects.
Status is a report, not a store. The spec is user intent; status is observed reality. Storing state the controller needs in status creates a dependency on a field that can be stale or overwritten.
// Wrong: assumes it has seen a transition, and blocks
func Reconcile(req) error {
if event.Type == "Added" {
createDatabase() // duplicates on resync
}
waitForDatabaseReady() // blocks a worker indefinitely
return nil
}
// Right: level-triggered, idempotent, requeues
func Reconcile(ctx, req) (Result, error) {
obj := get(req)
db, err := findDatabase(obj) // observe actual state
if db == nil {
if err := createDatabase(obj); err != nil {
return Result{}, err // controller retries with backoff
}
return Result{RequeueAfter: 10 * time.Second}, nil
}
if !db.Ready {
return Result{RequeueAfter: 10 * time.Second}, nil
}
return Result{}, updateStatus(obj, db)
}Getting these properties right is genuinely difficult, and getting them wrong produces failures that appear intermittently under load and are hard to reproduce.
When an Operator Earns Its Cost
The cases where nothing else works, because they require decisions made continuously in response to conditions:
Stateful systems with complex failover. A database that must promote a replica when the primary fails, verify replication state before accepting the promotion, and update service endpoints. This decision cannot be made at deployment time because it depends on runtime conditions.
Coordinated rolling upgrades. Systems where nodes must be upgraded in a specific order, with cluster health verified between steps, and the process halted if health degrades. The sequencing logic is operational.
Backup and restore orchestration. Quiescing writes, taking a consistent snapshot, verifying it, and resuming — with all of it coordinated across replicas.
Dynamic scaling based on domain metrics. Rebalancing shards when nodes join or leave, which requires understanding the data distribution.
Provisioning external resources. Creating a cloud database or message queue to match a resource declaration, with lifecycle tied to the Kubernetes object.
Certificate lifecycle management. Issuing, monitoring expiry, renewing, and distributing — a continuous obligation rather than a one-time action.
The common thread is that a human operator would need to make decisions repeatedly over the system’s life. Encoding that judgement is what an operator is for, and it is genuinely valuable when the judgement is real.
When Templating Is Sufficient
Conversely, the cases where an operator adds cost without capability:
Deploying a stateless application. Deployment, service, ingress, config. Rendered once, updated by the pipeline. There is no ongoing decision.
Environment configuration variation. Different values per environment is what templating and overlay tools exist for.
Setting sensible defaults. Achievable with templates, or with a mutating admission webhook if enforcement is needed — both substantially simpler than a controller.
Validating configuration. A validating admission webhook or a schema on the CRD. Validation does not require a control loop.
Simplifying an interface for developers. A template with a small input surface achieves this. If the only goal is a nicer interface, the operator is doing template rendering with extra steps.
A practical test: describe what the controller does on the second, third, and hundredth reconciliation of an unchanged object. If the honest answer is nothing meaningful, the continuous part is unused, and the value you wanted was the declarative interface — which templating provides.
Failure Modes Specific to Controllers
Controllers fail in ways ordinary applications do not, and these are worth anticipating.
Fighting other controllers. Two controllers managing the same field will each revert the other’s change, indefinitely. This produces continuous API writes and rapidly consumes rate limits. It is a common outcome when an operator manages a field that a platform default also sets.
Runaway reconciliation. A bug causing an object to requeue immediately, forever, produces a hot loop hammering the API server. Exponential backoff on error is the standard protection and it must be present.
Deletion deadlock. A finaliser that cannot complete blocks object deletion permanently. If the finaliser waits on an external system that is gone, the object cannot be removed without manually stripping the finaliser — which surprises users and generates support requests.
Permission scope. Operators frequently request cluster-wide permissions because it is simpler than namespace scoping. That makes the controller a high-value target and a broad blast radius.
Version skew between controller and CRD. A controller expecting fields a stored object lacks, or an older controller encountering newer objects, produces errors that are confusing to diagnose.
Cascading operator dependencies. An operator that requires another operator, which requires a third, produces an installation order requirement and a failure mode where one broken component blocks unrelated work.
CRD Design Decisions You Cannot Undo
The API you expose is the part you cannot change once users depend on it, so a few decisions deserve care.
Version from the start. Begin at v1alpha1 and communicate the stability expectation. Retrofitting versioning after users depend on an unversioned API requires a conversion path you would rather not build.
Keep the spec minimal. Every field is a commitment. Fields are easy to add and effectively impossible to remove. Starting narrow and expanding on demonstrated need produces a better API than exposing everything the underlying system supports.
Never make status writable by users. Status reflects observation. A user-writable status field will be written, and the controller will overwrite it, and someone will report that as a bug.
Use structural schemas with validation. Reject invalid configuration at submission rather than discovering it during reconciliation, where the error surfaces in controller logs nobody reads.
Include observedGeneration in status. It lets users determine whether the controller has processed their latest change, which is otherwise unknowable and is the most frequent source of confusion.
Use standard conditions. A conventional condition structure means existing tooling can interpret your resource’s state without special handling.
That observedGeneration point is small and matters disproportionately. Without it, a user who applies a change has no way to distinguish “the controller has not seen this yet” from “the controller processed it and nothing changed.”
Operating the Operator
An operator is production software and needs the same treatment.
Emit Kubernetes events for significant actions. Users debug by describing their resource. Events are where they will look, and an operator that logs internally while producing no events is opaque to the people using it.
Instrument reconciliation. Duration, error rate, queue depth, and requeue rate per resource type. Rising requeue rate is the signal that something is stuck.
Alert on the controller being down. A stopped controller produces no errors — resources simply stop converging, silently. This is a failure mode with no natural symptom.
Run with restricted permissions. Namespace-scoped where possible, and only the verbs actually used.
Test against a real API server. Reconciliation logic interacts with API semantics in ways mocks do not reproduce. Integration tests against a real control plane catch what unit tests cannot.
Plan the upgrade path. How the controller upgrades, whether stored objects need conversion, and what happens during the window where both versions might run.
Common Pitfalls
Building an operator to render templates. The control loop is unused after the first pass.
Edge-triggered logic. Events are unreliable. Act on observed state.
Non-idempotent reconciliation. Produces duplicates on resync and restart.
Blocking inside reconcile. Consumes workers and stalls unrelated objects.
Finalisers without a failure path. Undeletable objects requiring manual intervention.
Cluster-wide permissions by default. Unnecessary blast radius.
No events emitted. Makes the operator undebuggable by its users.
Conclusion
Operators are the correct tool for encoding operational judgement that must be exercised repeatedly — failover decisions, coordinated upgrades, backup consistency, shard rebalancing. For those problems nothing else works, and a well-built operator is genuinely valuable.
For deploying applications and varying configuration, templating is sufficient and dramatically cheaper. The test is simple: describe what your controller does on the hundredth reconciliation of an unchanged resource. If the answer is nothing, you wanted a declarative interface rather than a control loop.
If you do build one, the properties that determine correctness are idempotency, level-triggered logic, requeue instead of blocking, and status as observation rather than storage. And treat the CRD as a public API from the first version, because the fields you expose are the ones you will support indefinitely.
Frequently Asked Questions
When is an operator the right choice over a Helm chart? When the system requires ongoing operational decisions after deployment — failover, rebalancing, coordinated upgrades, backup orchestration. If the logic runs once at install, use the chart.
Can a CRD be used without writing a controller? Yes, and it is a legitimate pattern for storing structured configuration that other tooling reads. Without a controller, nothing acts on it, which is sometimes exactly what you want.
What breaks most often in operator implementations? Non-idempotent reconciliation and edge-triggered assumptions. Both produce intermittent failures under load that are difficult to reproduce, because they depend on event timing.
Should operators be namespace-scoped or cluster-scoped? Namespace-scoped where the domain permits, since it limits both permissions and failure impact. Cluster-scoped is appropriate for genuinely cluster-wide concerns like certificate management.
Are finalisers necessary? Only when external cleanup must happen before the object disappears. They introduce deletion deadlock risk, so include a timeout or failure path that permits deletion to proceed.
Which framework should be used? The mainstream controller frameworks handle the difficult parts — watches, caching, work queues, rate limiting — and are strongly preferable to writing against the API directly. Language choice matters less than using a framework at all.
How is an operator tested properly? Integration tests against a real API server, exercising the full reconcile path including restart and resync. Unit tests on the logic are useful and do not catch API interaction problems, which is where most bugs live.
