A Custom Resource Definition Cannot Change Shape Without a Plan
Key takeaway: A CRD schema change applies retroactively to every existing object of that type. Unlike an application deployment, there is no way to migrate custom resources gradually — the schema and the stored data must agree the instant the new version takes effect.
Why This Differs From Ordinary API Changes
Changing a REST API’s request shape affects only future requests; existing stored data is untouched until something rewrites it. A Kubernetes CRD schema change is different because the schema validates objects that already exist in etcd, continuously, on every read and write — there is no grace period where old objects are exempt from a new schema’s constraints.
Add a required field to a CRD without a migration path, and every existing object lacking that field either becomes invalid immediately or is silently treated inconsistently depending on how the conversion is configured. Neither outcome is acceptable for a production cluster with real workloads depending on those resources.
The Versioning Mechanism Kubernetes Provides
CRDs support multiple simultaneous API versions with a designated storage version and conversion webhooks translating between them. This is the mechanism that makes safe evolution possible, and it is more machinery than most CRD authors initially expect to need.
| Approach | Safety | Complexity |
|---|---|---|
| Single version, breaking change | Unsafe — breaks existing objects | Lowest |
| Single version, only additive optional fields | Safe for additions | Low |
| Multiple versions with conversion webhook | Safe for any change | Higher |
| Multiple versions, no webhook, manual migration | Safe if migration completes first | Moderate, risky timing |
Additive optional fields are the easy case and cover a surprising share of real evolution needs — adding a new optional field with a sensible default requires no conversion logic at all, because existing objects remain valid simply by lacking the new field.
Genuine breaking changes — renaming a field, changing a type, restructuring a nested object — require a conversion webhook translating between versions on every read, which is real infrastructure to build, test and operate correctly. Getting this wrong silently corrupts data on conversion, which is a substantially worse failure than a validation error would be.
Planning Before the First Breaking Change
The cost of supporting multiple CRD versions is high enough that avoiding the need is usually better than building the machinery to handle it well. Design the initial schema with room to grow — additional optional fields, extensible structures — specifically to delay the point where a genuine breaking change becomes necessary.
When a breaking change is unavoidable, plan the full deprecation lifecycle before writing the conversion webhook: introduce the new version alongside the old, migrate consumers deliberately, monitor for anything still using the old version, and only remove the old version once nothing depends on it. Removing a version while something still requests it is an outage, not a cleanup step.
Testing Conversion Correctness
A conversion webhook bug corrupts data silently rather than failing loudly, because the webhook’s job is specifically to transform data between shapes — a subtle transformation error looks like success from the API’s perspective. Test round-trip conversion explicitly: convert an object from the old version to the new and back, and assert the result is identical to the original, for every field and every combination of optional fields present or absent.
The Bottom Line
Design initial CRD schemas with room for additive growth to delay the need for breaking changes. When a breaking change is unavoidable, build and thoroughly round-trip test a conversion webhook rather than assuming a schema change is safe, because unlike an application deployment, the change applies retroactively to every object already stored in the cluster.
