Table of Contents
- Git Is Not the Point
- Push Versus Pull Changes the Guarantees
- Drift Is Constant and Mostly Invisible
- Repository Structure Determines Blast Radius
- Secrets Are the Unsolved Part
- Promotion Between Environments
- Rollback Is Not Just Git Revert
- What to Adopt in Which Order
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: GitOps is continuous reconciliation against a declared desired state, with divergence detected and reported. A pipeline that applies manifests from a repository is version-controlled deployment — useful, and not the same thing.
Git Is Not the Point
A team stores Kubernetes manifests in a repository. A pipeline runs on merge and applies them. Someone describes this as GitOps.
It is a reasonable deployment approach and it lacks the property that makes GitOps distinct: the cluster is not being continuously compared against the repository. Between pipeline runs, the actual state can diverge arbitrarily and nothing notices.
The specific mechanism that matters is reconciliation. An agent observes the desired state in the repository, observes the actual state in the cluster, and corrects the difference — continuously, not only when a merge occurs. That loop is what makes the repository authoritative rather than merely historical.
The distinction has practical consequences. With apply-on-merge, someone who patches a deployment directly during an incident has changed production, and that change persists indefinitely, undocumented, until the next deployment overwrites it unexpectedly. With continuous reconciliation, the same patch is reverted within minutes and reported as drift — which is both a safety property and an accurate record of what production actually runs.
Push Versus Pull Changes the Guarantees
Two architectures, with meaningfully different properties.
Push. A pipeline holds cluster credentials and applies changes from outside. This is how most CI-driven deployment works.
Pull. An agent inside the cluster watches the repository and applies changes itself. No external system holds cluster credentials.
The pull model’s advantages are concrete. Cluster credentials never leave the cluster, which removes a high-value secret from the CI system — historically a significant attack path. Reconciliation is continuous rather than triggered. The agent works without inbound network access to the cluster, which suits restricted environments. And multiple clusters can subscribe to the same repository without the pipeline needing credentials for each.
The costs are real too. An agent must be installed and operated in every cluster. Debugging spans the repository, the agent, and the cluster. And the pipeline no longer knows whether a deployment succeeded, since it only merged a commit — deployment status must be observed rather than returned.
That last point catches teams out. In a push model, a failed apply fails the pipeline. In a pull model, the merge succeeds and the deployment may fail silently minutes later, which means deployment status needs its own monitoring rather than being implicit in the build result.
Drift Is Constant and Mostly Invisible
Configuration diverges from its declaration continuously, through mechanisms that are individually reasonable.
Incident intervention. Someone scales a deployment or patches an image during an outage. Correct in the moment, undocumented afterwards.
Other controllers. Autoscalers modify replica counts. Admission webhooks inject sidecars and defaults. Certificate controllers rotate secrets. All legitimate, and all producing state that differs from what the repository declares.
Manual experimentation. A change made to test something and never reverted.
Partial application failures. An apply that succeeded for some resources and failed for others leaves a mixed state that no single place records.
Field defaulting. The API server populates fields the manifest omitted, so the live object never exactly equals the declared one.
The last two categories make naive drift detection noisy, which is why implementations that report every difference get ignored. Distinguishing meaningful drift from expected divergence is the difference between a useful signal and an alert nobody reads.
The practical approach is to declare which fields are managed by other controllers and exclude them from comparison — replica counts where an autoscaler is active, injected containers, and rotated secret data. What remains is drift that indicates something happened outside the declared process, which is the signal you wanted.
Repository Structure Determines Blast Radius
Structure decisions are difficult to change later and directly affect risk.
Application code and manifests in one repository. Simple, and a manifest change triggers the application pipeline unnecessarily. It also means the deployment history is entangled with code history.
Separate configuration repository. Manifests live apart from application code. Cleaner separation, and it requires coordination — a change spanning code and configuration is two pull requests.
One repository per environment. Strong isolation between production and everything else. Promotion means copying between repositories, which is explicit and auditable, and duplicates content.
One repository, directory per environment. Common and convenient. The risk is that a change intended for staging can reach production through a path error, so directory-level access control matters.
Monorepo for all clusters and applications. Excellent visibility and a single point where a mistake affects everything.
A structure that works well for most organisations: one configuration repository, with a directory per environment, where the production directory requires additional review approval. This keeps promotion visible as a diff between directories while making production changes deliberately harder than staging changes.
Whichever structure is chosen, the property to preserve is that reading the repository tells you what is deployed. Structures where the answer requires understanding a rendering process, an overlay chain, and a set of controller behaviours have lost the main benefit.
Secrets Are the Unsolved Part
Storing everything declaratively in Git conflicts directly with not storing secrets in Git.
The available approaches, with honest assessments:
Encrypted secrets in the repository. Encrypt values, commit the ciphertext, decrypt in the cluster. Fully declarative and self-contained. The difficulty is key management and rotation — rotating the encryption key means re-encrypting everything, and the ciphertext history remains in Git forever.
Sealed secrets. Encrypted with a cluster-specific public key, decryptable only by that cluster. Good properties, and secrets become cluster-bound, which complicates disaster recovery to a new cluster.
External secret references. The repository contains a reference; an operator fetches the value from a secret manager at runtime. This is the cleanest separation and the most widely adopted approach. It introduces a runtime dependency on the secret manager and means the repository no longer fully describes the deployed state.
Cluster-native secrets managed outside the flow. Simple and abandons declarative management for the most sensitive configuration.
There is no option here that is both fully declarative and operationally clean. External secret references are the pragmatic default for most teams — accepting that secrets are the documented exception to the repository being complete, in exchange for straightforward rotation and no key material in version control.
Promotion Between Environments
Getting a change from staging to production is where GitOps implementations most commonly become awkward.
Directory copy. Changes are made in the staging directory and copied to production when validated. Explicit, reviewable, and repetitive.
Image tag promotion. Both environments reference the same manifests with different image tags. Promotion means updating a tag. Simple and it assumes configuration is otherwise identical, which it rarely is.
Overlay with environment-specific patches. A shared base with per-environment differences expressed as patches. This handles genuine configuration variation well and makes the effective result harder to read, since you must mentally apply the patches.
Automated promotion on validation. A pipeline that promotes when checks pass. Efficient and requires substantial confidence in the checks.
The principle worth preserving is that promotion should be a reviewable diff. If promotion happens through a mechanism where nobody sees what changed, the audit trail that GitOps provides has been given up for convenience.
A related discipline: pin image tags to digests rather than mutable tags. A deployment referencing latest or a moving version tag is not declaratively specified, because the same commit produces different results at different times. Digest pinning is what makes the repository state correspond to an actual deployable artefact.
Rollback Is Not Just Git Revert
Reverting a commit and letting reconciliation apply it works for straightforward changes and fails for several common cases.
Database migrations. Application code reverts, the schema does not. A revert can leave code expecting the old schema against a migrated database. This requires migration design that tolerates both versions rather than a deployment mechanism fix.
Stateful workloads. Reverting a StatefulSet change may not restore prior state, and some changes to persistent volume claims cannot be reversed in place.
Deleted resources. Reverting a commit that removed a resource recreates it as a new object, losing whatever state the original held.
Non-reversible external effects. Anything the deployment triggered — messages published, webhooks called, third-party state changed — is unaffected by a revert.
Reconciliation ordering. A revert applies changes in whatever order the controller processes them, which may not be the reverse of how they were applied.
The practices that make rollback dependable: design changes to be backward compatible, so old and new versions can coexist. Separate schema migrations from application deployment, applying expand-migrate-contract so each step is independently safe. Test rollback deliberately rather than assuming it works. And use progressive delivery — canary or blue-green — so a bad change affects a fraction of traffic and rolling back means shifting traffic rather than reverting state.
What to Adopt in Which Order
A sequence where each step is independently valuable:
Manifests in version control, applied by a pipeline. Establishes review and history. Most teams have this.
Digest-pinned images. Makes the declared state correspond to specific artefacts.
A reconciliation agent in the cluster. Converts apply-on-merge into continuous reconciliation. This is the step that makes it GitOps.
Drift detection with sensible exclusions. Report divergence, excluding fields other controllers legitimately manage.
Automated sync with a manual override. Reconciliation applies changes automatically, with the ability to pause during incidents.
Health assessment before declaring success. A deployment that applied is not necessarily a deployment that works. The agent should evaluate readiness.
Progressive delivery. Canary analysis before full rollout, once the basics are reliable.
The third step is the pivotal one. Everything before it is version-controlled deployment; everything from it onward has the property that the repository is authoritative and divergence is visible.
Common Pitfalls
Calling apply-on-merge GitOps. Without continuous reconciliation, the repository is not authoritative between deployments.
Drift detection without exclusions. Noise from autoscalers and injected defaults trains people to ignore it.
Mutable image tags. The same commit produces different deployments at different times.
Secrets in the repository unencrypted. The most common serious mistake in GitOps adoption.
Assuming revert equals rollback. Migrations, state, and external effects do not revert.
No deployment status monitoring in pull mode. The merge succeeds and the deployment may fail silently.
Directory structure without access control. A path error becomes a production change.
Conclusion
GitOps is continuous reconciliation with visible drift, not manifests in a repository. The distinguishing property is that an agent constantly compares actual state against declared state, which makes the repository authoritative and makes undocumented changes both temporary and reported.
Adopting it well means pinning images to digests so the declared state is unambiguous, structuring the repository so reading it tells you what is deployed, excluding controller-managed fields from drift comparison so the signal stays useful, and monitoring deployment status separately because the pipeline no longer reports it.
Secrets remain the genuine exception. External secret references are the pragmatic answer, accepting that the repository is complete except for the values you deliberately keep out of it.
And treat rollback as a design problem rather than a deployment feature. Reverting a commit reverts manifests; it does not revert migrations, persistent state, or anything already sent elsewhere.
Frequently Asked Questions
Is GitOps only for Kubernetes? The pattern generalises to anything with a declarative API and a reconciliation loop, and the mature tooling is Kubernetes-focused. Cloud infrastructure GitOps exists and is less mature.
Should application code and manifests share a repository? Separate repositories give cleaner separation and require coordinating two pull requests for changes spanning both. A single repository is simpler for small teams and couples deployment history to code history.
How should secrets be handled? External secret references from a secret manager, for most teams. Encrypted-in-repository approaches are fully declarative and make rotation harder. Neither is fully satisfying.
What happens when someone changes the cluster directly? With continuous reconciliation, the change is reverted within the sync interval and reported as drift. This is the intended behaviour and it needs communicating, because engineers who patch during incidents will be surprised.
Can reconciliation be paused during an incident? Yes, and the mechanism should be explicit and documented. Pausing to permit emergency manual changes is legitimate; forgetting to resume afterwards is the common failure.
Does GitOps slow down deployment? The reconciliation interval adds latency between merge and deployment, typically seconds to minutes. Most implementations support webhook-triggered sync for immediate application when needed.
How is a failed deployment detected in pull mode? The agent’s health assessment, exposed as metrics and alerts. This must be configured deliberately — unlike a push pipeline, nothing fails visibly when the deployment does.
