Table of Contents
- What State Actually Holds
- The Failure Modes Nobody Plans For
- Blast Radius Is a State Decision
- Secrets in State
- Drift and the Limits of Plan
- Modules Without the Abstraction Tax
- Automation That Does Not Corrupt State
- Testing Infrastructure Code
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: State is the authoritative record of which real resource corresponds to which configuration block. Losing or corrupting it does not destroy infrastructure — it makes the infrastructure unmanageable, which is frequently worse because recovery is manual and slow.
What State Actually Holds
State is often described as a cache. That undersells it in a way that leads to poor decisions.
State records the mapping between the resources declared in your configuration and the actual resources that exist. When you declare a database and apply, state records the identifier of the database that was created. On the next run, state is what tells the tool that this configuration block corresponds to that specific database rather than to a new one.
Remove that mapping and the tool concludes the database does not exist. The next apply creates a second one. The original continues running, unmanaged, and now you have two databases and no record of which configuration governs which.
State also holds resource attributes as last observed, dependency relationships, output values other configurations may consume, and — importantly — sensitive values that were part of resource creation.
That combination makes it a production data store with specific requirements: it must be durable, it must not be corrupted by concurrent writes, it must be versioned so you can recover from a bad write, and it must be protected because it contains secrets. Treating it as a file that happens to live somewhere is how the failure modes in the next section occur.
The Failure Modes Nobody Plans For
Concurrent applies corrupting state. Two engineers, or a pipeline and an engineer, applying simultaneously produces interleaved writes. State locking prevents this and must be enabled deliberately — a state file in object storage without a lock mechanism has no protection.
Local state. State on a laptop is unavailable to anyone else, unbacked up, and lost when the machine is. This is fine for experimentation and a serious problem the moment two people are involved.
A partial apply. Interrupted mid-run, some resources are created and recorded, some are created and not recorded, and some are not created. Reconciling this requires understanding exactly what happened, and the tool cannot do it for you.
Manual changes to real resources. Someone modifies a resource through the console. State still reflects the old configuration. The next plan shows a change it intends to make, which may not be what anyone wants.
Resources deleted outside the tool. State says the resource exists. It does not. Depending on the resource type this produces either a clean recreation or a confusing error.
Provider version changes altering behaviour. A provider upgrade can change how a resource is represented, producing spurious diffs or requiring state migration.
Refactoring configuration without moving state. Renaming a resource block, or moving it into a module, changes its address. Without an explicit state move, the tool sees the old resource as removed and the new one as needing creation — which means destroying and recreating a production resource.
That last one is the most common way teams destroy something unintentionally, because the configuration change looks purely cosmetic.
Blast Radius Is a State Decision
How you partition state determines what a mistake can affect, and this is among the most consequential early architectural decisions.
One state for everything. Simple, and every apply touches the whole estate. Plans are slow, lock contention is constant, and a mistake can affect anything. Any team applying anything blocks every other team.
State per environment. The minimum sensible separation. Production is unaffected by staging operations.
State per environment per component. Networking, data stores, compute, and applications separated. Small blast radius, faster plans, and it introduces cross-state dependencies.
State per team per component per environment. Maximum isolation, maximum number of state files to manage, and coordination overhead for changes spanning boundaries.
The trade-off is between blast radius and coordination cost, and most organisations converge on something like per-environment-per-component.
Cross-state references deserve care because they create coupling. Reading another state’s outputs makes your configuration depend on that state’s structure and creates an ordering requirement. The alternative — looking resources up by tag or name through data sources — reduces coupling at the cost of depending on naming conventions being maintained. Neither is clearly better, and the choice should be deliberate rather than accidental.
A separate principle worth holding: infrastructure that changes at different rates belongs in different states. Networking that changes quarterly and application configuration that changes daily should not share a lock.
Secrets in State
State contains sensitive values, and this is frequently overlooked.
Database passwords generated during creation, private keys, connection strings, and any variable marked sensitive still appear in state — marking a value sensitive suppresses it from console output rather than from storage. Resource attributes that happen to be sensitive are recorded as observed.
The consequences: state must be encrypted at rest, access to state must be as restricted as access to the secrets it contains, state must never be committed to version control, and anyone who can read state can read those secrets regardless of what other access controls exist.
Approaches that reduce exposure: generate secrets outside the tool and reference them from a secret manager rather than creating them in configuration. Where the tool must create a credential, write it directly to a secret manager and have applications read it from there. And restrict state access to the automation identity plus a small number of humans, rather than everyone who can read the repository.
The practical test worth applying: list who can read your state backend and ask whether that group should have your production database passwords. The answer is frequently no, and the discrepancy is usually unnoticed.
Drift and the Limits of Plan
A plan compares configuration against state, then refreshes state against reality. Its accuracy depends on what the provider can observe.
Where plan is unreliable:
Attributes the provider does not read back. Some resource properties are write-only or not exposed by the API. Changes to them are invisible.
Resources modified by other systems. Autoscalers adjusting capacity, other automation, or platform-managed defaults produce differences that appear as drift the plan wants to revert.
Computed values. Attributes determined at creation appear as known-after-apply, which conceals whether a change is significant.
Side effects outside the resource model. Data written to a created database, configuration applied inside a provisioned instance, and anything done by a provisioner is not tracked.
The operational habits that matter: read plans carefully rather than approving them, particularly the destroy and replace lines. Run plan on a schedule against production to detect drift before someone discovers it during an unrelated change. Use lifecycle rules to ignore attributes legitimately managed elsewhere, so genuine drift is not buried in expected noise. And require a plan review as an explicit step for production changes rather than applying directly.
A plan showing an unexpected destroy-and-recreate on a stateful resource should stop the process entirely. That output is the tool telling you it is about to lose data, and it appears more often than teams expect — typically from a changed attribute that forces replacement.
Modules Without the Abstraction Tax
Modules enable reuse and are frequently over-applied in ways that add cost without benefit.
Where they earn their complexity: genuinely repeated patterns across environments or teams, encapsulating defaults and conventions, and providing a narrow interface over a complex resource set.
Where they add cost without benefit: wrapping a single resource with no added logic, which produces indirection and nothing else. Deeply nested module hierarchies, where tracing a value through four layers is required to understand a change. Modules with dozens of variables that expose everything the underlying resources support, which is a passthrough with extra steps. And modules used once, where the abstraction serves no reuse.
Practices that keep modules useful: version them and pin the version at each call site, so a module change does not alter every consumer simultaneously. Keep the variable surface small — every variable is a commitment. Prefer composition over nesting. And document what the module creates rather than only how to call it, because the consumer needs to know what they are responsible for.
A useful heuristic: a module should be worth understanding as a unit. If reading its implementation is necessary to use it correctly, the abstraction is not carrying its weight.
Automation That Does Not Corrupt State
Running infrastructure code from a pipeline requires more care than running application deployments.
Plan on the pull request, apply on merge. The plan output is the review artefact — reviewing configuration without the plan means reviewing intent without consequence.
Apply the reviewed plan, not a fresh one. Saving the plan and applying that exact file guarantees you apply what was approved. Re-planning at apply time can produce a different result if anything changed in between.
Serialise applies per state. Concurrent applies against one state are the primary corruption risk. The pipeline must enforce this even where state locking exists, because a lock failure mid-run is worse than queuing.
Never approve your own plan for production. A second reader catches destroy operations the author normalised.
Automation identity separate from human identities. With permissions scoped to what the automation actually manages.
Alert on drift detected by scheduled plans. Drift found by a scheduled run is information; drift found during an unrelated change is a surprise during a change.
Retain plan output. The record of what was intended, alongside the apply result, is what makes incident investigation possible.
Testing Infrastructure Code
Infrastructure testing is genuinely harder than application testing because the thing being tested is external state, and the pragmatic approach is layered.
Static validation. Syntax, formatting, and provider schema validation. Fast, cheap, and catches typos.
Policy checks on the plan. Rules evaluated against planned changes — no public storage, encryption required, no destroy of production data stores, required tags present. This is the highest-value layer because it runs before anything changes and catches the mistakes that matter.
Unit tests on module logic. Verifying that inputs produce the expected planned resources, without applying.
Integration tests in a disposable environment. Actually applying to a real environment, verifying behaviour, and destroying it. Slow and expensive, and the only way to know the configuration works.
Compliance verification against the real environment. Checking deployed reality rather than intended configuration, which catches drift and manual changes.
The policy layer deserves emphasis. A rule that rejects any plan destroying a production database is a few lines of policy and prevents the most expensive category of infrastructure mistake. Most teams have review as the only control at that point, and review is a person reading output under time pressure.
Common Pitfalls
Local state beyond experimentation. Unavailable, unbacked, and lost with the machine.
No state locking. Concurrent applies corrupt state.
Refactoring without state moves. Renaming a block can destroy and recreate a production resource.
Assuming sensitive marking protects state. It suppresses output, not storage.
One state for everything. Slow plans, constant contention, unlimited blast radius.
Applying a freshly generated plan. You apply something other than what was reviewed.
Approving your own production plan. No second reader for destroy operations.
Conclusion
State is the record that makes infrastructure manageable, and it deserves the treatment a production data store receives: remote storage with locking, versioning so a bad write is recoverable, encryption because it holds secrets, and access restricted to match the sensitivity of what it contains.
Partition it deliberately, because the partition determines blast radius. Per-environment is the minimum; per-environment-per-component is where most organisations settle. Keep infrastructure that changes at different rates in different states.
Then put a policy layer in front of applies. A rule rejecting plans that destroy production data stores costs almost nothing and prevents the most expensive mistakes, which are otherwise caught only by a person reading plan output carefully at the end of a long day.
And read the plan. Specifically, read the destroy and replace lines, because that is the tool warning you before it does something irreversible.
Frequently Asked Questions
Where should state be stored? Remote object storage with versioning enabled, encryption at rest, and a locking mechanism. Managed backends that include locking and history are worth the cost for the operational properties.
How should a state file be partitioned? At minimum per environment. Per component within environment is better for blast radius and plan speed. Components that change at very different rates should not share a state.
What happens if state is lost? Infrastructure keeps running and becomes unmanaged. Recovery means importing every resource back into a new state, which is manual and slow. Versioned backends make this largely avoidable.
Are secrets really stored in state? Yes. Marking a variable sensitive suppresses console output only. Encrypt state, restrict access, and prefer referencing secrets from a secret manager over creating them in configuration.
How is a resource renamed safely? With an explicit state move operation, or a move block in configuration where supported. Renaming without moving state destroys and recreates the resource.
Should modules be used for everything? No. Modules earn their cost for genuinely repeated patterns. Wrapping a single resource with no added logic produces indirection without benefit.
What is the most valuable safeguard to add? Policy checks on the plan. Rejecting plans that destroy production data stores, create public storage, or omit encryption prevents the highest-cost mistakes automatically rather than relying on careful reading.
