Terraform’s configuration language gets all the attention, but every serious Terraform incident I have worked traces back to state — corrupted, contested, drifted or deleted. Here are five from real engagements, anonymised, with the pattern that would have prevented each.
Disaster 1: The concurrent apply
Two engineers, one module, no state locking — the second apply read stale state and destroyed a database the first had just modified. The fix is boring and absolute: a backend with locking (Azure Blob with lease, S3 with DynamoDB), no exceptions, including "quick fixes" from laptops. If your backend supports locking and you have ever seen the lock error, the system works — the error is the feature.
Disaster 2: The manual portal change
A firewall rule added in the portal during an incident, invisible to Terraform, silently deleted by the next apply four days later — taking down the integration it had been added to fix. Drift detection on a schedule (plan in CI nightly, alert on any diff) turns four days of invisibility into one morning. The deeper fix is cultural: the portal is read-only in production, and incidents that need infrastructure changes get an expedited pipeline path, not an exemption.
# Nightly drift check — alert on any unexpected plan outputname: drift-detectionon: schedule: - cron: '0 5 * * *'jobs: plan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - run: terraform init -input=false - run: terraform plan -detailed-exitcode -input=false -out=drift.plan id: plan continue-on-error: true - name: Alert on drift if: steps.plan.outputs.exitcode == 2 run: ./notify-drift.sh drift.planDisaster 3: The mega-state
One state file for an entire platform: 3,100 resources, four-minute plans, and a blast radius where any mistake anywhere could touch everything. Splitting state along team-ownership and change-frequency boundaries (network rarely changes; app infra changes daily) cut plan time to seconds and made the blast radius legible. The rule of thumb that has survived: if two resources are always changed by different people, they belong in different states.
Disaster 4: The secret in the state
State files store attribute values in plaintext — including the database password a module accepted as a variable. The state backend was readable by forty people. Secrets belong in a vault, referenced at deploy time, never passed through Terraform values; and the state backend deserves the same access control as the secrets it might accidentally contain.
Disaster 5: The deleted backend
A cleanup script deleted "unused" storage — including the state container. Recovery took a weekend of terraform import archaeology. State backends need three protections: deletion locks at the platform level, versioning enabled so any overwrite is recoverable, and replication to a second region. It is a few lines of configuration that converts a weekend into a non-event.