Twelve months ago we turned on unattended remediation for a 14,000-seat customer. Not "the bot suggests, a human clicks" — genuinely unattended: detection, diagnosis, action, verification, closure. The number everyone asks about is 71% of incident volume closed without a human touch. The more interesting numbers are the ones underneath it.
This post is the honest ledger: three categories where automation outperformed our expectations, two where it failed in ways that taught us something, and the four guard rails we now consider non-negotiable.
Where it worked better than expected
Disk exhaustion, service restarts and certificate renewals were the obvious wins — deterministic diagnosis, reversible actions, easily verified outcomes. What surprised us was password lockout storms. The pattern "one user, five lockouts, three sources" has a diagnosis tree only four levels deep, but a human takes eleven minutes to walk it because the evidence lives in three consoles. The playbook walks it in nine seconds, and it never forgets to check for the stale credential on the forgotten mobile device — which is the answer 60% of the time.
The second surprise was how quickly the auto-closed incident data improved our problem management. When the same playbook fires forty times a month against the same asset group, that is a problem record writing itself. Our top ten recurring root causes now come with frequency data nobody had to collate.
Where it failed and what it cost
Failure one: a remediation loop. A service restart playbook fixed a symptom whose cause was a memory leak, so it fired every 40 minutes for two days — technically "resolving" 70 incidents while masking a degradation that a human would have escalated after the second occurrence. The fix was a recurrence breaker: the same playbook firing against the same CI more than N times in a window now halts automation and forces an escalation with the full history attached.
Failure two: a verification gap. A disk cleanup playbook reported success after freeing space — but on one server class, the space it freed was immediately consumed by the very log spew that caused the alert. The playbook verified the wrong thing (free space now) instead of the right thing (free space stable). Verification steps now have to assert a condition holding over time, not a snapshot.
The four non-negotiable guard rails
- Recurrence breakers on every playbook — automation that fires repeatedly is evidence of a missed root cause, and it must escalate rather than continue.
- Time-held verification — success means the condition holds for a defined window, not that a command returned zero.
- Blast-radius ceilings — a playbook may touch one CI freely, a service group with approval, and never an entire class in one execution.
- A kill switch that on-call actually knows about — one command that halts all unattended execution, tested monthly like a fire drill.
The metric that matters
We stopped celebrating "tickets auto-closed" in month three. The metric that predicts customer satisfaction is human-touched incident quality: when automation handles the routine 71%, the remaining 29% get engineers with time to think. MTTR on the hard incidents — the ones automation escalates — dropped 41% over the year, and that is the number the CIO renews on.
Automation did not replace our engineers. It replaced the part of their day that was beneath them.