advanced
Rollbacks
Plan fast recovery with versioned artifacts, reversible migrations, health checks, and clear rollback ownership.
Rollback is a designed recovery path, not a panic button. It depends on immutable artifacts, compatible schema changes, traffic controls, observability, and clear ownership during incidents.
Recovery options include:
- **Code rollback** — redeploy previous artifact version
- **Config rollback** — revert feature flags or env settings
- **Data repair** — forward-fix when schema cannot revert
- **Traffic shift** — blue-green or canary reversal
Trigger: error rate > SLO for 5m
Action: route traffic to previous revision + disable flag X
Owner: on-call + release captain
Practice restores before production incidents. Irreversible migrations and external side effects (payments, emails) may require forward-fix instead of revert.
On interviews: distinguish code, config, data, and flag rollback; when forward-fix is safer; rehearsed runbooks.
Common pitfalls: no immutable artifacts to return to; migrations that break old code paths; rollback without verifying health of the previous version.
The trade-off is keeping rollback simple versus shipping schema and API changes that complicate recovery.
Checklist:
- Keep deploy artifacts immutable.
- Use backward-compatible migrations.
- Define rollback triggers and owners.
- Rehearse recovery on staging.