Part 11 of the DevOps Dirty Dozen Series: Semper vigilans — always vigilant.

Insight: Encourages ongoing attention and adaptation.

Automation is not finished when it runs successfully once. Monitoring is not complete when the dashboard loads. A platform is not healthy because the last deployment worked. Systems change. Dependencies move. Credentials expire. Traffic shifts. Teams reorganize. What was safe last quarter can become fragile without anyone touching it directly.

The “set it and forget it” mentality treats operational systems as static. DevOps and SRE work assumes the opposite: systems are alive, and living systems require care.

This anti-pattern is dangerous because it rarely fails loudly at first. It decays quietly. The pipeline still runs, but with old assumptions. The dashboard still renders, but nobody knows whether the queries are meaningful. The alert still exists, but the owning team changed names two reorganizations ago. The automation is present, but the confidence is gone.

A glowing dashboard covered in dust while a warning light blinks in the corner

A dashboard nobody reviews is not observability. It is decoration with a refresh interval.

The Anatomy Of Set It And Forget It

This anti-pattern shows up anywhere a team deploys a mechanism and stops revisiting whether it still does the job.

  1. Forgotten Automation: A pipeline, cron job, or script keeps running for years. Nobody remembers who owns it, what assumptions it makes, or what would break if it stopped.

  2. Stale Monitoring: Dashboards remain online but no longer reflect current architecture. Metrics point at retired services, renamed namespaces, old labels, or incomplete data sources.

  3. Unreviewed Alerts: Alerts fire into channels nobody watches, page teams that no longer own the service, or trigger for conditions that stopped being actionable months ago.

  4. Aging Credentials: Tokens, certificates, IAM roles, kubeconfigs, and service accounts continue to exist because removing them feels risky. Eventually they become security debt.

  5. Static Runbooks: A runbook captures the system as it existed during one incident. The next time it is needed, commands fail because paths, names, APIs, or permissions changed.

  6. Policy Drift: Tagging, encryption, backup, and retention rules are documented once but never verified continuously. Compliance becomes a snapshot instead of a practice.

An automation conveyor belt slowly drifting away from the system it is supposed to manage

Automation does not stay correct by existing. It stays correct by being maintained.

The Cost Of Neglect

The cost of set-it-and-forget-it operations is usually paid during incidents, audits, migrations, and upgrades.

  1. False Confidence: Teams believe a control exists because the artifact exists. The backup job exists. The alert exists. The runbook exists. Nobody has tested whether it works.

  2. Slow Incident Response: During an outage, stale documentation and broken automation waste precious minutes. The team discovers decay at the worst possible time.

  3. Security Exposure: Old credentials and forgotten access paths accumulate. If nobody owns review and cleanup, the environment gets more permissive over time.

  4. Upgrade Fragility: Systems that have not been revisited become harder to upgrade. Unknown dependencies turn routine maintenance into archaeology.

  5. Operational Learning Stops: A system that is never reviewed cannot improve. Teams repeat old patterns because the operating model never gets challenged.

A clean system diagram fading and cracking over time

Operational decay is quiet until the moment it becomes urgent.

A Real-World Example: The Backup Nobody Restored

A team had a nightly backup job for a critical internal application. The job ran successfully for years. Green status. Logs uploaded. Retention configured. Everyone assumed the system was protected.

During a storage failure, the team attempted a restore and discovered three problems at once: the backup included application data but not a required configuration directory, the restore command referenced an old namespace, and the service account used for recovery no longer had the needed permissions.

The backup had not failed. The operational assumption had failed.

The job was created when the application was simpler. Over time, the architecture changed, the namespace changed, permissions changed, and the restore path was never rehearsed. The backup pipeline preserved the appearance of resilience while the actual recovery capability decayed.

The fix was not merely to repair the backup. The fix was to schedule restore tests, assign ownership, document recovery objectives, and track the backup system as production infrastructure.

Why Teams Fall Into The Trap

Set-it-and-forget-it behavior usually comes from pressure, not laziness.

  1. New Work Is More Visible Than Maintenance: Shipping a new pipeline gets attention. Reviewing an old one rarely does.

  2. Success Creates Complacency: If something has worked for a long time, teams assume it will keep working. Longevity becomes mistaken for reliability.

  3. Ownership Gets Blurry: People move teams. Services change hands. Vendors change APIs. The artifact remains, but the ownership model does not.

  4. Maintenance Has No Calendar: If review is not scheduled, it depends on memory. Memory is not an operational control.

  5. Nobody Wants To Touch The Old Thing: Older automation often lacks tests, documentation, and clear rollback. Teams avoid it because changing it feels riskier than ignoring it.

Ownership labels fading from several operational components

When ownership fades, maintenance becomes optional by accident.

Building A Practice Of Vigilance

The antidote is not constant anxiety. It is scheduled, boring, repeatable review.

  1. Assign Owners To Operational Artifacts: Pipelines, dashboards, alerts, secrets, runbooks, backup jobs, and policies need owners just like services do.

  2. Review On A Cadence: Quarterly is often enough for stable systems. High-risk systems may need monthly review. The key is that review is scheduled, not remembered.

  3. Test Recovery Paths: Backups, restores, failovers, rollback steps, and incident runbooks should be rehearsed before they are needed.

  4. Expire What Should Not Live Forever: Credentials, exceptions, temporary firewall rules, feature flags, and elevated permissions should have expiration dates.

  5. Measure Artifact Usefulness: Which dashboards drove decisions? Which alerts resulted in action? Which runbooks were used successfully? Retire or repair the rest.

  6. Treat Automation As Code: Automation should have version control, review, tests where feasible, clear ownership, and deprecation paths.

A calendar with recurring review checkpoints for backups, alerts, credentials, and runbooks

Vigilance becomes sustainable when it is scheduled.

Applying The Scientific Method

Operational vigilance can be treated as a testable practice.

  1. Ask: Which operational artifacts do we rely on but rarely verify?

  2. Hypothesize: “If we run quarterly restore tests, we will find recovery gaps before incidents.”

  3. Test: Pick one backup, one runbook, one dashboard, or one alert group and review it end to end.

  4. Observe: Did it work? Was ownership clear? Were permissions current? Did the artifact still match the system?

  5. Iterate: Update, assign ownership, schedule the next review, or retire the artifact.

Carl Sagan’s Baloney Detection Kit

Set-it-and-forget-it thinking survives on assumptions. Challenge them:

  • “The backup is green.” — Has anyone restored from it recently?

  • “The dashboard exists.” — Who uses it, and what decision does it support?

  • “The alert will page us.” — Does it route to the right team, and is the condition still actionable?

  • “That credential is probably still needed.” — By whom? For what system? When was it last used?

  • “The runbook worked last time.” — Has the system changed since then?

A loop showing configure, operate, verify, update, and repeat

The loop is not complete until the system is verified again.

Moving Forward Together

The strongest operational systems are not the ones that never change. They are the ones that are revisited often enough to stay true.

DevOps is full of useful artifacts: automation, dashboards, runbooks, policies, tests, alerts, backups, templates, modules, and deployment pipelines. None of them remain valuable by default. They remain valuable because teams keep them aligned with reality.

Semper vigilans is not a call to panic. It is a call to responsible stewardship. Build the system, automate the work, document the path — then come back and prove it still works.

What operational artifact is your team trusting because it existed last quarter? When was the last time someone verified it end to end?

References