Part 7 of the DevOps Dirty Dozen Series: Nullius boni sine socio iucunda possessio est — no good is enjoyable without a companion.

Insight: Points to the value of teamwork over individual heroics.

Every engineering organization has a story about the person who saved production at 2 AM. They knew the undocumented system. They remembered the one flag. They could read logs nobody else understood. They jumped in, fixed the outage, and became the name everyone invoked the next time things went sideways.

That person may be talented. They may be dedicated. They may have prevented real damage. But when an organization repeatedly depends on that person, it is not witnessing excellence. It is exposing fragility.

Hero culture is what happens when operational success depends on exceptional individuals instead of sustainable systems. It feels reassuring in the moment because someone always seems able to save the day. Underneath, it is a slow-moving failure mode: knowledge concentrates, teams disengage, risk hides, and the hero burns out.

One engineer standing under a bright spotlight while a tired team fades into the background

If one person is always under the spotlight, the system is already telling you where it is fragile.

The Anatomy Of Hero Culture

Hero culture rarely starts with bad intent. It often starts with competence. Someone knows the system better than everyone else. They respond quickly. They care deeply. Leaders trust them. Teams route hard problems to them because it works.

The pattern becomes dangerous when that individual capability replaces team capability.

Hero culture looks like this:

  1. The Same Names In Every Incident: The incident bridge starts with ten people and ends with two people actually doing the work. Everyone else watches because the real knowledge lives in a few heads.

  2. Undocumented Fixes: The hero fixes the issue faster than anyone can document it. The incident closes. The system remains just as mysterious as before.

  3. Escalation By Personality: Teams do not escalate to a role, runbook, or owning team. They escalate to a person. “Call Alex” becomes the operational model.

  4. Praise For Rescue, Silence For Prevention: The engineer who restores service gets visible recognition. The engineer who eliminated the class of failure before it happened receives little attention.

  5. Knowledge As A Bottleneck: Critical details about deployments, failover, networking, credentials, and recovery procedures live in private memory, old shell history, or undocumented chat threads.

Many operational paths narrowing through one engineer before reaching production

When all paths pass through one person, that person is no longer just helping. They are the bottleneck.

The Cost Of Hero Culture

Hero culture is expensive because the bill arrives in burnout, fragility, and stalled learning.

  1. Burnout Becomes Inevitable: The hero cannot ever fully disconnect. Vacations are interrupted. Weekends are conditional. Sleep depends on whether the system behaves. Eventually, the person who cared most becomes the person most likely to leave.

  2. Teams Stop Learning: If every hard problem is handed to the expert, everyone else loses the chance to build judgment. The team becomes dependent not because people are incapable, but because the system routes learning away from them.

  3. Risk Is Hidden From Leadership: As long as the hero keeps saving the day, leadership sees recovery, not fragility. The organization mistakes survival for resilience.

  4. Incidents Become Harder To Reproduce And Prevent: A fix performed from memory is difficult to audit. If nobody can explain exactly what changed, the organization cannot reliably prevent recurrence.

  5. Succession Becomes A Crisis: When the hero leaves, changes roles, gets sick, or is simply unavailable, the organization discovers that the recovery plan was a person.

A pager glowing red beside an empty coffee cup and a dim laptop

Heroics feel heroic once. Repeated heroics become exhaustion with a pager attached.

A Real-World Example: The One Engineer Who Knew The Network

A platform team I worked around had a recurring pattern during network incidents. Whenever routing behaved strangely, everyone waited for one senior engineer. He knew the history: the old firewall exceptions, the nonstandard BGP behavior, the one load balancer rule nobody wanted to touch, the Terraform module that had drifted from reality.

He was fast. Too fast, honestly. He could restore service before the rest of the bridge understood the failure mode. For a while, that looked like operational strength.

Then he took a week off.

During that week, a certificate rotation triggered a networking path that had not been documented. The team had dashboards, logs, and access. What they did not have was the mental map. The incident lasted hours longer than it should have because the real runbook was on vacation.

The painful part was not that one person knew too much. The painful part was that the organization had allowed that to remain true because his competence made the weakness easy to ignore.

The fix was not to blame him. He had been carrying the system. The fix was to convert hero knowledge into team knowledge: diagrams, failure-mode runbooks, paired incident reviews, ownership rotation, and explicit handoff of the undocumented paths. The goal was not to make him less valuable. It was to make the team more capable.

Why Hero Culture Persists

Hero culture persists because it produces short-term wins that hide long-term damage.

  1. It Works During The Incident: When production is down, speed matters. Calling the person who knows the answer is rational in the moment. The anti-pattern forms when the organization never follows up by distributing that knowledge.

  2. Recognition Systems Reward Rescue: Many organizations celebrate visible recovery more than invisible prevention. The person who fixes the outage is praised. The person who wrote the runbook that avoided three future outages is forgotten.

  3. Documentation Feels Slower Than Doing: The hero can fix the problem in ten minutes. Writing the runbook, explaining the context, and pairing with another engineer takes an hour. Under pressure, the hour loses.

  4. Leaders Confuse Dependency With Ownership: A leader may say, “This system has an owner,” when what they really mean is, “Only one person understands it.” Ownership should create clarity. Dependency creates risk.

  5. Teams Normalize Interruptions: Once the hero is always available, the organization treats interruption as normal operating procedure. The cost is invisible because it is paid by one person’s attention, sleep, and health.

A circular loop from incident to hero rescue to praise to no systemic fix and back to incident

Hero culture is a loop: rescue, praise, forget, repeat.

Turning Heroics Into Team Capability

The answer is not to shame the hero. Most heroes are created by organizational gaps, not ego. The answer is to turn individual expertise into shared capability.

  1. Pair During Incidents: If one person is driving the fix, another person should shadow and document. The goal is not to slow recovery. The goal is to ensure the same incident does not require the same person next time.

  2. Write Runbooks From Real Incidents: The best runbooks come from actual recovery work. After an incident, capture the commands, decision points, rollback criteria, and verification steps while the context is fresh.

  3. Rotate Ownership Deliberately: Ownership rotation does not mean everyone owns everything. It means critical systems should have more than one capable operator. Primary and secondary ownership should be explicit.

  4. Reward Prevention Publicly: Celebrate deleted alerts, simplified architecture, successful game days, and completed postmortem action items. Make prevention visible enough to compete with rescue.

  5. Make Escalation Role-Based: Escalate to the owning team, on-call role, or incident function — not to a favorite individual. If the process only works when one person answers the phone, the process is not real.

  6. Protect Recovery Time: If someone carries a major incident, they need recovery time afterward. Sustainable operations require rest as deliberately as they require coverage.

A group of engineers lifting the same platform together instead of one person holding it alone

The goal is not fewer experts. The goal is more shared capability.

Applying The Scientific Method

Hero culture can be measured and reduced like any other systemic risk.

  1. Observe: Who gets called during incidents? Which systems depend on one or two people? Which alerts require specific tribal knowledge?

  2. Hypothesize: “If we pair during incidents and write runbooks from real recovery steps, repeat escalations to the same person will decrease over the next quarter.”

  3. Test: Pick one high-risk system. Assign primary and secondary ownership. Run a game day where the usual expert observes but does not drive.

  4. Measure: Did the secondary owner recover the system? Was the runbook sufficient? Which steps still required private knowledge?

  5. Iterate: Update the runbook, train another person, and repeat until the system is no longer dependent on one individual.

Carl Sagan’s Baloney Detection Kit

Hero culture survives on comforting myths. Challenge them directly:

  • “Nobody else can do it.” — Is that true, or has nobody else been given time, access, and context?

  • “Documentation will slow us down.” — Slower than a four-hour outage when the expert is unavailable?

  • “They like being the expert.” — Maybe. But liking mastery is not the same as consenting to permanent interruption.

  • “We have an owner.” — Do you have ownership, or dependency? Can the system be operated when that person is unreachable?

  • “We will document it later.” — Later usually means never unless time is explicitly scheduled and protected.

A network map copied from one engineer's notebook onto a shared team wall

The knowledge is not safe until it moves from private memory into shared practice.

Moving Forward Together

Hero culture is seductive because it gives the organization a story: when things go wrong, someone exceptional will save us. But sustainable DevOps and SRE practice is not built on exceptional rescue. It is built on systems that ordinary teams can operate under pressure.

The best engineers should absolutely be valued. But valuing them means refusing to turn them into single points of failure. It means giving them time to teach, document, simplify, and rest. It means rewarding the quiet work that makes future heroics unnecessary.

The measure of operational maturity is not how often the same hero saves production. It is how rarely production requires a hero at all.

Who does your organization always call when things go wrong? What would happen if they were unavailable tomorrow? The answer is not a staffing concern. It is an architectural and operational risk.

References