Rescue Stories

Working With the Team That Built It

Getting the truth from people who expect to be blamed for it.

Chapter 8 of 1211 min readOpen access

Working With the Team That Built It is the part of a rescue with no technical component and the highest information yield. The people who wrote the system hold what Chapter 3 could not recover from the text, and every question you ask them sounds, from where they sit, like the opening of a case against them.

Key takeaways

  • Ask about decisions and constraints, never about code quality. "What did you know on the day you wrote this" is answerable; "why is this like this" is heard as an accusation and returns an architecture lecture.
  • Separate the review of an event from the evaluation of a person, explicitly and out loud. Google's postmortem practice works because the output is an owned action rather than a named cause, and that separation has to be stated to be believed.
  • Make the first consequence of a truthful answer something good. Fix the thing they name first, credit them for it in front of the sponsor, and the second interview will be twice as productive as the first.
  • The person with the most knowledge is often the one with the most exposure, and they are usually right to be worried. Say plainly what you are and are not reporting on, then hold to it exactly.
  • Notice periods are the real deadline. Every departure closes a channel on evidence that exists nowhere else, and no document written after the fact recovers it.

Read this beside Chapter 3, which set out the interview questions and their timing, and Chapter 7, which is the parallel conversation with the people funding the work. Chapter 11 is where this relationship becomes the handover.

The engineer who wrote the payment module has been asked to help you for two hours a week. They have watched you spend a month reading their work and talking to their sponsor. Nobody has told them whether they still have a job in September.

You ask why the retry logic is the way it is. They tell you it needs refactoring, that they always knew it was a problem, and that they raised it at the time. None of that is an answer to your question. All of it is a rational answer to the question they actually heard.

Everyone in the room has an incentive and yours is not neutral

Chapter 1 made the point in passing and it needs its own treatment here, because this is the chapter where the incentives are load-bearing.

The original engineers need the story to be that the system was underfunded and rushed. The engineering manager needs it to be that the team performed well under impossible constraints. The sponsor needs it to be that the money was spent on something recoverable. The vendor, if there was one, needs it to be that scope changed. Each of those is usually partly true, which is what makes them durable.

Your own incentive is worth stating too, in the room, because it is visible whether or not you name it. You were hired to recommend a course of action, and the more broken the system is, the larger the engagement. Anyone in the building can work that out. Naming it first is cheap and it buys more candour than any assurance about your character.

There is a fact from Chapter 1 that helps here, and it should be said early. Across 1,471 IT projects, Flyvbjerg and Budzier measured a mean cost overrun of 27%, with one project in six overrunning by 200%. Overruns are the base rate of this industry rather than a verdict on this team. That sentence, delivered early and honestly, does more for an interview than any amount of reassurance about intentions.

Ask about decisions, not about code

The single change that improves these conversations most is grammatical. Ask about a decision, on a date, with the evidence open.

"Why is this like this" invites a defence of a design. "What did you know on the day you wrote this" invites a description of constraints, and constraints are what you are actually after. The second question has a property the first lacks. It cannot be answered wrongly. Whatever they knew that day is a fact about the past, and facts about the past do not reflect on present competence.

Bring the diff. Open their commit. Read the date and the message aloud, then ask what was happening that week. You will get the release that was due, the customer who escalated, the person who left mid-sprint, the instruction from someone senior that was never written down. That is the why column filling, and none of it exists in the repository.

Then ask the highest-yield question in a rescue, which Chapter 3 introduced. What would you have fixed with two more weeks. It extracts a private defect list without asking anyone to confess to a defect, because the frame is a resource they were not given rather than a mistake they made.

Two habits to keep. Write the answer down in front of them and read it back, because half of what you think you heard is wrong and they will correct it for free. And never ask a question whose answer you already have from the repository, unless you say so first. One caught test question ends candour for the rest of the engagement.

The blameless review is a technique, not a courtesy

Blamelessness gets treated as etiquette. It is a mechanism for producing accurate data, and Google's site reliability practice is explicit that a blameless review is what makes people willing to escalate and describe what actually happened.

The mechanism has three parts, and dropping any one collapses it. The review names the sequence of events and the contributing conditions, not the person who typed the command. Its output is an action with an owner and a date, not a finding of cause. And the same standard applies to the people conducting it, which on a rescue means publishing your own errors in the same format, in the same document, without softening them.

Say the separation out loud, in the first session, in one sentence: this review is about the event, and it is not an input to anyone's performance evaluation. Then honour it exactly. You will be asked to break it, usually by the sponsor, usually in private. The first time an interview answer appears in a status report attached to a name, the channel closes permanently, and the people watching will have noticed before you do.

There is a limit worth being honest about. If your engagement genuinely includes an assessment of the team, you cannot promise the separation, and pretending otherwise is a worse offence than saying so. In that case say what you are reporting on, what you are not, and to whom. People can work with a stated boundary. They cannot work with a friendly one that turns out to be false.

Culture is a variable you can read in a week

DORA's capability guidance on generative organisational culture describes Westrum's six behaviours: high cooperation, messengers are trained, risks are shared, bridging is encouraged, failure leads to inquiry, and novelty is implemented. DORA's finding is that a high-trust generative culture predicts software delivery and organisational performance.

Read those six as a diagnostic rather than an aspiration. In week one you can observe most of them without asking a single question. Does bad news travel upward before it becomes an incident, or does it appear only after a customer reports it. When something breaks, does the first meeting ask what happened or who did it. Are the people who raise risks the ones being given the difficult work, or the ones being managed out.

What you find changes the plan rather than the diagnosis. In a pathological setting the evidence you gather is systematically distorted toward whatever is safe to say, so weight documents over interviews, prefer the repository to the retrospective, and expect the real explanation to arrive privately, late, and from the most junior person present.

It also changes what a handover can be. A team that cannot say bad things out loud cannot be handed an operational responsibility safely, and Chapter 11 has to plan around that rather than hope it improves.

The two people you must not lose

Every stalled delivery has one or two people whose departure would remove the last copy of something important. Identify them in week one and treat their calendar as the binding constraint on the schedule.

Naur's argument is the reason. A theory of a program transfers through working contact with someone who holds it, and he considered its reconstruction from documentation alone strictly impossible. That is not a claim about documentation quality. It is a claim about what documents can carry. Write better ones anyway, and do not expect them to substitute.

So the sequencing rule is uncomfortable and correct: interviews before analysis when a departure is scheduled. If the senior engineer leaves in three weeks, their time is worth more than three weeks of your reading, and the reading can happen after they are gone while the conversation cannot.

Retention is not in your gift. Raise it anyway, once, with numbers rather than sentiment. A short extension of one contract is cheap compared with the cost of reconstructing the theory it carries, and the sponsor is the only person who can authorise it. That conversation belongs in the same meeting as Chapter 7's options, because it is a cost with an expiry date.

What to do when someone is protecting a decision

Occasionally you will meet real resistance rather than caution: an engineer who steers every conversation away from one area, or a manager who supplies documents that answer a question you did not ask.

Do not confront it. Route around it. Let the evidence accumulate instead, because the repository records what happened regardless of what anyone says, and a decision that somebody is protecting usually has a paper trail in the diffs, the incident record, or the schema. Two hours in version control is worth more than four attempts to get a straight answer, and it costs nothing in goodwill.

When you do have to raise it, raise the artifact and not the person. Bring the migration that was started and abandoned, ask what stopped it, and leave the sentence there. Most of the time the answer is mundane, being a reprioritisation nobody wrote down, and the reluctance was about how it would sound rather than about what it was.

Chapter summary

The people who built a stalled system hold the evidence that cannot be recovered from the text and have the strongest reasons to guard it, so the conversation has to be engineered rather than trusted to good intentions. Name the incentives in the room, including your own, and use the industry base rate from Flyvbjerg and Budzier to make an overrun a statistical fact rather than a verdict on this team. Ask about decisions on dates with the diff open, never about code quality, and close with what they would have fixed given two more weeks, because that question extracts a defect list without asking for a confession. Treat blamelessness as a data-collection technique with three parts: events rather than people, actions with owners as the output, and the same standard applied to your own mistakes, stated once and honoured exactly, or declared impossible if your engagement really does include an assessment. Read Westrum's six behaviours as a diagnostic in the first week, and let a pathological reading change how you weight interviews against documents rather than change what you conclude. Identify the one or two people whose departure would delete the last copy of a theory, run their interviews before your own analysis, and put the cost of a short extension in front of the only person who can authorise it. And when someone protects a decision, route around them into the repository, then raise the artifact rather than the person.

The team is now telling you what happened, and the plan is agreed with the people paying for it. What remains is the hardest scheduling problem in a rescue. Chapter 9 is Sequencing Repairs Without Freezing Delivery: how to interleave repair and delivery in the same week without either stalling, why batch size decides whether that is possible, and how to order the repairs so each one makes the next one cheaper.

Sources

  1. Postmortem Culture: Learning from Failure, in Site Reliability Engineering: How Google Runs Production SystemsGoogle, O'Reilly Media · 2017 · Official documentation · verified
  2. Capabilities: Generative organizational cultureDORA · Official documentation · verified
  3. Programming as Theory Building (1985), reprinted as Appendix B of Agile Software Development: The Cooperative GamePeter Naur, in Alistair Cockburn (Addison-Wesley) · 1985 · Research paper · verified
  4. Why Your IT Project Might Be Riskier Than You ThinkarXiv · 2013-03-28 · Research paper · verified