Reading the Evidence About Your Own Job

Reading the Evidence About Your Own Job is the one skill in this book whose absence poisons everything else in it. Every later chapter tells you to do something different on the basis of a claim about the world. If you cannot check such a claim, you will eventually make a five-year decision on a number that dissolves the moment somebody clicks through it.

Key takeaways

• Four questions decide whether a number is evidence about you: who measured it, on whom, when, and what exactly they counted. A claim that fails any one of them is an anecdote wearing a percentage sign.
• The claim that 92% of developers use AI coding tools daily traces to a Wakefield Research survey for GitHub fielded 14 to 29 March 2023, among 500 non-student, non-manager US developers at firms with 1,000-plus employees, asking whether they had ever used such tools at work or in their own time. The word daily was added by retelling.
• Some claims have no reachable primary at all, and when the chain ends in outlets citing each other, the correct move is to stop rather than to soften the number and use it anyway.
• The Stanford Digital Economy Lab's Canaries paper reports a 16% relative employment decline for 22-to-25-year-olds in AI-exposed occupations in its 13 November 2025 version, having reported 13% in August 2025. Rounding it up in transmission is the most common distortion in this whole subject.
• Evidence classes answer different questions. METR's randomised trial found experienced developers 19% slower, the three-firm field experiments found 26.08% more tasks completed, and the disagreement between them is the finding rather than a scandal.

Read this before Chapter 3, which spends this evidence on a decision about what to learn deeply, and beside Chapter 13, which puts refresh intervals on the same inputs. Chapter 13 is the schedule. This chapter is how to weigh one claim.

I spent an afternoon reading the pages that currently rank for the career questions this book is about. They are good-looking pages. Several are long, structured and carry a reference list.

I came away with nine numbers I could not use. Three of them were not merely unsourced. They were demonstrably different from what the underlying study had measured, and two of the three came from the best-sourced page in the whole set.

That is not a story about bad writers. It is a story about how numbers travel.

Four questions, and one of them is usually fatal

Before a claim about your labour market changes what you do, put it through four questions in this order.

Who measured it. Not who published the page you are reading: who ran the survey, the trial or the analysis. Then, on whom. A sample of 500 and a national payroll panel support wildly different sentences. So does one company's staff.

Then, when. A measurement of AI use from early 2023 describes a different world from one taken in 2026, and this subject moves fast enough that a three-year-old adoption figure is a historical artifact. Then, and this is the one that usually kills the claim, what exactly did they count. "Has used", "uses daily", "prefers" and "is required to use" are four different questions, and the difference between them is where distortion lives.

The decision rule for the whole chapter is one sentence: if you cannot answer all four questions in under five minutes of clicking, do not let the number change your behaviour.

The 92% that never measured daily use

Take the most repeated statistic in this subject. It appears as some version of "92% of US developers use AI coding tools daily", often in a page with a reference list.

The primary is reachable. GitHub published the underlying research, run by Wakefield Research, and the study was fielded between 14 and 29 March 2023 among 500 US-based developers who were neither students nor managers, all at companies with more than 1,000 employees. It found that 92% of them had used AI coding tools, at work or in their personal time. The split was 67% both, 25% at work only, 6% personal only.

Now run the four questions. Who: a research firm commissioned by a vendor with an interest in the answer. On whom: 500 people, one country, large employers only. When: three years ago, before the assistants most people now use existed. What was counted: whether they had ever used one, in any context.

The retelling adds a frequency the survey never asked about. Nobody lied. Somebody compressed "have used at work or personally" into "use daily" because it reads better, and then four other people copied the compression.

A claim whose chain ends in outlets quoting each other

The second failure mode is worse than distortion, and it is more common than people expect: the chain has no end.

There is a widely recited claim that most of the new code at one very large engineering organisation is now generated by a model, attributed to a public statement by its chief executive. I could not reach a primary for it. The trade press cites other trade press, one of the most quoted articles sits behind a server that refuses automated readers, and I found no transcript, filing or company publication carrying the figure. The percentage also grows in the retelling, in a tidy sequence that looks like measurement and behaves like folklore.

What I can say honestly is the qualitative version. Large engineering organisations now report that most new code originates from a model and is accepted by a human, and the reported proportions rise fast enough that any specific figure is stale within a quarter.

Notice what that sentence gives up. It cannot be used to win an argument. That is the cost of being accurate, and it is a cost worth paying, because a reader who catches you on one unsourced number stops believing the sourced ones.

How 16% becomes 20% without anybody lying

The third pattern is drift, and it happens to good research.

The Stanford Digital Economy Lab's paper Canaries in the Coal Mine, using administrative payroll data from the largest US payroll provider, reports a 16% relative employment decline for workers aged 22 to 25 in AI-exposed occupations after controlling for firm-level shocks, in its version dated 13 November 2025. Its August 2025 version reported 13% on the same design. The paper also finds that adjustment runs through employment rather than pay, that declines concentrate where AI automates rather than augments, and that results hold when technology firms and remote-amenable occupations are excluded.

That is a careful, specific, heavily qualified finding. In transmission it becomes "entry-level engineering employment fell about 20%", with the age band gone, the relative comparison gone, the controls gone and the number rounded up past both published versions.

Every qualifier dropped makes the claim easier to repeat and less able to guide a decision. The 16% figure tells a 23-year-old something real and tells a 34-year-old almost nothing. The 20% version tells everybody something false.

Evidence classes, and what each one can settle

Most confusion in this subject is people comparing two numbers that were never answering the same question. Class the source first.

That table is the whole method. Ask what class a number belongs to, then ask whether the sentence being built on it is a sentence that class can support.

When two good studies disagree, the disagreement is the finding

Here is the case that teaches the method best, because both sides are strong.

METR published a randomised controlled trial on 10 July 2025. Sixteen experienced open-source developers worked 246 real issues in repositories they already knew, and they took 19% longer to complete issues when allowed AI tools. They had forecast a 24% speed-up beforehand, and after the study they still believed they had been 20% faster. METR states explicitly that this does not establish that AI fails to speed up most developers, and lists sampling and learning-curve limitations.

Now the other side. Cui and colleagues, published in Management Science, ran three field experiments across 4,867 developers and found a 26.08% increase in completed tasks with assistant access. Gains ran 27 to 39% for recent hires and more junior developers, against 8 to 13% for senior ones. Brynjolfsson, Li and Raymond, in the Quarterly Journal of Economics in May 2025, found a 15% average productivity gain in a customer-support setting, with less experienced workers improving both speed and quality while the most experienced saw small speed gains and small declines in quality.

Both results are real. They differ because the populations differ, the tasks differ and the codebases differ: expert developers in code they wrote, against a mixed workforce on assigned tasks. The synthesis is not an average of the two numbers. It is a shape, meaning the gain shrinks and can invert as expertise and codebase familiarity rise.

That shape is what you can act on. The two headline percentages, waved separately, are how people end up believing whichever number they read second.

Cite the people who stress-tested their own result

There is one move I would look for in any source before trusting it on this subject, and it is rarer than it should be.

The same Stanford lab published a follow-up on 9 February 2026 examining alternative explanations for its own finding. It reports that AI-exposed occupations are on average less exposed to interest rates, so rate rises do not explain the pattern; that under firm-time fixed effects the decline is statistically significant only after 2024; that the decline reached about 16% by October 2025; and that AI is not the sole determinant while remaining the leading driver they identify.

That is a research group publicly attacking its own headline. My read is simple: a source that stress-tests its own result has earned a great deal more of your trust than one that repeats the result louder.

Use the same standard on career content, including this book. Any writer telling you what to do with the next five years should be visibly willing to say which parts of their argument would be falsified, and by what.

Vendor research is usable when you say whose it is

The last class needs its own rule, because the reflex to dismiss it is as lazy as the reflex to swallow it.

Faros AI's AI Productivity Paradox Report 2025, published on 23 July 2025, draws on telemetry across more than 10,000 developers and 1,255 teams and states its method, including the correlation tests it ran. That is real data at a scale no academic group holds. It is also research by a company that sells engineering-productivity analytics, and the report's conclusion is that engineering productivity needs analytics. Both facts belong in any sentence that uses it, and Chapter 7 uses its numbers that way.

Scale is not the only trap. A January 2026 paper from IBM Research reports that 88% of surveyed enterprise developers saw productivity gains, 77% of them at 25% or better, which sounds decisive until you read that the sample is 57 developers inside a single organisation. That is a genuinely useful first-party account of one enterprise. It is not a rate you can apply to your own team.

So the rule is disclosure rather than exclusion. Name the publisher, name their interest, name the sample, and let the reader weigh it.

The objection: nobody has time to check footnotes

True, and the answer is that you do not check everything. You check what you are about to act on.

Almost every number you encounter has no consequence for you. Read it, note it, move on. The check is for the small set of claims that would change a decision: what to learn next, whether to change specialism, whether to leave, whether to stop investing in something you are good at. That set is a handful per year.

For those, five minutes is genuinely enough. Find the primary, read the method paragraph, check the sample and the date, and check the exact question asked. If the trail runs into a page you cannot open or a citation loop, you have your answer, and the answer is that this number does not get a vote.

There is a cheaper habit that catches most of it. Notice adverbs of frequency and absolute quantifiers, because "daily", "always" and "most" are where compression happens. When you see one, ask what the questionnaire actually said.

Chapter summary

A claim about your own labour market earns the right to change your behaviour only after four questions: who measured it, on whom, when, and what exactly they counted. The most repeated statistic in this subject fails the fourth. GitHub's own published research, fielded between 14 and 29 March 2023 among 500 US developers at firms of more than 1,000 employees, found that 92% had ever used AI coding tools at work or personally, and the retelling silently added a frequency nobody asked about. A second pattern is worse, being a chain that never terminates, as with the much-quoted claim about one organisation's share of model-generated code, where the trade press cites the trade press. A third is drift, where the Stanford Digital Economy Lab's carefully qualified 16% relative employment decline for 22-to-25-year-olds in AI-exposed occupations, from its 13 November 2025 version, becomes an unqualified 20% for everyone. Classing the source prevents most errors, because trials, telemetry, surveys and payroll data each answer a different question. METR's trial found 16 experienced developers 19% slower while still believing they were 20% faster, and Cui and colleagues found 26.08% more tasks completed across 4,867 developers with the largest gains among juniors. The reconciliation is a shape rather than an average: the gain shrinks as expertise and familiarity rise. Prefer sources that stress-test their own findings, name vendor interests rather than dismissing vendor data, and spend the check only on the few claims a year that would change a decision.

Chapter 3 is where this gets spent. Depth, Breadth, and the Half-Life of a Skill takes the evidence that survives the four questions and turns it into a decision about what to learn deeply, which is the first irreversible bet most engineers make after reading a headline.