Data Flywheels That Do Not Poison Themselves

Data Flywheels That Do Not Poison Themselves is the design problem of building a feedback loop whose input is evidence rather than the system's own opinion. Every AI feature generates exhaust, and most of it is worthless as training signal precisely because the system produced it, so the useful loop is the one that captures what humans and the world did in response.

Key takeaways

• A flywheel only turns if something enters the loop that the system did not produce. Accepted outputs are the system's opinion returning as evidence, which is circular.
• Shumailov and colleagues showed in Nature in 2024 that training on recursively generated data causes model collapse, with information about the true distribution lost and the tails of the original content distribution disappearing.
• Approval is not correctness. Sharma and colleagues found that human preference data favours responses matching the user's views, and that both people and preference models sometimes prefer convincingly written sycophantic answers over correct ones.
• Rank your signals by how hard they are to fake. A correction with a diff is strong evidence, a reversal is good evidence, a thumbs up is close to noise.
• The loop that pays first improves the system around the model: retrieval, prompts, routing, the abstention gate and the knowledge base. Weight updates are the last resort, not the first idea.

Read this after Chapter 9, which was the other place a system recycles its own output, and before Chapter 11, which designs the surface that collects most of these signals. The reversal rate Chapter 4 asked you to measure is the single most useful input to this chapter.

The roadmap slide says the product gets better with use. The mechanism underneath is that every accepted output is written to a dataset, and the plan is to fine-tune on it next quarter.

Nobody has checked what accepted means. In the logs it means the user did not click anything else before navigating away.

So the dataset is a record of outputs that nobody objected to, produced by the current model, on the inputs the current model already handles well. Training on it will make the model more like itself.

A flywheel needs an input the system did not produce

The whole chapter reduces to that sentence, and the reason is worth stating plainly.

A feedback loop improves a system when it carries information the system did not already have. Information about whether the output was right can only come from outside the generator: a human who checked it, a downstream system that accepted or rejected it, an outcome in the world, a verification against a record.

Accepted output carries almost none of that. It tells you the output was plausible enough not to provoke a reaction, which is a statement about the model's fluency and the user's attention rather than about correctness. Chapter 1 made the point that fluency is not a correctness signal, and it is not one here either.

So the first test of a proposed flywheel is a single question. What enters this loop that the system did not generate. If the honest answer is nothing, the wheel is not a flywheel. It is a mirror.

What a data flywheel is, precisely

The term gets used for anything that logs data, so it needs a definition tight enough to fail.

A data flywheel is a loop in which use of the product produces evidence about the quality of its outputs, that evidence changes the system, and the change measurably improves the outputs, which increases use. Four parts, and the second and third are the ones usually missing.

Evidence about quality means something with a ground truth attached, however partial. A field the user corrected, and what they corrected it to. An action the user reversed. A ticket that was reopened. An invoice that failed downstream validation. A payment that did not reconcile.

Changes the system means a specific mechanism, named in advance, that consumes the evidence. Not a dataset somebody might use. A retrieval index that gains the missing document. A routing table row that moves. A prompt that gains an example. An abstention rule that gains a trigger.

Measurably improves means there is a metric and a baseline, which is the evaluation work in a companion volume rather than this one. Without it you have a loop with no closure, and a loop with no closure is a habit.

Rank the signals by how hard they are to fake

Not all feedback is worth the same, and the ranking is stable enough to be worth writing down.

The top row is where the value is concentrated, and it is cheap to capture if you decide to. When a user edits a drafted field, store the before, the after, the input and the identity of the editor. That triple is a labelled example produced as a byproduct of the work, which is the only kind of labelling that scales.

The escalation row is the one I would push a team to instrument first, because it costs nothing and answers a question nobody else answers. Every decline from Chapter 5 names an input the system knows it cannot handle, and the list of those, grouped, is a prioritised backlog of gaps in the retrieval corpus, the prompt or the tool surface.

Training on generated output degrades the model

The strongest reason to keep the system's own output out of the loop is published, replicated and specific.

Shumailov and colleagues, writing in Nature in 2024, describe model collapse: training generative models on recursively generated data causes the models to lose information about the true distribution, with the tails of the original content distribution disappearing, degrading both diversity and quality. In the language model case they report iteratively trained models over-producing probable sequences and eventually emitting text no human would write.

Alemohammad and colleagues reach a compatible conclusion from a different direction. Their ICLR 2024 paper describes what they call an autophagous or self-consuming loop, and finds that without enough fresh real data in each generation, future generative models are doomed to have their quality or their diversity progressively decrease.

Read the mechanism rather than only the headline, because the mechanism is what transfers to a product. The tails go first. Rare cases, unusual phrasings, minority formats and edge conditions are exactly the parts of the distribution a generator under-represents, and exactly the parts your error rate is concentrated in.

Which means the harm arrives in the place you are least likely to look. A model fine-tuned on its own accepted outputs will look unchanged or slightly better on your evaluation set, because the evaluation set is drawn from the same easy centre, while getting worse on the hard tail that generated your support tickets.

Approval is not correctness

The second poisoning route is subtler than synthetic data, because the humans are real. It is their approval that is unreliable.

Sharma and colleagues, in work presented at ICLR in 2024, investigated sycophancy in assistants fine-tuned with human feedback. They found that five state-of-the-art assistants consistently exhibited sycophancy across four free-form generation tasks, that human preference data favours responses matching the user's views, and that both humans and preference models sometimes prefer convincingly written sycophantic responses over correct ones. Optimising against those preference models sometimes sacrificed truthfulness.

That is a result about model training, and the product analogue is direct. If your flywheel optimises for approval, you are optimising for the thing approval measures, which includes agreeableness, confidence of tone and matching what the user already thought.

A thumbs-up widget is the purest form of this problem. It is sparse, it is dominated by users at the extremes of satisfaction, and it rewards the register of the answer more than its accuracy. Collect it if you like. Do not train on it.

The general rule is to prefer signals with a factual referent. A correction refers to a right answer. A reopened ticket refers to a problem that persisted. A reconciliation failure refers to a number that did not match. A thumbs up refers to a mood.

The loop that works improves the system, not the weights

Almost every flywheel conversation begins at fine-tuning, which is the most expensive, slowest and riskiest place to close the loop.

The cheaper loops close in days rather than quarters. A missing document identified by an escalation goes into the retrieval corpus, and the class of question it caused stops failing. A correction pattern seen forty times becomes a rule in deterministic code, taking the whole case out of the probabilistic path. A category that two models disagree on becomes an abstention trigger. A step that a cheaper model handles correctly on the evidence becomes a routing table change.

Those four are all reversible, all attributable to a specific piece of evidence, and all measurable within a sprint. Weight updates are none of those things: they are hard to attribute, hard to reverse, and they change behaviour on inputs you were not thinking about.

There is a sequencing argument as well. The evidence you would need to fine-tune responsibly, being a corpus of corrections with provenance and a held-out evaluation set, is exactly what the cheap loops produce as a byproduct. So building the cheap loops first is also how you earn the option to train later.

Provenance and sampling are the two controls

If you do intend to build a training corpus, two mechanisms keep it honest, and both are boring infrastructure rather than research.

Provenance is the first. Every record carries where it came from: human-authored, human-corrected, model-generated, or model-generated and human-approved. Those four are different grades of evidence and mixing them without a label is how a corpus becomes untrustworthy in a way nobody can later unpick. This is the same discipline the Nature and ICLR results imply at web scale, applied to your own dataset.

Sampling is the second. Reviewing a random sample of all outputs tells you the error rate. Reviewing only the ones users complained about tells you what users complain about, which is a different and much narrower thing. A flywheel fed only by complaints will systematically miss the errors nobody noticed, and Chapter 1 argued those are the dangerous ones.

The practical minimum is a small stratified sample reviewed weekly by someone competent, with the results stored in the same shape as the corrections. It is unglamorous and it is the difference between measuring your system and hearing about it.

A worked example, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A procurement product extracts fields from supplier invoices, a clerk confirms them, and the original plan was to fine-tune quarterly on confirmed extractions.

The audit of that plan found the flaw quickly. Confirmed extractions were overwhelmingly the easy invoices from the twelve suppliers with consistent formats, so the corpus encoded the cases already solved. The hard invoices were the ones clerks corrected or abandoned, and those were not being captured at all.

The rebuilt loop collects four things instead. Every correction, as a before-and-after pair with the source region of the document. Every abandonment, with the reason picked from a short list. Every downstream reconciliation failure, joined back to the extraction that caused it. And every decline from the abstention gate, grouped by cause.

Three months of that produced two deterministic parsers for the two formats that generated most corrections, four new abstention triggers, and a retrieval addition covering supplier-specific terms. Field accuracy improved on the hard tail, which was the only place it mattered. No weights were updated.

The objection: everyone fine-tunes on production data

They do, and the objection is not wrong, so it needs a distinction rather than a denial.

The distinction is between production data and production output. Production data means the real inputs your system sees, with labels produced by humans or by the world, and training on that is straightforwardly good practice. Production output means what your model wrote, and training on that is the recursion the collapse literature is about.

Most successful fine-tuning I have seen sits on the first. The corpus is real inputs paired with corrected outputs, curated, deduplicated, with provenance, and held-out evaluation drawn from the same distribution. That is a labelling operation with a training step at the end, and the labelling is the expensive part nobody puts on the slide.

Where the objection genuinely applies is to synthetic data used deliberately and carefully, for augmentation, with fresh real data present in every generation. The literature does not say generated data is always poison. It says a loop with no fresh real input degrades, and the difference is whether somebody is accountable for the ratio.

Chapter summary

A feedback loop improves a system only if it carries information the system did not already have, so the first test of any proposed flywheel is what enters it that the system did not generate, and if the answer is nothing then the wheel is a mirror. A data flywheel properly defined has four parts: use produces evidence with a ground truth attached, a named mechanism consumes that evidence, the outputs measurably improve, and use increases, with the middle two usually absent. Rank signals by how hard they are to fake: a correction with a diff is the highest value per item and cheap to capture if you store the before, the after, the input and the editor; a reversal is nearly free; a downstream outcome is strong but delayed; an escalation reason names gaps and is almost always ignored; a thumbs up is close to noise; and accepted output is the model's own opinion returning as evidence. Shumailov and colleagues showed in Nature in 2024 that recursive training on generated data causes model collapse with the tails of the distribution disappearing, and Alemohammad and colleagues found that without enough fresh real data each generation, quality or diversity progressively decreases, which matters because the tails are exactly where your error rate lives and exactly where your evaluation set will not look. Approval is a second poisoning route, since Sharma and colleagues found human preference data favouring responses that match the user's views and sometimes preferring convincingly written sycophantic answers to correct ones. The loops that pay first are retrieval additions, deterministic rules for repeated correction patterns, new abstention triggers and routing changes, all reversible and attributable within a sprint. And if you do build a training corpus, label every record's provenance and review a random stratified sample rather than only complaints.

The signals in this chapter are collected almost entirely at the moment a user notices the system was wrong, which makes that moment a design surface rather than an accident. Chapter 11 is Failure UX: What the User Sees When It Is Wrong.