The retraction audit.
Every retracted paper with a DOI, matched to every citation it received after it was retracted. The set defines itself: a paper is in because a journal retracted it, not because I picked it.
| Campaign | 001, the retraction audit |
|---|---|
| Corpus | Every retraction in the Retraction Watch database, via Crossref (67,815 retraction records), matched against the OpenAlex citation graph. |
| Data cost | $0. Every source is open. |
| Budget | Under $50,000 for all three phases, nearly all of it for people: building the dataset and experts who check the machine. |
| Method | Published before each run. Raw results published with the findings. |
| Output | Open dataset, tables by journal and field, a ledger of journal replies, and a plain-language report. |
The clearest signal the record has.
A retraction is a journal saying, in public, that a paper should not be relied on. If the record can’t carry that message, it can’t carry subtler ones. And retractions are written down, dated and machine-readable, so the check can be run whole and run again by anyone.
The stakes are real. Reviews and treatment guidance are built from the literature one citation at a time. In one large study of biomedicine, only 5.4% of citation contexts citing a paper after its retraction said it had been retracted.1 Small studies have read samples of the rest; nobody had read them all to see how often they still use the retracted work as support. That is the job.
Count, read, tell.
- Done
1Count
Match every retracted paper to every citation it got after the retraction. Mechanical, no judgement calls.
Out: 32,626 of 51,414 retracted papers cited again; 262,998 citations in all, leaving out paperwork retractions (corrected 16 Oct 2026 from 286,050).
- Census read · human check next
2Read
For every citing paper with open full text, find the passage that cites the retracted work. Did the author use it as support, mention it in passing, or flag it? Experts check a random sample of the machine’s calls.
Census: 40.7% of 51,626 citations cited the retracted paper as support. Results →
- After Phase 2
3Tell
Send each journal that published a citation resting on retracted work the evidence. Then publish who corrected, who replied, and who said nothing.
Out: a public ledger of replies.
I read every citation I could.
Every citation made after retraction to a PubMed-indexed paper retracted from 1990 to 2023, wherever the citing paper is open access: 55,688 citations to 12,864 retracted papers. Three AI models read each one; blind samples measure how often they are wrong. A check by two human labellers comes next, and until then every figure here is provisional.
cited the retracted paper as support. 95% CI 39.9–41.5%; 38.1% after correcting for the error measured on a blind sample.
flagged it as retracted or unreliable next to the citation. 95% CI 2.6–4.3%.
still cited it as support when their own reference list said “retracted” (40.4% of 3,628).
51,626 citations to 12,328 retracted papers; citations to administrative retractions (out of date, replaced, moved journal) are reported apart. Search every citation →
What the census found
- Support fades slowly.43.6% of citations in the first two years after retraction cited the paper as support; after ten years it was still 34.9%. Leaving out citing papers submitted before the retraction moves the overall rate by 0.1 point.
- Misconduct is cited like honest error, and flagged less.Papers retracted for misconduct, paper mills or plagiarism were cited as support 41.2% of the time; papers retracted for error, 40.0%. Only 1.9% of citations to the first group flagged the problem, against 6.5% for error.
- Reviews lean on it most.More than half the citations in reviews, meta-analyses and systematic reviews cited the retracted paper as support. An exploratory check, done by AI, found 472 meta-analyses, systematic reviews and guidelines that counted a retracted study as data.
- Most citations are passing ones.55.9% were one reference in a bundle, background, or a description. They are counted, but not as support.
How it was run
| Frame | 22,553 retracted papers in Retraction Watch (indexed in PubMed, retracted 1990–2023). 16,019 (71%) were cited in a later year: 130,315 citations. |
|---|---|
| Citations | 58,358 citing papers had open full text in Europe PMC. The citing passage was found in 96%; notices were set aside, leaving 55,688 citations read, 43% of all later citations. |
| Reading | Three Cloudflare models (Jev, Clef, Gemma) answer two questions about each passage, and a fixed rule turns the answers into a label. The models are never told the paper was retracted. |
| Check | On blind random samples of 300 and 150 citing papers, labelled by a separate AI reader, the panel agreed on 92.3% and 94.7%. On 8,944 citations also checked by Hsiao and Schneider, whose labels were screened by keyword and confirmed by hand, it agreed on 97.7% about whether the retraction was acknowledged. Two human labellers checking 200 more is the next step. |
| Robustness | Six alternative cuts of the data, from first passage only to citing papers received after the retraction, all give 36–41%. |
| Cost | About $30 in compute for the whole census. |
| Data | Search every citation · citations.csv · census.json · codebook.md · protocol and code |
Limits: open-access biomedical papers only, so 57% of later citations are counted but not read; reweighting for them moves the rate by about one point. Labels judge the citing text, not whether the citing paper's conclusions depend on it, and “cited as support” also covers using a method or saying results agree. The pilot that came first (1,002 sampled citations, 38%) keeps its data: results.json · labels.csv.
How the count works.
- Retractions come from the Retraction Watch database, distributed by Crossref, which records the retraction date.
- Citations come from the OpenAlex graph, by year. A citation counts as post-retraction if it falls in a calendar year after the retraction year, so citations in the same year are left out and the count is low, not high.
- Coverage: 53,272 of the 53,685 retracted works with DOIs, retracted in 2024 or earlier, matched in OpenAlex (16 Oct 2026).
- Paperwork retractions are left out: those that find no fault in the science (withdrawn as out of date, temporary removal, publisher error, replaced at the same DOI). That is 1,858 papers and 23,251 citations, by the same rule the Phase 2 census used.
- Accuracy of the count: 200 counted citations, drawn at random and checked against the citing papers’ own reference lists (16 Oct 2026), were about 96% real; the rest were wrong links, a retraction notice, a whole journal issue, or duplicate copies of one paper. The true total is probably about 254,000 (likely 247,000 to 260,000). Correction, 16 Oct 2026: the first count (286,050, run of 13 Oct) included them.
- A post-retraction citation is a flag, not an accusation. Authors often never saw the notice. The audit measures that failure of the system, not of scientists.
- Anyone named in the results can reply, and the reply is attached to the record.
The data behind every number on this site: results.json and feed.json. The full dataset publishes with the tables by journal and field.
- Hsiao T-K, Schneider J. Continued use of retracted papers: temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine. Quantitative Science Studies 2021;2(4):1144–1169. 722 of 13,252 post-retraction citation contexts in PubMed Central acknowledged the retraction.