db = FileAttachment("../../_data/reviews_public.json").json()
// The six tiers, and the palette the citation post already uses for them.
TIER = ["top3","general_ps","ps_subfield","area_studies","interdisc","non_ps"]
TIERLAB = ({ top3: "Top 3 (AJPS/APSR/JOP)", general_ps: "General political science",
ps_subfield: "PS subfield", area_studies: "Area studies",
interdisc: "Interdisciplinary", non_ps: "Outside political science" })
TIERCOL = ({ top3: "#2a78d6", general_ps: "#eb6834", ps_subfield: "#1baf7a",
area_studies: "#eda100", interdisc: "#e87ba4", non_ps: "#008300" })
INK = "#6b1b1b"
// 2016 has no reports, and a band scale with a gap in it prints a duplicate tick.
// Fill the sequence so every year has a bar (2016 is a real zero) and the axis is honest.
YEARS = d3.range(db.first_year, db.last_year + 1)
byYear = YEARS.map(y => {
const r = db.by_year.find(d => d.year === y)
return { year: y, n_reports: r ? r.n_reports : 0 }
})
PHD = 2016 // Ph.D., University of Washington
TENURE = 2023I counted every referee report I have written
publishing
peer review
transparency
data viz
Ten years of peer review, logged report by report: 126 reports for 63 journals, 71,603 words. Whether that leaves me in credit or in debt turns out to depend entirely on an assumption nobody states — and the journals that tell me least are the ones I write least for.
A while ago I put my whole publication record online, and then asked whether it mattered where the papers landed. Both were about what I have received from peer review. Now I address what I have given.
I went back through every portal, decision letter and folder I could find and logged each report individually: which journal, which round, how many words, what I recommended, and what the journal told me afterwards, including whether they shared the other reviewers’ reports. My census included 126 reports, 63 journals, 95 manuscripts, 71,603 words, 2015 to 2026.
This post describes the record with commentary and introduces to how it was built. The arguments the follow and a bunch of other ideas I have I’ll save for later posts.
How much, and is that a lot?
2026 is hollow and dashed because the year is not over; 2016 is a real zero. The two vertical markers are my Ph.D. (2016) and tenure (2023). The flat line is the benchmark: Djupe and Walker's 2024 survey of 637 political scientists finds that faculty in PhD-granting programmes average about six reviews a year, and about 5.8 for associate professors like me. My mean over 2021–25 is 14.4, and 2025 was 29.
So: roughly two and a half times the disciplinary average, and 2025 alone lands in the far-right “20+” bin of their distribution — the tail they describe as shrinking. I am in the group that does roughly half of all reviewing while being about a fifth of the people.
One honest measurement issue. Djupe and Walker’s question asks you to “check your records to be accurate”, but almost nobody has records. Re-reviews of a paper you saw before are exactly what recall loses first. I have records, so my number is more complete than theirs; therefore, some of that 2.4× gap is likely to be measurement rather than labour. Counting only first-round reports I am still at 10.8 a year, about 1.8×, which is likely a more accurate number.
Where it goes, and the long tail
Sorted by scope, not prestige. "Outside political science" means the journal belongs to another discipline — psychology, communication, economics, public policy — rather than being genuinely cross-field.
17% of my reports are for the top three. The bulk sit in subfield journals and outside political science: interdisciplinary and non-PS venues together are 46 of 126 reports.
My long tail of reviewing once outside of political science probably distinguishes me from a lot of other political science reviewers that tend to focus more squarely in their domain. I have refereed for 46 of the 63 journals exactly once — and 27 of those 46 sit outside core political science. These are journals I do not have a relationship with; they are journals that found me for a specialty (and most often because I was cited), mostly post-Soviet politics, corruption, survey methods or disinformation. Most of them I have never submitted to, which in ledger terms is pure giving.
I have received a lot more non-PS requests recently. In 2015–21, 27% of my reports were for journals outside core political science. In 2022–26 it is 52%, and in 2025 it was 69% — my busiest year was also my least disciplinary one. That matters for everything below, because those are the venues that tell me least, am least likely to re-review for, and for which I write the shortest reports.
Because “outside political science” is a quarter of the work and lumps four disciplines together, it is worth breaking open:
I expected economics to be one of the big ones. It is the smallest, below a sociology and methods pair I had not thought about at all.
A round-two report is a third of a round-one report
Median length by round: 656 → 216 → 98 → 78 words. 73% of my reports are first reviews. The darker bar is the report proper; the gap above it is the reference list, which is small because most reports have none — see below.
This matters more than it looks, because every survey of reviewing asks for one number. Djupe and Walker’s Q17 is “how many article reviews have you conducted… in the past year?”, with nothing said about whether a re-review of a paper you already saw counts. Someone who did ten first reviews and someone who did five firsts plus five re-reviews both answer “ten” and have done very different amounts of work.
I would add one field:
How many article reviews have you conducted…? Count every report you submitted, including repeat reviews of a revised version of a paper you had reviewed before.
Of those, how many were first reviews of a manuscript that was new to you?
As for what I recommend to journal editors: of 84 first-round recommendations with a real term, 43 were reject and 41 were major revisions. Not one was accept or minor. I have never once opened with an “accept” recommendation. I knew that to be true before doing this exercise.
Who gets a reading list
I had the parser separate the reference list from the report proper, which turned out to be worth doing. Only 31 of my 123 counted reports cite anything at all, and references are 2.7% of everything I have written as a referee. So “report length” and “report length excluding citations” are almost the same number — but which reports carry a bibliography is not random.
First reports only. Small cells — the top-3 row is 1 of 14. Citing is almost entirely a first-round activity: 32% of first reports, 5% of second ones.
It runs opposite to length. I write the longest reports for the top three and almost never provide a reading list (1 of 14). I write the shortest reports for journals furthest from political science and hand those a reading list about half the time.
That is not inconsistent, I think. At a top-3 journal the authors already know the literature and what I can add is scrutiny. At an area-studies or interdisciplinary venue I am often there precisely because I know a political science literature the paper has not met, so the useful thing I can do is name it1.
Citing is also almost entirely a first-round act — 32% of first reports against 5% of second ones. By round two the argument about what manuscripts’ authors should have read is over.
I write more for journals closer to home
First-round reports only, so this is not a round-mix artefact — and the mix runs the other way: the top three have the lowest share of first-round reports (67%) and the longest ones.
A first report for a top-3 journal runs 960 words; for a journal outside political science, 528. Nearly double. That is not an something I did consciously, but I think it makes sense, given career incentives in the discipline.
Two registered reports, and what they cost
I have written about registered reports as an author — that political science offers the format at about eleven journals and uses it almost nowhere. The log lets me say something about the other side of the desk, because two of these 126 reports were for registered reports: one at the Journal of Politics and one at Research & Politics.
They are 2 of 95 manuscripts, and they look nothing alike.
The R&P one stopped at Stage 1. I assessed a design, it got its in-principle acceptance, and that was the last I had to do with it. It was also the rare observational registered report — a natural experiment rather than an experiment — which is the kind the format is supposed not to suit and evidently sometimes does.
The JoP one I saw the whole way through, and my own three reports on it is the most honest picture I can give of what refereeing this format is actually like:
- Stage 1, round 1 — 937 words, recommending reject. The heaviest single report in that journal’s column of my record.
- Stage 1, round 2 — 44 words, recommending accept. Four dozen words. The argument was over, and the article had been a borderline reject. I amended my opinion.
- Stage 2 — 366 words, minor revisions. The results had arrived and there was not much left to say about them, because the design had already been agreed.
That is the format working as advertised, and it is visible in the word counts: the scrutiny is front-loaded into a stage where it can still change the study. Across all three referees the manuscript consumed nine reports, six of them before the in-principle acceptance.
One things I had expected. The first is that a registered report can be more total refereeing than a standard submission, not less — nine reports is the second-highest count on any manuscript I have touched. The work moves earlier; it does not obviously shrink.
The ledger
Here is the question the whole exercise was for: have I done my share? I have 126 reports given. Against that, my own submissions consumed referee labour — 86 attempts at journals, which used 163 referees that I know of.
That sounds like a clean subtraction. It is not.
The same record, five ways. Dark bar is the accounting I think is defensible: first-round reports on both sides, with the obligation divided among coauthors.
The spread is −132 to +57 on one unchanged record, and every step is arguable:
- Do you charge a flat three reviews per submission? As far as I can tell, that norm comes from a blog post by Ben Lauderdale, not from any study, and when Djupe and Walker actually asked, agreement was lukewarm. It also charges me for desk rejects: 8 of my 13 consumed no referee at all. My submissions have really cost 2.47 reviews each, not three.
- Do you divide by coauthors? A report consumed by a four-author paper is not one report I owe. Dividing flips the sign. But it should probably only count coauthors who are themselves in the reviewer pool, and graduate students largely are not — which I cannot easily compute (lots of difficult cases), so I have not.
- Do you compare like with like? My giving side counts re-reviews; the consuming side counts distinct referees. Only first-round-against-first-round is honest.
So the defensible number to my mind is +23, a modest surplus — and I want to be clear that I chose the assumptions that produce it, which is exactly the problem. “Am I pulling my weight” has no agreed denominator. I am going to right a little R-package for a future blog post that will allow people to enter their info and get assumptions back.
One known bound, stated rather than hidden: 36 of my submissions went through at least one round of revisions, so roughly 43 later rounds of refereeing happened on my behalf that my ledger does not count. At two referees each that is ~86 further reports — enough to reverse the sign. Collecting them is an extension to this project.
Where I do not go back
Caveat that belongs here and not in a footnote: "did not go back" conflates I declined with they never asked again, and my invitation log is too thin to separate them. This also only covers journals I said yes to once, so it shows where I did not return, not where I never went.
I have a rule I only half-knew I had: if a journal never shows me the other referees’ reports and never tells me the outcome, I do not review for it again. The log says I keep it. I have re-reviewed for 39% of the journals that share the reports and 5% of the ones that do not.
Journals where the practice is established; cells are small, and the top-3 row rests on a single journal. The ordering survived me rebuilding the classification three times.
The gradient is not prestige. It is distance from my own discipline. Political science journals return the other reports; area-studies, interdisciplinary and non-PS journals mostly do not. Which has an uncomfortable corollary for anyone whose work crosses fields: the further your expertise takes you from your home discipline, the worse the terms you referee on. 46 of my 126 reports are for exactly those venues.
I rebuilt the journal classification three times while writing this — moving Regulation & Governance into political science, and a bunch of communication journals out of it — each time on disciplinary grounds without looking at what it would do. The ordering came out the same every time.
Put the three gradients together and they all run along the same axis. As a journal sits further from political science, it is less likely to show me the other reports, I write it a shorter report, and I am more likely to hand it a reading list. And that is the part of my reviewing that has been growing. I think this exercise will largely curtail my broader reviewing.
How this was built
The data is a local R project based on a data standard I developed that I will post about soon; Claude terminal helped me build out the coding pipeline for it.
Two follow-ups, both of which this post is deliberately not:
Data standards for a publication record like this. A blank cell turned out to be doing four different jobs, so it needed four codes — could not have existed, never established, recoverable but not yet fetched, and withheld, where the journal had the thing and would not show me. There are thirteen rating scales across these journals and three of them run backwards. And your own labour is less retrievable than you would hope.
My reports against my co-referees’ on the same manuscript. The log holds 151 co-referee reports beside my own 126, on the same papers and the same rounds, so the paired comparison — am I harsher, longer, earlier? — is computable.
Tell me what is wrong with it. Reply on LinkedIn or email me.