Appendix C — Persistent misses: a provisional Claude Opus 5.5 diagnostic
This page reports a small follow-up diagnostic, run and scored by Claude Opus 5.5 at David Reinstein’s request. All scores are a provisional first pass. David Reinstein has not yet reviewed them, or the two critiques the model disputes (below). The diagnostic is separate from the one-shot benchmark: its results are not pooled with, and do not change, the ratings or issue-coverage numbers reported in the main chapters and in Results: Critiques & Key Issues.
Question. The earlier one-shot runs (GPT-5.2 Pro, Claude Opus 4.6, Claude Sonnet 4) missed some critiques that human evaluators raised (see Results: Critiques & Key Issues). Do these misses persist with a current model? Where they do, is the cause:
- failing to generate the point;
- failing to prioritise it;
- failing to recognise it when told?
The aim is a short ledger of cases where an expert caught something and the model did not. Such cases can seed harder evaluation items with known answers.
Design
Papers. We used six papers: Acemoglu et al. (2024), Williams et al. (2024), Dullaghan and Zhang (2022), Frech et al. (2023), Clancy (2024) and Green et al. (2025). Each was given as the same PDF used in the historical runs. Bénabou et al. (2023) and Peterman et al. (2025) remain excluded, as elsewhere in this paper.
Model. claude-opus-5-5 was called through the Anthropic Messages API at effort “high”, with adaptive thinking, the PDF as a native document, no tools and no refusal fallback. Each condition was run once, and no call was refused. The total cost was about US$14, including one condition-C rerun on November 6 (see the Williams leakage note below).
Prompt conditions.
- A, standard: the combined evaluation prompt with the key-issues list from Methods.
- B, prioritisation: “What is the single most important problem with this paper’s main claims? Then give your top five, ranked.”
- C, recognition: the evaluator’s critique stated as a claim: “An evaluator argues X. Is this right, and how much does it matter?” C was run only for critiques missed or only partly caught in both A and B.
Reference set. The target critiques were re-checked against the published Unjournal evaluations, not only the internal curation used in Results: Critiques & Key Issues. Two corrections followed:
- The Acemoglu point about the prior-election control came from the evaluation manager, who called it “not a major limitation”.
- The Frech point about the $40,000-per-QALY estimate is only partly supported by the published evaluations.
Results (provisional)
No longer missed. The two clearest earlier misses did not persist:
- Acemoglu et al. Both A and B note that the field experiment’s statistical significance depends on controlling for the 2018 vote share. A adds that the control is “pre-specified and sensible”, which is the balanced answer.
- Williams et al. Both conditions catch the gap between the headline 215 Mha and the 31.6 Mha of regrowth observed in the paper’s training data. B ranks it first. A calls it a “~7-fold multiple of the observed 16-year regrowth” that is “not reconciled”.
Also caught.
- Acemoglu: spillovers and the partisan framing of the field treatment.
- Williams: socioeconomic confounding.
- Frech: the mechanical GDP regression.
- Clancy: the omission of AI risk (rank 4 in B).
- Green: the imputation of 0.01 for unreported effects, and the non-systematic search.
Persistent or disputed.
| Paper | Critique | A / B | C: model’s verdict | Failure type | Status |
|---|---|---|---|---|---|
| Clancy (2024) | The ~2% discount rate is open to the same objection the paper uses to reject forecasters’ lower rates | partly / partly | right; important | internal consistency | persists |
| Dullaghan & Zhang (2022) | Consumer-acceptance answers conditional on >10 kt produced are read as unconditional | missed / missed | partly right; minor | conditional-probability logic, then prioritisation | persists |
| Dullaghan & Zhang (2022) | 1000× units error (FGF2, per kg vs per gram) in the table given to forecasters | missed / missed | cannot assess from the paper | needs external source | persists; not a fair one-shot test |
| Dullaghan & Zhang (2022) | Cost studies (TEAs) presented as predictions rather than conditional estimates | partly / partly | partly right; important | needs external source; prioritisation | partly persists |
| Williams (2024) | “Validation” compares 2030 predictions with 2000–2016 outcomes | missed / missed | partly right; important | needs external source (the table is in the supplement, not in the PDF used) | persists |
| Williams (2024) | Predictors measured over the outcome period (leakage), including soil data that is in the final model | partly / missed; soil leg not raised in either | partly right; minor (rerun with the full published wording) | generation (soil leg), then prioritisation | persists |
| Acemoglu (2024) | Use post-minus-pre change scores in the online experiment | missed / missed | disputes it: the headline specification already controls for the baseline outcome | none established | to be adjudicated |
| Green (2025) | Excluding studies with n<25 limits publication-bias detection | missed / missed | partly right; minor: the paper’s bias methods do not rely on small-study asymmetry | specification judgement | to be adjudicated |
| Frech (2023) | $40,000/QALY estimate stated without sensitivity analysis | partly / partly | partly right; minor | prioritisation | weak item |
Correction to the Williams leakage item (November 6). The first condition-C prompt paraphrased the critique from our internal curation and named only net primary production, burned area and road density. The model judged that version wrong: none of the three is in the final ten-variable model (Extended Data Fig. 2). But the published critique also names soil characteristics. The evaluators’ joint synthesis lists them under Primary Concern 1, and evaluator 2’s leakage section leads with the SoilGrids data. Soil organic carbon is in the final model; it ranks third by Gini importance, and the paper itself attributes its effect to carbon being “higher within and near forests”. Rerun with the full published wording, the model rates the critique partly right, minor. It accepts a “legitimate, partial leakage risk via SOC that the paper does not test”, rejects the other legs for the final model, and asks for a refit without the soil variables. It judges the point minor because forest density and distance to forest dominate. That last step is its own untested claim. The original output is kept, with the ledger marking it as superseded.
Detailed notes on each case, and on how the model’s ratings compare with the evaluators’, are folded into the case notes at the end of this page.
Failure types
- Internal consistency (Clancy). The model calls the regime-exit parameter “judgmental” but does not notice that the paper’s own reason for rejecting the forecasters’ implied rates also undercuts its ~2% rate. When told, it agrees and would rank the point mid-to-high. This is the cleanest case of a point the model fails to generate but recognises.
- Conditional-probability logic (Dullaghan and Zhang). Neither A nor B mentions the conditioning on >10 kt. When told, the model accepts a narrower version: the conditioning makes the null correlation non-diagnostic, and the low bar selects worlds where demand already existed. It then rates the point minor.
- Evidence outside the PDF (Dullaghan and Zhang; Williams). The units error, and the validation table, sit in reference material or a supplement that the one-shot input did not include. The earlier models received the same PDFs, so these were never fair one-shot tests.
- Prioritisation. For several items the model engages once prompted but ranks the point below other concerns. For Frech, the published evaluations make the same call. For the Williams soil-data leakage, which the evaluators list as a primary concern, the model does not.
- Wording of the critique (a design lesson). In the recognition condition, the verdict depends on how the critique is phrased. A paraphrase that dropped the strongest leg of the Williams leakage critique drew a confident “wrong”; the full wording drew “partly right”.
- Magnitude sanity checks now appear to be handled. For Williams, the model also proposes a mechanism: uncalibrated, class-balanced random-forest scores are summed into an “expected area”.
Possible test items
Each persistent miss suggests an item with a known answer:
- Clancy. Ask whether the reason for rejecting the forecasters’ implied discount rates also applies to the paper’s own rate. Known answer: yes. About 1.6 points of the ~2% come from a short-horizon AGI forecast applied as a permanent hazard.
- Dullaghan and Zhang. Ask what the conditional acceptance questions can and cannot show. Grade on whether the answer spots the selection created by the conditioning event.
- Williams. With the supplement attached, ask what period the validation table tests against. Known answer: 2030 predictions against 2000–2016 regrowth, so it does not test forward skill.
- Williams, leakage. Ask whether any predictor in the final model could encode the outcome, and how to test it. Grade on naming soil organic carbon, noting that the other contemporaneous predictors are not in the final model, and proposing a refit without the soil variables.
The two disputed critiques (Acemoglu change scores, Green small-study exclusion) become items only after they are adjudicated. If the model is right, they become “reject a mistaken critique” items.
Contamination
These papers and their published evaluations were public before Claude Opus 5.5’s training cutoff (about mid-2026). The model may therefore have seen them, so a catch is weaker evidence than a miss.
Two points argue against simple recall:
- the model still missed points that appear in the published evaluations, such as Clancy’s discount rate and the Dullaghan conditional forecasts. This is the stronger point: recall should have produced them;
- what it did catch can be found from the paper alone, such as the roughly 7× gap from the paper’s own figures, and the prior-election control in the main results table.
A weaker point: in condition C the model was shown published critiques, so it plainly saw them, and it disputed some rather than deferring. That only weakly suggests it was not recalling the evaluations, since a model can disagree after re-reading the paper either way.
Some influence on the catches cannot be ruled out. Cleaner tests would use papers evaluated after the cutoff, or disguised variants that change names, setting and numbers while keeping the flaw. Disguised variants are also how these cases could become test items the model cannot have memorised.
Other caveats
- Run-to-run variation. Each condition was run once, so catches and misses ranked low in a list could flip on a rerun.
- Scorer. The provisional scorer is the same model family as the model tested.
- Comparability. Condition A matches the prompt of the GPT-5.2 Pro key-issues run, but the model and interface differ, so this is not a like-for-like comparison with the earlier runs.
- Clancy coverage. One of the Clancy evaluation reports was not in our exports, so that paper’s critique set is incomplete.
Case notes
More detail on each case, for readers who want it. Each note is folded: click a heading to expand it. Evaluators are referred to by their number in the published evaluation. The model’s verdicts are quoted from the outputs in the results folder; like the rest of this page, they have not yet been reviewed.
The paper. Clancy asks whether faster science is worth it if it also speeds up dangerous biotechnology. A key parameter is p, the annual probability of staying in the current “epistemic regime”, where historical trends still let us forecast policy effects. In practice 1 − p works as the discount rate.
How p was chosen (§4.2).
- Forecasts of extinction and of very fast growth from the Existential Risk Persuasion Tournament (XPT) imply exit rates of only 0.05–0.7% a year (p = 0.993–0.9995).
- These are rejected as “probably too conservative”, partly because the XPT forecast only to 2100 and “it may be inappropriate to extrapolate their forecasts forward by hundreds of years”.
- The paper’s own rate instead sums 1.6% a year for AI (from an expert panel’s 10–45% credences for AGI by 2043), 0.08% for other extinction risk (a to-2100 estimate) and 0.3% for economic trend breaks: about 2%, so p = 0.98, applied as a constant hazard indefinitely.
The critique (evaluator 2). About 1.6 of the 2 points come from a credence about the next 20 years. Turning that into a permanent annual hazard is a longer extrapolation, from a shorter forecast window, than the one used to reject the XPT rates, so “it is subject to the same criticism”. Evaluator 2 suggested reporting results for lower rates.
Why it matters. It is a straight consistency argument, and the results are very sensitive to the rate. The author response accepts the point and reports the return to science at other rates: about 13,000× at 1%, about 330× at 2% (the paper’s choice) and about 4× at 5%. The main paper did not test sensitivity to p. The author’s defence, that a wrong rate cannot by itself turn “science is good” into “science is very bad”, holds for the sign of the result but not its size.
How the model handled it.
- A: listed p = 0.98 among “judgmental and loosely evidenced” parameters, “a sum of heterogeneous hazard guesses”.
- B: ranked p fourth, objecting that regime exit is “assumed independent of science”.
- In neither did it connect the paper’s reason for rejecting the XPT rates to the source of its own 2%.
- C: “right, important”. It extended the point (the 0.08% component is also a to-2100 estimate projected forward), added that the extrapolation cuts both ways (if AGI does not arrive by the 2040s, a constant 1.6% is not warranted either), and would rank it mid-to-high.
As a test item. A clean failure to generate a point the model recognises at once. Everything needed is on two adjacent pages of the paper. The skill is holding the paper to its own standard; the model’s habit is to critique each parameter in isolation.
The paper. A forecasting study of cultivated-meat production, with a small panel (about six forecasters). Two “signpost” questions on consumer acceptance are explicitly conditional on more than 10 cumulative kilotons being produced before the end of 2031: large-scale protests (18%) and a majority of a representative sample willing to try it (69%). The summary nonetheless concludes that “Consumer approval did not appear as a major constraint”, citing these numbers, and separately reports that consumer approval was “not significantly correlated” with the production forecasts.
The critique. Evaluator 1: the conditioning “effectively screens off correlation” and “undermines the claim that ‘consumer approval did not appear as a major constraint’”. An evaluation-manager note in the same published report, which evaluator 1 endorsed, gives the logic: a forecaster could expect consumer rejection and protests, yet think that in the unlikely world where consumers like it, 10 kt becomes far more likely.
Why it is valid. Conditioning on 10 kt being produced selects the worlds where demand was already adequate, so high conditional acceptance is partly built in and says little about whether acceptance is a constraint on getting there. The null correlation is also uninformative: the acceptance questions presuppose production, and with about six forecasters a correlation test has almost no power. The claim is unconditional; the evidence is conditional.
How much it matters. A clean conditional-probability error behind a policy-relevant conclusion (don’t worry about demand; fund supply). But evaluator 1 filed it under methods and reporting; their most serious concerns were about framing, such as cost studies presented as forecasts.
How the model handled it.
- A and B: discussed conditioning only on funding and on “no transformative AI”, never the 10 kt condition, although it is printed in the sentence making the claim.
- C: “partly right, minor”. It pushed back reasonably (conditional questions can separate demand from supply), then conceded the core: the 10 kt bar “may already select worlds where demand was adequate”, the null correlation is “unsurprising and non-diagnostic, especially with about six forecasters”, and the evidence supports only “conditional on early product, outright rejection seems unlikely”. It rated the point minor because the acceptance claim is “a one-bullet side observation”.
As a test item. Noticing what a conditioning event does to the evidence, in plain sight. Less clean than Clancy, since once told the model partly agrees and partly argues. It also shows a difference in prioritisation: the model asks whether the headline numbers rest on a point; evaluators also ask whether the paper claims more than its evidence supports.
How the 215 Mha is built. The random forest is trained on samples “balanced with respect to the levels of the dependent variable”: half regrowth, half non-regrowth. Each pixel’s score is read as “the probability of natural regeneration per pixel”, multiplied by pixel area and summed into an “expected area”.
Why that inflates the total. Regrowth occupies a small share of the eligible area; the paper’s own benchmark is 31.6 Mha over 2000–2016. Scores from a 50/50 training set reflect a 50% base rate, so they run high, and summing them over hundreds of millions of hectares inflates the total. A prior-shift correction, or recalibration (Platt or isotonic) against true prevalence, would address this. None is reported.
What the model said. In B, ranked first: the magnitudes are “largely set by the chosen class ratio, not by the real prevalence of regrowth”. It also called the ±0.22 Mha interval “spurious precision”.
Relation to the evaluations. The evaluators flag the gap and the missing hindcast, and explain it mainly by socioeconomic confounding (the model learned from places where social conditions also favoured regrowth). None of the published Williams evaluation documents mentions calibration, class balance or prevalence. Each explanation could account for part of the gap.
A second consequence. The 87.9% accuracy is measured on a balanced test set. At realistic prevalence, precision for “will regenerate” would be lower. Extended Data Fig. 4 already shows accuracy falling to about 75–82% away from training points.
The final model behind the maps, the 215 Mha and the 87.9% has ten biophysical variables (Extended Data Fig. 2): four bioclimatic principal components, biome, distance to forest, forest density, land use, soil organic carbon and soil pH.
- Net primary production and burned area were candidates in variable selection but did not make the final ten.
- Road density is socioeconomic and appears only in the comparison model with socioeconomic variables (accuracy 0.886 vs 0.880). Leakage there would inflate that model, which narrows a gap the paper already calls small.
- Forest density and distance to forest use 2000 tree cover for training and are updated to 2018 only for prediction, which is the right way round.
- Soil organic carbon is the remaining route. Evaluator 2 suggests the SoilGrids layers may have been built using about 2010 land-cover data that include forest classes, and soil organic carbon ranks third in importance. How much this matters depends on how SoilGrids was built, which cannot be checked from the paper. The test is the one the model proposes: refit without the soil variables.
So, as the rerun found: partly right. The other legs do not touch the headline; the soil leg does.
The critique. Evaluator 1, under suggested robustness checks: “The outcome measure should not be post-treatment means but the difference between pre-treatment and post-treatment.” The report quotes §4.6, that “the magnitudes … become somewhat smaller when we control for the pre-treatment values”, and asks why that specification is not the main one in the online experiment. The evaluation summary restates the concern; we found no correction or author response on it.
What the paper shows. Table 1 (online experiment) runs five specifications. Column 2 adds the baseline outcome; column 5 adds the baseline outcome and all controls. The headline vote effect, 3.7 points, equals columns 2 and 5 (0.037), against 0.054 with no controls (column 1), and the notes to Figure 3 say the headline uses “the full set of pre-treatment variables”. So the baseline-controlled specification appears to be the main one, which is the model’s reading. The §4.6 wording, which presents baseline control as an alternative, explains why the evaluator read it the other way.
Change scores versus controlling for the baseline. A change score is a restricted version of regressing on the baseline: it forces the baseline coefficient to equal one. Under randomisation both are unbiased, and controlling for the baseline is generally more efficient. The model makes this point correctly. It adds a smaller one: it is unclear whether the baseline items on valuing institutions match the endline items.
Status. If adjudication confirms the reading of Table 1, this becomes a “reject a plausible but mistaken critique” item: the model held its position with table citations rather than deferring.
The paper. A meta-analysis of RCTs aimed at reducing meat and animal-product consumption. Pooled effect SMD = 0.07 [0.02, 0.12] across 112 interventions. Inclusion requires at least 25 subjects per arm, or at least 10 clusters. Publication bias is handled with a Hedges (1992) step-function selection model (corrected estimate 0.01) and Mathur’s worst-case estimate from non-affirmative studies (0.02), with a significance funnel plot. Headline: reducing consumption is “an unsolved problem”.
The critique (evaluator 2). Excluding small studies is “unwarranted”: it “makes publication bias harder to correct for and harder to detect”, by narrowing the range of standard errors that bias methods exploit. At least 16 studies were dropped for size alone.
Background. Funnel-asymmetry methods (Egger regression, PET-PEESE, trim-and-fill) detect bias from the relation between effect size and standard error, so they need a wide spread of standard errors, and the smallest studies are the most informative. Selection models, which the paper uses, estimate publication probability mainly from the relative frequency of significant and non-significant results, and rely on that spread less.
The model’s counter-argument (“partly right, minor”).
- The paper uses selection models, not asymmetry methods.
- The cutoff is mild: at 25 per arm the standard error of an SMD is about 0.28, and Fig. 3 still spans roughly 0.04 to 0.6.
- Dropping small studies removes the place where small-study bias concentrates, even if it makes the remaining bias harder to model.
- The corrected estimates are already about zero, so undetected bias would push the true effect lower and reinforce the paper’s null headline.
It also points to a more useful gap: no bias correction is reported for the red and processed meat subset (SMD = 0.25), where the paper makes a positive claim.
Why both views are defensible. As a general methods point, evaluator 2 is right: the rule discards the most bias-revealing data and adds a researcher choice. As a question about this paper’s conclusions, the model is largely right, since any missed bias favours the paper’s own null headline. A good answer engages with which bias methods the paper uses and which claim is at stake.
Strengths.
- Long lists (11–13 issues per paper in A; five ranked in B) that cover most of what the evaluators raised.
- Careful internal cross-checks, citing specific appendix tables and figures.
- Sometimes a sharper mechanism than the evaluations give (the Williams calibration point; the Clancy extrapolation cutting both ways).
- When handed a critique, it pushes back with evidence rather than deferring: it disputed two of the nine shown in C.
Weaknesses.
- It criticises parameters one at a time rather than checking whether the paper applies its own standards consistently (Clancy).
- It misses selection and conditioning in plain sight (Dullaghan and Zhang).
- It judges importance by whether the headline number rests on a point, so it downgrades overreach and framing problems that evaluators treat as major. It found Clancy’s omission of AI risk but ranked it fourth; evaluator 3 called it the “major worry”.
- It cannot catch points that need sources outside the PDF; these are not really reasoning failures.
Ratings. Condition A also produced the usual 0–100 ratings. Overall ratings against the published evaluators’ ratings:
| Paper | Evaluators | Evaluator mean | Opus 5.5 (A) | Difference |
|---|---|---|---|---|
| Acemoglu et al. (2024) | 85, 88 | 86.5 | 66 | −20.5 |
| Clancy (2024) | 71, 80 | 75.5 | 66 | −9.5 |
| Green et al. (2025) | 75, 39 | 57 | 66 | +9 |
| Williams et al. (2024) | 50, 50 | 50 | 55 | +5 |
| Frech et al. (2023) | 51, 30 | 40.5 | 30 | −10.5 |
| Dullaghan and Zhang (2022) | no numeric ratings | — | 50 | — |
The mean difference is about −5 points, in the same direction as Claude Opus 4.6 in the main benchmark (Results: Ratings). The ratings are compressed: the three strongest papers all get 66, the paper the evaluators rated highest is marked down most, and on Green, where the two evaluators differ by 36 points, the model lands between them. With five papers and one run, this is illustration, not evidence.
Materials. The runner is scripts/run_headless_claude_misses.py. Model outputs, raw API responses, the claims used in condition C and the provisional ledger with verbatim quotes are in results/claude_opus_5_5_high_misses_2026-11-04/. A longer note is in docs/persistent_misses_opus55_2026-11-04.md.