Typed Diff Questions
A narrow question about a diff can be scored, where a whole code review can't. This was a test of whether a bank of typed questions gives a usable signal.
Independent research. Corpus sampling, model scoring, calibration and evaluation.
Each question about a hunk produces a score that can be calibrated and compared across runs, so the reviewer stops being one judgement and becomes a set of them.
The harness sampled hunks and compared Jev and Haiku against an Opus reference, with repeated scores, class-balanced measures and an always-negative baseline to show how much the label distribution was inflating agreement. The answer moved between repositories, and the reference was another model, so what it measures is agreement.
State
Three recorded scoring runs and a harness I can reuse. Whether a developer would call any of those findings a real bug is still open.