[All work](/work?type=work)

# Typed Diff Questions

A narrow question about a diff can be scored, where a whole code review can't. This was a test of whether a bank of typed questions gives a usable signal.

Independent research. Corpus sampling, model scoring, calibration and evaluation.

![Two columns of line blocks, the changed lines on the right offset and filled red.](https://cdn.sanity.io/images/qnuj1c4o/production/065ce62d55c2c59793a2b1c4c8ac343172ecd2ce-1600x900.png?w=1280\&q=80\&auto=format)

Each question about a hunk produces a score that can be calibrated and compared across runs, so the reviewer stops being one judgement and becomes a set of them.

The harness sampled hunks and compared Jev and Haiku against an Opus reference, with repeated scores, class-balanced measures and an always-negative baseline to show how much the label distribution was inflating agreement. The answer moved between repositories, and the reference was another model, so what it measures is agreement.

## State

Three recorded scoring runs and a harness I can reuse. Whether a developer would call any of those findings a real bug is still open.

[All work](/work?type=work)