Anthropic: Measure how often a new model is preferred over the baseline in each category, and check whether answer position affects the choice.
The model evaluation team ran a human preference study that compares two models: the baseline, Aspen, and a new candidate, Cedar. In each comparison a rater saw two answers to the same prompt, one in position A and one in position B, and chose the better answer or marked a tie. The team shared the export of all submissions and wants a summary before deciding whether Cedar is ready for wider testing.
Note: Raters can show position bias: they may choose the answer in one position more often, whatever the answer says.
anthropic_pairwise_comparisons.csv has one row per submission:
submission_idcomparison_idrater_idcategorymodel_a - the model whose answer was shown in position Amodel_b - the model whose answer was shown in position Bwinner - A, B, or tiesubmitted_atYou can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.
Assume this data comes directly from the study platform and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.
What's assessed: your results, and how clearly you explain what you see in the data and justify the choices you make.