Model Version Quality

Anthropic: Choose one quality metric from user signals, and use it to compare two model versions.

Assignment

For two weeks in April 2025, a candidate version of a Claude model answered the conversations of half of the users in a test, while the other half kept the current version. The model team wants one metric that says which version gave better answers, so they can use it for this decision and for future tests. They have shared every response from the test together with what the user did after receiving it.

There is no agreed quality metric yet, so choosing one is part of the task.

Note: a user can rate a response with a thumbs up or a thumbs down, but most users never rate anything. They can also ask for a new answer to the same message (regenerate the response), or copy the response text.

Deliverables

• Your quality metric: what it measures, what it is divided by, and why it reflects the quality of the answers better than the other signals.
• The value of your metric for each model version, and the difference between them.
• A short closing report for the model team: is the candidate version better, and is there anything they should double-check before relying on the numbers?

Data Description

anthropic_model_responses.csv has one row per model response:

• response_id
• user_id
• model_version - current or candidate; each user saw only one version during the test
• response_date
• feedback - thumbs_up, thumbs_down, or empty if the user didn't rate the response
• regenerated - 1 if the user asked for a new answer instead of this one, otherwise 0
• copied - 1 if the user copied the response text, otherwise 0

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the product's logging system and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: how well your definition fits the question, whether you apply it correctly to the data, and how clearly you justify your choices and conclusion. There is no single correct number — a definition different from the one in our solution is fine if it is well argued and applied consistently.