Checkpoint Release Decision

OpenAI: Decide which new model checkpoint, if any, should replace the production model, based on benchmark results and a safety gate.

Assignment

The research team has trained a new language model and saved six checkpoints during training. The release team must decide whether one of these checkpoints should replace the model that is now in production, and which one. They shared three exports from the evaluation system: the checkpoints, the benchmarks with the team's weights and release gate, and the evaluation runs. Each checkpoint and the production model were evaluated on every benchmark with two random seeds.

Note: A checkpoint is a saved copy of the model at one point in training. A random seed controls the random choices made during an evaluation run, so two seeds of the same checkpoint give slightly different scores.

Deliverables

• The weighted improvement of each checkpoint over the production model.
• The checkpoints that meet your release criteria. The release gate in openai_benchmarks.csv is fixed; choosing any other release criteria is part of the task.
• A short closing report of your findings for the release team, including which checkpoint you recommend, if any, and anything they should double-check before relying on the numbers.

Definitions.

• A model's score on a benchmark is the mean of its seed results.
• Improvement on a benchmark $= \dfrac{\text{checkpoint score} - \text{production score}}{\text{production score}} \times 100$, with the sign reversed for benchmarks where a lower score is better, so a positive improvement always means the checkpoint is better.
• Weighted improvement $= \sum \text{weight} \times \text{improvement}$, summed over all benchmarks.

Data Description

openai_checkpoints.csv has one row per model:

• checkpoint_id - 0 is the current production model
• checkpoint_name
• train_step - the training step at which the checkpoint was saved

openai_benchmarks.csv has one row per benchmark:

• benchmark
• metric
• higher_is_better - 1 if a higher score is better, 0 if a lower score is better
• weight - the weight of the benchmark in the weighted improvement
• release_gate - the minimum score a checkpoint must reach to be released, when the benchmark has one

openai_eval_runs.csv has one row per evaluation run:

• run_id
• checkpoint_id
• benchmark
• seed
• harness_version - version of the evaluation code used for the run
• status
• score
• run_at

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the evaluation system and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: your results, and how clearly you explain what you see in the data and justify the choices you make.