Debugging a Training Run

Anthropic: Find out what went wrong in a short training run from its logs and configuration, and decide which problem to fix first.

Assignment

The pretraining team trained a small language model for 240 steps as a test before a larger run. The run finished without any error, but the final validation loss is higher than the team expected and the training loss looks unstable. The training code is not available. The team shared the logs of the run and the configuration the run was started with, and wants to know what to fix before the larger run starts.

Note: In one training step, the model reads one batch of examples and updates its weights. The learning rate sets the size of each update. The training data is split into shards, and one pass over all shards is one epoch.

Deliverables

• The problems in this run, each supported by evidence from the logs and the configuration.
• A ranking of these problems by how much they hurt the run, and the one the team should fix first.
• A short closing report of your findings for the pretraining team, including anything they should double-check before relying on the numbers.

Data Description

anthropic_train_log.csv has one row per training step:

• step
• shard_id - the data shard the batch came from
• loss - training loss on that batch
• grad_norm - size of the gradient used in the update
• learning_rate - the learning rate used in that step
• step_time_sec - time the step took, in seconds

anthropic_eval_log.csv has one row per evaluation on a held-out validation set:

• step
• val_loss
• val_accuracy - share of validation tokens the model predicts correctly

anthropic_shard_info.csv has one row per data shard:

• shard_id
• n_examples
• source - where the data of the shard comes from

anthropic_run_events.csv has one row per system event recorded during the run:

• step
• event

anthropic_run_config.json contains the settings the run was started with, including:

• the learning rate schedule - warmup means the learning rate rises from near 0 to its peak over the first steps; cosine decay means it then falls smoothly along a cosine curve
• grad_clip_norm - the gradient clipping limit: when the gradient norm is above it, the gradient is scaled down to this size before the update

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the training system and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: your results, and how clearly you explain what you see in the data and justify the choices you make.