API Retry Policy Test

OpenAI: Define a failed API request, and use it to compare a new client retry policy with the old one.

Assignment

For one hour on September 29, 2026, half of the client applications in a test used a new version of the SDK (the client library that applications use to call the API) with a new retry policy, while the other half kept the old one. Clients retry failed calls, and every attempt is logged on its own row. All attempts of one call share the same idempotency key, an identifier the client creates once and reuses on every retry. The API platform team wants to know whether the new policy made the API more reliable for customers, and has shared the list of clients in the test and the API log for that hour.

There is no agreed definition of the API failure rate yet, so choosing one is part of the task.

Note: with the old policy, the SDK retries a failed call once, immediately. With the new policy, it retries up to 3 times and waits longer before each retry (about 1, 2, and 4 seconds). Neither policy retries a call that the API rejected because of a problem in the call itself.

Deliverables

• Your definition of the API failure rate: what counts as one request, which outcomes count as failures, which requests are counted, and why.
• The value of your metric for each retry policy, and the difference between them.
• A short closing report for the API platform team: did the new policy make the API more reliable for customers, and is there anything they should double-check before relying on the numbers?

Data Description

openai_api_clients.csv has one row per client application in the test:

• client_id
• retry_policy - old or new

openai_api_request_log.csv has one row per attempt:

• attempt_id
• idempotency_key
• client_id
• attempt_number - the position of the attempt among the attempts of its request
• logged_by - the service that wrote the row
• http_status - the standard HTTP status code of the response: 200 means success; codes from 400 to 499 mean the API rejected the call, for example 401 (not authorized), 404 (not found), 422 (invalid input), or 429 (the client sent too many calls in a short time and was rate limited); codes from 500 to 599 mean an error on the server side
• logged_at

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the API logs and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: how well your definition fits the question, whether you apply it correctly to the data, and how clearly you justify your choices and conclusion. There is no single correct number — a definition different from the one in our solution is fine if it is well argued and applied consistently.