Feature Launch Impact

OpenAI: Estimate the effect of a feature that was launched country by country without a randomized test, and recommend whether to launch it everywhere.

Assignment

Voice conversations were added to the ChatGPT mobile app country by country, without a randomized test. Four countries got voice in the week of 5 May 2025, four more in the week of 26 May, and four had not received it by the end of June. The product team shared weekly app metrics per country, the rollout plan, and a list of market events that the team keeps for context. They want to know whether the launch increased engagement and whether to launch voice in the remaining countries.

Choosing the primary engagement metric is part of the task.

Note: In a randomized test (an A/B test), a feature is given to a random group of users and compared with everyone else at the same time. Here the feature was given to whole countries on different dates instead.

Deliverables

• Your primary engagement metric, with the reason you chose it, and the estimated effect of voice on it in the launched countries.
• How the effect changes over the weeks after launch, and the effect the team should expect in the long run.
• A short closing report of your findings for the product team, including your recommendation on launching voice in the remaining countries, how much they can trust the estimate, and anything they should double-check before relying on the numbers.

Data Description

openai_weekly_app_metrics.csv has one row per country and week, from the week of 31 March to the week of 23 June 2025:

• week_start - the Monday that starts the week
• country
• weekly_active_users - users who used the mobile app at least once that week
• avg_daily_active_users - the average number of users active on a day of that week
• messages_sent - typed messages sent that week
• voice_sessions - voice conversations started that week

openai_voice_rollout.csv has one row per country:

• country
• region
• launch_week - the week voice became available; empty if it had not launched by the end of the period

openai_market_events.csv has one row per market event:

• start_date
• end_date
• country
• description

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the product analytics warehouse and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: your results, and how clearly you explain what you see in the data and justify the choices you make.