ChatGPT Stickiness Test

OpenAI: Define what "stickier" means for a feature test, and use it to compare the two groups.

Assignment

For four weeks, from March 3 to March 30, 2025, half of the users in a test had access to Projects, a ChatGPT feature that groups related chats and files in one place. The other half used ChatGPT without it. The Product lead wants to know whether Projects made ChatGPT stickier, and has shared the list of test users and their daily activity.

There is no agreed definition of "stickier" yet, so choosing one is part of the task.

Deliverables

• Your definition of "stickier": what you measure, which users you count, and over which period, with the reason for your choice.
• The value of your metric for each group, and the difference between the groups.
• A short closing report for the Product lead: did Projects make ChatGPT stickier, and is there anything they should double-check before relying on the numbers?

Data Description

openai_projects_test_users.csv has one row per user in the test:

• user_id
• variant - control (no access to Projects) or treatment (access to Projects)
• is_employee - 1 if the account belongs to an OpenAI employee, otherwise 0

openai_daily_activity.csv has one row per user per day on which the user sent at least one message:

• user_id
• activity_date
• messages_sent - all messages the user sent that day
• project_messages - how many of those messages were sent inside a project

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the product's logging system and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: how well your definition fits the question, whether you apply it correctly to the data, and how clearly you justify your choices and conclusion. There is no single correct number — a definition different from the one in our solution is fine if it is well argued and applied consistently.