Headline Engagement Analysis

BuzzFeed: Validate content strategies through z-tests and exploratory-confirmatory analysis. Python, Scipy, TextBlob, Pandas.

Assignment

The editorial and growth team at BuzzFeed wants to identify which headline characteristics drive higher engagement (measured by click-through rate, CTR).
The company ran large-scale randomized A/B tests between 2013–2015, comparing different versions of headlines (and sometimes images).

Your task is to explore these experiments and answer research questions about headline effectiveness. Specifically, the project investigates whether features like numbers, question framing, length, sentiment, or timing impact engagement, and whether results replicate across exploratory and confirmatory experiments.

Data Description

The dataset is provided by the Upworthy Research Archive, which contains millions of randomized headline tests. For this project, we use four curated datasets:

• upworthy-archive-confirmatory-packages-03.12.2020.csv — confirmatory experiments designed to validate prior findings
• upworthy-archive-exploratory-packages-03.12.2020.csv — exploratory experiments designed to generate hypotheses
• upworthy-archive-holdout-packages-03.12.2020.csv — holdout data for validation and robustness checks
• upworthy-archive-undeployed-packages-01.12.2021.csv — experiments that were not deployed to production

Each dataset includes fields such as:

• headline — text of the tested headline
• impressions — number of times the headline was shown
• clicks — number of times the headline was clicked
• eyecatcher_id — ID for images used in experiments
• clickability_test_id — experiment identifier

Questions Investigated

The project answers six core research questions:

1. Do headlines containing numbers perform better than those without?
2. Does framing a headline as a question increase engagement?
3. Are shorter headlines more effective than longer ones?
4. Do emotionally charged headlines (positive/negative) perform differently from neutral ones?
5. When the same image is used, do textual differences alone drive engagement outcomes?
6. Do results from exploratory experiments replicate in confirmatory ones?

Practicalities

Analyze the provided data and provide insights using appropriate statistical methods, including:

• Two-proportion z-tests for comparing CTR between groups
• Chi-square tests for categorical comparisons
• Exploratory vs confirmatory replication checks

Make sure the solution includes:

• Clear data exploration steps
• Properly explained statistical methodology
• Concise interpretations of results for business stakeholders

The focus is not only on producing results but also on showing the reasoning and methodology behind A/B testing and its role in decision-making.