Moderation Review Cap

OpenAI: Find the lowest classifier threshold that stays within the daily review limit, and check that it still catches enough violations.

Assignment

The Integrity data science team is setting the score threshold for a content moderation classifier. Items with a score at or above the threshold are sent to human reviewers, who can handle at most 12 items per day. The team also needs the threshold to catch at least 80% of all violations. They shared two exports from the review system for three days of items — one with classifier scores, one with review decisions — and asked for a recommendation before the threshold changes.

Note: Recall is the share of all violations that the classifier sends to review. A violation with a score below the threshold is missed.

Deliverables

• The lowest threshold that keeps every day within the review limit.
• The recall at that threshold, and the range of thresholds that meet both the review limit and the recall target.
• A short closing report of your findings for the Integrity team, including your recommendation on which threshold to use and anything they should double-check before relying on the numbers.

Definitions.

• An item is sent to review when its score is at or above the threshold.
• Test thresholds from 0.00 to 1.00 in steps of 0.01.
• Recall $= \dfrac{\text{violations sent to review}}{\text{all violations}}$

Data Description

openai_moderation_scores.csv has one row per scored item:

• item_id
• scored_date
• classifier_score - from 0 to 1; a higher score means the item is more likely to be a violation

openai_moderation_reviews.csv has one row per review:

• review_id
• item_id
• reviewer_id
• reviewed_at
• decision - violation or no_violation
• appeal_decision - the decision after the author of the item appealed the review decision; empty if there was no appeal

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the review system and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: your results, and how clearly you explain what you see in the data and justify the choices you make.