RAG Answer Support

Perplexity: Define how well the chosen passages support an answer, and use it to compare two ways of choosing them.

Assignment

Perplexity's answer engine finds candidate passages for each user question with two retrievers, a keyword retriever and an embedding retriever. The language model that writes the answer can read only 300 tokens of passages for each question, so a rule decides which passages it gets.

• Current rule: put the results of both retrievers into one list, sort it by score from highest to lowest, and add passages in that order until the next passage does not fit in the 300 tokens.
• New rule: within each question, rescale each retriever's scores to a 0–1 range, add up the two rescaled scores of each passage so that every passage appears once, remove passages that are copies of another candidate article, and then fill the 300 tokens in the same way.

The answer quality team ran both rules on an evaluation set of 8 questions, with the facts a good answer to each question needs and the passages that state each fact. They want to know whether the new rule gives the answer model passages that support the answers better.

There is no agreed measure of how well the chosen passages support an answer yet, so choosing one is part of the task.

Note: A retriever is a search component that returns the passages most relevant to a question, each with a relevance score. The keyword retriever scores passages by the words they share with the question; the embedding retriever scores them by similarity of meaning, using numeric vector representations of the text. A token is a small piece of text, often a word or part of a word, that the model reads. Each passage has a known number of tokens.

Deliverables

• Your measure of answer support: what counts as a supported fact, what it is divided by, and which questions you include, with the reason for your choices.
• The value of your measure for each rule, and the difference between them.
• A short closing report for the answer quality team: does the new rule support the answers better, and is there anything they should double-check before relying on the numbers?

Data Description

perplexity_queries.csv has one row per question:

• query_id
• query_text

perplexity_required_facts.csv has one row per fact that a good answer needs:

• fact_id
• query_id
• fact_name

perplexity_passages.csv has one row per candidate passage that the retrievers returned. A passage can be used only for the question it was retrieved for:

• passage_id
• query_id
• domain - the website the passage comes from
• published_date
• n_tokens - the number of tokens in the passage
• text

perplexity_passage_facts.csv has one row for each required fact that a passage states:

• passage_id
• fact_id
• stated_value - the value of the fact as the passage states it

perplexity_context_selections.csv has one row per passage that a rule chose for a question:

• rule - current or new
• query_id
• passage_id
• selection_order - the order in which the rule added the passage

Practicalities

You can solve this in Python, SQL, or a mix of both — StrataScratch notebooks support both languages in the same notebook.

Assume this data comes directly from the search system's evaluation exports and hasn't been cleaned, so review it carefully before relying on the numbers. When something looks off, work out what it is and decide how to handle it; when a definition or a final decision is genuinely ambiguous, choose a reasonable option and explain why. There's often more than one valid way to compute a result, so explain the approach you take and the reasoning behind your choices.

What's assessed: how well your definition fits the question, whether you apply it correctly to the data, and how clearly you justify your choices and conclusion. There is no single correct number — a definition different from the one in our solution is fine if it is well argued and applied consistently.