Refusing when the evidence is thin

The most valuable answer is sometimes "I do not know"

Retrieval always returns something. Ask a corpus of baking recipes about tax law and it will hand back its five least-irrelevant chunks with a straight face — similarity is relative, and the top result of a bad set is still the top result.

A system that answers anyway is worse than one that has no documents at all, because its wrong answers arrive with citations attached.

Two gates, and both are needed:

1. Before the call — if the best similarity is below a floor, do not ask the model. The evidence is not there, and paying for a fluent guess is the worst outcome available.
2. After the call — if the answer cites nothing, treat it as ungrounded. It came from the model's own memory rather than your documents, which is exactly what RAG existed to prevent.

Your turn: write answer(question, scored_chunks, floor=0.5) returning (text, grounded). Below the floor, refuse without calling the model at all.

import re

import fake_llm

REFUSAL = "I do not have anything in these documents that answers that."


def answer(question, scored_chunks, floor=0.5):
    """Return (text, grounded). scored_chunks is [(score, text), ...] best first."""
    return (REFUSAL, False)