Exam-style
A RAG pipeline scores 0.95 on faithfulness across the evaluation set, yet a manual review of fifty answers finds eight of them factually wrong. What is the most likely explanation?
Reveal the answer
B. Retrieval is returning the wrong passages and the answers faithfully reflect them, so correctness against ground truth also needs scoring
Faithfulness only asks whether the answer follows from the passages that were retrieved, so an answer grounded perfectly in the wrong passage scores high and is still wrong. Correctness compares the answer to ground truth and therefore catches the whole pipeline, which is why the two metrics diverge exactly on retrieval misses. Anything that involves the model inventing detail, including a high temperature or a slack grounding threshold, would push faithfulness down rather than leave it at 0.95.
Q. A RAG answer scores high on faithfulness but is still wrong. How?
A. Faithfulness means the answer follows from the retrieved context; correctness means it matches ground truth. An answer can be perfectly faithful to the wrong retrieved ChunkingSplitting documents into retrievable pieces before embedding them – small enough to match precisely, big enough to still make sense. . Grade both.
Why? Faithfulness catches generation problems; correctness catches the whole pipeline. They diverge on retrieval misses.