# Answer Relevancy and Correctness

Distinguishing whether an answer addresses the question from whether what it says is true — two separate failures.

**Stands alone:** nothing in the curriculum requires this first, and nothing yet builds on it.

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

This concept matters because every time you ask an artificial intelligence for help, you face two distinct risks: it might talk past your question, or it might lie with confidence. Understanding the difference between these two failures is the foundation of knowing whether any automated system you build or use actually works. Without this distinction, a system can appear to respond politely and fluently while providing information that is either irrelevant to your specific need or factually wrong. Mastering this separation allows you to design tests that catch these errors independently, ensuring that you are not misled by impressive-sounding but empty or incorrect output. It is the first step toward reliably measuring the quality of any system that generates text.


## The intuition

Imagine you walk into a store and ask a clerk: 'Do you have a blue cotton kurta in size M?' Now consider three possible replies:

1. 'We have kurtas in the back, but I think they're all polyester, and we only go up to size L.'
2. 'Yes, we have blue cotton in size M — right over there on the third shelf.'
3. 'The weather is lovely today, and the parking lot is almost full.'

Reply 1 addresses the topic of kurtas, sizes, and fabric, so it is somewhat relevant to your inquiry. However, it does not confirm whether the specific item exists; it implies the answer might be no, but without directly stating it. It is partially relevant but unclear on correctness.

Reply 2 is both relevant and correct. It directly answers 'yes' to your request and provides a location. If the clerk is truthful, this is a perfect response.

Reply 3 is neither relevant nor correct. It ignores your question entirely. It might be true that the weather is lovely, but it does not help you find a kurta.

Now consider two more subtle failures:
- An answer that is relevant but false: 'Yes, we have that kurta in red.' This response addresses your request for a kurta (relevant) and confirms availability, but if the red one is out of stock while the blue one is not, the factual claim is wrong (incorrect).
- An answer that is true but irrelevant: 'The store closes at 9 PM.' This statement may be factually true, but it does not answer whether the kurta is in stock.

This distinction is crucial. You must check an answer against two different standards independently: does it address the question asked (relevancy), and are its claims supported by facts or evidence (correctness)? An AI system can excel at one while failing the other.


## An analogy

Think of a customer service representative on a phone line. You call and ask: 'What is the refund policy for a damaged item?'

The representative has two distinct jobs: to understand your question and stay on topic (relevancy), and to provide accurate information (correctness). These are separate skills that can fail independently.

A representative who says, 'Our return window is 30 days,' is being relevant because they are discussing returns, which relates to your question. However, if the actual policy is 14 days, they are incorrect.

A representative who says, 'We have a loyalty program with points,' is being irrelevant. They heard the word 'policy' and went off on a tangent about a different topic, even if every word they say about the loyalty program is factually true.

Now push this analogy to its limit: imagine the representative is not a human but a machine that has read millions of customer-service transcripts. It has learned patterns of what a 'good answer' looks like — fluent, polite, and on-topic-sounding — but it does not have a fixed database of policies. Its 'knowledge' is a statistical guess about what words usually follow each other. This is where the analogy becomes crucial: the machine can produce an answer that sounds perfectly relevant and confident but is entirely fabricated. A human representative, when unsure, can say 'Let me check'; a machine often cannot easily admit uncertainty unless specifically trained to. This is why relevancy and correctness are evaluated separately: a fluent-sounding answer that addresses the question can still be completely wrong in its facts, and evaluation systems must catch both types of failures using different methods.


## Definition

Answer relevancy measures whether a generated response directly addresses the question that was asked; answer correctness measures whether the specific claims made in the response are true and supported by evidence or reference data. These are two independent axes of quality that must be scored separately to accurately judge an AI's performance.


## How it connects

This concept is the entry point to the broader discipline of LLM Evaluation. It connects to neighbouring topics in your library: Word Overlap Metrics compares a generated answer to a correct example by counting shared words — but it cannot judge whether the answer is relevant to the question, only whether it resembles a fixed reference string. Deterministic Validators give unambiguous pass/fail results on format and structure, which is the correct layer for catching irrelevancy of a mechanical kind (like answering in the wrong language) but never for judging truth. Gold and Adversarial Sets require you to label expected behaviour — and labelling relevancy vs correctness separately is how you build a dataset that can expose both failure modes.


## The mathematics

The branch of mathematics this concept rests on is not advanced — it is arithmetic and elementary set theory, the mathematics of counting and categorising. There are no derivatives and no vectors here. The point of this concept is to *separate* two qualities, so the mathematics is about counting what matches and what does not, and expressing that as a simple fraction.

To measure relevancy, you start with the question and the answer. A 'relevant' answer, roughly speaking, covers what was asked. A crude but workable way to measure this is to count how many of the important words (the key terms or concepts) in the question appear in the answer. If the question is 'What is the refund policy for damaged items?' and the answer is 'Our return window is 30 days', the key terms might be 'refund', 'policy', 'damaged'. The answer contains none of these exact words. And yet it is relevant — it talks about 'return' which is a synonym for refund. This is exactly why exact word counting fails, and why a human or a model must judge relevance instead. But whatever the judge is, the mathematical operation at the core is: check whether the *content* of the answer addresses each part of the question, and sum up how many parts are addressed.

To measure correctness, you compare the claims in the answer against a reference or evidence. The operation is comparison, one claim at a time. The mathematics is again counting: how many of the claims are supported; how many are contradicted; how many are not mentioned at all. A correctness score might be the fraction of claims that are true relative to the total number of claims.

Both measures, at their simplest, are ratios — a count of successes divided by a count of opportunities. You will do the arithmetic by hand with real numbers in the next stage.

Let Q be the set of parts in the question that must be addressed (key concepts or sub-questions). Let R be the set of those parts that the answer actually addresses.

Relevancy score = |R| / |Q|

The vertical bars mean 'the number of items in the set'. The score is between 0 and 1; 1 means every part of the question was addressed.

Let C be the set of claims in the answer. Let T be the subset of those claims that are true, when checked against evidence or a reference.

Correctness score = |T| / |C|

Again, this is a fraction between 0 and 1; 1 means every claim in the answer is supported by evidence.

In practice, the 'sets' of parts and claims are not always easy to define, and the counting may involve a model or a human judge rather than exact rules. But the underlying mathematics is still this basic ratio of count — it is a fraction, a ratio, the same arithmetic you have just done by hand.


## A worked example

```
You ask a system: 'What is the refund policy for a damaged item within 14 days?' The system answers: 'You can return damaged items within 30 days for a full refund.'

We will score each axis separately.

Step 1: Relevancy check. Key parts of the question: (1) refund policy, (2) damaged item, (3) 14-day window.   # We break the question into three distinct parts that a relevant answer must address.
Step 2: Answer coverage: (1) mentions 'refund' — yes. (2) mentions 'damaged items' — yes. (3) mentions '14 days' — no, it says '30 days'.   # Two of three parts are addressed, but the third is not — the answer contradicts the question's premise rather than ignoring it.
Step 3: Relevancy score = 2 addressed parts ÷ 3 total parts = 0.67   # A crude relevancy ratio. Not the final word, but it shows the counting logic.
Step 4: Correctness check against the reference policy: 'Damaged items can be returned within 14 days for a full refund.'   # Now we compare claims to evidence, not to the question.
Step 5: Answer claims: (1) 'can return damaged items' — matches reference. (2) 'within 30 days' — contradicts reference's 14 days. (3) 'full refund' — matches reference.   # Each claim in the answer is checked one by one against the reference.
Step 6: Correctness score = 2 true claims ÷ 3 total claims = 0.67   # Same ratio logic, applied to a different comparison.

Result: Relevancy score = 0.67, correctness score = 0.67. The answer is partially relevant and partially correct — but the two failures are different in nature. Relevancy loses points for missing the 14-day window; correctness loses points for contradicting it. The fix is also different: to improve relevancy, the answer should explicitly address the 14-day constraint; to improve correctness, the answer should state 14 days, not 30, regardless of how relevant it sounds.
```


## From scratch, in Python

```python
def check_relevancy(question_parts, answer):
    """Check if each part of the question is addressed in the answer."""
    addressed = []
    for part in question_parts:
        # A naive verbatim match; real systems use synonyms and semantics
        if part.lower() in answer.lower():
            addressed.append(True)
        else:
            addressed.append(False)
    count = 0
    for ok in addressed:
        if ok:
            count = count + 1
    return count / len(question_parts), addressed

def check_correctness(claims, reference):
    """Check if each claim in the answer is supported by the reference."""
    supported = []
    for claim in claims:
        if claim.lower() in reference.lower():
            supported.append(True)
        else:
            supported.append(False)
    count = 0
    for ok in supported:
        if ok:
            count = count + 1
    return count / len(claims), supported

question_parts = ["refund policy", "damaged item", "14 days"]
answer = "You can return damaged items within 30 days for a full refund."
claims = ["can return damaged items", "within 30 days", "full refund"]
reference = "Damaged items can be returned within 14 days for a full refund."

rel_score, rel_checks = check_relevancy(question_parts, answer)
cor_score, cor_checks = check_correctness(claims, reference)

print("Relevancy parts:", rel_checks)
print(f"Relevancy score: {rel_score:.2f}")
print("Correctness claims:", cor_checks)
print(f"Correctness score: {cor_score:.2f}")
```


## The same thing, with a framework

While RAGAS is the common library, the essence is:

```python
from ragas import evaluate
from ragas.metrics import answer_relevancy, answer_correctness
from datasets import Dataset

# A dataset with one row: question, answer, and reference evidence
data = {
    "question": ["What is the refund policy for a damaged item within 14 days?"],
    "answer": ["You can return damaged items within 30 days for a full refund."],
    "reference": ["Damaged items can be returned within 14 days for a full refund."]
}
dataset = Dataset.from_dict(data)

# Run the evaluation
results = evaluate(dataset, metrics=[answer_relevancy, answer_correctness])

# Score is a DataFrame of per-metric values
print(results)
```

Underneath, this calls a model judge with a prompt asking separately for a relevancy score and a correctness score. The model evaluates semantically rather than by verbatim string matching. The scores are still ratios or categorical scores computed from the model's judgement, but the 'comparison' is done by the model's understanding, not by `in`.

The framework version is what a developer actually writes in production for RAG evaluation. It gives you a per-example score for each metric, which you then aggregate and track across versions. The key point to take from this is that the library is doing the same counting and comparing you did by hand — just using a model to decide what counts as 'addressed' and what counts as 'supported'.


## What the code does

This code takes the same logic and packages it into two **functions**. A function is a block of reusable code that we give inputs to and get a result from. We define `check_relevancy` to take two inputs: `question_parts` (a list of strings) and `answer` (a string). Inside the function, we create an empty list called `addressed` — a list is a collection of items kept in order, written with square brackets. We then use a **for loop** to go through each `part` in `question_parts` one at a time. For each part, we check whether it appears in the answer using the `in` operator, which checks for a substring inside the text — 'refund policy'.lower() is looked for anywhere inside the lowercased answer. If it is there, we append (add to the end of) the value True to the list; if not, False. After the loop, we count the number of True values by starting a counter at 0 and adding 1 each time we find a True. Finally, we return two things separated by a comma: the ratio (count divided by the total length of the list) and the list of True/False values.

The second function, `check_correctness`, is identical in shape but takes a list of `claims` and the `reference` text, and checks whether each claim is supported by the reference. This mirrors the two axes of evaluation: relevancy compares to the question, correctness compares to the reference.

At the bottom of the file we set up the data — two lists and two strings — and call each function. The results are packed into variables. When we `print`, we show both the score and the detailed list of which parts/claims passed. This lets you see not just the number but exactly which parts of the question were missed and which claims were unsupported.

The crucial limitation is the verbatim match. If the answer said 'return window' instead of 'refund policy', the relevancy check would mark it as not addressed even though it is a synonymous phrase. Real evaluation systems use model-based judges that understand synonyms and paraphrase. But the *structure* of the scoring — the ratio of successes to opportunities — is exactly what the code shows, and it is the same structure used by RAGAS or TruLens under the hood.


## Common misconceptions

**A relevant answer is automatically correct** — If the answer talks about the right topic, it probably has the right facts.

Relevancy and correctness are independent. An answer can be fully on-topic and address every part of the question while making factual errors in every sentence. The fluency of an LLM makes this worse: it can produce confident, well-structured, on-topic prose that is entirely fabricated. In fact: An answer can be relevant and false, or true and irrelevant. Evaluation must score the two axes separately to catch both failures.

**Correctness just means the format is right** — If the answer is in the right format and mentions the right topic keywords, it is correct.

Format and keyword presence are about surface structure, not truth. A model can answer 'within 30 days' in perfect format. Correctness is about whether the content is supported by evidence or a reference, not whether it looks professional. In fact: Correctness must be grounded — every claim in the answer needs to be checked against a reference or evidence, not just the text's surface characteristics.

**Relevancy and correctness can be merged into one score** — A single overall score is enough; we can just average the two.

A single averaged score hides which axis failed. A system scoring 0.5 could be perfectly relevant and entirely false, or perfectly true and entirely irrelevant, or a mix. The failure mode is different, and the fix is different. Regressions in one axis can be masked by the other. In fact: Score the axes independently, set separate thresholds, and report the two scores separately in every evaluation run.

**If the model knows the answer, correctness is automatic** — A factual answer that the model 'knows' is automatically correct.

Knowledge is not retrieval. A model trained on a corpus has a statistical memory, not a database of verified facts. It can be confidently wrong, especially on niche topics, recent events, or fine-grained specifics. Correctness requires checking against evidence, not trusting the model's confidence. In fact: Correctness must be measured by comparison against a reference, gold answer, or retrieved evidence — never by the model's own confidence.

**If the answer is true, it is relevant** — A factually true statement is always a good answer to the question.

Relevancy is about addressing the specific question, not about being true in the abstract. A true statement that ignores the question ('the store closes at 9 PM' when asked about a refund policy) is still an irrelevant answer. In an agentic system, this can be dangerous: an agent that outputs true but irrelevant tool results has failed its job. In fact: Relevancy is scored against the question alone (what was asked) while correctness is scored against evidence (what is true). A true statement can fail relevancy, and a relevant statement can fail correctness.


## In short

Answer relevancy (does the answer address the question?) and answer correctness (are the claims true?) are two independent axes of LLM evaluation. Score them separately, using a ratio of addressed parts to total parts for relevancy and true claims to total claims for correctness. Ground correctness in evidence or a reference; judge relevancy against the question alone. A fluent, on-topic, confident answer can still be entirely false. Merging the two scores hides the failure mode and makes regressions invisible.


## Where this is used

This concept is used in evaluating every kind of LLM system, and it is used differently depending on the context.

**In retrieval-augmented generation (RAG):** This is the most common place where relevancy and correctness are deliberately separated. In a RAG system, the model first retrieves documents (context) and then generates an answer from them. The book *Transformers and Large Language Models* decomposes RAG quality into context precision, context recall, answer relevancy, and answer correctness. The two answer-side axes are exactly the concepts in this page. The evaluation works like this: for a given question, the system retrieves documents and produces an answer. A relevancy evaluator checks whether the answer addresses the question; a correctness evaluator checks whether each claim in the answer is supported by the retrieved documents. This is how you catch the classic RAG failure where the retriever pulls the right documents but the answer generation produces fluent, relevant-sounding text that is factually unsupported. Developers implement this with libraries like RAGAS or TruLens, which provide built-in evaluators that use an LLM as the judge — they prompt a model to score relevancy and correctness separately, and they report the two scores independently rather than merging them.

**In large language models (general generation):** When a model answers a question without retrieval, there is no external evidence to ground correctness. The evaluation then splits the question differently. Relevancy is still about whether the answer addresses the question. Correctness, however, can only be judged by an external reference — a gold-standard answer produced by a human expert — or by a model judge with its own internal knowledge. This is where the distinction becomes operationally hard: a model judge that checks correctness without a reference is essentially guessing whether the model's claims are true, and human evaluators are needed on a sample of cases to keep the judge calibrated. The developer's task is to build an evaluation dataset with one question, one or more gold answers, and separate rubric items for relevancy and correctness. They then run a judge model on each item and monitor the scores across versions of the system — a regression gate that blocks deployment if relevancy or correctness drops below a threshold.

**In small language models running on devices:** A small model (say, a keyword spotter or a tiny on-device assistant) usually does not do open-ended generation, so relevancy and correctness take a different shape. Relevancy becomes 'does the response match the intent class that was detected' — for example, if the user asks about weather, the model's response must be a weather response and not a music response. This is enforced with a deterministic validator: check the output format and the intent label. Correctness, if the model gives a factual answer (like a conversion or a time), is checked with a simple comparison against the computed value. The developer writes a deterministic validator that checks the output pattern matches the expected format, and a numerical comparison for any facts the model states. This is where the two axes get checked by code rather than by a judge model — the failure modes are more mechanical and cheaper to catch.

**In agentic systems that plan, call tools, and act in loops:** This is where relevancy and correctness become most consequential. An agent receives a user request, plans a sequence, calls tools (like a database or a search API), and then produces a final answer. Relevancy here applies at several levels: is each tool call the relevant tool for the current step, and is the final response relevant to the original request? Correctness applies to the tool results used — the agent must not misreport what a tool returned. A system that has excellent relevancy (it plans the right steps and responds on-topic) but poor correctness (it fabricates the tool result, or misstates a number it pulled from the database) is catastrophically dangerous. Evaluation frameworks for agents break scoring into per-step relevancy (did the agent choose the right tool) and per-step correctness (did the agent accurately state what the tool returned). Developers use trace-based evaluation: they log each step of the agent's run and run an LLM-as-judge over the trace, scoring each step for both axes.

**In everyday applied machine learning (recommendation, vision, forecasting, fraud, speech):** The concepts appear wherever a model generates natural language output. In a recommendation system, the generated explanation ('we recommend this book because you liked similar thrillers') must be relevant to the item and correct about why it was recommended — a common failure being a template that mentions a genre the user has never engaged with. In fraud detection, a system that flags a transaction and explains the flag must have the explanation relevant to the actual fraud signal (not a generic message) and correct about which rule fired. In computer vision, a caption for an image must describe the image (correctness) and the caption must describe what was asked about (relevancy — if the user asks about the background, the model must not describe the foreground). In speech, a transcription service's confidence scores and question-answering over meetings depend on both axes. In each case, the developer builds a small evaluation set with labelled relevancy and correctness scores, runs the system on it, and uses the two scores as regression gates.

**What building software with this looks like in practice:** A developer building a RAG pipeline reaches for evaluation libraries like RAGAS or a custom LLM-as-judge prompt. They define two rubric prompts: one that asks the judge, 'Does the answer address the question? Score 0–5', and one that asks, 'Is every claim in the answer supported by the retrieved context? Score 0–5'. They then run this over a curated dataset, log the two scores separately, and set thresholds: for example, relevancy must be at least 4.0 and correctness at least 4.5 on average to deploy. When the retriever is changed (say, swapping an embedding model), the developer reruns the evaluation and watches the two scores move independently: a better retriever might raise correctness (more relevant context) while leaving relevancy flat (the generator was already good at answering). When done badly — when a team merges relevancy and correctness into a single score — a system can pass evaluation while producing fluent, relevant-sounding but factually wrong answers, and the regression goes undetected until users complain.


---

Read this on the web: https://basavapurushottam.com/learn-ai/answer-relevancy-and-correctness
