RAG eval¶

The eval sub-package scores RAG quality with the metrics established by the RAGAS framework — faithfulness, answer relevancy, context relevancy, hit rate, MRR, MAP, NDCG, context recall, hallucination rate, semantic similarity. Each evaluator takes an EvalInput (query, answer, contexts, optional reference and labelled relevant chunks) and returns an EvalResult (metric name, score in [0,1], optional reason string).

All implementations satisfy:

type Evaluator interface {
    Evaluate(ctx, *EvalInput) (*EvalResult, error)
    Name() string
}

eval.GetEvaluator(name, model) resolves a metric by name. LLM-judged evaluators require a non-nil model and return an explicit error if called without one (this avoids crashing later when the LLM call fails). BatchEvaluator runs a slice of evaluators sequentially and returns all results.

Metrics¶

faithfulness — fraction of answer claims supported by context¶

rag:
  eval:
    type: faithfulness
    model: ${LLM_REF}

RAGAS-style claim decomposition:

1. LLM decomposes the answer into atomic factual claims.
2. LLM checks each claim against the joined context, returning YES or NO per claim.
3. Score = supported / total.

Empty answer → 1.0 (nothing to be unfaithful about). No context → 0.0. Unparseable verdict → unsupported (conservative). Source: pkg/rag/eval/eval.go (FaithfulnessEvaluator).

answer_relevancy — does the answer answer the query?¶

rag:
  eval:
    type: answer_relevancy
    model: ${LLM_REF}

RAGAS approach: ask the LLM to back-generate up to 3 questions the answer would be a direct response to, then measure how close those reconstructed questions are to the original. If an EmbedFunc is wired (set in code, not YAML), uses cosine similarity between query embedding and back-generated question embeddings; otherwise asks the LLM to grade match on 0-10. Source: pkg/rag/eval/eval.go (AnswerRelevancyEvaluator).

context_relevancy — is the retrieved context relevant per chunk?¶

rag:
  eval:
    type: context_relevancy
    model: ${LLM_REF}

In one LLM call, rates each retrieved chunk 0-10 for relevance to the query; averages the normalised [0,1] scores. Empty context → 0.0. Unrated chunk stays 0 (an unrated chunk is not assumed relevant). Source: pkg/rag/eval/eval.go (ContextRelevancyEvaluator).

correctness — answer vs reference answer¶

rag:
  eval:
    type: correctness
    model: ${LLM_REF}

LLM rates the answer 0-10 against a provided reference. Score = 0 if no reference. Use when you have ground-truth answers (e.g. test sets). Source: pkg/rag/eval/eval.go (CorrectnessEvaluator).

hallucination_rate — fraction of unsupported claims¶

rag:
  eval:
    type: hallucination_rate
    model: ${LLM_REF}

Reuses the faithfulness claim decomposition. Score = unsupported / total (HIGHER IS WORSE). Pairs with faithfulness to surface the same data as both a quality and a defect metric. Source: pkg/rag/eval/eval.go (HallucinationRateEvaluator).

context_recall — recall against labelled relevant chunks¶

rag:
  eval:
    type: context_recall

No LLM. Fraction of EvalInput.RelevantChunks (ground-truth relevant texts) that appear in the retrieved set, tolerant to whitespace/case and partial containment. Score = 0 with reason "recall undefined" when no labels are provided. Source: pkg/rag/eval/eval.go (ContextRecallEvaluator).

hit_rate — did retrieval return at least one relevant chunk?¶

rag:
  eval:
    type: hit_rate

No LLM. Score = 1.0 if any context contains the reference (or matches a labelled relevant chunk), else 0. The simplest retrieval quality metric. Source: pkg/rag/eval/eval.go (HitRateEvaluator).

mrr — Mean Reciprocal Rank¶

rag:
  eval:
    type: mrr

No LLM. Score = 1 / rank of the first relevant context (1-based); 0 if none relevant. Source: pkg/rag/eval/eval.go (MRREvaluator).

ndcg — Normalised Discounted Cumulative Gain¶

rag:
  eval:
    type: ndcg

No LLM. Computes DCG with rel / log2(rank+1), divides by ideal DCG. Relevance is binary (1 if context matches reference or a labelled chunk, else fractional word-overlap with the reference). Source: pkg/rag/eval/eval.go (NDCGEvaluator).

map — Mean Average Precision¶

rag:
  eval:
    type: map

No LLM. For each relevant context at rank r, adds relevant_so_far / r; divides by total relevant. Source: pkg/rag/eval/eval.go (MAPEvaluator).

semantic_similarity — embedding-based answer-vs-reference¶

# Configured in code, not YAML; needs an embed function.
eval.NewSemanticSimilarityEvaluator(embedFunc)

Cosine similarity between answer embedding and reference embedding. Returns 0 when reference or embedder is missing. Source: pkg/rag/eval/eval.go (SemanticSimilarityEvaluator).

Choosing between them¶

Group metrics by what they tell you:

For a standard offline RAG eval run, the canonical set is:

rag:
  eval:
    metrics: [faithfulness, answer_relevancy, context_relevancy, context_recall]
    model: ${JUDGE_LLM_REF}

For pure retrieval ablations (no LLM cost):

rag:
  eval:
    metrics: [hit_rate, mrr, ndcg, map, context_recall]

See also¶

• generate — the answer being evaluated; mode choice often affects faithfulness more than retrieval changes.
• retrieve — retrieval changes show up in context_relevancy, mrr, hit_rate, context_recall.
• Eval harness audit — what's wired vs not in the broader eval system.