git:20260724.d4db756 to git:20260728.1bce424

---

name: ragas-evaluation

description: "Authors and runs Ragas - RAG-pipeline evaluation framework with metrics organized into RAG (Faithfulness, Response Relevancy, Context Precision/Recall, Context Entities Recall, Noise Sensitivity), Natural Language Comparison (Factual Correctness, Semantic Similarity, BLEU/ROUGE/CHRF/Exact Match), Agents/Tool-Use (Topic Adherence, Tool Call Accuracy/F1, Agent Goal Accuracy), General Purpose (Aspect Critic, Rubrics-based Scoring), Nvidia (Answer Accuracy, Context Relevance, Response Groundedness), and Summarization. Use when the user evaluates a RAG pipeline (retriever + generator) and needs the deepest metric variety in the OSS LLM-eval space."

---

# ragas-evaluation

[rg-gh]: https://github.com/explodinggradients/ragas

The model: assemble a dataset (question + answer + retrieval

contexts + ground truth), import the metrics relevant to the

evaluation goal, run `evaluate()`, and inspect per-metric per-row

scores (per [rg-gh][rg-gh]).

## When to use

- The repo uses LangChain / LlamaIndex / Haystack / direct

retriever→LLM RAG pipelines.

For non-RAG prompt evals, prefer `promptfoo-evaluation`.

For pytest-native LLM evals with a managed dashboard, prefer

`deepeval-evaluation`.

## Step 1 - Install

Per [rg-gh][rg-gh]:

```bash

pip install ragas

```

Or from source:

```bash

pip install git+https://github.com/explodinggradients/ragas

```

## Step 2 - Custom metric quickstart

Per [rg-gh][rg-gh] (verbatim):

```python

import asyncio

from openai import AsyncOpenAI

from ragas.metrics import DiscreteMetric

from ragas.llms import llm_factory

# Setup your LLM

client = AsyncOpenAI()

llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator

metric = DiscreteMetric(

name="summary_accuracy",

allowed_values=["accurate", "inaccurate"],

prompt="""Evaluate if the summary is accurate and captures

key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""

)

# Score your application's output

async def main():

score = await metric.ascore(

llm=llm,

response="The summary of the text is..."

)

print(f"Score: {score.value}")

print(f"Reason: {score.reason}")

if __name__ == "__main__":

asyncio.run(main())

```

`DiscreteMetric` is the pattern for custom rubric-based scoring;

the built-in metrics in Step 3 follow a similar shape but are

preconfigured.

## Step 4 - Dataset shape

Ragas accepts a Hugging Face `Dataset` or `pandas.DataFrame` with

columns matching the metrics being run:

| Column | Required by |

|---|---|

| `question` | All RAG metrics |

| `answer` | Response Relevancy, Faithfulness, NL Comparison |

| `contexts` (list of strings) | Context Precision/Recall, Faithfulness |

| `ground_truth` | Context Recall, Factual Correctness, Answer Accuracy |

| `reference_contexts` | Context-comparison metrics |

See the per-metric pages on [docs.ragas.io][rg-docs] for exact

required-column lists.

[rg-docs]: https://docs.ragas.io/

## Step 5 - Integration with retrieval frameworks

Ragas integrates with LangChain + LlamaIndex retrieval pipelines -

the integration code captures `contexts` from the retriever and

`answer` from the generator into the evaluation dataset

automatically. Consult the per-framework integration docs on

[docs.ragas.io][rg-docs] when wiring; APIs evolve faster than this

skill body and the canonical doc is the source of truth.

## Step 6 - CI integration

Ragas does not ship a first-party CI action. Pattern: run

`evaluate()` in a pytest fixture or a CLI script, compare per-metric

scores against thresholds, fail CI on regression.

```python

from ragas import evaluate

from ragas.metrics import faithfulness, answer_relevancy

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])

assert result["faithfulness"] >= 0.85

assert result["answer_relevancy"] >= 0.80

```

## Anti-patterns

| Anti-pattern | Why it fails | Fix |

|---|---|---|

| Run all 30+ metrics on every PR | Cost + latency explode | Pick 3 - 5 metrics per pipeline (Step 3) |

| Faithfulness without `contexts` column | Metric returns NaN / errors | Pass `contexts` per dataset spec (Step 4) |

| Pin nothing | Ragas + judge-model versions both drift | Pin both in requirements + CI env |

| Skip Aspect Critic for product-specific concerns | Built-in metrics miss the requirement | Custom Aspect Critic + rubric (Step 3) |

## Limitations

- Many metrics require a judge LLM → cost scales with metric count

× dataset size.

- Multimodal metrics need the multimodal extras (`pip install

ragas[multimodal]`); check the per-metric doc on

[docs.ragas.io][rg-docs].

- API surface evolves - pin versions in requirements; the canonical

doc is the source of truth (this skill body curates the mainstream

patterns but does not re-litigate per-method signatures).

## References

- [rg-gh][rg-gh] - repository + install

- [rg-metrics][rg-metrics] - full metric catalog

- [rg-docs][rg-docs] - full documentation including per-metric pages,

integration guides

- `deepeval-evaluation`,

`promptfoo-evaluation` -

sister tools (different framework styles)