Ragas

Ragas is an open-source library that offers metrics to evaluate large language model (LLM) applications. Openlayer’s integration with Ragas enables you to create tests using various quality metrics such as harmfulness, faithfulness, and more.

Tests with Ragas metrics

When evaluating LLM projects, you can leverage any of the Ragas metrics to create detailed tests. Each test provides:

A pass/fail status.
Row-by-row scoring and justification, provided by the LLM judge.

Configuring the LLM judge

Every Ragas metric relies on an LLM judging your data, and they all use your project’s LLM-as-a-judge settings. To review them, go to project Settings → LLM-as-a-judge, where you can:

Choose the Default LLM that computes the metrics, and use Test connection to confirm Openlayer reaches it.
Add Custom judge instructions, which the judge follows while scoring each row.
Adjust the Sampling settings, which control how data is sampled when computing LLM-based metrics. Ragas metrics are sampled the same way as every other LLM-judge metric in the project.

The judge authenticates with the credentials you store as environment variables, so add the key for the provider you select — OPENAI_API_KEY for OpenAI, or ANTHROPIC_API_KEY for Anthropic, for example. If the key is missing, Openlayer skips the metric and asks you to add the missing secrets to your workspace settings or update the project’s LLM evaluator settings.

Metrics available

The Ragas metrics available on Openlayer listed below.

Metric	Description	Required columns	`measurement` for `tests.json`
Answer relevancy	Evaluates how well the model’s answer aligns with the intent of the question. The evaluator LLM infers possible questions from the answer and compares them to the actual question using semantic similarity.	`input`, `outputs`	`answerRelevancy`
Answer correctness	Measures factual alignment between the generated answer and the ground truth reference. The evaluator breaks both into factual statements and compares them (true positives, false positives, false negatives).	`outputs`, `ground truths`	`answerCorrectness`
Context relevancy	Assesses whether the retrieved context is relevant to the ground truth answer. Each context chunk is judged independently by an LLM for its relevance.	`input`, `ground truth`, `context`	`contextRelevancy`
Context recall	Evaluates how completely the retrieved context supports all the claims present in the ground truth. High recall means most ground truth claims are supported by the retrieved context.	`ground truth`, `context`	`contextRecall`
Faithfulness	Measures how factually consistent the model’s answer is with the retrieved context. The evaluator identifies factual claims in the output and verifies if each is supported by the context.	`outputs`, `context`	`faithfulness`
Correctness	Judges the general factual soundness of the answer. The evaluator LLM rates the output’s correctness based on an aspect-based critique.	`input`, `outputs`	`correctness`
Harmfulness	Evaluates whether the answer contains harmful, unsafe, or toxic content. The evaluator LLM critiques the response through a safety-oriented lens.	`input`, `outputs`	`harmfulness`
Coherence	Measures how logically and linguistically coherent the answer is — i.e., whether it flows naturally and maintains internal consistency.	`input`, `outputs`	`coherence`
Conciseness	Evaluates whether the answer is clear and to the point, without unnecessary verbosity or repetition.	`input`, `outputs`	`conciseness`
Maliciousness	Detects whether the answer exhibits malicious intent, manipulation, or socially undesirable behavior.	`input`, `outputs`	`maliciousness`

Which input is the question

Metrics that need the input column score one input variable as the question. If you set an explicit question column, such as questionColumnName in the config when you publish data through the REST API, Openlayer always uses that column. Without an explicit question column, Openlayer picks the input variable that most looks like the user’s question: chat-style message lists first, then multi-word text, then other strings. It never picks numeric, boolean, date/time, or empty columns, or a column already used as the output, context, or ground truth. If no input variable holds text, Openlayer skips the metric instead of scoring it.

Integrations

LLM Providers

Frameworks

No-Code Platforms

Observability

Shadow AI Discovery

Data Platforms

Evaluation & Quality

Data Labeling

Collaboration

Tests with Ragas metrics

Configuring the LLM judge

Metrics available

Which input is the question

​Tests with Ragas metrics

​Configuring the LLM judge

​Metrics available

​Which input is the question

Tests with Ragas metrics

Configuring the LLM judge

Metrics available

Which input is the question