​
Definition

The hallucination test measures the extent to which the generated answer contains information that is not supported by or contradicts the given context. This metric is essentially the complement of faithfulness, identifying when your LLM generates unsupported or fabricated information.

​
Taxonomy

  • Task types: LLM.
  • Availability: and .

​
Why it matters

  • Hallucination detection is critical for maintaining trust and accuracy in AI-generated responses, especially in high-stakes applications.
  • This metric helps identify when your model is making up facts, providing unsupported claims, or contradicting the given context.
  • It’s essential for RAG (Retrieval-Augmented Generation) systems where responses should be strictly grounded in the provided information.
  • Lower hallucination scores indicate better adherence to factual accuracy and context consistency.

​
Required columns

To compute this metric, your dataset must contain the following columns:
  • Outputs: The generated answer/response from your LLM
  • Context: The provided context or background information
This metric relies on an LLM evaluator judging your submission. On Openlayer, you can configure the underlying LLM used to compute it. Check out the OpenAI or Anthropic integration guides for details.

​
Test configuration examples

If you are writing a tests.json, here are a few valid configurations for the hallucination test: 
[
  {
    "name": "Hallucination below 0.1",
    "description": "Ensure that generated responses have minimal hallucination with a score below 0.1",
    "type": "performance",
    "subtype": "metricThreshold",
    "thresholds": [
      {
        "insightName": "metrics",
        "insightParameters": null,
        "measurement": "hallucination",
        "operator": "<",
        "value": 0.1
      }
    ],
    "subpopulationFilters": null,
    "mode": "development",
    "usesValidationDataset": true,
    "usesTrainingDataset": false,
    "usesMlModel": false,
    "syncId": "b4dee7dc-4f15-48ca-a282-63e2c04e0689"
  }
]