Skip to main content
Tests in Openlayer let you codify expectations for your AI system and data. They help you ensure that your data, models, and outputs remain reliable, safe, and compliant over time. .

​
Where tests apply

Tests are a common layer across all workflows in Openlayer:

Observability

Monitor live requests after deployment.

Offline testing

Validate changes in CI/CD pipelines.

Data quality monitoring

Track the health of your tables and features.
Together, these allow you to test your system both pre- and post-production.

​
Getting started with tests

There are three levels of entry, from fastest to most flexible:

​
1. Apply a bundle

Bundles are pre-packaged sets of tests for common use cases. They allow you to apply broad coverage with a single step, without having to configure each test individually. Examples include:
  • Agentic bundle: evaluate the performance of agentic and RAG systems with metrics like faithfulness, relevance, and more.
  • Usage bundle: track system usage via cost, tokens, latency, and more.
  • OWASP bundle: check for common security issues such as prompt injection, hallucinations, and more.
  • EU AI Act bundle: align with regulatory requirements, including fairness, transparency, and more.
  • Data quality bundle: catch data quality issues such as missing values, duplicates, anomalies, and more.
We are continuously adding and improving bundles, so stay tuned for updates.

​
2. Pick individual tests from the catalog

Openlayer provides 100+ individual tests. You can browse all tests and assemble your own test suite for fine-grained control.

​
3. Define your own tests

If the built-in catalog does not cover your use case, you can create custom tests with custom metrics. This allows you to encode domain-specific checks alongside the standard ones.
By combining bundles, catalog tests, and custom metrics, you can build a test suite that fits your system today and adapts as it evolves.
⌘I