How to create and run invariance tests


Reach out

Currently, invariance tests are only available for NLP. Let us know if you'd like it to be available for other data types as well!

Creating the test

To create an invariance test, first, go to the corresponding project page.


Don't have a project yet?

All tests live inside a project. If you haven’t created a project yet, make sure you create it first. In case you missed it, here is a tutorial about it.

On the upper right corner of the Tests section, you will find a Create a test button. Click on Create a test to create a test.

You will be redirected to the test creation page. The first thing you’ll see, at the top of the page is the test category to select from.


Reach out

If you would like to use other testing frameworks, feel free to reach out so that we can accommodate your needs!

To create an invariance test, select Invariance. After selecting it, the Configuration and Data sections should appear below it.

Invariance tests run on top of synthetic datasets generated in the platform.


Don't have a synthetic dataset yet?

Check out our how-to guide to learn more about generating synthetic datasets for NLP using Openlayer.

The assertion that invariance tests strive to verify is that the model's predictions remain invariant (i.e., unaltered) even though the inputs are perturbed.

Once you’ve defined the test parameters, you can click on Create at the bottom of the page.

Running the test

After creating a test, on the test panel on the test page, the newly created test is added to the table. Hover over the test and click on Run test to run the test.

After the test finished running, you will be able to see information such as the number of passes and the number of failures.

Click on Open to look at the results in detail.

That's it! You've generated and run an invariance test!