Openlayer supports tests based on custom metrics that you write.

This guide shows how you can write a custom metric in Python and push it to your project using the Openlayer CLI.

​
Prerequisites

We are going to use the Openlayer CLI to push the custom metric to your project.

Therefore, to follow this guide, you must:

  1. Install the Openlayer CLI.
  2. Login with the Openlayer CLI with the command openlayer login.
  3. Link your working directory to an Openlayer project with the command openlayer link.

​
Write the metric

For each custom metric, you must prepare a directory structured as follows:

Below are instructions for each component.

​
Push the metric to Openlayer

Now that you have written your custom metric, you can push it to your Openlayer project.

To push the custom metric, you can run the openlayer metrics push command:

openlayer metrics push -d metric_name

where metric_name is the directory created in the previous section.

After you push your metrics to the platform, you will see them appear in the project metric settings page like so:

In the example above, we pushed/updated a single metric: metric_name.

However, you can also push multiple custom metrics at once. Assuming you have the following directory structure with all your custom metrics:

metrics
├── metric_name_1
├── metric_name_2
└── ...

where each metric_name_i subdirectory with a custom metric, you can push/update them all at once with:

openlayer metrics push -d metrics

​
Development mode

Once you have pushed your metrics to Openlayer, any new commit you push to Openlayer will run your selected custom metrics. You can find which metrics are selected in the project’s metric settings page. Newly pushed custom metrics are selected by default.

You can view the logs for your custom metric computation in the commit overview page:

​
Monitoring mode

Once you’ve pushed your metrics to Openlayer, you can create tests in monitoring mode on your custom metrics. Any custom metric with at least one test associated with it will be run.

You will see the option to create tests on your metric in the “Create tests” page:

​
(Optional) Pre-compute the metric

Typically, the custom metrics are run by Openlayer using the information you provide in the config.json (namely, the installCommand and the runCommand).

However, in development mode, you have the option to run your custom metrics before pushing to Openlayer. In this case, Openlayer will only leverage the results you computed instead of executing your code.

This allows you to execute your code in any environment. The only requirement is that you need to have already generated outputs for the current model + dataset pair (using openlayer batch).

In order to run your custom metrics on a commit, make sure your metrics directory is in the same place as your openlayer.json.

# if you don't have metrics/ alongside your dev commit, fetch them:
openlayer metrics pull --selected

# from the directory containing your openlayer.json
openlayer metrics run