Learn how to create custom metrics for your project
Write the run.py
run.py
file contains the custom metric logic.Make sure to create a class that inherits from the BaseMetric
from Openlayer’s Python SDK and
implements the compute_on_dataset
method.Below is a sample run.py
for a metric similar to the accuracy:List the dependencies
requirements.txt
file lists all dependencies for your run.py
.Make sure to include
openlayer>=0.2.0a26
as a requirement as well.Prepare the config.json
config.json
specifies how to prepare the environment and run your custom metric. Furthermore, it also
provides additional information about the metric, which is displayed on the platform.For example, your config.json
could look like:metric_name
is the directory created in the previous section.
After you push your metrics to the platform, you will see them appear in the
project metric settings page like so:
metric_name
.However, you can also push multiple custom metrics at once. Assuming you have the
following directory structure with all your custom metrics:metric_name_i
subdirectory with a custom metric, you can push/update them all
at once with:installCommand
and the runCommand
).
However, in development mode, you have the option to run your custom metrics
before pushing to Openlayer. In this case, Openlayer will only leverage the results you
computed instead of executing your code.
This allows you to execute your code in any environment. The only requirement is that
you need to have already generated outputs for the current model + dataset pair
(using openlayer batch).
In order to run your custom metrics on a commit, make sure your metrics directory is
in the same place as your openlayer.json.