openlayer.json

The openlayer.json contains the information Openlayer needs to validate your artifacts, run your AI system on your datasets, and evaluate your tests.

This guide shows how you can write the openlayer.json for your project.

The openlayer.json file has five parts:

• taskType
• model
• datasets
• testsPath
• metrics

​

taskType

Type: string

The taskType must be one of llm-base, tabular-classification, tabular-regression, and text-classification. It corresponds to your Openlayer project’s task type.

Example:

{
  "taskType": "llm-base",
  ...
}

This is needed so that Openlayer can validate the information provided in the model and datasets sections.

​

model

Type: object

The model part of the openlayer.json specifies the commands Openlayer will use to generate predictions with your AI system, and metadata about it.

​

Object attributes

​

modelType

Type: string, required

The type of model. Must be one of shell or full.

You must specify it as full if you are providing a script in batchCommand to run your model and get its predictions. You must specify it as shell if you already computed the model predictions and are uploading model metadata only.

​

runtime

Type: string

The environment runtime to execute the commands specified in installCommand and batchCommand.

This is only required if you want Openlayer to run your model to get its outputs. Refer to the Configuring output generation page for more information. Currently, the supported runtimes are:

​

installCommand

Type: string

The command that gets executed before the run script. Serves the purpose of installing the dependencies needed by your batchCommand script.

For more information about the installCommand, refer to the Configuring output generation guide.

Examples:

​

batchCommand

Type: string

The command that executes your script to get your model predictions.

In general, if you are using one of Openlayer’s SDKs to write your script, it is followed by the placeholder arguments --dataset-path {{ path }} --dataset-name {{ name }}.

For more information about the batchCommand and the placeholder arguments, refer to the Configuring output generation guide.

Examples:

​

outputDirectory

Type: string, default output

Directory where the file with model outputs will be saved.

​

metadata

Type: object

Object with model metadata.

​

datasets

Type: array of Dataset objects

The datasets part of the openlayer.json has an array of Dataset objects. Openlayer will iterate over this array to get your model’s outputs for each dataset.

The Dataset object has a set of common attributes and a set of attributes that depend on the taskType.

​

Dataset object common attributes

The common attributes must always be present, regardless of the taskType.

​

name

Type: string, required

Dataset name.

​

label

Type: string, required

Dataset label. Must be one of validation or training.

​

path

Type: string, required

Path to the dataset file.

​

groundTruthColumnName

Type: string | null

Name of the dataset column with the ground truths.

​

metadata

Type: object

Object with dataset metadata.

​

Dataset object task-specific attributes

The additional attributes you must specify for a dataset depend on the taskType of your Openlayer project.

• llm-base
• tabular-classification
• tabular-regression
• text-classification

​

testsPath

Type: string

Path to a JSON file with test configurations. This field is not needed if you are only creating tests via the UI.

Read more about test configurations on the tests.json guide.

Example:

{
  "testsPath": "tests.json",
  ...
}

​

metrics

Type: object

The metrics part of the openlayer.json allows you to control the metric settings for your project. You can control which metrics are “starred” and which are “selected” for your project, which defines the metrics that appear on the top panel of the project and metrics that should be computed, respectively.

​

Object attributes

​

settings

Type: array of Setting objects

​

Setting object attributes

​

key

Type: string

Metric name. For example, "conciseness" or "accuracy".

​

starred

Type: bool

Bool indicating if the metric is “starred.” Starred metrics are the ones shown on the top panel of your project.

​

selected

Type: bool

Bool indicating if the metric is “selected.” Selected metrics are computed, which allow you to create tests based on them. Unselected metrics are skipped.

Example:

{
  "metrics": {
    "settings": [
      {
        "key": "conciseness",
        "starred": true,
        "selected": true
      },
      {
        "key": "maxCost",
        "starred": false,
        "selected": true
      }
    ]
  }
  ...
}

​

Examples

Below are a few examples of openlayer.json. For additional examples, check out our Template gallery.