Skip to main content
The openlayer.json contains the information Openlayer needs to validate your artifacts, run your AI system on your datasets, and evaluate your tests. This guide shows how you can write the openlayer.json for your project.
If you prefer, you can pick a template from our Template gallery that feels closest to your use case and make edits to its openlayer.json.
The openlayer.json file has five parts:

​
taskType

Type: string The taskType must be one of llm-base, tabular-classification, tabular-regression, and text-classification. It corresponds to your Openlayer project’s task type. Example:
openlayer.json
{
  "taskType": "llm-base",
  ...
}
This is needed so that Openlayer can validate the information provided in the model and datasets sections.

​
model

Type: object The model part of the openlayer.json specifies the commands Openlayer will use to generate predictions with your AI system, and metadata about it.

​
Object attributes

​
modelType

Type: string, required The type of model. Must be one of shell or full. You must specify it as full if you are providing a script in batchCommand to run your model and get its predictions. You must specify it as shell if you already computed the model predictions and are uploading model metadata only.

​
runtime

Type: string The environment runtime to execute the commands specified in installCommand and batchCommand. This is only required if you want Openlayer to run your model to get its outputs. Refer to the Configuring output generation page for more information. Currently, the supported runtimes are:
RuntimeAvailable options
Pythonpython_3_13, python_3_12, python_3_11, python_3_10, python_3_8
NodeJSnode_20

​
installCommand

Type: string The command that gets executed before the run script. Serves the purpose of installing the dependencies needed by your batchCommand script. For more information about the installCommand, refer to the Configuring output generation guide. Examples: 
{
  ...
  "model": {
      "installCommand": "pip install -r requirements.txt",
      ...
  }
}

​
batchCommand

Type: string The command that executes your script to get your model predictions. In general, if you are using one of Openlayer’s SDKs to write your script, it is followed by the placeholder arguments --dataset-path {{ path }} --dataset-name {{ name }}. For more information about the batchCommand and the placeholder arguments, refer to the Configuring output generation guide. Examples: 
{
  ...
  "model": {
      "batchCommand": "python run.py --dataset-path {{ path }} --dataset-name {{ name }}",
      ...
  }
}

​
outputDirectory

Type: string, default output Directory where the file with model outputs will be saved.

​
metadata

Type: object Object with model metadata.

​
datasets

Type: array of Dataset objects The datasets part of the openlayer.json has an array of Dataset objects. Openlayer will iterate over this array to get your model’s outputs for each dataset. The Dataset object has a set of common attributes and a set of attributes that depend on the taskType.

​
Dataset object common attributes

The common attributes must always be present, regardless of the taskType.

​
name

Type: string, required Dataset name.

​
label

Type: string, required Dataset label. Must be one of validation or training.

​
path

Type: string, required Path to the dataset file.

​
groundTruthColumnName

Type: string | null Name of the dataset column with the ground truths.

​
metadata

Type: object Object with dataset metadata.

​
Dataset object task-specific attributes

The additional attributes you must specify for a dataset depend on the taskType of your Openlayer project.
  • llm-base
  • tabular-classification
  • tabular-regression
  • text-classification

​
inputVariableNames

Type: array[string], requiredArray of input variable names. Each input variable should be in a dataset column.

​
testsPath

Type: string Path to a JSON file with test configurations. This field is not needed if you are only creating tests via the UI. Read more about test configurations on the tests.json guide. Example:
openlayer.json
{
  "testsPath": "tests.json",
  ...
}

​
metrics

Type: object The metrics part of the openlayer.json allows you to control the metric settings for your project. You can control which metrics are “starred” and which are “selected” for your project, which defines the metrics that appear on the top panel of the project and metrics that should be computed, respectively.

​
Object attributes

​
settings

Type: array of Setting objects

​
Setting object attributes

​
key

Type: string Metric name. For example, "conciseness" or "accuracy".

​
starred

Type: bool Bool indicating if the metric is “starred.” Starred metrics are the ones shown on the top panel of your project.

​
selected

Type: bool Bool indicating if the metric is “selected.” Selected metrics are computed, which allow you to create tests based on them. Unselected metrics are skipped. Example:
openlayer.json
{
  "metrics": {
    "settings": [
      {
        "key": "conciseness",
        "starred": true,
        "selected": true
      },
      {
        "key": "maxCost",
        "starred": false,
        "selected": true
      }
    ]
  }
  ...
}

​
Examples

Below are a few examples of openlayer.json. For additional examples, check out our Template gallery. 
{
  "taskType": "llm-base",
  "model": {
    "modelType": "full",
    "runtime": "python_3_10",
    "installCommand": "pip install -r requirements.txt",
    "batchCommand": "python run.py --dataset-path {{ path }} --dataset-name {{ name }}",
    "outputDirectory": "output"
  },
  "datasets": [
    {
      "name": "validation_set_october_november",
      "label": "validation",
      "path": "dataset.json",
      "inputVariableNames": ["userQuery"],
      "groundTruthColumnName": "groundTruth"
    }
  ]
}