openlayer.json - Openlayer

The openlayer.json contains the information Openlayer needs to validate your artifacts, run your AI system on your datasets, and evaluate your tests. This guide shows how you can write the openlayer.json for your project.

If you prefer, you can pick a template from our Template gallery that feels closest to your use case and make edits to its openlayer.json.

The openlayer.json file has five parts:

taskType
model
datasets
testsPath
metrics

`taskType`

Type: string The taskType must be one of llm-base, tabular-classification, tabular-regression, and text-classification. It corresponds to your Openlayer project’s task type. Example:

openlayer.json

{
  "taskType": "llm-base",
  ...
}

This is needed so that Openlayer can validate the information provided in the model and datasets sections.

`model`

Type: object The model part of the openlayer.json specifies the commands Openlayer will use to generate predictions with your AI system, and metadata about it.

Object attributes

`modelType`

Type: string, required The type of model. Must be one of shell or full. You must specify it as full if you are providing a script in batchCommand to run your model and get its predictions. You must specify it as shell if you already computed the model predictions and are uploading model metadata only.

`runtime`

Type: string The environment runtime to execute the commands specified in installCommand and batchCommand. This is only required if you want Openlayer to run your model to get its outputs. Refer to the Configuring output generation page for more information. Currently, the supported runtimes are:

Runtime	Available options
Python	`python_3_13`, `python_3_12`, `python_3_11`, `python_3_10`, `python_3_8`
NodeJS	`node_20`

`installCommand`

Type: string The command that gets executed before the run script. Serves the purpose of installing the dependencies needed by your batchCommand script. For more information about the installCommand, refer to the Configuring output generation guide. Examples:

{
  ...
  "model": {
      "installCommand": "pip install -r requirements.txt",
      ...
  }
}

{
  ...
  "model": {
      "installCommand": "npm i && npx tsc",
      ...
  }
}

`batchCommand`

Type: string The command that executes your script to get your model predictions. In general, if you are using one of Openlayer’s SDKs to write your script, it is followed by the placeholder arguments --dataset-path {{ path }} --output-dir {{ outputDirectory }}/{{ name }}. For more information about the batchCommand and the placeholder arguments, refer to the Configuring output generation guide. Examples:

{
  ...
  "model": {
      "batchCommand": "python run.py --dataset-path {{ path }} --output-dir {{ outputDirectory }}/{{ name }}",
      ...
  }
}

{
  ...
  "model": {
      "batchCommand": "node run.js --dataset-path {{ path }} --output-dir {{ outputDirectory }}/{{ name }}",
      ...
  }
}

`outputDirectory`

Type: string, default output Directory where the file with model outputs will be saved.

`metadata`

Type: object Object with model metadata.

`datasets`

Type: array of Dataset objects The datasets part of the openlayer.json has an array of Dataset objects. Openlayer will iterate over this array to get your model’s outputs for each dataset. The Dataset object has a set of common attributes and a set of attributes that depend on the taskType.

`Dataset` object common attributes

The common attributes must always be present, regardless of the taskType.

`name`

Type: string, required Dataset name.

`label`

Type: string, required Dataset label. Must be one of validation, training, or fine-tuning. The non-validation label depends on the taskType: llm-base projects use fine-tuning (the platform rejects training for LLM datasets), while tabular and text classification projects use training (and reject fine-tuning).

`path`

Type: string, required Path to the dataset file. The accepted file formats are .csv, .tsv, and .json (a JSON array of row objects). Note that .jsonl (newline-delimited JSON) is not supported — convert it to a JSON array first.

`metadata`

Type: object Object with dataset metadata.

`Dataset` object task-specific attributes

The additional attributes you must specify for a dataset depend on the taskType of your Openlayer project.

llm-base
tabular-classification
tabular-regression
text-classification

`inputVariableNames`

Type: array[string], requiredArray of input variable names. Each input variable should be in a dataset column.

`outputColumnName`

Type: string, required for shell modelsName of the dataset column that holds the model’s (precomputed) output for each row.This is required when your model has "modelType": "shell" — i.e., you have already computed the outputs and are uploading them rather than letting Openlayer run your model. Without it, any test that depends on the model output (for example, the LLM-as-a-judge and Ragas metrics) silently skips or errors, since Openlayer cannot find the generated text to evaluate.For full models, Openlayer runs your batchCommand and populates the output column for you, so you do not need to set outputColumnName.

`groundTruthColumnName`

Type: string | nullName of the dataset column with the ground truths. This attribute is specific to llm-base datasets — for classification task types, use labelColumnName instead.

`categoricalFeatureNames`

Type: array[string] | []Array containing the names of all categorical features in the dataset.For example, [“Gender”, “Geography”].

`classNames`

Type: array[string], requiredArray of class names indexed by label integer in the dataset.For example, [“Retained”, “Exited”] when class 0 is "Retained" and class 1 is "Exited".

`featureNames`

Type: array[string] | [], requiredArray of all input feature names.

`labelColumnName`

Type: string, requiredName of the dataset column with the ground-truth class label (an integer index into classNames). For classification task types, use labelColumnName for the ground truths — not groundTruthColumnName.

`predictionsColumnName`

Type: stringName of the dataset column with the model’s predicted class label (an integer index into classNames).

`predictionScoresColumnName`

Type: stringName of the dataset column with the model’s per-class predicted probabilities. Each row’s value is a list of floats — one score per class, ordered to match classNames. For example, [0.1, 0.9] for a two-class problem.

For tabular and text classification, the featureNames, categoricalFeatureNames, and classNames you set on each dataset must also be set on the top-level model object. If they are present on the datasets but missing from the model, the server fails the commit.

`categoricalFeatureNames`

Type: array[string] | []Array containing the names of all categorical features in the dataset.For example, [“Gender”, “Geography”].

`featureNames`

Type: array[string] | [], requiredArray of all input feature names.

`targetColumnName`

Type: string, requiredName of the dataset column with the ground-truth (numeric) target value. For regression task types, use targetColumnName for the ground truths — not groundTruthColumnName (which the server rejects as an unknown field for tabular regression).

`predictionsColumnName`

Type: stringName of the dataset column with the model’s predicted (numeric) value. The values must be floats (e.g. 275000.0, not 275000) — a column of integers is rejected at commit with “values … that are not floats”.

For tabular regression, the featureNames and categoricalFeatureNames you set on each dataset must also be set on the top-level model object. If they are present on the datasets but missing from the model, the server fails the commit.

`classNames`

`textColumnName`

Type: string, requiredName of the column with the text.

`labelColumnName`

`predictionsColumnName`

Type: stringName of the dataset column with the model’s predicted class label (an integer index into classNames).

`predictionScoresColumnName`

For text classification, the classNames you set on each dataset must also be set on the top-level model object. If it is present on the datasets but missing from the model, the server fails the commit.

testsPath

Type: string Path to a JSON file with test configurations. This field is not needed if you are only creating tests via the UI. Read more about test configurations on the tests.json guide. Example:

openlayer.json

{
  "testsPath": "tests.json",
  ...
}

metrics

Type: object The metrics part of the openlayer.json allows you to control the metric settings for your project. You can control which metrics are “starred” and which are “selected” for your project, which defines the metrics that appear on the top panel of the project and metrics that should be computed, respectively.

Object attributes

`settings`

Type: array of Setting objects

`Setting` object attributes

`key`

Type: string Metric name. For example, "conciseness" or "accuracy".

`starred`

Type: bool Bool indicating if the metric is “starred.” Starred metrics are the ones shown on the top panel of your project.

`selected`

Type: bool Bool indicating if the metric is “selected.” Selected metrics are computed, which allow you to create tests based on them. Unselected metrics are skipped. Example:

openlayer.json

{
  "metrics": {
    "settings": [
      {
        "key": "conciseness",
        "starred": true,
        "selected": true
      },
      {
        "key": "maxCost",
        "starred": false,
        "selected": true
      }
    ]
  }
  ...
}

Examples

Below are a few examples of openlayer.json. For additional examples, check out our Template gallery.

View example openlayer.json

{
  "taskType": "llm-base",
  "model": {
    "modelType": "full",
    "runtime": "python_3_10",
    "installCommand": "pip install -r requirements.txt",
    "batchCommand": "python run.py --dataset-path {{ path }} --output-dir {{ outputDirectory }}/{{ name }}",
    "outputDirectory": "output"
  },
  "datasets": [
    {
      "name": "validation_set_october_november",
      "label": "validation",
      "path": "dataset.json",
      "inputVariableNames": ["userQuery"],
      "groundTruthColumnName": "groundTruth"
    }
  ]
}

{
  "taskType": "llm-base",
  "model": {
    "modelType": "full",
    "runtime": "node_20",
    "installCommand": "npm i && npx tsc",
    "batchCommand": "node run.js --dataset-path {{ path }} --output-dir {{ outputDirectory }}/{{ name }}",
    "outputDirectory": "output"
  },
  "datasets": [
    {
      "name": "validation_set_october_november",
      "label": "validation",
      "path": "dataset.json",
      "inputVariableNames": ["userQuery"],
      "groundTruthColumnName": "groundTruth"
    }
  ]
}

{
  "taskType": "llm-base",
  "model": {
    "modelType": "shell"
  },
  "datasets": [
    {
      "name": "validation_set_october_november",
      "label": "validation",
      "path": "dataset.json",
      "inputVariableNames": ["userQuery"],
      "outputColumnName": "modelOutput",
      "groundTruthColumnName": "groundTruth"
    }
  ]
}

{
  "taskType": "tabular-classification",
  "model": {
    "modelType": "shell",
    "featureNames": ["CreditScore", "Age", "Gender", "Geography"],
    "categoricalFeatureNames": ["Gender", "Geography"],
    "classNames": ["Retained", "Exited"]
  },
  "datasets": [
    {
      "name": "validation_set",
      "label": "validation",
      "path": "dataset.csv",
      "featureNames": ["CreditScore", "Age", "Gender", "Geography"],
      "categoricalFeatureNames": ["Gender", "Geography"],
      "classNames": ["Retained", "Exited"],
      "labelColumnName": "Exited",
      "predictionsColumnName": "prediction",
      "predictionScoresColumnName": "predictionScores"
    }
  ]
}

{
  "taskType": "tabular-regression",
  "model": {
    "modelType": "shell",
    "featureNames": ["sqft", "bedrooms", "bathrooms", "age_years", "lot_size"],
    "categoricalFeatureNames": []
  },
  "datasets": [
    {
      "name": "validation_set",
      "label": "validation",
      "path": "dataset.csv",
      "featureNames": ["sqft", "bedrooms", "bathrooms", "age_years", "lot_size"],
      "categoricalFeatureNames": [],
      "targetColumnName": "price",
      "predictionsColumnName": "prediction"
    }
  ]
}

​taskType

​model

​Object attributes

​modelType

​runtime

​installCommand

​batchCommand

​outputDirectory

​metadata

​datasets

​Dataset object common attributes

​name

​label

​path

​metadata

​Dataset object task-specific attributes

​inputVariableNames

​outputColumnName

​groundTruthColumnName

​categoricalFeatureNames

​classNames

​featureNames

​labelColumnName

​predictionsColumnName

​predictionScoresColumnName

​categoricalFeatureNames

​featureNames

​targetColumnName

​predictionsColumnName

​classNames

​textColumnName

​labelColumnName

​predictionsColumnName

​predictionScoresColumnName

​testsPath

​metrics

​Object attributes

​settings

​Setting object attributes

​key

​starred

​selected

​Examples

`taskType`

`model`

Object attributes

`modelType`

`runtime`

`installCommand`

`batchCommand`

`outputDirectory`

`metadata`

`datasets`

`Dataset` object common attributes

`name`

`label`

`path`

`metadata`

`Dataset` object task-specific attributes

`inputVariableNames`

`outputColumnName`

`groundTruthColumnName`

`categoricalFeatureNames`

`classNames`

`featureNames`

`labelColumnName`

`predictionsColumnName`

`predictionScoresColumnName`

`categoricalFeatureNames`

`featureNames`

`targetColumnName`

`predictionsColumnName`

`classNames`

`textColumnName`

`labelColumnName`

`predictionsColumnName`

`predictionScoresColumnName`

testsPath

metrics

Object attributes

`settings`

`Setting` object attributes

`key`

`starred`

`selected`

Examples