Learn how to write the openlayer.json
config for your project
openlayer.json
contains the information Openlayer needs to
validate your artifacts, run your AI system on your datasets, and evaluate your tests.
This guide shows how you can write the openlayer.json
for your project.
openlayer.json
.openlayer.json
file has five parts:
taskType
string
The taskType
must be one of llm-base
,
tabular-classification
, tabular-regression
, and text-classification
. It corresponds
to your Openlayer project’s task type.
Example:
model
and datasets
sections.
model
object
The model
part of the openlayer.json
specifies the commands Openlayer will use to generate
predictions with your AI system, and metadata about it.
modelType
string
, required
The type of model. Must be one of shell
or full
.
You must specify it
as full
if you are providing a script in batchCommand
to run your model and get its predictions.
You must specify it as shell
if you
already computed the model predictions and are uploading model metadata
only.
runtime
string
The environment runtime to execute the commands specified in installCommand
and batchCommand
.
This is only required if you want Openlayer to run your model to get its outputs. Refer
to the Configuring output generation page for more information.
Currently, the supported runtimes are:
Runtime | Available options |
---|---|
Python | python_3_10 , python_3_8 |
NodeJS | node_20 |
installCommand
string
The command that gets executed before the run script. Serves the
purpose of installing the dependencies needed by your batchCommand
script.
For more information about the installCommand
, refer to the Configuring output generation guide.
Examples:
batchCommand
string
The command that executes your script to get your model predictions.
In general, if you are using one of Openlayer’s SDKs to write your script,
it is followed by the
placeholder arguments --dataset-path {{ path }} --dataset-name {{ name }}
.
For more information about the batchCommand
and the placeholder arguments, refer to the Configuring output generation guide.
Examples:
outputDirectory
string
, default output
Directory where the file with model outputs will be saved.
metadata
object
Object with model metadata.
datasets
array
of Dataset objects
The datasets
part of the openlayer.json
has an array of Dataset
objects. Openlayer
will iterate over this array to get your model’s outputs for each dataset.
The Dataset
object has a set of common attributes and a set of attributes that
depend on the taskType
.
Dataset
object common attributestaskType
.
name
string
, required
Dataset name.
label
string
, required
Dataset label. Must be one of validation
or training
.
path
string
, required
Path to the dataset file.
groundTruthColumnName
string | null
Name of the dataset column with the ground truths.
metadata
object
Object with dataset metadata.
Dataset
object task-specific attributestaskType
of your
Openlayer project.
inputVariableNames
array[string]
, requiredArray of input variable names. Each input variable should be in a dataset column.string
Path to a JSON file with test configurations. This field is not needed if you are
only creating tests via the UI.
Read more about test configurations on the
tests.json guide.
Example:
object
The metrics
part of the openlayer.json
allows you to control the metric settings for your
project. You can control which metrics are “starred” and which are “selected” for your project,
which defines the metrics that appear on the top panel of the project and metrics that should be computed, respectively.
settings
array
of Setting
objects
Setting
object attributeskey
string
Metric name. For example, "conciseness"
or "accuracy"
.
starred
bool
Bool indicating if the metric is “starred.” Starred metrics are the ones shown on the top
panel of your project.
selected
bool
Bool indicating if the metric is “selected.” Selected metrics are computed, which allow you
to create tests based on them. Unselected metrics are skipped.
Example:
openlayer.json
. For additional examples, check out our
Template gallery.
View example openlayer.json