Publish inference

from openlayer import Openlayer

# Let's say we want to stream the following row, which represents a model prediction:
data = {
  "user_query": "what's the meaning of life?",
  "output": "42",
  "tokens": 7,
  "cost": 0.02,
  "timestamp": 1620000000
}

# Prepare the config for the data, which depends on your project's task type. In this
# case, we have an LLM project:
from openlayer.types.inference_pipelines import data_stream_params

config = data_stream_params.ConfigLlmData(
    input_variable_names=["user_query"],
    output_column_name="output",
    num_of_token_column_name="tokens",
    cost_column_name="cost",
    timestamp_column_name="timestamp",
    prompt=[{"role": "user", "content": "{{ user_query }}"}],
)

client = Openlayer()
data_stream_response = client.inference_pipelines.data.stream(
    inference_pipeline_id="182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
    rows=[data],
    config=config,
)

{
  "success": true
}

Use this endpoint to stream individual inference data points to Openlayer. If you want to upload many inferences in one go, please use the batch upload method instead.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your workspace API key. See Find your API key for more information.

Path Parameters

inferencePipelineId

string

required

The inference pipeline id (a UUID).

Body

application/json

rows

object[]

required

A list of inference data points with inputs and outputs

Example:

[
  {
    "user_query": "what is the meaning of life?",
    "output": "42",
    "tokens": 7,
    "cost": 0.02,
    "timestamp": 1620000000
  }
]

config

object

required

Configuration for the data stream. Depends on your Openlayer project task type.

config.outputColumnName

string

required

Name of the column with the model outputs.

Example:

"output"

config.numOfTokenColumnName

string | null

Name of the column with the total number of tokens.

Example:

"num_tokens"

config.contextColumnName

string

Name of the column with the context retrieved. Applies to RAG use cases. Providing the context enables RAG-specific metrics.

Example:

"context"

config.costColumnName

string

Name of the column with the cost associated with each row.

Example:

"cost"

config.groundTruthColumnName

string

Name of the column with the ground truths.

Example:

"ground_truth"

config.inferenceIdColumnName

string

Name of the column with the inference ids. This is useful if you want to update rows at a later point in time. If not provided, a unique id is generated by Openlayer.

Example:

"id"

config.inputVariableNames

string[]

Array of input variable names. Each input variable should be a dataset column.

Example:

["user_query"]

config.latencyColumnName

string

Name of the column with the latencies.

Example:

"latency"

config.metadata

object

Object with metadata.

config.prompt

object[]

Prompt for the LLM.

Example:

[
  {
    "role": "user",
    "content": "{{ user_query }}"
  }
]

config.questionColumnName

string

Name of the column with the questions. Applies to RAG use cases. Providing the question enables RAG-specific metrics.

Example:

"question"

config.timestampColumnName

string

Name of the column with the timestamps. Timestamps must be in UNIX sec format. If not provided, the upload timestamp is used.

Example:

"timestamp"

Example:

{
  "prompt": [
    {
      "role": "user",
      "content": "{{ user_query }}"
    }
  ],
  "inputVariableNames": ["user_query"],
  "outputColumnName": "output",
  "timestampColumnName": "timestamp",
  "costColumnName": "cost",
  "numOfTokenColumnName": "tokens"
}

Response

200

application/json

Status OK.

success

boolean

required

Was this page helpful?

Delete inference pipeline Update inference

from openlayer import Openlayer

# Let's say we want to stream the following row, which represents a model prediction:
data = {
  "user_query": "what's the meaning of life?",
  "output": "42",
  "tokens": 7,
  "cost": 0.02,
  "timestamp": 1620000000
}

# Prepare the config for the data, which depends on your project's task type. In this
# case, we have an LLM project:
from openlayer.types.inference_pipelines import data_stream_params

config = data_stream_params.ConfigLlmData(
    input_variable_names=["user_query"],
    output_column_name="output",
    num_of_token_column_name="tokens",
    cost_column_name="cost",
    timestamp_column_name="timestamp",
    prompt=[{"role": "user", "content": "{{ user_query }}"}],
)

client = Openlayer()
data_stream_response = client.inference_pipelines.data.stream(
    inference_pipeline_id="182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
    rows=[data],
    config=config,
)

{
  "success": true
}

Reference

SDKs

CLI

REST API

Authorizations

Path Parameters

Body

Response