Publishing data - Openlayer

You must set up a way to publish the requests your AI system is receiving to the Openlayer platform to use Openlayer’s monitoring mode.

This guide expands on the use of Openlayer’s SDKs to publish data to the Openlayer platform. Alternatively, you can use Openlayer’s REST API, as discussed in the Monitoring overview.

Data publishing methods

The data publishing methods are categorized as streamlined approaches and the manual approach.

The streamlined approaches exist for common AI patterns and frameworks. To use them, you need to wrap or decorate your code a certain way, and Openlayer automatically captures relevant data and metadata, such as the number of tokens, cost, latency, etc. This data is then published to the Openlayer platform.

The manual approach is system-agnostic. It is equivalent to hitting the relevant endpoint from Openlayer’s REST API but via Openlayer’s SDKs.

Streamlined approaches

There is a streamlined approach for each of the frameworks below:

To monitor chat completions and completion calls to OpenAI LLMs, you need to:

# 1. Set the environment variables
import os

os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY_HERE"
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_PROJECT_NAME"] = "YOUR_OPENLAYER_PROJECT_NAME_HERE"

# 2. Wrap the OpenAI client with Openlayer's llm_monitors.OpenAIMonitor
import openai
from openlayer import llm_monitors

openai_client = openai.OpenAI()
llm_monitors.OpenAIMonitor(client=openai_client)

# 3. From now on, every chat completion/completion call with
# the `openai_client`is monitored by Openlayer. E.g.,
completion = openai_client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "How are you doing today?"},
    ]
)

That’s it! Now, your calls are being published to Openlayer, along with metadata, such as latency, number of tokens, cost estimate, and more.

See full Python example

See full TypeScript example

To monitor chat completions and completion calls to Azure OpenAI LLMs, you need to:

Python

# 1. Set the environment variables
import os

os.environ["AZURE_OPENAI_ENDPOINT"] = "YOUR_AZURE_OPENAI_ENDPOINT_HERE"
os.environ["AZURE_OPENAI_API_KEY"] = "YOUR_AZURE_OPENAI_API_KEY_HERE"
os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"] = "YOUR_AZURE_OPENAI_DEPLOYMENT_NAME_HERE"
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_PROJECT_NAME"] = "YOUR_OPENLAYER_PROJECT_NAME_HERE"

# 2. Wrap the Azure OpenAI client with Openlayer's llm_monitors.AzureOpenAIMonitor
import openai
from openlayer import llm_monitors

azure_client = openai.AzureOpenAI(
    api_key=os.environ.get("AZURE_OPENAI_API_KEY"),
    api_version="2024-02-01",
    azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
)
llm_monitors.AzureOpenAIMonitor(client=azure_client)

# 3. From now on, every chat completion/completion call with
# the `azure_client`is monitored by Openlayer. E.g.,
completion = azure_client.chat.completions.create(
    model=os.environ.get("AZURE_OPENAI_DEPLOYMENT_NAME"),
    messages=[
        {"role": "user", "content": "How are you doing today?"},
    ]
)

That’s it! Now, your calls are being published to Openlayer, along with metadata, such as latency, number of tokens, and more.

See full Python example

To monitor chat completions models and chains built with LangChain, you need to:

# 1. Set the environment variables
import os

os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY_HERE"
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_PROJECT_NAME"] = "YOUR_PROJECT_NAME_HERE"

# 2. Instantiate the `OpenlayerHandler`
from openlayer.integrations import langchain_callback

openlayer_handler = langchain_callback.OpenlayerHandler()

# 3. Pass the handler to your LLM/chain invocations
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI

chat = ChatOpenAI(max_tokens=25, callbacks=[openlayer_handler])
chat.invoke([HumanMessage(content="What's the meaning of life?")])

That’s it! Now, your calls are being published to Openlayer, along with metadata, such as latency, number of tokens, cost estimate, and more.

See full Python example

To trace a multi-step LLM system (such as a RAG system or LLM chains), you just need to decorate all the functions you are interested in adding to a trace with Openlayer’s decorator. For example:

Python

import os
from openlayer.tracing import tracer

# Set the environment variables
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_PROJECT_NAME"] = "YOUR_OPENLAYER_PROJECT_NAME_HERE"

# Decorate all the functions you want to trace
@tracer.trace()
def main(user_query: str) -> str:
    context = retrieve_context(user_query)
    answer = generate_answer(user_query, context)
    return answer

@tracer.trace()
def retrieve_context(user_query: str) -> str:
    return "Some context"

@tracer.trace()
def generate_answer(user_query: str, context: str) -> str:
    return "Some answer"

# Every time the main function is called, the data is automatically
# streamed to your Openlayer project. E.g.:
main("What is the meaning of life?")

You can use the decorator together with the other streamlined methods. For example, if your generate_answer function uses a wrapped version of the OpenAI client, the chat completion calls will get added to the trace under the generate_answer function step.

See full Python example

To monitor runs from OpenAI Assistants, you need to:

Setup Openlayer’s OpenAIMonitor:

import openai
import os
from openlayer import llm_monitors

# Set the environment variables
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY_HERE"
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_PROJECT_NAME"] = "YOUR_OPENLAYER_PROJECT_NAME_HERE"

openai_client = openai.OpenAI()
monitor = llm_monitors.OpenAIMonitor(client=openai_client)

Create OpenAI assistant and thread:

# Create the assistant
assistant = openai_client.beta.assistants.create(
    name="Data visualizer",
    description="You are great at creating and explaining beautiful data visualizations.",
    model="gpt-4",
    tools=[{"type": "code_interpreter"}],
)

# Create a thread
thread = openai_client.beta.threads.create(
    messages=[
        {
        "role": "user",
        "content": "Create a data visualization of the american GDP.",
        }
    ]
)

Create and monitor a run:

import time

# Run assistant on thread
run = openai_client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id
)

# Keep polling the run results
while run.status != "completed":
    run = openai_client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)

    # Monitor the run with the Openlayer `monitor`. If complete, the thread is sent to Openlayer
    monitor.monitor_thread_run(run)

    time.sleep(5)

That’s it! Now, your calls are being published to Openlayer, along with metadata, such as latency, number of tokens, cost estimate, and more.

See full Python example

See full TypeScript example

Manual approach

To manually stream data to Openlayer, you must:

Retrieve an existing inference pipeline:

import openlayer
from openlayer.tasks import TaskType

client = openlayer.OpenlayerClient("YOUR_API_KEY_HERE")

project = client.create_project(
    name="YOUR_PROJECT_NAME_HERE",
    task_type=TaskType.LLM,
)

inference_pipeline = project.create_inference_pipeline(
    name="YOUR_INFERENCE_PIPELINE_NAME_HERE",
)

Stream data:

# If you have a data stream, where `data` is a single row as a dictionary
inference_pipeline.stream_data(
    stream_data=data,
    stream_config=config,
)

See full Python example

See full TypeScript example

You can also send additional information, such as the latency and the number of tokens used in the inference. Refer to the Data config guide for your task type of interest for more information about the config in the code snippet above.

Documentation

​Data publishing methods

​Streamlined approaches

See full Python example

See full TypeScript example

See full Python example

See full Python example

See full Python example

See full Python example

See full TypeScript example

​Manual approach

See full Python example

See full TypeScript example

Data publishing methods

Streamlined approaches

Manual approach