Groq

If you are building an AI system with Groq LLMs and want to evaluate it, you can use the SDKs to make Openlayer part of your workflow.

This integration guide shows how you can do it.

​

Evaluating Groq LLMs

You can set up Openlayer tests to evaluate your Groq LLMs in development and monitoring.

​

Development

In development mode, Openlayer becomes a step in your CI/CD pipeline, and your tests get automatically evaluated after being triggered by some events.

Openlayer tests often rely on your AI system’s outputs on a validation dataset. As discussed in the Configuring output generation guide, you have two options:

1. either provide a way for Openlayer to run your AI system on your datasets, or
2. before pushing, generate the model outputs yourself and push them alongside your artifacts.

For AI systems built with Groq LLMs, if you are not computing your system’s outputs yourself, you must provide your API credentials.

To do so, navigate to “Settings” > “Workspace secrets,” and click on “Add secret” to add your GROQ_API_KEY.

If you don’t add the required Groq API key, you’ll encounter a “Missing API key” error when Openlayer tries to run your AI system to get its outputs:

​

Monitoring

To use the monitoring mode, you must set up a way to publish the requests your AI system receives to the Openlayer platform. This process is streamlined for Groq LLMs.

To set it up, you must follow the steps in the code snippet below:

# 1. Set the environment variables
import os

os.environ["GROQ_API_KEY"] = "YOUR_GROQ_API_KEY_HERE"
os.environ["OPENLAYER_API_KEY"] = "YOUR_OPENLAYER_API_KEY_HERE"
os.environ["OPENLAYER_INFERENCE_PIPELINE_ID"] = "YOUR_OPENLAYER_INFERENCE_PIPELINE_ID_HERE"

# 2. Import the `trace_groq` function and wrap the Groq client
import groq
from openlayer.lib import trace_groq

groq_client = trace_groq(groq.Groq())

# 3. From now on, every chat completion call with
# the `groq_client` is traced by Openlayer. E.g.,
completion = groq_client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Explain the importance of fast language models",
        }
    ],
    model="llama3-8b-8192",
)

Once the code is set up, all your Groq AI LLM calls are automatically published to Openlayer, along with metadata, such as latency, number of tokens, cost estimate, and more.

If you navigate to the “Requests” page of your Openlayer inference pipeline, you can see the traces for each request.

After your AI system requests are continuously published and logged by Openlayer, you can create tests that run at a regular cadence on top of them.

Refer to the Monitoring overview, for details on Openlayer’s monitoring mode, to the Publishing data guide, for more information on setting it up, or to the Tracing guide, to understand how to trace more complex systems.