Log context for RAG systems

Retrieval-Augmented Generation (RAG) systems depend on a retriever that fetches context documents before passing them to an LLM. To best evaluate RAG quality, Openlayer needs to know what context was retrieved for each request. Once context is logged, you can use tests that leverage metrics such as context recall (did the retriever surface the right information?), context relevancy (how relevant was the retrieved content to the query?), and more.

How to log context

There are two main ways to provide context to Openlayer:

The context must be passed as a list of strings (List[str]). Each string should represent a retrieved chunk/document.

1. Use `@trace` with `context_kwarg`

If your function receives the retrieved context as a keyword argument, you can tell Openlayer which argument contains it.

from openlayer.lib import trace

@trace(context_kwarg="context")
def generate_answer(query: str, context: list[str]) -> str:
    return llm_call(query, context)

2. Call `log_context` directly

If you do not want to rely on a kwarg, or if you get the context at different points in the pipeline, you can log it explicitly:

from openlayer.lib import trace
from openlayer.lib.tracing import log_context

@trace()
def generate_answer(query: str) -> str:
    context = retrieve_context(query)
    # Log the retrieved context manually
    log_context(context)

    return llm_call(query, context)

Track users and sessions

Log question for RAG systems

⌘I

​How to log context

​1. Use @trace with context_kwarg

​2. Call log_context directly

How to log context

1. Use `@trace` with `context_kwarg`

2. Call `log_context` directly