Trace multimodal data

Not every AI system runs on text alone. A claims agent reads a photo of a receipt, a support bot listens to a voice note, a document pipeline parses a PDF. Attachments let your traces carry that unstructured data. The media itself is uploaded to your workspace storage, and the trace keeps a reference to it — so the platform can show you the actual image, play the actual audio, and page through the actual document next to the rest of the trace.

A monitoring record whose trace shows an audio recording and a document alongside the generated text

Attachments work the same way in the Python and TypeScript SDKs, and both write the same attachment format.

Enable attachment uploads

Attachment uploads are disabled by default. Until you turn them on, attachments are recorded in the trace but never uploaded — and anything that was not uploaded cannot be displayed.

from openlayer.lib import init

init(attachment_upload_enabled=True)

import { configure } from "openlayer/lib/tracing/tracer";

configure({ attachmentUploadEnabled: true });

With uploads enabled, Openlayer uploads each attachment when the trace completes and stores the resulting reference in the trace data. If you attach media that already lives at an external URL, also enable URL uploads so Openlayer fetches it and keeps its own copy:

init(attachment_upload_enabled=True, url_upload_enabled=True)

configure({ attachmentUploadEnabled: true, urlUploadEnabled: true });

Without it, an external URL is recorded as-is. That keeps your trace pointing at a resource Openlayer cannot read — if the URL later expires or sits behind authentication, the media is gone.

Attachments require openlayer>=0.17.0 in Python (url_upload_enabled requires openlayer>=0.17.9) and openlayer>=0.32.0 in TypeScript. Uploading media sends that data to Openlayer, so enable uploads only when your privacy requirements allow it.

Attach media to a step

Call log_attachment() in Python or logAttachment() in TypeScript inside any traced function to attach media to the step currently being recorded:

from openlayer.lib import init, trace
from openlayer.lib.tracing import log_attachment

init(attachment_upload_enabled=True)


@trace()
def triage_expense_claim(claim_id: str) -> str:
    # Attach the artifacts the claim arrived with
    log_attachment("receipt.png", metadata={"source": "mobile upload"})
    log_attachment("voice_note.wav", metadata={"channel": "voicemail"})
    log_attachment("policy.pdf")

    return review(claim_id)

import trace, { configure, logAttachment } from "openlayer/lib/tracing/tracer";

configure({ attachmentUploadEnabled: true });

const triageExpenseClaim = trace(async function triageExpenseClaim(
  claimId: string,
): Promise<string> {
  // Attach the artifacts the claim arrived with
  logAttachment("receipt.png", { metadata: { source: "mobile upload" } });
  logAttachment("voice_note.wav", { metadata: { channel: "voicemail" } });
  logAttachment("policy.pdf");

  return review(claimId);
});

The Python helpers live in openlayer.lib.tracing, not in openlayer.lib. In TypeScript, logAttachment is exported from openlayer/lib/tracing/tracer.

The helper accepts a file path, raw bytes, or an Attachment you built yourself. Python also accepts a file-like object, and TypeScript accepts a Buffer, Uint8Array, or ArrayBuffer:

# A path — the media type is detected from the extension
log_attachment("invoices/june.pdf")

# Raw bytes — pass a name and media type so the media renders correctly
log_attachment(chart_png, name="chart.png", media_type="image/png")

// A path — the media type is detected from the extension
logAttachment("invoices/june.pdf");

// Raw bytes — pass a name and media type so the media renders correctly
logAttachment(chartPng, { name: "chart.png", mediaType: "image/png" });

Every attachment can carry metadata. Use it for whatever you need to filter or debug on later: the upload channel, a page count, an audio duration, a document revision.

Build attachments explicitly

For more control, construct an Attachment and pass it to the helper:

from openlayer.lib.tracing import Attachment, log_attachment

log_attachment(Attachment.from_file("receipt.png", name="Receipt"))
log_attachment(Attachment.from_url("https://example.com/receipt.png"))
log_attachment(
    Attachment.from_bytes(png, name="chart.png", media_type="image/png")
)
log_attachment(
    Attachment.from_base64(b64, name="clip.wav", media_type="audio/wav")
)

import { Attachment } from "openlayer/lib/tracing/attachments";
import { logAttachment } from "openlayer/lib/tracing/tracer";

logAttachment(Attachment.fromFile("receipt.png", { name: "Receipt" }));
logAttachment(Attachment.fromUrl("https://example.com/receipt.png"));
logAttachment(
  Attachment.fromBytes(png, { name: "chart.png", mediaType: "image/png" }),
);
logAttachment(
  Attachment.fromBase64(b64, { name: "clip.wav", mediaType: "audio/wav" }),
);

Python	TypeScript	Use it for	Notes
`from_file()`	`fromFile()`	Media on local disk	Media type is guessed from the extension; size and checksum are computed; the path is recorded
`from_url()`	`fromUrl()`	Media hosted somewhere else	Only fetched into Openlayer when URL uploads are enabled
`from_bytes()`	`fromBytes()`	Media generated in memory	Name and media type are required; in TypeScript the bytes are copied, so you can reuse a buffer
`from_base64()`	`fromBase64()`	Media already base64-encoded	Useful for provider payloads that are encoded in transit

from_file() / fromFile() records the file’s absolute local path in the trace (filePath). If you don’t want local paths in your traces, read the file and use from_bytes() / fromBytes() instead.

Media in inputs and outputs

Attachments don’t have to hang off a step. You can pass one to a traced function, or return one, and it is uploaded and displayed like any other attachment:

from openlayer.lib import trace
from openlayer.lib.tracing import Attachment


@trace()
def transcribe(recording: Attachment) -> str:
    return speech_to_text(recording.get_bytes())


transcribe(
    Attachment.from_bytes(audio, name="caller.wav", media_type="audio/wav")
)

import { Attachment } from "openlayer/lib/tracing/attachments";
import trace from "openlayer/lib/tracing/tracer";

const transcribe = trace(async function transcribe(
  recording: Attachment,
): Promise<string> {
  return speechToText(recording.getBytes());
});

await transcribe(
  Attachment.fromBytes(audio, { name: "caller.wav", mediaType: "audio/wav" }),
);

The platform renders an input or output as media when the value is an attachment, when it is an object whose values are attachments (for example { "audio": attachment }), or when it is an array of content items. A content item nested inside another object is not rendered as media — put the bare attachment there instead.

Multimodal messages

To model a message that is itself part text and part media — the shape a vision or audio model actually receives — use content items:

from openlayer.lib import init, trace
from openlayer.lib.tracing import Attachment, create_step
from openlayer.lib.tracing.content import (
    AudioContent,
    FileContent,
    ImageContent,
    TextContent,
)
from openlayer.lib.tracing.enums import StepType

init(attachment_upload_enabled=True)


@trace()
def answer_claim_question(question: str) -> str:
    receipt = Attachment.from_file("receipt.png")
    voice_note = Attachment.from_file("voice_note.wav")
    policy = Attachment.from_file("policy.pdf")

    with create_step(
        name="Claim assistant", step_type=StepType.CHAT_COMPLETION
    ) as step:
        answer = call_your_model(question, receipt, voice_note, policy)
        step.log(
            inputs={
                "prompt": [
                    {
                        "role": "user",
                        "content": [
                            TextContent(text=question),
                            ImageContent(attachment=receipt),
                            AudioContent(attachment=voice_note),
                            FileContent(attachment=policy),
                        ],
                    }
                ]
            },
            output=answer,
        )

    return answer

import { Attachment } from "openlayer/lib/tracing/attachments";
import {
  AudioContent,
  FileContent,
  ImageContent,
  TextContent,
  type ContentItem,
} from "openlayer/lib/tracing/content";
import trace, { configure } from "openlayer/lib/tracing/tracer";

configure({ attachmentUploadEnabled: true });

const answerClaimQuestion = trace(async function answerClaimQuestion(
  message: ContentItem[],
): Promise<string> {
  return callYourModel(message);
});

await answerClaimQuestion([
  new TextContent("Is this claim within policy?"),
  new ImageContent(Attachment.fromFile("receipt.png")),
  new AudioContent(Attachment.fromFile("voice_note.wav")),
  new FileContent(Attachment.fromFile("policy.pdf")),
]);

There are four content items — TextContent, ImageContent, AudioContent, and FileContent — and a single message can mix as many as you need. Openlayer renders the message in order, so the text and the media it refers to stay together.

Automatic capture

Some integrations attach media for you once uploads are enabled.

OpenAI (Python)

If you send multimodal messages through a traced OpenAI client in Python, Openlayer reads the content array and converts it to attachments, on both the Chat Completions and Responses APIs:

Python

import openai
from openlayer.lib import init

init(attachment_upload_enabled=True)

client = openai.OpenAI()  # auto-traced

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is the total?"},
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        }
    ],
)

Images (image_url, input_image), audio (input_audio), and files (file) are all recognized, whether they arrive as a URL, a base64 data URL, or an uploaded file ID. Generated images in Responses API output are captured the same way.

Azure AI Speech (Python and TypeScript)

The Azure AI Speech integration attaches synthesized audio to each synthesis step, and the audio you pass for recognition to each recognition step, in both SDKs.

How attachments appear in Openlayer

Uploaded attachments are rendered wherever the trace is shown — in the row, in the row detail view, and on the individual step:

Media	What you get
Images	Rendered inline, click to open full size
Audio	A player you can scrub; only one clip plays at a time
PDFs	A page-by-page viewer, click to open the full document
Everything else	A card with the file name and size, and a download button

Every attachment can be downloaded, whatever its type. Downloads keep the attachment’s name and extension as part of the filename.

How uploads work

When: attachments are uploaded when the trace completes, before the trace is published, so the published trace already carries each attachment’s storageUri.
Deduplication: attachments are deduplicated by MD5 checksum, so attaching the same image to three steps costs one upload.
Failures: a failed upload is logged and the trace is still published — you get the trace without that media rather than an exception.
Where: everything found on a step’s attachments and in its inputs and outputs is uploaded, including nested steps.

The attachment format

Attachments are plain JSON inside your trace, so any client that can publish a row can publish an attachment. This is what both SDKs write:

{
  "id": "1f1d3e0c-6a19-4d1e-9f3c-2b7c0d84a501",
  "name": "receipt.png",
  "mediaType": "image/png",
  "storageUri": "s3://openlayer-assets/.../attachments/ebb63407.png",
  "sizeBytes": 35016,
  "checksumMd5": "ebb634079cb8555ceef04266b5a976ee",
  "metadata": { "source": "mobile upload" }
}

storageUri is what makes an attachment displayable — it is the reference to the copy in your workspace storage. To obtain one for media you upload yourself, request a presigned URL, upload the bytes to it, and keep the storageUri that comes back:

curl -X POST \
  "https://api.openlayer.com/v1/storage/presigned-url?objectName=receipt.png" \
  -H "Authorization: Bearer $OPENLAYER_API_KEY"

The response contains the url to upload to and the storageUri to record. How you upload depends on your deployment’s storage:

Openlayer Cloud (S3): the response also contains form fields. Send a multipart POST to url with every field first and the file last, in a part named file.
Self-hosted on GCS, Azure Blob Storage, or Oracle Object Storage: there are no fields. Send a PUT of the raw bytes to url, with a Content-Type header. Azure also requires x-ms-blob-type: BlockBlob.
Self-hosted with local storage: send a multipart POST to url with the file in a part named file.

You can then put the attachment in a column when you stream the row:

{
  "config": {
    "inputVariableNames": ["question", "receipt"],
    "outputColumnName": "output",
    "inferenceIdColumnName": "inferenceId"
  },
  "rows": [
    {
      "inferenceId": "claim-4417",
      "question": "Is this meal claim within policy?",
      "receipt": {
        "storageUri": "...",
        "mediaType": "image/png",
        "name": "receipt.png"
      },
      "output": "The claim is within policy."
    }
  ]
}

The column holding the attachment must be listed in inputVariableNames. A row containing an attachment in an undeclared column is still accepted — the request returns success — but the platform has no column to attach it to, so it is never displayed.

A column can also hold a full multimodal message, mixing media with text:

[
  { "type": "text", "text": "Is this meal claim within policy?" },
  {
    "type": "image",
    "attachment": {
      "storageUri": "...",
      "mediaType": "image/png",
      "name": "receipt.png"
    }
  }
]

Complete example

An expense claim that arrives as a photo, a voice note, and a policy document — attached to the trace, then answered by a model:

import base64

import openai
from openlayer.lib import init, trace
from openlayer.lib.tracing import log_attachment

# Attachment uploads are off by default
init(attachment_upload_enabled=True)

client = openai.OpenAI()  # auto-traced


def as_data_url(path: str, media_type: str) -> str:
    with open(path, "rb") as file:
        encoded = base64.b64encode(file.read()).decode("utf-8")
    return f"data:{media_type};base64,{encoded}"


@trace()
def read_receipt(receipt_path: str) -> str:
    """The image sent to OpenAI becomes an attachment automatically."""
    image_url = as_data_url(receipt_path, "image/png")
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Read this receipt."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            }
        ],
    )
    return response.choices[0].message.content


@trace()
def triage_expense_claim(claim_id: str) -> str:
    # Attach the artifacts the claim arrived with
    log_attachment("receipt.png", metadata={"source": "mobile upload"})
    log_attachment("voice_note.wav", metadata={"channel": "voicemail"})
    log_attachment("policy.pdf", metadata={"revision": 4})

    extracted = read_receipt("receipt.png")
    return f"Claim {claim_id} triaged. {extracted}"


if __name__ == "__main__":
    print(triage_expense_claim("CLM-4417"))

import * as fs from "fs";

import { Attachment } from "openlayer/lib/tracing/attachments";
import { AudioContent, TextContent } from "openlayer/lib/tracing/content";
import trace, { configure, logAttachment } from "openlayer/lib/tracing/tracer";

// Attachment uploads are off by default
configure({ attachmentUploadEnabled: true });

const triageExpenseClaim = trace(async function triageExpenseClaim(
  claimId: string,
  receipt: Attachment,
): Promise<Array<TextContent | AudioContent>> {
  // Attach the artifacts the claim arrived with
  logAttachment("voice_note.wav", { metadata: { channel: "voicemail" } });
  logAttachment("policy.pdf", { metadata: { revision: 4 } });

  const summary = await summarizeClaim(claimId, receipt);
  const spokenSummary = await textToSpeech(summary); // audio bytes

  // Return a multimodal message: the text plus its spoken version
  return [
    new TextContent(summary),
    new AudioContent(
      Attachment.fromBytes(spokenSummary, {
        name: "summary.mp3",
        mediaType: "audio/mpeg",
      }),
    ),
  ];
});

await triageExpenseClaim(
  "CLM-4417",
  Attachment.fromBytes(fs.readFileSync("receipt.png"), {
    name: "receipt.png",
    mediaType: "image/png",
  }),
);

Run it with your credentials set, and the trace arrives with the attachments you logged, the media passed in as inputs, and the media in the output. In Python, the receipt image OpenAI received is attached too.

Troubleshooting

My attachment does not appear in the platform

Check that uploads are enabled first (attachment_upload_enabled=True in Python, attachmentUploadEnabled: true in TypeScript) — they are off by default, and without them nothing is uploaded. If the attachment came from from_url() / fromUrl(), you also need URL uploads enabled, otherwise Openlayer never fetches a copy it can display.

The attachment is in the trace but not rendered as media

Check where it sits. An input or output renders as media when it is an attachment, an object whose values are attachments, or an array of content items. A single content item nested inside another object is not rendered — use the bare attachment there instead.

An attachment silently disappeared from the trace

Attachments with no data and no reference are dropped rather than published — most often because a file path does not exist. The SDK logs a warning when this happens. In Python, enable debug logging while you debug:

import logging

logging.getLogger("openlayer").setLevel(logging.DEBUG)

Will attachment uploads slow down my application?

Uploads happen when the trace completes, after your traced function returns, so your code isn’t held up while the media uploads. A failed upload is logged and the trace is still published — you get the trace without the media rather than an exception.

Looking to add non-media context to your traces instead? See Add metadata to traces.

Get started

Workspace setup

Governance

Observability

Offline testing

Tests

Gateway

Data quality monitoring

Administration

Notifications

Other resources

Trace multimodal data

Enable attachment uploads

Attach media to a step

Build attachments explicitly

Media in inputs and outputs

Multimodal messages

Automatic capture

OpenAI (Python)

Azure AI Speech (Python and TypeScript)

How attachments appear in Openlayer

How uploads work

The attachment format

Complete example

Troubleshooting

​Enable attachment uploads

​Attach media to a step

​Build attachments explicitly

​Media in inputs and outputs

​Multimodal messages

​Automatic capture

​OpenAI (Python)

​Azure AI Speech (Python and TypeScript)

​How attachments appear in Openlayer

​How uploads work

​The attachment format

​Complete example

​Troubleshooting

Enable attachment uploads

Attach media to a step

Build attachments explicitly

Media in inputs and outputs

Multimodal messages

Automatic capture

OpenAI (Python)

Azure AI Speech (Python and TypeScript)

How attachments appear in Openlayer

How uploads work

The attachment format

Complete example

Troubleshooting