Azure AI Speech

Openlayer integrates with Azure AI Speech, Microsoft’s service for speech-to-text, speech translation, and text-to-speech. Each recognition, translation, or synthesis call made with the Azure Speech SDK becomes a step in your Openlayer trace, with the recognized text or synthesized-audio statistics, latency, the result reason, and failure details. You can optionally attach the audio itself.

Trace Azure AI Speech on your server only. Keep your Azure Speech key and your Openlayer API key in server-side configuration and out of browser or mobile code. For browser apps, issue short-lived Speech tokens from your backend and use SpeechConfig.fromAuthorizationToken in the browser.

Monitoring Azure AI Speech

To use monitoring mode, instrument your code to publish the Speech calls your AI system makes to the Openlayer platform.

Installation

Install Openlayer and the Azure Speech SDK:

pip install openlayer azure-cognitiveservices-speech

npm install openlayer microsoft-cognitiveservices-speech-sdk

Azure AI Speech tracing requires openlayer>=0.34.0 in Python and openlayer>=0.32.0 in TypeScript. In TypeScript, the Azure Speech SDK is an optional peer dependency, so installing openlayer alone does not add it.

Set your credentials and inference pipeline in the environment:

OPENLAYER_API_KEY=YOUR_OPENLAYER_API_KEY
OPENLAYER_INFERENCE_PIPELINE_ID=YOUR_OPENLAYER_INFERENCE_PIPELINE_ID
AZURE_SPEECH_KEY=YOUR_AZURE_SPEECH_KEY
AZURE_SPEECH_REGION=YOUR_AZURE_SPEECH_REGION

Trace Speech calls

import os

import azure.cognitiveservices.speech as speechsdk

from openlayer.lib import init

# Auto-instruments every Speech client created afterwards
init()

speech_config = speechsdk.SpeechConfig(
    subscription=os.environ["AZURE_SPEECH_KEY"],
    region=os.environ["AZURE_SPEECH_REGION"],
)
speech_config.speech_recognition_language = "en-US"
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioOutputConfig(filename="greeting.wav"),
)
synthesizer.speak_text("Hello! How can I help you today?")

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioConfig(filename="greeting.wav"),
)
result = recognizer.recognize_once()
print(result.text)

import * as sdk from "microsoft-cognitiveservices-speech-sdk";
import { traceAzureSpeech } from "openlayer/lib/integrations/azureSpeechTracer";

const speechConfig = sdk.SpeechConfig.fromSubscription(
  process.env.AZURE_SPEECH_KEY ?? "",
  process.env.AZURE_SPEECH_REGION ?? "",
);
speechConfig.speechRecognitionLanguage = "en-US";
speechConfig.speechSynthesisVoiceName = "en-US-JennyNeural";

// Wrap each recognizer or synthesizer; the client is patched in place
const synthesizer = traceAzureSpeech(
  new sdk.SpeechSynthesizer(
    speechConfig,
    sdk.AudioConfig.fromAudioFileOutput("greeting.wav"),
  ),
);

// The Speech SDK is callback-based: wrap calls in a Promise to await them
const result = await new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
  synthesizer.speakTextAsync(
    "Hello! How can I help you today?",
    resolve,
    reject,
  ),
);
synthesizer.close();

In Python, init() instruments every SpeechRecognizer, TranslationRecognizer, and SpeechSynthesizer created afterwards. To trace a single client instead, pass it to trace_azure_speech(client) from openlayer.lib. In TypeScript, wrap each client with traceAzureSpeech(). If your resource uses a custom endpoint (for example, an Azure AI Services *.cognitiveservices.azure.com domain), create the configuration with SpeechConfig(subscription=..., endpoint=...) in Python or SpeechConfig.fromEndpoint(new URL(endpoint), key) in TypeScript. Tracing works the same way.

Trace a full voice turn

Wrap the function that handles a conversational turn so recognition and synthesis appear as steps of the same trace:

from openlayer.lib import trace


@trace()
def voice_turn(audio_path: str) -> str:
    recognizer = speechsdk.SpeechRecognizer(
        speech_config=speech_config,
        audio_config=speechsdk.audio.AudioConfig(filename=audio_path),
    )
    heard = recognizer.recognize_once().text

    reply = f"You said: {heard}"
    speechsdk.SpeechSynthesizer(
        speech_config=speech_config, audio_config=None
    ).speak_text(reply)
    return reply


voice_turn("caller.wav")

import * as fs from "fs";

import trace from "openlayer/lib/tracing/tracer";

function recognize(audioPath: string): Promise<sdk.SpeechRecognitionResult> {
  const recognizer = traceAzureSpeech(
    new sdk.SpeechRecognizer(
      speechConfig,
      sdk.AudioConfig.fromWavFileInput(fs.readFileSync(audioPath)),
    ),
  );
  return new Promise<sdk.SpeechRecognitionResult>((resolve, reject) =>
    recognizer.recognizeOnceAsync(resolve, reject),
  ).finally(() => recognizer.close());
}

function speak(text: string): Promise<sdk.SpeechSynthesisResult> {
  const replySynthesizer = traceAzureSpeech(
    new sdk.SpeechSynthesizer(speechConfig, null as any),
  );
  return new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
    replySynthesizer.speakTextAsync(text, resolve, reject),
  ).finally(() => replySynthesizer.close());
}

const voiceTurn = trace(async function voiceTurn(
  audioPath: string,
): Promise<string> {
  const heard = await recognize(audioPath);
  const reply = `You said: ${heard.text}`;
  await speak(reply);
  return reply;
});

await voiceTurn("caller.wav");

See full Python example

See full TypeScript example

Attach audio to traces

Audio is not sent to Openlayer by default. When you enable attachment uploads, synthesized audio is attached to each synthesis step automatically. For recognition, pass the audio you are transcribing explicitly, because the Speech SDK’s AudioConfig does not expose its source:

from openlayer.lib import init

init(attachment_upload_enabled=True)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioConfig(filename="caller.wav"),
)
# `openlayer_audio` accepts a file path, bytes, or an Attachment
result = recognizer.recognize_once(openlayer_audio="caller.wav")

import * as fs from "fs";

import { configure } from "openlayer/lib/tracing/tracer";

configure({ attachmentUploadEnabled: true });

const audio = fs.readFileSync("caller.wav");
// `inputAudio` accepts bytes, a file path, or an Attachment. It is attached
// to every recognition step of this recognizer.
const recognizer = traceAzureSpeech(
  new sdk.SpeechRecognizer(
    speechConfig,
    sdk.AudioConfig.fromWavFileInput(audio),
  ),
  { inputAudio: audio },
);

The audio is uploaded to Openlayer storage when the trace completes and appears as an audio player on the step: recognition audio under Inputs, synthesized audio under Output. A file path is read as bytes, so your local path is never recorded. Synthesized audio is labeled by its actual encoding. MP3, Ogg, WebM, and WAV output keep their format. Headerless PCM, µ-law, and A-law output (the raw-* formats) is wrapped in a WAV container so it can be played. Formats without a playable container, such as raw Opus frames, are attached as files that you can download.

Uploading audio sends that data to Openlayer. Enable it only when your privacy and security requirements allow it. See Trace multimodal data for how attachments are stored and displayed.

Data captured

Speech calls appear on the Data page of your Openlayer data source as steps named Azure Speech Recognition, Azure Speech Translation, or Azure Speech Synthesis, with the Azure_Speech provider.

Field	Recognition and translation	Synthesis
Inputs	Recognition language, translation target languages, optional audio	The text or SSML that was synthesized
Output	Recognized text, and translations for translation	Audio duration and size, and the audio when uploads are on
Model	Custom endpoint ID, or `speech-to-text`	The voice name, such as `en-US-Ava:DragonHDLatestNeural`
Model parameters	Non-secret settings such as region, language, and endpoint ID	Non-secret settings such as region, voice, and output format
Metadata	Result ID, reason, offset and duration, no-match or cancellation info	Result ID, reason, and cancellation info

Azure reports most failures as canceled results rather than errors, so failed calls are traced too, with the cancellation reason, error code, and error details. Your Azure subscription key and authorization token are never recorded. Openlayer reads only an allowlist of non-secret settings, and it redacts credentials from error details: the Speech SDK can include your endpoint URL, with any token in its query string, in connection errors. Speech steps report 0 tokens, so their cost shows as $0.

Supported calls

Client	Python	TypeScript
`SpeechRecognizer` and `TranslationRecognizer`	`recognize_once`, `recognize_once_async`	`recognizeOnceAsync`
`SpeechSynthesizer`	`speak_text`, `speak_ssml`, `speak_text_async`, `speak_ssml_async`	`speakTextAsync`, `speakSsmlAsync`

Continuous recognition, streaming synthesis requests, and batch transcription are not traced.

Troubleshooting

No Speech steps appear in Openlayer

In Python, call init() before creating your Speech clients, or pass each client to trace_azure_speech(). In TypeScript, wrap every recognizer and synthesizer with traceAzureSpeech(). Also check that OPENLAYER_API_KEY and OPENLAYER_INFERENCE_PIPELINE_ID are set.

Recognition and synthesis show up as separate traces

Wrap the function that makes both calls with @trace() in Python or trace() in TypeScript. In TypeScript, await each Speech call inside that function so its step is recorded under the right parent.

The trace has no audio player

Enable attachment uploads (attachment_upload_enabled=True in Python, attachmentUploadEnabled: true in TypeScript) before the trace is created. Recognition steps also need the audio passed explicitly, as openlayer_audio in Python or inputAudio in TypeScript.

The Azure Speech SDK cannot be resolved in TypeScript

Install microsoft-cognitiveservices-speech-sdk. It is an optional peer dependency, so installing openlayer alone does not add it to your application.

Canceled and NoMatch results appear in my traces

This is expected. Azure returns canceled and NoMatch outcomes as results, and Openlayer traces them so you can investigate the reason and latency.

Integrations

LLM Providers

Frameworks

No-Code Platforms

Observability

Data Platforms

Evaluation & Quality

Data Labeling

Collaboration

Monitoring Azure AI Speech

Installation

Trace Speech calls

Trace a full voice turn

See full Python example

See full TypeScript example

Attach audio to traces

Data captured

Supported calls

Troubleshooting

​Monitoring Azure AI Speech

​Installation

​Trace Speech calls

​Trace a full voice turn

See full Python example

See full TypeScript example

​Attach audio to traces

​Data captured

​Supported calls

​Troubleshooting

Monitoring Azure AI Speech

Installation

Trace Speech calls

Trace a full voice turn

Attach audio to traces

Data captured

Supported calls

Troubleshooting