> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure AI Speech

> Learn how to monitor Azure AI Speech with Openlayer

<img width="700" style={{ borderRadius: "0.5rem" }} src="https://mintcdn.com/openlayer-44/4cu8TDdOg1L6h2C1/images/integrations/azure_speech_hero.png?fit=max&auto=format&n=4cu8TDdOg1L6h2C1&q=85&s=2c4a863b9c881cb0ff09d74a0380c7ef" alt="Azure AI Speech hero" data-path="images/integrations/azure_speech_hero.png" />

Openlayer integrates with [Azure AI Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/overview),
Microsoft's service for speech-to-text, speech translation, and text-to-speech. Each
recognition, translation, or synthesis call made with the Azure Speech SDK becomes a step
in your Openlayer trace, with the recognized text or synthesized-audio statistics, latency,
the result reason, and failure details. You can optionally attach the audio itself.

<Warning>
  Trace Azure AI Speech on your server only. Keep your Azure Speech key and your
  Openlayer API key in server-side configuration and out of browser or mobile
  code. For browser apps, issue short-lived Speech tokens from your backend and
  use `SpeechConfig.fromAuthorizationToken` in the browser.
</Warning>

## Monitoring Azure AI Speech

To use [monitoring mode](/monitoring/overview), instrument your code to publish the Speech
calls your AI system makes to the Openlayer platform.

### Installation

Install Openlayer and the Azure Speech SDK:

<CodeGroup>
  ```bash Python theme={null}
  pip install openlayer azure-cognitiveservices-speech
  ```

  ```bash TypeScript theme={null}
  npm install openlayer microsoft-cognitiveservices-speech-sdk
  ```
</CodeGroup>

<Note>
  Azure AI Speech tracing requires `openlayer>=0.34.0` in Python and
  `openlayer>=0.32.0` in TypeScript. In TypeScript, the Azure Speech SDK is an
  optional peer dependency, so installing `openlayer` alone does not add it.
</Note>

Set your credentials and inference pipeline in the environment:

```bash theme={null}
OPENLAYER_API_KEY=YOUR_OPENLAYER_API_KEY
OPENLAYER_INFERENCE_PIPELINE_ID=YOUR_OPENLAYER_INFERENCE_PIPELINE_ID
AZURE_SPEECH_KEY=YOUR_AZURE_SPEECH_KEY
AZURE_SPEECH_REGION=YOUR_AZURE_SPEECH_REGION
```

### Trace Speech calls

<CodeGroup>
  ```python Python theme={null}
  import os

  import azure.cognitiveservices.speech as speechsdk

  from openlayer.lib import init

  # Auto-instruments every Speech client created afterwards
  init()

  speech_config = speechsdk.SpeechConfig(
      subscription=os.environ["AZURE_SPEECH_KEY"],
      region=os.environ["AZURE_SPEECH_REGION"],
  )
  speech_config.speech_recognition_language = "en-US"
  speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

  synthesizer = speechsdk.SpeechSynthesizer(
      speech_config=speech_config,
      audio_config=speechsdk.audio.AudioOutputConfig(filename="greeting.wav"),
  )
  synthesizer.speak_text("Hello! How can I help you today?")

  recognizer = speechsdk.SpeechRecognizer(
      speech_config=speech_config,
      audio_config=speechsdk.audio.AudioConfig(filename="greeting.wav"),
  )
  result = recognizer.recognize_once()
  print(result.text)
  ```

  ```typescript TypeScript theme={null}
  import * as sdk from "microsoft-cognitiveservices-speech-sdk";
  import { traceAzureSpeech } from "openlayer/lib/integrations/azureSpeechTracer";

  const speechConfig = sdk.SpeechConfig.fromSubscription(
    process.env.AZURE_SPEECH_KEY ?? "",
    process.env.AZURE_SPEECH_REGION ?? "",
  );
  speechConfig.speechRecognitionLanguage = "en-US";
  speechConfig.speechSynthesisVoiceName = "en-US-JennyNeural";

  // Wrap each recognizer or synthesizer; the client is patched in place
  const synthesizer = traceAzureSpeech(
    new sdk.SpeechSynthesizer(
      speechConfig,
      sdk.AudioConfig.fromAudioFileOutput("greeting.wav"),
    ),
  );

  // The Speech SDK is callback-based: wrap calls in a Promise to await them
  const result = await new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
    synthesizer.speakTextAsync(
      "Hello! How can I help you today?",
      resolve,
      reject,
    ),
  );
  synthesizer.close();
  ```
</CodeGroup>

In Python, `init()` instruments every `SpeechRecognizer`, `TranslationRecognizer`, and
`SpeechSynthesizer` created afterwards. To trace a single client instead, pass it to
`trace_azure_speech(client)` from `openlayer.lib`. In TypeScript, wrap each client with
`traceAzureSpeech()`.

If your resource uses a custom endpoint (for example, an Azure AI Services
`*.cognitiveservices.azure.com` domain), create the configuration with
`SpeechConfig(subscription=..., endpoint=...)` in Python or
`SpeechConfig.fromEndpoint(new URL(endpoint), key)` in TypeScript. Tracing works the same
way.

### Trace a full voice turn

Wrap the function that handles a conversational turn so recognition and synthesis appear
as steps of the same trace:

<CodeGroup>
  ```python Python theme={null}
  from openlayer.lib import trace


  @trace()
  def voice_turn(audio_path: str) -> str:
      recognizer = speechsdk.SpeechRecognizer(
          speech_config=speech_config,
          audio_config=speechsdk.audio.AudioConfig(filename=audio_path),
      )
      heard = recognizer.recognize_once().text

      reply = f"You said: {heard}"
      speechsdk.SpeechSynthesizer(
          speech_config=speech_config, audio_config=None
      ).speak_text(reply)
      return reply


  voice_turn("caller.wav")
  ```

  ```typescript TypeScript theme={null}
  import * as fs from "fs";

  import trace from "openlayer/lib/tracing/tracer";

  function recognize(audioPath: string): Promise<sdk.SpeechRecognitionResult> {
    const recognizer = traceAzureSpeech(
      new sdk.SpeechRecognizer(
        speechConfig,
        sdk.AudioConfig.fromWavFileInput(fs.readFileSync(audioPath)),
      ),
    );
    return new Promise<sdk.SpeechRecognitionResult>((resolve, reject) =>
      recognizer.recognizeOnceAsync(resolve, reject),
    ).finally(() => recognizer.close());
  }

  function speak(text: string): Promise<sdk.SpeechSynthesisResult> {
    const replySynthesizer = traceAzureSpeech(
      new sdk.SpeechSynthesizer(speechConfig, null as any),
    );
    return new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
      replySynthesizer.speakTextAsync(text, resolve, reject),
    ).finally(() => replySynthesizer.close());
  }

  const voiceTurn = trace(async function voiceTurn(
    audioPath: string,
  ): Promise<string> {
    const heard = await recognize(audioPath);
    const reply = `You said: ${heard.text}`;
    await speak(reply);
    return reply;
  });

  await voiceTurn("caller.wav");
  ```
</CodeGroup>

<CardGroup cols={2}>
  <Card title="See full Python example" icon="python" iconType="duotone" href="https://colab.research.google.com/github/openlayer-ai/openlayer-python/blob/main/examples/tracing/azure-speech/azure_speech_tracing.ipynb" />

  <Card title="See full TypeScript example" icon="js" iconType="duotone" href="https://github.com/openlayer-ai/openlayer-ts/blob/main/examples/azure-speech-tracing.ts" />
</CardGroup>

## Attach audio to traces

Audio is **not** sent to Openlayer by default. When you enable attachment uploads,
synthesized audio is attached to each synthesis step automatically. For recognition, pass
the audio you are transcribing explicitly, because the Speech SDK's `AudioConfig` does not
expose its source:

<CodeGroup>
  ```python Python theme={null}
  from openlayer.lib import init

  init(attachment_upload_enabled=True)

  recognizer = speechsdk.SpeechRecognizer(
      speech_config=speech_config,
      audio_config=speechsdk.audio.AudioConfig(filename="caller.wav"),
  )
  # `openlayer_audio` accepts a file path, bytes, or an Attachment
  result = recognizer.recognize_once(openlayer_audio="caller.wav")
  ```

  ```typescript TypeScript theme={null}
  import * as fs from "fs";

  import { configure } from "openlayer/lib/tracing/tracer";

  configure({ attachmentUploadEnabled: true });

  const audio = fs.readFileSync("caller.wav");
  // `inputAudio` accepts bytes, a file path, or an Attachment. It is attached
  // to every recognition step of this recognizer.
  const recognizer = traceAzureSpeech(
    new sdk.SpeechRecognizer(
      speechConfig,
      sdk.AudioConfig.fromWavFileInput(audio),
    ),
    { inputAudio: audio },
  );
  ```
</CodeGroup>

The audio is uploaded to Openlayer storage when the trace completes and appears as an audio
player on the step: recognition audio under **Inputs**, synthesized audio under **Output**.
A file path is read as bytes, so your local path is never recorded.

Synthesized audio is labeled by its actual encoding. MP3, Ogg, WebM, and WAV output keep
their format. Headerless PCM, µ-law, and A-law output (the `raw-*` formats) is wrapped in a
WAV container so it can be played. Formats without a playable container, such as raw Opus
frames, are attached as files that you can download.

<Warning>
  Uploading audio sends that data to Openlayer. Enable it only when your privacy
  and security requirements allow it. See [Trace multimodal
  data](/monitoring/multimodal) for how attachments are stored and displayed.
</Warning>

## Data captured

Speech calls appear on the **Data** page of your Openlayer data source as steps named
**Azure Speech Recognition**, **Azure Speech Translation**, or **Azure Speech Synthesis**,
with the `Azure_Speech` provider.

| Field            | Recognition and translation                                           | Synthesis                                                    |
| ---------------- | --------------------------------------------------------------------- | ------------------------------------------------------------ |
| Inputs           | Recognition language, translation target languages, optional audio    | The text or SSML that was synthesized                        |
| Output           | Recognized text, and translations for translation                     | Audio duration and size, and the audio when uploads are on   |
| Model            | Custom endpoint ID, or `speech-to-text`                               | The voice name, such as `en-US-Ava:DragonHDLatestNeural`     |
| Model parameters | Non-secret settings such as region, language, and endpoint ID         | Non-secret settings such as region, voice, and output format |
| Metadata         | Result ID, reason, offset and duration, no-match or cancellation info | Result ID, reason, and cancellation info                     |

Azure reports most failures as canceled results rather than errors, so failed calls are
traced too, with the cancellation reason, error code, and error details.

Your Azure subscription key and authorization token are never recorded. Openlayer reads
only an allowlist of non-secret settings, and it redacts credentials from error details:
the Speech SDK can include your endpoint URL, with any token in its query string, in
connection errors.

Speech steps report 0 tokens, so their cost shows as \$0.

## Supported calls

| Client                                         | Python                                                             | TypeScript                         |
| ---------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------- |
| `SpeechRecognizer` and `TranslationRecognizer` | `recognize_once`, `recognize_once_async`                           | `recognizeOnceAsync`               |
| `SpeechSynthesizer`                            | `speak_text`, `speak_ssml`, `speak_text_async`, `speak_ssml_async` | `speakTextAsync`, `speakSsmlAsync` |

Continuous recognition, streaming synthesis requests, and batch transcription are not
traced.

## Troubleshooting

<AccordionGroup>
  <Accordion title="No Speech steps appear in Openlayer">
    In Python, call `init()` before creating your Speech clients, or pass each
    client to `trace_azure_speech()`. In TypeScript, wrap every recognizer and
    synthesizer with `traceAzureSpeech()`. Also check that `OPENLAYER_API_KEY`
    and `OPENLAYER_INFERENCE_PIPELINE_ID` are set.
  </Accordion>

  <Accordion title="Recognition and synthesis show up as separate traces">
    Wrap the function that makes both calls with `@trace()` in Python or
    `trace()` in TypeScript. In TypeScript, `await` each Speech call inside that
    function so its step is recorded under the right parent.
  </Accordion>

  <Accordion title="The trace has no audio player">
    Enable attachment uploads (`attachment_upload_enabled=True` in Python,
    `attachmentUploadEnabled: true` in TypeScript) before the trace is created.
    Recognition steps also need the audio passed explicitly, as
    `openlayer_audio` in Python or `inputAudio` in TypeScript.
  </Accordion>

  <Accordion title="The Azure Speech SDK cannot be resolved in TypeScript">
    Install `microsoft-cognitiveservices-speech-sdk`. It is an optional peer
    dependency, so installing `openlayer` alone does not add it to your
    application.
  </Accordion>

  <Accordion title="Canceled and NoMatch results appear in my traces">
    This is expected. Azure returns canceled and `NoMatch` outcomes as results,
    and Openlayer traces them so you can investigate the reason and latency.
  </Accordion>
</AccordionGroup>