Text-to-speech guide for iOS

The MediaPipe Text-to-Speech task lets you synthesize natural-sounding speech audio from text on iOS (15.0+) using neural TTS models such as Kokoro-82M. These instructions show you how to use the Text-to-Speech in Swift applications.

For more information about the capabilities, .litertlm model bundles, and supported voices of this task, see the Overview.

Code example

The MediaPipe Tasks example code is a basic implementation of a Text-to-Speech app for iOS. The example synthesizes the text that you enter with a voice that you choose, either as a complete utterance or as a stream of audio chunks, and plays the audio as it arrives.

You can use the app as a starting point for your own iOS app, or refer to it when modifying an existing app. You can refer to the Text-to-Speech example code on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

1. Clone the git repository using the following command:

   git clone https://github.com/google-ai-edge/mediapipe-samples
   

2. Optionally, configure your git instance to use sparse checkout, so you have only the files for the Text-to-Speech example app:

   cd mediapipe-samples
   git sparse-checkout init --cone
   git sparse-checkout set examples/text_to_speech/ios
   

After creating a local version of the example code, add the model to the app as described in the example's README. Then open the project in Xcode and run the app. For instructions, see the Setup guide for iOS.

Key components

The following files contain the main code for the Text-to-Speech example app:

• TTSViewModel.swift: Creates the Text-to-Speech task, lists the available voices, and runs unary and streaming synthesis.
• TTSAudioPlayer.swift: Plays the synthesized audio with AVAudioEngine and can export it as a WAV file.
• ContentView.swift: Implements the SwiftUI user interface.

Setup

This section describes key steps for setting up your Xcode or Swift Package Manager (SPM) project to use Text-to-Speech. For general information on setting up your development environment for MediaPipe tasks, see the Setup guide for iOS.

Dependencies

Text-to-Speech uses the MediaPipeTasksAudio library, which you can install using Swift Package Manager (SPM).

To add the MediaPipeTasksAudio library to your Xcode project:

1. With your project open in Xcode, select File > Add Package Dependencies....
2. In the search field in the upper right, enter the repository URL: https://github.com/google-ai-edge/mediapipe
3. Select the mediapipe package and choose your dependency rule.
4. Click Add Package, then select MediaPipeTasksAudio and add it to your application target.

Then import the library in your Swift source files:

import MediaPipeTasksAudio

Model

The MediaPipe Text-to-Speech task requires a .litertlm model bundle. This guide uses kokoro_int8_gpu_fp16.litertlm. See the task overview Models section for the other supported models.

Bundle kokoro_int8_gpu_fp16.litertlm in your app resources (or download it into your app's Application Support / Documents container) and pass its path to TTSBaseOptions(modelAssetPath:). Keep the original filename, which the task uses to detect the model type:

guard let modelPath = Bundle.main.path(forResource: "kokoro_int8_gpu_fp16", ofType: "litertlm") else {
    fatalError("kokoro_int8_gpu_fp16.litertlm not found in app bundle.")
}

Create the task

Use createTextToSpeech(options:) (or TextToSpeech(options:)) with TextToSpeechOptions, which takes a TTSBaseOptions with the model path and delegate:

import Foundation
import MediaPipeTasksAudio

let cacheDir = FileManager.default.temporaryDirectory
    .appendingPathComponent("mediapipe_tts_cache").path

let options = TextToSpeechOptions(
    baseOptions: TTSBaseOptions(
        modelAssetPath: modelPath,
        delegate: .gpu
    ),
    defaultVoiceId: "af_heart",
    speakingRate: 1.0,
    targetBucket: 64,
    cacheDirectory: cacheDir
)

let tts = try createTextToSpeech(options: options)

To use PocketTTS or Phonon, pass that model's bundle path and set defaultVoiceId to one of its voices (or leave it unset). Both models run on CPU, so they ignore the delegate setting.

Configuration options

This task has the following configuration options for iOS apps:

Task options (TextToSpeechOptions)

Per-utterance options (SynthesisOptions)

Inspect available voices

Call listVoices() to query all voices packaged inside the loaded .litertlm bundle:

let voices = try tts.listVoices()
for voice in voices {
    print("\(voice.voiceId) (\(voice.displayName)) - locale=\(voice.locale), \(voice.sampleRateHz) Hz")
}

Run the task

You can call the non-blocking synthesizeAsync(_:options:) method from Swift structured concurrency, or call synthesize(_:options:) synchronously on a background dispatch queue.

Async and synchronous synthesis

// Non-blocking async call (recommended):
let audioResult = try await tts.synthesizeAsync(
    "Hello from MediaPipe Text to Speech on Apple platforms!"
)

// Synchronous blocking call (run off the main thread):
let syncResult = try tts.synthesize("Synthesized synchronously on a worker thread.")

Fast bilingual voice switching

Pass SynthesisOptions(voiceId:) to switch between bundled Kokoro voices across languages in < 1 ms. Leave locale as nil so the engine automatically resolves the target language from the voice prefix:

// Synthesize English ("af_heart") followed by Spanish ("ef_dora"):
let enResult = try await tts.synthesizeAsync(
    "Hello! Let's switch to Spanish.",
    options: SynthesisOptions(voiceId: "af_heart")
)

let esResult = try await tts.synthesizeAsync(
    "Hola desde MediaPipe Text to Speech en iOS.",
    options: SynthesisOptions(voiceId: "ef_dora")
)

Streaming speech synthesis

For lower Time-To-First-Audio (TTFA) on multi-sentence text, iterate over tts.synthesizeStream(_:options:), which returns an AsyncThrowingStream<AudioChunk, Error>. Each AudioChunk (samples, sampleRateHz, chunkIndex, isFinal) is yielded as soon as its clause finishes vocoding, so you can schedule it onto AVAudioPlayerNode immediately:

for try await chunk in tts.synthesizeStream(
    "Streaming synthesis plays the first clause immediately. Subsequent clauses synthesize in the background.",
    options: SynthesisOptions(voiceId: "af_heart")
) {
    try playAudioResult(
        AudioResult(samples: chunk.samples, sampleRateHz: chunk.sampleRateHz),
        playerNode: playerNode,
        engine: audioEngine
    )
}

When streaming tokens incrementally from an on-device LLM, open a TTSStream with tts.openStream(options:), push text fragments with pushText(_:), mark end-of-input with finishInput(), and pull synthesized AudioChunks with readNext():

let stream = try tts.openStream(options: SynthesisOptions(voiceId: "af_heart"))
defer { stream.close() }

try stream.pushText("Hello from an ")
try stream.pushText("on-device LLM! ")
try stream.pushText("Each completed clause streams out.")
try stream.finishInput()

while let chunk = try stream.readNext() {
    try playAudioResult(
        AudioResult(samples: chunk.samples, sampleRateHz: chunk.sampleRateHz),
        playerNode: playerNode,
        engine: audioEngine
    )
}

Barge-in cancellation

If the user interrupts speech playback ("barge-in"), you can abort an in-flight call immediately:

• Breaking out of for try await chunk in tts.synthesizeStream(...) or canceling the enclosing Swift Task automatically cancels and closes the underlying TTSStream using onTermination.
• synthesizeAsync(_:options:) uses withTaskCancellationHandler internally: canceling the Swift Task automatically calls tts.cancel() and throws CancellationError.
• You can also call tts.cancel() (or stream.cancel()) directly from any thread to abort an in-flight synthesis pass immediately.

let synthesisTask = Task {
    for try await chunk in tts.synthesizeStream("Long utterance that may be interrupted by the user...") {
        try playAudioResult(
            AudioResult(samples: chunk.samples, sampleRateHz: chunk.sampleRateHz),
            playerNode: playerNode,
            engine: audioEngine
        )
    }
}

// On user barge-in:
synthesisTask.cancel() // Immediately cancels the TTSStream

Handle and play results

synthesize and synthesizeAsync return an AudioResult struct containing:

• samples: [Float] array of mono PCM samples in [-1.0, 1.0]
• sampleRateHz: Int (24000 Hz for all supported models)

You can schedule the PCM samples onto an AVAudioEngine / AVAudioPlayerNode for playback:

import AVFoundation

func playAudioResult(_ result: AudioResult, playerNode: AVAudioPlayerNode, engine: AVAudioEngine) throws {
    guard let format = AVAudioFormat(
        standardFormatWithSampleRate: Double(result.sampleRateHz),
        channels: 1
    ),
    let buffer = AVAudioPCMBuffer(
        pcmFormat: format,
        frameCapacity: AVAudioFrameCount(result.samples.count)
    ) else { return }

    buffer.frameLength = buffer.frameCapacity
    if let channelData = buffer.floatChannelData?[0] {
        result.samples.withUnsafeBufferPointer { src in
            channelData.initialize(from: src.baseAddress!, count: src.count)
        }
    }

    if !engine.isRunning {
        try engine.start()
    }
    playerNode.scheduleBuffer(buffer, completionHandler: nil)
    playerNode.play()
}