Prepare input data

MediaPipe Tasks automatically handles model-specific input preprocessing—including image resizing, rotation, and value normalization for vision models, and audio resampling, channel mixing, buffering, and spectrogram framing for audio models. Before running inference, wrap your platform's image, video, camera, or microphone buffers in the input container expected by your target SDK.

Vision Tasks

Prepare vision input data

Wrap still images, decoded video frames, and live camera streams in MPImage (Android and iOS), mp.Image (Python), or standard DOM media elements (Web), and configure frame rotation or regions of interest.

Audio Tasks

Prepare audio input data

Convert prerecorded audio files and live microphone PCM buffers into AudioData ring buffers or normalized Float32 sample arrays across Web, Android, iOS, and Python.

Prepare vision input data

Vision tasks accept still images, decoded video frames, and live camera feeds (on Android and iOS, Universal Embedder also accepts MPImage for image embeddings, while Web and Python pass encoded image bytes). Each task automatically resizes, normalizes, and converts color spaces to match its underlying model bundle.

Platform	Input type	Supported media sources & orientation handling
Web	`HTMLImageElement`, `HTMLVideoElement`, `HTMLCanvasElement`, `ImageBitmap`, `OffscreenCanvas`, `ImageData`	Pass browser media elements directly to `detect()` or `detectForVideo()` without an intermediate wrapper.
Android	`MPImage`	Build from `ARGB_8888` `Bitmap` (`BitmapImageBuilder`, including CameraX `ImageProxy.toBitmap()`), `RGBA_8888` `android.media.Image` (`MediaImageBuilder`), or `ByteBuffer` (`ByteBufferImageBuilder`). Apply clockwise sensor rotation in 90° multiples with `ImageProcessingOptions`.
iOS	`MPImage`	Initialize from `UIImage`, `CVPixelBuffer`, or camera `CMSampleBuffer` with a non-mirrored `UIImage.Orientation` (`.up`, `.down`, `.left`, or `.right`).
Python	`mp.Image`	Load files with `mp.Image.create_from_file()` or wrap RGB/RGBA NumPy arrays with `mp.Image(image_format=mp.ImageFormat.SRGB, data=...)`. Convert OpenCV `BGR` frames to `RGB` first.

Vision input code examples

Select your platform and running mode to prepare still images, video frames with monotonic millisecond timestamps, or live camera streams (shown here with HandLandmarker; all Vision tasks accept the same media inputs):

Web

npmnpm install @mediapipe/tasks-visionSetup guide·API reference

Image

import type { HandLandmarker, HandLandmarkerResult } from "@mediapipe/tasks-vision";

export async function detectFromImageElement(
  handLandmarker: HandLandmarker,
  imgElement: HTMLImageElement,
): Promise<HandLandmarkerResult> {
  await imgElement.decode();
  return handLandmarker.detect(imgElement);
}

export async function detectFromBlob(
  handLandmarker: HandLandmarker,
  fileOrBlob: Blob,
): Promise<HandLandmarkerResult> {
  const bitmap = await createImageBitmap(fileOrBlob);
  try {
    return handLandmarker.detect(bitmap);
  } finally {
    bitmap.close();
  }
}

Video

import type { HandLandmarker, HandLandmarkerResult } from "@mediapipe/tasks-vision";

export async function analyzeVideoClip(
  handLandmarker: HandLandmarker,
  video: HTMLVideoElement,
  onResult: (result: HandLandmarkerResult, timestampMs: number) => void,
): Promise<void> {
  let lastTimestampMs = -1;

  const processNextFrame = () => {
    if (video.paused || video.ended) return;
    const timestampMs = Math.round(video.currentTime * 1000);
    if (timestampMs > lastTimestampMs) {
      lastTimestampMs = timestampMs;
      onResult(handLandmarker.detectForVideo(video, timestampMs), timestampMs);
    }
    video.requestVideoFrameCallback(processNextFrame);
  };

  await video.play();
  video.requestVideoFrameCallback(processNextFrame);
}

Livestream

import type { HandLandmarker, HandLandmarkerResult } from "@mediapipe/tasks-vision";

export async function startWebcamStream(
  handLandmarker: HandLandmarker,
  webcam: HTMLVideoElement,
  onResult: (result: HandLandmarkerResult) => void,
): Promise<void> {
  webcam.srcObject = await navigator.mediaDevices.getUserMedia({
    video: { facingMode: "user", width: { ideal: 1280 }, height: { ideal: 720 } },
  });
  await webcam.play();

  let lastVideoTime = -1;
  const renderLoop = () => {
    if (webcam.readyState >= 2 && webcam.currentTime !== lastVideoTime) {
      lastVideoTime = webcam.currentTime;
      onResult(handLandmarker.detectForVideo(webcam, performance.now()));
    }
    requestAnimationFrame(renderLoop);
  };
  renderLoop();
}

Android (Kotlin)

Gradleimplementation("com.google.mediapipe:tasks-vision:latest.release")Setup guide·MPImage API

Image

import android.graphics.Bitmap
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult

fun detectBitmap(handLandmarker: HandLandmarker, bitmap: Bitmap): HandLandmarkerResult {
    // Hardware-backed bitmaps must be copied to ARGB_8888 before wrapping
    val argbBitmap = if (bitmap.config == Bitmap.Config.ARGB_8888) {
        bitmap
    } else {
        bitmap.copy(Bitmap.Config.ARGB_8888, /* isMutable = */ false)
    }
    val mpImage = BitmapImageBuilder(argbBitmap).build()
    return handLandmarker.detect(mpImage)
}

Video

import android.content.Context
import android.graphics.Bitmap
import android.media.MediaMetadataRetriever
import android.net.Uri
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult

fun detectVideoFrame(
    context: Context,
    videoUri: Uri,
    timestampUs: Long,
    handLandmarker: HandLandmarker
): HandLandmarkerResult? {
    val retriever = MediaMetadataRetriever()
    try {
        retriever.setDataSource(context, videoUri)
        val frame = retriever.getFrameAtTime(timestampUs, MediaMetadataRetriever.OPTION_CLOSEST) ?: return null

        val argbFrame = if (frame.config == Bitmap.Config.ARGB_8888) {
            frame
        } else {
            frame.copy(Bitmap.Config.ARGB_8888, false)
        }
        val mpImage = BitmapImageBuilder(argbFrame).build()
        return handLandmarker.detectForVideo(mpImage, timestampUs / 1000L)
    } finally {
        retriever.release()
    }
}

Livestream

import android.os.SystemClock
import androidx.camera.core.ImageProxy
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.tasks.vision.core.ImageProcessingOptions
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker

fun analyzeCameraFrame(imageProxy: ImageProxy, handLandmarker: HandLandmarker) {
    imageProxy.use { proxy ->
        val mpImage = BitmapImageBuilder(proxy.toBitmap()).build()
        val imageProcessingOptions = ImageProcessingOptions.builder()
            .setRotationDegrees(proxy.imageInfo.rotationDegrees)
            .build()

        handLandmarker.detectAsync(mpImage, imageProcessingOptions, SystemClock.uptimeMillis())
    }
}

Android (Java)

Gradleimplementation 'com.google.mediapipe:tasks-vision:latest.release'Setup guide·MPImage API

Image

import android.graphics.Bitmap;
import com.google.mediapipe.framework.image.BitmapImageBuilder;
import com.google.mediapipe.framework.image.MPImage;
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker;
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult;

HandLandmarkerResult detectBitmap(HandLandmarker handLandmarker, Bitmap bitmap) {
  Bitmap argbBitmap = (bitmap.getConfig() == Bitmap.Config.ARGB_8888)
      ? bitmap
      : bitmap.copy(Bitmap.Config.ARGB_8888, /* isMutable= */ false);
  MPImage mpImage = new BitmapImageBuilder(argbBitmap).build();
  return handLandmarker.detect(mpImage);
}

Video

import android.content.Context;
import android.graphics.Bitmap;
import android.media.MediaMetadataRetriever;
import android.net.Uri;
import com.google.mediapipe.framework.image.BitmapImageBuilder;
import com.google.mediapipe.framework.image.MPImage;
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker;
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult;
import java.io.IOException;

HandLandmarkerResult detectVideoFrame(
    Context context,
    Uri videoUri,
    long timestampUs,
    HandLandmarker handLandmarker) throws IOException {
  MediaMetadataRetriever retriever = new MediaMetadataRetriever();
  try {
    retriever.setDataSource(context, videoUri);
    Bitmap frame = retriever.getFrameAtTime(timestampUs, MediaMetadataRetriever.OPTION_CLOSEST);
    if (frame == null) {
      return null;
    }

    Bitmap argbFrame = (frame.getConfig() == Bitmap.Config.ARGB_8888)
        ? frame
        : frame.copy(Bitmap.Config.ARGB_8888, false);
    MPImage mpImage = new BitmapImageBuilder(argbFrame).build();
    return handLandmarker.detectForVideo(mpImage, timestampUs / 1000L);
  } finally {
    retriever.release();
  }
}

Livestream

import android.graphics.Bitmap;
import android.os.SystemClock;
import androidx.camera.core.ImageProxy;
import com.google.mediapipe.framework.image.BitmapImageBuilder;
import com.google.mediapipe.framework.image.MPImage;
import com.google.mediapipe.tasks.vision.core.ImageProcessingOptions;
import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker;

void analyzeCameraFrame(ImageProxy imageProxy, HandLandmarker handLandmarker) {
  try {
    Bitmap bitmap = imageProxy.toBitmap();
    MPImage mpImage = new BitmapImageBuilder(bitmap).build();
    ImageProcessingOptions imageProcessingOptions = ImageProcessingOptions.builder()
        .setRotationDegrees(imageProxy.getImageInfo().getRotationDegrees())
        .build();

    handLandmarker.detectAsync(mpImage, imageProcessingOptions, SystemClock.uptimeMillis());
  } finally {
    imageProxy.close();
  }
}

iOS

SwiftPMgoogle-ai-edge/mediapipe (MediaPipeTasksVision)Setup guide·MPImage API

Image

import MediaPipeTasksVision
import UIKit

func detectImage(_ uiImage: UIImage, handLandmarker: HandLandmarker) throws -> HandLandmarkerResult {
    // Uses uiImage.imageOrientation by default (or pass orientation: .up/.down/.left/.right)
    let mpImage = try MPImage(uiImage: uiImage)
    return try handLandmarker.detect(image: mpImage)
}

Video

import AVFoundation
import MediaPipeTasksVision

func analyzeVideo(
    asset: AVAsset,
    videoTrack: AVAssetTrack,
    handLandmarker: HandLandmarker,
    onResult: (HandLandmarkerResult, Int) -> Void
) throws {
    let reader = try AVAssetReader(asset: asset)
    let trackOutput = AVAssetReaderTrackOutput(
        track: videoTrack,
        outputSettings: [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
    )
    reader.add(trackOutput)
    reader.startReading()

    while let sampleBuffer = trackOutput.copyNextSampleBuffer() {
        let mpImage = try MPImage(sampleBuffer: sampleBuffer, orientation: .up)
        let pts = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        let timestampMs = Int(CMTimeGetSeconds(pts) * 1000)
        let result = try handLandmarker.detect(videoFrame: mpImage, timestampInMilliseconds: timestampMs)
        onResult(result, timestampMs)
    }
}

Livestream

import AVFoundation
import MediaPipeTasksVision
import UIKit

final class CameraFeedService: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let handLandmarker: HandLandmarker

    init(handLandmarker: HandLandmarker) {
        self.handLandmarker = handLandmarker
    }

    // Configure AVCaptureVideoDataOutput with kCVPixelFormatType_32BGRA
    func captureOutput(
        _ output: AVCaptureOutput,
        didOutput sampleBuffer: CMSampleBuffer,
        from connection: AVCaptureConnection
    ) {
        guard let mpImage = try? MPImage(sampleBuffer: sampleBuffer, orientation: .up) else { return }

        let pts = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        let timestampMs = Int(CMTimeGetSeconds(pts) * 1000)
        try? handLandmarker.detectAsync(image: mpImage, timestampInMilliseconds: timestampMs)
    }
}

Python

pippip install mediapipe opencv-pythonSetup guide·mp.Image API

Image

import cv2
import mediapipe as mp

def detect_from_file(hand_landmarker: mp.tasks.vision.HandLandmarker, path: str):
    mp_image = mp.Image.create_from_file(path)
    return hand_landmarker.detect(mp_image)

def detect_from_opencv_frame(hand_landmarker: mp.tasks.vision.HandLandmarker, path: str):
    bgr_frame = cv2.imread(path)
    if bgr_frame is None:
        raise ValueError(f"Could not decode image: {path}")
    rgb_frame = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2RGB)
    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)
    return hand_landmarker.detect(mp_image)

Video

import cv2
import mediapipe as mp

def analyze_video_file(hand_landmarker: mp.tasks.vision.HandLandmarker, video_path: str):
    cap = cv2.VideoCapture(video_path)
    try:
        fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
        frame_index = 0
        while cap.isOpened():
            ok, bgr_frame = cap.read()
            if not ok:
                break
            rgb_frame = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2RGB)
            mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)
            timestamp_ms = round((frame_index * 1000.0) / fps)
            frame_index += 1
            yield hand_landmarker.detect_for_video(mp_image, timestamp_ms)
    finally:
        cap.release()

Livestream

import time
import cv2
import mediapipe as mp

def stream_webcam(hand_landmarker: mp.tasks.vision.HandLandmarker, camera_index: int = 0):
    cap = cv2.VideoCapture(camera_index)
    try:
        while cap.isOpened():
            ok, bgr_frame = cap.read()
            if not ok:
                break
            rgb_frame = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2RGB)
            mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)
            timestamp_ms = time.monotonic_ns() // 1_000_000
            hand_landmarker.detect_async(mp_image, timestamp_ms)
    finally:
        cap.release()

Rotation and region of interest: On Web, Android, and Python, Vision tasks accept an optional ImageProcessingOptions argument per inference call to rotate the frame clockwise in 90° increments (0, 90, 180, or 270) and—for ImageClassifier and ImageEmbedder only—crop a normalized regionOfInterest ([0.0, 1.0]; ObjectDetector does not support a region of interest). On iOS, pass a non-mirrored UIImage.Orientation (.up, .down, .left, or .right) when constructing MPImage; mirrored orientations (.upMirrored, .downMirrored, .leftMirrored, and .rightMirrored) are not supported.

Prepare audio input data

Audio tasks automatically resample, buffer, and frame PCM audio to match the model's sample rate and window size. Input container types depend on the task and platform:

Audio Classifier uses AudioData (a typed PCM float ring buffer paired with AudioDataFormat) on Android, iOS, and Python, or a Float32Array + optional sample rate on Web. (Universal Embedder also accepts AudioData on Android and iOS, Float32Array on Web, and list[float] in Python.)
Audio Transcriber accepts mono Float32 PCM sample arrays normalized to [-1.0, 1.0] directly (Float32Array on Web, float[] on Android, UnsafePointer<Float> on iOS, and np.ndarray in Python), resampling automatically when sampleRateHz / sample_rate_hz is set in the task options. (Text-to-Speech is a text-in, audio-out task that accepts UTF-8 text strings and returns normalized 24 kHz mono Float32 PCM samples.)

Platform	Input type	Supported media sources & format notes
Web	`Float32Array` (+ sample rate for `AudioClassifier` / `UniversalEmbedder`, or mono `Float32Array` for `AudioTranscriber`)	Decode audio files with `AudioContext.decodeAudioData()` (`audioBuffer.getChannelData(0)`) or stream live microphone chunks with `AudioWorkletNode`.
Android	`AudioData` (or `float[]` for `AudioTranscriber`)	Create with `AudioData.create(format, sampleCounts)` and populate from `float[]`, 16-bit `short[]`, or live `android.media.AudioRecord` buffers.
iOS	`AudioData` & `MPPFloatBuffer` (or `UnsafePointer<Float>` for `AudioTranscriber`)	Read files with `AVAudioFile` (`.pcmFormatFloat32`) or tap the microphone with `AudioRecord` / `AVAudioEngine`.
Python	`AudioData` (or `np.ndarray` for `AudioTranscriber`)	Normalize PCM WAV or microphone NumPy arrays to `[-1.0, 1.0]` and call `AudioData.create_from_array(samples, sample_rate)`.

Audio input code examples

Select your platform and audio mode to prepare complete prerecorded clips or contiguous live microphone streams (shown here with AudioClassifier):

Web

npmnpm install @mediapipe/tasks-audioSetup guide·API reference

Audio clips

import type { AudioClassifier, AudioClassifierResult } from "@mediapipe/tasks-audio";

export async function classifyAudioUrl(
  audioClassifier: AudioClassifier,
  audioUrl: string,
): Promise<AudioClassifierResult[]> {
  const audioCtx = new AudioContext({ sampleRate: 16000 });
  try {
    const response = await fetch(audioUrl);
    const audioBuffer = await audioCtx.decodeAudioData(await response.arrayBuffer());
    const pcmSamples: Float32Array = audioBuffer.getChannelData(0);
    return audioClassifier.classify(pcmSamples, audioBuffer.sampleRate);
  } finally {
    await audioCtx.close();
  }
}

Audio stream

Define an AudioWorkletProcessor in its own module that collects fixed-size mono PCM windows (15600 samples = 0.975 s at 16 kHz, the window YAMNet expects) and posts each one to the main thread:

class PcmChunkProcessor extends AudioWorkletProcessor {
  constructor({ processorOptions }) {
    super();
    this.buffer = new Float32Array(processorOptions?.windowSize ?? 15600);
    this.offset = 0;
  }
  process(inputs) {
    const input = inputs[0]?.[0];
    if (!input) return true;
    for (const sample of input) {
      this.buffer[this.offset++] = sample;
      if (this.offset === this.buffer.length) {
        this.port.postMessage(this.buffer.slice());
        this.offset = 0;
      }
    }
    return true;
  }
}
registerProcessor("pcm-chunk-processor", PcmChunkProcessor);

Register that module with audioWorklet.addModule(), route the microphone through it, and classify every window it posts:

import type { AudioClassifier, AudioClassifierResult } from "@mediapipe/tasks-audio";

export async function startMicrophoneClassification(
  audioClassifier: AudioClassifier,
  onResult: (results: AudioClassifierResult[]) => void,
): Promise<void> {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  const audioCtx = new AudioContext({ sampleRate: 16000 });
  await audioCtx.audioWorklet.addModule("pcm-chunk-processor.js");

  const source = audioCtx.createMediaStreamSource(stream);
  const worklet = new AudioWorkletNode(audioCtx, "pcm-chunk-processor");
  worklet.port.onmessage = ({ data: pcmWindow }: MessageEvent<Float32Array>) => {
    onResult(audioClassifier.classify(pcmWindow, audioCtx.sampleRate));
  };
  source.connect(worklet);
}

Android (Kotlin)

Gradleimplementation("com.google.mediapipe:tasks-audio:latest.release")Setup guide·AudioData API

Audio clips

import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifierResult
import com.google.mediapipe.tasks.components.containers.AudioData
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat

fun classifyPcmClip(
    audioClassifier: AudioClassifier,
    pcmFloatSamples: FloatArray,
    sampleRateHz: Float = 16000f
): AudioClassifierResult {
    val format = AudioDataFormat.builder()
        .setNumOfChannels(1)
        .setSampleRate(sampleRateHz)
        .build()

    val audioData = AudioData.create(format, pcmFloatSamples.size)
    audioData.load(pcmFloatSamples)
    return audioClassifier.classify(audioData)
}

Audio stream

import android.media.AudioRecord
import android.os.SystemClock
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier
import com.google.mediapipe.tasks.components.containers.AudioData
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat

fun startAudioRecord(
    audioClassifier: AudioClassifier,
    sampleRate: Int = 16000
): Pair<AudioRecord, AudioData> {
    val audioRecord = audioClassifier.createAudioRecord(
        /* numChannels = */ 1,
        /* sampleRate = */ sampleRate,
        /* requiredInputBufferSize = */ sampleRate
    )
    val audioData = AudioData.create(AudioDataFormat.create(audioRecord.format), sampleRate)
    audioRecord.startRecording()
    return audioRecord to audioData
}

// Invoke periodically on a background ScheduledExecutorService
fun pollMicrophoneFrame(
    audioRecord: AudioRecord,
    audioData: AudioData,
    audioClassifier: AudioClassifier
) {
    if (audioData.load(audioRecord) > 0) {
        audioClassifier.classifyAsync(audioData, SystemClock.uptimeMillis())
    }
}

Android (Java)

Gradleimplementation 'com.google.mediapipe:tasks-audio:latest.release'Setup guide·AudioData API

Audio clips

import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifierResult;
import com.google.mediapipe.tasks.components.containers.AudioData;
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat;

AudioClassifierResult classifyPcmClip(
    AudioClassifier audioClassifier, float[] pcmFloatSamples, float sampleRateHz) {
  AudioDataFormat format = AudioDataFormat.builder()
      .setNumOfChannels(1)
      .setSampleRate(sampleRateHz)
      .build();

  AudioData audioData = AudioData.create(format, pcmFloatSamples.length);
  audioData.load(pcmFloatSamples);
  return audioClassifier.classify(audioData);
}

Audio stream

import android.media.AudioRecord;
import android.os.SystemClock;
import com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier;
import com.google.mediapipe.tasks.components.containers.AudioData;
import com.google.mediapipe.tasks.components.containers.AudioData.AudioDataFormat;

void streamMicrophoneChunk(AudioClassifier audioClassifier) {
  int sampleRate = 16000;
  AudioRecord audioRecord = audioClassifier.createAudioRecord(
      /* numChannels= */ 1,
      /* sampleRate= */ sampleRate,
      /* requiredInputBufferSize= */ sampleRate
  );
  AudioData audioData = AudioData.create(AudioDataFormat.create(audioRecord.getFormat()), sampleRate);
  audioRecord.startRecording();

  // Poll on a background executor and forward non-empty reads
  if (audioData.load(audioRecord) > 0) {
    audioClassifier.classifyAsync(audioData, SystemClock.uptimeMillis());
  }
}

iOS

SwiftPMgoogle-ai-edge/mediapipe (MediaPipeTasksAudio)Sample code·Setup guide

Audio clips

import AVFoundation
import MediaPipeTasksAudio

func classifyAudioFile(
    at audioURL: URL,
    audioClassifier: AudioClassifier
) throws -> AudioClassifierResult? {
    let audioFile = try AVAudioFile(forReading: audioURL, commonFormat: .pcmFormatFloat32, interleaved: false)
    let frameCount = AVAudioFrameCount(audioFile.length)
    guard let pcmBuffer = AVAudioPCMBuffer(
        pcmFormat: audioFile.processingFormat,
        frameCapacity: frameCount
    ) else { return nil }
    try audioFile.read(into: pcmBuffer)

    let sampleCount = UInt(pcmBuffer.frameLength)
    let format = AudioDataFormat(channelCount: 1, sampleRate: audioFile.processingFormat.sampleRate)
    let audioData = AudioData(format: format, sampleCount: sampleCount)
    let floatBuffer = MPPFloatBuffer(data: pcmBuffer.floatChannelData?[0], length: sampleCount)
    try audioData.load(buffer: floatBuffer, offset: 0, length: sampleCount)
    return try audioClassifier.classify(audioClip: audioData)
}

Audio stream

import AVFoundation
import Foundation
import MediaPipeTasksAudio

// Acquire AVAudioSession record permission before calling startRecording()
func startMicrophoneStream(audioClassifier: AudioClassifier) throws -> (AudioRecord, AudioData) {
    let sampleRate = 16000.0
    let bufferLength: UInt = 16000
    let format = AudioDataFormat(channelCount: 1, sampleRate: sampleRate)
    let audioRecord = try AudioRecord(audioDataFormat: format, bufferLength: bufferLength)
    let audioData = AudioData(format: format, sampleCount: bufferLength)
    try audioRecord.startRecording()
    return (audioRecord, audioData)
}

// Invoke periodically on a background dispatch queue
func pollMicrophoneBuffer(
    audioRecord: AudioRecord,
    audioData: AudioData,
    audioClassifier: AudioClassifier
) throws {
    try audioData.load(audioRecord: audioRecord)
    let timestampMs = Int(ProcessInfo.processInfo.systemUptime * 1000)
    try audioClassifier.classifyAsync(audioBlock: audioData, timestampInMilliseconds: timestampMs)
}

Python

pippip install mediapipe numpy scipySetup guide·AudioData API

Audio clips

import mediapipe as mp
import numpy as np
from scipy.io import wavfile

AudioData = mp.tasks.components.containers.AudioData

def normalize_wav_to_mono_float32(raw_pcm: np.ndarray) -> np.ndarray:
    if np.issubdtype(raw_pcm.dtype, np.floating):
        samples = raw_pcm.astype(np.float32)
    elif raw_pcm.dtype == np.uint8:
        samples = (raw_pcm.astype(np.float32) - 128.0) / 128.0
    else:
        scale = float(1 << (8 * raw_pcm.dtype.itemsize - 1))
        samples = raw_pcm.astype(np.float32) / scale
    if samples.ndim > 1:
        samples = samples.mean(axis=1)
    return np.clip(samples, -1.0, 1.0)

def classify_wav_file(audio_classifier: mp.tasks.audio.AudioClassifier, wav_path: str):
    sample_rate, raw_pcm = wavfile.read(wav_path)
    samples = normalize_wav_to_mono_float32(raw_pcm)
    audio_data = AudioData.create_from_array(samples, sample_rate)
    return audio_classifier.classify(audio_data)

Audio stream

import time
import mediapipe as mp
import numpy as np

AudioData = mp.tasks.components.containers.AudioData

def on_microphone_int16_chunk(
    audio_classifier: mp.tasks.audio.AudioClassifier,
    int16_buffer: np.ndarray,
    sample_rate: int = 16000,
) -> None:
    float_chunk = np.clip(int16_buffer.astype(np.float32) / 32768.0, -1.0, 1.0)
    audio_data = AudioData.create_from_array(float_chunk, sample_rate)
    timestamp_ms = time.monotonic_ns() // 1_000_000
    audio_classifier.classify_async(audio_data, timestamp_ms)

Audio permissions and threading: Always request microphone permission (android.permission.RECORD_AUDIO on Android, NSMicrophoneUsageDescription + AVAudioSession.requestRecordPermission on iOS, or a user gesture before getUserMedia on Web) before starting live audio capture. Keep audio capture callbacks non-blocking by forwarding PCM buffers onto a dedicated worker queue for inference.

Prepare input data Stay organized with collections Save and categorize content based on your preferences.

Prepare vision input data

Vision input code examples

Web

Image

Video

Livestream

Android (Kotlin)

Image

Video

Livestream

Android (Java)

Image

Video

Livestream

iOS

Image

Video

Livestream

Python

Image

Video

Livestream

Prepare audio input data

Audio input code examples

Web

Audio clips

Audio stream

Android (Kotlin)

Audio clips

Audio stream

Android (Java)

Audio clips

Audio stream

iOS

Audio clips

Audio stream

Python

Audio clips

Audio stream

Prepare input data