LiteRT-LM is the production-ready orchestration layer to run LLMs with LiteRT, engineered for high-performance, cross-platform execution.
- Cross-Platform Support: Run on Android, iOS, Web, Desktop, and IoT (e.g. Raspberry Pi).
- Hardware Acceleration: Get peak performance and system stability by leveraging GPU and NPU accelerators across diverse hardware.
- Multi-Modality: Build with LLMs and embedding models that have vision and audio support.
- Multimodal Embeddings (New): Generate on-device vector representations across text, images, video, and audio for semantic search and RAG. See Embedding Models.
- Tool Use: Function calling support for agentic workflows with constrained decoding for improved accuracy.
- Broad Model Support: Run Gemma, EmbeddingGemma 2, Llama, Phi-4, Qwen and more.
What's New
For the latest release details and updates, refer to the GitHub Release Notes.
On-Device GenAI Showcase
The Google AI Edge Gallery is an experimental app designed to showcase on-device Generative AI capabilities running entirely offline using LiteRT-LM.
- Google Play: Use LLMs locally on supported Android devices.
- App Store: Experience on-device AI on your iOS device.
- GitHub Source: View the source code for the gallery app to learn how to integrate LiteRT-LM inside your own projects.
Featured Model: EmbeddingGemma 2
- Model Size: 485 MB (740M Omnimodal), 388 MB (440M Text-Vision), 165 MB (270M Text)
- Try it in your browser with the Multimodal Search Web Demo.
Additional technical details and integration examples are in the Embedding Models Guide and the HuggingFace model card.
Platform (Device) Backend Text Latency (ms) Text + Vision Latency (ms) Text + Audio Latency (ms) Text + Vision + Audio Latency (ms) Android (Pixel 11 Pro) TPU 8.3 49 193.5 149 Android (S26 Ultra) CPU 27.1 175 305 322 GPU 25.9 119 350 324 iOS (iPhone 18 Pro) CPU 41.8 191 445 400 GPU 11.6 69.8 165 150 Linux (Arm 2.3 & 2.8 GHz, NVIDIA GeForce RTX 4090) CPU 105 825 947 1251 GPU 7.6 23.9 169 112 macOS (MacBook Pro M5) CPU 31.3 151 314 432 GPU 9.5 37.3 195 131 Windows (Intel Core Ultra Series 3) CPU 71.7 338 813 708 GPU 19.2 62.5 284 205 NPU 13.3 49.8 124.3 108.6 Web (MacBook Pro M5) WebGPU 21.8 107 228 226 IoT (Raspberry Pi 5 16GB) CPU 161 1761 946 2304
Featured Model: Gemma-4-E2B
- Model Size: 2.58 GB
Additional technical details are in the HuggingFace model card
Platform (Device) Backend Prefill (tk/s) Decode (tk/s) Time to First Token (seconds) Peak CPU Memory (MB) Android (S26 Ultra) CPU 557 47 1.8 1733 GPU 3808 52 0.3 676 iOS (iPhone 17 Pro) CPU 532 25 1.9 607 GPU 2878 56 0.3 1450 Linux (Arm 2.3 & 2.8 GHz, NVIDIA GeForce RTX 4090) CPU 260 35 4 1628 GPU 11234 143 0.1 913 macOS (MacBook Pro M4 Max) CPU 901 42 1.1 736 GPU 7835 160 0.1 1623 Windows (Intel LunarLake) CPU 435 30 2.4 3505 GPU 3751 48 0.3 3540 IoT (Raspberry Pi 5 16GB) CPU 133 8 7.8 1546
Start Building
LiteRT-LM provides APIs for several programming languages and platforms to help you build on-device AI applications quickly. Select a guide below to get started:
| Language | Status | Best For... | Documentation |
|---|---|---|---|
| CLI | ✅ Stable |
Getting started with LiteRT-LM in less than 1 min. | CLI Guide |
| Python | ✅ Stable |
Rapid prototyping, development, on desktop, Android & Raspberry Pi. | Python Guide |
| Kotlin | ✅ Stable |
Native Android apps and JVM-based desktop tools. Optimized for Coroutines. | Kotlin Guide |
| Swift | 🚀 Early Preview |
Native iOS and macOS integration with specialized Metal support. | Swift Guide |
| JavaScript (web) | 🚀 Early Preview |
Deploy models directly in web browsers with high performance. | JavaScript Guide |
| Flutter | 🚀 Community |
Cross-platform
Flutter apps
using community
flutter_gemma. |
Flutter Guide |
| C++ | ✅ Stable |
High-performance, cross-platform core logic and embedded systems. | C++ Guide |
Build from Source
If you want to customize LiteRT-LM or build it for a specific hardware configuration, you can compile it directly from the source code. For step-by-step instructions on how to set up your environment and build the framework, refer to the LiteRT-LM Build and Run Guide on GitHub.
Supported Backends & Platforms
| Acceleration | Android | iOS | macOS | Windows | Linux | IoT |
|---|---|---|---|---|---|---|
| CPU | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| GPU | ✅ | ✅ | ✅ | ✅ | ✅ | - |
| NPU | ✅ | - | - | 🚀 | - | - |
Supported Models
The following table lists a subset of models supported by LiteRT-LM. For the latest models and performance data, visit the LiteRT Community on Hugging Face.
| Model | Type | Size (MB) | Details | Device | CPU Prefill (tk/s) | CPU Decode (tk/s) | GPU Prefill (tk/s) | GPU Decode (tk/s) |
|---|---|---|---|---|---|---|---|---|
| Gemma4-E2B | Chat | 2583 | Model Card | Samsung S26 Ultra | 557 | 47 | 3808 | 52 |
| iPhone 17 Pro | 532 | 25 | 2878 | 57 | ||||
| MacBook Pro M4 Max | 901 | 42 | 7835 | 160 | ||||
| Gemma4-E4B | Chat | 3654 | Model Card | Samsung S26 Ultra | 195 | 18 | 1293 | 22 |
| iPhone 17 Pro | 159 | 10 | 1189 | 25 | ||||
| MacBook Pro M4 Max | 277 | 27 | 2560 | 101 | ||||
| Gemma-3n-E2B | Chat | 2965 | Model Card | MacBook Pro M3 | 233 | 28 | - | - |
| Samsung S24 Ultra | 111 | 16 | 816 | 16 | ||||
| Gemma-3n-E4B | Chat | 4235 | Model Card | MacBook Pro M3 | 170 | 20 | - | - |
| Samsung S24 Ultra | 74 | 9 | 548 | 9 | ||||
| Gemma3-1B | Chat | 1005 | Model Card | Samsung S24 Ultra | 177 | 33 | 1191 | 24 |
| FunctionGemma | Base | 289 | Model Card | Samsung S25 Ultra | 2238 | 154 | - | - |
| phi-4-mini | Chat | 3906 | Model Card | Samsung S24 Ultra | 67 | 7 | 314 | 10 |
| Qwen2.5-1.5B | Chat | 1598 | Model Card | Samsung S25 Ultra | 298 | 34 | 1668 | 31 |
| Qwen3-0.6B | Chat | 586 | Model Card | Vivo X300 Pro | 165 | 9 | 580 | 21 |
| Qwen2.5-0.5B | Chat | 521 | Model Card | Samsung S24 Ultra | 251 | 30 | - | - |
Embedding Models
| Model | Modalities | Size (MB) | Details |
|---|---|---|---|
| EmbeddingGemma 2 (740M) | Text, Images, Video, Audio | 485 | Model Card |
| EmbeddingGemma 2 Text-Vision (440M) | Text, Images | 388 | Model Card |
| EmbeddingGemma 2 Text (270M) | Text | 165 | Model Card |
Report Issues
If you encounter a bug or have a feature request, report at LiteRT-LM GitHub Issues.