LiteRT-LM is the production-ready orchestration layer to run LLMs with LiteRT, engineered for high-performance, cross-platform execution.
- Cross-Platform Support: Run on Android, iOS, Web, Desktop, and IoT (e.g. Raspberry Pi).
- Hardware Acceleration: Get peak performance and system stability by leveraging GPU and NPU accelerators across diverse hardware.
- Multi-Modality: Build with LLMs that have vision and audio support.
- Tool Use: Function calling support for agentic workflows with constrained decoding for improved accuracy.
- Broad Model Support: Run Gemma, Llama, Phi-4, Qwen and more.
What's New (v0.16.0)
This release is a quick follow up to
v0.15.0
(which brought Apple Foundation Framework integration, CLI configuration, and
JavaScript API Updates).
- C API Prebuilts: Added the first versioned C API shared library prebuilts for all supported platforms. This allows natively integrating LiteRT-LM into your applications and creating language bindings without the hassle of building shared libraries.
- Experimental YNNPACK Delegate: Added the experimental YNNPACK delegate, enabled for linux arm64 builds in the LiteRT-LM CLI and Python API.
On-Device GenAI Showcase
The Google AI Edge Gallery is an experimental app designed to showcase on-device Generative AI capabilities running entirely offline using LiteRT-LM.
- Google Play: Use LLMs locally on supported Android devices.
- App Store: Experience on-device AI on your iOS device.
- GitHub Source: View the source code for the gallery app to learn how to integrate LiteRT-LM inside your own projects.
Featured Model: Gemma-4-E2B
- Model Size: 2.58 GB
Additional technical details are in the HuggingFace model card
Platform (Device) Backend Prefill (tk/s) Decode (tk/s) Time to First Token (seconds) Peak CPU Memory (MB) Android (S26 Ultra) CPU 557 47 1.8 1733 GPU 3808 52 0.3 676 iOS (iPhone 17 Pro) CPU 532 25 1.9 607 GPU 2878 56 0.3 1450 Linux (Arm 2.3 & 2.8 GHz, NVIDIA GeForce RTX 4090) CPU 260 35 4 1628 GPU 11234 143 0.1 913 macOS (MacBook Pro M4 Max) CPU 901 42 1.1 736 GPU 7835 160 0.1 1623 Windows (Intel LunarLake) CPU 435 30 2.4 3505 GPU 3751 48 0.3 3540 IoT (Raspberry Pi 5 16GB) CPU 133 8 7.8 1546
Start Building
LiteRT-LM provides APIs for several programming languages and platforms to help you build on-device AI applications quickly. Select a guide below to get started:
| Language | Status | Best For... | Documentation |
|---|---|---|---|
| CLI | ✅ Stable |
Getting started with LiteRT-LM in less than 1 min. | CLI Guide |
| Python | ✅ Stable |
Rapid prototyping, development, on desktop, Android & Raspberry Pi. | Python Guide |
| Kotlin | ✅ Stable |
Native Android apps and JVM-based desktop tools. Optimized for Coroutines. | Kotlin Guide |
| Swift | 🚀 Early Preview |
Native iOS and macOS integration with specialized Metal support. | Swift Guide |
| JavaScript (web) | 🚀 Early Preview |
Deploy models directly in web browsers with high performance. | JavaScript Guide |
| Flutter | 🚀 Community |
Cross-platform
Flutter apps
using community
flutter_gemma. |
Flutter Guide |
| C++ | ✅ Stable |
High-performance, cross-platform core logic and embedded systems. | C++ Guide |
Build from Source
If you want to customize LiteRT-LM or build it for a specific hardware configuration, you can compile it directly from the source code. For step-by-step instructions on how to set up your environment and build the framework, refer to the LiteRT-LM Build and Run Guide on GitHub.
Supported Backends & Platforms
| Acceleration | Android | iOS | macOS | Windows | Linux | IoT |
|---|---|---|---|---|---|---|
| CPU | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| GPU | ✅ | ✅ | ✅ | ✅ | ✅ | - |
| NPU | ✅ | - | - | 🚀 | - | - |
Supported Models
The following table lists models supported by LiteRT-LM. For more detailed performance numbers and model cards, visit the LiteRT Community on Hugging Face.
| Model | Type | Size (MB) | Details | Device | CPU Prefill (tk/s) | CPU Decode (tk/s) | GPU Prefill (tk/s) | GPU Decode (tk/s) |
|---|---|---|---|---|---|---|---|---|
| Gemma4-E2B | Chat | 2583 | Model Card | Samsung S26 Ultra | 557 | 47 | 3808 | 52 |
| iPhone 17 Pro | 532 | 25 | 2878 | 57 | ||||
| MacBook Pro M4 Max | 901 | 42 | 7835 | 160 | ||||
| Gemma4-E4B | Chat | 3654 | Model Card | Samsung S26 Ultra | 195 | 18 | 1293 | 22 |
| iPhone 17 Pro | 159 | 10 | 1189 | 25 | ||||
| MacBook Pro M4 Max | 277 | 27 | 2560 | 101 | ||||
| Gemma-3n-E2B | Chat | 2965 | Model Card | MacBook Pro M3 | 233 | 28 | - | - |
| Samsung S24 Ultra | 111 | 16 | 816 | 16 | ||||
| Gemma-3n-E4B | Chat | 4235 | Model Card | MacBook Pro M3 | 170 | 20 | - | - |
| Samsung S24 Ultra | 74 | 9 | 548 | 9 | ||||
| Gemma3-1B | Chat | 1005 | Model Card | Samsung S24 Ultra | 177 | 33 | 1191 | 24 |
| FunctionGemma | Base | 289 | Model Card | Samsung S25 Ultra | 2238 | 154 | - | - |
| phi-4-mini | Chat | 3906 | Model Card | Samsung S24 Ultra | 67 | 7 | 314 | 10 |
| Qwen2.5-1.5B | Chat | 1598 | Model Card | Samsung S25 Ultra | 298 | 34 | 1668 | 31 |
| Qwen3-0.6B | Chat | 586 | Model Card | Vivo X300 Pro | 165 | 9 | 580 | 21 |
| Qwen2.5-0.5B | Chat | 521 | Model Card | Samsung S24 Ultra | 251 | 30 | - | - |
Report Issues
If you encounter a bug or have a feature request, report at LiteRT-LM GitHub Issues.