CLI Configuration

Learn how to configure global defaults and per-model settings for the LiteRT-LM CLI using configuration files.

Overview

The LiteRT-LM CLI provides a centralized configuration system (config.json) to manage model execution parameters, hardware backend selection, sampling options, and performance optimizations across commands.

Why Use Configuration Files?

Configuration files solve two key operational challenges:

1. Eliminate Repetitive Flags: Instead of passing long command-line flags (such as --backend=gpu --cpu-thread-count=16 --max-num-tokens=2048) every time you execute litert-lm run or litert-lm benchmark, you can define persistent global or per-model defaults.
2. Configure OpenAI-Compatible Server: When running the OpenAI-Compatible Server (litert-lm serve), standard OpenAI API requests (e.g., POST /v1/chat/completions) do not include LiteRT-LM-specific parameters like hardware backend selection (npu vs gpu), vision/audio backends, or Multi-Token Prediction (MTP / speculative decoding). config.json allows the server to automatically resolve these settings based on the requested model identifier.

Configuration File Location

By default, the CLI looks for a JSON configuration file at:

You can override the configuration file path with the --config /path/to/config.json to any CLI command to specify an explicit configuration file.

Configuration Structure

The configuration file (config.json) is a JSON object with two primary top-level sections:

{
  "default": {
    /* Global ModelConfig settings applied to all models */
  },
  "models": {
    "model-id-1": {
      /* ModelConfig overrides for model-id-1 */
    },
    "model-id-2": {
      /* ModelConfig overrides for model-id-2 */
    }
  }
}

• default: A ModelConfig object containing default settings applied to any model executed by the CLI.
• models: A map of model IDs to ModelConfig objects. Settings specified for a model override any matching fields in default.

Settings Precedence

When resolving execution parameters for a model, LiteRT-LM applies settings according to a strict hierarchy defined in config.json (see Configuration Structure) and runtime command-line arguments. Higher-priority sources override lower-priority ones:

OpenAI-Compatible Server Integration

When running the OpenAI-Compatible Server (litert-lm serve), the server dynamically resolves configuration parameters for incoming HTTP requests:

1. A client sends an OpenAI-compatible request, for example:

   curl http://localhost:9379/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "gemma4-e2b",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'
   

2. The server extracts the "model" identifier ("gemma4-e2b").

3. The server checks config.json for a matching key under models.gemma4-e2b. If found, it merges those settings over the global default configuration.

4. The underlying engine is initialized or reused with the resolved hardware acceleration (gpu/npu), thread counts, and speculative decoding options.

This enables you to define custom server aliases with tailored performance profiles (such as lightweight CPU models vs high-performance GPU/MTP models) without modifying client code.

Supported Configuration Keys

All properties in a ModelConfig object are optional. The following table summarizes all supported configuration keys, their data types, constraints, supported CLI versions, and descriptions.

Example Configuration

Here is a config.json example demonstrating global defaults and per-model overrides:

{
  "default": {
    "backend": "gpu",       // use GPU as the main backend, by default
    "cpu_thread_count": 16, // when using CPU, use 16 threads, by default
  },
  "models": {
    "gemma4-e2b": {
      "speculative_decoding": true,  // enable MTP
    },
    "gemma4-26b": {
      "max_num_tokens": 65536,   // set larger context for agentic use case
      "thinking": true,          // enable thinking by default
      "thinking_budget": 4096    // set a thinking budget
    }
  }
}