Optimize inference speed with prefix caching

Prefix caching is a feature that reduces inference time by storing and reusing the intermediate LLM state of processing a shared and recurring prompt prefix part. To enable prefix caching, you only have to separate the static prefix from the dynamic suffix in your API request.

Prefix caching currently only supports text-only input, so you shouldn't use this feature if you're providing an image in your prompt.

Implement prefix caching

To enable prefix caching for Prompt API, add the shared portion of the prompt into the promptPrefix field, as shown in the following code snippet:

Kotlin

val promptPrefix = "Reverse the given sentence: "
val dynamicSuffix = "Hello World"

val result = generativeModel.generateContent(
  generateContentRequest(TextPart(dynamicSuffix)) {
    promptPrefix = PromptPrefix(promptPrefix)
  }
)

Java

String promptPrefix = "Reverse the given sentence: ";
String dynamicSuffix = "Hello World";

GenerateContentResponse response = generativeModelFutures.generateContent(
    new GenerateContentRequest.Builder(new TextPart(dynamicSuffix))
    .setPromptPrefix(new PromptPrefix(promptPrefix))
    .build())
    .get();

In the preceding snippet, the dynamicSuffix is passed as the main content, and the promptPrefix is provided separately.

Estimated performance gains

Without prefix caching

With prefix cache-hit

(Prefix cache-miss may occur when prefix is used for the first time)

Pixel 9 with 300-token fixed prefix and a 50-token dynamic suffix prompt

0.82 seconds

0.45 seconds

Pixel 9 with a 1,000-token fixed prefix and a 100-token dynamic suffix prompt

2.11 seconds

0.5 seconds

Storage considerations

With prefix caching, cache files are saved on the client application's private storage, which increases your app's storage usage. Encrypted cache files and their associated metadata, including original prefix text, are stored. Keep the following storage considerations in mind:

  • The number of caches is managed by an LRU (Least Recently Used) mechanism. Least used caches are deleted automatically when exceeding the max total cache amount.
  • Prompt cache sizes are dependent on the length of the prefix.

  • To clear all caches created from prefix caching, use the generativeMode.clearCaches() method.