InterpreterApi

interface InterpreterApi : AutoCloseable

Interface to TensorFlow Lite model interpreter, excluding experimental methods.

An InterpreterApi instance encapsulates a pre-trained TensorFlow Lite model, in which operations are executed for model inference.

For example, if a model takes only one input and returns only one output: 

try (InterpreterApi interpreter =
    new InterpreterApi.create(file_of_a_tensorflowlite_model)) {
  interpreter.run(input, output);
}

If a model takes multiple inputs or outputs: 

Object[] inputs = {input0, input1, ...};
Map<Integer, Object> map_of_indices_to_outputs = new HashMap<>();
FloatBuffer ith_output = FloatBuffer.allocateDirect(3 * 2 * 4);  // Float tensor, shape 3x2x4.
ith_output.order(ByteOrder.nativeOrder());
map_of_indices_to_outputs.put(i, ith_output);
try (InterpreterApi interpreter =
    new InterpreterApi.create(file_of_a_tensorflowlite_model)) {
  interpreter.runForMultipleInputsOutputs(inputs, map_of_indices_to_outputs);
}

If a model takes or produces string tensors: 

String[] input = {"foo", "bar"};  // Input tensor shape is [2].
String[][] output = new String[3][2];  // Output tensor shape is [3, 2].
try (InterpreterApi interpreter =
    new InterpreterApi.create(file_of_a_tensorflowlite_model)) {
  interpreter.runForMultipleInputsOutputs(input, output);
}

Note that there's a distinction between shape [] and shape[1]. For scalar string tensor outputs: 

String[] input = {"foo"};  // Input tensor shape is [1].
ByteBuffer outputBuffer = ByteBuffer.allocate(OUTPUT_BYTES_SIZE);  // Output tensor shape is [].
try (Interpreter interpreter = new Interpreter(file_of_a_tensorflowlite_model)) {
  interpreter.runForMultipleInputsOutputs(input, outputBuffer);
}
byte[] outputBytes = new byte[outputBuffer.remaining()];
outputBuffer.get(outputBytes);
// Below, the `charset` can be StandardCharsets.UTF_8.
String output = new String(outputBytes, charset);

Orders of inputs and outputs are determined when converting TensorFlow model to TensorFlowLite model with Toco, as are the default shapes of the inputs.

When inputs are provided as (multi-dimensional) arrays, the corresponding input tensor(s) will be implicitly resized according to that array's shape. When inputs are provided as java.nio.Buffer types, no implicit resizing is done; the caller must ensure that the java.nio.Buffer byte size either matches that of the corresponding tensor, or that they first resize the tensor via resizeInput. Tensor shape and type information can be obtained via the Tensor class, available via getInputTensor and getOutputTensor.

WARNING:InterpreterApi instances are not thread-safe.

WARNING:An InterpreterApi instance owns resources that must be explicitly freed by invoking close

The TFLite library is built against NDK API 19. It may work for Android API levels below 19, but is not guaranteed.

Summary

Nested types

An options class for controlling runtime interpreter behavior.

Enum to represent where to get the TensorFlow Lite runtime implementation from.

Public functions
Unit

Explicitly updates allocations for all tensors, if necessary.
Unit

Release resources associated with the InterpreterApi instance.
java-static InterpreterApi!
create(byteBuffer: ByteBuffer, options: InterpreterApi.Options!)

Constructs an InterpreterApi instance, using the specified model and options.
java-static InterpreterApi!
create(modelFile: File, options: InterpreterApi.Options!)

Constructs an InterpreterApi instance, using the specified model and options.
Int

Gets index of an input given the op name of the input.
Tensor!
getInputTensor(inputIndex: Int)

Gets the Tensor associated with the provided input index.
Int

Gets the number of input tensors.
Tensor!
getInputTensorFromSignature(inputName: String!, signatureKey: String!)

Gets the Tensor associated with the provided input name and signature method name.
Long!

Returns native inference timing.
Int

Gets index of an output given the op name of the output.
Tensor!
getOutputTensor(outputIndex: Int)

Gets the Tensor associated with the provided output index.
Int

Gets the number of output Tensors.
Tensor!
getOutputTensorFromSignature(outputName: String!, signatureKey: String!)

Gets the Tensor associated with the provided output name in specific signature method.
Array<String!>!
getSignatureInputs(signatureKey: String!)

Gets the list of SignatureDefs inputs for method signatureKey.
Array<String!>!

Gets the list of SignatureDef exported method names available in the model.
Array<String!>!
getSignatureOutputs(signatureKey: String!)

Gets the list of SignatureDefs outputs for method signatureKey.
Unit
resizeInput(index: Int, dims: IntArray)

Resizes the input at the given index of the native model to the given dims.
Unit
resizeInput(index: Int, dims: IntArray, strict: Boolean)

Resizes the input at the given index of the native model to the given dims.
Unit
run(input: Any!, output: Any!)

Runs model inference if the model takes only one input, and provides only one output.
Unit
runForMultipleInputsOutputs(
    inputs: Array<Any!>,
    outputs: (Mutable)Map<Int!, Any!>
)

Runs model inference if the model takes multiple inputs, or returns multiple outputs.
Unit
runSignature(
    inputs: (Mutable)Map<String!, Any!>,
    outputs: (Mutable)Map<String!, Any!>
)

Same as runSignature but doesn't require passing a signatureKey, assuming the model has one SignatureDef.
Unit
runSignature(
    inputs: (Mutable)Map<String!, Any!>,
    outputs: (Mutable)Map<String!, Any!>,
    signatureKey: String!
)

Runs model inference based on SignatureDef provided through signatureKey.

Public functions

allocateTensors

fun allocateTensors(): Unit

Explicitly updates allocations for all tensors, if necessary.

This will propagate shapes and memory allocations for dependent tensors using the input tensor shape(s) as given.

Note: This call is *purely optional*. Tensor allocation will occur automatically during execution if any input tensors have been resized. This call is most useful in determining the shapes for any output tensors before executing the graph, e.g., 

interpreter.resizeInput(0, new int[]{1, 4, 4, 3}));
interpreter.allocateTensors();
FloatBuffer input = FloatBuffer.allocate(interpreter.getInputTensor(0).numElements());
// Populate inputs...
FloatBuffer output = FloatBuffer.allocate(interpreter.getOutputTensor(0).numElements());
interpreter.run(input, output)
// Process outputs...

Note: Some graphs have dynamically shaped outputs, in which case the output shape may not fully propagate until inference is executed.

Throws
java.lang.IllegalStateException

if the graph's tensors could not be successfully allocated.

close

@Override
fun close(): Unit

Release resources associated with the InterpreterApi instance.

create

java-static fun create(byteBuffer: ByteBuffer, options: InterpreterApi.Options!): InterpreterApi!

Constructs an InterpreterApi instance, using the specified model and options. The model will be read from a ByteBuffer.

Parameters
byteBuffer: ByteBuffer

A pre-trained TF Lite model, in binary serialized form. The ByteBuffer should not be modified after the construction of an InterpreterApi instance. The ByteBuffer can be either a MappedByteBuffer that memory-maps a model file, or a direct ByteBuffer of nativeOrder() that contains the bytes content of a model.
options: InterpreterApi.Options!

A set of options for customizing interpreter behavior.
Throws
java.lang.IllegalArgumentException

if byteBuffer is not a MappedByteBuffer nor a direct ByteBuffer of nativeOrder.

create

java-static fun create(modelFile: File, options: InterpreterApi.Options!): InterpreterApi!

Constructs an InterpreterApi instance, using the specified model and options. The model will be loaded from a file.

Parameters
modelFile: File

A file containing a pre-trained TF Lite model.
options: InterpreterApi.Options!

A set of options for customizing interpreter behavior.
Throws
java.lang.IllegalArgumentException

if modelFile does not encode a valid TensorFlow Lite model.

getInputIndex

fun getInputIndex(opName: String!): Int

Gets index of an input given the op name of the input.

Throws
java.lang.IllegalArgumentException

if opName does not match any input in the model used to initialize the interpreter.

getInputTensor

fun getInputTensor(inputIndex: Int): Tensor!

Gets the Tensor associated with the provided input index.

Throws
java.lang.IllegalArgumentException

if inputIndex is negative or is not smaller than the number of model inputs.

getInputTensorCount

fun getInputTensorCount(): Int

Gets the number of input tensors.

getInputTensorFromSignature

fun getInputTensorFromSignature(inputName: String!, signatureKey: String!): Tensor!

Gets the Tensor associated with the provided input name and signature method name.

Parameters
inputName: String!

Input name in the signature.
signatureKey: String!

Signature key identifying the SignatureDef, can be null if the model has one signature.
Throws
java.lang.IllegalArgumentException

if inputName or signatureKey is null or empty, or invalid name provided.

getLastNativeInferenceDurationNanoseconds

fun getLastNativeInferenceDurationNanoseconds(): Long!

Returns native inference timing.

Throws
java.lang.IllegalArgumentException

if the model is not initialized by the interpreter.

getOutputIndex

fun getOutputIndex(opName: String!): Int

Gets index of an output given the op name of the output.

Throws
java.lang.IllegalArgumentException

if opName does not match any output in the model used to initialize the interpreter.

getOutputTensor

fun getOutputTensor(outputIndex: Int): Tensor!

Gets the Tensor associated with the provided output index.

Note: Output tensor details (e.g., shape) may not be fully populated until after inference is executed. If you need updated details *before* running inference (e.g., after resizing an input tensor, which may invalidate output tensor shapes), use allocateTensors to explicitly trigger allocation and shape propagation. Note that, for graphs with output shapes that are dependent on input *values*, the output shape may not be fully determined until running inference.

Throws
java.lang.IllegalArgumentException

if outputIndex is negative or is not smaller than the number of model outputs.

getOutputTensorCount

fun getOutputTensorCount(): Int

Gets the number of output Tensors.

getOutputTensorFromSignature

fun getOutputTensorFromSignature(outputName: String!, signatureKey: String!): Tensor!

Gets the Tensor associated with the provided output name in specific signature method.

Note: Output tensor details (e.g., shape) may not be fully populated until after inference is executed. If you need updated details *before* running inference (e.g., after resizing an input tensor, which may invalidate output tensor shapes), use allocateTensors to explicitly trigger allocation and shape propagation. Note that, for graphs with output shapes that are dependent on input *values*, the output shape may not be fully determined until running inference.

Parameters
outputName: String!

Output name in the signature.
signatureKey: String!

Signature key identifying the SignatureDef, can be null if the model has one signature.
Throws
java.lang.IllegalArgumentException

if outputName or signatureKey is null or empty, or invalid name provided.

getSignatureInputs

fun getSignatureInputs(signatureKey: String!): Array<String!>!

Gets the list of SignatureDefs inputs for method signatureKey.

getSignatureKeys

fun getSignatureKeys(): Array<String!>!

Gets the list of SignatureDef exported method names available in the model.

getSignatureOutputs

fun getSignatureOutputs(signatureKey: String!): Array<String!>!

Gets the list of SignatureDefs outputs for method signatureKey.

resizeInput

fun resizeInput(index: Int, dims: IntArray): Unit

Resizes the input at the given index of the native model to the given dims.

Throws
java.lang.IllegalArgumentException

if index is negative or is not smaller than the number of model inputs; or if error occurs when resizing the input at the given index.

resizeInput

fun resizeInput(index: Int, dims: IntArray, strict: Boolean): Unit

Resizes the input at the given index of the native model to the given dims.

When `strict` is True, only unknown dimensions can be resized. Unknown dimensions are indicated as `-1` in the array returned by `Tensor.shapeSignature()`.

Throws
java.lang.IllegalArgumentException

if index is negative or is not smaller than the number of model inputs; or if error occurs when resizing the input at the given index. Additionally, the error occurs when attempting to resize a tensor with fixed dimensions when `strict` is True.

run

fun run(input: Any!, output: Any!): Unit

Runs model inference if the model takes only one input, and provides only one output.

Warning: The API is more efficient if a Buffer (preferably direct, but not required) is used as the input/output data type. Please consider using Buffer to feed and fetch primitive data for better performance. The following concrete Buffer types are supported:

  • ByteBuffer - compatible with any underlying primitive Tensor type.
  • FloatBuffer - compatible with float Tensors.
  • IntBuffer - compatible with int32 Tensors.
  • LongBuffer - compatible with int64 Tensors.
Note that boolean types are only supported as arrays, not Buffers, or as scalar inputs.
Parameters
input: Any!

an array or multidimensional array, or a Buffer of primitive types including int, float, long, and byte. Buffer is the preferred way to pass large input data for primitive types, whereas string types require using the (multi-dimensional) array input path. When a Buffer is used, its content should remain unchanged until model inference is done, and the caller must ensure that the Buffer is at the appropriate read position. A null value is allowed only if the caller is using a Delegate that allows buffer handle interop, and such a buffer has been bound to the input Tensor.
output: Any!

a multidimensional array of output data, or a Buffer of primitive types including int, float, long, and byte. When a Buffer is used, the caller must ensure that it is set the appropriate write position. A null value is allowed, and is useful for certain cases, e.g., if the caller is using a Delegate that allows buffer handle interop, and such a buffer has been bound to the output Tensor (see also Interpreter.Options#setAllowBufferHandleOutput(boolean)), or if the graph has dynamically shaped outputs and the caller must query the output Tensor shape after inference has been invoked, fetching the data directly from the output tensor (via asReadOnlyBuffer).
Throws
java.lang.IllegalArgumentException

if input is null or empty, or if an error occurs when running inference.
java.lang.IllegalArgumentException

(EXPERIMENTAL, subject to change) if the inference is interrupted by setCancelled(true).

runForMultipleInputsOutputs

fun runForMultipleInputsOutputs(
    inputs: Array<Any!>,
    outputs: (Mutable)Map<Int!, Any!>
): Unit

Runs model inference if the model takes multiple inputs, or returns multiple outputs.

Warning: The API is more efficient if Buffers (preferably direct, but not required) are used as the input/output data types. Please consider using Buffer to feed and fetch primitive data for better performance. The following concrete Buffer types are supported:

  • ByteBuffer - compatible with any underlying primitive Tensor type.
  • FloatBuffer - compatible with float Tensors.
  • IntBuffer - compatible with int32 Tensors.
  • LongBuffer - compatible with int64 Tensors.
Note that boolean types are only supported as arrays, not Buffers, or as scalar inputs.

Note: null values for individual elements of inputs and outputs is allowed only if the caller is using a Delegate that allows buffer handle interop, and such a buffer has been bound to the corresponding input or output Tensor(s).

Parameters
inputs: Array<Any!>

an array of input data. The inputs should be in the same order as inputs of the model. Each input can be an array or multidimensional array, or a Buffer of primitive types including int, float, long, and byte. Buffer is the preferred way to pass large input data, whereas string types require using the (multi-dimensional) array input path. When Buffer is used, its content should remain unchanged until model inference is done, and the caller must ensure that the Buffer is at the appropriate read position.
outputs: (Mutable)Map<Int!, Any!>

a map mapping output indices to multidimensional arrays of output data or Buffers of primitive types including int, float, long, and byte. It only needs to keep entries for the outputs to be used. When a Buffer is used, the caller must ensure that it is set the appropriate write position. The map may be empty for cases where either buffer handles are used for output tensor data, or cases where the outputs are dynamically shaped and the caller must query the output Tensor shape after inference has been invoked, fetching the data directly from the output tensor (via asReadOnlyBuffer).
Throws
java.lang.IllegalArgumentException

if inputs is null or empty, if outputs is null, or if an error occurs when running inference.

runSignature

fun runSignature(
    inputs: (Mutable)Map<String!, Any!>,
    outputs: (Mutable)Map<String!, Any!>
): Unit

Same as runSignature but doesn't require passing a signatureKey, assuming the model has one SignatureDef. If the model has more than one SignatureDef it will throw an exception.

runSignature

fun runSignature(
    inputs: (Mutable)Map<String!, Any!>,
    outputs: (Mutable)Map<String!, Any!>,
    signatureKey: String!
): Unit

Runs model inference based on SignatureDef provided through signatureKey.

See run for more details on the allowed input and output data types.

Parameters
inputs: (Mutable)Map<String!, Any!>

A map from input name in the SignatureDef to an input object.
outputs: (Mutable)Map<String!, Any!>

A map from output name in SignatureDef to output data. This may be empty if the caller wishes to query the Tensor data directly after inference (e.g., if the output shape is dynamic, or output buffer handles are used).
signatureKey: String!

Signature key identifying the SignatureDef.
Throws
java.lang.IllegalArgumentException

if inputs is null or empty, if outputs or signatureKey is null, or if an error occurs when running inference.