Announcement: All noncommercial projects registered to use Earth Engine before April 15, 2025 must verify noncommercial eligibility to maintain access. If you have not verified by September 26, 2025, your access may be on hold.

Hosted Model Payload Formats Supported on Earth Engine

AI-generated Key Takeaways

To use custom-trained models in Earth Engine, configure them for compatibility with Vertex AI for seamless inference requests.
Earth Engine supports various payload formats, including gRPC and HTTP, for sending inference requests to your model, with gRPC being preferred for TensorFlow models due to its lower latency.
When using gRPC, select the appropriate format (GRPC_TF_TENSORS, GRPC_SERIALIZED_TF_TENSORS, or GRPC_SERIALIZED_TF_EXAMPLES) based on your model's requirements and whether it needs a gRPC prediction handler.
For HTTP payloads, Earth Engine offers options like SERIALIZED_TF_TENSORS (default and efficient for TensorFlow), RAW_JSON (flexible for PyTorch and AutoML), and ND_ARRAYS (for PyTorch with numerical data).
Carefully choose the payload format that best aligns with your model framework and data structure for optimal performance and compatibility.

Your custom trained model must be configured such that Earth Engine can send well-formed, interpretable inference requests to the model as it is hosted on Vertex AI.

Supported Model Inputs

Earth Engine constructs requests on your behalf when performing inference. Specify the payload format that EE will send requests with using the payloadFormat parameter when you instantiate the model connector with ee.Model.fromVertexAi.

gRPC Prediction Payloads

All hosted TensorFlow models can send predictions over the gRPC protocol. This is the preferred way to connect hosted models with Earth Engine as it will result in lower prediction latency and higher reliability.

GRPC_TF_TENSORS

Use the GRPC_TF_TENSORS payload format to use gRPC with TensorFlow models. All properties and or bands will be encoded in a single PredictRequest. This PredictRequest will be converted to a dictionary of tensors for your model to use.

GRPC_SERIALIZED_TF_TENSORS

Use the GRPC_SERIALIZED_TF_TENSORS format if you want to migrate a Cloud AI Platform model that was previously already integrated with Earth Engine without having to modify the model. You will need to re-upload and re-deploy if container_grpc_ports is not set on your model in Vertex AI.

GRPC_SERIALIZED_TF_EXAMPLES

Use the GRPC_SERAILZED_TF_EXAMPLES for models that support tf.Example protocol buffers. Earth Engine will send a single tensor named "input" that contains the utf-8 encoded proto ByteString of an Example proto.

HTTP API Payloads

Vertex AI supports connecting to HTTP inference endpoints. Earth Engine supports several of the common HTTP payload formats. By default all Vertex AI custom models support the HTTP inference API.

SERIALIZED_TF_TENSORS

This is the default payloadFormat when connecting to a hosted model in Vertex AI. This payload format is the most efficient of the HTTP payload formats when using TensorFlow models.

Earth Engine will construct the inputs in the following way: for every band and property required for your inference request will be a single key-value pair in the instances object sent to your hosted model.

Each key will be the band or property name and each value will be a Base64 encoded TensorProto ByteString as a string_val.

RAW_JSON

For other model frameworks the most flexible format we can send is a JSON dictionary of named inputs and values. This payload format works well with PyTorch and AutoML models by default.

However do note that all numerical values will be converted into JSON strings. For example, to represent the number we encode 12.345 this as the string "12.345". Large inference payloads are not well supported with this payload format.

ND_ARRAYS

This is similar to RAW_JSON payload format but will omit the keys and only pass in a list of numbers in the same format of calling to_list() on a NumPy ndarray. This payload format works well with PyTorch natively.