Earth Engine provides ee.Model as a connector to models hosted on
Vertex AI. Specifically, Earth Engine will send image or table data as online prediction requests to a
trained model deployed on a Vertex AI endpoint. The model outputs are then available as Earth
Engine images or tables.
TensorFlow is an open source ML platform that supports advanced ML methods such as deep learning. The Earth Engine API provides methods for importing/exporting imagery, training and testing data in TFRecord format. See the ML examples page for demonstrations that use TensorFlow with data from Earth Engine. See the TFRecord page for details about how Earth Engine writes data to TFRecord files.
ee.Model
The ee.Model package handles interaction with hosted machine learning models.
Connecting to models hosted on Vertex AI
A new ee.Model instance can be created with
ee.Model.fromVertexAi. This is an ee.Model object that packages Earth Engine data into tensors,
forwards them as predict requests to
Vertex AI then automatically reassembles the
responses into Earth Engine data types.
Earth Engine supports TensorFlow (e.g. SavedModel format), PyTorch and AutoML models. To prepare a model for hosting, save it, import it, then deploy the model to an endpoint.
To interact with Earth Engine, a hosted model's inputs/outputs need to be compatible with a
supported interchange format. For hosted TensorFlow models, this is the TensorProto
interchange format, specifically serialized TensorProtos in base64 (reference). This can be done programmatically, as shown on the
ML examples page, after training and before
saving, or by reloading saved models. Other supported format include JSON (RAW_JSON
mode) and multi-dimensional arrays (ND_ARRAYS). See
the ee.Model.fromVertexAi() reference docs
for details.
To use a model with ee.Model.fromVertexAi(), you must have sufficient permissions
to use the model. Specifically, you (or anyone who uses the model) needs at least the
Vertex AI user role. You control permissions for your Cloud Project using
Identify and Access Management (IAM) controls.
Regions
When deploying your model and endpoint, you will need to specify which region to deploy to.
The us-central1 region is recommended since it will likely perform best due to
proximity to Earth Engine servers, but almost any region will work. See the
Vertex AI docs for
details about Vertex AI regions and what features each one supports.
If you are migrating from AI Platform, note that Vertex AI does not have a global endpoint,
and ee.Model.fromVertexAi() does not have a region parameter.
Costs
Image Predictions
Use model.predictImage() to make predictions on an ee.Image using a
hosted model. The return type of predictImage() is an ee.Image which
can be added to the map, used in other computations, exported, etc. The image is forwarded to
the hosted model in tiles. You control how the image is tiled using the
inputTileSize, inputOverlapSize, and
outputTileSize parameters. For example, a fully convolutional model may segment
256x256xChannels shaped inputs (scale is unchanged), but the edges of the tiles can be
discarded by the specified overlap. Alternatively, an image classification model may accept
256x256xChannels shaped inputs, but return a 1x1x1 output of probability (reduced resolution)
of some tile characteristic. Note that Earth Engine will always forward each band in the input
Image as a 3D - specifically (inputTileSize[0], inputTileSize[1], 1) shaped - float32 tensor,
even if the band's values are scalar.
Nearly all convolutional models will expect input of a fixed projection (that of the data on
which the model was trained). In this case, set the fixInputProj parameter to
true in your call to ee.Model.fromVertexAi() and specify the
data's projection in the proj parameter. When visualizing predictions, use
caution when zooming out on a model that has a fixed input projection. As explained in the
EE reprojecting guide, zooming to
a large spatial scope can result in requests for too much data, too many concurrent requests,
or too much billable compute.
The input Image's projection and the model's fixed projection will affect the output values. See the EE Reprojection guide.
Table Predictions
Use model.predictProperties() to make predictions on an
ee.FeatureCollection. Inputs are stored as properties of the table. The inputs
and outputs can be scalar string values, scalar boolean values, or numeric values of any
shape, from scalars to multidimensional arrays. The outputs of the model are stored as new
properties in the output table.