Note: The REST API contains new and advanced features that may not be suitable for all users. If you are new to Earth Engine, please get started with the JavaScript guide.
Earth Engine can load images from Cloud Optimized GeoTiffs (COGs) in Google Cloud Storage (learn more). This notebook demonstrates how to create Earth Engine assets backed by COGs. An advantage of COG-backed assets is that the spatial and metadata fields of the image will be indexed at asset creation time, making the image more performant in collections. (In contrast, an image created through ee.Image.loadGeoTIFF
and put into a collection will require a read of the GeoTiff for filtering operations on the collection.) A disadvantage of COG-backed assets is that they may be several times slower than standard assets when used in computations.
To create a COG-backed asset, make a POST
request to the Earth Engine CreateAsset
endpoint. As shown in the following, this request must be authorized to create an asset in your user folder.
Start an authorized session
To be able to make an Earth Engine asset in your user folder, you need to be able to authenticate as yourself when you make the request. You can use credentials from the Earth Engine authenticator to start an AuthorizedSession
. You can then use the AuthorizedSession
to send requests to Earth Engine.
import ee
from google.auth.transport.requests import AuthorizedSession
ee.Authenticate() # or !earthengine authenticate --auth_mode=gcloud
# Specify the cloud project you want associated with Earth Engine requests.
ee_project = 'your-project'
session = AuthorizedSession(
ee.data.get_persistent_credentials().with_quota_project(ee_project)
)
Request body
The request body is an instance of an EarthEngineAsset. This is where the path to the COG is specified, along with other useful properties. Note that the image is a small area exported from the composite made in this example script. See this doc for details on exporting a COG.
Earth Engine will determine the bands, geometry, and other relevant information from the metadata of the TIFF. The only other fields that are accepted when creating a COG-backed asset are properties
, start_time
, and end_time
.
import json
from pprint import pprint
# Request body as a dictionary.
request = {
'type': 'IMAGE',
'gcs_location': {
'uris': ['gs://ee-docs-demos/COG_demo.tif']
},
'properties': {
'source': 'https://code.earthengine.google.com/d541cf8b268b2f9d8f834c255698201d'
},
'startTime': '2016-01-01T00:00:00.000000000Z',
'endTime': '2016-12-31T15:01:23.000000000Z',
}
pprint(json.dumps(request))
Send the request
Make the POST request to the Earth Engine projects.assets.create
endpoint.
# Specify the cloud project you want write the asset to.
project_folder = 'your-project'
# Specify a folder (or ImageCollection) name and the new asset name.
asset_id = 'cog-collection/your-cog-asset'
url = 'https://earthengine.googleapis.com/v1alpha/projects/{}/assets?assetId={}'
response = session.post(
url = url.format(project_folder, asset_id),
data = json.dumps(request)
)
pprint(json.loads(response.content))
Details on COG-backed assets
Permissions
The ACLs of COG-backed Earth Engine assets and the underlying data are managed separately. If a COG-backed asset is shared in Earth Engine, it is the owner's responsibility to ensure that the data in GCS is shared with the same parties. If the data is not visible, Earth Engine will return an error of the form "Failed to load the GeoTIFF at gs://my-bucket/my-object#123456
" (123456 is the generation of the object).
Generations
When a COG-backed asset is created, Earth Engine reads the metadata of the TIFF in Cloud Storage and creates an asset store entry. The URI associated with that entry can have a generation. See the object versioning docs for details on generations. If a generation is specified (e.g., gs://foo/bar#123
), Earth Engine will use it. If a generation is not specified, Earth Engine will use the latest generation of the object.
That means that if the object in GCS is updated, Earth Engine will return a "Failed to load the GeoTIFF at gs://my-bucket/my-object#123456
" error because the expected object no longer exists (unless the bucket enables multiple object versions). This policy is designed to keep metadata of the asset in sync with the metadata of the object.
Configuration
In terms of how a COG should be configured, the TIFF MUST be:
Tiled, where the tile dimensions are either:
- 16x16
- 32x32
- 64x64
- 128x128
- 256x256
- 512x512
- 1024x1024
Arranged so that all IFDs are at the beginning.
For best performance:
- Use tile dimensions of 256x256.
- Include power of 2 overviews.
- Compress the files:
- Use DEFLATE (or LZW).
- Set the compressor's zlevel to the maximum of 9.
- Use a predictor of 2. Sometimes a predictor of 3 will do better.
See your software's documentation for how to create the best COG. For example, GDAL has this page: GTiff – GeoTIFF File Format.