Consumption Monitoring

This page describes how to create charts to monitor Earth Engine compute and storage consumption using Cloud Monitoring.

There are other ways to monitor Earth Engine usage from the Cloud Console, which are not the focus of the document but include:

• The APIs & Services > Metrics page, which shows basic metrics including traffic (number of requests), errors and latency (per API method, response code or credentials).
• The APIs & Services > Quotas & System Limits page, which shows the amount of stored assets in bytes and the number of read requests for the assets.
• The APIs & Services > Credentials page, which shows which credentials (e.g, service accounts) have been used to access the API.

View consumption in Cloud Monitoring

Chart metrics in Metrics Explorer

1. Go to the Monitoring > Metrics Explorer page in the Cloud Console.

   Go to Metrics Explorer

2. Select the name of your project if it is not already selected at the top of the page.

   • Note: You must have the appropriate Identity and Access Management (IAM) permissions to access monitoring data.

3. Click Select a metric to choose a metric to add to the chart.

   • Earth Engine metrics are under the Earth Engine Cloud Project resource.
   • By default, only resources and metrics that were active in the past hour are visible. Adjust the time range or un-check the "Active" filter to see more metrics.

4. Once you've selected a metric, click Apply.

5. In the top pane, configure the drop-down filters to set how to visualize the data.

   • By default, the explorer will show a rate aggregation for compute metrics. See the Units and Aligners section for details on choosing a different Aligner and showing explicit units.
   • For example, to see the total completed batch compute used per workload_tag (see Workload tags section) over the past week, you could choose the following settings. Here, each data point represents the total amount of EECU-hours each completed batch task used.
     
     

The Cloud Monitoring documentation provides more guides on using Cloud Monitoring. In particular, the Select the metrics to chart page provides a detailed overview of different ways to build queries, and the Filtering and aggregation page provides more information about configuring the time series.

Available Metrics

Units and Aligners

By default, compute metrics will be displayed as a unitless rate of the average EECU-seconds used per-second over the Min interval (default 1 minute).

To see the raw EECU-time used with explicit units, click the Aggregation field in your query and choose "Configure aligner" from the resulting menu. This replaces the aggregation operation with two new operations: Grouping and Alignment function. Choosing "Grouping: Sum" and "Alignment function: Sum" will construct a graph with explicit units, representing the total EECU-time being used at each data point. See the Aligner reference for a list of possible Aligners.

Workload Tags

Workload tags are labels for monitoring specific computations within Earth Engine. Use setDefaultWorkloadTag to bind all computations in the script to a default workload tag unless one is explicitly set with ee.data.setWorkloadTag, in which case the default is overridden. These methods set the workload_tag label for specific computations and export tasks.

You can then monitor and track tagged computations in the Metrics Explorer using the Earth Engine Cloud Project > Project > Used EECUs metric, and grouping or filtering by workload_tag.

For example, to monitor the EECUs used for an image computation and/or and export:

// Cloud masking function.
function maskL8sr(image) {
  var qaMask = image.select('QA_PIXEL').bitwiseAnd(parseInt('11111', 2)).eq(0);
  var saturationMask = image.select('QA_RADSAT').eq(0);

  // Apply the scaling factors to the appropriate bands.
  var opticalBands = image.select('SR_B.').multiply(0.0000275).add(-0.2);
  var thermalBands = image.select('ST_B.*').multiply(0.00341802).add(149.0);

  // Replace the original bands with the scaled ones and apply the masks.
  return image.addBands(opticalBands, null, true)
      .addBands(thermalBands, null, true)
      .updateMask(qaMask)
      .updateMask(saturationMask);
}

// Set a default workload tag.
ee.data.setDefaultWorkloadTag('landsat-compositing')
var collection = ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')
                     .filterDate('2020-01-01', '2021-01-01')
                     .map(maskL8sr);
var composite = collection.select('SR_B.').median();
print(composite);

// Set a new workload tag.
ee.data.setWorkloadTag('export-jobs');
Export.image.toAsset(composite);
ee.data.resetWorkloadTag(); // Reset to landsat-compositing

ee.data.resetWorkloadTag(true); // Reset back to empty

import ee
import geemap.core as geemap

# Authenticate, then initialize with your Cloud Project.
ee.Initialize(project='your-project')

# Set a default workload tag.
ee.data.setDefaultWorkloadTag('landsat-compositing')
composite = (
    ee.ImageCollection('LANDSAT/LC08/C02/T1_L2')
    .filterDate('2020-01-01', '2021-01-01')
    .median()
)

# Set a workload tag for export.
ee.data.setWorkloadTag('export-jobs')
ee.batch.Export.image.toAsset(composite).start()
ee.data.resetWorkloadTag()

# Alternatively, use a workload tag with with.
with ee.data.workloadTagContext('export-jobs'):
  ee.batch.Export.image.toAsset(composite).start()

In this example, all computations are annotated with the landsat-compositing tag (set as default), and the export gets its own workload tag since ee.data.setWorkloadTag is called before running it. Use ee.data.resetWorkloadTag to set back to the default tag or to set the default tag back to an empty string.