Logging

Fleet Engine offers a simple logging service that lets you save its API requests and response payloads. With these logs, you can debug your integration, create monitoring metrics, and analyze traffic patterns.

Fleet Engine sends the logs as platform logs to Cloud Logging, so that you can use the Cloud Logging tools to access them.

What Fleet Engine logs

Fleet Engine sends multiple pieces of information to Cloud Logging, such as:

  • All authenticated REST and gRPC requests and responses
  • Error responses
  • Requests, responses, and error messages from calls initiated by the Driver SDK to the Fleet Engine.

See documentation for available log messages and schema in the Logging Reference.

Use restricted and default log buckets to comply with data retention policies

Using "restricted" and "default" buckets ensures a clear picture of your restricted and unrestricted data usage. Some of the log data that Fleet Engine sends to Google Maps Platform may only be retained for a limited period of time, according to the Mobility Service Specific Terms. To make sure you retain restricted data for only the amount of time allowed, Google recommends that you label restricted data as TOS_RESTRICTED and send it to a dedicated bucket called "restricted". From there, you can control how long it's retained and purge it automatically on expiration using Cloud Logging settings. For example, restricted-use logs should only be retained for 30 days.

Send all remaining unrestricted log data to the "default" bucket, where it can remain indefinitely.

Get a complete view by merging restricted and unrestricted logs

Restricted-use logs contain the restricted-use data and a reference to the default log so that they can be considered together. The restricted-use log contains the default log's insertId as a reference in the parent_insert_id field. You can use this field to join the data from both logs and get the complete picture.

See documentation for all available log messages and schema in the Logging Reference.

Enable Cloud Logging

Fleet Engine automatically enables default logs for new Mobility customers, starting with projects created on February 10, 2022. You can confirm whether logging is enabled by using the following query in the Logs Explorer :

resource.type:"fleetengine.googleapis.com"

If you don't see any logs for that query, Cloud Logging might not have been enabled for your project. Contact Support if you want to enable the feature.

Enable restricted-use logs

Restricted-use logs are enabled on request. To enable these logs for your GCP project, complete the following steps:

Prepare your project to receive restricted-use logs

  1. In the Google Cloud console, open the Log Router page.
  2. Update the _Default logging bucket to exclude restricted-use logs.
    1. Select the _Default logging bucket and then choose Edit sink.
    2. On the "Choose logs to filter out of sink" section, click the "Add Exclusion" button:
    3. Exclusion filter name: ExcludeRestrictedLogs
    4. Exclusion filter: labels.restriction="TOS_RESTRICTED"
    5. Click "Update sink".
  3. Update the Restricted logging bucket to store the restricted-use logs.
    1. From the Log Router page, select "Create sink".
    2. Create a sink with the following settings:
    3. Sink details:
      1. Name: RestrictedLogs
      2. Description: Routes Fleet Engine restricted-use logs
    4. Sink destination:
      1. Sink service: Logging bucket
      2. Select log bucket: Create new log bucket
      3. Name: Restricted
      4. Description: Contains Fleet Engine restricted-use logs
      5. Retention period: 30 days
      6. Note: The retention period must not exceed 30 days.
    5. Logs to include in sink: leave empty
    6. Logs to filter out of sink: Click "Add Exclusion"
      1. Exclusion filter name: ExcludeNonRestrictedLogs
      2. Exclusion filter: NOT (resource.type = "fleetengine.googleapis.com/Fleet" OR resource.type = "fleetengine.googleapis.com/DeliveryFleet") NOT (labels.restriction = "TOS_RESTRICTED")
    7. Click "Create sink"

Contact support to enable restricted-use logs

Then, contact support and provide the following information in your support request:

  1. Project ID(s) to enable:
  2. Email address of the person requesting the change:
    1. Note: This person should have Edit access to the GCP Projects listed above.
  3. By enabling restricted-use Google Maps Content in Cloud Logging, you agree to adhere to the Google Maps Platform terms and Mobility service specific terms, including caching and permitted use requirements as they relate to Google Maps Content. Yes / No

Once the support team receives your request, it sends confirmation that logging has been enabled for your project

Access your logs

Cloud Logs are structured around the LogEntry format. Fleet Engine sends logs to Cloud Logging with the LogEntry's resource.type set to fleetengine.googleapis.com. You can use the Logs Explorer to write queries for viewing your logs.

For example, to view all RPCs to Fleet Engine that returned an error, use the following Logs Explorer query:

resource.type:"fleetengine.googleapis.com"
severity=ERROR

To view logs of RPCs made to the UpdateVehicle method for the project example-project-id, use the following Logs Explorer query:

logName="projects/project-id/logs/fleetengine.googleapis.com%2Fupdate_vehicle"

The following example shows a LogEntry for the UpdateVehicle log. The RPC request and response are located inside the jsonPayload field:

    {
      "insertId": "c6b85fbc927343fc8a85338c57a65733",
      "jsonPayload": {
        "request": {
          "header": {4},
          "updateMask": "deviceSettings",
          "vehicleId": "uniqueVehicleId",
          "vehicle": {2}
        },
        "response": {
          "name": "providers/example-project-id/vehicles/uniqueVehicleId",
          "availableCapacity": 2,
          "state": "VEHICLE_STATE_OFFLINE",
          "maximumCapacity": 2,
          "vehicleType": {1},
          "supportedTrips": {1}
        },
        "@type": "type.googleapis.com/maps.fleetengine.v1.UpdateVehicleLog"
      },
      "resource": {
        "type": "fleetengine.googleapis.com/Fleet",
        "labels": {2}
      },
      "timestamp": "2021-01-01T00:00:00.000000000Z",
      "labels": {2},
      "logName": "projects/example-project-id/logs/fleetengine.googleapis.com%2Fupdate_vehicle",
      "receiveTimestamp": "2021-01-01T00:00:00.000000000Z"
    }

If an RPC error is returned, the responseVehicle field is cleared and the errorResponse field is set and populated within jsonPayload:

    {
      "insertId": "c6b85fbc927343fc8a85338c57a65733",
      "jsonPayload": {
        "errorResponse": {
          "httpStatusCode": 404,
          "code": "NOT_FOUND",
          "message": "No entity with id invalidVehicleId exists"
        },
        "@type": "type.googleapis.com/maps.fleetengine.v1.UpdateVehicleLog",
        "request": {
          "vehicle": {3},
          "updateMask": "deviceSettings",
          "vehicleId": "fakeVehicleId",
          "header": {4}
        }
      },
      "resource": {
        "type": "fleetengine.googleapis.com/Fleet",
        "labels": {2}
      },
      "timestamp": "2021-01-01T00:00:00.000000000Z",
      "severity": "ERROR",
      "labels": {2}
      "logName": "projects/example-project-id/logs/fleetengine.googleapis.com%2Fupdate_vehicle",
      "receiveTimestamp": "2021-01-01T00:00:00.000000000Z"
    }

For more information about the logging query language, see Logging query language . For information about how you can use your logs to create metrics, see Overview of logs-based metrics.

Manage logging costs

After logging is enabled, you are responsible for setting up how you would like to route, store, and retain your logs. You may incur extra Google Cloud Platform charges for log ingestion and retention if you exceed the usage and retention limits for no charge. However, you can control logging costs by doing one of the following:

Reduce logging usage

You can limit the amount of log data ingestion by excluding certain log entries.

Export or route logs

You can route logs to other GCP or external destinations to avoid the default ingestion and storage costs. Make sure you turn off log ingestion, as described below, to avoid ingestion costs.

Turn off log ingestion to avoid charges

Reducing logging usage, or exporting or routing logs, are preferred over turning off log ingestion. However, if you don't intend to use Fleet Engine logs, you can avoid potential Cloud Logging charges by turning off ingestion. By default, Fleet Engine logs are routed to the _Default log bucket.

The following command updates the _Default logging bucket to not ingest Fleet Engine logs.

gcloud logging sinks update _Default \
--log-filter='NOT LOG_ID("cloudaudit.googleapis.com/activity") \
AND NOT LOG_ID("externalaudit.googleapis.com/activity") \
AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") \
AND NOT LOG_ID("externalaudit.googleapis.com/system_event") \
AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") \
AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency") \
AND NOT resource.type:"fleetengine.googleapis.com"''

For more information, see: Cloud Logging Exclusions and Excluding logs. Cloud Logging Exports and Exporting logs

Use the Logs Explorer

To use the Logs Explorer, open the Cloud Console, select Logging, and then Logs Explorer. To see a list of all the Fleet Engine logs available, click the Fleet Engine Resource Type. Some API logs are labeled with a Trip ID and a Vehicle ID. You can use these labels to select logs for the trips or vehicles that interest you.

Log labels

Filter logs by vehicle ID

In the Logs Explorer, you can use the following query to restrict logs to a specific vehicle: none resource.type="fleetengine.googleapis.com/Fleet" labels.vehicle_id="vehicle_id" Filter vehicle

Filter logs by trip ID

In the Logs Explorer, you can use the following query to restrict logs to a specific trip: none resource.type="fleetengine.googleapis.com/Fleet" labels.trip_id=~"trip_id" Note: To make sure that your query filter captures all logs, use the =~ operator in your query. If there are multiple trips assigned to a vehicle, the trips IDs are separated by commas. This filter matches any log that includes the value in the label, rather than a strict equality check.

Filter logs for a vehicle over a specific time period

In the Logs Explorer, you can use the following query to restrict logs to those for a vehicle over a specific time period: none resource.type="fleetengine.googleapis.com/Fleet" labels.vehicle_id="vehicle_id" timestamp>="2020-09-24T20:00:00.000Z" timestamp<"2020-09-24T21:00:00.000Z" Note: Time values in this example are in Coordinated Universal Time (UTC) format. To specify a time zone, replace the letter Z at the end of the timestamp with +hh:mm or -hh:mm. For more information, see the Cloud documentation.

Log-based metrics example

The following example shows how to use log-based metrics to track the number of trips created over time.

  1. In the Cloud Console, select Logging and then Logs Explorer to open the Logs Explorer. Then apply the following filter:

    resource.type="audited_resource"
    resource.labels.method="maps.fleetengine.v1.TripService.ReportBillableTrip"
    
  2. In the Query Results pane, select the Actions drop-down and then select Create Metric.

    Create metric

  3. In the Metric Editor dialog:

    • Specify a metric name (for example, billable_trips).
    • Specify a metric description (for example, The number of Billable Trip calls).
    • Leave the Units option blank. _ Leave the Type option as Counter.

    Then select the Create Metric button.

  4. On the Logs-Based Metrics page, you should see a message confirming that the metric was created successfully, and the new metric should appear in the User-define metrics section. The metric will now be populated as matching logs are generated.

  5. Select the vertical drop down on the right side of the new metric and then select View in Metrics Explorer.

    View metric

  6. On the left pane under Build Your Query, set the Resource Type to Fleet Engine and search for the billable_trips metric.

    Search metric

    The graph on the right shows the rate of billable_trips calls.

Use BigQuery

BigQuery is a powerful tool for performing analytics. It can be used to store longer-term logs and to perform ad hoc SQL-like queries against the data.

Route logs to BigQuery

To take advantage of BigQuery, logs must be routed to a BigQuery data store, as follows:

  1. In the Cloud Console, select Logging and then Logs Explorer.

  2. Create a filter that isolates Fleet Engine logs. In the Logs Field Explorer, select the Fleetengine.googleapis.com/Fleet resource type.

    Create filter

  3. In the Query Results pane, click the Actions drop-down and choose Create Sink.

    Create sink

  4. In the Select sink service dialog, select BigQuery dataset.

    Select sink

  5. In the Edit Sink dialog, specify the following options:

    • Specify an sink name (for example, FleetEngineLogsSink).
    • Leave Sink Service as BigQuery.
    • Select the Use Partitioned Tables option. This will boost query performance.
    • Under Sink Destination, select Create New BigQuery Data Set, and then specify a BigQuery data set name (for example, FleetEngineLogs).
    • Click the Create Sink button.

    Edit sink

Your logs should now begin to populate the BigQuery data set. You can see the data in the BigQuery section of the Cloud Console.

BigQuery section

Several tables under the FleetEngineLogs data set will be populated automatically, one for each log type:

  • CreateVehicle
  • GetVehicle
  • ListVehicles
  • SearchVehicles
  • UpdateVehicle
  • CreateTrip
  • GetTrip
  • UpdateTrip
  • ListTrips

The table names use the following pattern:

project_id.data_set.log_name

For example, if the project is called test_project and the dataset name is FleetEngineLogs, the CreateTrip table has the following name:

test_project.FleetEngineLogs.fleetengine_googleapis_com_create_trip

Example queries

This section shows examples of queries you can create.

Trips created per hour

The following query counts the number of CreateTrips logs and groups them by hour. sql SELECT TIMESTAMP_TRUNC(timestamp, HOUR) as hour, count(*) as num_trips_created FROM `ProjectId.FleetEngineLogs.fleetengine_googleapis_com_create_trip` GROUP BY hour ORDER by hour

Number of stops per vehicle per hour

The following query generates a count of the stops that a vehicle served, broken down by hour.

For example, this query could tell us that in the last hour:

  • Vehicle A completed 10 stops in hour 12 and 8 stops in hour 13.
  • Vehicle B completed 5 stops in hour 11 and 7 stops in hour 12.
  • Vehicle C completed 12 stops in hour 13 and 9 stops in hour 14.

    SELECT
      jsonpayload_v1_updatevehiclelog.request.vehicleid AS vehicle,
      TIMESTAMP_TRUNC(timestamp, HOUR) AS hour,
      COUNT(*) AS num_stops
    FROM
      `ProjectId.FleetEngineLogs.fleetengine_googleapis_com_update__vehicle`
    WHERE
    ARRAY_LENGTH(jsonpayload_v1_updatevehiclelog.request.vehicle.remainingvehiclejourneysegments) > 0
      AND jsonpayload_v1_updatevehiclelog.request.vehicle.remainingvehiclejourneysegments[
    OFFSET
      (0)].stop.state = 'VEHICLE_STOP_STATE_LOG_ARRIVED'
    GROUP BY
      1,
      2
    ORDER BY
      2
    

Data Studio dashboards

BigQuery can be integrated with business intelligence tools and dashboards can be created for business analytics.

The following example shows how to build a dashboard on which ttrips and vehicle movements can be visualized on a map.

  1. Launch a new Datastudio dashboard and select BigQuery as the data connection.

    Data connection

  2. Select Custom Query and select the Cloud Project to which it should be billed.

    Select project

  3. Enter the following query into the query box.

    Enter query

    SELECT
     timestamp,
     labels.vehicle_id,
    jsonpayload_v1_updatevehiclelog.response.lastlocation.location.latitude AS lat,
    jsonpayload_v1_updatevehiclelog.response.lastlocation.location.longitude AS lng
    FROM
    `ProjectId.FleetEngineLogs.fleetengine_googleapis_com_update_vehicle`
    
  4. Select Chart Type as Bubble Map, and then select the location field.

    Chart type

  5. Select Create Field.

    Create field

  6. Name the field and add the following formula: CONCAT(lat, ",", lng).

    Then set type to Geo->Latitude, Longitude.

    Set type

  7. You can add controls to the dashboard to filter data. For example, select the Date-range filter.

    Add controls

  8. Edit the date range box to select a default date range.

    Date range

  9. You can add additional Drop-down list controls for vehicle_id.

    Drop-down list

With these controls, you can visualize the movement of the vehicle or the movement within a trip.