Integrate App Actions with Android widgets

Figure 1. Launching a widget for GET_EXERCISE_OBSERVATION.

For many intents, the best response is to deliver a simple answer, brief confirmation, or quick interactive experience to the user. You can display an Android app widget in Google Assistant to fulfill these kinds of intents.

This guide covers fulfilling Assistant user queries using widgets, and how to enhance your widget experience for Assistant with the App Actions Widgets Extension library.

Benefits

Widgets are miniature application views that can be embedded on Android surfaces, such as the launcher or lock screen. With App Actions, you increase the impact of your widgets by making them eligible for display in Assistant:

  1. Discovery: Proactively display widgets in response to users' natural language queries.
  2. Engagement: Display widgets in hands-free contexts, such as when Assistant provides personal results on the lock screen, and on Android Auto.
  3. Retention: Enable users to pin the widgets displayed in Assistant to their launcher. Pinning functionality requires the Widgets Extension library.

How Assistant displays widgets

There are two ways users can invoke widgets on Assistant:

  • Explicitly requesting a widget by name, or
  • Speaking a query to Assistant that triggers a built-in intent (BII) or custom intent configured for widget fulfillment.

Explicit invocation

To explicitly invoke widgets for any installed app, users can ask Assistant things like:

  • "Hey Google, show ExampleApp widget."
  • "Widgets from ExampleApp."

Assistant displays these widgets with the generic introduction: "ExampleApp says, here's a widget". While Assistant natively returns widgets requested in this manner with no work required by the app developer, this invocation method requires the user to have explicit knowledge of the widget to request. To simplify widget discovery, use the intent fulfillment method detailed in the following section.

Intent fulfillment

Make your widgets easier to find by using them to fulfill the natural language queries users perform on Assistant. For example, you could return a widget whenever a user triggers the GET_EXERCISE_OBSERVATION BII in your fitness app by asking, "Hey Google, how many miles have I run this week on ExampleApp?" In addition to simplifying discovery, integrating widgets with App Actions offers these advantages:

  • Parameter access: Assistant provides the intent parameters extracted from the user query to your widget, enabling tailored responses.
  • Custom TTS introductions: You can provide a text-to-speech (TTS) string for Assistant to announce when displaying your widget.
  • Widget pinning: Assistant displays an Add this widget button near your widget, allowing users to easily pin your widgets to their launcher.

Custom TTS introductions and widget pinning require adding the Widgets Extension library to your project.

Implement Widgets

To implement widget fulfillment for your intents, follow these steps:

  1. Implement an Android widget by following the steps described in Create a simple widget.
  2. In your app's shortcuts.xml resource file, add an <app-widget> element to your capability containing fulfillment details and BII <parameter> tags. Update your widget to handle the parameters.
  3. (Recommended) Add the optional Widgets Extension library to your Android project to enable custom TTS introductions and widget pinning.

The following section describes the <app-widget> schema for shortcuts.xml.

Widget schema

<app-widget> elements are defined as fulfillments within <capability> elements in shortcuts.xml. They require the following attributes, unless noted as optional:

Shortcuts.xml tagContained inAttributes
<app-widget> <capability>
  • android:identifier
  • android:targetClass
<parameter> <app-widget>
<extra> <app-widget>
  • android:name (only applicable for TTS)
  • android:value (optional)

Widget schema description

<app-widget>

Top level widget fulfillment element.

Attributes:

  • android:identifier: Unique identifier for this fulfillment. This value should be unique across the <app-widget> and <intent> fulfillment elements defined within a <capability>.
  • android:targetClass: The full class name of your AppWidgetProvider to handle the intent.

<parameter>

Maps a BII parameter to an intent <parameter> value. You can define zero or more parameters for each <app-widget> element. During fulfillment, Assistant passes parameters by updating the extras for the widget instance as key/value pairs with the following format:

  • Key: the android:key defined for the parameter.
  • Value: The value the BII extracts from a user’s voice input.

You access these extras by calling getAppWidgetOptions() on the associated AppWidgetManager object which returns a Bundle containing the name of the triggering BII and its parameters. See Extract parameter values for details.

For more information on BII parameter matching, see Parameter data and matching.

<extra>

Optional tag declaring that a custom TTS introduction should be used for this widget. This tag requires the following attribute values:

  • android:name: "hasTts"
  • android:value: "true"

Sample code

The following XML demonstrates a widget fulfillment configuration for a GET_EXERCISE_OBSERVATION BII capability:

shortcuts.xml

<capability android:name="actions.intent.GET_EXERCISE_OBSERVATION">
  <app-widget
    android:identifier="GET_EXERCISE_OBSERVATION_1"
    android:targetClass="com.exampleapp.providers.exampleAppWidgetProvider"
    android:targetPackage="com.exampleapp">
    <parameter
      android:name="exerciseObservation.aboutExercise.name"
      android:key="exercisename">
    </parameter>
    <extra android:name="hasTts" android:value="true"/>
  </app-widget>
</capability>

You can specify multiple <app-widget> elements or use a combination of <app-widget> and <intent> elements per capability. This approach lets you provide a customized experience based on different combinations of parameters provided by users. For example, if the user does not specify a drop-off location in their query, you can direct them to the activity in your app that shows options for setting the pick-up and drop-off locations. See the Fallback intents section for more information about defining fallback intents.

Extract parameter values

In the following sample AppWidgetProvider class, the private function updateAppWidget() is used to extract the BII name and parameters from the widget options Bundle:

package com.example.exampleapp;

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

/**
 * Implementation of App Widget functionality.
 */
public class MyAppWidget extends AppWidgetProvider {

    @Override
    public void onUpdate(Context context, AppWidgetManager appWidgetManager, int[] appWidgetIds) {
        // There may be multiple widgets active, so update all of them
        for (int appWidgetId : appWidgetIds) {
            updateAppWidget(context, appWidgetManager, appWidgetId);
        }
    }

    private static void updateAppWidget(Context context, AppWidgetManager appWidgetManager, int appWidgetId) {

        CharSequence widgetText = context.getString(R.string.appwidget_text);

        // Construct the RemoteViews object
        RemoteViews views = new RemoteViews(context.getPackageName(), R.layout.my_app_widget);
        views.setTextViewText(R.id.appwidget_text, widgetText);

        // Extract the name and parameters of the BII from the widget options.
        Bundle optionsBundle = appWidgetManager.getAppWidgetOptions(appWidgetId);
        String bii =
                optionsBundle.getString(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_BII); // "actions.intent.CREATE_TAXI_RESERVATION"
        Bundle params =
                optionsBundle.getBundle(AppActionsWidgetExtension.EXTRA_APP_ACTIONS_PARAMS);

        if (params != null && params.containsKey(("dropoff"))){
            String dropoffLocation = params.getString("dropoff");
            // Build your RemoteViews with the extracted BII parameter
            // ...
        }

        appWidgetManager.updateAppWidget(appWidgetId, views);
    }
}

Widgets Extension library

The App Actions Widgets Extension library enhances your widgets for voice-forward Assistant experiences. This Maven library allows you to provide a custom text-to-speech (TTS) introduction for each widget, enabling Assistant to announce a summary of the content being visually rendered to users. It also enables launcher pinning, making it easy for users to save the widgets displayed in Assistant to their launcher screens.

Get started by adding the library to the dependencies section of the build.gradle file for your app module:

app/build.gradle

dependencies {
  //...
  implementation "com.google.assistant.appactions:widgets:0.0.1"
}

Custom introductions

After importing the Widgets Extension library, you can provide custom TTS introductions for your widgets. To add your definition to the widget's AppWidgetProvider, open the class in your IDE and import the Widgets Extension library:

import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

Next, use the library to define your introduction strings and update the widget:

ExampleAppWidget

package com.example.exampleapp;

//... Other module imports
import com.google.assistant.appactions.widgets.AppActionsWidgetExtension;

/**
 * Implementation of App Widget functionality.
 */
public class MyAppWidget extends AppWidgetProvider {

  static void updateAppWidget(Context context, AppWidgetManager appWidgetManager,
    int appWidgetId) {

    AppActionsWidgetExtension appActionsWidgetExtension = AppActionsWidgetExtension.newBuilder(appWidgetManager)
      .setResponseSpeech("Hello world")  // TTS to be played back to the user.
      .setResponseText("Hello world!")  // Response text to be displayed in Assistant.
      .build();

      // Update widget with TTS.
      appActionsWidgetExtension.updateWidget(appWidgetId);

      // Update widget UI.
      appWidgetManager.updateAppWidget(appWidgetId, views);
    }
  //...
}

Launcher pinning

This library enables the Add this widget button to be displayed with your widget in Assistant. Pinning requires adding the following receiver definition to AndroidManifest.xml:

AndroidManifest.xml

<application>
  <receiver android:name="com.google.assistant.appactions.widgets.pinappwidget.PinAppWidgetBroadcastReceiver"
    android:exported="false">
    <intent-filter>
      <action android:name="com.google.assistant.appactions.widgets.COMPLETE_PIN_APP_WIDGET" />
    </intent-filter>
  </receiver>
  <service
    android:name=
    "com.google.assistant.appactions.widgets.pinappwidget.PinAppWidgetService"
    android:enabled="true"
    android:exported="true">
    <intent-filter>
      <action
        android:name="com.google.assistant.appactions.widgets.PIN_APP_WIDGET" />
    </intent-filter>
  </service>
</application>

Inventory availability

BIIs supporting inline inventory or web inventory can extend these inventories to your widget fulfillments.

Inline inventory

The following code sample demonstrates a START_EXERCISE BII capability configured for inline inventory and widget fulfillment:

shortcuts.xml

<capability
  android:name="actions.intent.START_EXERCISE">
  <app-widget
    android:identifier="START_EXERCISE_1"
    android:targetClass="com.example.exampleapp.StartExerciseAppWidgetProvider">
    <parameter
      android:name="exercise.name"
      android:key="exerciseName"
      android:shortcutMatchRequired="true">
    </parameter>
  </app-widget>
</capability>

<shortcut android:shortcutId="RunningShortcut">
  <intent
    android:action="android.intent.action.VIEW"
    android:targetClass="com.example.exampleapp.StartExcerciseActivity" />
  <capability-binding
    android:capability="actions.intent.START_EXERCISE"
    android:parameter="exercise.name"
    android:value="running;runs" />
</shortcut>

In the preceding sample, when a user triggers this capability by asking Assistant, “Start running with ExampleApp ”, the option bundle for the <app-widget> fulfillment will contain the following key-value pair:

  • Key = “exerciseName”
  • Value = “RunningShortcut”

Web inventory

See the following sample code for a capability enabled for web inventory and widget fulfillment:

shortcuts.xml

<shortcuts>
  <capability
    android:name="actions.intent.START_EXERCISE">
    <app-widget
      android:identifier="START_EXERCISE_1"
      android:targetClass="com.example.exampleapp.CreateTaxiAppWidgetProvider">
      <parameter
        android:name="exercise.name"
        android:key="exerciseName"
        android:mimeType="text/*">
        <data android:pathPattern="https://exampleapp.com/exercise/.*" />
      </parameter>
    </app-widget>
  </capability>
</shortcuts>

Testing

Use the App Actions Test Tool, a feature of the Google Assistant plugin for Android Studio, to test widgets on a physical or virtual device. To use the test tool, follow these steps:

  1. Connect your test device with your app running.
  2. In Android Studio, go to Tools > App Actions > App Actions Test Tool.
  3. Click Create preview.
  4. In Android Studio, run your app on your test device.
  5. Use the Assistant app on your test device to test your App Action. For example, you can say something like "Hey Google, how many miles have I run this week on ExampleApp?".
  6. Observe the behavior of your app, or use the Android Studio debugger, to verify the desired action result.

Quality guidelines

This section highlights key requirements and best practices when you integrate App Actions with widgets.

Content in widgets

  • (Required) Do not show ads in your widgets.
  • Widget content should be completely focused on fulfilling the intent — don't try to fulfill multiple intents with one widget, or add irrelevant content.

Handling authentication

  • (Required) Where user authentication is needed to complete a user flow, return a widget that explains that the user needs to continue in the app. In-line user authentication in Google Assistant is not supported for App Actions.
  • If users are permitting your app to show data via widgets, you can return an error widget at runtime for unauthorized users.

Fallback intents

  • (Required) In your shortcuts.xml, always provide a fallback <intent> in addition to your widget fulfillment for a given capability. A fallback intent is an <intent> element with no required <parameter> values. This enables Assistant to fulfill an Action when the user query does not contain parameters required by the other fulfillment elements defined in the capability. The exception to this is when there are no required parameters for that capability, in which case only the widget fulfillment is needed.
  • The fallback intent should open your app to the relevant screen in your app, not the home screen.

The following sample code demonstrates a <capability> with a fallback <intent> supporting a primary <app-widget> fulfillment:

shortcuts.xml

<shortcuts>
  <capability
    android:name="actions.intent.CREATE_TAXI_RESERVATION">
    <!-- Widget is primary fulfillment, determined by declaration order. -->
    <app-widget
      android:identifier="CREATE_TAXI_RESERVATION_1"
      android:targetClass="com.example.myapplication.CreateTaxiAppWidgetProvider">
      <parameter
        android:name="taxiReservation.dropoffLocation.name"
        android:key="dropoff">
      </parameter>
    </app-widget>
    <!-- Fallback intent with no parameters needed to successfully execute. -->
    <intent
      android:identifier="CREATE_TAXI_RESERVATION_3"
      android:action="myapplication.intent.CREATE_TAXI_RESERVATION_1"
      android:targetClass="com.example.myapplication.TaxiReservationActivity">
    </intent>
  </capability>
</shortcuts>