Build your web app

A web app is the user interface (UI) for an Action that uses Interactive Canvas. You can use existing web technologies (such HTML, CSS, JavaScript, and WebAssembly) to design and develop your web app. For the most part, Interactive Canvas can render web content like a browser, but there are a few restrictions enforced for user privacy and security. Before you begin designing your UI, consider the design principles outlined in Design guidelines. We recommend using Firebase hosting to deploy your web app.

The HTML and JavaScript for your web app do the following:

This page goes over the recommended ways to build your web app, how to enable communication between your Conversational Action and your web app, and general guidelines and restrictions.

Although you can use any method to build your UI, Google recommends using the following libraries:

Architecture

Google strongly recommends using a single-page application architecture. This approach allows for optimal performance and supports continuous conversational user experience. Interactive Canvas can be used in conjunction with front-end frameworks like Vue, Angular, and React, which help with state management.

HTML file

The HTML file defines how your UI looks. This file also loads the Interactive Canvas API, which enables communication between your web app and your Conversational Action.

HTML

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <title>Interactive Canvas Sample</title>

    <!-- Disable favicon requests -->
    <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;,">

    <!-- Load Interactive Canvas JavaScript -->
    <script src="https://www.gstatic.com/assistant/interactivecanvas/api/interactive_canvas.min.js"></script>

    <!-- Load PixiJS for graphics rendering -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/pixi.js/4.8.7/pixi.min.js"></script>

    <!-- Load Stats.js for fps monitoring -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/stats.js/r16/Stats.min.js"></script>

    <!-- Load custom CSS -->
    <link rel="stylesheet" href="css/main.css">
  </head>
  <body>
    <div id="view" class="view">
      <div class="debug">
        <div class="stats"></div>
        <div class="logs"></div>
      </div>
    </div>
    <!-- Load custom JavaScript after elements are on page -->
    <script src="js/log.js"></script>
    <script type="module" src="js/main.js"></script>
  </body>
</html>
    

Communicate between Conversational Action and web app

After you've built your web app and Conversational Action and loaded in the Interactive Canvas library in your web app file, you need to define how your web app and Conversational Action interact. To do this, modify the files that contain your web app logic.

action.js

This file contains the code to define callbacks and invoke methods through interactiveCanvas. Callbacks allow your web app to respond to information or requests from the Conversational Action, while methods provide a way to send information or requests to the Conversational Action.

Add interactiveCanvas.ready(callbacks); to your HTML file to initialize and register callbacks:

JavaScript

/**
* This class is used as a wrapper for Google Assistant Canvas Action class
* along with its callbacks.
*/
export class Action {
 /**
  * @param  {Phaser.Scene} scene which serves as a container of all visual
  * and audio elements.
  */
 constructor(scene) {
   this.canvas = window.interactiveCanvas;
   this.gameScene = scene;
   const that = this;
   this.intents = {
     GUESS: function(params) {
       that.gameScene.guess(params);
     },
     DEFAULT: function() {
       // do nothing, when no command is found
     },
   };
 }

 /**
  * Register all callbacks used by the Interactive Canvas Action
  * executed during game creation time.
  */
 setCallbacks() {
   const that = this;
   // Declare the Interactive Canvas action callbacks.
   const callbacks = {
     onUpdate(data) {
       const intent = data[0].google.intent;
       that.intents[intent ? intent.name.toUpperCase() : 'DEFAULT'](intent.params);
     },
   };
   // Called by the Interactive Canvas web app once web app has loaded to
   // register callbacks.
   this.canvas.ready(callbacks);
 }
}
    

main.js

The main.js JavaScript module imports the files action.js and scene.js and creates instances of each of them when the web app loads. This module also registers callbacks for Interactive Canvas.

JavaScript

import {Action} from './action.js';
import {Scene} from './scene.js';
window.addEventListener('load', () => {
  window.scene = new Scene();
  // Set Google Assistant Canvas Action at scene level
  window.scene.action = new Action(scene);
  // Call setCallbacks to register Interactive Canvas
  window.scene.action.setCallbacks();
});
    

scene.js

The scene.js file constructs the scene for your web app. The following is an excerpt from scene.js:

JavaScript

const view = document.getElementById('view');

// initialize rendering and set correct sizing
this.radio = window.devicePixelRatio;
this.renderer = PIXI.autoDetectRenderer({
  transparent: true,
  antialias: true,
  resolution: this.radio,
  width: view.clientWidth,
  height: view.clientHeight,
});
this.element = this.renderer.view;
this.element.style.width = `${this.renderer.width / this.radio}px`;
this.element.style.height = `${(this.renderer.height / this.radio)}px`;
view.appendChild(this.element);

// center stage and normalize scaling for all resolutions
this.stage = new PIXI.Container();
this.stage.position.set(view.clientWidth / 2, view.clientHeight / 2);
this.stage.scale.set(Math.max(this.renderer.width,
    this.renderer.height) / 1024);

// load a sprite from a svg file
this.sprite = PIXI.Sprite.from('triangle.svg');
this.sprite.anchor.set(0.5);
this.sprite.tint = 0x00FF00; // green
this.sprite.spin = true;
this.stage.addChild(this.sprite);

// toggle spin on touch events of the triangle
this.sprite.interactive = true;
this.sprite.buttonMode = true;
this.sprite.on('pointerdown', () => {
  this.sprite.spin = !this.sprite.spin;
});
    

Support touch interactions

Your Interactive Canvas Action can respond to your user's touch as well as their vocal inputs. Per the Interactive Canvas design guidelines, you should develop your Action to be "voice-first". That being said, some smart displays support touch interactions.

Supporting touch is similar to supporting conversational responses; however, instead of a vocal response from the user, your client-side JavaScript looks for touch interactions and uses those to change elements in the web app.

You can see an example of this in the sample, which uses the Pixi.js library:

JavaScript

…
this.sprite = PIXI.Sprite.from('triangle.svg');
…
this.sprite.interactive = true; // Enables interaction events
this.sprite.buttonMode = true; // Changes `cursor` property to `pointer` for PointerEvent
this.sprite.on('pointerdown', () => {
  this.sprite.spin = !this.sprite.spin;
});
    

Add more features

Now that you've learned the basics, you can enhance and customize your Action with Canvas-specific methods. You can choose to develop your Action with the client fulfillment model or the server-side fulfillment model when you create your Actions project. For more information about these options, see Enable Interactive Canvas.

If you select the client fulfillment model option, you can use the following in your Action:

If you select the server fulfillment model option, you can use the following in your Action:

Some Interactive Canvas APIs are not recommended for use with a particular fulfillment model. The following table shows the APIs enabled when you select the client fulfillment option, and whether these APIs are are recommended or discouraged for each model:

API name Supported in server fulfillment model? Supported in client fulfillment model?
sendTextQuery() Yes Supported but not recommended (see sendtextQuery() for more information)
outputTts() Yes Yes
triggerScene() No Yes
createIntentHandler(), expect(), clearExpectations(), prompt() No Yes
createNumberSlot(),createTextSlot, createConfirmationSlot, createOptionsSlot() No Yes
setHomeParam(), getHomeParam(), setUserParam(), getUserParam() No Yes

The following sections explain how to implement APIs for the client and server-side fulfillment models in your Interactive Canvas Action.

Build with client-side fulfillment

You can implement the following Interactive Canvas APIs in your web app logic:

outputTts()

This API allows you to output text-to-speech (TTS) from a device without sending a static prompt from Actions Builder or invoking a webhook. If no server-side logic associated with the TTS is required, you can use outputTts() from the client-side to skip a trip to your server and provide a faster response to your users.

Client-side outputTts()can interrupt or cancel server-side TTS. You can avoid interrupting server-side TTS by taking these precautions:

  • Avoid calling outputTts() at the beginning of the session; instead, use server-side TTS in your Action’s first conversational turn.
  • Avoid calling outputTts() consecutively without user action in between.

The following snippet shows how to use outputTts() to output TTS from the client-side:

interactiveCanvas.outputTts(
      '<speak>This is an example response.</speak>', true);

You can also use outputTts() with onTtsMark() to place SSML markers into the text sequence. Using onTtsMark() syncs your web app animation or game state at specific points of a SSML TTS string, as shown in the following snippet:

interactiveCanvas.outputTts(
      '<speak>Speak as <mark name="number" /> number <break time="700ms"/>' +
      '<say-as interpret-as="cardinal">12345</say-as> <break time="300ms"/> ' +
      'Speak as <mark name="digits" /> digits <break time="700ms"/>' +
      '<say-as interpret-as="characters">12345</say-as></speak>', true);

In the previous example, the two marks customizing your response are sent to the web app with the TTS.

Handle intent fulfillment on the client

In the server fulfillment model for Interactive Canvas, all intents have to be handled by a webhook, which increases latency in your Action. Instead of calling a webhook, you can handle fulfillment of intents within your web app.

To handle intents client-side, you can use the following APIs:

  • createIntentHandler(): A method that allows you to define intent handlers in your web app code for custom intents defined in Actions Builder.
  • expect(): A method that activates/registers the intent handler so that a user can match the intent.
  • clearExpectations(): A method that clears the expectations for all currently activated intents so that the intents cannot be matched, even if a user says an utterance that matches the intent.
  • deleteHandler(): A method that disables individual intent handlers so those intents cannot be matched.

With these APIs, you can selectively activate or disable intents for different states of your Interactive Canvas Action. You must use expect() on intent handlers to activate those intents.

Activate intent handlers

Activating an intent handler is a two-step process. First, you must define the intent in Actions Builder. Next, to make the intent matchable, you need to call expect() on the intent handler.

To configure and activate an intent handler on the client-side, follow these steps:

  1. Open your project in the Actions console and add a Custom intent.
  2. Select Yes for Is this a global intent?

  3. Configure your intent and click Save.

  4. Define the handler for the intent in your web app logic, as shown in the following snippet:

    /**
    * Define handler for intent.
    */
    const bookTableIntent = interactiveCanvas.createIntentHandler('reserveTable',
      matchedIntent => {
        console.log("Intent match handler to reserve a table was triggered!");
      });
    
    /**
    * Define handler for intent with an argument.
    */
    const bookTableIntent = interactiveCanvas.createIntentHandler('reserveTable',
      matchedIntent => {
        const numberOfPeople = matchedIntent.getIntentArg('numberOfPeople');
        console.log(`Intent match handler to reserve a table for ${number of people} was triggered!`);
      });
    
  5. Call the expect() method to register the intent handler, as shown in the following snippet:

    /**
    * Define handler for intent and expect() it.
    */
    const bookTableIntent = interactiveCanvas.createIntentHandler('reserveTable',
      matchedIntent => {
        console.log("Intent match handler to reserve a table was triggered!");
      });
    var handler = interactiveCanvas.expect(bookTableIntent);
    
Disable intent handlers

After you define an intent handler, you can activate or deactivate the intent as needed for your Action. When you call expect() to activate an intent, it returns an object with a deleteHandler() method, which you can use to disable the newly-created handler. The intent handler definition persists even if the intent is not currently active, so you can re-activate the intent when needed.

To disable an intent handler, call deleteHandler() on the intent handler, as shown in the following snippet:

    /**
    * Define handler for intent and expect() it.
    */
    const bookTableIntent = interactiveCanvas.createIntentHandler('reserveTable',
      matchedIntent => {
        console.log("Intent match handler to reserve a table was triggered!");
      });
    var handler = interactiveCanvas.expect(bookTableIntent);
    
    // Delete the handler for `bookTableIntent`.
    handler.deleteHandler();
    

You can call expect() to re-add a disabled intent handler, as shown in the following snippet:

    // Re-add the `bookTableIntent` handler.
    handler = interactiveCanvas.expect(bookTableIntent);

To disable intents in bulk, you can use the clearExpectations() method, which disables all currently activated intents. The following snippet shows how to clear the expectations for all intent handlers:

interactiveCanvas.clearExpectations();

Handle slot filling on the client

Instead of adding slot filling to a scene within Actions Builder, you can handle slot filling directly in your web app.

To handle slot filling on the client-side, you must first create a slot using one of the following APIs:

  • createNumberSlot(callback, hints): A method that allows you to define a number slot in your web app code. Used to prompt the user for a number.
  • createTextSlot(callback, hints): A method that allows you to define a text slot in your web app code. Used to prompt the user for a word.
  • createConfirmationSlot(callback, hints): A method that allows you to define a confirmation slot in your web app code. Used to prompt the user for confirmation (yes/no).
  • createOptionsSlot(options, callback, hints): A method that allows you to define an options slot in your web app code. Used to prompt the user to select from a list of predefined options.

When you create a slot, you can optionally define triggerHints, which are keywords that improve the natural language understanding (NLU) system for your Action. These keywords should be short words that the user might say when filling a slot. For example, the triggerHints keyword for a number slot could be years. When a user replies to a question about their age in the conversation with the response "I am thirty years old", your Action is more likely to recognize that the user is filling the slot appropriately.

After creating a slot, you can prompt the user for a slot using the prompt API:

  • prompt(tts, slot): A method that will output TTS to the user, prompting them for an expected slot to be filled.

Calling prompt() returns a promise with the status and value of the filled slot.

Create number slot

A number slot allows you to prompt a user for a number during the conversation. For more information about slot filling, see the Slot filling section of the Actions Builder documentation.

To prompt the user to fill a number slot on the client-side, follow these steps:

  1. Call the createNumberSlot() method to create a number slot in your web app logic:

    /**
     * Create number slot.
     */
    const triggerHints = { associatedWords: ['guess number', 'number'] };
    const slot = interactiveCanvas.createNumberSlot(
      number => {
        console.log(`Number guessed: ${number}.`);
      }, triggerHints);
    
    
  2. Call the prompt() method to prompt the user for the slot, and handle the slot value from the returned promise, as shown in the following snippet:

    const promptPromise = interactiveCanvas.prompt(
      { text: 'What number am I thinking of between 1 and 10?' }, slot);
    
    promptPromise.then(
      answer => {
        if (answer.status == interactiveCanvas.AnswerStatus.ANSWERED) {
          // answer === {value: 5, status: ANSWERED}
          // Do something with answer.value
        } else {
          console.error('Promise returned unsuccessful status ' + answer.status);
        }
      });
    
Create text slot

A text slot allows you to prompt a user for a word during the conversation. For more information about slot filling, see the Slot filling section of the Actions Builder documentation.

To prompt the user to fill a text slot on the client-side, follow these steps:

  1. Call the createTextSlot() method to create a text slot in your web app logic:

    /**
     * Create text slot.
     */
    const triggerHints = { associatedWords: ['favorite color', 'color'] };
    const slot = interactiveCanvas.createTextSlot(
      text => {
        console.log(`Favorite color: ${text}.`);
      }, triggerHints);
    
    
  2. Call the prompt() method to prompt the user for the slot, and handle the slot value from the returned promise, as shown in the following snippet:

    const promptPromise = interactiveCanvas.prompt(
      { text: 'What is your favorite color?' }, slot);
    
    promptPromise.then(
      answer => {
        if (answer.status == interactiveCanvas.AnswerStatus.ANSWERED) {
          // answer === {value: "red", status: ANSWERED}
          // Do something with answer.value
        } else {
          console.error('Promise returned unsuccessful status ' + answer.status);
        }
      });
    
Create confirmation slot

A confirmation slot allows you to prompt a user for confirmation (the user can respond "Yes" or "No" to fill the slot). For more information about slot filling, see the Slot filling section of the Actions Builder documentation.

To prompt the user to fill a confirmation slot on the client-side, follow these steps:

  1. Call the createConfirmationSlot() method to create a confirmation slot in your web app logic:

    /**
     * Create confirmation slot (boolean).
     */
    const triggerHints = { associatedWords: ['user confirmation', 'confirmation'] };
    const slot = interactiveCanvas.createConfirmationSlot(
      yesOrNo => {
        console.log(`Confirmation: ${yesOrNo}`);
      }, triggerHints);
    
    
  2. Call the prompt() method to prompt the user for the slot, and handle the slot value from the returned promise, as shown in the following snippet:

    const promptPromise = interactiveCanvas.prompt(
      { text: 'Do you agree to the Terms of Service?' }, slot);
    
    promptPromise.then(
      answer => {
        if (answer.status == interactiveCanvas.AnswerStatus.ANSWERED) {
          // answer === {value: true, status: ANSWERED}
          // Do something with answer.value
        } else {
          console.error('Promise returned unsuccessful status ' + answer.status);
        }
      });
    
Create options slot

An options slot allows you to prompt the user to select from a list of predefined options. For more information about slot filling, see the Slot filling section of the Actions Builder documentation.

To prompt the user to fill an options slot on the client-side, follow these steps:

  1. Call the createOptionsSlot() method to create an options slot in your web app logic:

    /**
     * Create options slot (list selection).
     */
    const triggerHints = { associatedWords: ['select fruit', 'choose fruit'] };
    // Define selectable options
    const options = [{
      key: 'apple',
      synonyms: ['apple', 'large apple', 'gala apple'],
    }, {
      key: 'banana',
      synonyms: ['banana', 'green banana', 'plantain'],
    }];
    const slot = interactiveCanvas.createOptionsSlot(
      options,
      selectedOption => {
        console.log(`You have selected ${selectedOption} as your fruit.`);
      }, triggerHints);
    
    
  2. Call the prompt() method to prompt the user for the slot, and handle the slot value from the returned promise, as shown in the following snippet:

    const promptPromise = interactiveCanvas.prompt(
      { text: 'Would you like a banana or an apple?' }, slot);
    
    promptPromise.then(
      answer => {
        if (answer.status == interactiveCanvas.AnswerStatus.ANSWERED) {
          // answer === {value: 'apple', status: ANSWERED}
          // Do something with answer.value
        } else {
          console.error('Promise returned unsuccessful status ' + answer.status);
        }
      });
    

triggerScene()

The triggerScene() API allows you to transition to another scene in your Interactive Canvas Action from your client-side fulfillment. With triggerScene(), you can also switch from client-side fulfillment to server-side fulfillment when the user needs to access a system scene in Actions Builder that requires a webhook. For example, you can call triggerScene()when a user needs to link their account or receive notifications; then, you can return from that scene to client-side fulfillment with a Canvas prompt.

The following snippet shows how to implement triggerScene() in your Action:

interactiveCanvas.triggerScene('SceneName').then((status) => {
  console.log("sent the request to trigger scene.");
}).catch(e => {
  console.log("Failed to trigger a scene.");
})

Home and user storage on the client

Instead of using a webhook to get and set home and user storage values, you can call client-side APIs to handle home and user storage in your web app. Your web app can then use these stored values across multiple sessions (for example, in prompts and conditions), and can access values for a specific household or user when needed. Using these APIs can reduce latency in your Interactive Canvas Action because you no longer need to call a webhook to get and set storage values.

Home and user storage in the web app follows the same general principles as storage in the webhook. For more information about home and user storage, see the documentation for Home storage and User storage.

Client-side home storage

Home storage allows you to store values for household users based on the home graph, and is shared across all sessions in the household. For example, if a user plays an Interactive Canvas game in a household, the score of the game can be stored in home storage, and other household members can continue playing the game with the stored score.

To enable your Action to support home storage, follow these steps:

  1. In the Actions console, navigate to Deploy > Directory information > Additional Information.
  2. Check the Yes box for Do your Actions use home storage?

To write a value to home storage in your web app, call the setHomeParam() method, as shown in the following snippet:

interactiveCanvas.setHomeParam('familySize',  10).then(
      result => {
        console.log('Set home param success');
      },
      fail => {
        console.error(err);
      });

To read a value from home storage in your web app, call the getHomeParam() method, as shown in the following snippet:

interactiveCanvas.getHomeParam('familySize').then(
      value => {
        console.log(JSON.stringify(result));
      },
      err => {
        console.error(err);
      }
  );

To clear all existing home storage, call the resetHomeParam() method, as shown in the following snippet:

interactiveCanvas.resetHomeParam();
Client-side user storage

User storage allows you to store parameter values for a specific, verified user across multiple sessions. For example, if a user is playing a game, the score of the game can be stored for that user. In a subsequent gameplay session, the user can continue playing the game with the same score.

To write a value to user storage in your web app, call the setUserParam() method, as shown in the following snippet:

interactiveCanvas.setUserParam('color',  'blue').then(
      result => {
        console.log('Set user param success');
      },
      err => {
        console.error(err);
      });

To read a value from user storage in your web app, call the getUserParam() method, as shown in the following snippet:

interactiveCanvas.getUserParam('color').then(
      value => {
        console.log(JSON.stringify(result));
      },
      err => {
        console.error(err);
      }
  );

To clear all existing user storage, call the resetUserParam() method, as shown in the following snippet:

interactiveCanvas.resetUserParam();

setCanvasState()

The setCanvasState() method allows you to send state data from your Interactive Canvas web app to your fulfillment, and notifies Assistant that the web app has updated its state. The web app sends its updated state as a JSON object.

Calling setCanvasState() does not invoke an intent. After invoking setCanvasState(), if sendTextQuery() is invoked or the user query matches an intent in the conversation, the data that was set with setCanvasState() in the previous conversational turn is then available in subsequent turns of conversation.

In the following snippet, the web app uses setCanvasState() to set Canvas state data:

JavaScript

this.action.canvas.setCanvasState({ score: 150 })
    
Reference Canvas state from webhook

You can reference stored Canvas state values in your fulfillment code. To reference the value, use conv.context.canvas.state.KEY syntax, where KEY is the key given when the Canvas state value was set.

For example, if you previously stored a high score value for a game in Canvas state as the parameter score, reference that value using conv.context.canvas.state.score to access that value in fulfillment:

Node.js

app.handle('webhook-name', conv => {
    console.log(conv.context.canvas.state.score);
})
    
Reference Canvas state within prompts

You can reference stored Canvas state values in a prompt. To reference the value, use $canvas.state.KEY syntax, where KEY is the key given when the Canvas state value was set.

For example, if you previously stored a high score value for a game in Canvas state as the parameter score, reference that value using $canvas.state.score to access that value in a prompt:

JSON

{
  "candidates": [{
    "first_simple": {
      "variants": [{
        "speech": "Your high score is $canvas.state.score."
      }]
    }
  }]
}
    
Reference Canvas state within conditions

You can also reference stored Canvas state values in conditions. To reference the value, use the canvas.state.KEY syntax, where KEY is the key given when the Canvas state value was set.

For example, if you previously stored a high score value for a game in Canvas state as the parameter score and want to compare it with the value 999 in a condition, you can reference the stored value in your condition using canvas.state.score. Your condition expression looks like the following:

Condition syntax

canvas.state.score >= 999
    

sendTextQuery()

The sendTextQuery() method sends text queries to the Conversational Action to programmatically match an intent. This sample uses sendTextQuery() to restart the triangle-spinning game when the user clicks a button. When the user clicks the "Restart game" button, sendTextQuery() sends a text query that matches the Restart game intent and returns a promise. This promise results in SUCCESS if the intent is triggered and BLOCKED if it is not. The following snippet matches the intent and handles the success and failure cases of the promise:

JavaScript

…
/**
* Handle game restarts
*/
async handleRestartGame() {
    console.log(`Request in flight`);
    this.button.texture = this.button.textureButtonDisabled;
    this.sprite.spin = false;
    const res = await this.action.canvas.sendTextQuery('Restart game');
    if (res.toUpperCase() !== 'SUCCESS') {
        console.log(`Request in flight: ${res}`);
        return;
    }
    console.log(`Request in flight: ${res}`);
    this.button.texture = this.button.textureButtonDisabled;
    this.sprite.spin = false;
}
…
    

If the promise results in SUCCESS, the Restart game webhook handler sends a Canvas response to your web app:

JavaScript

…
app.handle('restart', conv => {
  conv.add(new Canvas({
    data: {
      command: 'RESTART_GAME'
    }
  }));
});
…
    

This Canvas response triggers the onUpdate() callback, which executes the code in the RESTART_GAME code snippet below:

JavaScript

…
RESTART_GAME: (data) => {
    this.scene.button.texture = this.scene.button.textureButton;
    this.scene.sprite.spin = true;
    this.scene.sprite.tint = 0x00FF00; // green
    this.scene.sprite.rotation = 0;
},
…
    

Build with server-side fulfillment

You can implement the following Interactive Canvas APIs in your webhook:

Enable full-screen mode

By default, Interactive Canvas web apps include a header at the top of the screen with the name of your Action. You can use enableFullScreen to remove the header and replace it with a temporary toast that appears in the loading screen, which allows your user to have a full-screen experience while interacting with your Action. The toast message shows the Action's display name, the developer's name, and instructions for exiting the Action, and the toast color changes depending on what the user selects as a theme on their device.

Figure 1. A toast message on the loading screen for an Action.

If a user interacts with your Action frequently, the toast message temporarily stops appearing on the loading screen. If the user does not engage with your Action for a while, the toast message reappears when they launch the Action.

You can enable full-screen mode in your webhook or in a static prompt in Actions Builder.

To enable full-screen mode in your webhook, follow this step:

  1. Set the enableFullScreen field to true in the first canvas response returned by the webhook in a session. The following snippet is an example implementation using the Node.js client library:

     const { conversation, Canvas } = require('@assistant/conversation');
     const functions = require('firebase-functions');
    
     const app = conversation();
    
     app.handle('invocation_fullscreen', conv => {
       conv.add(new Canvas(
         {
           url: 'https://example-url.com',
           enableFullScreen: true
         }));
     });
    
     exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);
    

To enable full-screen mode in a static prompt in Actions Builder, follow these steps:

  1. Open your project in the Actions console.
  2. Click Develop in the navigation bar and open the prompt that includes the first canvas response.
  3. Set enable_full_screen to true, as shown in the following snippet:

     {
      "candidates": [
        {
          "canvas": {
            "url": "https://example-url.com",
            "enable_full_screen": true
          }
        }
      ]
    }
    

continueTtsDuringTouch

By default, when a user taps the screen while using an Interactive Canvas Action, the TTS stops playing. You can enable the TTS to continue playing when users touch the screen with continueTtsDuringTouch. This behavior cannot be toggled on and off in the same session.

You can implement this behavior in your webhook or in a static prompt in Actions Builder.

To enable TTS to continue after the user taps the screen in your webhook, follow this step:

  • Set the continueTtsDuringTouch field to true in the first canvas response returned by the webhook in a session. The following snippet is an example implementation using the Node.js client library:

    const { conversation, Canvas } = require('@assisant/conversation');
    const functions = require('firebase-functions');
    
    const app = conversation();
    
    app.handle('intent-name', conv => {
      conv.add(new Canvas(
        {
          url: 'https://example-url.com',
          continueTtsDuringTouch: true
        }));
    });
    
    exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);
    

To enable TTS to continue after the user taps the screen in a static prompt in Actions Builder, follow these steps:

  1. Open your project in the Actions console.
  2. Click Develop in the navigation bar and open the prompt the includes the first canvas response.
  3. Set continue_tts_during_touch to true, as shown in the following snippet:

      {
       "candidates": [
         {
           "canvas": {
             "url": "https://example-url.com",
             "continue_tts_during_touch": true
           }
         }
       ]
     }
    

Callbacks

You can implement the following callbacks in your Interactive Canvas Action:

onUpdate()

The onUpdate()callback passes data from your webhook to your web app to update the web app appropriately. You should only use this callback with the server-side fulfillment model of Interactive Canvas development.

For more information about onUpdate(), see Pass data to update the web app.

onTtsMark()

The onTtsMark() callback is called when custom <mark> tags included in the Speech Synthesis Markup Language (SSML) of your response are read out to the user during Text to Speech (TTS). You can use onTtsMark() in both the server-side and client-side fulfillment development models.

In the following snippets, onTtsMark() synchronizes the web app's animation with the corresponding TTS output. When the Action has said to the user, "Sorry, you lost," the web app spells out the correct word and displays the letters to the user.

In the following example, the webhook handler revealWord includes a custom mark in the response to the user when they've lost the game:

JavaScript

…
app.handle('revealWord', conv => {
  conv.add(new Simple(`<speak>Sorry, you lost.<mark name="REVEAL_WORD"/> The word is ${conv.session.params.word}.</speak>`));
  conv.add(new Canvas());
});
…
    

The following code snippet then registers the onTtsMark() callback, checks the name of the mark, and executes the revealCorrectWord() function, which updates the web app:

JavaScript

…
setCallbacks() {
    // declare Assistant Canvas Action callbacks
    const callbacks = {
        onTtsMark(markName) {
            if (markName === 'REVEAL_WORD') {
                // display the correct word to the user
                that.revealCorrectWord();
            }
        },
    }
    callbacks.onUpdate.bind(this);
}
…
    

onInputStatusChanged()

The onInputStatusChanged() callback notifies you when the input status changes in your Interactive Canvas Action. Input status changes indicate when the microphone opens and closes or when Assistant is processing a query. The following events can cause the input status to change:

  • The user speaking to your Action
  • The user inputting text on the Android Google Search App (AGSA)
  • The web app using the sendTextQuery() API to send a text query to the Action
  • The Action writing to home storage and other Assistant events

The primary use case for this callback is synchronizing your Action with the user’s voice interactions. For example, if a user is playing an Interactive Canvas game and opens the microphone, you can pause the game while the user speaks. You can also wait until the microphone is open to send a text query to Assistant to ensure it's received.

This API reports the following statuses:

  • LISTENING - Indicates that the microphone is open.
  • IDLE - Indicates that the microphone is closed.
  • PROCESSING - Indicates that Assistant is currently executing a query, and the microphone is closed.

The API reports the input status to your Action each time the status changes.

While any transition between states is possible, the following flows are common:

  • IDLE>LISTENING>PROCESSING>IDLE - The user says a query, the query is processed, and the microphone closes.
  • IDLE>PROCESSING>IDLE - The web app uses the sendTextQuery() API to send a text query to the Action.
  • IDLE>LISTENING>IDLE - The user opens the microphone but does not say a query.

To use this feature in your Action, add onInputStatusChanged() to your web app code, as shown in the following snippet:

onInputStatusChanged(inputStatus) {
   console.log("The new input status is: ", inputStatus);
}

The onInputStatusChanged() callback passes back a single enum parameter, inputStatus. You can check this value to see the current input status. The inputStatus can be LISTENING, PROCESSING, or IDLE.

Next, add onInputStatusChanged() to the callbacks object to register it, as shown in the following snippet:

 /**
  * Register all callbacks used by the Interactive Canvas Action
  * executed during game creation time.
  */
 setCallbacks() {
   const that = this;
   // Declare the Interactive Canvas action callbacks.
   const callbacks = {
     onUpdate(data) {
       const intent = data[0].google.intent;
       that.intents[intent ? intent.name.toUpperCase() : 'DEFAULT'](intent.params);
     },
     onInputStatusChanged(inputStatus) {
       console.log("The new input status is: ", inputStatus);
     },
   };
   // Called by the Interactive Canvas web app once web app has loaded to
   // register callbacks.
   this.canvas.ready(callbacks);
 }
}

Troubleshooting

While you can use the simulator in the Actions console to test your Interactive Canvas Action during development, you can also see errors that occur within your Interactive Canvas web app on users' devices in production. You can view these errors in your Google Cloud Platform logs.

To see these error messages in your Google Cloud Platform logs, follow these steps:

  1. Open your Actions project in the Actions console.
  2. Click Test in the top navigation.
  3. Click the View logs in Google Cloud Platform link.

Errors from your users' devices appear in chronological order in the logs viewer.

Error types

There are three types of web app errors you can see in the Google Cloud Platform logs:

  • Timeouts that occur when ready is not called within 10 seconds
  • Timeouts that occur when the promise returned by onUpdate() is not fulfilled within 10 seconds
  • JavaScript runtime errors that are not caught within your web app

View JavaScript error details

The details of JavaScript runtime errors within your web app aren't available by default. To see the details of JavaScript runtime errors, follow these steps:

  1. Ensure that you've configured the appropriate cross-origin resource sharing (CORS) HTTP response headers in your web app files. For more information, see Cross-origin resource sharing.
  2. Add crossorigin="anonymous" to your imported <script> tags in your HTML file, as shown in the following code snippet:
<script crossorigin="anonymous" src="<SRC>"></script>

Guidelines and restrictions

Take the following guidelines and restrictions into consideration as you develop your web app:

  • No cookies
  • No local storage
  • No geolocation
  • No camera usage
  • No audio or video recording
  • No popups
  • Stay under the 200 MB memory limit
  • Take the Action name header into account when rendering content (occupies upper portion of screen)
  • No styles can be applied to videos
  • Only one media element may be used at a time
  • No Web SQL database
  • No support for the SpeechRecognition interface of the Web Speech API.
  • Dark mode setting not applicable
  • Video playback is supported on smart displays. For more information on the supported media container formats and codecs, see Google Nest Hub codecs.

Cross-origin resource sharing

Because Interactive Canvas web apps are hosted in an iframe and the origin is set to null, you must enable cross-origin resource sharing (CORS) for your web servers and storage resources. This process allows your assets to accept requests from null origins.