Join us live on October 8th for the virtual Google Assistant Developer Day. Register now.

Canvas prompts

To relay information about your web app to Google Assistant, you must include a Canvas response in your fulfillment. A Canvas response can do either of the following:

  • Render the full-screen web app on the user's device
  • Pass data to update the web app

The following sections describe how to return a Canvas response for each scenario.

Enable Interactive Canvas

You must configure your Action in a specific way to use Interactive Canvas. Creating an Action that uses Interactive Canvas requires additional configuration in the Actions console (and, for the Actions SDK, modifications to your settings.yaml file). To see the full procedure for creating and configuring an Interactive Canvas Action with the Actions SDK, see Create a project.

When using Actions Builder, follow these additional steps to enable Interactive Canvas:

  1. Click Deploy in the top navigation of the Actions console.
  2. In the Category section of Additional information, select the Games & fun card.
  3. At the bottom of the page, check the option under Interactive Canvas. Click Save.

When using the Actions SDK, follow these additional steps to enable Interactive Canvas:

  1. Set the category field in your settings.yaml file to GAMES_AND_TRIVIA to best describe and help users discover your Action.
  2. Set the usesInteractiveCanvas field in your settings.yaml file to true.

Check surface capability

The Interactive Canvas framework runs only on Assistant devices that provide a visual interface, so your Action needs to check for the INTERACTIVE_CANVAS capability on the user's device. Configure your logic so that, if the Interactive Canvas capability isn't available, you send the user an alternate response type.

Configure your Action's logic to do the following:

  1. Check that the user's device supports the INTERACTIVE_CANVAS capability. If it does, send the user a Canvas response.
  2. If the Interactive Canvas capability is unavailable, check if the user's device supports the capability RICH_RESPONSE. If it does, send the user a rich response instead.
  3. If the rich response capability is unavailable, send the user a simple response.

The following snippet checks for various surface capabilities:

const supportsRichResponse = conv.device.capabilities.includes("RICH_RESPONSE");
const supportsInteractiveCanvas = conv.device.capabilities.includes("INTERACTIVE_CANVAS");
if (supportsInteractiveCanvas) {
  // Respond with a Canvas response
  conv.add(new Canvas({
    url: 'https://example.web.app',
  }));
} else if (supportsRichResponse) {
  // Respond with a rich response
  conv.add(new Card({
    title: 'Card title',
    image: new Image({
      url: 'https://example.com/image.png',
      alt: 'Alt text',
    }),
    button: new Link({
      name: 'Link name',
      open: {
        url: 'https://example.com/',
      },
    }),
  }));
} else {
  // Respond with a simple response
  conv.add('Example simple response.');
}

Render the web app

An Action that uses Interactive Canvas includes a web app with customized visuals that you send to users as a response. Once the web app renders, users continue to interact with it through voice, text, or touch until the conversation is over.

Your first Canvas response must contain the URL of the web app. This type of Canvasresponse tells Google Assistant to render the web app at that address on the user's device. Typically, you send the first Canvas response immediately after the user invokes your Action.

When the web app loads, the Interactive Canvas library loads, and the web app registers a callback handler with the Interactive Canvas API.

The following snippets show how to construct Canvas responses in both Actions Builder and your webhook:

YAML

candidates:
  - first_simple:
       variants:
         - speech: >-
             Welcome! Do you want me to change color or pause spinning? You can
             also tell me to ask you later.
     canvas:
       url: 'https://your-web-app.com'
    

JSON

{
  "candidates": [
    {
      "first_simple": {
        "variants": [
          {
            "speech": "Welcome! Do you want me to change color or pause spinning? You can also tell me to ask you later."
          }
        ]
      },
      "canvas": {
        "url": "https://your-web-app.com"
      }
    }
  ]
}
    

Node.js

app.handle('welcome', (conv) => {
  conv.add('Welcome! Do you want me to change color or pause spinning? ' +
    'You can also tell me to ask you later.');
  conv.add(new Canvas({
    url: `https://your-web-app.com`,
  }));
});
    

JSON

{
  "session": {
    "id": "session_id",
    "params": {}
  },
  "prompt": {
    "override": false,
    "firstSimple": {
      "speech": "Welcome! Do you want me to change color or pause spinning? You can also tell me to ask you later.",
      "text": "Welcome! Do you want me to change color or pause spinning? You can also tell me to ask you later."
    },
    "canvas": {
      "data": [],
      "suppressMic": false,
      "url": "https://your-web-app.com"
    }
  }
}
    

Pass data to update the web app

After you send the initial Canvas response, you can use additional Canvas responses to provide updates to data, which your web app's custom logic uses to make changes to your web app. When you send a Canvas response that passes data to the web app, the following steps occur:

  1. When the intent is matched within a scene, it triggers an event, and the corresponding webhook event handler is called. A Canvas response containing a data field with a JSON payload is then sent back as a response.
  2. The data field is passed to an onUpdate callback and used to update the web app.
  3. Your Conversational Action can send a new Canvas response and provide information in the data field to send new updates or load new states.

The following snippets show how to pass data in a Canvas response in both Actions Builder and your webhook:

YAML

candidates:
  - first_simple:
      variants:
        - speech: 'Ok, I''m spinning. What else?'
    canvas:
      data:
        - command: SPIN
          spin: true
    

JSON

{
  "candidates": [
    {
      "first_simple": {
        "variants": [
          {
            "speech": "Ok, I'm spinning. What else?"
          }
        ]
      },
      "canvas": {
        "data": [
          {
            "command": "SPIN",
            "spin": true
          }
        ]
      }
    }
  ]
}
    

Node.js

app.handle('start_spin', (conv) => {
  conv.add(`Ok, I'm spinning. What else?`);
  conv.add(new Canvas({
    data: {
      command: 'SPIN',
      spin: true,
    },
  }));
});

    

JSON

{
  "session": {
    "id": "session_id",
    "params": {}
  },
  "prompt": {
    "override": false,
    "firstSimple": {
      "speech": "Ok, I'm spinning. What else?",
      "text": "Ok, I'm spinning. What else?"
    },
    "canvas": {
      "data": {
        "command": "SPIN",
        "spin": true
      },
      "suppressMic": false,
      "url": ""
    }
  }
}
    

Sample fulfillment

The following excerpt from the Interactive Canvas sample fulfillment code shows how to implement Canvas:

const {
  conversation,
  Canvas,
  Simple
} = require('@assistant/conversation');
const functions = require('firebase-functions');

const tints = {
  blue: 0x0000FF,
  green: 0x00FF00,
  red: 0xFF0000,
};

const app = conversation();

app.handle('welcome', conv => {
  conv.add(new Simple('Welcome! Do you want me to change color or pause spinning? ' +
    'You can also tell me to ask you later.'));
  conv.add(new Canvas({
    url: 'https://your-web-app.com'
  }));
});

app.handle('changeColor', conv => {
  const color = conv.intent.params.color? conv.intent.params.color.resolved : null;
  if (color in tints) {
    conv.add(new Simple(`Ok, I changed my color to ${color}. What else?`));
    conv.add(new Canvas({
      data: {
        command: 'TINT',
        tint: tints[color]
        }
    }));
    return;
  }
  conv.add(new Simple(`Sorry, I don't know that color. Try red, blue, or green!`));
  conv.add(new Canvas({
    data: {
      query: conv.intent.query
    }
  }));
});

app.handle('startSpin', conv => {
  conv.add(new Simple(`Ok, I'm spinning. What else?`));
  conv.add(new Canvas({
    data: {
      command: 'SPIN',
      spin: true
    }
  }));
});

app.handle('stopSpin', conv => {
  conv.add(new Simple('Ok, I paused spinning. What else?'));
  conv.add(new Canvas({
    data: {
      command: 'SPIN',
      spin: false
    }
  }));
});

app.handle('restart', conv => {
  conv.add(new Simple('Do you want me to change color or pause spinning? You can also tell me to ask you later.'));
  conv.add(new Canvas({
    data: {
      command: 'RESTART_GAME'
    }
  }));
});

exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);

Guidelines and restrictions

Keep the following guidelines and restrictions for Canvas responses in mind when building your fulfillment:

  • Each webhook handler in your fulfillment must include Canvas. If the webhook response does not include Canvas, your web app closes.
  • You only need to include your web app URL in the first Canvas response you send to the user.
  • The Canvas response must be 50 KB or smaller in size.