Conversational Actions were deprecated on June 13, 2023. For more information, see Conversational Actions sunset.

Package google.assistant.embedded.v1alpha2

Index

EmbeddedAssistant (interface)
AssistConfig (message)
AssistRequest (message)
AssistResponse (message)
AssistResponse.EventType (enum)
AudioInConfig (message)
AudioInConfig.Encoding (enum)
AudioOut (message)
AudioOutConfig (message)
AudioOutConfig.Encoding (enum)
DebugConfig (message)
DebugInfo (message)
DeviceAction (message)
DeviceConfig (message)
DeviceLocation (message)
DialogStateIn (message)
DialogStateOut (message)
DialogStateOut.MicrophoneMode (enum)
ScreenOut (message)
ScreenOut.Format (enum)
ScreenOutConfig (message)
ScreenOutConfig.ScreenMode (enum)
SpeechRecognitionResult (message)

EmbeddedAssistant

Service that implements the Google Assistant API.

Assist

Assist
`rpc Assist(AssistRequest) returns (AssistResponse)` Initiates or continues a conversation with the embedded Assistant Service. Each call performs one round-trip, sending an audio request to the service and receiving the audio response. Uses bidirectional streaming to receive results, such as the `END_OF_UTTERANCE` event, while sending audio. A conversation is one or more gRPC connections, each consisting of several streamed requests and responses. For example, the user says Add to my shopping list and the Assistant responds What do you want to add?. The sequence of streamed requests and responses in the first gRPC message could be: AssistRequest.config AssistRequest.audio_in AssistRequest.audio_in AssistRequest.audio_in AssistRequest.audio_in AssistResponse.event_type.END_OF_UTTERANCE AssistResponse.speech_results.transcript "add to my shopping list" AssistResponse.dialog_state_out.microphone_mode.DIALOG_FOLLOW_ON AssistResponse.audio_out AssistResponse.audio_out AssistResponse.audio_out The user then says bagels and the Assistant responds OK, I've added bagels to your shopping list. This is sent as another gRPC connection call to the `Assist` method, again with streamed requests and responses, such as: AssistRequest.config AssistRequest.audio_in AssistRequest.audio_in AssistRequest.audio_in AssistResponse.event_type.END_OF_UTTERANCE AssistResponse.dialog_state_out.microphone_mode.CLOSE_MICROPHONE AssistResponse.audio_out AssistResponse.audio_out AssistResponse.audio_out AssistResponse.audio_out Although the precise order of responses is not guaranteed, sequential `AssistResponse.audio_out` messages will always contain sequential portions of audio. Authorization Scopes Requires the following OAuth scope: `https://www.googleapis.com/auth/assistant-sdk-prototype` For more information, see the Auth Guide.

rpc Assist(AssistRequest) returns (AssistResponse)

Initiates or continues a conversation with the embedded Assistant Service. Each call performs one round-trip, sending an audio request to the service and receiving the audio response. Uses bidirectional streaming to receive results, such as the END_OF_UTTERANCE event, while sending audio.

A conversation is one or more gRPC connections, each consisting of several streamed requests and responses. For example, the user says Add to my shopping list and the Assistant responds What do you want to add?. The sequence of streamed requests and responses in the first gRPC message could be:

AssistRequest.config
AssistRequest.audio_in
AssistRequest.audio_in
AssistRequest.audio_in
AssistRequest.audio_in
AssistResponse.event_type.END_OF_UTTERANCE
AssistResponse.speech_results.transcript "add to my shopping list"
AssistResponse.dialog_state_out.microphone_mode.DIALOG_FOLLOW_ON
AssistResponse.audio_out
AssistResponse.audio_out
AssistResponse.audio_out

The user then says bagels and the Assistant responds OK, I've added bagels to your shopping list. This is sent as another gRPC connection call to the Assist method, again with streamed requests and responses, such as:

AssistRequest.config
AssistRequest.audio_in
AssistRequest.audio_in
AssistRequest.audio_in
AssistResponse.event_type.END_OF_UTTERANCE
AssistResponse.dialog_state_out.microphone_mode.CLOSE_MICROPHONE
AssistResponse.audio_out
AssistResponse.audio_out
AssistResponse.audio_out
AssistResponse.audio_out

Although the precise order of responses is not guaranteed, sequential AssistResponse.audio_out messages will always contain sequential portions of audio.

Authorization Scopes

Requires the following OAuth scope:

https://www.googleapis.com/auth/assistant-sdk-prototype

For more information, see the Auth Guide.

AssistConfig

Specifies how to process the AssistRequest messages.

Fields
`audio_out_config`	`AudioOutConfig` Required Specifies how to format the audio that will be returned.
`screen_out_config`	`ScreenOutConfig` Optional Specifies the desired format to use when server returns a visual screen response.
`dialog_state_in`	`DialogStateIn` Required Represents the current dialog state.
`device_config`	`DeviceConfig` Device configuration that uniquely identifies a specific device.
`debug_config`	`DebugConfig` Optional Debugging parameters for the whole `Assist` RPC.
Union field `type`. `type` can be only one of the following:
`audio_in_config`	`AudioInConfig` Specifies how to process the subsequent incoming audio. Required if `AssistRequest.audio_in` bytes will be provided in subsequent requests.
`text_query`	`string` The text input to be sent to the Assistant. This can be populated from a text interface if audio input is not available.

AssistRequest

The top-level message sent by the client. Clients must send at least two, and typically numerous AssistRequest messages. The first message must contain a config message and must not contain audio_in data. All subsequent messages must contain audio_in data and must not contain a config message.

Fields

Fields
Union field `type`. Exactly one of these fields must be specified in each `AssistRequest`. `type` can be only one of the following:
`config`	`AssistConfig` The `config` message provides information to the recognizer that specifies how to process the request. The first `AssistRequest` message must contain a `config` message.
`audio_in`	`bytes` The audio data to be recognized. Sequential chunks of audio data are sent in sequential `AssistRequest` messages. The first `AssistRequest` message must not contain `audio_in` data and all subsequent `AssistRequest` messages must contain `audio_in` data. The audio bytes must be encoded as specified in `AudioInConfig`. Audio must be sent at approximately real-time (16000 samples per second). An error will be returned if audio is sent significantly faster or slower.

Union field type. Exactly one of these fields must be specified in each AssistRequest. type can be only one of the following:

config

AssistConfig

The config message provides information to the recognizer that specifies how to process the request. The first AssistRequest message must contain a config message.

audio_in

bytes

The audio data to be recognized. Sequential chunks of audio data are sent in sequential AssistRequest messages. The first AssistRequest message must not contain audio_in data and all subsequent AssistRequest messages must contain audio_in data. The audio bytes must be encoded as specified in AudioInConfig. Audio must be sent at approximately real-time (16000 samples per second). An error will be returned if audio is sent significantly faster or slower.

AssistResponse

The top-level message received by the client. A series of one or more AssistResponse messages are streamed back to the client.

Fields
`event_type`	`EventType` Output-only Indicates the type of event.
`audio_out`	`AudioOut` Output-only The audio containing the Assistant's response to the query.
`screen_out`	`ScreenOut` Output-only Contains the Assistant's visual response to the query.
`device_action`	`DeviceAction` Output-only Contains the action triggered by the query with the appropriate payloads and semantic parsing.
`speech_results[]`	`SpeechRecognitionResult` Output-only This repeated list contains zero or more speech recognition results that correspond to consecutive portions of the audio currently being processed, starting with the portion corresponding to the earliest audio (and most stable portion) to the portion corresponding to the most recent audio. The strings can be concatenated to view the full in-progress response. When the speech recognition completes, this list will contain one item with `stability` of `1.0`.
`dialog_state_out`	`DialogStateOut` Output-only Contains output related to the user's query.
`debug_info`	`DebugInfo` Output-only Debugging info for developer. Only returned if request set `return_debug_info` to true.

EventType

Indicates the type of event.

Enums

EVENT_TYPE_UNSPECIFIED No event specified.

END_OF_UTTERANCE This event indicates that the server has detected the end of the user's speech utterance and expects no additional speech. Therefore, the server will not process additional audio (although it may subsequently return additional results). The client should stop sending additional audio data, half-close the gRPC connection, and wait for any additional results until the server closes the gRPC connection.

Enums
`EVENT_TYPE_UNSPECIFIED`	No event specified.
`END_OF_UTTERANCE`	This event indicates that the server has detected the end of the user's speech utterance and expects no additional speech. Therefore, the server will not process additional audio (although it may subsequently return additional results). The client should stop sending additional audio data, half-close the gRPC connection, and wait for any additional results until the server closes the gRPC connection.

AudioInConfig

Specifies how to process the audio_in data that will be provided in subsequent requests. For recommended settings, see the Google Assistant SDK best practices.

Fields

Fields
`encoding`	`Encoding` Required Encoding of audio data sent in all `audio_in` messages.
`sample_rate_hertz`	`int32` Required Sample rate (in Hertz) of the audio data sent in all `audio_in` messages. Valid values are from 16000-24000, but 16000 is optimal. For best results, set the sampling rate of the audio source to 16000 Hz. If that's not possible, use the native sample rate of the audio source (instead of re-sampling).

encoding

Encoding

Required Encoding of audio data sent in all audio_in messages.

sample_rate_hertz

int32

Required Sample rate (in Hertz) of the audio data sent in all audio_in messages. Valid values are from 16000-24000, but 16000 is optimal. For best results, set the sampling rate of the audio source to 16000 Hz. If that's not possible, use the native sample rate of the audio source (instead of re-sampling).

Encoding

Audio encoding of the data sent in the audio message. Audio must be one-channel (mono).

Enums
`ENCODING_UNSPECIFIED`	Not specified. Will return result `google.rpc.Code.INVALID_ARGUMENT`.
`LINEAR16`	Uncompressed 16-bit signed little-endian samples (Linear PCM). This encoding includes no header, only the raw audio bytes.
`FLAC`	`FLAC` (Free Lossless Audio Codec) is the recommended encoding because it is lossless--therefore recognition is not compromised--and requires only about half the bandwidth of `LINEAR16`. This encoding includes the `FLAC` stream header followed by audio data. It supports 16-bit and 24-bit samples, however, not all fields in `STREAMINFO` are supported.

AudioOut

The audio containing the Assistant's response to the query. Sequential chunks of audio data are received in sequential AssistResponse messages.

Fields

Fields
`audio_data`	`bytes` Output-only The audio data containing the Assistant's response to the query. Sequential chunks of audio data are received in sequential `AssistResponse` messages.

audio_data

bytes

Output-only The audio data containing the Assistant's response to the query. Sequential chunks of audio data are received in sequential AssistResponse messages.

AudioOutConfig

Specifies the desired format for the server to use when it returns audio_out messages.

Fields

Fields
`encoding`	`Encoding` Required The encoding of audio data to be returned in all `audio_out` messages.
`sample_rate_hertz`	`int32` Required The sample rate in Hertz of the audio data returned in `audio_out` messages. Valid values are: 16000-24000.
`volume_percentage`	`int32` Required Current volume setting of the device's audio output. Valid values are 1 to 100 (corresponding to 1% to 100%).

encoding

Encoding

Required The encoding of audio data to be returned in all audio_out messages.

sample_rate_hertz

int32

Required The sample rate in Hertz of the audio data returned in audio_out messages. Valid values are: 16000-24000.

volume_percentage

int32

Required Current volume setting of the device's audio output. Valid values are 1 to 100 (corresponding to 1% to 100%).

Encoding

Audio encoding of the data returned in the audio message. All encodings are raw audio bytes with no header, except as indicated below.

Enums
`ENCODING_UNSPECIFIED`	Not specified. Will return result `google.rpc.Code.INVALID_ARGUMENT`.
`LINEAR16`	Uncompressed 16-bit signed little-endian samples (Linear PCM).
`MP3`	MP3 audio encoding. The sample rate is encoded in the payload.
`OPUS_IN_OGG`	Opus-encoded audio wrapped in an ogg container. The result will be a file which can be played natively on Android and in some browsers (such as Chrome). The quality of the encoding is considerably higher than MP3 while using the same bitrate. The sample rate is encoded in the payload.

DebugConfig

Debugging parameters for the current request.

Fields

Fields
`return_debug_info`	`bool` When this field is set to true, the `debug_info` field in `AssistResponse` may be populated. However it will significantly increase latency of responses.

return_debug_info

bool

When this field is set to true, the debug_info field in AssistResponse may be populated. However it will significantly increase latency of responses.

DebugInfo

Debug info for developer. Only returned if request set return_debug_info to true.

Fields

Fields
`aog_agent_to_assistant_json`	`string` The original JSON response from an Action-on-Google agent to Google server. See AppResponse. It will only be populated if the request maker owns the AoG project and the AoG project is in preview mode.

aog_agent_to_assistant_json

string

The original JSON response from an Action-on-Google agent to Google server. See AppResponse. It will only be populated if the request maker owns the AoG project and the AoG project is in preview mode.

DeviceAction

The response returned to the device if the user has triggered a Device Action. For example, a device which supports the query Turn on the light would receive a DeviceAction with a JSON payload containing the semantics of the request.

Fields

Fields
`device_request_json`	`string` JSON containing the device command response generated from the triggered Device Action grammar. The format is given by the `action.devices.EXECUTE` intent for a given trait.

device_request_json

string

JSON containing the device command response generated from the triggered Device Action grammar. The format is given by the action.devices.EXECUTE intent for a given trait.

DeviceConfig

Required Fields that identify the device to the Assistant.

Fields
`device_id`	`string` Required Unique identifier for the device. The id length must be 128 characters or less. Example: DBCDW098234. This MUST match the device_id returned from device registration. This device_id is used to match against the user's registered devices to lookup the supported traits and capabilities of this device. This information should not change across device reboots. However, it should not be saved across factory-default resets.
`device_model_id`	`string` Required Unique identifier for the device model. The combination of device_model_id and device_id must have been previously associated through device registration.

DeviceLocation

There are three sources of locations. They are used with this precedence:

This DeviceLocation, which is primarily used for mobile devices with GPS .
Location specified by the user during device setup; this is per-user, per device. This location is used if DeviceLocation is not specified.
Inferred location based on IP address. This is used only if neither of the above are specified.

Fields

Fields
`coordinates`	`LatLng` Latitude and longitude of device.

coordinates

LatLng

Latitude and longitude of device.

DialogStateIn

Provides information about the current dialog state.

Fields
`conversation_state`	`bytes` Required This field must always be set to the `DialogStateOut.conversation_state` value that was returned in the prior `Assist` RPC. It should only be omitted (field not set) if there was no prior `Assist` RPC because this is the first `Assist` RPC made by this device after it was first setup and/or a factory-default reset.
`language_code`	`string` Required Language of the request in IETF BCP 47 syntax (for example, "en-US"). See Language Support for more information. If you have selected a language for this `device_id` using the Settings menu in your phone's Google Assistant app, that selection will override this value.
`device_location`	`DeviceLocation` Optional Location of the device where the query originated.
`is_new_conversation`	`bool` Optional If true, the server will treat the request as a new conversation and not use state from the prior request. Set this field to true when the conversation should be restarted, such as after a device reboot, or after a significant lapse of time since the prior query.

DialogStateOut

The dialog state resulting from the user's query. Multiple of these messages may be received.

Fields
`supplemental_display_text`	`string` Output-only Supplemental display text from the Assistant. This could be the same as the speech spoken in `AssistResponse.audio_out` or it could be some additional information which aids the user's understanding.
`conversation_state`	`bytes` Output-only State information for the subsequent `Assist` RPC. This value should be saved in the client and returned in the `DialogStateIn.conversation_state` field with the next `Assist` RPC. (The client does not need to interpret or otherwise use this value.) This information should be saved across device reboots. However, this value should be cleared (not saved in the client) during a factory-default reset.
`microphone_mode`	`MicrophoneMode` Output-only Specifies the mode of the microphone after this `Assist` RPC is processed.
`volume_percentage`	`int32` Output-only Updated volume level. The value will be 0 or omitted (indicating no change) unless a voice command such as Increase the volume or Set volume level 4 was recognized, in which case the value will be between 1 and 100 (corresponding to the new volume level of 1% to 100%). Typically, a client should use this volume level when playing the `audio_out` data, and retain this value as the current volume level and supply it in the `AudioOutConfig` of the next `AssistRequest`. (Some clients may also implement other ways to allow the current volume level to be changed, for example, by providing a knob that the user can turn.)

MicrophoneMode

Possible states of the microphone after a Assist RPC completes.

Enums
`MICROPHONE_MODE_UNSPECIFIED`	No mode specified.
`CLOSE_MICROPHONE`	The service is not expecting a follow-on question from the user. The microphone should remain off until the user re-activates it.
`DIALOG_FOLLOW_ON`	The service is expecting a follow-on question from the user. The microphone should be re-opened when the `AudioOut` playback completes (by starting a new `Assist` RPC call to send the new audio).

ScreenOut

The Assistant's visual output response to query. Enabled by screen_out_config.

Fields

Fields
`format`	`Format` Output-only The format of the provided screen data.
`data`	`bytes` Output-only The raw screen data to be displayed as the result of the Assistant query.

format

Format

Output-only The format of the provided screen data.

data

bytes

Output-only The raw screen data to be displayed as the result of the Assistant query.

Format

Possible formats of the screen data.

Enums
`FORMAT_UNSPECIFIED`	No format specified.
`HTML`	Data will contain a fully-formed HTML5 layout encoded in UTF-8, e.g. `<html><body><div>...</div></body></html>`. It is intended to be rendered along with the audio response. Note that HTML5 doctype should be included in the actual HTML data.

ScreenOutConfig

Specifies the desired format for the server to use when it returns screen_out response.

Fields

Fields
`screen_mode`	`ScreenMode` Current visual screen-mode for the device while issuing the query.

screen_mode

ScreenMode

Current visual screen-mode for the device while issuing the query.

ScreenMode

Possible modes for visual screen-output on the device.

Enums
`SCREEN_MODE_UNSPECIFIED`	No video mode specified. The Assistant may respond as if in `OFF` mode.
`OFF`	Screen is off (or has brightness or other settings set so low it is not visible). The Assistant will typically not return a screen response in this mode.
`PLAYING`	The Assistant will typically return a partial-screen response in this mode.

SpeechRecognitionResult

The estimated transcription of a phrase the user has spoken. This could be a single segment or the full guess of the user's spoken query.

Fields

Fields
`transcript`	`string` Output-only Transcript text representing the words that the user spoke.
`stability`	`float` Output-only An estimate of the likelihood that the Assistant will not change its guess about this result. Values range from 0.0 (completely unstable) to 1.0 (completely stable and final). The default of 0.0 is a sentinel value indicating `stability` was not set.

transcript

string

Output-only Transcript text representing the words that the user spoke.

stability

float

Output-only An estimate of the likelihood that the Assistant will not change its guess about this result. Values range from 0.0 (completely unstable) to 1.0 (completely stable and final). The default of 0.0 is a sentinel value indicating stability was not set.