gemini-live-api

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

LiveAPI Service Skill

LiveAPI服务技能

This skill provides instructions for generating a LiveAPI client service class that connects to the Gemini Enterprise Live API over WebSockets. The generated client handles bidirectional streaming, bearer-token authentication via Application Default Credentials (ADC), transparent session resumption, and

ClientMessage

ServerMessage

proto exchange.

The skill also produces a demo frontend + backend service so the user can interactively validate the generated client (text, audio, video, transcription, and interrupt handling).

本技能提供生成LiveAPI客户端服务类的指导说明，该类通过WebSocket连接Gemini Enterprise Live API。生成的客户端支持双向流式传输、通过应用默认凭据（ADC）进行Bearer令牌认证、透明会话恢复，以及

ClientMessage

ServerMessage

proto的交互。

本技能还会生成一个演示前端+后端服务，方便用户交互式验证生成的客户端（支持文本、音频、视频、转录及中断处理）。

Prerequisites

前提条件

Before running the generation flow, ensure the following are available on the host:

A Google Cloud project with the Vertex AI / Gemini Enterprise Agent Platform APIs enabled.
Application Default Credentials configured on the host running the generated client:
bash
```
gcloud auth application-default login
```
A destination output folder supplied by the user (e.g.
```
/tmp/liveapi_out
```
) where the generated code, environment, and demo will be written. Never mutate the host's system Python environment.
The user's chosen implementation language (Python is the default and reference language for this skill).

在运行生成流程前，请确保主机满足以下条件：

已启用Vertex AI / Gemini Enterprise Agent Platform API的Google Cloud项目。
运行生成客户端的主机已配置应用默认凭据：
bash
```
gcloud auth application-default login
```
用户提供的目标输出文件夹（例如
```
/tmp/liveapi_out
```
），生成的代码、环境及演示内容将写入该文件夹。切勿修改主机的系统Python环境。
用户选定的实现语言（Python是本技能的默认参考语言）。

Reference Files

参考文件

Provided files in

references/

(do not treat these as standalone skills — they are loaded on demand):

```
client_server_messages.md
```
: Public reference for the
```
ClientMessage
```
/
```
ServerMessage
```
schemas used by the Live API.

client_server_messages.proto

: The proto definition generated from

client_server_messages.md

```
session_manager.md
```
: Describes how to correctly handle sessions, buffering, and resumption on disconnection.

references/

目录下提供的文件（请勿将这些文件视为独立技能——它们会按需加载）：

client_server_messages.md

：Live API所用

ClientMessage

ServerMessage

schema的公开参考文档。

client_server_messages.proto

：从

client_server_messages.md

生成的proto定义文件。

```
session_manager.md
```
：描述如何正确处理会话、缓冲以及断开连接后的恢复操作。

Steps

步骤

Step 1: Copy the reference files

步骤1：复制参考文件

Copy

client_server_messages.md

client_server_messages.proto

, and

session_manager.md

from this skill's

references/

folder into the user's destination output folder. These files become the source of truth for the generated client.

将本技能

references/

目录下的

client_server_messages.md

、

client_server_messages.proto

和

session_manager.md

复制到用户的目标输出文件夹。这些文件将作为生成客户端的事实依据。

Step 2: Reconcile with the public documentation

步骤2：与公开文档同步

Examine the public documents linked from

client_server_messages.md

. If there are any discrepancies between the public documents and the copied

client_server_messages.md

client_server_messages.proto

, update the copies in the destination folder so the generated client compiles and runs against the current server contract.

检查

client_server_messages.md

中链接的公开文档。如果公开文档与复制的

client_server_messages.md

client_server_messages.proto

存在差异，请更新目标文件夹中的副本，确保生成的客户端能根据当前服务器协议编译并运行。

Step 3: Implement the client class

步骤3：实现客户端类

Implement a class in the user's chosen language that:

Imports the local

client_server_messages.proto

types (

ClientMessage

ServerMessage

Opens a WebSocket connection to the Live API endpoint.
Exposes async methods so the user can send and receive data to/from the model.

For languages that require an isolated runtime (e.g. Python), create an isolated environment (e.g.

venv

) inside the destination folder and generate a bash script (e.g.

setup.sh

) that recreates the environment and installs dependencies. Never install into the system interpreter or the user's global site-packages, and never instruct the user to run

sudo pip install

使用用户选定的语言实现一个类，该类需满足：

导入本地的

client_server_messages.proto

类型（

ClientMessage

、

ServerMessage

）。

打开与Live API端点的WebSocket连接。
暴露异步方法，方便用户与模型之间发送和接收数据。

对于需要隔离运行时的语言（例如Python），请在目标文件夹内创建一个隔离环境（例如

venv

），并生成一个bash脚本（例如

setup.sh

）用于重建环境并安装依赖。切勿安装到系统解释器或用户全局site-packages，也切勿指示用户运行

sudo pip install

。

Initialization parameters

初始化参数

The user provides the following at construction time:

```
project_id
```
```
location
```
```
model_id
```
```
config
```
: a
```
ClientMessage
```
with the
```
setup
```
field populated.

用户在构造时需提供以下参数：

```
project_id
```
```
location
```
```
model_id
```
```
config
```
：一个填充了
```
setup
```
字段的
```
ClientMessage
```
。

Authentication

认证

Obtain a bearer token via Application Default Credentials, attach it to the WebSocket connect request as

Authorization: Bearer <token>

, refresh the token before or upon expiry, and reuse the refreshed token on every reconnection (including

go_away

and unexpected disconnects). Do not hard-code a long-lived API key as the only auth mechanism.

通过应用默认凭据获取Bearer令牌，将其作为

Authorization: Bearer <token>

附加到WebSocket连接请求中，在令牌过期前刷新，并在每次重新连接（包括

go_away

和意外断开）时使用刷新后的令牌。请勿将长期有效的API密钥作为唯一的认证机制硬编码。

Public async API

公开异步API

The class MUST expose the following async methods, gated on receipt of a

setup_complete

ServerMessage

before sending:

send_realtime_data(data)

: send realtime input.

data

is a

ClientMessage

carrying a

realtime_input

field.

```
send_client_content(data)
```
: send non-realtime, turn-based content that contributes to history.
```
data
```
is a
```
ClientMessage
```
carrying a
```
client_content
```
field.
```
receive()
```
: yield
```
ServerMessage
```
instances parsed from the WebSocket stream.

Do not expose synchronous blocking variants as the primary API surface.

该类必须暴露以下异步方法，且需在收到

setup_complete

类型的

ServerMessage

后才能发送请求：

send_realtime_data(data)

：发送实时输入。

data

是携带

realtime_input

字段的

ClientMessage

。

```
send_client_content(data)
```
：发送非实时、基于轮次的内容，这些内容会被纳入对话历史。
```
data
```
是携带
```
client_content
```
字段的
```
ClientMessage
```
。
```
receive()
```
：从WebSocket流中解析并生成
```
ServerMessage
```
实例。

请勿将同步阻塞变体作为主要API接口。

Step 4: Write a test file

步骤4：编写测试文件

Once the client is implemented, generate a test file that initializes the connection and exercises sending

text

audio

, and

video

data and receiving the responses. Ask the user for any information required to run the test (project, model, media samples).

客户端实现完成后，生成一个测试文件，用于初始化连接并测试发送

text

、

audio

和

video

数据以及接收响应。向用户询问运行测试所需的任何信息（项目、模型、媒体样本）。

Step 5: Generate

how_to_run.md

步骤5：生成

how_to_run.md

Provide a

how_to_run.md

in the destination folder that documents the generated class. Include full examples showing how to build

ClientMessage

payloads for every supported modality, how to send them, and how to receive data from the model.

在目标文件夹中提供

how_to_run.md

文档，记录生成的类。包含完整示例，展示如何为每种支持的模态构建

ClientMessage

负载、如何发送这些负载，以及如何从模型接收数据。

Step 6: Generate a demo frontend + backend service

步骤6：生成演示前端+后端服务

Create scripts that deploy the implementation as a service with both a frontend UI and a backend service (any language). The service MUST reuse the

ClientMessage

ServerMessage

protos from Step 1 for wire traffic. Through the UI the user should be able to:

Start a new connection / close the current connection.
Select the model to use.
Select input sources (audio and/or video from camera or screenshot) and stream them to the model.
Send a text message to the model.
Hear model audio and see the interleaved model and user transcription / conversation history.

While implementing audio and transcription playback, follow the guidance in Live API best practices.

创建脚本，将实现部署为包含前端UI和后端服务的服务（可使用任意语言）。该服务必须重用步骤1中的

ClientMessage

ServerMessage

proto进行网络通信。用户通过UI应能：

启动新连接/关闭当前连接。
选择要使用的模型。
选择输入源（来自摄像头或截图的音频和/或视频）并将其流式传输到模型。
向模型发送文本消息。
收听模型音频，并查看交错显示的模型和用户转录/对话历史。

在实现音频和转录播放时，请遵循Live API最佳实践中的指导。

Handling the

interrupt

signal

处理

interrupt

信号

When a

ServerMessage

server_content

arrives with

interrupted: true

, the UI MUST:

Ensure played audio and its corresponding transcription remain time-aligned.
Immediately stop the currently playing model audio and stop appending to the in-progress transcription bubble.
Clear the unplayed audio buffer and any pending unrendered transcription so stale content does not bleed into the next turn.
Start new chat bubbles for the next user and model turns.

当

ServerMessage

的

server_content

携带

interrupted: true

时，UI必须：

确保播放的音频及其对应的转录保持时间对齐。
立即停止当前播放的模型音频，并停止向正在进行的转录气泡追加内容。
清除未播放的音频缓冲区和任何待渲染的转录内容，避免陈旧内容混入下一轮对话。
为下一轮用户和模型对话启动新的聊天气泡。

Handling the transcription

finished

signal

处理转录

finished

信号

For streamed

input_transcription

output_transcription

chunks, append to the currently active bubble while

finished

is unset, and close that bubble and start a fresh one when

finished

is observed. Route

input_transcription

text to user-role bubbles and

output_transcription

text to model-role bubbles.

对于流式传输的

input_transcription

output_transcription

片段，当

finished

未设置时，追加到当前活动的气泡中；当观察到

finished

设置为真时，关闭该气泡并启动一个新气泡。将

input_transcription

文本路由到用户角色气泡，将

output_transcription

文本路由到模型角色气泡。

Step 7: Generate

how_to_test_with_ui.md

步骤7：生成

how_to_test_with_ui.md

Write

how_to_test_with_ui.md

describing how to launch and use the demo service. It MUST include:

The exact shell command(s) or script invocation(s) to start the backend service.
The exact shell command(s) or script invocation(s) to start the frontend UI.
The host and port (e.g.
```
http://localhost:PORT
```
) the user should open in their browser.
How to start a session, select a model, choose input sources (mic, camera, screen), send a text message, and observe model audio and transcription in the UI.

编写

how_to_test_with_ui.md

文档，说明如何启动和使用演示服务。该文档必须包含：

启动后端服务的确切shell命令或脚本调用方式。
启动前端UI的确切shell命令或脚本调用方式。
用户应在浏览器中打开的主机和端口（例如
```
http://localhost:PORT
```
）。
如何启动会话、选择模型、选择输入源（麦克风、摄像头、屏幕）、发送文本消息，以及在UI中观察模型音频和转录内容。

Validation Checklist

验证清单

Before considering the generation complete, verify each item:

client_server_messages.md

client_server_messages.proto

, and

session_manager.md

were copied into the destination folder.

The generated client imports the local proto-generated
```
ClientMessage
```
and
```
ServerMessage
```
types.

The client connects to the Live API WebSocket at

wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent

(or the

wss://aiplatform.googleapis.com/...

global variant), and formats the setup

model

field as

projects/{project_id}/locations/{location}/publishers/google/models/{model_id}

Authentication uses ADC-provided bearer tokens sent as
```
Authorization: Bearer <token>
```
, is refreshed before expiry, and reattached on every reconnect.
Public async methods
```
send_realtime_data
```
,
```
send_client_content
```
, and
```
receive
```
are present, correctly typed, and gated on
```
setup_complete
```
.
Transparent session resumption is enabled (
```
session_resumption.transparent = true
```
), the latest
```
new_handle
```
is tracked, sent-message indexing starts at 1, the buffer is pruned via
```
last_consumed_client_message_index
```
, and buffered messages are replayed on reconnect (including on
```
go_away
```
and WebSocket close codes 1000 / 1006).
If using python, an isolated environment (e.g.
```
venv
```
) plus a
```
setup.sh
```
and
```
requirements.txt
```
(or equivalent) exist inside the destination folder; no changes were made to system or user-global Python.

how_to_run.md

and

how_to_test_with_ui.md

are present, and the demo UI reuses the same

ClientMessage

ServerMessage

protos.

Interrupt handling and transcription
```
finished
```
handling behave as described above.
The client does not target
```
generativelanguage.googleapis.com
```
and does not authenticate via API key in a query string.

在认为生成完成前，请验证以下每一项：

client_server_messages.md

、

client_server_messages.proto

和

session_manager.md

已复制到目标文件夹。

生成的客户端导入了本地proto生成的
```
ClientMessage
```
和
```
ServerMessage
```
类型。

客户端连接到Live API WebSocket端点

wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent

（或全局变体

wss://aiplatform.googleapis.com/...

），并将

setup

字段中的

model

格式化为

projects/{project_id}/locations/{location}/publishers/google/models/{model_id}

。

认证使用ADC提供的Bearer令牌，以
```
Authorization: Bearer <token>
```
形式发送，在过期前刷新，并在每次重新连接时重新附加。
公开异步方法
```
send_realtime_data
```
、
```
send_client_content
```
和
```
receive
```
已存在，类型正确，并受
```
setup_complete
```
限制。
已启用透明会话恢复（
```
session_resumption.transparent = true
```
），跟踪最新的
```
new_handle
```
，发送消息索引从1开始，通过
```
last_consumed_client_message_index
```
修剪缓冲区，并且在重新连接时重播缓冲消息（包括
```
go_away
```
和WebSocket关闭代码1000/1006的情况）。
如果使用Python，目标文件夹内存在隔离环境（例如
```
venv
```
）以及
```
setup.sh
```
和
```
requirements.txt
```
（或等效文件）；未对系统或用户全局Python进行任何修改。

how_to_run.md

和

how_to_test_with_ui.md

已存在，且演示UI重用了相同的

ClientMessage

ServerMessage

proto。

中断处理和转录
```
finished
```
处理的行为符合上述描述。
客户端未以
```
generativelanguage.googleapis.com
```
为目标，也未通过查询字符串中的API密钥进行认证。

gemini-live-api

Original

Translation

LiveAPI Service Skill

LiveAPI服务技能

Prerequisites

前提条件

Reference Files

参考文件

Steps

步骤

Step 1: Copy the reference files

步骤1：复制参考文件

Step 2: Reconcile with the public documentation

步骤2：与公开文档同步

Step 3: Implement the client class

步骤3：实现客户端类

Initialization parameters

初始化参数

Authentication

认证

Public async API

公开异步API

Step 4: Write a test file

步骤4：编写测试文件

Step 5: Generate
`how_to_run.md`

步骤5：生成
`how_to_run.md`

Step 6: Generate a demo frontend + backend service

步骤6：生成演示前端+后端服务

Handling the
`interrupt`
signal

处理
`interrupt`
信号

Handling the transcription
`finished`
signal

处理转录
`finished`
信号

Step 7: Generate
`how_to_test_with_ui.md`

步骤7：生成
`how_to_test_with_ui.md`

Validation Checklist

验证清单

gemini-live-api

Original

Translation

LiveAPI Service Skill

LiveAPI服务技能

Prerequisites

前提条件

Reference Files

参考文件

Steps

步骤

Step 1: Copy the reference files

步骤1：复制参考文件

Step 2: Reconcile with the public documentation

步骤2：与公开文档同步

Step 3: Implement the client class

步骤3：实现客户端类

Initialization parameters

初始化参数

Authentication

认证

Public async API

公开异步API

Step 4: Write a test file

步骤4：编写测试文件

Step 5: Generate how_to_run.md

步骤5：生成how_to_run.md

Step 6: Generate a demo frontend + backend service

步骤6：生成演示前端+后端服务

Handling the interrupt signal

处理interrupt信号

Handling the transcription finished signal

处理转录finished信号

Step 7: Generate how_to_test_with_ui.md

步骤7：生成how_to_test_with_ui.md

Validation Checklist

验证清单

Step 5: Generate
`how_to_run.md`

步骤5：生成
`how_to_run.md`

Handling the
`interrupt`
signal

处理
`interrupt`
信号

Handling the transcription
`finished`
signal

处理转录
`finished`
信号

Step 7: Generate
`how_to_test_with_ui.md`

步骤7：生成
`how_to_test_with_ui.md`