Back to Details

google-cloud-solution-agentic-ai-bidirectional-streaming

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Live bidirectional multimodal streaming agentic AI solution

实时双向多模态流Agentic AI解决方案

This skill guides agents through the workflow to design and implement a tailored multi-product solution in the cloud for a live, bidirectional multimodal streaming workload, use case, or requirement.
本Skill指导Agent完成工作流，为实时双向多模态流工作负载、用例或需求，在云端设计并实现定制化多产品解决方案。

Workflow

工作流

The solution design and implementation workflow consists of the following phases:
  • Phase 1: Requirements discovery and analysis: Analyze the workload's requirements, constraints, dependencies, and current state.
  • Phase 2: Solution design: Build a technology stack, architecture, and deployment configuration for the workload based on Google Cloud design best practices and recommendations.
  • Phase 3: Implementation plan: Generate automation and instructions to deploy the solution.
  • Phase 4: Solution validation: Validate that the deployment meets the requirements of the workload.
解决方案设计与实施工作流包含以下阶段：
  • 阶段1：需求挖掘与分析：分析工作负载的需求、约束、依赖关系及当前状态。
  • 阶段2：解决方案设计：基于Google Cloud设计最佳实践与建议，为工作负载构建技术栈、架构及部署配置。
  • 阶段3：实施计划：生成自动化脚本与部署说明，用于部署解决方案。
  • 阶段4：解决方案验证：验证部署是否满足工作负载的需求。

Supporting links

参考链接

Use these references to answer user questions that might deviate from the workflow phases.
若用户问题偏离工作流阶段，可使用以下参考资料解答。

Phase 1: Requirements discovery and analysis

阶段1：需求挖掘与分析

  1. Discover requirements: Understand the functional and non-functional requirements, business goals, and current state (if any) of the workload, including its architecture, dependencies, and constraints. Use the following questions to guide the requirements discovery process:
    • What are the primary input modalities (audio, video, or text) and what is the target latency for real-time, narrated feedback?
    • Do you require real-time safety monitoring, hazard detection, or visual inspection? If so, then what specific safety hazards, operational risks, or incorrect steps need to be monitored and detected in the video stream?
    • What existing systems, knowledge bases, product documentation, or schematic repositories must the AI agents access for grounded guidance?
    • What are the client-side device constraints and network limitations?
  2. Identify components: Based on the requirements analysis, identify the components of the workload and their relationships. Also identify any cross-cloud components, hybrid components, or on-prem components that the solution needs to integrate with.
  3. Generate component decomposition: Generate a technical decomposition of the components of the workload.
  4. Ask for confirmation: Ask the user to confirm whether the generated technical decomposition matches their workload requirements.
  5. Iterate: If the user requests changes, then generate an updated technical decomposition, and ask the user to confirm the changes. Continue iterating until the user confirms the technical decomposition.
  1. 挖掘需求：理解工作负载的功能与非功能需求、业务目标及当前状态（若存在），包括其架构、依赖关系与约束。可通过以下问题引导需求挖掘过程：
    • 主要输入模态是什么（音频、视频或文本）？实时语音反馈的目标延迟是多少？
    • 是否需要实时安全监控、危险检测或视觉检查？如果需要，视频流中需要监控和检测哪些具体的安全隐患、操作风险或错误步骤？
    • AI Agent必须访问哪些现有系统、知识库、产品文档或原理图存储库以提供可靠指导？
    • 客户端设备约束与网络限制有哪些？
  2. 识别组件：基于需求分析，识别工作负载的组件及其关系，同时识别解决方案需要集成的跨云组件、混合组件或本地组件。
  3. 生成组件分解：生成工作负载组件的技术分解文档。
  4. 请求确认：请用户确认生成的技术分解是否符合其工作负载需求。
  5. 迭代优化：若用户要求修改，则生成更新后的技术分解文档，并请用户确认修改内容。持续迭代直至用户确认技术分解文档。

Phase 2: Solution design

阶段2：解决方案设计

  1. Retrieve relevant Google Cloud documentation:
    Important: Use the content that you retrieve from Google Cloud documentation to ground the guidance that you generate in the remaining steps of this phase.
  2. Map components to Google Cloud products:
    • Cloud Run networking:
      • Recommended primary configuration: Regional External Application Load Balancer combined with Cloud Armor for HTTP/HTTPS/WebSocket ingress, and Direct VPC egress for Cloud Run private network access.
      • Alternative product 1: Global External Application Load Balancer
        • Pros: Single anycast IP, global IPv6 termination, and low-latency routes to globally distributed backend services.
        • Cons: Terminates TLS globally at edge locations, which might not comply with strict regional data residency regulations.
      • Alternative product 2: Internal Application Load Balancer
        • Pros: Securely exposes Cloud Run services internally within the VPC to meet internal ingress criteria, terminates TLS with trusted certificates, and supports Cloud Armor backend security policies.
        • Cons: Requires that you configure serverless network endpoint groups (NEGs) as backends and manage load balancer resources.
      • Alternative product 3: Private Service Connect interface
        • Pros: Secure private VPC connections for Gemini Enterprise Agent Runtime that uses network attachments.
        • Cons: Limited to RFC 1918 routable subnet ranges, requires proxy setup for non-routable/internet destinations.
    • Frontend:
      • Recommended primary product: Cloud Run
      • Alternative product 1: Firebase App Hosting
        • Pros: Automated builds and deployment pipeline from GitHub, optimized for modern framework integrations.
        • Cons: Less control over container configurations, limits customization of low-level networking.
      • Alternative product 2: Google Kubernetes Engine (GKE)
        • Pros: Maximum control over routing, scaling, and custom container runtimes.
        • Cons: Significant infrastructure management complexity and cost overhead.
    • Agent development framework:
      • Recommended primary product: Agent Development Kit (ADK).
    • Agent-to-agent communication:
      • Recommended primary product: Agent2Agent (A2A) protocol.
    • Runtime for your agent:
      • Recommended primary product: Cloud Run
      • Alternative product 1: Gemini Enterprise Agent Runtime
        • Pros: Fully managed Python runtime, built-in memory/sessions, secure code execution sandbox.
        • Cons: Limited to Python, does not support hosting custom MCP servers, less control over container environment.
      • Alternative product 2: Google Kubernetes Engine (GKE)
        • Pros: Maximum infrastructure control, stateful pods, custom scaling.
        • Cons: High operational complexity and overhead.
    • Model runtime:
      • Recommended primary product: Gemini Enterprise Agent Platform
      • Alternative product 1: Cloud Run
        • Pros: Serverless hosting for containerized open/custom models like Gemma.
        • Cons: Cannot serve Google Gemini models, manual instance scaling overhead.
      • Alternative product 2: Google Kubernetes Engine (GKE)
        • Pros: Maximum control over inference server on GPU/TPU nodes, cheap for predictable high volume.
        • Cons: Cannot run Google Gemini models, high cluster management overhead.
    • Model selection:
      • Recommended primary product: Gemini Flash with Gemini Live API
      • Alternative product 1: Gemini Pro
        • Pros: Highest capability for reasoning, complex instructions, context tracking, and multi-agent coordination.
        • Cons: Higher request cost and latency, which makes it less suitable for real-time conversational requirements.
    • VPC connection to the database: The architect agent sends queries through this connector to securely access resources in the Virtual Private Cloud (VPC) network used for storage resources in this architecture.
      • Recommended primary product: Serverless VPC Access connector.
      • Alternative product 1: Direct VPC egress
        • Pros: Lower latency, lower resource cost, and avoids throughput scaling bottlenecks.
        • Cons: Requires specific routing and subnet configurations.
    • Caching:
      • Recommended primary product: Memorystore for Redis Cluster
    • Database:
      • Recommended primary product: Google Cloud Databases Use the recommendations listed on this page to help the user choose the appropriate database option.
      • Alternative product 1: Compute Engine (for self-hosted databases)
        • Pros: Full control over database configurations, OS accessibility, and custom database engines or extensions.
        • Cons: High operational overhead to manually manage backups, patching, scaling, and high availability.
  3. Create architecture diagram: Create an architecture diagram that shows the components, their relationships, and data and control flows.
  4. Generate design recommendations: Generate design guidance based on the following Google Cloud best practices and recommendations:
    • Security, privacy, and compliance:
      • To limit access to the app, disable the default run.app URL of the frontend Cloud Run service and configure a regional external Application Load Balancer with Cloud Armor security policies to handle request filtering, rate limiting, and DDoS protection.
      • Enforce the principle of least privilege when you configure IAM permissions for resources in the topology.
      • To protect sensitive multimodal data (such as voice prints and video), enforce TLS encryption for all bidirectional WebSocket connections.
      • Secure Agent2Agent (A2A) communication using authenticated extended agent cards, and attach OpenID Connect (OIDC) identity tokens to let IAM validate that only authorized agents access the data.
      • Incorporate human-in-the-loop flows to let supervisors monitor, pause, and override business-critical agent actions.
      • For more information about security considerations, see https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/security.md.txt
    • Reliability:
      • Build fault-tolerant agents employing decentralized designs where agents can operate independently to survive failures.
      • Simulate inter-agent coordination issues and unexpected behaviors in a replica staging environment before deploying to production.
      • Leverage regional multi-zone deployment of Cloud Run to automatically load-balance and survive zone outages.
      • Plan model capacity by monitoring standard quota rates, using Provisioned Throughput for business-critical production workloads.
      • For more information about reliability considerations, see https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/reliability.md.txt
    • Operational excellence:
      • Route agent logs to Cloud Logging in structured formats, integrating standard stdout/stderr streams.
      • Track complete agent workflows, reasoning loops, and execution paths using Cloud Trace and trace visualizers.
      • Perform continuous evaluation using tools like Agent Evaluation on Gemini Enterprise Agent Platform or ADK evaluation methodologies.
      • Centralize database tools and connection scaling policies using the MCP Database Toolbox.
      • For more information about operational excellence considerations, see https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/operational-excellence.md.txt
    • Cost optimization:
      • Reduce data ingestion costs by employing low-frequency frame sampling and compressing video to Base64 JPEGs.
      • Use context caching for requests containing long system prompts or static lookup databases to reduce input token costs.
      • Structure prompts to get concise responses to minimize generation token outputs.
      • Start with the most smaller and cost-efficient models. Upgrade to more powerful models with reasoning based on performance requirements.
      • For more information about cost optimization considerations, see https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/cost-optimization.md.txt
    • Performance efficiency:
      • Decouple incoming audio and video packets from the model's inference engine. Use a thread-safe, asynchronous First-In-First-Out (FIFO) buffer to keep the user interface responsive to interruptions.
      • To achieve sub-millisecond read speeds and prevent silences during real-time voice interactions, deploy an in-memory Memorystore for Redis Cluster database for the agent's schematic vault.
      • To optimize service performance, configure memory limits and CPU limits allocated to the Cloud Run instances based on live workloads.
      • For more information about performance considerations, see https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/performance-optimization.md.txt.
    • Sustainability:
  5. Draft solution architecture: Compile the requirements, technical decomposition, product mapping, architecture diagram, and design recommendations into a single Markdown file.
  6. Request review: Present the generated solution architecture to the user and request their feedback or approval.
  7. Iterate: If the user requests changes, generate an updated solution architecture and repeat steps 5-6 until the user approves the solution architecture.
  1. 检索相关Google Cloud文档:
    重要提示：利用从Google Cloud文档中检索到的内容，为本阶段后续步骤生成的指导提供依据。
  2. 将组件映射到Google Cloud产品:
    • Cloud Run网络:
      • 推荐主配置：区域外部应用负载均衡器结合Cloud Armor，处理HTTP/HTTPS/WebSocket入站流量，同时使用Direct VPC出口实现Cloud Run私有网络访问。
      • 替代方案1：全球外部应用负载均衡器
        • 优势：单任播IP、全球IPv6终止、面向全球分布式后端服务的低延迟路由。
        • 劣势：在边缘位置全局终止TLS，可能不符合严格的区域数据驻留法规。
      • 替代方案2：内部应用负载均衡器
        • 优势：在VPC内部安全暴露Cloud Run服务以满足内部入站要求，使用可信证书终止TLS，并支持Cloud Armor后端安全策略。
        • 劣势：需要将无服务器网络端点组（NEG）配置为后端，并管理负载均衡器资源。
      • 替代方案3：Private Service Connect接口
        • 优势：为Gemini Enterprise Agent Runtime提供安全的私有VPC连接，使用网络附件。
        • 劣势：仅限于RFC 1918可路由子网范围，需要为不可路由/互联网目标设置代理。
    • 前端:
      • 推荐主产品：Cloud Run
      • 替代方案1：Firebase App Hosting
        • 优势：从GitHub自动构建与部署流水线，针对现代框架集成优化。
        • 劣势：对容器配置的控制较少，限制底层网络的自定义。
      • 替代方案2：Google Kubernetes Engine (GKE)
        • 优势：对路由、扩缩容和自定义容器运行时拥有最大控制权。
        • 劣势：基础设施管理复杂度与成本开销较高。
    • Agent开发框架:
      • 推荐主产品：Agent Development Kit (ADK)。
    • Agent间通信:
      • 推荐主产品：Agent2Agent (A2A)协议。
    • Agent运行时:
      • 推荐主产品：Cloud Run
      • 替代方案1：Gemini Enterprise Agent Runtime
        • 优势：全托管Python运行时，内置内存/会话功能，安全代码执行沙箱。
        • 劣势：仅限于Python，不支持托管自定义MCP服务器，对容器环境的控制较少。
      • 替代方案2：Google Kubernetes Engine (GKE)
        • 优势：对基础设施拥有最大控制权，支持有状态Pod与自定义扩缩容。
        • 劣势：运营复杂度与开销较高。
    • 模型运行时:
      • 推荐主产品：Gemini Enterprise Agent Platform
      • 替代方案1：Cloud Run
        • 优势：为容器化开源/自定义模型（如Gemma）提供无服务器托管。
        • 劣势：无法部署Google Gemini模型，需要手动管理实例扩缩容。
      • 替代方案2：Google Kubernetes Engine (GKE)
        • 优势：对GPU/TPU节点上的推理服务器拥有最大控制权，针对可预测的高流量场景成本较低。
        • 劣势：无法运行Google Gemini模型，集群管理开销较高。
    • 模型选择:
      • 推荐主产品：Gemini Flash with Gemini Live API
      • 替代方案1：Gemini Pro
        • 优势：在推理、复杂指令处理、上下文跟踪及多Agent协调方面能力最强。
        • 劣势：请求成本与延迟较高，不太适合实时对话需求。
    • 数据库VPC连接：架构Agent通过此连接器发送查询，安全访问本架构中用于存储资源的Virtual Private Cloud (VPC)网络内的资源。
      • 推荐主产品：Serverless VPC Access连接器。
      • 替代方案1：Direct VPC出口
        • 优势：延迟更低、资源成本更低，避免吞吐量扩缩容瓶颈。
        • 劣势：需要特定的路由与子网配置。
    • 缓存:
      • 推荐主产品：Memorystore for Redis Cluster
    • 数据库:
      • 推荐主产品：Google Cloud Databases 使用此页面列出的建议帮助用户选择合适的数据库选项。
      • 替代方案1：Compute Engine（用于自托管数据库）
        • 优势：完全控制数据库配置、操作系统访问权限，支持自定义数据库引擎或扩展。
        • 劣势：手动管理备份、补丁、扩缩容与高可用性的运营开销较高。
  3. 创建架构图：创建架构图，展示组件、组件关系及数据与控制流。
  4. 生成设计建议：基于以下Google Cloud最佳实践与建议生成设计指导：
    • 安全、隐私与合规:
      • 为限制应用访问，禁用前端Cloud Run服务的默认run.app URL，配置区域外部应用负载均衡器并搭配Cloud Armor安全策略，处理请求过滤、速率限制与DDoS防护。
      • 配置拓扑中资源的IAM权限时，遵循最小权限原则。
      • 为保护敏感多模态数据（如声纹与视频），强制所有双向WebSocket连接使用TLS加密。
      • 使用经过认证的扩展Agent卡片保护Agent2Agent (A2A)通信，并附加OpenID Connect (OIDC)身份令牌，让IAM验证只有授权Agent可访问数据。
      • 融入人工介入流程，让管理员能够监控、暂停并覆盖关键业务的Agent操作。
      • 有关安全注意事项的更多信息，请参阅 https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/security.md.txt
    • 可靠性:
      • 构建容错Agent，采用去中心化设计，使Agent可独立运行以应对故障。
      • 在部署到生产环境前，在副本 staging 环境中模拟Agent间协调问题与意外行为。
      • 利用Cloud Run的区域多区域部署，自动负载均衡并应对区域故障。
      • 通过监控标准配额速率、为关键业务生产工作负载使用Provisioned Throughput来规划模型容量。
      • 有关可靠性注意事项的更多信息，请参阅 https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/reliability.md.txt
    • 运营卓越:
      • 将Agent日志以结构化格式路由到Cloud Logging，集成标准stdout/stderr流。
      • 使用Cloud Trace与跟踪可视化工具跟踪完整的Agent工作流、推理循环与执行路径。
      • 使用Gemini Enterprise Agent Platform上的Agent Evaluation或ADK评估方法等工具进行持续评估。
      • 使用MCP Database Toolbox集中管理数据库工具与连接扩缩容策略。
      • 有关运营卓越注意事项的更多信息，请参阅 https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/operational-excellence.md.txt
    • 成本优化:
      • 采用低频率帧采样并将视频压缩为Base64 JPEG格式，降低数据 ingestion 成本。
      • 对包含长系统提示或静态查找数据库的请求使用上下文缓存，降低输入令牌成本。
      • 构建提示以获取简洁响应，最小化生成令牌输出。
      • 从更小、更具成本效益的模型开始，根据性能需求升级到更强大的推理模型。
      • 有关成本优化注意事项的更多信息，请参阅 https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/cost-optimization.md.txt
    • 性能效率:
      • 将传入的音频与视频数据包与模型推理引擎解耦。使用线程安全的异步先进先出（FIFO）缓冲区，保持用户界面可响应中断。
      • 为实现亚毫秒级读取速度并防止实时语音交互期间出现静默，为Agent的原理图库部署内存型Memorystore for Redis Cluster数据库。
      • 根据实时工作负载配置分配给Cloud Run实例的内存限制与CPU限制，优化服务性能。
      • 有关性能注意事项的更多信息，请参阅 https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/performance-optimization.md.txt.
    • 可持续性:
  5. 起草解决方案架构：将需求、技术分解、产品映射、架构图及设计建议整合到单个Markdown文件中。
  6. 请求评审：向用户展示生成的解决方案架构，并请求反馈或批准。
  7. 迭代优化：若用户要求修改，则生成更新后的解决方案架构，重复步骤5-6直至用户批准。

Phase 3: Implementation plan

阶段3：实施计划

  1. Retrieve relevant implementation resources:
    Important: Use these resources as the technical foundation for the IaC and deployment instructions you generate in the remaining steps of this phase.
  2. Identify deployment prerequisites: Document prerequisites for the deployment, including the following:
    • Projects and billing associations
    • Required Google Cloud APIs
    • Required IAM permissions
    • Any other prerequisites
  3. Generate Infrastructure as Code (IaC): Generate code, like Terraform, and deployment scripts to automate the provisioning of the proposed Google Cloud resources.
  4. Write deployment instructions: Draft sequential, step-by-step deployment instructions to execute the IaC and initialize the workload components.
  5. Request review: Present the generated deployment instructions to the user for feedback and confirmation.
  6. Iterate: If the user requests changes, then generate an updated implementation plan and repeat steps 4-5 until the user approves the implementation plan.
  1. 检索相关实施资源:
    重要提示：将这些资源作为生成基础设施即代码（IaC）与部署说明的技术基础，用于本阶段后续步骤。
  2. 识别部署先决条件：记录部署的先决条件，包括：
    • 项目与计费关联
    • 所需Google Cloud API
    • 所需IAM权限
    • 其他任何先决条件
  3. 生成基础设施即代码（IaC）：生成Terraform等代码及部署脚本，自动化部署提议的Google Cloud资源。
  4. 编写部署说明：起草按顺序排列的分步部署说明，用于执行IaC并初始化工作负载组件。
  5. 请求评审：向用户展示生成的部署说明，请求反馈与确认。
  6. 迭代优化：若用户要求修改，则生成更新后的实施计划，重复步骤4-5直至用户批准。

Phase 4: Solution validation

阶段4：解决方案验证

  1. Retrieve relevant verification resources:
    Important: Use these resources and their verification patterns as the starting point for the validation checks and verification scripts that you generate in the remaining steps of this phase.
  2. Define validation checks: Outline validation steps to verify that the deployed infrastructure meets the workload's requirements:
    • Deployment dry-run: Commands like 
      terraform plan
      to preview changes.
    • Connectivity and routing: Verification of network paths, load balancer routing, and service endpoints.
    • Security policies: Verification of restricted access, firewall rules, and IAM enforcement.
  3. Generate verification scripts: Draft lightweight scripts or command-line instructions, such as using 
    curl
    or 
    gcloud
    , that the user can run to perform these validation checks.
  4. Compile validation report: Document the validation steps, verification scripts, and expected outcomes in a single Markdown file.
  5. Conduct validation and finalize: Assist the user in executing the validation checks and troubleshooting any deployment issues. After the solution is validated successfully, request final approval from the user.
  6. Iterate: If the user requests changes, then generate an updated validation plan and repeat steps 4-5 until the user approves the validation plan.
  1. 检索相关验证资源:
    重要提示：将这些资源及其验证模式作为起点，生成本阶段后续步骤中的验证检查与验证脚本。
  2. 定义验证检查：概述验证步骤，确认部署的基础设施是否满足工作负载需求：
    • 部署预演：使用
      terraform plan
      等命令预览变更。
    • 连通性与路由：验证网络路径、负载均衡器路由及服务端点。
    • 安全策略：验证访问限制、防火墙规则及IAM实施情况。
  3. 生成验证脚本：起草轻量级脚本或命令行说明（如使用
    curl
    或
    gcloud
    ），供用户运行以执行这些验证检查。
  4. 编制验证报告：将验证步骤、验证脚本及预期结果记录在单个Markdown文件中。
  5. 执行验证并最终确认：协助用户执行验证检查并排查任何部署问题。解决方案验证成功后，请求用户最终批准。
  6. 迭代优化：若用户要求修改，则生成更新后的验证计划，重复步骤4-5直至用户批准。