ag2-multimodal-input

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Multimodal inputs

多模态输入

When to use

使用场景

The user wants the agent to process non-text input: an image to describe, audio to transcribe, video to summarise, or a PDF / document to extract from. The same factory pattern works across providers; per-provider support varies.

当用户需要Agent处理非文本输入时：比如描述图片、转录音频、总结视频，或从PDF/文档中提取信息。同一工厂模式适用于各服务商，但不同服务商的支持范围有所差异。

60-second recipe

60秒快速上手

python

from ag2 import Agent
from ag2.config import GeminiConfig
from ag2.events import ImageInput

agent = Agent(
    "vision",
    "You describe images.",
    config=GeminiConfig(model="gemini-3-flash-preview"),
)

image = ImageInput("https://example.com/photo.jpg")
reply = await agent.ask("Describe this image in detail.", image)
print(reply.body)

Multiple inputs in one ask are fine:

python

reply = await agent.ask(
    "Compare these two images.",
    ImageInput("https://example.com/before.jpg"),
    ImageInput("https://example.com/after.jpg"),
)

python

from ag2 import Agent
from ag2.config import GeminiConfig
from ag2.events import ImageInput

agent = Agent(
    "vision",
    "You describe images.",
    config=GeminiConfig(model="gemini-3-flash-preview"),
)

image = ImageInput("https://example.com/photo.jpg")
reply = await agent.ask("Describe this image in detail.", image)
print(reply.body)

一次请求中支持传入多个输入：

python

reply = await agent.ask(
    "Compare these two images.",
    ImageInput("https://example.com/before.jpg"),
    ImageInput("https://example.com/after.jpg"),
)

Input factories

输入工厂类

Factory	Formats
`ImageInput(...)`	JPEG, PNG, GIF, WebP
`AudioInput(...)`	WAV, MP3, OGG, FLAC, AAC
`VideoInput(...)`	MP4, WebM, MOV, MKV, MPEG
`DocumentInput(...)`	PDF, TXT, HTML, Markdown, CSV, JSON, Office formats

Each accepts the same four data sources:

python

from ag2.events import ImageInput

ImageInput("https://example.com/photo.jpg")     # URL
ImageInput(path="photo.jpg")                    # local file
ImageInput(data=raw_bytes, media_type="image/png")  # bytes
ImageInput(file_id="file-abc123")               # provider-uploaded

工厂类	支持格式
`ImageInput(...)`	JPEG、PNG、GIF、WebP
`AudioInput(...)`	WAV、MP3、OGG、FLAC、AAC
`VideoInput(...)`	MP4、WebM、MOV、MKV、MPEG
`DocumentInput(...)`	PDF、TXT、HTML、Markdown、CSV、JSON、Office格式

每个工厂类均支持以下四种数据来源：

python

from ag2.events import ImageInput

ImageInput("https://example.com/photo.jpg")     # URL
ImageInput(path="photo.jpg")                    # 本地文件
ImageInput(data=raw_bytes, media_type="image/png")  # 字节数据
ImageInput(file_id="file-abc123")               # 服务商已上传文件ID

Provider matrix

服务商支持矩阵

Input type	OpenAI	OpenAI Responses	Gemini	Anthropic
Text	✓	✓	✓	✓
Image (URL)	✓	✓	✓	✓
Image (binary)	✓	✓	✓	✓
Audio (URL)	–	–	✓	–
Audio (binary)	✓	–	✓	–
Video (URL)	–	–	✓	–
Video (binary)	–	–	✓	–
Document (URL)	–	✓	✓	✓
Document (binary)	✓	✓	✓	✓
File ID	✓	✓	✓	✓

Unsupported combinations raise

UnsupportedInputError

with a clear message.

Gemini has the broadest multimodal support. If you don't know which provider to pick for a multimodal task, start there.

输入类型	OpenAI	OpenAI Responses	Gemini	Anthropic
文本	✓	✓	✓	✓
图片（URL）	✓	✓	✓	✓
图片（二进制）	✓	✓	✓	✓
音频（URL）	–	–	✓	–
音频（二进制）	✓	–	✓	–
视频（URL）	–	–	✓	–
视频（二进制）	–	–	✓	–
文档（URL）	–	✓	✓	✓
文档（二进制）	✓	✓	✓	✓
文件ID	✓	✓	✓	✓

不支持的组合会抛出

UnsupportedInputError

并给出明确提示。

Gemini的多模态支持范围最广。如果不确定为多模态任务选择哪个服务商，建议从Gemini开始。

Provider-specific niceties

各服务商专属特性

Gemini — YouTube URLs work directly

Gemini — 直接支持YouTube URL

python

from ag2.events import VideoInput

video = VideoInput("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
reply = await agent.ask("Summarize this video.", video)

python

from ag2.events import VideoInput

video = VideoInput("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
reply = await agent.ask("Summarize this video.", video)

Gemini — large files (> 20MB) via Google Files API

Gemini — 大文件(>20MB)需通过Google Files API上传

python

from google import genai
from ag2.events import VideoInput
import time

client = genai.Client()
uploaded = client.files.upload(file="large_video.mp4")
while uploaded.state.name == "PROCESSING":
    time.sleep(2)
    uploaded = client.files.get(name=uploaded.name)

video = VideoInput(uploaded.uri)

python

from google import genai
from ag2.events import VideoInput
import time

client = genai.Client()
uploaded = client.files.upload(file="large_video.mp4")
while uploaded.state.name == "PROCESSING":
    time.sleep(2)
    uploaded = client.files.get(name=uploaded.name)

video = VideoInput(uploaded.uri)

Gemini —

vendor_metadata

Gemini —

vendor_metadata

参数

Key	Purpose
`media_resolution`	`MEDIA_RESOLUTION_LOW/MEDIUM/HIGH/ULTRA_HIGH` — token vs cost
`video_metadata`	Clipping ( `start_offset` , `end_offset` ) and `fps`
`display_name`	Display name for the file

python

ImageInput(data=raw, media_type="image/jpeg", vendor_metadata={"media_resolution": "MEDIA_RESOLUTION_LOW"})

VideoInput(path="lecture.mp4", vendor_metadata={
    "video_metadata": {"start_offset": "60s", "end_offset": "120s", "fps": 0.5},
})

键名	用途
`media_resolution`	`MEDIA_RESOLUTION_LOW/MEDIUM/HIGH/ULTRA_HIGH` — 平衡令牌消耗与成本
`video_metadata`	视频剪辑( `start_offset` 、 `end_offset` )与帧率 `fps` 设置
`display_name`	文件的显示名称

python

ImageInput(data=raw, media_type="image/jpeg", vendor_metadata={"media_resolution": "MEDIA_RESOLUTION_LOW"})

VideoInput(path="lecture.mp4", vendor_metadata={
    "video_metadata": {"start_offset": "60s", "end_offset": "120s", "fps": 0.5},
})

OpenAI — image detail

OpenAI — 图像细节设置

python

ImageInput(data=raw, media_type="image/png", vendor_metadata={"detail": "low"})  # "low" | "high" | "auto"

python

ImageInput(data=raw, media_type="image/png", vendor_metadata={"detail": "low"})  # "low" | "high" | "auto"

Anthropic — File ID + prompt caching

Anthropic — 文件ID + 提示缓存

python

import anthropic
from ag2.events import ImageInput, DocumentInput

client = anthropic.Anthropic()
uploaded = client.beta.files.upload(file=("photo.jpg", open("photo.jpg", "rb"), "image/jpeg"))

python

import anthropic
from ag2.events import ImageInput, DocumentInput

client = anthropic.Anthropic()
uploaded = client.beta.files.upload(file=("photo.jpg", open("photo.jpg", "rb"), "image/jpeg"))

filename determines block type (image vs document)

文件名决定块类型(图片 vs 文档)

image = ImageInput(file_id=uploaded.id, filename="photo.jpg")

Cache an attachment so subsequent turns skip re-uploading

缓存附件，后续对话无需重新上传

doc = DocumentInput(path="report.pdf", vendor_metadata={"cache_control": {"type": "ephemeral"}})

undefined

doc = DocumentInput(path="report.pdf", vendor_metadata={"cache_control": {"type": "ephemeral"}})

undefined

FilesAPI

— upload lifecycle, provider-agnostic

FilesAPI

— 跨服务商的上传生命周期管理

For any provider that has a file API (

OpenAIConfig

OpenAIResponsesConfig

AnthropicConfig

GeminiConfig

python

from ag2 import FilesAPI
from ag2.config import OpenAIResponsesConfig

files = FilesAPI(OpenAIResponsesConfig(model="gpt-5-mini"))

uploaded = await files.upload(path="report.pdf", purpose="assistants")
print(uploaded.file_id)

对于支持文件API的服务商(

OpenAIConfig

、

OpenAIResponsesConfig

、

AnthropicConfig

、

GeminiConfig

)：

python

from ag2 import FilesAPI
from ag2.config import OpenAIResponsesConfig

files = FilesAPI(OpenAIResponsesConfig(model="gpt-5-mini"))

uploaded = await files.upload(path="report.pdf", purpose="assistants")
print(uploaded.file_id)

Or from bytes (filename required)

或从字节数据上传(需指定文件名)

uploaded = await files.upload(data=b"...", filename="hello.txt", purpose="assistants")

List, read, delete

列出、读取、删除文件

all_files = await files.list() data = await files.read(uploaded.file_id) # NotImplementedError on Gemini await files.delete(uploaded.file_id)


Pass the `file_id` to `DocumentInput`, `ImageInput`, etc.:

```python
from ag2.events import DocumentInput

doc = DocumentInput(file_id=uploaded.file_id)
reply = await agent.ask("Summarize this report.", doc)

all_files = await files.list() data = await files.read(uploaded.file_id) # Gemini不支持此方法，会抛出NotImplementedError await files.delete(uploaded.file_id)


将`file_id`传入`DocumentInput`、`ImageInput`等：

```python
from ag2.events import DocumentInput

doc = DocumentInput(file_id=uploaded.file_id)
reply = await agent.ask("Summarize this report.", doc)

Going deeper

深入学习

website/docs/user-guide/multimodal/inputs.mdx

— full provider matrix and

vendor_metadata

reference.

website/docs/user-guide/advanced/files.mdx

—

FilesAPI

reference (upload / list / read / delete).

For tools that return images / binary back to the LLM, see
```
ag2-add-custom-tool
```
(
```
ImageInput
```
,
```
BinaryInput
```
,
```
ToolResult
```
).

website/docs/user-guide/multimodal/inputs.mdx

— 完整的服务商支持矩阵与

vendor_metadata

参考文档。

website/docs/user-guide/advanced/files.mdx

—

FilesAPI

参考文档(上传/列出/读取/删除)。

若需工具向LLM返回图片/二进制数据，请查看
```
ag2-add-custom-tool
```
(涉及
```
ImageInput
```
、
```
BinaryInput
```
、
```
ToolResult
```
)。

Common pitfalls

常见陷阱

Picking a provider that doesn't support your input type — silently you'll get
```
UnsupportedInputError
```
. Check the matrix; Gemini is broadest.
FilesAPI.read()
on Gemini — raises
```
NotImplementedError
```
. Gemini doesn't expose download.
Calling
files.upload(data=...)
without
filename=
— raises
```
ValueError
```
. Filename is required for in-memory uploads.
Supplying more than one source to a factory — not an error. The factory resolves in priority order
```
url
```
>
```
file_id
```
>
```
path
```
>
```
data
```
, so extra sources are silently ignored. Pass exactly one to get what you intend. Supplying zero sources raises
```
ValueError
```
.
Anthropic
ImageInput(file_id=...)
without
filename=
— Anthropic decides block type (image vs document) by filename extension. Pass it.
Gemini
vendor_metadata
keys are nested —
```
video_metadata
```
itself takes a dict. Check the doc table for shape.
Forgetting to wait for Gemini file processing — large uploads have a
```
PROCESSING
```
state. Poll
```
client.files.get(name=...)
```
until ready before referencing the URI.

选择不支持目标输入类型的服务商 — 会静默抛出
```
UnsupportedInputError
```
。请查看支持矩阵；Gemini的支持范围最广。
在Gemini上调用
FilesAPI.read()
— 会抛出
```
NotImplementedError
```
。Gemini不提供下载接口。
调用
files.upload(data=...)
时未指定
filename=
— 会抛出
```
ValueError
```
。内存上传必须指定文件名。
向工厂类传入多个数据来源 — 不会报错，但工厂类会按优先级
```
url
```
>
```
file_id
```
>
```
path
```
>
```
data
```
解析，多余的来源会被静默忽略。为确保预期效果，请仅传入一个来源。若未传入任何来源，会抛出
```
ValueError
```
。
Anthropic的
ImageInput(file_id=...)
未指定
filename=
— Anthropic通过文件扩展名判断块类型(图片 vs 文档)，请务必传入文件名。
Gemini的
vendor_metadata
参数为嵌套结构 —
```
video_metadata
```
本身需要传入字典。请参考文档表格中的结构。
忘记等待Gemini文件处理完成 — 大文件上传后会处于
```
PROCESSING
```
状态。在引用URI前，需轮询
```
client.files.get(name=...)
```
直到状态就绪。

ag2-multimodal-input

Original

Translation

Multimodal inputs

多模态输入

When to use

使用场景

60-second recipe

60秒快速上手

Input factories

输入工厂类

Provider matrix

服务商支持矩阵

Provider-specific niceties

各服务商专属特性

Gemini — YouTube URLs work directly

Gemini — 直接支持YouTube URL

Gemini — large files (> 20MB) via Google Files API

Gemini — 大文件(>20MB)需通过Google Files API上传

Gemini —
`vendor_metadata`

Gemini —
`vendor_metadata`
参数

OpenAI — image detail

OpenAI — 图像细节设置

Anthropic — File ID + prompt caching

Anthropic — 文件ID + 提示缓存

filename determines block type (image vs document)

文件名决定块类型(图片 vs 文档)

Cache an attachment so subsequent turns skip re-uploading

缓存附件，后续对话无需重新上传

`FilesAPI`
— upload lifecycle, provider-agnostic

`FilesAPI`
— 跨服务商的上传生命周期管理

Or from bytes (filename required)

或从字节数据上传(需指定文件名)

List, read, delete

列出、读取、删除文件

Going deeper

深入学习

Common pitfalls

常见陷阱

ag2-multimodal-input

Original

Translation

Multimodal inputs

多模态输入

When to use

使用场景

60-second recipe

60秒快速上手

Input factories

输入工厂类

Provider matrix

服务商支持矩阵

Provider-specific niceties

各服务商专属特性

Gemini — YouTube URLs work directly

Gemini — 直接支持YouTube URL

Gemini — large files (> 20MB) via Google Files API

Gemini — 大文件(>20MB)需通过Google Files API上传

Gemini — vendor_metadata

Gemini — vendor_metadata参数

OpenAI — image detail

OpenAI — 图像细节设置

Anthropic — File ID + prompt caching

Anthropic — 文件ID + 提示缓存

filename determines block type (image vs document)

文件名决定块类型(图片 vs 文档)

Cache an attachment so subsequent turns skip re-uploading

缓存附件，后续对话无需重新上传

FilesAPI — upload lifecycle, provider-agnostic

FilesAPI — 跨服务商的上传生命周期管理

Or from bytes (filename required)

或从字节数据上传(需指定文件名)

List, read, delete

列出、读取、删除文件

Going deeper

深入学习

Common pitfalls

常见陷阱

Gemini —
`vendor_metadata`

Gemini —
`vendor_metadata`
参数

`FilesAPI`
— upload lifecycle, provider-agnostic

`FilesAPI`
— 跨服务商的上传生命周期管理