evaluate-multimodal

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Evaluate Your Multimodal Agent

评估您的多模态 Agent

This recipe helps you evaluate agents that process images, audio, PDFs, or other non-text inputs.

本指南将帮助您评估处理图像、音频、PDF 或其他非文本输入的 Agent。

Step 1: Identify Modalities

步骤 1：确定模态

Read the codebase to understand what your agent processes:

Images: classification, analysis, generation, OCR
Audio: transcription, voice agents, audio Q&A
PDFs/Documents: parsing, extraction, summarization
Mixed: multiple input types in one pipeline

阅读代码库，了解您的 Agent 会处理哪些内容：

图像：分类、分析、生成、OCR
音频：转录、语音 Agent、音频问答
PDF/文档：解析、信息提取、摘要生成
混合模态：在同一条流水线中处理多种输入类型

Step 2: Read the Relevant Docs

步骤 2：阅读相关文档

Use the

langwatch

CLI to fetch the right pages:

bash

langwatch scenario-docs                            # Index: locate multimodal pages
langwatch scenario-docs multimodal/audio-to-text   # Audio testing patterns
langwatch scenario-docs multimodal/multimodal-files # Generic file analysis patterns
langwatch docs                                     # LangWatch docs index
langwatch docs evaluations/experiments/sdk         # Experiment SDK basics
langwatch docs evaluations/evaluators/list         # Browse evaluator types

For PDF evaluation specifically, reference the pattern from

sdks/python/examples/pdf_parsing_evaluation.ipynb

Download/load documents
Define extraction pipeline
Use LangWatch experiment SDK to evaluate extraction accuracy

使用

langwatch

CLI 获取所需页面：

bash

langwatch scenario-docs                            # 索引：定位多模态页面
langwatch scenario-docs multimodal/audio-to-text   # 音频测试模式
langwatch scenario-docs multimodal/multimodal-files # 通用文件分析模式
langwatch docs                                     # LangWatch 文档索引
langwatch docs evaluations/experiments/sdk         # Experiment SDK 基础用法
langwatch docs evaluations/evaluators/list         # 浏览评估器类型

对于 PDF 评估，请参考

sdks/python/examples/pdf_parsing_evaluation.ipynb

中的模式：

下载或加载文档
定义提取流水线
使用 LangWatch Experiment SDK 评估提取准确性

Step 3: Set Up Evaluation by Modality

步骤 3：按模态搭建评估流程

Image Evaluation

图像评估

LangWatch's LLM-as-judge evaluators can accept images. Create an evaluation that:

Loads test images
Runs the agent on each image
Uses an LLM-as-judge evaluator to assess output quality

python

import langwatch

experiment = langwatch.experiment.init("image-eval")

for idx, entry in experiment.loop(enumerate(image_dataset)):
    result = my_agent(image=entry["image_path"])
    experiment.evaluate(
        "llm_boolean",
        index=idx,
        data={
            "input": entry["image_path"],  # LLM-as-judge can view images
            "output": result,
        },
        settings={
            "model": "openai/gpt-5-mini",
            "prompt": "Does the agent correctly describe/classify this image?",
        },
    )

LangWatch 的 LLM-as-judge 评估器可以接收图像。创建一个符合以下要求的评估流程：

加载测试图像
对每张图像运行 Agent
使用 LLM-as-judge 评估器评估输出质量

python

import langwatch

experiment = langwatch.experiment.init("image-eval")

for idx, entry in experiment.loop(enumerate(image_dataset)):
    result = my_agent(image=entry["image_path"])
    experiment.evaluate(
        "llm_boolean",
        index=idx,
        data={
            "input": entry["image_path"],  # LLM-as-judge can view images
            "output": result,
        },
        settings={
            "model": "openai/gpt-5-mini",
            "prompt": "Does the agent correctly describe/classify this image?",
        },
    )

Audio Evaluation

音频评估

Use Scenario's audio testing patterns:

Audio-to-text: verify transcription accuracy
Audio-to-audio: verify voice agent responses

Read the dedicated guide:

bash

langwatch scenario-docs multimodal/audio-to-text

使用 Scenario 的音频测试模式：

音频转文本：验证转录准确性
音频到音频：验证语音 Agent 的响应

阅读专门指南：

bash

langwatch scenario-docs multimodal/audio-to-text

PDF/Document Evaluation

PDF/文档评估

Follow the pattern from the PDF parsing evaluation example:

Load documents (PDFs, CSVs, etc.)
Define extraction/parsing pipeline
Evaluate extraction accuracy against expected fields
Use structured evaluation (exact match for fields, LLM judge for summaries)

遵循 PDF 解析评估示例中的模式：

加载文档（PDF、CSV 等）
定义提取或解析流水线
根据预期字段评估提取准确性
使用结构化评估（字段精确匹配、摘要使用 LLM 评审器）

File Analysis

文件分析

For agents that process arbitrary files, read the file analysis guide:

bash

langwatch scenario-docs multimodal/multimodal-files

对于处理任意文件的 Agent，请阅读文件分析指南：

bash

langwatch scenario-docs multimodal/multimodal-files

Step 4: Generate Domain-Specific Test Data

步骤 4：生成特定领域的测试数据

For each modality, generate or collect test data that matches the agent's actual use case:

If it's a medical imaging agent → use relevant medical image samples
If it's a document parser → use real document types the agent encounters
If it's a voice assistant → record realistic voice prompts

对于每种模态，请生成或收集与 Agent 实际使用场景相匹配的测试数据：

如果是医学影像 Agent → 使用相关医学图像样本
如果是文档解析器 → 使用 Agent 实际会遇到的真实文档类型
如果是语音助手 → 录制逼真的语音提示

Step 5: Run and Iterate

步骤 5：运行并迭代

Run the evaluation, review results, fix issues, re-run until quality is acceptable.

运行评估、审查结果、修复问题，然后重新运行，直至质量达到可接受水平。

Common Mistakes

常见错误

Do NOT evaluate multimodal agents with text-only metrics. Use image-aware judges
Do NOT skip testing with real file formats. Synthetic descriptions aren't enough
Do NOT forget to handle file loading errors in evaluations
Do NOT use generic test images. Use domain-specific ones matching the agent's purpose
Always read the relevant
```
langwatch scenario-docs ...
```
page for the modality before writing code; multimodal patterns differ a lot from text-only ones

不要使用纯文本指标评估多模态 Agent。应使用能够识别图像的评审器
不要跳过使用真实文件格式的测试。仅有合成描述是不够的
不要忽略处理评估过程中的文件加载错误
不要使用通用测试图像。应使用与 Agent 用途相匹配的特定领域图像
编写代码前，务必阅读对应模态的
```
langwatch scenario-docs ...
```
页面；多模态模式与纯文本模式存在很大差异