- 先评估任务,再选通道:根据「目标信息的性质、什么工具能直接拿到」决定起点,选最轻且能直达的方案。
- 确保信息的真实性,一手信息优于二手信息:搜索引擎和聚合平台是信息发现入口。当多次搜索尝试后没有质的改进时,升级到更根本的获取方式:定位一手来源(官网、官方平台、原始页面)。
| 场景 | 通道 |
|---|
| 只需搜索摘要或关键词结果,或需要发现信息来源 | WebSearch |
| URL 已知,读取页面内容 | Jina(默认,底层执行 JS 渲染);需要读取 HTML 源码中的结构化字段(meta、JSON-LD 等)时改用 WebFetch(不执行 JS 渲染) |
| URL 是 PDF | Jina |
| 非公开内容,或已知静态层无效的平台(小红书、微信公众号等公开内容也被反爬限制) | 浏览器 CDP(直接,跳过静态层) |
| 需要动态内容、登录态、交互操作,或需要像人一样在浏览器内自由导航探索 | 浏览器 CDP |
浏览器 CDP 不要求 URL 已知——可从任意入口出发,通过页面内搜索、点击、跳转等方式找到目标内容。
Jina:调用方式为
(URL 前加前缀,不保留原网址 http 前缀)。限 20 RPM,更节省 AI 上下文。
底层用 Puppeteer 渲染页面(能处理 JS/SPA),再用 Readability 算法提取主文章内容转为 Markdown,会过滤导航、广告、侧边栏等噪声。适合文章、博客、文档、PDF 等以正文为核心的页面;对视频页、数据面板、商品页等非文章结构页面,可能提取到错误区块。拿到结果后判断内容是否符合任务预期,不符合则切换访问策略。
WebFetch:直接获取原始 HTML,不执行 JS。meta、JSON-LD 等结构化字段通常由服务端静态嵌入 HTML,WebFetch 可直接读取;若字段由 JS 动态注入,WebFetch 同样拿不到。请求时加 header
Accept: text/markdown, text/html
,支持该协议的网站直接返回 Markdown。
选择逻辑:默认用 Jina——Jina 底层执行 JS,能处理动态渲染页面。只有当任务可能需要读取 HTML 源码时,才用 WebFetch。Jina 和 WebFetch 均无法处理时(无法获取所需信息、报错、需登录)→ 升级浏览器层。
降级禁止:进入更重的通道后,不得回头用轻量工具完成同一目标——等同于重走已知不通的路。浏览器层遇到阻碍应在层内解决(如处理登录),而不是绕回。唯一例外:浏览器操作中衍生的新子目标,可重新选择通道。
进入浏览器层后,区分任务性质:
- 操作型(导航、填表、点击):用 accessibility tree 感知界面,无法识别时才截图辅助
- 内容型(读帖子、看资讯、分析页面):accessibility tree 读文字结构,同时判断图片是否承载核心信息——是则提取图片 URL 定向读取
图片判断:社交媒体、图文博客、截图类内容,默认图片有价值,主动去取;工具类、导航类页面,默认 accessibility tree 够用。
- Evaluate the task first, then choose the channel: Decide the starting point based on "the nature of the target information, which tool can directly obtain it", and select the lightest and most direct solution.
- Ensure information authenticity, first-hand information is better than second-hand: Search engines and aggregation platforms are information discovery entrances. When multiple search attempts don't bring qualitative improvement, upgrade to a more fundamental acquisition method: locate first-hand sources (official websites, official platforms, original pages).
| Scenario | Channel |
|---|
| Only need search summaries or keyword results, or need to discover information sources | WebSearch |
| URL is known, read page content | Jina (default, executes JS rendering at the bottom layer); use WebFetch (does not execute JS rendering) instead when needing to read structured fields in HTML source code (meta, JSON-LD, etc.) |
| URL is a PDF | Jina |
| Non-public content, or platforms where static layer is known to be ineffective (public content on Xiaohongshu, WeChat Official Accounts, etc. is also restricted by anti-scraping) | Browser CDP (direct, skips static layer) |
| Need dynamic content, login state, interactive operations, or need to freely navigate and explore in the browser like a human | Browser CDP |
Browser CDP does not require a known URL — it can start from any entry point, find target content through in-page search, clicking, jumping, etc.
Jina: Called in the format
(add prefix before URL, do not retain the original http prefix). Limited to 20 RPM, saves AI context more efficiently.
It uses Puppeteer to render pages (can handle JS/SPA) at the bottom layer, then uses the Readability algorithm to extract main article content and convert it to Markdown, filtering out noise such as navigation, ads, and sidebars. Suitable for pages centered on main text such as articles, blogs, documents, PDFs; for non-article structured pages such as video pages, data panels, product pages, it may extract incorrect blocks. After obtaining the result, judge whether the content meets the task expectations, switch the access strategy if not.
WebFetch: Directly obtains original HTML without executing JS. Structured fields such as meta and JSON-LD are usually statically embedded in HTML by the server, and WebFetch can read them directly; if the fields are dynamically injected by JS, WebFetch cannot obtain them either. Add header
Accept: text/markdown, text/html
when requesting, and websites supporting this protocol will return Markdown directly.
Selection Logic: Use Jina by default — Jina executes JS at the bottom layer and can handle dynamically rendered pages. Only use WebFetch when the task may require reading HTML source code. When neither Jina nor WebFetch can handle it (cannot obtain required information, error occurs, login required) → upgrade to browser layer.
Downgrade Forbidden: After entering a heavier channel, do not go back to use lightweight tools to complete the same target — it is equivalent to retracing a known blocked path. When encountering obstacles in the browser layer, solve them within the layer (such as handling login), instead of going back. The only exception: new sub-goals derived from browser operations can re-select channels.
After entering the browser layer, distinguish task types:
- Operational (navigation, form filling, clicking): Use accessibility tree to perceive the interface, only use screenshots for assistance when recognition fails
- Content-based (reading posts, viewing news, analyzing pages): Use accessibility tree to read text structure, and simultaneously judge whether images carry core information — extract image URLs for targeted reading if yes
Image Judgment: For social media, graphic blogs, screenshot-based content, images are considered valuable by default, actively fetch them; for tool-based, navigation-based pages, accessibility tree is considered sufficient by default.