opencli-explorer

Original🇨🇳 Chinese
Translated

Use when creating a new OpenCLI adapter from scratch, adding support for a new website or platform, or exploring a site's API endpoints via browser DevTools. Covers API discovery workflow, authentication strategy selection, YAML/TS adapter writing, and testing.

148installs
Added on

NPX Install

npx skill4agent add jackwener/opencli opencli-explorer

SKILL.md Content (Chinese)

View Translation Comparison →

CLI-EXPLORER — Complete Guide to Exploratory Adapter Development

This document teaches you (or an AI Agent) how to add commands for a new website to OpenCLI.
From scratch to release, it covers the full process of API discovery, solution selection, adapter writing, and testing & verification.
[!TIP] Just want to quickly generate a command for a specific page? Check CLI-ONESHOT.md (~150 lines, 4 steps to complete). This document applies to the full process of exploring a new site from scratch.

Must-read for AI Agent Developers: Explore with Browsers

[!CAUTION] You (AI Agent) must open the target website via a browser to explore!
Do not rely solely on the
opencli explore
command or static analysis to discover APIs.
You have browser tools available, you must actively use them to browse web pages, observe network requests, and simulate user interactions.

Why?

Many APIs are lazy-loaded (network requests are only triggered when the user clicks a certain button/tab). Deep data such as subtitles, comments, and follow lists will not appear in the Network panel when the page first loads. If you do not actively browse and interact with the page, you will never discover these APIs.

AI Agent Exploration Workflow (Must Follow)

StepToolWhat to Do
0. Open browser
browser_navigate
Navigate to the target page
1. Observe page
browser_snapshot
Observe interactive elements (buttons/tabs/links)
2. First packet capture
browser_network_requests
Filter JSON API endpoints, record URL pattern
3. Simulate interaction
browser_click
+
browser_wait_for
Click buttons like "subtitles", "comments", "follow"
4. Second packet capture
browser_network_requests
Compare with step 2, find newly triggered APIs
5. Verify API
browser_evaluate
fetch(url, {credentials:'include'})
test return structure
6. Write codeWrite adapter based on confirmed API

Common Mistakes

❌ Wrong Practice✅ Correct Practice
Only use
opencli explore
command, wait for results to come out automatically
Open the page with browser tools, browse actively
Directly
fetch(url)
in code without checking actual browser requests
Confirm API is available in browser first, then write code
Capture packets directly after page opens, expect all APIs to appearSimulate click interactions (expand comments/switch tabs/load more)
Give up when encountering HTTP 200 with empty dataCheck if Wbi signature or Cookie authentication is required
Fully rely on
__INITIAL_STATE__
to get all data
__INITIAL_STATE__
only has first screen data, deep data needs API calls

Practical Success Case: Implement "Follow List" Adapter in 5 Minutes

The following is the complete process of actually discovering the Bilibili follow list API using the above workflow:
1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
2. browser_network_requests → Discovered:
   GET /x/relation/followings?vmid={uid}&pn=1&ps=24  →  [200]
   GET /x/relation/stat?vmid={uid}                    →  [200]
3. browser_evaluate → Verify API:
   fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
   → { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
4. Conclusion: Standard Cookie API, no Wbi signature required
5. Write following.ts → Build passes in one go
Key Decision Points:
  • Directly access the
    fans/follow
    page (not the homepage), the following API will be triggered as soon as the page loads
  • No
    /wbi/
    in the URL → no signature required → use
    fetchJson
    directly instead of
    apiGet
  • API returns
    code: 0
    + non-empty
    list
    → Tier 2 Cookie strategy confirmed

Core Workflow

 ┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. API Discovery │ ──▶ │ 2. Select Strategy │ ──▶ │ 3. Write Adapter │ ──▶ │ 4. Test │
 └─────────────┘     └─────────────┘     └──────────────┘     └────────┘
   explore             cascade             YAML / TS            run + verify

Step 1: API Discovery

1a. Automated Discovery (Recommended)

OpenCLI has built-in Deep Explore, which automatically analyzes website network requests:
bash
opencli explore https://www.example.com --site mysite
Output to
.opencli/explore/mysite/
:
FileContent
manifest.json
Site metadata, framework detection (Vue2/3, React, Next.js, Pinia, Vuex)
endpoints.json
Discovered API endpoints, sorted by score, including URL pattern, method, response type
capabilities.json
Inferred functions (
hot
,
search
,
feed
…), including confidence and recommended parameters
auth.json
Authentication method detection (Cookie/Header/no authentication), candidate strategy list

1b. Manual Packet Capture Verification

Explore's automatic analysis may not be perfect, use verbose mode for manual confirmation:
bash
# Open the target page in the browser, observe network requests
opencli explore https://www.example.com --site mysite -v

# Or directly use evaluate to test API
opencli bilibili hot -v   # View data flow of each step of the existing command pipeline
Key information to pay attention to in packet capture results:
  • URL pattern:
    /api/v2/hot?limit=20
    → This is the endpoint you need to call
  • Method:
    GET
    /
    POST
  • Request Headers: Cookie? Bearer? Custom signature headers (X-s, X-t)?
  • Response Body: JSON structure, especially the path where the data is located (
    data.items
    ,
    data.list
    )

1c. Advanced API Discovery Heuristics

Before diving into complex packet capture interception, try the following methods in order of priority:
  1. Suffix brute force method (
    .json
    )
    : For complex sites like Reddit, just add
    .json
    after the URL (e.g.
    /r/all.json
    ), you can get extremely clean REST data directly with
    fetch
    when carrying cookies (Tier 2 Cookie strategy is extremely fast). In addition, fully functional sites like Xueqiu can also use this pure API method to get data very simply, making it a golden benchmark for you to build simple YAML.
  2. Global state lookup method (
    __INITIAL_STATE__
    )
    : Many Server-Side Rendered (SSR) websites (such as Xiaohongshu, Bilibili) mount the full data of the homepage or detail page to the global window object. Instead of intercepting network requests, you can directly get the entire data tree via
    page.evaluate('() => window.__INITIAL_STATE__')
    .
  3. Active interaction trigger method: Many deep APIs (such as video subtitles, replies under comments) are lazy-loaded. When you can't find data in static packet capture, try to actively click the corresponding button on the page (such as "CC", "Expand All") during the
    evaluate
    step or when manually setting breakpoints, to trigger hidden Network Fetch.
  4. Framework detection and Store Action interception: If the site uses Vue + Pinia, you can use the
    tap
    step to call the action, letting the frontend framework complete the complex authentication signature encapsulation for you.
  5. Low-level XHR/Fetch interception: Last resort, when all the above methods fail, use a TypeScript adapter for non-intrusive request capture.

1d. Framework Detection

Explore automatically detects frontend frameworks. If you need manual confirmation:
bash
# When the target website is already open
opencli evaluate "(()=>{
  const vue3 = !!document.querySelector('#app')?.__vue_app__;
  const vue2 = !!document.querySelector('#app')?.__vue__;
  const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
  const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
  return JSON.stringify({vue3, vue2, react, pinia});
})()"
Sites using Vue + Pinia (such as Xiaohongshu) can bypass signatures directly via Store Action.

Step 2: Select Authentication Strategy

OpenCLI provides 5 levels of authentication strategies. Use the
cascade
command for automatic detection:
bash
opencli cascade https://api.example.com/hot

Strategy Decision Tree

Can you get data directly with fetch(url)?
  → ✅ Tier 1: public (public API, no browser required)
  → ❌ Can you get data with fetch(url, {credentials:'include'}) carrying cookies?
       → ✅ Tier 2: cookie (most common, fetch within evaluate step)
       → ❌ → Can you get data after adding Bearer / CSRF header?
              → ✅ Tier 3: header (e.g. Twitter ct0 + Bearer)
              → ❌ → Does the site have Pinia/Vuex Store?
                     → ✅ Tier 4: intercept (Store Action + XHR interception)
                     → ❌ Tier 5: ui (UI automation, last resort)

Comparison of Each Strategy

TierStrategySpeedComplexityUse CaseExample
1
public
⚡ ~1sSimplestPublic API, no login requiredHacker News, V2EX
2
cookie
🔄 ~7sSimpleCookie authentication is sufficientBilibili, Zhihu, Reddit
3
header
🔄 ~7sMediumRequires CSRF token or BearerTwitter GraphQL
4
intercept
🔄 ~10sRelatively highRequests have complex signaturesXiaohongshu (Pinia + XHR)
5
ui
🐌 ~15s+HighestNo API, pure DOM parsingLegacy websites

Step 2.5: Preparation (Before Writing Code)

Find Templates First: Start with the Most Similar Existing Adapter

Do not write from scratch. First check what adapters already exist for the same site:
bash
ls src/clis/<site>/    # Check what already exists
cat src/clis/<site>/feed.ts   # Read the most similar one
The most efficient way is to copy the most similar adapter, then modify 3 parts:
  1. name
    → New command name
  2. API URL → The endpoint you discovered in Step 1
  3. Field mapping → Correspond to the fields of the new API

Platform SDK Cheat Sheet

Before writing a TS adapter, check if there are ready-made helper functions available for reuse for your target site:

Bilibili (
src/clis/bilibili/utils.ts
)

FunctionPurposeWhen to Use
fetchJson(page, url)
Fetch with cookie + JSON parsingNormal Cookie-tier API
apiGet(page, path, {signed, params})
API call with Wbi signatureInterface with
/wbi/
in URL
getSelfUid(page)
Get UID of current logged in user"My xxx" type commands
resolveUid(page, input)
Parse UID input by user (supports number/URL)
--uid
parameter processing
wbiSign(page, params)
Low-level Wbi signature generationUsually not used directly, already encapsulated in
apiGet
stripHtml(s)
Remove HTML tagsClean rich text fields
How to judge if
apiGet
is needed
? Check the Network request URL:
  • Contains
    /wbi/
    or
    w_rid=
    → Must use
    apiGet(..., { signed: true })
  • No → Use
    fetchJson
    directly
Other sites (Twitter, Xiaohongshu, etc.) do not have dedicated SDKs for now, just use
page.evaluate
+
fetch
directly.

Step 3: Write Adapter

YAML vs TS? Check the Decision Tree First

Does your pipeline have an evaluate step (embedded JS code)?
  → ✅ Use TypeScript (src/clis/<site>/<name>.ts), automatically dynamically registered on save
  → ❌ Pure declarative (navigate + tap + map + limit)?
       → ✅ Use YAML (src/clis/<site>/<name>.yaml), automatically registered on save
ScenarioChoiceExample
Pure fetch/select/map/limitYAML
v2ex/hot.yaml
,
hackernews/top.yaml
navigate + evaluate(fetch) + mapYAML (assess complexity)
zhihu/hot.yaml
navigate + tap + mapYAML ✅
xiaohongshu/feed.yaml
,
xiaohongshu/notifications.yaml
Has complex JS logic (Pinia state reading, conditional branches)TS
xiaohongshu/me.ts
,
bilibili/me.ts
XHR interception + signatureTS
xiaohongshu/search.ts
GraphQL / pagination / Wbi signatureTS
bilibili/search.ts
,
twitter/search.ts
Rule of Thumb: If you find that there are more than 10 lines of JS embedded in YAML, it is more maintainable to switch to TS.

Common Pattern: Pagination API

Many APIs use
pn
(page number) +
ps
(page size) for pagination. Standard processing pattern:
typescript
args: [
  { name: 'page', type: 'int', required: false, default: 1, help: 'Page number' },
  { name: 'limit', type: 'int', required: false, default: 50, help: 'Page size (max 50)' },
],
func: async (page, kwargs) => {
  const pn = kwargs.page ?? 1;
  const ps = Math.min(kwargs.limit ?? 50, 50); // Respect API ps limit
  const payload = await fetchJson(page,
    `https://api.example.com/list?pn=${pn}&ps=${ps}`
  );
  return payload.data?.list || [];
},
The
ps
limit for most sites is 20~50. Exceeding it will be silently truncated or return an error.

Method A: YAML Pipeline (Declarative, Recommended)

File path:
src/clis/<site>/<name>.yaml
, automatically registered once placed.

Tier 1 — Public API Template

yaml
# src/clis/v2ex/hot.yaml
site: v2ex
name: hot
description: V2EX Hot Topics
domain: www.v2ex.com
strategy: public
browser: false

args:
  limit:
    type: int
    default: 20

pipeline:
  - fetch:
      url: https://www.v2ex.com/api/topics/hot.json

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      replies: ${{ item.replies }}

  - limit: ${{ args.limit }}

columns: [rank, title, replies]

Tier 2 — Cookie Authentication Template (Most Used)

yaml
# src/clis/zhihu/hot.yaml
site: zhihu
name: hot
description: Zhihu Hot List
domain: www.zhihu.com

pipeline:
  - navigate: https://www.zhihu.com       # Load page first to establish session

  - evaluate: |                            # Send request in browser, automatically carries cookie
      (async () => {
        const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
          credentials: 'include'
        });
        const d = await res.json();
        return (d?.data || []).map(item => {
          const t = item.target || {};
          return {
            title: t.title,
            heat: item.detail_text || '',
            answers: t.answer_count,
          };
        });
      })()

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      heat: ${{ item.heat }}
      answers: ${{ item.answers }}

  - limit: ${{ args.limit }}

columns: [rank, title, heat, answers]
Key Point: The
fetch
inside the
evaluate
step runs in the browser page context, automatically carries
credentials: 'include'
, no need to handle cookies manually.

Advanced — With Search Parameters

yaml
# src/clis/zhihu/search.yaml
site: zhihu
name: search
description: Zhihu Search

args:
  query:
    type: str
    required: true
    positional: true
    description: Search query
  limit:
    type: int
    default: 10

pipeline:
  - navigate: https://www.zhihu.com

  - evaluate: |
      (async () => {
        const q = encodeURIComponent('${{ args.query }}');
        const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${{ args.limit }}', {
          credentials: 'include'
        });
        const d = await res.json();
        return (d?.data || [])
          .filter(item => item.type === 'search_result')
          .map(item => ({
            title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
            type: item.object?.type || '',
            author: item.object?.author?.name || '',
            votes: item.object?.voteup_count || 0,
          }));
      })()

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      type: ${{ item.type }}
      author: ${{ item.author }}
      votes: ${{ item.votes }}

  - limit: ${{ args.limit }}

columns: [rank, title, type, author, votes]

Tier 4 — Store Action Bridge (
tap
step, recommended for intercept strategy)

Suitable for Vue + Pinia/Vuex sites (such as Xiaohongshu), no need to write XHR interception code manually:
yaml
# src/clis/xiaohongshu/notifications.yaml
site: xiaohongshu
name: notifications
description: "Xiaohongshu Notifications"
domain: www.xiaohongshu.com
strategy: intercept
browser: true

args:
  type:
    type: str
    default: mentions
    description: "Notification type: mentions, likes, or connections"
  limit:
    type: int
    default: 20

columns: [rank, user, action, content, note, time]

pipeline:
  - navigate: https://www.xiaohongshu.com/notification
  - wait: 3
  - tap:
      store: notification       # Pinia store name
      action: getNotification   # Store action to call
      args:                     # Action arguments
        - ${{ args.type | default('mentions') }}
      capture: /you/            # URL pattern to capture response
      select: data.message_list # Extract sub-path from response
      timeout: 8
  - map:
      rank: ${{ index + 1 }}
      user: ${{ item.user_info.nickname }}
      action: ${{ item.title }}
      content: ${{ item.comment_info.content }}
  - limit: ${{ args.limit | default(20) }}
The
tap
step automatically completes
: Inject fetch + XHR dual interception → Find Pinia/Vuex store → Call action → Capture response matching URL → Clean up interception.
If the store or action is not found, a
hint
will be returned listing all available store actions for easy debugging.
Tap ParameterRequiredDescription
store
Pinia store name (e.g.
feed
,
search
,
notification
)
action
Store action method name
capture
URL substring match (matches network request URL)
args
Parameter array passed to action
select
Path extracted from captured JSON (e.g.
data.items
)
timeout
Timeout in seconds waiting for network response (default 5s)
framework
pinia
or
vuex
(auto detected by default)

Method B: TypeScript Adapter (Programmatic)

Suitable for scenarios that require embedded JS code to read Pinia state, XHR interception, GraphQL, pagination, complex data conversion, etc.
File path:
src/clis/<site>/<name>.ts
. The file will be dynamically scanned and registered at runtime (do not manually
import
in
index.ts
).

Tier 3 — Header Authentication (Twitter)

typescript
// src/clis/twitter/search.ts
import { cli, Strategy } from '../../registry.js';

cli({
  site: 'twitter',
  name: 'search',
  description: 'Search tweets',
  strategy: Strategy.HEADER,
  args: [{ name: 'query', required: true, positional: true }],
  columns: ['rank', 'author', 'text', 'likes'],
  func: async (page, kwargs) => {
    await page.goto('https://x.com');
    const data = await page.evaluate(`
      (async () => {
        // Extract CSRF token from Cookie
        const ct0 = document.cookie.split(';')
          .map(c => c.trim())
          .find(c => c.startsWith('ct0='))?.split('=')[1];
        if (!ct0) return { error: 'Not logged in' };

        const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
        const headers = {
          'Authorization': 'Bearer ' + decodeURIComponent(bearer),
          'X-Csrf-Token': ct0,
          'X-Twitter-Auth-Type': 'OAuth2Session',
        };

        const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
        const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
        const res = await fetch(url, { headers, credentials: 'include' });
        return await res.json();
      })()
    `);
    // ... parse data
  },
});

Tier 4 — XHR/Fetch Dual Interception (Common pattern for Twitter/Xiaohongshu)

typescript
// src/clis/xiaohongshu/user.ts
import { cli, Strategy } from '../../registry.js';

cli({
  site: 'xiaohongshu',
  name: 'user',
  description: 'Get user notes',
  strategy: Strategy.INTERCEPT,
  args: [{ name: 'id', required: true }],
  columns: ['rank', 'title', 'likes', 'url'],
  func: async (page, kwargs) => {
    await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
    await page.wait(5);

    // Low-level XHR/Fetch interception: capture all requests containing 'v1/user/posted'
    await page.installInterceptor('v1/user/posted');

    // Trigger backend API: simulate human user scrolling to the bottom 2 times
    await page.autoScroll({ times: 2, delayMs: 2000 });

    // Extract all intercepted JSON response bodies
    const requests = await page.getInterceptedRequests();
    if (!requests || requests.length === 0) return [];

    let results = [];
    for (const req of requests) {
      if (req.data?.data?.notes) {
        for (const note of req.data.data.notes) {
           results.push({
             title: note.display_title || '',
             likes: note.interact_info?.liked_count || '0',
             url: `https://explore/${note.note_id || note.id}`
           });
        }
      }
    }

    return results.slice(0, 20).map((item, i) => ({
      rank: i + 1, ...item,
    }));
  },
});
Core idea of interception: Do not construct the signature yourself, but use
installInterceptor
to hijack the site's own
XMLHttpRequest
and
fetch
, let the site send the request, and we directly extract the parsed
response.json()
at the bottom layer.
Cascading requests (e.g. BVID→CID→subtitles) full template and key points can be found in the Advanced Mode: Cascading Requests section below.

Step 4: Testing

Build pass ≠ Functional normal.
npm run build
only validates TypeScript / YAML syntax, not runtime behavior.
Each new command is only completed after actually running and confirming the output is correct.

Required Checklist

bash
# 1. Build (confirm syntax is correct)
npm run build

# 2. Confirm command is registered
opencli list | grep mysite

# 3. Actually run the command (most important!)
opencli mysite hot --limit 3 -v        # verbose to view data flow of each step
opencli mysite hot --limit 3 -f json   # JSON output to confirm complete fields

tap Step Debugging (Exclusive for intercept strategy)

Do not guess store name / action name. First explore with evaluate, then write YAML.

Step 1: List all Pinia stores

After opening the target website in the browser:
bash
opencli evaluate "(() => {
  const app = document.querySelector('#app')?.__vue_app__;
  const pinia = app?.config?.globalProperties?.\$pinia;
  return [...pinia._s.keys()];
})()"
# Output: ["user", "feed", "search", "notification", ...]

Step 2: Check store action names

Intentionally write a wrong action name, tap will return all available actions:
⚠  tap: Action not found: wrongName on store notification
💡 Available: getNotification, replyComment, getNotificationCount, reset

Step 3: Confirm capture pattern with network requests

bash
# Open the target page in the browser, view network requests
# Find the URL feature of the target API (e.g. "/you/mentions", "homefeed")

Full Workflow

 ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. Navigate  │ ──▶ │ 2. Explore store │ ──▶ │ 3. Write YAML   │ ──▶ │ 4. Test │
 │    to target page  │     │ name/action  │     │    tap step   │     │ Run verification │
 └──────────────┘     └──────────────┘     └──────────────┘     └────────┘

Verbose Mode & Output Verification

bash
opencli bilibili hot --limit 1 -v          # View data flow of each pipeline step
opencli mysite hot -f json | jq '.[0]'     # Confirm JSON can be parsed
opencli mysite hot -f csv > data.csv       # Confirm CSV can be imported

Step 5: Submit and Release

Put the file into
src/clis/<site>/
to register automatically (no manual import required for YAML or TS), then:
bash
opencli list | grep mysite                            # Confirm registration
git add src/clis/mysite/ && git commit -m "feat(mysite): add hot" && git push
Architecture Concept: OpenCLI has built-in Zero-Dependency jq data flow — all parsing is done in native JS within
evaluate
, the outer YAML uses
select
/
map
for extraction, no dependency on system
jq
binary.

Advanced Mode: Cascading Requests

When the target data requires multi-step API chained fetching (e.g.
BVID → CID → subtitle list → subtitle content
), you must use a TS adapter. YAML cannot handle this multi-step logic.

Template Code

typescript
import { cli, Strategy } from '../../registry.js';
import type { IPage } from '../../types.js';
import { apiGet } from './utils.js'; // Reuse platform SDK

cli({
  site: 'bilibili',
  name: 'subtitle',
  strategy: Strategy.COOKIE,
  args: [{ name: 'bvid', required: true }],
  columns: ['index', 'from', 'to', 'content'],
  func: async (page: IPage | null, kwargs: any) => {
    if (!page) throw new Error('Requires browser');

    // Step 1: Establish Session
    await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);

    // Step 2: Extract intermediate ID from page (__INITIAL_STATE__)
    const cid = await page.evaluate(`(async () => {
      return window.__INITIAL_STATE__?.videoData?.cid;
    })()`);
    if (!cid) throw new Error('Cannot extract CID');

    // Step 3: Call next-level API with intermediate ID (automatic Wbi signature)
    const payload = await apiGet(page, '/x/player/wbi/v2', {
      params: { bvid: kwargs.bvid, cid },
      signed: true, // ← Auto generate w_rid
    });

    // Step 4: Detect risk control degradation (null assertion)
    const subtitles = payload.data?.subtitle?.subtitles || [];
    const url = subtitles[0]?.subtitle_url;
    if (!url) throw new Error('subtitle_url is empty, suspected risk control degradation');

    // Step 5: Fetch final data (CDN JSON)
    const items = await page.evaluate(`(async () => {
      const res = await fetch(${JSON.stringify('https:' + url)});
      const json = await res.json();
      return { data: json.body || json };
    })()`);

    return items.data.map((item, idx) => ({ ... }));
  },
});

Key Points

StepNotes
Extract intermediate IDPrioritize getting from
__INITIAL_STATE__
to avoid extra API calls
Wbi signatureBilibili
/wbi/
interfaces mandatory check
w_rid
, pure
fetch
will get 403
Null assertionEven with HTTP 200, core fields may be empty strings (risk control degradation)
CDN URLUsually starts with
//
, remember to add
https:
JSON.stringify
Must use it to escape when splicing URL into evaluate to avoid injection

Common Pitfalls

PitfallSymptomSolution
Missing
navigate
evaluate throws
Target page context
error
Add
navigate:
step before evaluate
Nested field access
${{ item.node?.title }}
does not work
Flatten data in evaluate, do not use optional chaining in templates
Missing
strategy: public
Public API also starts browser, 7s → 1sAdd
strategy: public
+
browser: false
for public APIs
evaluate returns stringmap step receives
""
instead of array
Pipeline has auto-parse, but it is recommended to reshape with
.map()
in evaluate
Search parameters are URL encoded
${{ args.query }}
is double encoded by browser
Manually encode with
encodeURIComponent()
in evaluate
Cookie expiredReturns 401 / empty dataLog in to the target site again in the browser
Extension tab residualExtra
chrome-extension://
tab in Chrome
Automatically cleaned; if residual, close manually
TS evaluate format
() => {}
reports
result is not a function
page.evaluate()
in TS must use IIFE:
(async () => { ... })()
Page asynchronous loadingevaluate gets empty data (store state not updated yet)Use polling in evaluate to wait for data to appear, or increase
wait
time
YAML embedded large JSDifficult debugging, string escaping issuesCommands with more than 10 lines of JS use TS adapter instead
Risk control interception (pseudo 200)Core data in the obtained JSON is
""
(empty string)
Very easy to be misjudged. Must add assertion! If there is no core data, immediately request to upgrade authentication Tier and reconfigure Cookie
API not foundThe results scored by the
explore
tool cannot get deep data
Click page buttons to trigger lazy loaded data, then combine with
getInterceptedRequests
to obtain

Automatically Generate Adapters with AI Agent

The fastest way is to let AI Agent complete the whole process:
bash
# One-click: Explore → Analyze → Synthesize → Register
opencli generate https://www.example.com --goal "hot"

# Or execute step by step:
opencli explore https://www.example.com --site mysite           # Discover API
opencli explore https://www.example.com --auto --click "字幕,CC"  # Simulate click to trigger lazy loaded API
opencli synthesize mysite                                        # Generate candidate YAML
opencli verify mysite/hot --smoke                                # Smoke test
The generated candidate YAML is saved in
.opencli/explore/mysite/candidates/
, which can be directly copied to
src/clis/mysite/
and fine-tuned.

Record Workflow

record
is a manual recording solution for pages that "cannot be automatically discovered with
explore
" (require login operations, complex interactions, in-SPA routing).

Working Principle

opencli record <url>
  → Open automation window and navigate to target URL
  → Inject fetch/XHR interceptor to all tabs (idempotent, can be injected repeatedly)
  → Poll every 2s: automatically inject when new tab is found, drain capture buffers of all tabs
  → Stop on timeout (default 60s) or Enter press
  → Analyze captured JSON requests: deduplicate → score → generate candidate YAML
Interceptor Features:
  • Patch both
    window.fetch
    and
    XMLHttpRequest
    at the same time
  • Only capture responses with
    Content-Type: application/json
  • Filter responses with plain objects with less than 2 keys (avoid tracking/ping)
  • Cross-tab isolation: each tab has independent buffer, drained separately during polling
  • Idempotent injection: when injecting twice in the same tab, restore original functions first then re-patch, no lost captured data

Usage Steps

bash
# 1. Start recording (recommended to give enough operation time with --timeout)
opencli record "https://example.com/page" --timeout 120000

# 2. Operate the page normally in the pop-up automation window:
#    - Open lists, search, click items, switch tabs
#    - All operations that trigger network requests will be captured

# 3. Press Enter to stop after completing operations (or wait for timeout to stop automatically)

# 4. View results
cat .opencli/record/<site>/captured.json        # Raw capture
ls  .opencli/record/<site>/candidates/          # Candidate YAML

Page Types and Capture Expectations

Page TypeExpected Capture VolumeDescription
List/Search PageHigh (5~20+)Each search/page turn triggers new requests
Detail Page (Read-only)Low (1~5)First screen data is returned at once, subsequent operations go through form/redirect
In-SPA route跳转MediumRoute switching triggers new interfaces, but first screen requests are sent before injection
Pages requiring loginDepends on operationEnsure Chrome is logged into the target site
Note: If the page sends most requests before navigation is completed (Server-Side Rendering / SSR hydration), the interceptor will miss these requests. Solution: After the page is loaded, manually trigger operations that generate new requests (search, page turn, switch tabs, expand collapsed items, etc.).

Candidate YAML → TS CLI Conversion

The generated candidate YAML is a starting point, usually needs to be converted to TypeScript (especially for internal systems such as tae):
Candidate YAML Structure (auto generated):
yaml
site: tae
name: getList          # Name inferred from URL path
strategy: cookie
browser: true
pipeline:
  - navigate: https://...
  - evaluate: |
      (async () => {
        const res = await fetch('/approval/getList.json?procInsId=...', { credentials: 'include' });
        const data = await res.json();
        return (data?.content?.operatorRecords || []).map(item => ({ ... }));
      })()
Convert to TS CLI (refer to the style of
src/clis/tae/add-expense.ts
):
typescript
import { cli, Strategy } from '../../registry.js';

cli({
  site: 'tae',
  name: 'get-approval',
  description: 'View reimbursement form approval process and operation records',
  domain: 'tae.alibaba-inc.com',
  strategy: Strategy.COOKIE,
  browser: true,
  args: [
    { name: 'proc_ins_id', type: 'string', required: true, positional: true, help: 'Process instance ID (procInsId)' },
  ],
  columns: ['step', 'operator', 'action', 'time'],
  func: async (page, kwargs) => {
    await page.goto('https://tae.alibaba-inc.com/expense/pc.html?_authType=SAML');
    await page.wait(2);
    const result = await page.evaluate(`(async () => {
      const res = await fetch('/approval/getList.json?taskId=&procInsId=${kwargs.proc_ins_id}', {
        credentials: 'include'
      });
      const data = await res.json();
      return data?.content?.operatorRecords || [];
    })()`);
    return (result as any[]).map((r, i) => ({
      step: i + 1,
      operator: r.operatorName || r.userId,
      action: r.operationType,
      time: r.operateTime,
    }));
  },
});

Conversion Key Points

  1. Extract dynamic IDs in URL (
    procInsId
    ,
    taskId
    , etc.) as
    args
  2. The real body structure in
    captured.json
    is used to determine the correct data path (e.g.
    content.operatorRecords
    )
  3. Tae system uniformly uses
    { success, content, errorCode, errorMsg }
    outer wrapper, data fetching should go through
    content.*
  4. Authentication method: cookie (
    credentials: 'include'
    ), no extra header required
  5. Put the file into
    src/clis/<site>/
    , no manual registration required, automatically discovered after
    npm run build

Troubleshooting

SymptomCauseSolution
0 requests capturedInterceptor injection failed, or page has no JSON APICheck if daemon is running:
curl localhost:19825/status
Low capture volume (1~3 requests)Page is read-only detail page, first screen data was sent before injectionManually trigger more requests (search/page turn), or use list page instead
0 candidate YAMLAll captured JSON have no array structureDirectly view
captured.json
and write TS CLI manually
New tab is not interceptedTab is closed within polling intervalShorten
--poll 500
Data is not continuous when running record for the second timeNormal, each
record
starts a new automation window
No processing required