cli-eval

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Overview

概述

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

通过 CLI 创建和运行评估套件、实时查看基准测试进度、查看记分卡、比较模型性能，并将评估运行集成到 CI 工作流中。

Quick install

快速安装

bash

npm install -g omniroute   # or: npx omniroute
omniroute --version

bash

npm install -g omniroute   # or: npx omniroute
omniroute --version

Subcommands

子命令

eval

eval

Example:

bash

omniroute eval

示例：

bash

omniroute eval

eval suites

eval suites

Example:

bash

omniroute eval suites

示例：

bash

omniroute eval suites

eval list

eval list

Example:

bash

omniroute eval list

示例：

bash

omniroute eval list

eval get <suiteId>

eval get <suiteId>

Example:

bash

omniroute eval get <suiteId>

示例：

bash

omniroute eval get <suiteId>

eval create

eval create

Flags:

```
--file <path>
```

Example:

bash

omniroute eval create

参数：

```
--file <path>
```

示例：

bash

omniroute eval create

eval run <suiteId>

eval run <suiteId>

Flags:

```
-m, --model <id>
```
```
--combo <name>
```
```
--concurrency <n>
```
```
--tag <tag>
```
```
--watch
```

Example:

bash

omniroute eval run <suiteId>

参数：

```
-m, --model <id>
```
```
--combo <name>
```
```
--concurrency <n>
```
```
--tag <tag>
```
```
--watch
```

示例：

bash

omniroute eval run <suiteId>

eval list

eval list

Flags:

```
--suite <id>
```
```
--status <s>
```
```
--since <ts>
```
```
--limit <n>
```

Example:

bash

omniroute eval list

参数：

```
--suite <id>
```
```
--status <s>
```
```
--since <ts>
```
```
--limit <n>
```

示例：

bash

omniroute eval list

eval get <runId>

eval get <runId>

Example:

bash

omniroute eval get <runId>

示例：

bash

omniroute eval get <runId>

eval results <runId>

eval results <runId>

Flags:

```
--failed
```

Example:

bash

omniroute eval results <runId>

参数：

```
--failed
```

示例：

bash

omniroute eval results <runId>

eval cancel <runId>

eval cancel <runId>

Flags:

```
--yes
```

Example:

bash

omniroute eval cancel <runId>

参数：

```
--yes
```

示例：

bash

omniroute eval cancel <runId>

eval scorecard <runId>

eval scorecard <runId>

Example:

bash

omniroute eval scorecard <runId>

示例：

bash

omniroute eval scorecard <runId>

simulate [prompt]

simulate [prompt]

Flags:

```
--file <path>
```
```
-m, --model <id>
```
```
--combo <name>
```
```
--reasoning-effort <level>
```
```
--thinking-budget <n>
```
```
--explain
```

Example:

bash

omniroute simulate [prompt]

参数：

```
--file <path>
```
```
-m, --model <id>
```
```
--combo <name>
```
```
--reasoning-effort <level>
```
```
--thinking-budget <n>
```
```
--explain
```

示例：

bash

omniroute simulate [prompt]

OmniRoute — CLI Evals

OmniRoute — CLI 评估

Requires the

omniroute

CLI. See CLI entry-point skill for install + global flags.

需要使用

omniroute

CLI。有关安装和全局参数，请参阅 CLI 入口点 skill。

What are evals?

什么是评估？

Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.

评估（eval）是自动化测试套件，用于根据预期答案或评分标准对 LLM 输出进行评分。OmniRoute 将套件和运行结果存储在其本地数据库中。

Eval suites

评估套件

bash

omniroute eval suites list                       # List all eval suites
omniroute eval suites list --json                # JSON output

omniroute eval suites get <suiteId>              # Full suite definition

bash

omniroute eval suites list                       # List all eval suites
omniroute eval suites list --json                # JSON output

omniroute eval suites get <suiteId>              # Full suite definition

Create a suite

创建套件

bash

omniroute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}

Rubric options:

exact-match

contains

llm-judge

regex

--samples-file

format (one JSON object per line):

jsonl

{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}

bash

omniroute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}

评分标准（rubric）选项：

exact-match

、

contains

、

llm-judge

、

regex

。

--samples-file

格式（每行一个 JSON 对象）：

jsonl

{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}

Run an eval

运行评估

bash

omniroute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

omniroute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)

The run is asynchronous. Use

--watch

for a live terminal dashboard or poll manually:

bash

RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
omniroute eval get $RUN_ID

bash

omniroute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

omniroute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)

运行过程是异步的。使用

--watch

可查看实时终端仪表板，也可以手动轮询：

bash

RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
omniroute eval get $RUN_ID

Manage runs

管理运行

bash

omniroute eval list                              # List all eval runs
omniroute eval list --json

omniroute eval get <runId>                       # Run details (status, model, score)
omniroute eval results <runId>                   # Per-sample results
omniroute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
omniroute eval cancel <runId>                    # Cancel a running eval

bash

omniroute eval list                              # List all eval runs
omniroute eval list --json

omniroute eval get <runId>                       # Run details (status, model, score)
omniroute eval results <runId>                   # Per-sample results
omniroute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
omniroute eval cancel <runId>                    # Cancel a running eval

Scorecard output

记分卡输出

bash

omniroute eval scorecard <runId> --output json

Response fields per sample:

json

{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}

bash

omniroute eval scorecard <runId> --output json

每个样本的响应字段：

json

{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}

Comparing models

模型对比

Run the same suite against multiple models and compare:

bash

for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done

对多个模型运行同一套件并进行比较：

bash

for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done

CI integration

CI 集成

bash

undefined

bash

undefined

Run and fail CI if score drops below threshold

SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score') python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"

undefined

SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score') python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"

undefined

Errors

错误

suites create

fails with

invalid rubric

→ use one of:

exact-match

contains

llm-judge

regex

suites run

returns

model not found

→ verify model ID with

omniroute models --search <name>

eval get

shows

status: failed

→ check

omniroute logs --search eval

for error details

```
scorecard
```
returns empty results → the run may still be
```
running
```
; poll
```
omniroute eval get <runId>
```
until
```
status
```
is
```
completed
```

suites create

失败并提示

invalid rubric

→ 使用以下选项之一：

exact-match

、

contains

、

llm-judge

、

regex

suites run

model not found

→ 使用

omniroute models --search <name>

验证模型 ID

eval get

显示

status: failed

→ 使用

omniroute logs --search eval

查看错误详情

```
scorecard
```
返回空结果 → 运行可能仍处于
```
running
```
状态；轮询
```
omniroute eval get <runId>
```
，直至
```
status
```
变为
```
completed
```