evaluations

Original：🇺🇸 English

Translated

Compatibility router for LangWatch evaluation requests. Use only when the user asks for evaluations without making it clear whether they mean pre-deployment experiments or production online evaluations. Routes the request to the focused companion skill and does not implement either workflow itself.

3installs

Sourcelangwatch/skills

Added on2026-07-27

NPX Install

npx skill4agent add langwatch/skills evaluations

SKILL.md Content

View Translation Comparison →

Route an Evaluation Request

This is a compatibility skill. Do not build an experiment, monitor, or guardrail from this skill.

Classify the user's intent:

Intent	Correct skill
Batch test a dataset, compare prompts or models, benchmark, create a CI quality gate	`experiments`
Score live traces or threads, monitor production quality, create a guardrail	`online-evaluations`

If the request remains ambiguous after inspecting context — a bare "make me an eval" that names neither a dataset nor live traffic — do not create anything yet. This choice picks what gets tested, so it is the user's to make, not a default's. Ask it as a question card and stop; the answer arrives as the next message.

Where

langy-card

blocks render, ask it as a

choices

block — the only sanctioned question format — last in the reply:

markdown

```langy-card
{
  "kind": "choices",
  "blockId": "eval-kind",
  "question": "What should this evaluate?",
  "options": [
    { "id": "experiment", "label": "A dataset, before deployment" },
    { "id": "online", "label": "Live production traffic" }
  ]
}
```

Neither option names an existing entity, so neither carries a

ref

. Without that channel, ask the same question as one short line of prose.

A rejected field value is not this kind of choice. If a create later fails with a

validation_error

whose reason names the field and an

expected

list, correct that exact field from the list and retry once — never turn a fixable slug into a question for the user.

Then hand off:

If the correct companion skill is available, load it and follow it instead of continuing here.

If

experiments

is missing, tell the user to install it with:

bash

npx skills@1.5.19 add langwatch/skills/experiments

If

online-evaluations