ae-evaluation

Use

ae-cli evaluation +<command> [options]

for existing operations and the resource commands below for new capabilities. All 121 commands target the Agent Evaluation platform. AE Agent resource management belongs to

ae-agent

; this skill manages evaluation assets and agents under test.

Read only the relevant reference

Before executing a command, read the reference for the operation. Do not load all references. References contain command inputs, JSON contracts, and task-specific workflows; use command

--help

for additional flags.

Task	Reference
Create/import datasets, manage fields and cases, publish versions	datasets.md
Configure LLM judges/templates, publish evaluator versions, calibrate or optimize prompts	evaluators.md
Create/configure/start/retry/cancel runs, inspect results/traces, export results	runs.md
Configure/test models and agents, manage credentials	connections.md
Change resource visibility or administer project members	access-and-members.md
Download imported/exported files to disk	files.md
Create/update/delete recurring evaluation schedules	schedules.md
Connect/import AI Monitor data, configure TE reporting, discover TE Claude catalog	integrations.md

Endpoint and authentication

In both personal terminals and AE sandboxes, evaluation commands default to the current AE Host and call
```
<current-host>/agenteval/api/v1/...
```
(an optional trailing
```
/agent
```
is removed from the host). Use
```
--host
```
to select another AE Host. Evaluation requests use the HTTP or HTTPS scheme configured for that AE Host, consistent with other AE commands. Legacy
```
--endpoint
```
and
```
AE_EVAL_BASE_URL
```
values may only match the current AE Host; other values fail before credentials are sent. Remove these settings and use
```
--host
```
to select a host.
Session exchange and evaluation requests reject redirects; session exchange has a 30-second timeout.
Path resource IDs must contain only letters, digits, hyphens, or underscores; separators and URL escape syntax are rejected locally.
Use the online platform. Do not suggest localhost/loopback endpoints or obtain tokens from a local
```
agent-test/.env
```
.
Authentication uses
```
AE_EVAL_ACCESS_TOKEN
```
when explicitly set; it is sent only as
```
Authorization: Bearer
```
and must never appear in command arguments or output.
In an AE sandbox, when
```
AE_EVAL_ACCESS_TOKEN
```
is unset, ae-cli automatically exchanges the sandbox-provisioned
```
cli-token
```
through te-claude for a short-lived evaluation session. The CLI token is never sent to Agent Evaluation. If the token can access multiple TE projects, set
```
AE_EVAL_TE_PROJECT_ID
```
to the authorized TE project id.
Outside a sandbox, when the token is unset, the CLI immediately returns
```
EVAL_ACCESS_TOKEN_MISSING
```
with
```
token_missing
```
onboarding guidance and sends no network request. Module availability and service health remain unchecked. Direct the user to obtain a session token through the platform web console (
```
sessionStorage.getItem('agent.evaluation.session-token')
```
) or
```
POST /agenteval/api/v1/auth/session
```
. Never request tokens in chat.
Successful commands return JSON by default; use the actual error envelope to diagnose failures.

Evaluation availability and onboarding

AI Evaluation Center is an optional deployment module. CLI command/Skill availability does not establish that the server module is installed or that the user has access.

When an error includes
```
error.details.onboarding
```
, follow the status-specific scope below. Do not blindly repeat the failed operation, change flags, refresh tokens, or use direct HTTP calls to bypass a denial. This is a host-Agent instruction, not a persistent CLI circuit breaker. Independent local work can continue.
First preserve local work: when asked to organize and upload a dataset, finish and save the local CSV/JSON if possible. Confirm its actual file path. If a write times out, loses its response, or receives a gateway/server failure, explain that the write result is unconfirmed: the server may already have accepted it. Do not claim either success or failure and do not resubmit blindly. Once connectivity is restored, use authorized read-only status checks with any known resource/import ID to reconcile the result. If no reliable identifier or lookup exists, ask the administrator/support to verify before another write.
Read
```
onboarding.status
```
, explain the evidence in the user's language, and follow its
```
next_steps
```
:
- ```
module_not_detected
```
  : stop further evaluation commands on the affected host until deployment/routing is verified. The API route was not detected (generic route 404 or HTML fallback). The optional module may be missing or its routing may be wrong. Contact the ThinkingData service representative (CSM or Technical Support) to verify deployment/routing, install the module if needed, and grant access. Do not assert that installation is certainly absent.
- ```
access_not_granted
```
  : stop only the denied operation. Independent authorized operations may continue; for example, a denied delete does not prohibit an authorized dataset read. Do not run other commands just to probe or bypass the denial. Stop a whole project or module only when that access scope is confirmed. Ask the administrator or ThinkingData support to check the required permission (READER / EDITOR / ADMIN as appropriate). A 403 alone proves neither cluster-wide denial nor module installation; do not recommend reinstallation or token rotation.
- ```
token_missing
```
  : stop remote evaluation operations on this host until a token is configured. The external terminal has no token; no request was sent and endpoint availability was not checked. Follow the conditional web-console instructions; if the app launcher entry is missing, request administrator/support verification instead of sending the user to a nonexistent page. This status does not establish the health of every evaluation service.
- ```
availability_unknown
```
  : stop blind retries of the failed operation. Network errors, timeouts, server failures, invalid responses, and missing sandbox session routes cannot establish installation state. Ask for host/network/service/routing verification; do not present an outage as an installation requirement. After connectivity is restored, authorized read-only status checks may reconcile an unconfirmed write; they must not resubmit it.
Briefly relate the three capabilities to the user's task: Datasets manage single/multi-turn test cases and versions, with CSV/JSON imports and AI Monitor traces; Evaluators provide templates/custom LLM judges, human calibration, and prompt optimization; Evaluation Runs test live Agents or score preset outputs, schedule regressions, and inspect low-score cases and traces. Translate this explanation naturally for Chinese or English users; CLI envelopes remain English.
If the user asks only about capabilities, explain them without a network call. If they also request current datasets/runs, apply the status-specific stop scope above when the first call reports onboarding; do not make another call just to demonstrate another capability.
Existing business error codes, error type (
```
api
```
), and exit code remain unchanged. Onboarding is additive; ordinary business 404 errors (such as
```
DATASET_NOT_FOUND
```
) do not indicate a missing module. Older CLIs may omit onboarding; use their diagnostic evidence cautiously and do not infer installation from a bare 404/5xx.

IDs, versions, and mutations

Discover real IDs with list/get commands. UUIDs are version-7 strings; never guess IDs, JSON payload shapes, frequency fields, or
```
lock_version
```
values.
Published dataset, evaluator, and evaluation-task versions are immutable and separate from drafts. Runs and schedules pin versions; existing published versions cannot be overwritten. Publishing unchanged assets returns
```
ASSET_VERSION_NO_CHANGES
```
.
Read operations can run once inputs are known. Writes require user intent; high-risk operations retain confirmation unless the user authorizes bypassing it with
```
--yes
```
.
Credential enabling/disabling, run cancellations, deletes, ownership transfers, template disabling, access-scope changes, and member admission/permission changes are high-risk writes. Check command help for risk and role requirements; references identify ADMIN-only operations.
Before each mutation requiring
```
--lock-version
```
, read the current resource. Every successful mutation increments its version; re-read before the next mutation.
Use the corresponding get command for runs, datasets, evaluators, models, agents, credentials, dataset cases, TE report config, and AI Monitor connection. Schedules use
```
+list_schedules
```
; calibration cases use
```
+list_calibration_cases
```
.
Field mutations use the dataset version; case mutations use the case version. First creation of TE report config or AI Monitor connection omits
```
--lock-version
```
; updates require the current value, including 0.
Async dataset imports, evaluation-set imports, and result exports require a fresh unique
```
--idempotency-key
```
per attempt. Poll the relevant status until completion; resource creation alone does not establish readiness.
Prefer
```
--dry-run
```
before broad/destructive writes whose targets have not been validated. It previews the request without executing it.
Evaluation errors redact the submitted credential values and session tokens; server-provided error messages and hints are not persisted in CLI logs. Stable business error codes are preserved unchanged; non-sensitive diagnostics remain available in the JSON error envelope.
Case data is redacted from command logs for both single-case and batch writes. Sensitive JSON parse failures report the flag and syntax guidance without echoing the input.
Credential secrets are write-only. Never echo secrets, session claims, or sensitive trace evidence; use redacted dry-run output.

Evaluation lifecycle

Select a published dataset version with
```
+list_dataset_versions
```
.
Choose
```
LIVE_AGENT
```
(default) to call an Agent/model, or
```
PRESET_RESULT
```
to score outputs stored in the dataset. PRESET_RESULT must omit agent/model config IDs.
```
IMPORTED_OUTPUT
```
is legacy data only; never recommend it.
Create a draft run, bind at least one ACTIVE evaluator (prefer pinned published versions), and configure complete field mapping plus result filter fields. An explicit empty result filter is required when no fields are selected.
Prefer
```
+validate_run
```
before starting. It runs the start preflight without a body or lock version. Read the latest run, then
```
+start_run
```
with its current lock version.
Inspect overview/distribution, drill into cases, and use the run reference for retry, trace, and export semantics.

Read runs.md for the executable sequence and mapping examples. Setup of datasets, evaluators, or connections needs only the matching reference above.

Transport transition

The existing platform contract exposes REST operations rather than evaluation capabilities registered in this repository. New commands retain that transport for typed input, permission confirmation, and file handling. They are Transitional; see the matching references for migration records. Check deployment Gateway coverage before extending these commands; prefer existing equivalent capabilities.

Transition status: transitional Owning module: Agent Evaluation (agent-test) Current transport: authenticated REST /agenteval/api/v1 Gateway target: TBD (evaluation capability registration) Review after: 2027-01-21 Exit condition: once equivalent evaluation capabilities are available, migrate selected typed/file commands to L2 or replace simple wrappers with L3; retain existing + commands only where compatibility or orchestration justifies them.

This record also applies to the locale and process-evidence updates in the existing evaluator/run references. It is a migration plan, not a claim that a particular deployed Gateway has been inspected.

ae-evaluation

NPX Install

Tags

SKILL.md Content

ae-evaluation

Read only the relevant reference

Endpoint and authentication

Evaluation availability and onboarding

IDs, versions, and mutations

Evaluation lifecycle

Transport transition