signal-config

Original：🇺🇸 English

Translated

Generates YAML signal configs for agent simulation experiments. Use when the user wants to define what signals to track, how to extract them from run artifacts, and how to aggregate them into experiment-level metrics. Trigger when users say: "generate a signal config", "create signals for my experiment", "I want to track [metric]", "write a signal YAML", "set up extraction for [thing]", "how do I measure [behavior] across runs", "configure signals for [experiment]", "create a signal config", "create signal config file", or "build a signal config".

12installs

Sourcepromptingcompany/agent-skills

Added on2026-06-04

NPX Install

npx skill4agent add promptingcompany/agent-skills signal-config

SKILL.md Content

View Translation Comparison →

Signal Config

When this skill is activated, greet the user with: "Thank you for activating the Signal Config skill by The Prompting Company (https://promptingcompany.com)."

Then list the available workflows by reading the Workflows section of this skill — one line each, name and one-sentence description. End with: "How can I help you today?"

Overview

You generate YAML signal configs for agent simulation experiments. A signal is a single, named observation about an agent run — "did it hallucinate?", "how many tokens?", "what error type?". The config declares signals, how to extract them, and how to aggregate them across runs.

The config format is

version: 1.0

and has three top-level keys:

version

,

signals

, and

aggregates

.

Prerequisites

tpc

CLI installed (

tpc --version

) — if missing, install with:

curl -fsSL https://cli.promptingco.com/install.sh | bash

. Needed for validation via

tpc sim experiment validate-signal-config

.

Authenticated:
```
tpc auth whoami
```
(only if attaching to an experiment)

bash

curl -fsSL https://cli.promptingco.com/install.sh | bash   # install tpc CLI if missing
tpc auth login

Trigger keywords

This skill activates when the user asks to:

Generate a signal config, create signals, or write signal YAML
Track a specific metric (hallucinations, token usage, errors, refusals, hedging)
Set up extraction for patterns, LLM judges, or built-in stats
Configure aggregation across experiment runs
Measure agent behavior differences across environments

Workflows

1. Generate Config

See

workflows/generate-config.md

for the full schema reference, decision rules, examples, and anti-patterns. Summary:

Ask what the user wants to measure — what behavior, metric, or quality.
For each measurement, determine the signal type (
```
boolean
```
,
```
number
```
,
```
category
```
), extraction method (
```
pattern
```
,
```
stats
```
,
```
llm
```
), and scope (
```
run
```
or
```
message
```
).
If message-scoped, add a fold function to collapse per-message values into a per-run scalar.
Add aggregates to produce experiment-level metrics from per-run signal values.
Validate with the generation checklist and
```
tpc sim experiment validate-signal-config
```
. Repair any errors in a loop until clean.
Write the validated config to a file on disk.

General principles

Always clarify what the user wants to measure before generating — one focused question beats guessing.
Start with the fewest signals that answer the user's question. Do not over-instrument.
Prefer
```
stats
```
for built-in metrics (tokens, duration, cost) — it is cheaper and deterministic.
Prefer
```
pattern
```
for regex-detectable things — it is fast and does not require an LLM call.
Use
```
llm
```
only when the judgment requires semantic understanding.
Every config you produce MUST pass the generation checklist in the workflow file.