langsmith-trace

Original：🇺🇸 English
Translated

LangSmith tracing and trace debugging for AgentSeek templates. Covers CLI installation, adding tracing to LangGraph apps, querying traces, and inspecting run IO. Use when debugging agent backends, investigating slow traces, or adding observability to a template.

16installs
Sourceob-labs/agentseek
Added on

NPX Install

npx skill4agent add ob-labs/agentseek langsmith-trace

Tags

Translated version includes tags in frontmatter
langsmith-tracingagent-debugginglanggraph-observabilitycli-toolstrace-inspection

SKILL.md Content

View Translation Comparison →

LangSmith Trace

Add tracing to your agent and query traces for debugging. Supports Python and TypeScript.
[!IMPORTANT] This skill is tuned for AgentSeek template backends (LangGraph + middleware stacks). For general LangSmith concepts, see the upstream langsmith-skills repo.
[!CAUTION] Never pass 
--api-key
as a CLI flag or expose API keys in shell commands / tool calls. The CLI reads 
LANGSMITH_API_KEY
from the environment automatically. Using 
--api-key <value>
leaks secrets into shell history, process listings, and agent tool-call logs. Always rely on the environment variable set in your shell profile or 
.env
file.

Installation & Setup

1. Install the CLI

bash
curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh
The binary installs to 
~/.local/bin/langsmith
. If 
langsmith
is not found after install, add to your shell profile:
bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

2. Set your API key (environment variable only)

Add to 
~/.zshrc
(or the template's 
.env
file) — the user must fill in their own key:
bash
export LANGSMITH_API_KEY=<your-key-here>  # starts with lsv2_pt_
Key format must start with 
lsv2_pt_
. Get yours at https://smith.langchain.com/settings.
The CLI reads this env var automatically — never use 
--api-key
flags. Do not echo, print, or pass the key value in any shell command or tool call argument.

3. Verify

bash
langsmith project list
If you see a JSON array of projects, you're set. Common failures:
  • command not found
     — 
    ~/.local/bin
    not in PATH (see step 1)
  • 401 Unauthorized — key is wrong or expired; regenerate at LangSmith settings
  • Empty array 
    []
     — valid auth but no projects yet; create one in the UI or run a traced app

Adding Tracing

LangGraph / LangChain apps (automatic)

Just set environment variables — no code changes needed:
bash
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-key-here>  # must be set; do NOT pass via --api-key flag
export LANGSMITH_PROJECT=my-project  # optional, defaults to "default"
For serverless (Python): also set 
LANGCHAIN_CALLBACKS_BACKGROUND=false
to ensure traces flush before function exit.

Non-LangChain apps

Use the 
@traceable
decorator (Python) or 
traceable()
wrapper (TypeScript) and wrap your LLM client:
python
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable
def my_pipeline(question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content
typescript
import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";

const client = wrapOpenAI(new OpenAI());

const myPipeline = traceable(async (question: string) => {
  const resp = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: question }],
  });
  return resp.choices[0].message.content || "";
}, { name: "my_pipeline" });

Debugging a Trace (Step-by-Step)

This is the recommended workflow for investigating agent behavior:

Step 1: Find the right project

bash
langsmith project list
Look at 
last_run_start_time
to find which project has recent activity. Don't assume — LangGraph apps default to the 
"default"
project, not a named one.

Step 2: List recent traces

bash
langsmith trace list --project default --limit 5

# Or with full hierarchy inline (combines steps 2+3):
langsmith trace list --project default --limit 5 --show-hierarchy

Step 3: Get the trace hierarchy

bash
langsmith trace get <trace-id> --project <name>
This returns the full run tree — use it to understand the agent's execution flow.

Step 4: Get IO for all runs in a trace

bash
langsmith run list --trace-ids <trace-id> --project <name> --include-io
This gives you inputs and outputs for every run (LLM calls, tool calls, middleware).

Step 5: Drill into a specific run

bash
langsmith run get <run-id> --include-io

AgentSeek Trace Structure

Our templates produce traces with this typical hierarchy:
<agent_name> (root chain)
├── SkillsMiddleware.before_agent
├── PatchToolCallsMiddleware.before_agent
├── MemoryMiddleware.before_agent
├── model (chain) ← LLM turn
│   ├── TodoListMiddleware.awrap_model_call
│   ├── SkillsMiddleware.awrap_model_call
│   ├── FilesystemMiddleware.awrap_model_call
│   ├── SubAgentMiddleware.awrap_model_call
│   ├── SummarizationMiddleware.awrap_model_call
│   ├── AnthropicPromptCachingMiddleware.awrap_model_call
│   ├── MemoryMiddleware.awrap_model_call
│   └── ChatOpenAI (llm) ← actual LLM call (inputs/outputs here)
├── TodoListMiddleware.after_model
├── tools (chain) ← tool execution
│   ├── FilesystemMiddleware.awrap_tool_call
│   └── <tool_name> (tool) ← actual tool (inputs/outputs here)
├── model (chain) ← next LLM turn
│   └── ... (same middleware stack)
└── TodoListMiddleware.after_model
Key points:
  • The actual LLM call is always the innermost 
    ChatOpenAI
    run
  • Tool results are in the 
    <tool_name>
    run (e.g., 
    generate_cover
    , 
    execute
    )
  • Middleware wrappers are transparent — they add latency but the IO you care about is at the leaf nodes

Gotchas

--full
vs 
--include-io
on individual runs

Problem: 
langsmith run get <id> --full
can return null for inputs/outputs, even though the data exists. Despite 
--full
being documented as equivalent to 
--include-metadata --include-io --include-feedback
, the underlying API behavior differs for individual run fetches in some CLI versions.
Solution: Always use 
--include-io
explicitly when inspecting specific runs:
bash
# DO THIS
langsmith run get <run-id> --include-io

# NOT THIS (may return null IO despite docs saying it includes --include-io)
langsmith run get <run-id> --full
--full
works reliably on 
trace export
and 
run list
, but has inconsistent behavior on individual 
run get
calls. If this is fixed in a future CLI version, 
--include-io
still works correctly — it's always safe.

Null IO even with 
--include-io

If inputs/outputs come back null even with 
--include-io
, the project has IO logging disabled. Check for these environment variables in the template's 
.env
:
bash
LANGCHAIN_HIDE_INPUTS=true    # hides inputs from traces
LANGCHAIN_HIDE_OUTPUTS=true   # hides outputs from traces
Remove or set to 
false
to enable IO capture for debugging.

Project confusion

LangGraph apps trace to the 
"default"
project unless 
LANGSMITH_PROJECT
is explicitly set. Always run 
langsmith project list
first and check 
last_run_start_time
to find where your traces actually landed.

Tips

  • Add 
    --format pretty
    for human-readable output during interactive debugging
  • Use 
    LANGSMITH_ENDPOINT
    env var if connecting to a self-hosted LangSmith instance
  • The middleware ordering in the trace tree is configurable — your template may differ slightly from the diagram above

CLI Reference

Full command reference: reference/cli-commands.md