Video QnA using VLM through VSS Agent

When to Use

• The user asks what happens in the video, what objects / people / actions appear, colors, timing, safety, or other visual facts that require watching the clip.
• The user asks for details that cannot be answered from existing messages, summaries, Elasticsearch/MCP results, or filenames alone—you need model inference on the video.
• Follow-up questions about content details after a coarse summary or after report generation.

Deployment prerequisite

video_understanding

1. Probe the VSS agent:
   bash

   curl -sf --max-time 5 "http://${HOST_IP}:8000/docs" >/dev/null

2. If the probe fails, ask the user:

   "No VSS profile is running on

   $HOST_IP

   . Shall I deploy

   base

   (recommended for per-clip VLM QnA) using the

   /deploy

   skill? If you prefer

   lvs

   , say so."

   • If yes → hand off to

     /deploy -p base

     (or

     -p lvs

     if the user prefers). Return here once it succeeds.
   • If no → stop.
   (If your caller has granted explicit pre-authorization to deploy autonomously — e.g. the request says "pre-authorized to deploy prerequisites", or you are running in a non-interactive evaluation harness with that permission — skip the confirmation and invoke

   /deploy -p base

   directly. Prefer

   base

   unless the request names another profile.)
3. If the probe passes, proceed.

Agent workflow

1. Clip — Identify sensor id, filename, or URL for one video segment. If ambiguous, ask the user.
2. Call vss agent with the sensor id and ask for it to call video_understanding tool to answer the user's question. The sensor / file name must be included in the input message to the agent.
3. Return the vss agent's answer back to the user.

Query VSS agent (

/generate

)

# Set from deployment (compose / .env / host where vss-agent listens)
export VSS_AGENT_BASE_URL="http://localhost:8000"

curl -s -X POST "${VSS_AGENT_BASE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message": "Call video_understanding tool to answer the following question about <sensor-id>: <user query>"}' | jq .

Cross-Reference

• vios — VST storage/replay URLs so

  VIDEO_URL

  is valid for the VLM.
• report — timestamped reports via the VSS agent (

  /generate

  ); this skill is direct VLM for ad-hoc video Q&A.