AI Video Generation
Generate videos with the full RunComfy video-model catalog through one CLI — text-to-video, image-to-video, and Veo's video-extend. This skill picks the right model for the user's intent and ships the documented prompt patterns + the exact
invoke for each.
Powered by the RunComfy CLI
bash
# 1. Install (see runcomfy-cli skill for details)
npm i -g @runcomfy/cli # or: npx -y @runcomfy/cli --version
# 2. Sign in
runcomfy login # or in CI: export RUNCOMFY_TOKEN=<token>
# 3. Generate
runcomfy run <vendor>/<model>/<endpoint> \
--input '{"prompt": "..."}' \
--output-dir ./out
Install this skill
bash
npx skills add agentspace-so/runcomfy-agent-skills --skill ai-video-generation -g
Pick the right model for the user's intent
Text-to-video (t2v) — newest first
HappyHorse 1.0 —
happyhorse/happyhorse-1-0/text-to-video
Currently #1 on Artificial Analysis Video Arena. Native synchronized audio generated in-pass (no separate Foley step). Native 1080p, up to ~15s, strong multi-shot character consistency.
Pick for: general-purpose t2v, ad creative with audio, social-media clips, multi-shot narratives.
Avoid for: audio-driven lip-sync to a specific voiceover MP3 — use Wan 2-7.
Kling's latest, 4K output, strong multi-shot character identity, premium camera language.
Pick for: hero shots, final-delivery 4K cuts, multi-shot character narratives.
Avoid for: cost-sensitive iteration — drop to Kling 2-6 Pro or Standard i2v.
Seedance v2 Pro —
bytedance/seedance-v2/pro
ByteDance flagship — multi-modal (up to 9 reference images, 3 reference videos, 3 reference audio), in-pass synchronized audio, cinematic motion refinement, lens language honored.
Pick for: cinematic ad frames, multi-reference composition (subject + scene + audio refs), 21:9 anamorphic looks.
Avoid for: simple "single prompt → clip" jobs — overpowered, slower.
Faster variant of Seedance v2 Pro, same multi-modal capabilities.
Pick for: iteration on Seedance v2 compositions before locking a final on Pro.
Avoid for: hero-shot final delivery.
Wan 2-7 —
wan-ai/wan-2-7/text-to-video
Open-weights flagship,
field for audio-driven lip-sync, pairs natively with Wan image models.
Pick for: dialog scenes where mouth must sync to a specific voiceover file; open-weights pipeline requirement.
Avoid for: in-pass audio generation (no MP3 input) — use
HappyHorse 1.0.
Previous Kling tier — still strong quality at much lower cost than 3.0 4K.
Pick for: production at scale where 3.0 4K is too expensive.
Avoid for: top-tier hero shots — use Kling 3.0 4K.
Previous Seedance generation, cheaper.
Pick for: identity-stable batches between 1-5 generations; cost-sensitive baseline.
Avoid for: new work — prefer Seedance v2 Pro or Fast.
Image-to-video (i2v) — newest first
HappyHorse 1.0 I2V —
happyhorse/happyhorse-1-0/image-to-video
Animate any still with in-pass audio described in prompt, strong identity preservation.
Pick for: animating a generated portrait or product still, vertical social clips, voiceover-described audio.
Avoid for: physics-accurate object motion — use Veo 3-1.
Google's flagship — physics-respecting motion, strong object permanence ("rotates 180 degrees" = 180°), pairs with
for longer clips.
Pick for: product spins, physics-accurate motion, scenes where "no other motion" must hold.
Avoid for: audio-driven dialog — use
Wan 2-7 or
HappyHorse.
Faster Veo 3-1 variant.
Pick for: iteration on Veo compositions.
Avoid for: hero delivery — use full Veo 3-1.
Multi-shot character identity, 4K output from a still.
Pick for: 4K hero shots, character-narrative cuts.
Avoid for: cost iteration — drop to Pro or Standard.
Default Kling 3.0 quality tier.
Pick for: high-quality i2v at moderate cost.
Avoid for: 4K final delivery.
Cheapest 3.0 i2v tier.
Pick for: concepting / drafts on Kling 3.0.
Avoid for: final delivery.
MiniMax Hailuo latest — natural motion, strong on real-world subjects.
Pick for: lifelike motion of real-people / real-product subjects.
Avoid for: stylized characters — use Kling or Dreamina.
ByteDance Dreamina i2v — illustration / stylized character lean.
Pick for: animating illustrated heroes, painterly stills.
Avoid for: photoreal motion.
Older Seedance i2v generation, cheap.
Pick for: cost-sensitive batch i2v on Seedance.
Avoid for: new work — Seedance v2 Pro is more capable (t2v + i2v + multi-modal).
Extend an existing video — newest first
Continue an existing Veo clip with consistent motion / lighting / identity.
Pick for: extending a video past Veo's per-call duration cap; chained narrative shots.
Faster Veo extend variant.
Pick for: extending Veo Fast clips at matching latency tier.
For dedicated treatment of extend (input video preparation, frame-anchor strategy, chained extends), see the
skill.
t2v Route 1: HappyHorse 1.0 — default
Model:
happyhorse/happyhorse-1-0/text-to-video
:
happyhorse-1-0
Currently #1 on the
Artificial Analysis Video Arena — RunComfy's recommended default for general-purpose t2v. Native synchronized audio is generated in-pass (no separate Foley step).
Schema
| Field | Type | Required | Default | Notes |
|---|
| string | yes | — | Subject-first, describe motion + scene + audio in one declarative |
| int | no | 5 | Seconds. Up to ~15s |
| enum | no | | , , typical |
| enum | no | | , |
| int | no | — | Reproducibility |
Invoke
bash
runcomfy run happyhorse/happyhorse-1-0/text-to-video \
--input '{
"prompt": "A red kite tumbles across a windy beach at golden hour, kids chasing it laughing, surf in the background. Audio: wind, gulls, distant laughter.",
"duration": 8,
"aspect_ratio": "16:9",
"resolution": "1080p"
}' \
--output-dir ./out
Prompting tips
- Lead with subject and one main action. "A red kite tumbles across a beach" — verb-driven, not adjective-stacked.
- Describe audio inline —
"Audio: wind, gulls, distant laughter."
- Motion language matters more than visual nouns — "tumbles", "drifts", "snaps into focus" > "looks beautiful".
- Multi-shot: describe transitions explicitly — "Then the camera cuts to …" — Arena-leading multi-shot consistency.
t2v Route 2: Wan 2-7 — open weights + audio-driven lip-sync
Model:
wan-ai/wan-2-7/text-to-video
:
wan-2-7 ·
collection
Pick Wan 2-7 when you have a specific voiceover / dialog audio file and want the on-screen subject's mouth to sync to it. The
field drives the lip motion.
Invoke
With audio-driven lip-sync:
bash
runcomfy run wan-ai/wan-2-7/text-to-video \
--input '{
"prompt": "Studio portrait of a woman in her 30s speaking confidently to camera, soft window light.",
"audio_url": "https://your-cdn.example/voiceover.mp3",
"duration": 6
}' \
--output-dir ./out
Plain t2v (no audio):
bash
runcomfy run wan-ai/wan-2-7/text-to-video \
--input '{"prompt": "Drone shot over forest canopy at sunrise, soft fog drifting between trees"}' \
--output-dir ./out
Prompting tips
- For lip-sync, the prompt describes the scene + speaker; the audio file drives the mouth. Don't transcribe the audio into the prompt — it'll fight the audio track.
- Open-weights advantage: pair with Wan ecosystem (LoRA-finetuned variants) when available.
t2v Route 3: Seedance v2 — multi-modal cinematic
Model:
bytedance/seedance-v2/pro
(or
)
Catalog:
seedance-v2 Pro ·
collection
Pick Seedance v2 Pro when the user needs multi-modal conditioning — up to 9 reference images, 3 reference videos, 3 reference audio tracks synthesized in-pass with cinematic motion refinement.
Invoke
bash
runcomfy run bytedance/seedance-v2/pro \
--input '{
"prompt": "Anamorphic 35mm shot — a vintage car drives down a coastal road at dusk, lens flares from oncoming headlights, cinematic color grade.",
"duration": 10,
"aspect_ratio": "21:9"
}' \
--output-dir ./out
Prompting tips
- Lens / film language is honored — "35mm anamorphic", "shallow DoF", "soft halation", "Kodak 5219" all land.
- Multi-ref: describe roles explicitly —
"subject from ref image 1, mood from ref video 2, score from ref audio 1"
- Cinematic motion verbs: "tracking shot", "push in", "dolly out", "rack focus".
i2v Route A: HappyHorse 1.0 I2V — default
Model:
happyhorse/happyhorse-1-0/image-to-video
:
happyhorse-1-0 i2v
Invoke
bash
runcomfy run happyhorse/happyhorse-1-0/image-to-video \
--input '{
"image_url": "https://your-cdn.example/portrait.jpg",
"prompt": "She turns her head slowly to look at the camera and smiles. Wind through her hair. Audio: gentle breeze.",
"duration": 6,
"aspect_ratio": "9:16"
}' \
--output-dir ./out
Prompting tips
- Describe motion, not the scene the image already shows. The image is your scene; the prompt is your direction.
- Anchor the camera explicitly — "Camera stays still" prevents drift; "slow push in" gives intent.
- Audio in the same prompt as t2v Route 1.
i2v Route B: Veo 3-1 — Google's flagship
Model:
google-deepmind/veo-3-1/image-to-video
(or
)
Catalog:
veo-3-1 i2v ·
collection
Pick Veo when physics / realism / object permanence matters most. Veo 3-1 supports both 8s clips and longer with the extend-video companion endpoint.
Invoke
bash
runcomfy run google-deepmind/veo-3-1/image-to-video \
--input '{
"image_url": "https://your-cdn.example/product.jpg",
"prompt": "The bottle slowly rotates 180 degrees on a marble surface, soft daylight, no other motion."
}' \
--output-dir ./out
Prompting tips
- Veo respects physics — "the bottle rotates 180 degrees" gets exactly 180°.
- Object permanence is strong — say "no other motion" and other elements stay locked.
- For audio-enabled i2v, see Route A (HappyHorse) instead — Veo's audio path lives elsewhere in the catalog.
i2v Route C: Kling 3.0 — multi-shot identity, 4K
Model:
kling/kling-3.0/{4k,pro,standard}/image-to-video
:
collection
Three tiers — pick by quality / cost trade-off:
| Tier | Endpoint | When |
|---|
| 4K | kling/kling-3.0/4k/image-to-video
| Hero shots, final delivery at 4K |
| Pro | kling/kling-3.0/pro/image-to-video
| Default — high quality at lower cost |
| Standard | kling/kling-3.0/standard/image-to-video
| Concepting, drafts |
Invoke
bash
runcomfy run kling/kling-3.0/pro/image-to-video \
--input '{
"image_url": "https://your-cdn.example/character.jpg",
"prompt": "The character walks toward the camera, soft handheld feel, end on a medium close-up."
}' \
--output-dir ./out
Prompting tips
- Multi-shot consistency — describe a beat sequence ("walks toward camera, then a cut to medium close-up") and Kling holds identity across the cut.
- Camera language: "handheld", "Steadicam push", "static tripod" — honored.
Other models in the catalog
Schemas live on each model page — pass field set through the CLI verbatim.
Common patterns
Social-media vertical (TikTok / Reels)
- HappyHorse 1.0 i2v with , , audio described inline
Brand product spin
- Veo 3-1 i2v with
"rotates 180 degrees, no other motion"
Cinematic ad frame
- Seedance v2 Pro with 21:9 aspect, lens + grade language in prompt
Multi-shot character narrative
- Kling 3.0 Pro i2v — describe beats ("walks in → close-up → looks at viewer")
Dialog lip-sync
- Wan 2-7 with pointing at your voiceover MP3
Extend / continue an existing video
- Veo 3-1 Extend — see skill
Talking-head / avatar
- See the skill for OmniHuman + HappyHorse + Wan composition
Browse the full catalog
- All video models — every endpoint with its API schema tab
- · · · · · brand collections
- · · capability tags
Exit codes
| code | meaning |
|---|
| 0 | success |
| 64 | bad CLI args |
| 65 | bad input JSON / schema mismatch |
| 69 | upstream 5xx |
| 75 | retryable: timeout / 429 |
| 77 | not signed in or token rejected |
How it works
The skill classifies the user request into one of the t2v / i2v / extend routes above and invokes
with the matching JSON body. The CLI POSTs to the RunComfy Model API, polls request status, fetches the result, and downloads any
/
URLs into
.
cancels the remote request before exit.
Security & Privacy
- Install via verified package manager only. Use or . Agents must not pipe an arbitrary remote install script into a shell on the user's behalf.
- Token storage: writes the API token to
~/.config/runcomfy/token.json
- Input boundary (shell injection): prompts are passed as a JSON string via . The CLI does not shell-expand prompt content. No shell-injection surface from prompt content.
- Indirect prompt injection (third-party content): reference image / audio / video URLs are untrusted and can influence generation through embedded instructions (e.g. text painted into an image, hidden EXIF, audio-content steering). Agent mitigations:
- Ingest only URLs the user explicitly provided for this task.
- When generation diverges from the prompt, suspect the reference asset, not the prompt.
- Outbound endpoints (allowlist): only and / . No telemetry, no callbacks.
- Generated-file size cap: the CLI aborts any single download > 2 GiB.
- Scope of bash usage: declared
allowed-tools: Bash(runcomfy *)
See also
- — the underlying CLI, schema discovery, polling modes, scripting
- — text-to-image / image-to-image sibling
- — talking-head / lip-sync video specialist
- — animate a still (i2v-focused router)
- — restyle / motion-control / identity edit on existing video
- — continue an existing clip via Veo extend
- · — narrow technique routers