wjs-dubbing-video

*_<lang>_dub.mp4

/wjs-burning-subtitles/render.py

When to use

• User has a target-language SRT (e.g.,

  entrevista.zh-CN.srt

  ) and wants the video to speak that language.
• User says "中文配音 / 配音 / 帮我做配音 / dub it / voice over".
• User has multiple speakers on camera and wants different voices per speaker.

When NOT to use

• No SRT yet → run

  /wjs-transcribing-audio

  then

  /wjs-translating-subtitles

  first.
• Source-language only TTS (rare; usually you translate first) → still use this skill, but pass the source SRT.
• Burn-in only, no audio change → skip to

  /wjs-burning-subtitles

  .

Number of speakers — default to one

[A]

[B]

Engine routing — by voice ID

scripts/dub.py

zh_..._bigtts

VOLC_TTS_APPID

VOLC_TTS_ACCESS_TOKEN

zh-CN-...Neural

en-US-...Neural

Volcano TTS (Chinese only)

https://openspeech.bytedance.com/api/v3/tts/unidirectional

X-Api-App-Id:       (env: VOLC_TTS_APPID)         # 10-digit speech App ID
X-Api-Access-Key:   (env: VOLC_TTS_ACCESS_TOKEN)  # 32-char token from speech console
X-Api-Resource-Id:  volc.service_type.10029       # see resource ID note below
Content-Type:       application/json

~/code/.env

set -a; source ~/code/.env; set +a

Resource ID — important quirk

seed-tts-2.0

*_bigtts

seed-tts-2.0

200 code=55000000 "resource ID is mismatched with speaker related resource"

volc.service_type.10029

dub.py

volc.service_type.10029

VOLC_TTS_RESOURCE

• 401 code=45000010 "load grant: requested grant not found in SaaS storage"

  — the App ID + key combo is valid against the gateway, but the user has not activated this resource. They must go to 火山引擎 → 语音技术 → 语音合成大模型 → 实例管理 and 开通 the service. No workaround.

• 403 code=45000030

  — the speaker isn't included in the user's instance bundle.

Response format

data

code: 20000000

code: 0

import base64, json, requests
audio = b""
r = requests.post(url, headers=h, json=payload, timeout=60, stream=True)
for line in r.iter_lines():
    if not line: continue
    evt = json.loads(line)
    if evt.get("code") not in (0, None, 20000000):
        raise RuntimeError(f"code={evt.get('code')} {evt.get('message')}")
    if evt.get("data"):
        audio += base64.b64decode(evt["data"])

Speaker catalog (verified working under

volc.service_type.10029

)

zh_female_gaolengyujie_moon_bigtts

zh_female_kailangjiejie_moon_bigtts

zh_female_shuangkuaisisi_moon_bigtts

zh_female_linjianvhai_moon_bigtts

zh_female_yuanqinvyou_moon_bigtts

zh_female_meilinvyou_moon_bigtts

zh_female_shuangkuaisisi_emo_v2_mars_bigtts

vv_uranus_bigtts

wenroushunv_moon_bigtts

qingxin_moon_bigtts

yingmaoxiaoyuan_moon_bigtts

tianxinxiaoling_moon_bigtts

shaoergushi_moon_bigtts

Audio params

speech_rate

-8

--rate -8%

-8

• emotion="calm"

  ,

  emotion_scale=4

  — contemplative, default for this skill's spiritual-content niche.

• emotion="gentle"

  — softer / more intimate.

• emotion="neutral"

  — flat / informational.

• emotion="sad"

  — melancholic. Use sparingly.

dub.py

VOLC_TTS_EMOTION

VOLC_TTS_EMOTION_SCALE

en_male_*_bigtts

en_female_*_bigtts

dub.py

edge-tts (Microsoft Edge neural TTS)

uvx

uvx

uv venv .venv
uv pip install --python .venv/bin/python edge-tts

edge_tts.Communicate(...)

scripts/dub.py

Voice selection — match the original speaker

Chinese voices (Volcano preferred, edge-tts fallback)

zh_female_gaolengyujie_moon_bigtts

speech_rate=-8

zh-CN-XiaoxiaoNeural

zh-CN-XiaoyiNeural

zh-CN-YunjianNeural

zh-CN-YunxiNeural

zh-CN-YunyangNeural

zh-HK-HiuMaanNeural

zh-TW-HsiaoChenNeural

English voices (edge-tts neural, all multilingual)

*Multilingual*

en-US-AvaMultilingualNeural

en-US-EmmaMultilingualNeural

en-US-AndrewMultilingualNeural

en-US-BrianMultilingualNeural

en-US-AriaNeural

en-US-GuyNeural

en-GB-SoniaNeural

en-GB-RyanNeural

en-AU-WilliamMultilingualNeural

fr-FR-VivienneMultilingualNeural

en-US-AvaMultilingualNeural

--rate -5% --pitch -3Hz

Aria

Guy

Picking heuristics

• Mature contemplative female speaker (yoga/spirituality/coaching):

  zh-CN-XiaoxiaoNeural

  with

  --rate=-8% --pitch=-10Hz

  (or Volcano

  gaolengyujie

  ).
• Mature professional male:

  zh-CN-YunyangNeural

  with

  --rate=-5%

  . Avoid Yunjian/Yunxi (too energetic).
• Young casual speaker: Defaults; no pitch shift.
• Western-mouth feel: one of the

  *MultilingualNeural

  voices.

Always sample before committing

dub.py

1. Pick the longest-text cue (worst stretch case) and one short/casual cue (timbre check).
2. Synthesize 3–4 voice/rate/pitch combos at 3–8s each.
3. Show the user the audio panel and ask: "选哪个 voice？rate/pitch 要调吗？确认后我再跑全片。" Wait for explicit pick.

scripts/sample_voices.py

code=45000010

Running dub.py

.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/dub.py [voice] [rate] [pitch]

# Mature Chinese contemplative female (Volcano):
.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/dub.py \
    zh_female_gaolengyujie_moon_bigtts -8% +0Hz

# Warm English caring female (edge-tts, multilingual):
.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/dub.py \
    en-US-AvaMultilingualNeural -5% -3Hz

# Default Chinese fallback (no Volcano creds needed):
.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/dub.py \
    zh-CN-XiaoxiaoNeural -8% -10Hz

1. Reads the SRT (auto-detects

   *.zh-CN.srt

   ,

   *.en.srt

   , etc., or pass

   --srt

   ).
2. Synthesizes one MP3 per cue under

   dub_work/seg_NN.mp3

   .
3. Probes each clip's actual duration with

   ffprobe

   .
4. For each cue: if TTS is longer than the SRT slot, chains

   atempo

   filters to speed it up; if shorter, pads with silence after.
5. Inserts silence segments for SRT gaps and any trailing tail so the output audio length exactly matches the source video.
6. Muxes the new audio into

   *_zh_dub.mp4

   /

   *_en_dub.mp4

   keeping the original video stream by

   -c:v copy

   .

<source-stem>_<lang>_dub.mp4

entrevista_zh_dub.mp4

/wjs-burning-subtitles/render.py

Filling awkward silences

1. Slow the native TTS rate. Changing

   --rate

   from

   +0%

   to

   -12%

   to

   -15%

   produces clean, natural-sounding slower speech (much better than time-stretching afterward). Try

   -12%

   first;

   -15%

   /

   -20%

   for very contemplative content.
2. Mild slow-stretch per cue. When a cue's TTS is still shorter than its slot, run

   atempo

   between 0.82× and 0.95×.

   dub.py

   does this automatically: when slack > 0.5s, it sets

   atempo = max(0.82, tts_dur / target_dur)

   and pads the remainder. Below 0.82× the voice starts sounding drugged; above 0.92× the stretch is essentially imperceptible.
3. Expand the target-language text in the worst cues. When the slot is so long that even 0.82× stretch leaves >2s of silence, the cleanest fix is to lengthen the translation. Add natural Mandarin particles ("嗯，", "其实", "也就是说", "你知道") or unpack a compressed phrase into its full meaning. This changes the on-screen subtitle, so confirm with the user before doing it. Edit the SRT, regenerate just those segments by deleting their

   dub_work/seg_NN.mp3

   and re-running

   dub.py

   .

-12%

Multi-speaker dubbing (opt-in)

Path 1 (recommended for on-camera speakers): visual diarization

scripts/visual_diarize.py

uv pip install --python .venv/bin/python mediapipe opencv-python

.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/visual_diarize.py \
    --video input.mp4 --srt input.en.srt \
    --out input.en.diarized.srt \
    --report diarization_report.json \
    --sample-fps 5 --num-speakers 2

1. Samples N frames per second (default 5).
2. Runs MediaPipe FaceLandmarker (Tasks API) for up to

   --num-speakers

   faces per frame, 478 landmarks each.
3. Measures mouth aperture per face as the vertical distance between inner upper lip (idx 13) and inner lower lip (idx 14).
4. Bins faces by horizontal screen position (x-quantiles) → speakers

   A

   ,

   B

   , ... left-to-right.
5. For every cue's [start, end] window, integrates per-speaker frame-to-frame mouth-aperture change. Highest mover wins the tag.
6. Writes a

   [A]

   /

   [B]

   -prefixed SRT plus a JSON report with per-cue scores and a confidence ratio (winner / runner-up).

/tmp/mp_models/face_landmarker.task

confidence_ratio < 1.5

Path 2 (fallback): manual tagging

1
00:00:00,000 --> 00:00:03,400
[A] So what about that AI rewrite thing?

2
00:00:03,400 --> 00:00:08,200
[B] Right — let me explain the workflow.

*.tagged.srt

/wjs-burning-subtitles

Routing voices in dub.py

--voice-map

speaker=voice

.venv/bin/python ~/.claude/skills/wjs-dubbing-video/scripts/dub.py \
    en-US-AndrewMultilingualNeural -3% +0Hz \
    --srt input.en.tagged.srt \
    --voice-map "A=en-US-BrianMultilingualNeural,B=en-US-AndrewMultilingualNeural"

• Two of the same gender: pick voices with audibly different timbre. Brian (casual) + Andrew (warm) works for two American males. Ava (warm female) + Emma (cheerful female) for two females.
• Mixed gender: Ava + Andrew is a clean default.
• Accent contrast: pair

  en-US-

  and

  en-GB-

  for distinctness.
• Chinese: mix Volcano voices like

  zh_female_gaolengyujie_moon_bigtts

  (mature) +

  zh_female_kailangjiejie_moon_bigtts

  (warm sister).

Limits

• A speaker is consistently off-camera while talking.
• Camera cuts or zooms make face position unstable across cues.
• Three or more speakers sit at similar horizontal positions (x-quantile binning is too coarse — switch to k-means on (x, y) or use audio-based diarization instead).

pyannote.audio

whisperx --diarize

Output

• <source-stem>_<lang>_dub.mp4

  — video stream-copied from source, audio replaced with the time-aligned dub track. Drop-in input for

  /wjs-burning-subtitles/render.py

  .

• dub_work/seg_NN.mp3

  — per-cue TTS clips (kept for resume / per-cue regen).

Downstream

• /wjs-burning-subtitles

  — to mix the original audio as a low-volume bed, burn the SRT, or both. The final encode happens there in one ffmpeg pass (no cascade). Pass

  --video <source.mp4> --dub <source_lang_dub.mp4> [--srt <srt>]

  to its

  render.py

  .
• The dub-only file (

  *_<lang>_dub.mp4

  ) is technically a finished video and can ship as-is, but it sounds dubbed (because it is). Mixing the original underneath gives the "professional translation" feel — do that in

  /wjs-burning-subtitles

  .

Anti-patterns

• ❌ Calling

  uvx edge-tts

  once per cue. Spawns a Python process each time; bing endpoint rate-limits or RSTs mid-render. Use the persistent library path in

  dub.py

  .
• ❌ Trusting

  audio_source

  without listening. Always sample a 30 s clip before committing.
• ❌ Stretching below 0.82× atempo. Voice starts sounding drugged. Add silence padding or expand text instead.
• ❌ Tagging single-speaker SRTs with

  [A]

  . Wastes time and the dub sounds the same. Default to one voice.
• ❌ Promising a Volcano voice without smoke-testing it on the user's instance. The doc lists many voices that error with

  code=55000000

  against typical SeedTTS-2.0 starter bundles. Always synth a 5-word smoke test before quoting.
• ❌ Parsing Volcano response as one JSON document. It's streaming NDJSON; the success terminator is

  code=20000000

  , not

  code=0

  . Concatenate base64-decoded

  data

  chunks for the full MP3.