Back to Details

audio-reactive-video-composition

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Audio-reactive video composition

音频反应式视频合成

Use this skill to turn an approved audio source into an auditable cue map and then into bounded visual behavior. The central contract is:
$$ \text{source audio} \rightarrow \text{measured features} \rightarrow \text{confidence-bearing cues} \rightarrow \text{reviewed anchors} \rightarrow \text{deterministic visual timeline} $$
Do not equate a detector output with editorial meaning. An onset is not automatically a beat, cut, drop, or lyric accent; an acoustic cluster is not automatically a verse or chorus.
使用本方法可将已获批准的音频源转换为可审计的cue map，进而生成受约束的视觉行为。核心流程为：
$$ \text{source audio} \rightarrow \text{measured features} \rightarrow \text{confidence-bearing cues} \rightarrow \text{reviewed anchors} \rightarrow \text{deterministic visual timeline} $$
请勿将检测器输出等同于编辑意图。Onset并非自动等同于beat、剪辑点、段落切换或歌词重音；声学聚类也并非自动等同于主歌或副歌。

Evidence stance

证据立场

  • Documented fact: behavior stated by official analysis libraries, standards, or cited research.
  • Production heuristic: a practical mapping that must be tested on the track and audience.
  • Empirical observation: a measured result from the supplied audio, analyzer run, render, or playback.
Analysis algorithms, defaults, and model behavior are volatile. Facts were verified 2026-07-12. Pin the decoder, library, model, parameters, and random seeds used for each production.
  • 已记录事实： 由官方分析库、标准或引用研究明确说明的行为。
  • 制作启发法： 需针对特定曲目和受众测试的实用映射规则。
  • 实证观察： 从提供的音频、分析器运行结果、渲染或回放中得到的测量结果。
分析算法、默认设置及模型行为易发生变化。所有事实均于2026-07-12验证。需固定每次制作所使用的解码器、库、模型、参数及随机种子。

Scope

范围

This skill owns source custody, analysis policy, confidence interpretation, rhythmic/non-rhythmic routing, cue promotion, feature-to-visual mappings, rational frame alignment, accessibility, and render QA.
It does not own music-video narrative or artist branding, music generation, mastering, source separation, transcription, lyric writing, or a HyperFrames/Remotion/FFmpeg-specific implementation.
本方法负责音频源管理、分析策略、置信度解读、节奏/非节奏路由、cue升级、特征到视觉的映射、合理帧对齐、可访问性及渲染QA。
不负责音乐视频叙事或艺人品牌塑造、音频生成、母带处理、源分离、转录、歌词创作，或HyperFrames/Remotion/FFmpeg特定实现。

Freeze the audio contract

固化音频协议

Record before analysis:
  • source path/URI, SHA-256, acquisition source, rights basis, and restrictions;
  • selected stream, codec, native sample rate, channels/layout, start timestamp, and duration;
  • decoder/resampler and versions;
  • canonical PCM format, channel/downmix policy, analysis sample rate, window/hop sizes, centering, and padding;
  • analyzer/library/model versions, priors, thresholds, and seeds;
  • trim offsets and source time origin;
  • master output frame rate as a rational number;
  • transcript/lyrics source, language, timing provenance, review status, and separate rights basis.
Use sample indices as the primary analysis clock where possible. Derived seconds should not replace exact source positions.
分析前需记录：
  • 源路径/URI、SHA-256、获取来源、授权依据及限制；
  • 选定的流、编码格式、原生采样率、声道/布局、起始时间戳及时长；
  • 解码器/重采样器及其版本；
  • 标准PCM格式、声道/下混策略、分析采样率、窗口/步长、居中方式及填充；
  • 分析器/库/模型版本、先验值、阈值及种子；
  • 修剪偏移量及源时间原点；
  • 主输出帧率（以有理数表示）；
  • 转录/歌词来源、语言、时序依据、审核状态及独立授权依据。
尽可能使用采样索引作为主要分析时钟。推导的秒数不应替代精确的源位置。

Interpret features conservatively

保守解读特征

Documented facts:
  • librosa.beat.beat_track
    estimates tempo from onset strength and selects beat positions consistent with it; it does not return calibrated beat confidence.
  • librosa.onset.onset_detect
    peak-picks an onset-strength envelope. Onsets estimate event attacks, not semantic accents.
  • Predominant local pulse can model changing tempo, but remains an estimate.
  • Essentia confidence values are algorithm-specific; they are not universal probabilities, and some routes return an unusable zero confidence.
  • Beat evaluation commonly accounts for half/double-tempo metrical ambiguity and uses tolerances rather than exact equality.
  • Structural boundaries and structural labels are separate tasks. Cluster labels such as A/B/C do not establish verse/chorus meaning.
  • RMS is an energy measure. EBU R128 loudness uses defined weighting and windows. Neither is a direct emotion score.
  • Spectral centroid, bandwidth, and contrast describe spectrum distribution; they do not mean happiness, tension, or quality.
  • Silence detection is relative to a declared reference and threshold.
  • pYIN voicing means pitch periodicity, not proof that a person is speaking or singing.
Keep raw candidates distinct from human-promoted anchors.
已记录事实：
  • librosa.beat.beat_track
    从onset强度估算 tempo，并选择与之匹配的beat位置；不返回校准后的beat置信度。
  • librosa.onset.onset_detect
    对onset强度包络进行峰值检测。Onset仅估算事件起始，而非语义重音。
  • 主导局部脉冲可模拟变化的tempo，但仍为估算值。
  • Essentia置信度值为算法特定值；并非通用概率，部分路由会返回无效的零置信度。
  • Beat评估通常会考虑半速/倍速的节拍歧义，并使用容差而非精确相等。
  • 结构边界与结构标签是独立任务。A/B/C等聚类标签并不代表主歌/副歌的含义。
  • RMS是能量测量指标。EBU R128响度使用定义的加权和窗口。两者均非直接的情感评分。
  • 频谱质心、带宽和对比度描述频谱分布；并不代表愉悦感、紧张感或质量。
  • 静音检测相对于声明的参考值和阈值。
  • pYIN有声化指音高周期性，并非证明有人在说话或唱歌。
需将原始候选与人工升级的锚点区分开。

Cue-map contract

Cue-map协议

Each cue should include:
json
{
  "id": "accent-014",
  "type": "onset",
  "time_samples": 417312,
  "time_seconds": 9.46399,
  "value": 0.82,
  "units": "normalized-onset-strength",
  "analyzer": "record exact library and algorithm",
  "confidence": null,
  "confidence_semantics": "not supplied by analyzer",
  "profile_hash": "sha256:...",
  "status": "promoted-anchor",
  "editorial_role": "single visual accent",
  "reason": "reviewed strong transient before section change"
}
For interval features, include exact end samples/times. Record rejected candidates rather than deleting evidence. Keep acoustic section names neutral until lyrics, metadata, or human review supports functional labels.
每个cue应包含：
json
{
  "id": "accent-014",
  "type": "onset",
  "time_samples": 417312,
  "time_seconds": 9.46399,
  "value": 0.82,
  "units": "normalized-onset-strength",
  "analyzer": "record exact library and algorithm",
  "confidence": null,
  "confidence_semantics": "not supplied by analyzer",
  "profile_hash": "sha256:...",
  "status": "promoted-anchor",
  "editorial_role": "single visual accent",
  "reason": "reviewed strong transient before section change"
}
对于区间特征，需包含精确的结束采样/时间。记录被拒绝的候选而非删除证据。在歌词、元数据或人工审核支持功能标签之前，保持声学段落名称中立。

Choose a routing mode

选择路由模式

Rhythmic

节奏模式

Use when the selected tracker has useful evidence, local tempo is stable enough, and beat/onset results agree. Periodic motion may follow beats; selected strong accents or reviewed section changes may motivate cuts.
当选定的跟踪器提供有效证据、局部tempo足够稳定且beat/onset结果一致时使用。周期性运动可跟随beat；选定的强重音或审核通过的段落切换可触发剪辑。

Mixed confidence

混合置信度模式

Use reliable rhythmic windows locally. Elsewhere shift to onsets, transcript phrases, energy, silence, or manual anchors. Never extrapolate a grid through a failed interval.
在局部使用可靠的节奏窗口。其他区域切换为onset、转录乐句、能量、静音或手动锚点。切勿通过失效区间外推网格。

Non-rhythmic

非节奏模式

Use reviewed speech/lyric phrases, silence, energy contour, spectral change, and manually confirmed macro anchors. This suits rubato, ambient work, spoken word, sparse recordings, and free improvisation.
使用审核通过的语音/歌词乐句、静音、能量轮廓、频谱变化及手动确认的宏观锚点。适用于自由节奏、氛围作品、口语、稀疏录音及自由即兴创作。

Map features to visual behavior

将特征映射到视觉行为

Production heuristics:
  • Give each feature a limited visual responsibility.
  • Normalize continuous values with robust per-track or per-section statistics, not one extreme maximum.
  • Smooth noise and use hysteresis before switching visual states.
  • Clamp every parameter and define neutral behavior for missing/invalid values.
  • Use macro section anchors, medium phrase anchors, and sparse accents rather than cutting on every event.
  • Reserve low-energy or silent spans for holds, reading, resets, and visual breathing room.
  • Protect lyric lines and important vocal phrases from competing cuts and overlays.
  • Preserve one cue map across aspect ratios; recompose layout, not timing.
Avoid quantitative-looking mappings that imply false measurement. If energy maps to scale, define the exact bounded range and do not call it emotion.
制作启发法：
  • 为每个特征分配有限的视觉职责。
  • 使用稳健的单曲目或单段落统计数据对连续值进行归一化，而非单一极端最大值。
  • 在切换视觉状态前平滑噪声并使用滞后机制。
  • 限制每个参数，并为缺失/无效值定义中性行为。
  • 使用宏观段落锚点、中观乐句锚点和稀疏重音，而非每个事件都触发剪辑。
  • 为低能量或静音时段预留停顿、阅读、重置及视觉喘息空间。
  • 保护歌词行和重要 vocal 短语免受竞争性剪辑和叠加干扰。
  • 在不同宽高比下保留同一cue map；重新构图布局，而非调整时序。
避免看似量化的映射，以免暗示虚假测量。若将能量映射到缩放比例，需明确定义精确的边界范围，且不得称之为情感。

Deterministic frame timing

确定性帧时序

Keep exact rates such as $30000/1001$ rational. For each cue, define a rounding policy. A conservative visual-response policy assigns the cue to the first output frame whose presentation time is at or after the audio event:
$$ f = \left\lceil t_{event} \cdot \frac{fps_{num}}{fps_{den}} \right\rceil $$
Record both the original event time and mapped frame. Verify actual frame presentation timestamps after encoding because an encoder may duplicate or drop frames to satisfy constant-frame-rate output.
The visual state at frame $f$ must derive from cue data and frame time, not wall-clock playback. Seed procedural mappings and freeze analyzer output before distributed rendering.
保留精确的帧率（如 $30000/1001$ 这类有理数）。为每个cue定义舍入策略。保守的视觉响应策略将cue分配给呈现时间大于或等于音频事件的第一帧：
$$ f = \left\lceil t_{event} \cdot \frac{fps_{num}}{fps_{den}} \right\rceil $$
记录原始事件时间和映射后的帧。编码后需验证实际帧呈现时间戳，因为编码器可能会复制或丢弃帧以满足恒定帧率输出。
帧 $f$ 的视觉状态必须源自cue数据和帧时间，而非实时播放时钟。在分布式渲染前为过程化映射设置种子并冻结分析器输出。

Lyrics, vocals, and captions

歌词、人声及字幕

Treat transcript or lyric timing as a separate evidence stream. Do not use pitch voicing as vocal detection. Human-review names, lyrics, line boundaries, and timing before they control typography.
Prerecorded synchronized media with meaningful speech needs accurate captions. Captions should include meaningful non-speech sound where needed. A stylized lyric layer does not automatically replace accessible captions or transcript.
将转录或歌词时序视为独立的证据流。请勿使用音高有声化作为人声检测依据。在让名称、歌词、行边界及时序控制排版前，需经过人工审核。
包含有意义语音的预录同步媒体需要准确的字幕。必要时字幕应包含有意义的非语音声音。风格化的歌词层并不能自动替代可访问的字幕或转录文本。

Safety, rights, and provenance

安全、授权及来源

WCAG 2.2 SC 2.3.1 limits flashing above three times in one second unless below general/red-flash thresholds. Test loops while looping and at the largest intended scale. Reduced motion does not make unsafe flashing safe.
Provide a lower-motion version when large displacement, zoom, shake, or dense event response may cause discomfort. Reduce event density and travel, not merely output FPS.
The musical work, lyrics, and sound recording can carry separate rights. Possession of a file does not grant synchronization, adaptation, or distribution rights. Record source and transformation provenance; do not claim metadata proves authenticity or permission.
WCAG 2.2 SC 2.3.1 限制每秒闪烁超过三次，除非低于通用/红光闪烁阈值。循环播放时测试循环效果，并在最大预期尺寸下测试。减少动效并不能使不安全的闪烁变得安全。
当存在大位移、缩放、抖动或密集事件响应可能导致不适时，需提供低动效版本。减少事件密度和移动范围，而非仅降低输出FPS。
音乐作品、歌词及录音可能拥有独立的授权。拥有文件并不意味着获得同步、改编或分发授权。记录来源和转换过程；不得声称元数据可证明真实性或权限。

QA

QA

  1. Re-run identical input/configuration and compare canonical cue-map hashes.
  2. Audition click-marked candidate and promoted anchors.
  3. Inspect tempo-level ambiguity and failed-confidence windows.
  4. Compare source timing against decoded/trimmed master and non-zero timestamps.
  5. Inspect frames immediately before, on, and after every macro anchor.
  6. Verify aspect variants share cue IDs and frame assignments.
  7. Probe output frame PTS, duration, and audio sync.
  8. Review captions/lyrics, reduced motion, and flash safety.
  9. Preserve source hash, profile, cue map, mapping table, render versions, and approvals.
  1. 重新运行相同的输入/配置，并比较标准cue-map哈希值。
  2. 试听标记了点击声的候选锚点和已升级锚点。
  3. 检查tempo层级歧义及置信度失效窗口。
  4. 比较源时序与解码/修剪后的主版本及非零时间戳。
  5. 检查每个宏观锚点前后的帧。
  6. 验证不同宽高比版本共享相同的cue ID和帧分配。
  7. 探查输出帧的PTS、时长及音频同步。
  8. 审核字幕/歌词、低动效版本及闪烁安全性。
  9. 保留源哈希、配置文件、cue map、映射表、渲染版本及审批记录。

Example 1: constant-tempo electronic visualizer

示例1：恒定tempo电子音乐可视化器

This is a complete example, not a mandatory formula.
Intent: 30-second 9:16 and 16:9 visualizer from an authorized 44.1 kHz stereo instrumental at 30 fps.
Approach: confirm stable tempo with a documented analyzer and inspect local pulse. Retain all beat/onset candidates. Promote every fourth beat for medium choreography, reviewed top-strength transients for sparse accents, and reviewed recurrence changes for macro transitions. Beat phase drives scale only from 1.000 to 1.035; low-band energy controls bounded depth; centroid controls a narrow texture-density range. A six-beat low-energy span holds title copy.
Map all anchors to the first frame at or after their sample time. Both aspect variants use identical cue IDs. QA click tracks, rerun hashes, PTS, duration, final-size text, and full-screen flashing.
Likely failure: half/double-tempo ambiguity. Repair by documenting the metrical level selected for visual periodicity without rewriting the raw detections.
这是完整示例，而非强制公式。
目标： 从授权的44.1 kHz立体声器乐制作30秒、30 fps的9:16和16:9可视化器。
方法： 使用已记录的分析器确认稳定tempo，并检查局部脉冲。保留所有beat/onset候选。每隔四个beat升级为中观编排锚点，审核通过的高强度瞬态升级为稀疏重音锚点，审核通过的重复变化升级为宏观过渡锚点。Beat相位仅驱动1.000至1.035的缩放比例；低频能量控制受限的深度；质心控制窄范围的纹理密度。六beat的低能量时段用于展示标题文本。
将所有锚点映射到采样时间对应的第一帧或之后的帧。两种宽高比版本使用相同的cue ID。QA点击音轨、重新运行哈希值、PTS、时长、最终尺寸文本及全屏闪烁情况。
可能的故障： 半速/倍速歧义。修复方法为记录为视觉周期性选择的节拍层级，而非改写原始检测结果。

Example 2: rubato spoken word

示例2：自由节奏口语作品

This is a complete example, not a mandatory formula.
Intent: 75-second poem with ambient bed at 24 fps.
Approach: global beat trackers disagree, so use the non-rhythmic route. Reviewed transcript line starts/ends are primary anchors; pauses reset the field; sustained loudness rises control subtle expansion; acoustic-change candidates remain review prompts. Each stanza establishes a stable visual field, and punctuation settles motion. No beat cuts or word-by-word scaling.
The vertical variant reflows text but retains timing. QA muted-caption comprehension, audio-only clarity, every line boundary, breath-adjacent silence, reduced motion, flashing, and separate text/recording rights.
Likely failure: an acoustic boundary lands inside a sentence. Keep it as a low-confidence event and reject it as an editorial anchor.
这是完整示例，而非强制公式。
目标： 制作75秒、24 fps的带氛围背景的诗歌视频。
方法： 全局beat跟踪器结果不一致，因此使用非节奏路由。审核通过的转录行起始/结束为主要锚点；停顿重置视觉场；持续响度上升控制细微扩展；声学变化候选仅作为审核提示。每个诗节建立稳定的视觉场，标点符号终止运动。不使用beat剪辑或逐字缩放。
垂直版本重新排版文本但保留时序。QA静音字幕的可理解性、纯音频清晰度、每个行边界、呼吸停顿、低动效版本、闪烁情况及文本/录音的独立授权。
可能的故障： 声学边界落在句子内部。将其标记为低置信度事件，并拒绝作为编辑锚点。

Sources

来源

Verified 2026-07-12:
2026-07-12验证：