How I built AI-generated debate videos with talking heads, native multilingual generation, and DCC-style kinetic subtitles — in an afternoon.
January 6, 2026 · Tim Jacoby
I wanted to create a video of us debating whether to scale up AI efforts at Meta. You as the experienced bear (measured caution), me as the bull (fortune favors the bold). Real talking heads. Real voices. Native Dutch translation. Professional subtitles. From nothing but our photos and a script.
Result: Two 2-minute videos (English + Dutch) with HeyGen Avatar IV talking heads, ElevenLabs voices with word-level timestamps, and DCC-style kinetic subtitles burned in. Total production time: ~15 minutes of API calls, automated.
The video production module is ~3K LOC out of Kagami's ~483K LOC Python codebase. It leverages the existing character system, voice infrastructure, and media pipeline — demonstrating the power of composable architecture.
┌─────────────────────────────────────────────────────────────────────────────┐ │ VIDEO PRODUCTION PIPELINE │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ SCRIPT (12 exchanges) │ │ │ │ │ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Character │────▶│ Voice │────▶│ Holodeck │ │ │ │ Protocol │ │ (TTS) │ │ (HeyGen) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ metadata.json word_timings[] shot_XXX.mp4 │ │ reference.jpg audio.mp3 │ │ │ │ │ │ │ │ │ ▼ ▼ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ │ Kinetic │ │ FFmpeg │ │ │ │ │ Subtitles │────▶│ Compose │ │ │ │ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ │ subtitles.ass final.mp4 │ │ │ │ │ └─────────────────────────────────────────────────────────────────▶│ │ │ └─────────────────────────────────────────────────────────────────────────────┘ EXTERNAL APIS: • ElevenLabs → TTS with word timestamps (/with-timestamps endpoint) • HeyGen → Avatar IV talking head generation • FFmpeg → Video composition + ASS subtitle burn-in
| Module | LOC | Purpose |
|---|---|---|
characters/voice.py |
706 | ElevenLabs TTS with word-level timestamps |
characters/protocol.py |
352 | Unified character metadata loading |
modes/holodeck.py |
432 | HeyGen Avatar IV integration |
subtitles/kinetic.py |
454 | DCC-style word-by-word reveal subtitles |
generation/dubbing.py |
599 | ElevenLabs dubbing (not used in final) |
produce_ai_debate.py |
440 | Orchestration script |
Each of Kagami's seven cognitive colonies evaluated the architecture independently. Here's the consensus:
| Colony | Grade | Assessment |
|---|---|---|
| 🔥 Spark (Innovation) | 95/100 | Novel integration of 3 APIs into seamless pipeline. Word timestamps from TTS is clever. |
| ⚒️ Forge (Build Quality) | 88/100 | Clean separation of concerns. Good abstractions. Retry logic added. |
| 🌊 Flow (Reliability) | 82/100 | Happy path works well. Error handling improved with graceful degradation. |
| 🔗 Nexus (Integration) | 97/100 | Excellent use of existing character system. Clean API abstractions. |
| 🗼 Beacon (Architecture) | 91/100 | Pipeline pattern is solid. Clear data flow. Well-documented. |
| 🌿 Grove (Documentation) | 85/100 | Code is readable with docstrings. Architecture document complete. |
| 💎 Crystal (Testing) | 92/100 | 105 tests passing. Comprehensive coverage across all modules. |
The /v1/text-to-speech/{voice_id}/with-timestamps endpoint returns character-level alignment data. Building word timings from this requires careful handling of punctuation and whitespace. The stability parameter must be exactly 0.0, 0.5, or 1.0 — custom values fail silently in some SDK versions.
Avatar IV is remarkably good at lip sync from photos. Key gotchas: content-type detection matters (a JPEG file with .png extension fails), and job polling can take 30-60 seconds per shot. Parallelization helps but watch for rate limits.
FFmpeg's ASS filter doesn't support all animation tags (\t transforms often fail). The solution is separate dialogue events per word — creates a word-by-word reveal that works reliably across all players.
Initially I tried dubbing the English video to Dutch. The result was uncanny — English lip movements with Dutch audio. Native generation (Dutch TTS → HeyGen with Dutch audio) produces much better results. Worth the extra API calls.
All items from the remediation plan have been completed:
Want the code? Happy to share the full kagami_studio package. It's ~15K LOC with characters, voices, video generation, subtitles, and more. Works with any ElevenLabs + HeyGen accounts.
This video production pipeline is a small piece of Kagami — a cognitive architecture I've been building that combines active inference, Byzantine consensus across specialized "colonies," and a unified character system for household AI. The video stuff is 0.6% of the codebase, but it demonstrates the power of composable systems: character metadata flows seamlessly from smart home voice control to video production.
The real magic isn't any single API integration — it's having the substrate ready so that "make a video of us debating in Dutch" becomes a tractable afternoon project.