SIGMAAI
Case studies
Case study

The Dub That Passed Every Word, But Still Lost The Speaker

Generative AI dubbing and lip-sync evaluation across thirty locales, with prosody and speaker-persona retention scored against the original English source.

The Dub That Passed Every Word, But Still Lost The Speaker
CoverageScreeningKPIs
30+ localesEnglish + locale proficiency (Medium-High), trustworthinessProsody match, speaker match, failure-mode diagnostics

The Problem

Generative dubbing can get every word right and still lose the person who said them. Prosody, timing, and speaker identity slip in ways a transcript never shows.

  • A dub can pass translation and lip-sync checks and still read as an impersonation of the original speaker instead of a match.
  • Locale drift compounds the risk. es-US and es-ES aren''t interchangeable, and neither are fr-CA and fr-FR, or en-AU and en-GB. A dub can nail one dialect and fail the one the client actually needed.

The Insight

The project architecture: dual-stream video input → managed human-in-the-loop rating (naturalness, clarity, translation, prosody, speaker-persona match, locale fidelity) → a failure-mode taxonomy isolating root-cause errors (hallucinations, pacing, lip-sync).

  • Prosodic benchmarking runs the comparison, not just the clip. Generalist rating pools typically score dubbed audio on its own. Sigma''s raters compared the original and generated videos side by side to judge whether the emotional transfer, not just the translation, held up.
  • Locale isolation was enforced through training, not left to the platform. es-US was judged against es-US, not against a Spanish dub in general — the same discipline applied across all thirty locales, including es-ES, fr-CA versus fr-FR, and en-AU versus en-GB.
  • A rating means nothing without a diagnosis. Where a high-volume rating pool stops at like or dislike, every flagged issue here required a written justification tied to a named failure mode: translation hallucination, voice-character mismatch, acoustic static, pacing.

The Stakes

  • Sigma had no access to the tasks raters submitted on the client''s own platform. Screening and calibration were the only defense against inconsistent judgment reaching the client.
  • A missed dialect check doesn''t surface until a model has already learned from it. One locale''s raters skipped accent evaluation early on; Sigma''s process caught the gap and corrected kickoff guidance before it could recur elsewhere.
  • Long-form video tightened the margin for error. Some clips ran seven to eight minutes, which meant a rater''s target of eight tasks an hour sometimes collided directly with the time it took to watch the material closely enough to judge it.

What We Built

  • Screened linguistic consultants to Medium-High English and Medium-High locale proficiency across thirty locales, plus trustworthiness screening, given Sigma had no visibility into submitted tasks.
  • Built a two-task evaluation flow: rate the dubbed video alone on naturalness, clarity, and overall quality with tagged, justified issues, then compare it directly against the original on prosody match and speaker match.
  • Onboarded the full team onto the client''s own annotation platform, with kickoff training built around the client''s existing task structure rather than a Sigma-built tool.
  • Corrected a dialect-assessment gap mid-project, after one locale''s raters skipped accent evaluation, updating kickoff guidance so every subsequent locale caught it from day one.

Why it Matters Now

  • For video AI and generative media leads, this is the evaluation layer that tests whether a dub retained the emotion, timing, and cadence of the original speaker, not just whether the words translated.
  • For media localization leads, an audit that scores audio and lip-sync alignment together, so a fix traces to the failure that caused it, not a generic quality complaint.
  • For model alignment and safety leads, a named failure taxonomy — translation hallucination, voice-character mismatch, acoustic static — with a mandatory diagnosis behind every flag, built to feed RLHF directly.

A translation can be checked in seconds. Whether it still sounds like the same person takes a human ear listening for the difference.