SIGMAAI
Case studies
Case study

Same Voice, Two Languages: Why Human Assessment Matters

Code-switched voice evaluation across ten locales for a multilingual voice assistant, judging whether the same persona holds on both sides of the language switch.

Same Voice, Two Languages: Why Human Assessment Matters
CoveragePricingReplicationKPIs
10 locales, EN-pairedPer hour5-wayVoice identity, accent fidelity, justified diagnostics

The Problem

A voice model can switch from English into another language mid-sentence. Getting the words right stopped being the hard part.

  • Transcription confirms what a model said, not whether it still sounds like the same speaker once it crosses into the other language.
  • A code-switch that drops a native accent, drifts into robotic prosody, or shifts to a different timbre reads as broken to a listener, even when every word transcribes clean.
  • Three teams need three different things. Client teams need the voice to hold its identity. Product teams need the accent right on both sides of the switch. Data operations need a scoring trail they can act on, not a single number.
  • Sigma's answer: native-ear evaluation of code-switched speech across ten locales, judging whether the persona holds across the language boundary, not just whether the words are audible.

The Insight

Judging a code-switch is a native-ear call twice over: does the English hold up, and does the target language hold up, in the same breath.

  • The two halves were scored separately. Locale-fluent evaluators judged the target-language half against a five-point native accent scale, run apart from the English half, since a bilingual native ear catches a slip a monolingual one would miss.
  • Any sub-optimal score triggered a mandatory justification, enforced by conditional logic built into the platform rather than left to an optional comment box. Every flagged clip came back with a reason an engineering team could act on.
  • "Native" isn't fixed. Whether an Indian-accented English reading counted as native or foreign depended on what the assistant was being built to sound like in that batch, so annotators worked from per-batch client guidance rather than one blanket rule.
  • One-sided and side-by-side batches asked different questions. One-sided batches judged a single voice against a pronunciation-teaching bar. Side-by-side batches judged two voices head-to-head, where a monolingual clip inside an expected language pair wasn't a defect to skip. It was a data point, and the guidance said so explicitly after early batches skipped it by mistake.

The Stakes

  • In the one-sided batches, the audio doubles as a pronunciation teacher. A learner hears what "correct" sounds like and imitates it. A model that produces a foreign accent teaches that accent forward, past the moment a listener hears it once.
  • An assistant that reads as a different speaker mid-sentence signals something wrong before a listener can name what it is. That failure shows up as a persona that didn't hold, not as a labeled error, which makes it easy to miss and hard to trace back to a fix.

What We Built

  • Native-ear evaluation across ten locales: de-DE, hi-IN, fr-FR, ko-KR, es-ES, id-ID, ja-JP, pt-BR, ar-XA, and it-IT, each paired against English, with the accent scale run separately for both halves of every clip.
  • Judgment on persona continuity, not just clarity. A generalist crowd catches static and mispronunciation. Ours asks whether a voice still sounds like the same person after the switch, and puts that question in front of native speakers rather than a general labeling pool.
  • Root-cause justification built into the platform logic itself, so a "poor" score with no explanation was structurally impossible rather than just discouraged.
  • Skip logic tight enough to protect the data. A clip that stayed entirely in English wasn't waved through as broken. It was scored as a real data point about how the model behaved with no switch to make, once the guidance was corrected mid-pilot to say so.
  • A team built at triple the headcount the 5-way replication called for on paper, because a Monday-load, Thursday-deadline pilot doesn't leave room to discover mid-week that a locale is short-staffed.
  • Same-week contingencies handled without slipping the deadline: a missing locale added and priced within a day, accounts reactivated after a client-side deactivation, and internal staff annotating alongside contracted workers to clear a stalled batch.
  • Worker-hour audits that separated real effort from an open browser tab. Cross-checking self-reported task rates against completed output found several annotators had left the platform open without working, undercounting their own efficiency rather than gaming the clock.
  • A live feedback loop on subtle category boundaries. "Robotic" versus "mechanical" voice is a real distinction to a trained ear, not one two adjectives explain on their own, so the kickoff was updated mid-pilot with audio examples once annotators started asking where the line sat.

Why it Matters Now

  • For speech and TTS teams, this is the evaluation layer that catches a persona break before a listener ever hears it, tied to a specific acoustic parameter engineering can act on.
  • For data operations leaders, this is proof that human judgment scales under a compressed deadline, even when the locale mix changes without warning.
  • For multilingual product leaders, the paired accent scale answers a sharper question than "does it sound native." It answers whether the same voice sounds native on both sides of the switch.

A model can switch languages instantly. Only a native ear can tell you if it just switched people.