| Coverage | Pricing | Replication | KPIs |
|---|---|---|---|
| 10 locales, EN-paired | Per hour | 5-way | Voice identity, accent fidelity, justified diagnostics |
The Problem
A voice model can switch from English into another language mid-sentence. Getting the words right stopped being the hard part.
- Transcription confirms what a model said, not whether it still sounds like the same speaker once it crosses into the other language.
- A code-switch that drops a native accent, drifts into robotic prosody, or shifts to a different timbre reads as broken to a listener, even when every word transcribes clean.
- Three teams need three different things. Client teams need the voice to hold its identity. Product teams need the accent right on both sides of the switch. Data operations need a scoring trail they can act on, not a single number.
- Sigma's answer: native-ear evaluation of code-switched speech across ten locales, judging whether the persona holds across the language boundary, not just whether the words are audible.
The Insight
Judging a code-switch is a native-ear call twice over: does the English hold up, and does the target language hold up, in the same breath.
- The two halves were scored separately. Locale-fluent evaluators judged the target-language half against a five-point native accent scale, run apart from the English half, since a bilingual native ear catches a slip a monolingual one would miss.
- Any sub-optimal score triggered a mandatory justification, enforced by conditional logic built into the platform rather than left to an optional comment box. Every flagged clip came back with a reason an engineering team could act on.
- "Native" isn't fixed. Whether an Indian-accented English reading counted as native or foreign depended on what the assistant was being built to sound like in that batch, so annotators worked from per-batch client guidance rather than one blanket rule.
- One-sided and side-by-side batches asked different questions. One-sided batches judged a single voice against a pronunciation-teaching bar. Side-by-side batches judged two voices head-to-head, where a monolingual clip inside an expected language pair wasn't a defect to skip. It was a data point, and the guidance said so explicitly after early batches skipped it by mistake.
The Stakes
- In the one-sided batches, the audio doubles as a pronunciation teacher. A learner hears what "correct" sounds like and imitates it. A model that produces a foreign accent teaches that accent forward, past the moment a listener hears it once.
- An assistant that reads as a different speaker mid-sentence signals something wrong before a listener can name what it is. That failure shows up as a persona that didn't hold, not as a labeled error, which makes it easy to miss and hard to trace back to a fix.
What We Built
- Native-ear evaluation across ten locales: de-DE, hi-IN, fr-FR, ko-KR, es-ES, id-ID, ja-JP, pt-BR, ar-XA, and it-IT, each paired against English, with the accent scale run separately for both halves of every clip.
- Judgment on persona continuity, not just clarity. A generalist crowd catches static and mispronunciation. Ours asks whether a voice still sounds like the same person after the switch, and puts that question in front of native speakers rather than a general labeling pool.
- Root-cause justification built into the platform logic itself, so a "poor" score with no explanation was structurally impossible rather than just discouraged.
- Skip logic tight enough to protect the data. A clip that stayed entirely in English wasn't waved through as broken. It was scored as a real data point about how the model behaved with no switch to make, once the guidance was corrected mid-pilot to say so.
- A team built at triple the headcount the 5-way replication called for on paper, because a Monday-load, Thursday-deadline pilot doesn't leave room to discover mid-week that a locale is short-staffed.
- Same-week contingencies handled without slipping the deadline: a missing locale added and priced within a day, accounts reactivated after a client-side deactivation, and internal staff annotating alongside contracted workers to clear a stalled batch.
- Worker-hour audits that separated real effort from an open browser tab. Cross-checking self-reported task rates against completed output found several annotators had left the platform open without working, undercounting their own efficiency rather than gaming the clock.
- A live feedback loop on subtle category boundaries. "Robotic" versus "mechanical" voice is a real distinction to a trained ear, not one two adjectives explain on their own, so the kickoff was updated mid-pilot with audio examples once annotators started asking where the line sat.
Why it Matters Now
- For speech and TTS teams, this is the evaluation layer that catches a persona break before a listener ever hears it, tied to a specific acoustic parameter engineering can act on.
- For data operations leaders, this is proof that human judgment scales under a compressed deadline, even when the locale mix changes without warning.
- For multilingual product leaders, the paired accent scale answers a sharper question than "does it sound native." It answers whether the same voice sounds native on both sides of the switch.
A model can switch languages instantly. Only a native ear can tell you if it just switched people.
