| Delivery | Coverage | Pricing | Duration | KPIs |
|---|---|---|---|---|
| Remote + secure facility | 12 locales: DE-DE, ES-US, EN-US, EN-GB, JA-JP, EN-IN, FR-FR, HI-IN, ID-ID, IT-IT, KO-KR, PT-BR | Per hour worked | Pilot to 6 month project | Span precision, replication reliability, throughput |
The Problem
A model can generate a laugh, a gasp, a strained pause. It can't yet say exactly when one starts or stops.
- Sentiment buckets don't hold up. Positive, negative, neutral collapses states that behave differently in a live conversation. A startled negative surprise and a charmed positive one call for different responses, not the same label.
- A mistimed non-verbal cue breaks conversational timing. A real-time assistant that treats a held breath as silence responds at the wrong moment, and a wrong moment in a voice interface reads as a malfunction.
- Whole-clip labeling hides the failure. One tag on an entire file says an emotion was present somewhere. It says nothing about where, which is what a model needs to time its own response.
The Insight
Marking a span against a waveform is a different discipline than describing a mood.
- 30+ categories, not 3. Positive, negative, and neutral is how crowdsourced annotation usually treats audio. Sigma's annotators worked against a psycholinguistic taxonomy built to separate adjacent states (contemplation from nostalgia, positive surprise from negative) because a model trained on the coarse version can't tell them apart either.
- Salience has to be taught, not assumed. Early batches over-labeled every audible breath. The guideline drew a hard line to mark only the emotions and non-verbal bursts distinct enough to matter. The team retrained against that standard before continuing.
- The timestamp is the deliverable, not the label. An annotator who can name an emotion but can't place its boundary on the waveform hasn't finished the task.
The Stakes
- An unmarked cue is invisible until a model acts on its absence. A conversational agent that can't isolate a sigh from silence pauses in the wrong place, and no dashboard metric catches that until a user notices.
- A generic mood label doesn't survive a safety review. Auditing an assistant's emotional register requires the specific state an annotator heard, not a bucket that could mean five different things.
- Platform failures compound quietly. Work lost to a broken save function or a mistimed session doesn't just cost hours, it risks compensating annotators for redone work rather than original output, which erodes both budget accuracy and annotator trust.
What We Built
- Locale-screened annotators scaled from a 3-market pilot to twelve. DE-DE, ES-US, EN-US, EN-GB, JA-JP, EN-IN, FR-FR, HI-IN, ID-ID, IT-IT, KO-KR, and PT-BR, each cleared for proficiency plus the English needed to work from source guidelines directly.
- Replication doubled as the program scaled. Single-annotator review at pilot launch moved to two-way replication in later batches, adding inter-annotator reliability once volume justified the cost.
- A throughput floor with room to reward pace. 10 tasks per hour minimum, with annotators encouraged toward 20 per hour — without loosening the salience standard that early batches had already tested.
- A quality cadence matched to how each team worked. Freelance annotators met weekly with PMs to monitor productivity, revisit guidelines, and flag technical issues; on-site teams ran daily flash meetings to catch the same problems in real time.
- A working glossary for the 37-category taxonomy. So annotators encountering an unfamiliar label like "empathetic pain" or "awe" had a reference, not a guess.
- A corrected salience standard, retrained mid-project. Once early batches showed annotators marking every audible sound rather than the emotions meant to register, the guideline was rewritten and the team recalibrated before continuing.
- A rebuilt workflow around platform failures Sigma caught at pilot stage. Labels disappearing on long audio files, session timers resetting below the task length, a save function that didn't preserve progress — each was logged, manually reset by the PM, and fixed before the program scaled past the pilot.
Why it Matters Now
- A real-time multimodal assistant that only parses words is deaf to the events that make a conversation feel natural. Span-level, taxonomy-mapped emotion data closes that gap.
- Research and safety teams get a shared, granular vocabulary. For what a model heard, not just what a transcript says was spoken.
- 178,288 tasks across twelve locales over six months prove the discipline travels at scale. This didn't stay a pilot. The same span-level standard held from the first batch through production volume, which matters the moment a voice interface goes global.
The word for a laugh was never the problem. Knowing where it started was.
