SIGMAAI
Case studies
Case study

The Word for a Laugh Was Never the Problem. Knowing Where It Started Was.

Span-level non-verbal emotion annotation across a 30+ category taxonomy, scaled from pilot to a 12-locale program.

The Word for a Laugh Was Never the Problem. Knowing Where It Started Was.
DeliveryCoveragePricingDurationKPIs
Remote + secure facility12 locales: DE-DE, ES-US, EN-US, EN-GB, JA-JP, EN-IN, FR-FR, HI-IN, ID-ID, IT-IT, KO-KR, PT-BRPer hour workedPilot to 6 month projectSpan precision, replication reliability, throughput

The Problem

A model can generate a laugh, a gasp, a strained pause. It can't yet say exactly when one starts or stops.

  • Sentiment buckets don't hold up. Positive, negative, neutral collapses states that behave differently in a live conversation. A startled negative surprise and a charmed positive one call for different responses, not the same label.
  • A mistimed non-verbal cue breaks conversational timing. A real-time assistant that treats a held breath as silence responds at the wrong moment, and a wrong moment in a voice interface reads as a malfunction.
  • Whole-clip labeling hides the failure. One tag on an entire file says an emotion was present somewhere. It says nothing about where, which is what a model needs to time its own response.

The Insight

Marking a span against a waveform is a different discipline than describing a mood.

  • 30+ categories, not 3. Positive, negative, and neutral is how crowdsourced annotation usually treats audio. Sigma's annotators worked against a psycholinguistic taxonomy built to separate adjacent states (contemplation from nostalgia, positive surprise from negative) because a model trained on the coarse version can't tell them apart either.
  • Salience has to be taught, not assumed. Early batches over-labeled every audible breath. The guideline drew a hard line to mark only the emotions and non-verbal bursts distinct enough to matter. The team retrained against that standard before continuing.
  • The timestamp is the deliverable, not the label. An annotator who can name an emotion but can't place its boundary on the waveform hasn't finished the task.

The Stakes

  • An unmarked cue is invisible until a model acts on its absence. A conversational agent that can't isolate a sigh from silence pauses in the wrong place, and no dashboard metric catches that until a user notices.
  • A generic mood label doesn't survive a safety review. Auditing an assistant's emotional register requires the specific state an annotator heard, not a bucket that could mean five different things.
  • Platform failures compound quietly. Work lost to a broken save function or a mistimed session doesn't just cost hours, it risks compensating annotators for redone work rather than original output, which erodes both budget accuracy and annotator trust.

What We Built

  • Locale-screened annotators scaled from a 3-market pilot to twelve. DE-DE, ES-US, EN-US, EN-GB, JA-JP, EN-IN, FR-FR, HI-IN, ID-ID, IT-IT, KO-KR, and PT-BR, each cleared for proficiency plus the English needed to work from source guidelines directly.
  • Replication doubled as the program scaled. Single-annotator review at pilot launch moved to two-way replication in later batches, adding inter-annotator reliability once volume justified the cost.
  • A throughput floor with room to reward pace. 10 tasks per hour minimum, with annotators encouraged toward 20 per hour — without loosening the salience standard that early batches had already tested.
  • A quality cadence matched to how each team worked. Freelance annotators met weekly with PMs to monitor productivity, revisit guidelines, and flag technical issues; on-site teams ran daily flash meetings to catch the same problems in real time.
  • A working glossary for the 37-category taxonomy. So annotators encountering an unfamiliar label like "empathetic pain" or "awe" had a reference, not a guess.
  • A corrected salience standard, retrained mid-project. Once early batches showed annotators marking every audible sound rather than the emotions meant to register, the guideline was rewritten and the team recalibrated before continuing.
  • A rebuilt workflow around platform failures Sigma caught at pilot stage. Labels disappearing on long audio files, session timers resetting below the task length, a save function that didn't preserve progress — each was logged, manually reset by the PM, and fixed before the program scaled past the pilot.

Why it Matters Now

  • A real-time multimodal assistant that only parses words is deaf to the events that make a conversation feel natural. Span-level, taxonomy-mapped emotion data closes that gap.
  • Research and safety teams get a shared, granular vocabulary. For what a model heard, not just what a transcript says was spoken.
  • 178,288 tasks across twelve locales over six months prove the discipline travels at scale. This didn't stay a pilot. The same span-level standard held from the first batch through production volume, which matters the moment a voice interface goes global.

The word for a laugh was never the problem. Knowing where it started was.