SIGMAAI
Case studies
Case study

The Assistant Is Only as Good as Its Grader

Validating AI-generated answers for two real-time, multimodal assistant prototypes — EN-US pilot, Madrid Secure Facility

The Assistant Is Only as Good as Its Grader
DeliveryCoveragePricingDurationKPIs
Madrid Secure FacilityEN-USTime & MaterialPilotFactuality, Coherence, Valid Responses

The Problem

Two prototype assistants could see, hear, and act in real time. Neither had a reliable way to prove their answers were right.

  • Category labels show what an assistant said, not whether it should have said it.
  • A multimodal assistant that takes actionable steps turns a wrong answer into a wrong action.

The Insight

Grading an answer isn't data entry. It's fact-checking and paraphrasing, applied segment by segment, sourced against live search when the tool didn't already know.

  • Corrected user-segment transcription and segmentation while watching the video and listening to the audio.
  • Categorized every assistant segment against the assistant's own answer taxonomy.
  • Built one to three valid and invalid alternate responses per assistant answer, factually checked, to test the delivered answer against real alternatives.
  • Ran the second prototype through category-only comparison with no content creation to benchmark head-to-head.

The Stakes

An assistant that misreads what's in front of it isn't a labeling error. It's a liability the moment it acts on it.

What We Built

Candidates were selected first for creativity, fact-checking, and paraphrasing, not transcription speed.

Mid-pilot, Sigma proactively flagged and corrected issues no KPI dashboard would have caught on its own:

  • Missing labels in the client's own taxonomy.
  • Screen real estate that shrank the transcription window.
  • Task timing set shorter than the work required.
  • No autosave — interrupted tasks lost all progress.
  • Paralinguistic markup and duplicated segments in source transcripts, cleaned manually before grading could start.

Why it Matters Now

  • A frontier lab powering a real-time, multimodal assistant at consumer scale needs a fact-checking layer that catches drift before launch, not after.
  • A governance layer for an agentic assistant needs a working template for grading right data source, correct answer, appropriate action — the same three-part test this pilot already ran twice.

An assistant that acts is only as trustworthy as the person who checked its answer.