| Delivery | Coverage | Pricing | Duration | KPIs |
|---|---|---|---|---|
| Madrid Secure Facility | EN-US | Time & Material | Pilot | Factuality, Coherence, Valid Responses |
The Problem
Two prototype assistants could see, hear, and act in real time. Neither had a reliable way to prove their answers were right.
- Category labels show what an assistant said, not whether it should have said it.
- A multimodal assistant that takes actionable steps turns a wrong answer into a wrong action.
The Insight
Grading an answer isn't data entry. It's fact-checking and paraphrasing, applied segment by segment, sourced against live search when the tool didn't already know.
- Corrected user-segment transcription and segmentation while watching the video and listening to the audio.
- Categorized every assistant segment against the assistant's own answer taxonomy.
- Built one to three valid and invalid alternate responses per assistant answer, factually checked, to test the delivered answer against real alternatives.
- Ran the second prototype through category-only comparison with no content creation to benchmark head-to-head.
The Stakes
An assistant that misreads what's in front of it isn't a labeling error. It's a liability the moment it acts on it.
What We Built
Candidates were selected first for creativity, fact-checking, and paraphrasing, not transcription speed.
Mid-pilot, Sigma proactively flagged and corrected issues no KPI dashboard would have caught on its own:
- Missing labels in the client's own taxonomy.
- Screen real estate that shrank the transcription window.
- Task timing set shorter than the work required.
- No autosave — interrupted tasks lost all progress.
- Paralinguistic markup and duplicated segments in source transcripts, cleaned manually before grading could start.
Why it Matters Now
- A frontier lab powering a real-time, multimodal assistant at consumer scale needs a fact-checking layer that catches drift before launch, not after.
- A governance layer for an agentic assistant needs a working template for grading right data source, correct answer, appropriate action — the same three-part test this pilot already ran twice.
An assistant that acts is only as trustworthy as the person who checked its answer.
