SIGMAAI
Case studies
Case study

Confidence Is a Number, Not a Feeling

An internal pilot testing whether an AI agent could safely triage video QA at scale, built after a 3,500-video, 6-week RFP landed, before committing to the timeline

Confidence Is a Number, Not a Feeling
DeliveryCoveragePricingDurationKPIs
Internal pilot, Sigma-ledEN, video + audioPer video (AI review)6 weeksAgreement, Precision, Safe-Automation Rate

The Problem

A major global tech company issued an RFP for 3,500 fully-annotated videos in 6 weeks. Manual review of every submission doesn''t scale to that volume, and off-the-shelf automation can''t weigh subjective, multimodal quality the way a person can — is this description reasonable, does this rating make sense, is this good enough.

  • Simple pass/fail automation misses nuance. Reviewing everything by hand costs 1:1 with volume, which breaks at 3,500 videos in six weeks.
  • Rather than bid the project blind, Sigma built and tested the QA agent internally first, on a small controlled dataset, before committing to that timeline live.

The Insight

Not every submission needs the same scrutiny. The agent should grade its own certainty, and only hand off the cases it isn''t sure about.

  • Built a confidence-based triage: the agent scores its own certainty (0.0 to 1.0) on every answer, and only uncertain or rejected cases route to a human.
  • Seeded known errors into half the test set deliberately, so accuracy could be measured against a real answer key, not gut feel.
  • Built the agent to self-audit: track where it disagreed with the human reviewer, group the mismatches into patterns, and revise its own prompt from there.
  • Shadow-tested every prompt revision against past videos before promoting it into the live workflow, so a change never shipped on faith.

The Stakes

If AI review runs too lenient, factually wrong labels ship into training data — the same hallucination problem this kind of system exists to prevent elsewhere. If it runs too strict, the volume savings disappear and cost matches manual review dollar for dollar.

What We Built

Sourced 500 candidate clips from YouTube-8M (Apache 2.0, permissively licensed) and selected 200 for testing. Two annotators built the set deliberately: faithfully answering half the videos'' questions, and planting intentional errors in the other half, so the agent''s error detection could be scored against a known answer key.

  • Evaluation standard, three parts: factually accurate, what a reasonable person would say, acceptable quality.
  • Confidence-scored decisions routed automatically: maximum-confidence approvals shipped without review, with periodic audit sampling; everything else went to a human.
  • A lean team by design: three annotators, plus one Sigma team member covering review, prompt engineering, and every other role needed to move fast on a self-funded pilot.

Kept one piece of scope narrow on purpose: for this test, a human reviewer''s correction was treated as final. In a live engagement, that same case would get one more AI pass or a second human reviewer before it''s called done.

Service Workflow

Five steps, in order, on every task — nothing skipped, nothing reordered:

12345
Submission IntakeBasic ValidationAI Confidence ReviewHuman Review of Edge CasesSelf-Reflection & Prompt Refinement

What We Measured

KPIResults
Agreement Rate93.4% overall agreement between the AI''s evaluation and the human reviewer''s, across every case tested
Precision at Max Confidence99% accuracy when the AI approved a submission with a 1.0 confidence score, its highest tier
Safe Automation RateRose from about 61% to 64.1% of submissions safely approved without human review, over 7 to 8 self-reflection cycles
Cost Per Video$0.0032 average cost of an AI-handled review decision

Why it Matters Now

  • An agentic governance layer needs proof an agent can grade its own trajectory, catch its own mismatches, and safely route only the uncertain cases to a human. This pilot''s self-reflection loop is that pattern, tested and measured.
  • A platform expanding into narrated, multimodal video content at scale needs a QA layer that judges subjective quality faster than manual review allows. This is that same judgment call, quantified.
  • A team building agentic retrieval or dataset governance needs confidence-based routing that knows when to trust itself and when to ask for a second opinion, with any change shadow-tested before it goes live.
  • A multimodal assistant reasoning over photos, screens, and context needs the same discipline proved here to score its own confidence, escalate what it''s unsure of, and improve from disagreement with human graders.

The point of a confidence score isn''t speed. It''s knowing exactly which decisions still need a person.