| Delivery | Coverage | Pricing | Duration | KPIs |
|---|---|---|---|---|
| Internal pilot, Sigma-led | EN, video + audio | Per video (AI review) | 6 weeks | Agreement, Precision, Safe-Automation Rate |
The Problem
A major global tech company issued an RFP for 3,500 fully-annotated videos in 6 weeks. Manual review of every submission doesn''t scale to that volume, and off-the-shelf automation can''t weigh subjective, multimodal quality the way a person can — is this description reasonable, does this rating make sense, is this good enough.
- Simple pass/fail automation misses nuance. Reviewing everything by hand costs 1:1 with volume, which breaks at 3,500 videos in six weeks.
- Rather than bid the project blind, Sigma built and tested the QA agent internally first, on a small controlled dataset, before committing to that timeline live.
The Insight
Not every submission needs the same scrutiny. The agent should grade its own certainty, and only hand off the cases it isn''t sure about.
- Built a confidence-based triage: the agent scores its own certainty (0.0 to 1.0) on every answer, and only uncertain or rejected cases route to a human.
- Seeded known errors into half the test set deliberately, so accuracy could be measured against a real answer key, not gut feel.
- Built the agent to self-audit: track where it disagreed with the human reviewer, group the mismatches into patterns, and revise its own prompt from there.
- Shadow-tested every prompt revision against past videos before promoting it into the live workflow, so a change never shipped on faith.
The Stakes
If AI review runs too lenient, factually wrong labels ship into training data — the same hallucination problem this kind of system exists to prevent elsewhere. If it runs too strict, the volume savings disappear and cost matches manual review dollar for dollar.
What We Built
Sourced 500 candidate clips from YouTube-8M (Apache 2.0, permissively licensed) and selected 200 for testing. Two annotators built the set deliberately: faithfully answering half the videos'' questions, and planting intentional errors in the other half, so the agent''s error detection could be scored against a known answer key.
- Evaluation standard, three parts: factually accurate, what a reasonable person would say, acceptable quality.
- Confidence-scored decisions routed automatically: maximum-confidence approvals shipped without review, with periodic audit sampling; everything else went to a human.
- A lean team by design: three annotators, plus one Sigma team member covering review, prompt engineering, and every other role needed to move fast on a self-funded pilot.
Kept one piece of scope narrow on purpose: for this test, a human reviewer''s correction was treated as final. In a live engagement, that same case would get one more AI pass or a second human reviewer before it''s called done.
Service Workflow
Five steps, in order, on every task — nothing skipped, nothing reordered:
| 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|
| Submission Intake | Basic Validation | AI Confidence Review | Human Review of Edge Cases | Self-Reflection & Prompt Refinement |
What We Measured
| KPI | Results |
|---|---|
| Agreement Rate | 93.4% overall agreement between the AI''s evaluation and the human reviewer''s, across every case tested |
| Precision at Max Confidence | 99% accuracy when the AI approved a submission with a 1.0 confidence score, its highest tier |
| Safe Automation Rate | Rose from about 61% to 64.1% of submissions safely approved without human review, over 7 to 8 self-reflection cycles |
| Cost Per Video | $0.0032 average cost of an AI-handled review decision |
Why it Matters Now
- An agentic governance layer needs proof an agent can grade its own trajectory, catch its own mismatches, and safely route only the uncertain cases to a human. This pilot''s self-reflection loop is that pattern, tested and measured.
- A platform expanding into narrated, multimodal video content at scale needs a QA layer that judges subjective quality faster than manual review allows. This is that same judgment call, quantified.
- A team building agentic retrieval or dataset governance needs confidence-based routing that knows when to trust itself and when to ask for a second opinion, with any change shadow-tested before it goes live.
- A multimodal assistant reasoning over photos, screens, and context needs the same discipline proved here to score its own confidence, escalate what it''s unsure of, and improve from disagreement with human graders.
The point of a confidence score isn''t speed. It''s knowing exactly which decisions still need a person.
