| Delivery | Evaluators | KPIs |
|---|---|---|
| Managed HITL secure facility | Up to 5 independent evaluators cross-checking each clip's trigger call | Criteria completeness, trigger accuracy |
The Problem
A proactive agent has to act without being asked. Most fail at the moment they're supposed to act, not at the answer itself.
- Reactive models wait for a prompt. Autonomous agents in wearables and robotics don't get one. They have to notice a visual or situational trigger and decide to act on their own.
- Interrupting at the wrong moment reads as a malfunction. An agent that speaks too early, too late, or over a task the user never needed help with breaks trust as fast as a wrong answer does.
- A single graded response hides the harder failure. Grading only what an agent said misses whether it should have said anything, and when.
The Insight
Teaching a model when to act starts with isolating the trigger, not grading the response, then capturing the human judgment behind that call in a form the model can train on.
- The trigger must be identified before anything else can be graded. Annotators find the exact user query or visual event complex enough to justify a proactive response, before touching the assistant's side of the interaction.
- A good response and a correct response aren't the same test. Success Criteria define what the assistant must include to be useful. Failure Criteria define what would make an autonomous action actively wrong, a distinct judgment call from grading the words.
- Visual context is part of the ground truth. The Relevant Context field captures what's on screen driving the trigger, so the training signal reflects why a proactive action was justified, not just that one occurred.
The Stakes
- A model with no failure signal keeps improvising past its guardrails. Success criteria without failure criteria gives a reward signal with no downside to avoid.
- An inconsistent dataset teaches the wrong lesson uniformly. If required and optional response elements aren't distinguished consistently across annotators, the model learns to treat both as equally necessary or equally optional.
What We Built
- A structured segmentation model built around triggers, not transcripts. Annotators isolate the Relevant Video segment and the Relevant User Query within it, capturing the same read a person makes instinctively, that this moment calls for a response, as a labeled data point before the Relevant Assistant Response fields are filled in.
- Explicit dual-trajectory annotation. Success Criteria, split into required and optional elements, and Failure Criteria captured separately, turning an annotator's judgment of a good outcome and a bad one into two distinct, model-usable signals.
- Kickoffs built around demonstration, not just documentation. Annotators walked through video-based examples and cleared questions before starting, with PMs weighting quality over speed from the outset.
- A client-derived style guide enforced per field. Phrasing rules matched to whether an annotator was writing Relevant Context, Success Criteria, or Failure Criteria, so tone stayed consistent across annotators.
- A pre-submission checklist built into the workflow. Annotators had to confirm every required field before the system allowed a submission, catching incomplete entries before they reached QA.
Why it Matters Now
- Agentic AI, smart glasses, and robotics teams need models that know when to act, not just what to say. Trigger segmentation turns a person's judgment of the right moment to intervene into training data the model can learn to reproduce directly, a human call made explicit rather than a pattern inferred after the fact.
- Alignment and safety teams get an explicit negative signal. Failure Criteria capture a trained annotator's judgment of what would make an autonomous action wrong, giving RLHF and DPO pipelines guardrail data drawn from a human call, the kind of signal a success-only dataset never produces.
- Multimodal by construction. Video, speech, and response span are triangulated in the same annotation pass, rather than graded as separate signals stitched together after the fact.
An agent that only learns what to say next never learns when to stay quiet.
