| Delivery | Coverage | Duration | KPIs |
|---|---|---|---|
| Sigma Spain, Remote | PT-PT + EN | 1-Day Quality Test | Creative Balance, Accuracy, Lead Time, Source Freshness |
The Problem
Models trained on unverified data learn to guess with confidence. The client needed a Document QA and table dataset that forced every answer to come strictly from the source text, in Portuguese, with nothing invented and nothing pulled from outside knowledge.
- A single wrong or too-simple example doesn''t just weaken the dataset. It signals the vendor can''t be trusted with the next batch.
- This wasn''t a production run. It was a 5-task test the client used to decide whether Sigma got the project at all.
The Insight
Teaching people what makes a good question is only half the guideline. The harder half is teaching them what a lazy question looks like, so they stop writing it.
- Built an explicit "what not to do" library: no simple lookups, no summaries, no questions the text can''t answer.
- Paired it with "what to do" examples: multi-fact questions that force the model to search and connect across a passage, not just find one line.
- Restricted every source to a short, cleared list — Wikipedia, Wikinews, Client Manuals, EUR-Lex — so nothing in the dataset carries copyright risk.
- Banned generative AI tools at every stage of writing questions and answers, with contract-level consequences for using one.
The Stakes
If annotators write easy questions, the model looks capable on a test it was never actually challenged by. If they invent an answer or source one outside the text, the client inherits a dataset that feeds the exact hallucination problem it was built to fix.
What We Built
Annotators were screened for bilingual fluency (Portuguese from Portugal, working English) and creativity first, then trained on how AI models fail, so they could recognize which questions would correctly test one.
- Document QA: sourced a text fragment, wrote a context-bound question, wrote the reference answer as either a lightly-edited excerpt or a synthesized summary depending on length.
- Tables: converted structured data to prose and prose to tables, always organizing only the data given, never adding a row the source didn''t have.
- Logged every task against the platform''s own timer, by duty and not by task, to keep the client''s invoice and Sigma''s billing an honest match.
Mid-test, Sigma caught and fixed:
- A platform bug where opening the Tutorial tab wiped in-progress work — guidelines moved to a standalone PDF so no one lost a task by reading instructions.
- A timer habit that would have under-reported active work if left uncorrected.
- The gap between technically correct and actually hard in early draft questions, closed with a sharper kickoff on how models behave under easy versus difficult prompts.
Service Workflow
Five steps, in order, on every task — nothing skipped, nothing reordered:
| 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|
| Source Selection | Prompt Engineering | Reference Authoring | Quality Audit | Automated Benchmarking |
What We Measured
| KPI | What It Tracks |
|---|---|
| Creative Balance | Ratio of creative, inferential prompts to simple factual prompts, within the allotted time per task |
| Annotation Accuracy | Share of reference answers 100% supported by the source text, with no outside bias |
| Task Lead Time | Average time to turn a one-page source fragment into a concise, accurate summary answer |
Why it Matters Now
- A frontier lab localizing a multimodal assistant into a new market needs native fluency and copyright-clean sourcing in the same annotator, not two separate vendors.
- A lab training on mixed text-and-table inputs needs a team that won''t quietly average the two skills — question design and structured-data discipline are different muscles, and this test proves Sigma hires for both.
- A lab that pilots vendors before awarding volume gets a template here: a small, graded test that surfaces tooling gaps and skill gaps before either one is expensive to fix.
A dataset built to stop hallucinations doesn''t get to be written by a machine, or by someone who didn''t notice the question was too easy to ask.
