Why gen AI quality requires rethinking human annotation standards
From accuracy to agreement: A new lens on quality Traditional AI annotation tasks (e.g. labeling a cat in an image) tend to yield high human agreement and low error rates. Annotators working with clear guidelines often achieve over 98% accuracy — sometimes even 99.99% — especially when backed by tech-assisted workflows. But these standards don’t […]
Precision in data annotation: What’s needed for gen AI models
Precision in gen AI data annotation Gen AI models learn to create novel content. However, for these models to be reliable and useful, their content should be grounded in accurate information and logical structures. In gen AI data annotation, precision extends beyond accurate facts; it also encompasses creativity and nuance. Precise outputs should be factually […]
Human touch in gen AI: Training models to capture nuance
Humanity in gen AI data annotation Data annotation is not just about accuracy and precision. It requires human expertise and careful oversight to ensure AI models interact with the world in a meaningful, relevant, and responsible way. Drawing from our most recent whitepaper, “Beyond accuracy: The new standards for quality in human data annotation for […]
Accelerating the new AI: Key insights from our latest whitepaper
The key to gen AI success? High-quality data powered by human expertise Building accurate and reliable generative AI models demands vast amounts of high-quality training data. Achieving this is easier said than done: it requires the right blend of efficient workflows, deep-domain knowledge, and human oversight. With over 30 years at the forefront of AI […]
Beyond words: 10 subtle layers of human context AI still struggles to understand
Irony and sarcasm What it is: Saying the opposite of what is meant, often with a tonal cue. Example: “Oh, fantastic job…” said with clear frustration. Why machines miss it: Literal interpretation of words leads to mislabeling intent. Pragmatic implicature What it is: Inferring meaning beyond explicit words, based on context. Example: “It’s cold in […]
Behind the scenes of creating the book ‘Nature’s Palette’ with AI
The concept: Blending art, science, and AI We began by brainstorming coffee table book ideas that would appeal to a broad, international adult audience. AI generated a host of possible topics and then provided additional data on the popularity of certain topics and categories. Humans narrowed these options down to the most appealing. A book […]
Golden datasets: Evaluating fine-tuned large language models
What is a golden dataset? A golden dataset is a curated collection of human-labeled data that serves as a benchmark for evaluating the performance of AI and ML models, particularly fine-tuned large language models. Because they are considered ground truth — the north star for correct answers — golden datasets must contain high-quality data that […]
How gen AI is transforming the role of human data annotation
5 key challenges of human data annotation in the gen AI era The potential of the global data collection and labeling market is immense, with a projected revenue of US$17 billion by 2030, growing at nearly 30% annually. Domain-specific models are driving rapid growth in specialized industry sectors, such as healthcare. Here’s why human data […]
Scaling generative AI: How companies are harnessing its power
Scaling generative AI: Benefits, risks, and limitations Before generative AI, traditional AI technology focused on solving well-defined problems. These traditional AI models were designed for specific tasks, such as text classification, entity extraction, and predictive modeling, limiting business applications to narrow domains. A glimpse back at McKinsey’s State of AI in 2022 reveals popular enterprise AI use […]