Gen AI data roadmap to kickstart your journey
1 – Preparing for gen AI begins with a data strategy
Data is the fuel of AI. For companies to fully leverage the potential of this technology, a strong data foundation is imperative. This involves addressing data management issues related to quality, security, transparency, integration, storage, and privacy, before rolling out a generative AI project.
A Deloitte survey among AI early adopters reveals that 75% of them “have increased their technology investments around data life cycle management due to gen AI.” Many of these companies have encountered unforeseen data challenges while scaling their AI initiatives. In fact, for 55% of the surveyed organizations, data-related issues caused them to avoid certain gen AI use cases.
Similarly, a MIT Technology Review Insights survey reveals that 72% of executives agree that data problems are the factor most likely to jeopardize their AI or ML goals.
So, how can businesses prepare their data strategies for gen AI?
- Enhance data quality practices, such as labeling or metadata tagging on internal company data to provide context. When it comes to AI readiness, the concept of ‘data liquidity’ — ensuring that data is accessible, usable, structured, and ready to be shared and combined — is a key differentiator.
- Develop an IT architecture that can integrate and support a growing volume of structured and unstructured data throughout its entire lifecycle.
- Identify the main data security risks related to business and customer data and set up ongoing mitigation practices.
- Invest in upskilling to build a capable workforce, focusing on technical profiles such as data engineers, system architects, software developers, and security experts.
2 – Building a ‘lighthouse’: Start with a pilot
A lighthouse project, a small-scale proof of concept, is a strategic approach for quickly evaluating the potential value and impact of generative AI initiatives. According to McKinsey, “encouraging a proof of concept is still the best way to quickly test and refine a valuable business case before scaling to adjacent use cases.”
Adopting a lighthouse approach is a good way to spark enthusiasm across your organization for gen AI and a culture of innovation. By demonstrating early success, companies can build momentum and a competitive edge.
At the data annotation level, creating a pilot can help organizations identify bottlenecks and areas for improvement in their data processes. It can also provide insights on specific data requirements and how to allocate resources effectively to scale the data annotation process.
3 – People, process, value: The 3 pillars of successful data annotation
Companies don’t have to do everything by themselves. Finding the right partners for your gen AI initiatives can be an excellent way to accelerate execution.
Data annotation, for instance, is one of the most time-consuming aspects of developing a generative AI model and requires high precision, accuracy, and domain expertise. It also might require a substantial workforce of qualified domain experts that can be ramped up and then ramped down, depending on the company’s demands. Most companies just don’t have that staff flexibility.
Jean-Claude Junqua identifies three pillars for successful data annotation and validation: people, process, and value.
Let’s take a closer look at each of them:
People
Effective data annotation for gen AI demands a diverse crowd of human annotators with precise skills and domain expertise that varies for each project.
For instance, Sigma AI’s approach to assembling data annotation teams involves a rigorous skill assessment process during the recruitment stage. Upon onboarding, annotators engage in a continuous feedback loop, receiving ongoing training and providing valuable insights to refine the data annotation process.
A human-in-the-loop methodology is crucial for mitigating ambiguities, validating data annotation, and iteratively improving the initial annotation guidelines and output.
Process
While human judgment and expertise are key for data annotation, well-structured processes are equally critical. “Companies must prioritize quality, establish a feedback loop, integrate automation tools, and ensure the accuracy of the data they provide,” Junqua says.
The first step in establishing robust processes is to clearly define the desired outcomes of your generative AI project. What’s the problem you are trying to solve? “Often, clients have a clear vision of their goals but may lack clarity on the specific steps involved, particularly to obtain the data they need,” explains Junqua. That’s why Sigma AI’s project managers play an important role in bridging that gap between what outcome is desired and what process is delivered.
Processes provide a framework for achieving specific goals, ensuring quality, and mitigating risks. These are some of the key processes of a data annotation project:
- Data gathering and preparation: Collect and analyze data. Ensure domain coverage, quality, consistency, and relevance
- Develop clear and comprehensive guidelines for annotators
- Implement quality assurance mechanisms to monitor and maintain data quality
- Establish review cycles for continuous feedback and improvement
- Leverage automation tools to speed up repetitive annotation tasks and improve efficiency
- Augment datasets with synthetic data if necessary
Value
With human expertise and well-structured processes in place, the next critical step is integrating an ethical lens into data annotation and validation.
Generative AI can be a transformative force for society, but it can only amplify the quality of the data it learns from. To create ethical and responsible AI that truly serves people, it’s key to prioritize the following areas:
- Ethical data sourcing: Ensure that data is obtained through ethical and legal processes, avoiding biases, equally representing all users, and protecting privacy.
- Bias mitigation: Identify and address biases within the data to prevent discriminatory or harmful outcomes. Assess the coverage and balance of training data, and ensure that annotation teams represent all groups within a population.
- Transparency and accountability: Be transparent about AI’s decision-making processes. Evaluate the impact of choices made in the creation of a model.
- Human oversight: Embrace a human-in-the-loop approach to ensure AI is used responsibly and ethically.
- Continuous evaluation: Regularly assess the ethical implications of AI’s applications and make necessary adjustments.
4 – Creating clear annotation guidelines
Data annotation guidelines are a central aspect of the annotation and validation process. Guidelines provide annotators with detailed instructions, definitions, and examples to handle the different situations within an AI project. By strictly following these guidelines, every annotator can be on the same page, ensuring consistency and quality.
With traditional AI, creating clear guidelines was a straightforward process: there was a unique valid response. Generative AI incorporates ambiguity and subjectivity, making annotation guidelines more complex but still crucial. As Abou Jaoude from Sigma AI points out, “A small oversight in a guideline could have catastrophic consequences. In a dataset of thousands, such a mistake could magnify and drastically alter the final outcome.”
Developing guidelines should be approached as a collaborative process between all the stakeholders of the project, incorporating multiple perspectives and encouraging annotators to provide feedback and highlight areas of uncertainty.
“It’s impossible to completely remove subjectivity from generative AI training. The solution? Iteration. You can’t predict every scenario, so start with a baseline, test it, and refine it. Set a goal, and work toward it. Continuously refining your guidelines is key.”
— Valentina Vendola, Manager at Sigma AI
Open and ongoing communication with customers is vital for companies such as Sigma AI to develop effective guidelines that align with specific needs. As Junqua explains, “We need to be able to understand exactly what the customer wants, and this depends on the use case and the output they want from their model.”
When subjectivity enters the equation, the project’s context becomes crucial, says Vendola. In a recent audio annotation project led by Sigma AI, multiple annotators listened to the same recording yet delivered vastly different interpretations due to their different cultural backgrounds. What one perceived as a friendly tone, another interpreted as defiance.
“We need to be clear about what we’re trying to achieve,” Vendola says. “Sometimes a single, definitive answer is best, but in other cases, we might need diverse perspectives.”
“We need to be clear about what we’re trying to achieve. Sometimes a single, definitive answer is best, but in other cases, we might need diverse perspectives.”
— Valentina Vendola, Manager at Sigma AI
5 – Bringing humans to the center
In the era of generative AI, humans take center stage.
When faced with countless possibilities, how do we determine the best output? Let’s think of a summary, for example. Gen AI can provide thousands of summaries for a particular paragraph. The only way to select the most effective is by aligning with what humans actually consider a good summary.
Creativity, empathy, and judgment are inherently human — and that’s precisely why people will continue to be essential for training gen AI models.
“Generative AI has shifted AI from handling repetitive tasks like classification to more complex areas where human judgment is crucial. Humans are the only ones that really can say if gen AI models are doing something that people would do, as these tasks often involve subjective elements,” says Junqua.
A human-in-the-loop approach is crucial at various stages of the annotation and validation process to provide machines with the context and nuance they need to make the right call.
Vendola explains, “In traditional AI, tasks were relatively straightforward, such as spell checking. But generative AI introduces multidimensional challenges. Beyond spelling and grammar, we must evaluate creativity, the relevance of concepts, syntax, and the answer’s overall coherence. How can we teach machines to assess these qualities?”
If traditional AI models were primarily focused on automating repetitive tasks, gen AI is now tackling more complex challenges. However, humans continue to excel at understanding language subtleties, a skill that machines cannot yet fully replicate. As Abou Jaoude says, “We need to be right there to give AI the human touch that it lacks.”