For a natural understanding project, a major global technology player faced a daunting data collection task: capture over 1,000 authentic, flowing conversations, all within an incredibly tight two-month deadline.

Their approach consisted of providing speakers with games to play as prompts, where participants would have to describe an image or a word to their conversation partner. This would encourage natural, free-flowing conversations, not limited to a script.

Finding the optimal balance of diverse speakers

The first challenge was sourcing speakers of very specific dialects — they needed not just native speakers, but those with at least 10 years of residency in the dialect’s region, and a broad representation of ages and genders.

Sigma was able to tap into its existing pool of over 25,000 annotators and linguists to quickly source speakers of the requested dialects with the required local experience.

But there were still a few challenges to solve. Given such a tight deadline, how could the dataset include the right balance of ages and genders needed to train the algorithm for all possible circumstances? Sigma collaborated with the client to identify the optimal distribution, maximizing speaker diversity while acknowledging real-world constraints. Their speech and language AI expertise also allowed Sigma to advise on incorporating diverse vocal pitch, timbre, and speaking speeds.

Project and product teams collaborate to deliver at scale

Right from the start, Sigma understood that efficient workflows were essential for delivering the project at scale and on time. This required a concerted effort from project managers and product development to optimize tools and processes. Sigma’s commitment to customization — of tools, teams, and processes — ensured they could adapt to the project’s specific demands and satisfy the client.

Sigma’s proactive approach to project management, with constant feedback loops with the client and participants, allowed them to anticipate and address potential issues before they escalated. For instance, seeing that speakers were struggling with multiple user interfaces, the product development team quickly integrated them into a single, seamless platform. Project managers then created and embedded video tutorials within the interface. These adaptations resulted in smoother participation, quicker recording setup, and more natural conversations.

Automated pair matching saves 25% of project management time

Two key challenges emerged in the data collection process. First, the logistics of scheduling gameplay between a large number of participants with different availabilities, while ensuring unique pairings for each 45-minute conversation. Second, maintaining participant motivation for these relatively short sessions. 

The most essential piece of the optimization puzzle was automating the process of matching unique pairs of players together. The product team created a custom interface to match participants’ availability and automatically send them calendar invitations, optimizing the most labor-intensive part of the project management process and saving 25% of the total project management time. 

This also solved the question of motivation – participants could set their own schedules, lose no time on overhead and coordination, and could be sure that their next conversation would happen as soon as possible. It also greatly improved the speakers’ satisfaction with the project and continued participation, because their game partners were more likely to show up consistently and on time thanks to the efficient, automated meeting planner.