Chris Phillips
Chief Solutions Officer/Business Unit Managing Director, Argos Multilingual
Language-related data collection projects are becoming more and more complex. With the power of neural networks, our clients are ever more specific and targeted with their requirements — forcing us to be more creative during project set-up and with our human-in-the-loop practices. We are also seeing a trend of clients demanding more quality over quantity. Fortunately for us, our programs have always been structured around quality, making us the preferred partner during these changing trends. Here is how we do it.
Sourcing the right talent
You want to ensure that contributors have the necessary expertise, whether it’s for data production or evaluation. Yet governing the demographics of “The Crowd” is challenging within a virtual environment. This means investing in screening, training, profiling, testing, and quality assurance. This will result in higher costs at the start but saves on rework and provides confidence for your clients.
Moreover, the success of NLP crowdsourcing depends on the quality of data fed to it. Even with articulate instructions, continuous quality control, and customized project-specific technology, it’s still a one-sided conversation between you and “The Crowd,” so make sure you are communicating and continuously sampling to guarantee you receive exactly what you need.
Be ready to support the crowd daily. Assign a single person or dedicated team to collect queries and categorize them. Ensure the entire program team meets regularly to review the queries and respond promptly. This will usually involve: supply chain, language leads, project managers, technical support, and finance.
Plan for attrition
Attrition is nearly inevitable in NLP data collection projects. Most of these projects are unique and require specialist expertise sourced specifically for the task. But even when we plan things out carefully, the size and scope of the project may evolve rapidly when you actually get down to work. Sometimes a specialist realizes the project isn’t in their wheelhouse when they begin their work. You may also need to replace people due to non-compliance with guidelines and quality. Budgeting in 10% attrition into your scope is a good starting point, but you need an experienced team to assess this comprehensively based on the project complexity.
Fair pay
As the saying goes: You get what you pay for. So, how do you secure the right talent and pay fair compensation without breaking the bank?
Research fair hourly compensation rates for the target region/locale. You will need different rates depending on the complexity of the task. Many clients will want rates per utterance, annotation, word, etc., so your next step is to test productivity to increase the hourly throughput.
Payment methods
Once you settle on the amount of remuneration allocated to hiring the appropriate resources, decide on the best payment solutions. It may be bank transfer, PayPal, Payoneer, cash, gift vouchers, etc.
How you choose to compensate your workers makes a difference. However, each payment method comes with its own challenges. It’s also important to assess the different fees associated with each method, availability, and talent preferences when devising your payment strategies.
Work closely with your finance team, but the more options you have available, the easier this will be during talent sourcing.
The problem with NLP data collection tools
Most tools focus on input and output. However, we have discovered that the throughput is even more important. If your tooling is cumbersome, your attrition rate will be higher. Trying to develop a catch-all platform is also the wrong approach.
You can’t predict the following project’s specifications, and forcing a project to adapt to the system may work from an input/output perspective but will more than likely impact throughput and productivity. Instead, develop tooling that has foundational functionality for onboarding, assigning, and monitoring but ensure you have resources to adapt the UX for project-specific workflows.
Conclusion
Though a popular and practical approach for collecting data and annotations, NLP crowdsourcing is not without its challenges. However, there are tried and true practices and principles that have proven effective in yielding high-quality data.
-
Top iSEO Mistakes Companies Make
Five recurring international SEO mistakes, from translating keywords instead of researching them to ignoring SEO during a website redesign.
-
How Pinterest Embraces the Change for its 400 Million+ Users
Pinterest separates a universal global message from locally recreated content, treating inclusion and accessibility as prerequisites, not afterthoughts.
-
Tech Innovations to Keep an Eye on: AI TM Cleanup and AI Backtranslation
AI TM cleanup and AI back translation apply semantic analysis to language assets, cutting review time while improving translation quality.
