Audio Evaluator - TSS English (New Zealand)
Overview
We are seeking English (New Zealand) speakers to evaluate the quality of text-to-speech (TTS) audio outputs for voice AI. Evaluators will assess audio samples across a range of quality dimensions using structured rubrics, and provide written feedback in English.
About the Role
Each task involves listening to AI-generated audio samples in your native language or other proficient language, as indicated by you, and assessing them using a provided rubric. You will complete your evaluations in a web-based interface. Tasks vary in format. Some examples of what you might be asked to do: Listen to a single audio sample and rate it across multiple quality dimensions on a scale (e.g., 1-5). Listen to a pair of audio samples and indicate which you prefer, along with how strong your preference is. Evaluate multiple audio samples within one task, answering a set of rubric questions for each. Write short English-language feedback explaining some of your ratings, describing what you heard, and noting specific issues. Rubrics cover a range of quality dimensions beyond overall preference, and depending on the task, may include areas such as pronunciation, pacing, emotion, speaker similarity, and more. Each task comes with specific instructions outlining what to listen for and how to apply the rubric. We provide detailed evaluation principles and expect evaluators to internalize and apply them reliably so that ratings are aligned across the team. This means carefully reading the guidelines for each task, asking clarifying questions when something is unclear, and applying the same standards throughout your work. We conduct periodic calibration check-ins, and you will be expected to grow more proficient with the guidelines as you gain experience. Commitment Up to 10 hours per week. Volume varies by language and project. Not every week will require the full 10 hours, and some weeks may have less work available. Typical turnaround for a study is 24 hours from assignment. Larger studies (100+ samples) will have extended deadlines. Some tasks are planned in advance; others are ad hoc with tighter turnaround. Contract length is approximately 1 month, with potential for extension based on project needs and performance.
Requirements
Reliable internet connection, computer with audio playback, and headphones. Ability to follow detailed guidelines precisely and maintain consistent quality across tasks. Must join the team's Slack workspace for ongoing communication and coordination. Engaged and communicative. We value evaluators who ask questions and actively participate. Preferred Prior experience with data annotation, transcription, linguistic evaluation, or audio-related work.
How to Apply
In your proposal, please include: Relevant Experience: Describe any prior work in data annotation, transcription, audio evaluation, linguistics, or related fields. Be specific about the type of work and your role or maybe you just think you will be great at it. Feel free to apply. Availability: Confirm you can commit up to 10 hours per week and are available to begin promptly.