Quality Evaluator – Fluent English (Remote, Estonia)

About the role

Anyone AI creates high-quality training and evaluation data for frontier AI models. It is looking for English-fluent experts with strong writing skills, critical thinking and extensive hands-on experience using large language models (LLMs). You will interact directly with AI models and compare their responses side by side to judge which best meets the user's needs; your evaluations and written explanations help improve the accuracy, helpfulness and reliability of AI systems. The work is remote, as an independent contractor on a part-time basis, for up to 3 months, at $7-$12 per hour.

Key responsibilities

• Interact with different AI models and tools across a wide range of prompts and topics.
• Compare two or more AI-generated responses side by side.
• Assess accuracy, helpfulness, instruction-following, clarity and overall response quality.
• Follow detailed rating guidelines and multi-step evaluation workflows.
• Write clear, well-reasoned explanations supporting each evaluation decision.
• Apply evaluation criteria consistently across tasks.

Required profile

• Fluency in English, with strong reading comprehension and writing skills.
• Extensive hands-on experience using LLMs and critically assessing their outputs.
• Strong analytical judgment and attention to detail, able to follow complex instructions carefully.
• Ability to explain evaluation decisions clearly and convincingly.
• Nice to have: previous experience in AI response evaluation, data annotation or quality assurance, and experience in editing, research, fact-checking or analytical writing.