Brazilian Portuguese Expert - Remote

Location

Type

Why This Role Exists

What You’ll Do

• Evaluate LLM-generated responses on their ability to effectively answer user queries
• Conduct fact-checking using trusted public sources and external tools
• Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies
• Assess reasoning quality, clarity, tone, and completeness of responses
• Ensure model responses align with expected conversational behavior and system guidelines
• Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines

Who You Are

• You hold a Bachelor’s degree
• You are a native speaker or have ILR 5/primary fluency (C2 on the CEFR scale) in Brazilian Portuguese
• You have significant experience using large language models (LLMs) and understand how and why people use them
• You have excellent writing skills and can clearly articulate nuanced feedback
• You have strong attention to detail and consistently notice subtle issues others may overlook
• You are adaptable and comfortable moving across topics, domains, and customer requirements
• You have a background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering)
• You have excellent college-level mathematics skills

Nice-to-Have Specialties

• Prior experience with RLHF, model evaluation, or data annotation work
• Experience writing or editing high-quality written content
• Experience comparing multiple outputs and making fine-grained qualitative judgments
• Familiarity with evaluation rubrics, benchmarks, or quality scoring systems

What Success Looks Like

• You identify factual inaccuracies, reasoning errors, and communication gaps in model responses
• You produce clear, consistent, and reproducible evaluation artifacts
• Your feedback leads to measurable improvements in response quality and user experience