Research Engineer, AI Evaluations

What We Do

Key Responsibilities

• Design benchmarks, rubrics and metrics, and choose between human, automated and hybrid evaluation based on reliability, cost and validity trade-offs.
• Deploy and serve models through APIs and self-hosted GPU inference, with versioned configs, logging and controls for reproducible results at scale.
• Analyse evaluation data with sound statistical methods (agreement, uncertainty, significance, disaggregation), and validate automated judges against human ground truth.
• Build and maintain modular evaluation pipelines and leaderboard infrastructure, covering sampling, inference, task packaging, scoring and reporting, with tested, documented code.
• Prototype new evaluation methods, and turn results into publications, technical reports and public releases.

Who You Are

• 4+ years in ML engineering or applied ML research, shipping production-quality systems.
• Strong software engineering, including clean, tested Python and end-to-end data and ML pipelines.
• Deep working knowledge of modern AI architectures and inference, including quality, latency and cost trade-offs.
• Rigorous grounding in probability and statistics on noisy real-world data.
• Ability to independently turn ambiguous questions into rigorous, executable technical work.
• Master's degree in Computer Science, Machine Learning, NLP or a related field preferred; equivalent industry or research experience will be considered.
• Strong differentiators
• Experience building benchmarks or evaluation systems.
• Experience with human judgment data.
• Experience with speech or multilingual NLP, especially Indian languages.
• Publications or open-source work in AI or related domains.
• Mindset
• Sceptical of numbers until you know how they were produced.
• Thrives in interdisciplinary teams spanning engineering, research, social science, operations and domain experts.
• Comfortable with ambiguity, messy data and evolving requirements, and maintains rigour under tight timelines.
• Cares about building AI that works for the people it is meant to serve.

What We Offer

• Flexible leave and vacation policy
• Hybrid work options
• Insurance as per industry standards
• Access to leaders in technology and social impact sectors
• A chance to contribute meaningfully to AI, digital equity, and rural development
• Karya celebrates diversity and is an equal opportunity employer. All applicants will be considered without regard to race, religion, gender identity, sexual orientation, disability, or any other protected status.

Preferred Qualifications

• Master's degree in Computer Science, Machine Learning, NLP or a related field preferred; equivalent industry or research experience will be considered.