Staff Software Engineer, AI/ML GenAI, Search Intelligence

Responsibilities

• Define the long-term technical roadmap and architectural standards for automated evaluation infrastructure.
• Establish company-wide metric definitions and taxonomies that faithfully capture user behavior and the multidimensional quality of GenAI experiences.
• Lead the design and implementation of highly scalable, production-grade autorater systems.
• Drive advanced statistical methodologies to validate autorater accuracy against production user signals and labeled ground truth.
• Partner with cross-functional teams to integrate statistically significant evaluations into the systems, while mentoring engineers in measurement best practices.
• - Define the long-term technical roadmap and architectural standards for automated evaluation infrastructure. - Establish company-wide metric definitions and taxonomies that faithfully capture user behavior and the multidimensional quality of GenAI experiences. - Lead the design and implementation of highly scalable, production-grade autorater systems. - Drive advanced statistical methodologies to validate autorater accuracy against production user signals and labeled ground truth. - Partner with cross-functional teams to integrate statistically significant evaluations into the systems, while mentoring engineers in measurement best practices.

Minimum qualifications:

Preferred qualifications:

Qualifications

• Minimum qualifications: - Bachelor's degree or equivalent practical experience. - 8 years of experience in software development. - 5 years of experience testing, and launching software products, and 3 years of experience with software design and architecture. - 5 years of experience leading ML design and optimizing ML infrastructure (e.g., model deployment, model evaluation, data processing, debugging, fine tuning). - 2 years of experience with state of the art GenAI techniques (e.g., LLMs, Multi-Modal, Large Vision Models) or with GenAI-related concepts (language modeling, computer vision). Preferred qualifications: - Master’s degree or PhD in Computer Science, Applied Mathematics, Statistics, Data Science, Machine Learning, or a quantitative field. - 8 years of experience architecting and operating large-scale distributed systems or ML infrastructure in Python, C++, or Java. - Experience with statistical modeling, hypothesis testing, sampling theory, reliability metrics, and data science tools (e.g., Pandas, PyTorch). - Experience designing measurements that capture GenAI user behavior, conversational experiences, and multidimensional product quality. Experience leading major engineering initiatives to production deployment and driving technical alignment across teams.