AI Benchmarking Lead, Performance Benchmarking Evaluation

Role and product

Benchmarking and audit quality

• Benchmark Seller Assistant AI models for relevance, correctness, and completeness.
• Evaluate and quality-check core-team audits, measure auditor accuracy, and improve evaluation reliability and consistency.
• Calibrate quality standards, provide actionable feedback, and drive continuous improvement in audit processes and methodologies.
• Identify rubric gaps and ambiguities; validate and categorize model failures to surface product issues.

Annotation workflows and SOPs

• Serve as point of contact for annotation tasks across ML data process areas and ensure quality execution and delivery.
• Understand dependencies across ML data workflows and explain customer impact.
• Modify annotation methods and update SOPs; document changes, secure approval, share knowledge, and audit adoption and execution.
• Test new SOPs and tools, provide feedback, and recommend improvements to support onboarding.

Metrics and process improvement

• Structure data collection, analyze results, and provide input for SOP changes.
• Collate, track, and report functional metrics with stakeholders such as program managers and applied scientists.
• Identify process and tooling issues and recommend improvements to productivity and quality metrics.

Basic qualifications

• Bachelor’s degree or equivalent in a related field.
• Experience in natural language data labeling, data annotation, linguistic annotation, or other forms of data markup.
• Proficiency in MS Excel, basic understanding of SQL and Python, and experience with Microsoft Office products and applications.
• Strong verbal and written communication skills in English.
• Knowledge of SOA and processes that deal with sellers.

Preferred qualifications

• 1 to 3 years of equivalent experience; annotation-related tasks across ML data process areas are preferred.
• Strong process documentation and analysis knowledge; stronger SQL querying and Python programming for data analysis are preferred.
• Strong analytical and problem-solving skills and ability to work independently and as part of a team are preferred.