Director, Product Management AI Quality & Evaluation

Role overview

Platform and evaluation pathways

• Own the shared AI evaluation platform's roadmap, adoption strategy, integrations, and vendor commercial and roadmap relationship.
• Partner with engineering to operate tracing, offline and online evaluation harnesses, calibrated LLM-as-judge pipelines, golden datasets, and CI regression suites.
• Create and maintain documented pathways for instrumenting systems, building datasets, running evaluations, and integrating them into releases.

Quality standards and review operations

• Define the AI quality taxonomy, including failure definitions and classifications, and steward the organization's shared quality language.
• Own human-review operations for calibration sets, ground-truth datasets, and adjudication, including reviewer sourcing, training, consistency, cost, and scale.
• Set minimum pre-release requirements for evaluations, tracing, documented failure taxonomy, and measured baselines.

Adoption and team building

• Help product teams move from existing quality practices to shared tools and pathways, making the shared route easier to use.
• Build the team supporting the AI quality and evaluation charter.

About FactSet

Requirements

• Product leadership experience sufficient to establish and lead an enterprise AI quality and evaluation function.
• Ability to define and operate quality measures, evaluation pathways, and platform adoption across multiple product teams.
• Ability to lead human-review operations, including reviewer sourcing, training, consistency measurement, cost, and scale.
• Ability to establish minimum AI release-quality requirements, including running evaluations, tracing, failure taxonomy, and measured baselines.