AI Quality and Evaluation Lead

Opportunity

Evaluation and release readiness

• Own risk tiers, metrics, thresholds, representative test suites, release evidence, monitoring expectations, and continuous improvement.
• Define use-case measures with product, underwriting, engineering, QA, risk, compliance, and security partners.
• Own methods for evaluation harnesses, golden datasets, adversarial and regression tests, human review, and automated scoring.
• Establish production quality gates, including UAT evidence, performance thresholds, guardrail validation, privacy and access checks, rollback criteria, kill-switch readiness, runbooks, and approvals.

Risk, monitoring, and quality operations

• Test for prompt injection, data leakage, unsupported claims, policy deviations, tool misuse, unsafe write-back, inappropriate automation, and model degradation.
• Monitor evaluation results, model and prompt versions, drift, latency, user feedback, exceptions, overrides, incidents, and platform costs.
• Create defect taxonomies and root-cause practices and work with product and engineering teams on durable fixes.
• Scale reusable scorecards, automated continuous evaluation, squad QA practices, analyst oversight, benchmarking, and audit-ready evidence.

Experience and qualifications

• Requires a relevant bachelor’s degree and 7+ years of progressive experience in a relevant technical, quality, data, AI governance, or risk discipline.
• Requires production AI or generative AI evaluation framework and release-criteria experience, including RAG, document intelligence, agents, observability, and human-in-the-loop controls.
• Requires Python, SQL, test automation, data analysis, statistical methods, evaluation tools, quality metrics, regression testing, drift monitoring, and risk-control knowledge.
• Requires ability to translate policy and risk appetite into measurable criteria and communicate trade-offs to technical and business stakeholders.
• Quality capability scaling, reusable standards, automated assets, monitoring, coaching, and leadership experience are required; model or credit risk exposure is an asset.
• Financial services, mortgage, lending, servicing, broker-channel, third-party partnership, or regulated technology experience is preferred.

Working environment

• Office environment with periods of high volume and tight timelines.
• Work includes prolonged sitting, repetitive keyboard and mouse use, screen viewing, and multitasking.

Team and relationships

Hours

• Hours of work are 8:30 a.m. to 5:00 p.m.

Benefits and development

• The posting cites competitive compensation, benefits, training, a modern office, teamwork, and social and community events.