Senior Manager- AI Safety and Evaluation Engineering

Role overview

AI evaluation and guardrails

• Define evaluation rubrics and scoring for GenAI performance, reliability, and safety across conversational AI, RAG, agents, and AI-enabled data access.
• Design benign, adversarial, deterministic, probabilistic, and edge-case test cases; architect test-data generation and curation.
• Build automated evaluators, including LLM-as-a-judge, targeted SLMs, and deterministic evaluators.
• Define development-stage evaluations for live systems and manage execution of evaluations and guardrails for GenAI use cases.
• Partner with GenAI and MLOps platform product and engineering leaders to automate evaluation and guardrail capabilities.

Monitoring, fairness, and risk

• Implement explainability methods such as Shapley Values and Integrated Gradients, plus bias-detection metrics for transparency and compliance.
• Design telemetry pipelines capturing inputs, outputs, and embeddings while maintaining PII and data privacy standards.
• Establish metrics such as latency, drift, hallucination rates, and semantic similarity, with circuit breakers for breached thresholds.
• Monitor drift and disparate impact in production predictive models.
• Analyze guardrail violations and adversarial activity; incorporate human feedback into model refinement.
• Create actionable Model Health reports and root-cause analysis for Model Risk, Cyber, and Ops Risk functions.

Leadership and standards

• Train and advise AI project teams on evaluation best practices and safety protocols.
• Maintain a unified safety stack and methodology across the bank’s AI portfolio.
• Serve as a technical contact for second-line risk and compliance teams, including Model Risk Management, Cybersecurity, Data Management, and Operational Risk.

Candidate requirements

• At least 7 years of experience in ML Engineering, Data Science, or AI evaluation, testing, or safety; financial-sector experience is ideal.
• Bachelor’s degree in information technology, Computer Science, Engineering, or a related discipline.
• Expert Python proficiency and deep knowledge of statistical testing, XAI libraries, and LLM evaluation frameworks such as Ragas, Giskard, or Arize.
• Proven ability to design observability and monitoring systems at scale.
• Experience leading cross-functional initiatives or serving as a technical mentor or consultant.

Benefits and development

• Benefits include medical, life, and personal accident insurance, paid leave and leave airfare, preferential loan and finance rates, discounts, and education assistance for certain job levels.
• Flexible and remote working options are available subject to eligibility and job requirements.
• Learning options include instructor-led courses, e-learning, on-the-job training, and professional development programs.