AI QA & Evaluation Engineer

Role purpose

Evaluation design and testing

• Define risk-based test strategies for generative AI, agents, predictive models, and automation integrations.
• Create versioned datasets with business experts, covering normal cases, ambiguity, rare exceptions, and sensitive brokerage scenarios.
• Measure grounded accuracy, retrieval relevance, citation correctness, structured output, task completion, and human escalation.
• Develop scoring rubrics and reference answers; calibrate automated judges against human assessments.
• Test prompt injection, data leakage, cross-client access, excessive permissions, and approval bypasses.
• Validate end-to-end connected-system outcomes, including duplicates, partial completion, timeouts, retries, and recovery.
• Test brokerage risks involving product terms, policies, trading advice, client messaging, and account information.
• Evaluate latency, concurrency, provider outages, degraded retrieval, and model changes; verify fallback and human handoff.
• Integrate regression evaluations into CI/CD and compare releases with approved baselines and documented thresholds.
• Reproduce quantitative results with the Quantitative Lead; check data separation, leakage controls, and segment stability.
• Maintain defect triage, release evidence, and exception records; challenge readiness and escalate unresolved failures.
• Use redacted production samples, incidents, and feedback to extend coverage while separating development and held-out evaluation data.

Essential qualifications

• At least 5 years in software QA, test automation, or quality engineering, including hands-on AI, ML, or data-driven system evaluation.
• Strong Python, API testing, SQL, automated test framework, and delivery-pipeline integration skills.
• Understand LLM variability, retrieval, agent tool execution, and limits of exact-match testing for generated outputs.
• Can design representative test data, evaluation rubrics, and meaningful thresholds with business and technical stakeholders.
• Working knowledge of statistical metrics, sampling, class imbalance, and training, validation, and test data separation.
• Experience testing access controls, sensitive-data handling, integration failures, and production-system recoverability.
• Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field, or equivalent relevant experience.

Preferred experience

• Financial services QA, especially brokerage platforms, payment processes, or regulated client communications.
• AI red teaming, human annotation, and multilingual evaluation experience.
• Experience assessing predictive models, drift monitoring, and model documentation.