Senior Quality Engineer II (AI & Data)

Responsibilities

• Data Pipeline Validation: Test the integrity of data ingestion and transformation processes to ensure models are trained on high-quality datasets.
• API & Model Testing: Conduct deep API testing using tools like APIDog to validate model responses, handling complex edge cases such as zero-value parameters or unauthorized access.
• Generative Model Output Accuracy: Define and implement evaluation frameworks to measure the quality of generative AI outputs including relevance, factual accuracy, coherence, and hallucination rate using techniques such as LLM-as-a-judge, human-in-the-loop scoring, and automated benchmark suites.
• AI Agent Response Validation: Design test strategies to assess the correctness and reliability of agentic AI responses, covering tool-call accuracy, multi-step reasoning integrity, goal completion rate, and graceful failure handling for out-of-scope or adversarial inputs.
• Security Testing: Proactively identify vulnerabilities like request interception and price manipulation to safeguard the system including prompt injection and jailbreak attempts specific to generative AI surfaces.
• UI Automation: Build and maintain scalable automation suites using low-code tools like mabl, aiming for high Mabl Test Coverage %.
• AI Tooling Advocacy: Champion the use of AI assistants like Jira Rovo for requirement analysis and test case generation to increase team velocity.
• Root Cause Analysis (RCA): Lead the RCA process for production bugs and developer-rejected tickets to continuously improve the "Definition of Done".
• Senior Experience: 8+ years of relevant work experience
• Data Proficiency: Strong SQL skills and experience testing data-heavy applications or ML models.
• Automation Mastery: Proficiency in automation tools (e.g., Selenium, mabl, Playwright)
• Security Mindset: Understanding of OWASP principles and practical experience in security testing (request interception, label tracking).
• Domain Knowledge: Experience in e-commerce and critical customer-flow analysis is highly preferred.