Sr. AI Engineer 4B

Role overview

Key responsibilities

• Build and maintain automated tests for functional, integration, and regression testing.
• Test REST APIs and AI/model endpoints, including request/response validation, contract testing, and negative testing.
• Evaluate LLM responses for expected outcomes and consistency, and flag hallucinations or incorrect outputs against defined datasets and criteria.
• Extend end-to-end automation across UI, API, and data layers.
• Integrate automated suites into CI/CD pipelines and test applications deployed on AWS or Azure.
• Track and triage defects with developers and DevOps; prepare test cases, data, reports, and coverage metrics.
• Support test strategy and planning for AI/GenAI features, and keep up with evolving AI/LLM testing practices.

Required qualifications

• At least 3 years of hands-on QA/test automation experience, especially in AI application development.
• Strong Java or Python programming skills for test script development.
• Hands-on experience with Selenium, Cypress, JMeter, or Playwright.
• REST API testing experience; Postman or RestAssured are examples of tools.
• Practical experience with functional, integration, and regression testing.
• Experience integrating automated tests into CI/CD pipelines, such as Jenkins, GitHub Actions, or Azure DevOps.
• Working knowledge of Git/GitHub; understanding of AWS and Azure cloud testing.
• Foundational understanding of AI concepts, LLMs, and AI-specific testing considerations.
• Familiarity with Jira or HP ALM for test/defect management.

Advantageous experience

• Exposure to Ragas, DeepEval, Promptfoo, LangSmith, or Langfuse for AI/LLM evaluation.
• Exposure to testing RAG pipelines, agentic workflows, or conversational AI/chatbot applications.
• Performance/load testing experience using JMeter; DevOps exposure including containerization is mentioned, but the source description ends mid-sentence.