AI Test Analyst
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking experienced AI Test Analysts to design, execute, and evaluate test scenarios for AI-powered applications, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, AI Agents, conversational AI, and classical Machine Learning solutions.
Key Skills for This Role
Full Job Posting
Overview
We are seeking experienced
Ai Test Analysts
to design, execute, and evaluate test scenarios for AI-powered applications, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, AI Agents, conversational AI, and classical Machine Learning solutions.
Key Responsibility
- Design and execute AI-specific test cases across multiple AI solution types, including RAG, AI Agents, conversational AI, Q&A systems, and ML applications.
- Develop and maintain golden datasets, evaluation datasets, and adversarial test cases.
- Perform testing across key AI quality dimensions:
- Accuracy
- Grounding/Faithfulness
- Robustness
- Fairness/Bias
• Refusal & Safety Behaviour
Evaluate AI model responses using human and LLM-as-a-Judge methodologies.
Analyse test results and recommend improvements.
Contribute to AI quality gates, acceptance criteria, and traceability matrices.
Work closely with automation, governance, and development teams to improve AI quality.
Qualifications
- Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
- 7–9 years of experience in Software QA or Testing.
- Hands-on experience evaluating AI/LLM-based applications.
- Experience testing RAG, conversational AI, AI assistants, or agentic systems.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Birbal AI
Senior Microsoft Dynamics 365 CRM Developer(TAMM)
Dubai, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
Customer Relationship Management Developer
Abu Dhabi, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
Senior CRM Developer
Dubai, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
CRM Developer
Dubai, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
CRM
Dubai, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
Senior CRM Developer
Sharjah, UAE
We are looking for an experienced Microsoft Dynamics 365 CRM Developer to design, develop, customize, and support Dynamics 365 CRM solutions. The ideal candidate will have strong technical expertise in Dynamics 365 custo
AI Governance & Risk Lead (Responsible AI)
Abu Dhabi, UAE
We are seeking an experienced AI Governance & Risk Lead to establish and oversee AI governance, Responsible AI practices, and risk-based assurance for enterprise AI solutions. The successful candidate will define AI gove
Senior Test Lead
Abu Dhabi, UAE
We are looking for an experienced Engagement / AI Test Lead to lead the establishment and delivery of a Managed AI QA Testing Service for a major government engagement in Abu Dhabi. This role combines delivery leadership
Senior Microsoft Dynamics 365 CRM Developer(TAMM)
Dubai, UAE
Customer Relationship Management Developer
Abu Dhabi, UAE
Senior CRM Developer
Dubai, UAE
CRM Developer
Dubai, UAE
CRM
Dubai, UAE
Senior CRM Developer
Sharjah, UAE
AI Governance & Risk Lead (Responsible AI)
Abu Dhabi, UAE
Senior Test Lead
Abu Dhabi, UAE