Quality Assurance Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking a Senior QA Automation Engineer to lead the validation and verification strategies for our clients AI transformation.
Key Skills for This Role
Full Job Posting
Overview
We are seeking a
Senior QA Automation Engineer
to lead the validation and verification strategies for our clients AI transformation.
In this role, you will define "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards required for the defense and enterprise sectors.
You will act as the bridge between Agile development and formal Systems Engineering.
Your mandate is to build automated testing frameworks that validate AI behaviors against "Ground Truth" datasets and ensure our AI agents pass rigorous
Test Readiness Reviews (TRR)
and
1. AI & LLM Validation
- **Non-Deterministic Testing**
- : Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against "Gold Standard" datasets
- **RAG Evaluation**
- : Implement automated metrics (e.g.,
RAGA
- S, faithfulness, answer relevance) to verify that Retrieval-Augmented Generation pipelines accurately cite technical and regulatory documentation
- **Prompt Regression**
- : Design regression suites to monitor "prompt drift," ensuring model updates do not degrade the quality of AI-generated engineering documents
2. Integration & System Verification
- **Enterprise Integration**
- : Build robust tests to validate data consistency between AI agents and critical systems (e.g.,
Sap S/4han
- A, Ariba), ensuring the integrity of Bill of Materials (BOM) and financial data
- **Performance Benchmarking**
- : Design tests to validate latency and throughput for forecasting models and risk-scoring engines using tools like
6
- **API & Security Validation**
- : Automate testing of secure API gateways, verifying
Role-Based Access Control (RBAC
) and PII redaction logic before data reaches AI models
.3. Governance & Traceability
- **V-Model Alignment**
- : Map automated test cases to "System Requirements" to create digital evidence for formal
Verification and Validation (V&V
- ) reports
- **Stage Gate Compliance**
- : Prepare "Test Readiness" packages for formal reviews, providing quantitative evidence that systems are stable enough to move from MVP to Production
- **Defect Lifecycle Management**
- : Manage the feedback loop between Requirements Quality Assistants and development teams, tracing AI logic defects back to specific model versions
What You’ll BringTechnical Requirements
- **Core Automation**
- : Expert proficiency in
Python (Pytest
) and standard libraries
(Selenium/Playwrigh
- t, Requests)
- **AI Evaluation**
- : Hands-on experience with LLM evaluation frameworks (e.g.,
DeepEval, TruLen
- s) and "Ground Truth" dataset management
- **Performance Engineering**
- : Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.)
- **Data Validation**
- : Expertise in
SQ
L and data quality tools (e.g.,
Great Expectation
- s) for Data Lakehouses and Vector Databases
- **CI/CD & DevOps**
- : Strong experience integrating quality gates into
GitLab CI/C
- D pipelines
- **Engineering Practices**
- : Deep understanding of modern QE practices, including
Shift Lef
t, Test Pyramid, and Mono-repo architectures
Professional Qualification
- **Experience**
- : 5+ years in QA Automation, with 2+ years focused on complex data-driven applications, ML models, or AI agents
- **Domain Expertise**
- : Background in
Defense, Aerospac
e, or highly regulated industries is a strong plus.
Familiarity with
IV&
- V processes is highly desirable
- **Analytical Mindset**
- : Ability to define pass/fail criteria for probabilistic systems and communicate "Confidence Levels" to engineering leadership
- This is a Fixed Term Contract role.
- .
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Arcus Search
Software Engineer - AEC
Abu Dhabi, UAE
We are looking for an AEC Software Engineer for a leading company in Abu Dhabi. This role would be a Fixed Term Contract. Responsibilities Develop software for Building Information Modeling (BIM) Create tools for archi
Product Manager
Abu Dhabi, UAE
The Role: You own one of the product domains, you define the vision of the product. You write the PRDs, define acceptance criteria, and work daily and lead engineering to ship features that platform users adopt. What You
Senior Solutions Architect
Abu Dhabi Emirate, UAE
The Role: We are looking for a talented and forward-thinking Solution Architect to lead the design and evolution of the architecture underpinning our SaaS and bespoke product portfolio. In this role, you will ensure that
Agentic AI Technical Lead
Dubai, UAE
Role: Agentic AI Technical Lead Location: Dubai Salary: 60,000 - 70,000 AED Benefits: Annual Bonus, Insurance, Tickets The role: You will help scale an AI-native engineering organisation building agentic platforms, leadi
Data Product Manager
Abu Dhabi Emirate, UAE
Arcus Search have partnered with a leading Financial Services organisation in the UAE, who are looking for a Data Product Manager to join and play a central role in shaping, evolving, and executing the strategic directio