Senior QA Automation Engineer

Overview

Job Summary

Key Responsibilities

• Lead the validation and verification of AI systems by architecting automated testing frameworks to evaluate non-deterministic outputs, including hallucination detection, consistency checks, and factual accuracy against gold standard datasets for Large Language Models (LLMs) and predictive engines.
• Design and implement automated evaluation metrics such as RAGAS, faithfulness, and answer relevance to validate Retrieval-Augmented Generation (RAG) pipelines, ensuring accurate citation of internal technical documentation and regulatory texts.
• Develop prompt regression suites to monitor prompt drift and maintain the quality of AI-generated engineering documents, mitigating degradation risks from underlying model or system instruction changes.
• Construct robust integration tests to validate data consistency between AI agents and critical enterprise systems (e.g., SAP S/4HANA, Ariba), ensuring no corruption of Bill of Materials (BOM) or financial data in supply chain applications.
• Create performance benchmarking tests to validate latency and throughput of forecasting models and risk scoring engines, aligning with real-time requirements for supply chain dashboards.
• Automate API validation for secure gateways, verifying Role-Based Access Control (RBAC) and Personally Identifiable Information (PII) redaction logic before data ingestion into AI models.
• Ensure alignment of automated test cases with System Requirements and User Needs, generating digital evidence for Verification and Validation (V&V) reports as per Systems Engineering Handbook standards.
• Prepare Test Readiness packages for Stage Gate reviews, providing quantitative evidence to demonstrate system stability and readiness for progression from Minimum Viable Product (MVP) to Production deployment.
• Manage the defect lifecycle between Requirements Quality Assistants and development teams, tracing defects in AI logic back to specific model versions or datasets for resolution.
• Bridge Agile development and formal Systems Engineering processes, ensuring AI agents meet rigorous Test Readiness Reviews (TRR) and Functional Configuration Audits (FCA) for defense-sector deployment.

Qualifications And Experience

• 5+ years of experience in QA Automation, with at least 2 years focused on testing complex data-driven applications, machine learning models, or AI agents.
• Experience in Defense, Aerospace, or highly regulated industries is a strong plus. Understanding of IV&V (Integration, Verification, and Validation) processes is highly desirable.
• Expert proficiency in Python for building custom test harnesses (Pytest) and standard automation libraries (Selenium/Playwright for UI, Requests for API).
• Expert proficiency in crafting Performance Test Plans and Implementations (e.g., Locust, Jmeter, K6).
• Experience utilizing frameworks for evaluating Large Language Models (e.g., DeepEval, TruLens, or custom Python evaluators) and understanding of 'Ground Truth' dataset creation and management.
• Proficiency with SQL and data validation tools (e.g., Great Expectations) to verify data quality within Data Lakehouses and Vector Databases.
• Strong experience integrating automated tests into GitLab CI/CD pipelines, enforcing 'Quality Gates' that prevent non-compliant code or models from merging.
• Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion) and linking automated test results to specific requirement IDs.
• Strong hands-on experience managing Test Reports and Artifacts (e.g., TestRail, Allure).
• String knowledge of maintaining code-based frameworks (e.g., Git, GitLab).
• Strong hands-on knowledge of modern Quality Engineering best practices for fast-paced development environments (e.g., Shift Left Approaches, Test Pyramid, Mono-repo architecture for automation projects).
• Ability to define pass/fail criteria for probabilistic systems and communicate 'Confidence Levels' to engineering leadership.
• Proven ability to collaborate with Data Scientists to understand model limitations and with Systems Engineers to understand formal acceptance criteria.

Required Technical Skills

• Expert proficiency in Python for building custom test harnesses using Pytest and standard automation libraries (Selenium/Playwright for UI testing, Requests for API testing).
• Expert proficiency in crafting Performance Test Plans and implementations using tools such as Locust, JMeter, and K6.
• Experience utilizing frameworks for evaluating Large Language Models (LLMs), including DeepEval, TruLens, or custom Python evaluators, with a strong understanding of Ground Truth dataset creation and management.
• Proficiency with SQL and data validation tools (e.g., Great Expectations) to verify data quality within Data Lakehouses and Vector Databases.
• Strong experience integrating automated tests into GitLab CI/CD pipelines, enforcing Quality Gates to prevent non-compliant code or models from merging.
• Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion) and the ability to link automated test results to specific requirement IDs.
• Hands-on experience managing test reports and artifacts using tools such as TestRail and Allure.
• String knowledge of maintaining code-based frameworks using Git and GitLab for version control.
• Strong hands-on knowledge of modern Quality Engineering best practices for fast-paced development environments, including Shift Left Approaches, Test Pyramid principles, and Mono-repo architecture for automation projects.
• Architecting automated frameworks to evaluate non-deterministic AI outputs for hallucination, consistency, and factual accuracy against Gold Standard datasets.
• Implementing automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify Retrieval-Augmented Generation (RAG) pipelines for accurate citation of internal technical documentation and regulatory texts.
• Designing regression suites to monitor prompt drift and ensure changes to underlying models or system instructions do not degrade the quality of AI-generated engineering documents.
• Building robust integration tests to validate data consistency between AI agents and critical enterprise systems (e.g., SAP S/4HANA, Ariba), ensuring no corruption of Bill of Materials (BOM) or financial data.
• Designing performance tests to validate latency and throughput of forecasting models and risk scoring engines for real-time supply chain dashboard requirements.
• Automating the testing of secure API gateways to verify Role-Based Access Control (RBAC) and Personally Identifiable Information (PII) redaction logic before data reaches AI models.

Company And Project Focus

• Location: Abu Dhabi, United Arab Emirates

• Company: Sister Company of the Client
• Project Focus: Integration, Verification, and Validation (IV&V) for the AI platform and Intelligent Supply Chain systems

Why This Role Matters