Quality Assurance Engineer

Role Overview

Core Responsibilities

• Design, build, and operate test automation frameworks that span web, mobile, API, backend services, and data pipelines — owning the architecture of the testing surfaces you build.
• Design and operate evaluation infrastructure for AI-powered products: systematic testing of LLM outputs, regression detection on model behaviour, golden-set management, automated grading where appropriate, and the practices that catch quality regressions before they reach production.
• Partner with AI Product Engineers on evaluation design — connecting their model-level evaluation work to the systematic, organisation-wide testing infrastructure you own.
• Build automated data validation across training data, evaluation datasets, and production data flows: schema validation, distribution checks, drift detection, and the practices that keep data trustworthy across consumers.
• Design and run performance testing across AI models, APIs, and user-facing applications: load profiling, latency under realistic conditions, capacity testing, and the practices that surface scalability limits before users do.
• Design and generate synthetic test data, adversarial inputs, and edge-case fixtures that exercise systems beyond the happy path — including inputs designed to probe AI failure modes such as prompt injection, jailbreaks, and grounding failure.
• Own the integration of automated testing into CI/CD pipelines: test selection, parallelisation, flake management, and the discipline that keeps test suites trustworthy as they grow.
• Own quality observability: test result analytics, failure pattern analysis, regression tracking, and the dashboards that make system quality transparent across teams.
• Contribute to non-functional testing across security, accessibility, and reliability — partnering with the relevant specialists to ensure these are systematically tested rather than checked off.
• Lead incident triage and post-mortem analysis for quality escapes: identify what testing should have caught the issue, build the test that does, and ensure systemic improvement rather than a one-off patch.
• Contribute to quality engineering standards across the AI Factory: testing conventions, framework choices, evaluation practices, and the technical bar that applies to anything shipped to production.
• Mentor engineers across the organisation on testing strategy, automation design, and AI evaluation practice. Champion a culture where engineers own the quality of what they ship.

Basic Qualifications

• 7+ years of QA and test automation experience, with a track record of designing and owning test automation frameworks at scale — not just writing tests within frameworks built by others.
• Strong programming foundation in at least one of Python, JavaScript / TypeScript, or Java — at the depth required to design test frameworks and evaluation tooling, not just script test cases.
• Strong expertise in modern test automation tooling across the relevant surfaces: Playwright, Cypress, Selenium, or equivalent for web; Appium, Detox, or equivalent for mobile; with the judgment to choose the right tool for the problem.
• Strong expertise in API testing and contract testing: REST, schema validation, contract verification, and the practices that catch integration failures early.
• Experience designing test strategy at the architectural level: deciding what to test, at what layer, with what coverage — and defending those decisions with evidence rather than habit.
• Strong CI/CD integration experience with GitHub Actions, GitLab CI, Jenkins, or equivalent: test selection, parallelisation, environment management, and flake reduction practices.
• Performance testing experience with k6, JMeter, Locust, or equivalent at production scale, including realistic load modelling and result interpretation.
• Demonstrated ability to think systematically about quality — including for non-deterministic systems where traditional pass/fail assertions are insufficient.
• Strong written and verbal communication — you can author test strategy documents, drive quality decisions in design reviews, and explain quality trade-offs clearly to engineers, product managers, and non-technical stakeholders.
• Proven ability to operate autonomously, take ownership, and deliver high-quality work end-to-end.

Preferred Qualifications

• Experience designing evaluation frameworks for AI / LLM-powered products: output quality assessment, retrieval and grounding evaluation, regression detection on model behaviour, and the practices that close the loop between measurement and product improvement.
• Hands-on experience with LLM evaluation tooling: ragas, DeepEval, Promptfoo, Braintrust, LangSmith, or equivalent — and the judgment to use them appropriately rather than ceremonially.
• Experience with LLM-as-judge evaluation patterns, including their limitations, calibration issues, and the practices that make automated grading trustworthy.
• Experience testing RAG systems specifically: retrieval quality, grounding correctness, citation accuracy, and end-to-end pipeline behaviour.
• Experience testing agent-based AI systems: tool use validation, multi-step reasoning evaluation, and failure recovery testing.
• Experience with data pipeline testing using Great Expectations, Soda, or equivalent — and with ML pipeline testing in Airflow, Kubeflow, MLflow, or equivalent.
• Experience with security testing practices: OWASP coverage, fuzzing, and adversarial testing — including AI-specific concerns such as prompt injection and jailbreak resistance.
• Experience with accessibility testing automation: axe-core, Pa11y, or equivalent, and the practices required for WCAG conformance at scale.
• Experience with chaos engineering or resilience testing in production-like environments.
• Experience with Arabic-language testing or right-to-left interface validation — relevant for government services in the UAE.
• Experience in a regulated or government-adjacent environment: audit trails, compliance frameworks, and the engineering discipline required to operate quality systems for sensitive data.

Our Stack

• We use the tools best suited to each problem.
• Current defaults reflect what works well for our use cases — not a mandated standard.
• Candidates should be productive across most of these areas and willing to operate across them.
• **Languages:**

Python, TypeScript / JavaScript, Java

• **Web and mobile testing:**

Playwright, Cypress, Appium, Detox

• **API testing:**
• Postman, Newman, Pact or equivalent contract testing, schema validation tooling
• **AI evaluation:**
• Custom evaluation harnesses, LLM-as-judge patterns, golden sets, ragas / DeepEval / Promptfoo or equivalent
• **Data validation:**
• Great Expectations, Soda, or equivalent
• **Performance testing:**
• k6, JMeter, Locust
• **CI/CD:**

GitHub Actions, GitLab CI, Jenkins

• **Observability:**
• Test result analytics, regression dashboards, structured logging across test infrastructure
• **Infrastructure:**
• Docker, Kubernetes, cloud platforms (Azure)

Technical Depth Expectations

• Candidates will be expected to demonstrate genuine depth in at least three of the following areas.
• Conceptual familiarity is not sufficient.
• **Test automation architecture —**
• framework design across web, mobile, API, and backend; test selection and parallelisation; flake management; and the structural decisions that keep test infrastructure trustworthy as it grows.
• **AI and LLM evaluation —**
• evaluation harness design, golden set management, regression detection on non-deterministic outputs, LLM-as-judge patterns and their limitations, and the practices that make AI quality measurable in production.
• **Performance and load testing —**
• realistic load modelling, capacity testing, latency analysis, and the depth to surface scalability limits in AI services and user-facing applications.
• **Data validation and pipeline testing —**
• schema validation, distribution checks, drift detection, and the practices that keep data trustworthy across training, evaluation, and production.
• **Security and adversarial testing —**
• OWASP coverage, fuzzing, prompt injection and jailbreak testing, and the practices appropriate for high-value government applications.
• **CI/CD integration —**
• test selection, parallelisation, environment management, and the discipline that keeps automated test suites fast and reliable as they scale.
• **Quality observability —**
• test analytics, regression tracking, failure pattern analysis, and the practices that turn quality data into systemic improvement.
• **Accessibility and inclusive testing —**
• WCAG conformance, assistive technology validation, and the practices that make government services genuinely usable.