{bc}
Apply
linkedin

QA AI Automation Engineer

Discovered MENA
Abu Dhabi, UAE
Full-time
Mid-Senior
Onsite
AED 28,000–32,000
Discovered 5 days ago
PythonpytestAPI integrationsAI and LLM output evaluationRAGAS, DeepEval, LangSmith, or custom scoringTest data preparation and golden datasets

Full Job Posting

Role overview

Hiring an experienced QA AI Automation Engineer to own testing strategy using real production data and synthetic test datasets.

The role continuously evaluates AI models and internal LLMs and works with AI, ML, and product teams in Abu Dhabi.

Compensation

  • Compensation is listed as AED 28,000–32,000 plus benefits per month.

Responsibilities

  • Build and maintain Python-based automation test suites and frameworks to test internal AI models using real-world and synthetic data.
  • Evaluate AI and LLM outputs for accuracy, consistency, edge cases, and failure modes.
  • Build and manage test datasets from production data and generated synthetic scenarios.
  • Write regression tests, golden-dataset evaluations, and performance benchmarks for latency, cost, and throughput.
  • Integrate tests into CI/CD pipelines with automated quality gates that stop deployments when thresholds drop.
  • Track model performance using dashboards and alerting systems to identify regressions.
  • Collaborate with AI, ML, and product teams to define use-case quality criteria and encode them in automated checks.

Requirements

  • Bachelor's or master's degree.
  • Strong Python skills, including scripting, testing frameworks such as pytest, and API integrations.
  • Experience testing AI and LLM outputs or validating ML systems using RAGAS, DeepEval, LangSmith, or custom scoring.
  • Experience cleaning real data, generating test data, and constructing golden datasets.
  • Hands-on CI/CD experience, including GitLab CI or GitHub Actions.
  • Understanding of SQL, NoSQL databases, data analysis, and vector search or embeddings.
  • Familiarity with FastAPI, Docker, Linux, Model Context Protocol, and GPU or CPU optimizations.
  • Understanding of model fine-tuning concepts, prompt engineering, and observability tools.
  • Strong attention to detail, conceptual thinking, initiative, and a results-oriented approach.

Preferred qualification

  • ISTQB CT AI certification is a plus.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Discovered MENA

SAP Project Coordinator

, UAE

Discovered TodayContract

Relationship Manager

, UAE

Discovered TodayFull-time

SAP Consultant Finance

, UAE

Discovered 3 days agoContract

Localization Engineer

, KSA

Discovered 3 days agoFull-time

Senior Agentic AI Engineer

, UAE

Discovered 4 days agoFull-time

SAP Consultant

, UAE

Discovered 5 days agoContract

Software Engineer (SDKs, Unity)

, UAE

Discovered 5 days agoFull-time

Senior IT Infrastructure Engineer

, KSA

Discovered 5 days agoFull-time