Senior AI QA Engineer

About the role

What you'll do

• Test agentic systems end to end, including tool use, trajectories, grounding, citations, refusal calibration, guardrails, and adversarial testing.
• Evaluate non-deterministic AI for hallucination, consistency, and accuracy against gold datasets, including calibrated LLM-as-judge scoring.
• Maintain prompt and model regression suites to catch quality drift before release.
• Run continuous evaluation on production traffic with quality monitoring, alerts, and incident response.
• Perform integration and API testing, secure gateway validation for RBAC and PII, and performance and load testing.

What we're looking for

• At least 5 years in QA or quality engineering, including 2 years testing ML, LLM, or AI agent systems.
• Hands-on with LLM or agent evaluation frameworks such as DeepEval, TruLens, RAGAS, or custom Python evaluators, and LLM-as-judge.
• Experience with agent observability and tracing, such as LangSmith, Langfuse, or OpenTelemetry.
• Expert Python skills, including Pytest and custom harnesses, plus Selenium or Playwright and API testing.

Work setup

• Abu Dhabi-based hybrid role.
• Contract-based engagement; onboarding is ASAP.