Senior AI Quality Engineer

Position description

Responsibilities

• Design and maintain LLM and agent evaluation frameworks, regression suites, golden datasets, and scoring rubrics.
• Build harnesses that catch hallucinations, tool-calling failures, prompt regressions, and unsafe or off-policy behavior.
• Define measurable agent-quality criteria for task completion, factual grounding, latency, cost, and reasoning quality.
• Integrate evaluations into CI/CD gates for model, prompt, and agent changes.
• Partner on observability and tracing for agent runs.
• Contribute API and end-to-end automation for AI features within larger product flows.
• Shape shared AI-testing practices and coach QA engineers across scrum teams.

Required experience and qualifications

• At least six years in QA, SDET, or test automation, with production automation delivered.
• Hands-on experience testing LLM-based or agentic systems, including evaluations, LLM-as-judge, prompt regression testing, or agent trajectory analysis.
• Prior experience in a shift-left, embedded QA model.
• Experience with a modern automation stack such as Playwright or Cypress and a typed language; TypeScript is preferred and Python is acceptable.
• Deep experience with Vitest, Jest, or Pytest and building custom test harnesses.
• API-first testing experience with REST and Postman or equivalent; fluency in HTTP-level testing, recording proxies, and service-to-service traffic.
• Working knowledge of CI/CD and evaluation integration; GitHub Actions is a plus.
• Familiarity with cloud secret managers and careful handling of sensitive test data in restore-from-production environments.
• Ability to reason about probabilistic systems, variance, sample sizes, confidence, and noisy results.

Preferred qualifications

• Experience with Langfuse, Braintrust, LangSmith, Ragas, or DeepEval is preferred.
• Familiarity with RAG systems, vector stores, or tool-calling frameworks is preferred.
• AI test-data strategy, synthetic data generation, and Datadog or similar observability experience are preferred.

Technology stack

• The listed stack includes TypeScript, Playwright, Vitest, Postman, GitHub Actions, Jira/Xray, Datadog, AWS, and emerging AI evaluation tools.

Compensation

• Annual base salary is CAD 100,000–130,000, with an annual bonus opportunity of 10%.

Benefits

• The posting lists medical, dental, and vision coverage; flexible spending accounts; paid time off and holidays; a 401(k) with company match and immediate vesting; company-funded life insurance; employee assistance programs; and mental-health benefits.

About Novara