Senior QA Engineer, AI Agents & Agentic Workflows

Responsibilities:

• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows, including agent routing, tool/function calls, and multi-turn conversations. • Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and completeness, using both human review and automated methods. • Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated regression suites that run as agents, prompts, models and tools change. • Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs, tool-call failures, context loss, looping or runaway behavior. • Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses. • Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and quantify them with metrics. • Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool and orchestration causes. • Re-test fixes, validate the end-user experience, and keep evals current as the product evolves. • Document test strategy, eval results and quality trends clearly for global stakeholders.
• Bachelor's degree in software engineering, computer science or a related field. • 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or agentic workflows. • Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and acceptance thresholds for AI responses. • Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets, scoring rubrics). • Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning, memory/context, RAG. • Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.
• Your health and well being are important to us at QAD. We provide programs that help you strike a healthy work-life balance.
• Opportunity to join a growing business, launching into its next phase of expansion and transformation.
• Collaborative culture of smart and hard-working people who support one another to get the job done.
• An atmosphere of growth and opportunity, where idea-sharing is always prioritized over level or hierarchy.
• Compensation packages based on experience and desired skill set
• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,
• including agent routing, tool/function calls, and multi-turn conversations.
• Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
• completeness, using both human review and automated methods.
• Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
• regression suites that run as agents, prompts, models and tools change.
• Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
• tool-call failures, context loss, looping or runaway behavior.
• Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
• Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
• quantify them with metrics.
• Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
• and orchestration causes.
• Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
• Document test strategy, eval results and quality trends clearly for global stakeholders.
• Bachelor's degree in software engineering, computer science or a related field.
• 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or
• agentic workflows.
• Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and
• acceptance thresholds for AI responses.
• Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets,
• scoring rubrics).
• Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning,
• memory/context, RAG.
• Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.

About QAD:

Qualifications

• Bachelor's degree in software engineering, computer science or a related field. • 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or agentic workflows. • Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and acceptance thresholds for AI responses. • Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets, scoring rubrics). • Understanding of agentic concepts: