Automated AI Tester

Location

Industry

Employment Typ

About Kayana

Important Work Arrangement Note

Role Overview

Automated AI QA Tester

Key Responsibilities

• Build, maintain, and scale automated test frameworks for AI-powered ordering and payment systems.
• Design
• conversation-based test scenarios
• for AI ordering assistants (menu browsing, cart building, checkout, cancellations)
• Develop AI-driven and automated E2E test suites that simulate real-world customer journeys.
• Validate
• payment flows
• (UPI, cards, wallets, failures, refunds, edge cases)
• Create synthetic user simulations and automated load scenarios to stress-test AI systems at scale.
• Evaluate
• AI responses for accuracy, consistency, and task completion
• Perform regression testing for AI model updates, prompt changes, and product releases.
• Identify issues in
• AI reasoning, API integrations, and transaction flows
• Integrate automated tests into
• CI/CD pipelines
• Collaborate closely with engineering, product, and AI teams to improve product quality and reliability.
• Leverage AI-assisted testing tools and frameworks to increase test coverage, automate test generation, and improve testing efficiency.

Required Skills

• **4+ years of experience**
• in QA automation, software testing, or test engineering.
• Strong experience designing and implementing automated E2E testing frameworks.
• Demonstrated experience using
• AI tools and AI-assisted testing methodologies
• for test creation, execution, analysis, and validation.
• Strong understanding of AI/LLM testing concepts, including Prompt testing, Hallucination detection, Response evaluation, Groundedness validation, AI safety testing &Evaluation metrics
• Experience with automation tools such as: Playwright, Selenium, Cypress & Postman
• Strong programming skills in Python and/or JavaScript.
• Experience with API testing and service integration validation.
• Familiarity with payment systems, fintech workflows, or transactional platforms.
• Ability to design structured test cases for conversational AI systems, chatbots, and AI agents.
• Experience validating multi-step user journeys across web, mobile, and backend systems.
• Strong debugging, root-cause analysis, and problem-solving skills.
• Nice to Have :
• Experience testing LLM-powered applications, AI agents, or autonomous workflows.
• Experience with Retrieval-Augmented Generation (RAG) systems and vector databases.
• Familiarity with AI evaluation frameworks such as: OpenAI Evals, DeepEval, LangSmith, Ragas, Similar evaluation platforms
• Experience in food ordering, restaurant technology, POS, e-commerce, or fintech products.
• Experience with CI/CD platforms such as: GitHub Actions, Jenkins, GitLab CI/CD
• Experience with performance, load, and reliability testing of AI-powered applications.

Soft Skills

• Strong analytical and investigative mindset.
• Exceptional attention to detail and edge-case discovery.
• Ability to work cross-functionally with engineering, product, and AI teams.
• Excellent written and verbal communication skills.
• Comfortable working in fast-paced startup environments with rapidly evolving products and requirements.
• Proactive ownership mindset with a focus on quality, reliability, and customer experience.

Why Join Us?

• At Kayana, you’ll be part of a fast-growing global company where your contributions will make a direct impact.
• We offer:
• A clear path for career growth.
• The opportunity to set up and shape the compliance framework for a growing international business.
• A supportive and collaborative environment where your success is celebrated.
• Starting salary: 8K AED per month