Application Engineer

Job Overview

Key Responsibilities

• Develop and execute test cases, scripts, and procedures for Conversational AI and Agentic AI applications.
• Collaborate with product managers, developers, data scientists, prompt engineers, and other stakeholders to ensure quality standards are met throughout the development lifecycle.
• Perform functional, regression, integration, performance, and end-to-end testing across conversational AI platforms and agent-based AI systems.
• Validate AI agent decision-making, task execution, multi-step workflows, reasoning paths, and interactions with external tools, APIs, and knowledge sources.
• Test LLM-powered applications for accuracy, relevance, consistency, hallucinations, guardrail compliance, and response quality.
• Evaluate agent orchestration capabilities, including multi-agent collaboration, task decomposition, context management, memory retention, and tool utilization.
• Identify, document, and track defects using JIRA and provide detailed reports on testing results.
• Analyze requirements and design comprehensive test strategies covering conversational flows, autonomous agent behaviors, integrations, and edge-case scenarios.
• Conduct prompt, response, and workflow validation to ensure optimal performance and user experience.
• Perform security, privacy, bias, and responsible AI testing to validate compliance with organizational and regulatory standards.
• Generate test reports, dashboards, and quality metrics for stakeholders, ensuring clear and actionable insights.
• Contribute to the continuous improvement of AI testing processes, frameworks, methodologies, and automation capabilities.

Required Qualifications

• 3-7 years of experience in software testing and quality assurance, with exposure to Conversational AI, Generative AI, Agentic AI, or similar AI technologies.
• Strong knowledge of QA methodologies, SDLC, STLC, defect management, and validation techniques.
• Experience testing conversational AI platforms, chatbots, voicebots, virtual assistants, or AI-driven applications.
• Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), prompt engineering concepts, and AI agent architectures.
• Experience with JIRA for defect tracking and project management.
• Strong proficiency in Excel for data analysis, reporting, and test metrics.
• Familiarity with API testing tools such as Postman, Swagger, or equivalent.
• Experience in functional, integration, performance, security, and user acceptance testing.
• Understanding of AI evaluation metrics, model behavior analysis, and conversational testing frameworks.
• Ability to work effectively in cross-functional teams and adapt to fast-paced environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Bachelor's/Master's in Engineering 2-5 years

Desired Qualifications

• Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, LangChain, or similar platforms.
• Experience with automated testing tools and scripting languages (Python, JavaScript, etc.).
• Exposure to NLP, NLU, LLM evaluation, prompt testing, and AI model validation.
• Knowledge of cloud AI platforms such as Google CCAI/Dialogflow, Azure AI, AWS Bedrock, Amazon Lex, or OpenAI-based solutions.
• Familiarity with version control systems (Git, GitHub, Azure DevOps).
• Experience developing AI test automation frameworks and evaluation pipelines.
• Understanding of Responsible AI principles, AI governance, model monitoring, and risk assessment.
• Personal Attributes
• Attention to detail and a commitment to delivering high-quality work.
• Strong communication and stakeholder management skills, both written and verbal.
• Self-motivated with a passion for AI innovation, quality engineering, and continuous improvement.
• Ability to manage multiple priorities in a dynamic, fast-paced environment.
• Curiosity to explore emerging AI technologies and evaluate complex AI agent behaviors.
• Strong critical thinking skills with the ability to identify risks, edge cases, and quality gaps in AI systems.
• Preferred Experience Areas
• Conversational AI Testing (Chatbots, Voicebots, Virtual Assistants)
• Generative AI & LLM Testing
• Agentic AI Testing
• RAG Validation
• Prompt Testing and Optimization
• AI Safety & Responsible AI Testing
• API and Integration Testing
• Test Automation
• Cloud-based AI Solutions