AI Service Experience Analyst

Position summary

Location

• The position is described as remote in the USA.

Experience design and AI oversight

• Own experience objectives, user journeys, conversational flows, agent instructions, escalation paths, and desired outcomes.
• Design, test, and optimize prompts, instructions, and interaction patterns for generative AI tools, agents, and knowledge assistants.
• Review agent performance and outputs, identify gaps and risks, validate behavior, and recommend corrective actions.
• Evaluate AI outputs for accuracy, relevance, clarity, consistency, usability, bias, risk, compliance, and alignment with business intent.

Knowledge, testing, and requirements

• Maintain and improve knowledge content, procedures, job aids, scripts, and source materials to support AI readiness and accurate retrieval.
• Define requirements, test scenarios, acceptance criteria, quality standards, and success measures for AI-enabled experiences.
• Test prompts, knowledge retrieval, agent responses, conversational experiences, and end-to-end service flows using feedback and performance data.
• Translate experience designs, prompt logic, knowledge needs, test findings, and performance gaps into engineering-ready requirements and recommendations.

Collaboration and performance

• Partner with technology, product, data, compliance, governance, operations, engineering, and vendors to support secure, scalable, governed solutions.
• Create reusable prompt frameworks, agent playbooks, knowledge standards, testing methods, documentation, and business guidance.
• Measure and communicate adoption, agent performance, efficiency, service and knowledge quality, and business value or ROI.
• Apply healthcare service journey principles to identify pain points and shape solutions for employees, members, providers, or stakeholders.
• Stay current with AI capabilities, evaluation methods, knowledge practices, healthcare payer trends, and industry practices.

Qualifications

• Experience designing or improving AI-enabled service experiences, conversational flows, user journeys, knowledge assistants, AI agents, or generative AI interactions.
• Strong understanding of prompt design, agent instructions, interaction patterns, knowledge retrieval, and AI response evaluation.
• Experience improving knowledge content, procedures, scripts, job aids, or source materials for accuracy, consistency, usability, and AI readiness.
• Ability to define requirements, acceptance criteria, quality standards, test scenarios, and success measures for digital or AI solutions.
• Analytical skills to assess AI outputs for accuracy, relevance, clarity, consistency, usability, bias, risk, and business alignment.
• Experience collaborating across technology, product, data, operations, compliance, governance, vendor, and engineering teams.
• Ability to translate business expectations, designs, knowledge needs, prompt logic, and test findings into clear requirements and recommendations.
• Strong documentation skills and comfort working in an evolving AI environment with a continuous-improvement and measurable-value focus.
• Healthcare, payer, member/provider service, or service journey design experience is a plus.

Compensation

• The listed salary range is USD 92,250–120,000 per year.