FLEX AI Software Engineer, Software Development Engineer in Test

Job Summary

Education and Experience Required

• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.
• 2+ years of experience in quality engineering, test automation, or SDET roles.
• Strong programming experience in Python, JavaScript, TypeScript, Java, or a comparable language, with hands-on experience in browser automation, API testing, automated test frameworks, performance testing, source control, code review, and CI/CD quality gates.
• Experience testing conversational AI, generative AI, retrieval-augmented generation, intelligent search, chatbot, or agent-based applications, including non-deterministic evaluation, cloud-native systems, containers, Kubernetes, observability, accessibility, and cross-functional delivery.

Preferred Qualifications

• Experience with commercial or open-source LLM evaluation platforms and with designing LLM-as-judge, deterministic, human-evaluation, prompt-testing, model-comparison, and retrieval-ranking workflows.
• Experience with responsible-AI validation, red teaming, adversarial datasets, contract testing, service virtualization, workflow orchestration, asynchronous systems, canary deployments, and production smoke testing.
• Experience with localization, accessibility, responsive and cross-browser testing, authentication and authorization testing, CORS, CSP, sensitive-data security testing, quality dashboards, and executive release-readiness reporting.

Core Work Activities

• Defines and owns the automated quality strategy for conversational AI applications and intelligent search experiences.
• Builds and maintains browser, API, integration, contract, smoke, regression, and end-to-end automated test suites.
• Develops automated evaluations for response correctness, relevance, grounding, hallucination risk, retrieval quality, ranking quality, tool selection, and tool-call accuracy.
• Validates multi-turn conversation consistency, safety guardrails, prompt-injection resistance, responsible-AI requirements, and sensitive-data handling.
• Converts business scenarios, customer journeys, and production defects into reusable regression datasets and automated tests.
• Develops performance, load, stress, spike, scalability, and reliability tests for AI applications and supporting services.
• Establishes measurable quality baselines for latency, throughput, error rates, concurrency, resource utilization, and release readiness.
• Integrates automated tests, AI evaluations, and quality thresholds into CI/CD pipelines and progressive-delivery workflows.
• Builds automated pre-deployment and post-deployment smoke tests that validate critical customer and system workflows.
• Manages test environments, synthetic data, fixtures, service mocks, test credentials, and repeatable test execution capabilities.
• Analyzes failures and production issues using logs, metrics, distributed traces, automation results, and AI evaluation data.
• Produces quality dashboards, defect insights, evaluation summaries, and release-readiness reports for engineering and business stakeholders.

Pay Range

• Pay Range: $41.97-$69.23 per hour

Schedule

• Schedule: Full Time
• This is a temporary position.
• The assignment duration is through September 11, 2027.

Workplace

• Located Remotely? Y
• Marriott Headquarters (HQ) supports a hybrid work environment that helps associates Be Connected. HQ based roles are generally hybrid and require regular in-office presence at the assigned location, including for candidates who live within commuting distance of HQ for a role listed as remote. Remote

Benefits

• All locations offer 401(k) plan, stock purchase plan, discounts at Marriott properties, commuter benefits, employee assistance plan, and childcare discounts.
• Full-time positions also offer coverage for medical, dental, vision, health care flexible spending account, dependent care flexible spending account, life insurance, disability insurance, accident insurance, adoption expense reimbursements, and paid parental leave.