greenhouse
AI Quality & Test Engineer
Simple Technology Solutions
Washington, USA
Senior · 8+ years experience
Onsite
Discovered 1 weeks ago
awsbedrockjestjirapythontypescript
Key Skills for This Role
awsbedrockjestjirapythontypescript
Full Job Posting
Education and Experience:
- Required
- Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related technical field, or equivalent professional experience in addition to the minimum below
- Minimum of 8 years of software quality engineering or test automation experience, including at least 2 years testing applications built on large language models or other AI/ML components
- Demonstrated experience building automated end-to-end test suites integrated into CI/CD pipelines and defining quality gates for a delivery team
- Experience leading user-acceptance testing with business or mission users, ideally in a federal or regulated environment
- Strong test automation skills in Python or TypeScript using frameworks such as Playwright, pytest, or Jest, plus API testing tools
- Practical knowledge of LLM evaluation techniques, including golden datasets, rubric and model-graded evaluation, retrieval quality metrics, and prompt and model regression testing
- Working knowledge of Amazon Bedrock and AWS application services sufficient to test and troubleshoot deployed AI features
- Fluency with GitHub, Jira, and CI/CD pipelines, and with reading use cases and specifications to derive tests
- Familiarity with security testing basics and federal compliance expectations for test evidence
- Skeptical in the best sense; assumes nothing works until it is demonstrated, including AI output that reads well
- Collaborative and constructive, treating defects as information for the team rather than blame
- Self-directed and comfortable owning outcomes on a small, senior team where everyone contributes across the lifecycle
- Curious and current on AI tooling; treats AI assistants as part of the team and holds their output to the same standard as human work
- Clear, candid communicator with federal clients and teammates; writes well and documents as a habit
- Bias toward fast delivery and early user feedback over conservative gating; learns from failure quickly
- Required Certifications (to be earned within 12 months of hire)
- AWS Certified AI Practitioner or AWS Certified Developer – Associate
- Claude Certified Developer – Foundations (CCDV-F) or Claude Certified Associate – Foundations (CCAO-F)
- Preferred Certifications
- ISTQB Certified Tester Foundation Level (CTFL) and ISTQB Certified Tester AI Testing (CT-AI)
- AWS Certified Solutions Architect – Associate
- CompTIA Security+
- Certified ScrumMaster (CSM) or Professional Scrum Master (PSM I)
What We Offer:
- Competitive compensation package
- Comprehensive health, dental, and vision insurance plans
- Professional development and training
- Performance-based incentives
- 401(k) retirement savings plan
- Work from home stipend
- Unlimited PTO
- Paid holidays
- Flexible work schedule
- Professional development budget to be used for training and certifications
- STS is committed to equal employment opportunity and merit-based employment practices. STS provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination, harassment, and retaliation in all employment practices and decisions in accordance with applicable federal, state, and local laws. Employment decisions at STS are based on individual qualifications, performance, skills, and business needs, without regard to race, color, religion, sex, national origin, age, disability, protected veteran status, sexual orientation, gender identity, genetic information, marital status, or any other status protected by applicable law. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, compensation, training, transfer, discipline, termination, layoff, recall, and leaves of absence. --- Applicants may request removal from our applicant database, or specific information about how the data is used by contacting [email protected].
Preferred Qualifications
- ISTQB Certified Tester Foundation Level (CTFL) and ISTQB Certified Tester AI Testing (CT-AI)
- AWS Certified Solutions Architect – Associate
- CompTIA Security+
- Certified ScrumMaster (CSM) or Professional Scrum Master (PSM I)
About Simple Technology Solutions
IT Services & Consulting95 employeesFounded 2013
HUBZone U.S. government contractor delivering cloud, data, AI, and application-modernization services to federal law-enforcement agencies.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer