Staff Software Development Engineer Test - Machine learning test
Role Overview
Tekion is seeking a Staff Software Development Engineer in Test for its AI Platform team.
The role builds shared evaluation infrastructure and frameworks for AI agents and ML models across Service, Sales, Finance and Insurance, and Analytics.
The role works with ML engineers, data scientists, the AI Platform team, and Product Management.
Key Skills for This Role
Full Job Posting
About Tekion
Tekion provides a cloud-native automotive platform spanning retail, enterprise, and partner operations.
The platform uses big data, machine learning, and AI to connect automotive manufacturers, retailers, partners, and consumers.
About the Role
Tekion is seeking a Staff Software Development Engineer in Test for its AI Platform team.
The role builds shared evaluation infrastructure and frameworks for AI agents and ML models across Service, Sales, Finance and Insurance, and Analytics.
The role works with ML engineers, data scientists, the AI Platform team, and Product Management.
Evaluation Platform
- Design, enhance, and own AI evaluation infrastructure as a shared capability across ML teams.
- Create, curate, and maintain evaluation datasets and golden or ground-truth sets across use cases and domains.
- Define metrics for accuracy, relevance, faithfulness, consistency, safety, and task success.
- Build automated scoring pipelines using LLM-as-a-judge, rubric-based, and reference-based methods.
Quality Monitoring and Delivery
- Validate user intents and measure response quality for AI-powered capabilities.
- Identify hallucinations, unsafe or biased outputs, and edge cases and design targeted evaluation suites.
- Build offline benchmarks and online production evaluation with monitoring, A/B testing, and drift detection.
- Establish CI/CD evaluation gates for model, prompt, and data changes.
- Develop dashboards and reporting for ML and product teams.
- Use AI and LLMs for automated judges, synthetic data generation, and evaluation tooling.
Quality Practices
- Champion evaluation and responsible AI quality practices across the organization.
Required Qualifications
- At least 8 years in SDET, quality engineering, ML engineering, or data science with hands-on evaluation or measurement systems experience.
- Strong Python programming skills for reusable evaluation pipelines and tooling.
- Deep understanding of ML and LLM evaluation, benchmark design, and non-deterministic system evaluation.
- Hands-on experience with LLM-as-a-judge, rubric-based scoring, or human-in-the-loop evaluation.
- Understanding of prompting, RAG, embeddings, tool use, hallucination, drift, prompt sensitivity, and bias.
- Experience designing and curating datasets, including labeling or annotation strategy and data quality.
- Strong statistical intuition and communication skills for interpreting results and supporting ML and product decisions.
Preferred Qualifications
- Experience with evaluation frameworks, online evaluation, guardrails, production monitoring, responsible AI, red-teaming, experiment tracking, or A/B testing.
- Prior experience establishing evaluation as a platform capability for multiple teams.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career