gem
Staff AI Evaluation Lead
Fetch
Remote, USA
Full-time
Senior Level
Remote
USD 129000-152000 yearly
Discovered 1 months ago
Full Job Posting
- Own programs: Lead complex, high-impact automation and evaluation initiatives across workflows or teams
- Design scalable solutions: Architect end-to-end workflows integrating datasets, evaluations, automations, and HITL processes; build the harness and reusable components other analysts build against; build evaluation pipelines that run against production without hands-on operation
- Establish standards: Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations; verify that projects you don't run are meeting them; own the process by which those standards get made and used
- Own metric definitions: Create and maintain the metric definitions the org evaluates against, validate them against real production behavior, and revise them as models, tooling, and system architectures change
- Set the bar for production: Define the quality bar a system clears before it reaches production and where human review stays in the loop, and pull a system back when it stops meeting the bar
- Allocate evaluation depth by risk: Decide which systems get what depth of evaluation given finite capacity, name the risk accepted on the rest, and make that tradeoff visible to project stakeholders
- Keep the evaluation stack current: Define when a change to a model or platform requires re-baselining across the org and own the process for doing it; evaluate new models and AI capabilities as they ship and decide what the org adopts
- Improve systems at scale: Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations, not only within your own programs
- Drive cross-functional alignment: Influence priorities and partner with Engineering, Product, and AI teams to deliver solutions
- Measure and communicate impact: Define success metrics and communicate performance and recommendations to leadership
- Elevate the team: Raise the bar through your work and actively mentor others by sharing approaches, guiding problem-solving, and enabling the team to build stronger automation and evaluation capabilities
- Drive innovation: Stay current on emerging tools and approaches; pilot and translate them into actionable improvements for the team
- 8+ years of professional experience AI, machine learning, operational automations, or a related field.
- Proven ability to lead complex automation or evaluation initiatives across systems or teams
- Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products
- Fluency in SQL, JSON, APIs, and scripting with AI assistance, and ownership of the technical direction of the evaluation stack
- Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems you have built
- Experience with data pipelines, APIs, and production systems
- Experience in influencing cross-functional stakeholders and aligning priorities
- Demonstrated ability to define metrics and drive measurable business impact
- Experience mentoring or leading technical contributors
- Experience evaluating agentic systems in production at scale
- Experience setting technical standards adopted across an organization
- Equity: We offer full-time employees equity in Fetch, so that everyone can benefit from Fetch’s growth.
- 401k Match: Dollar-for-dollar match up to 4%.
- Benefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets.
- Continuing Education: Fetch provides ten thousand per year in education reimbursement.
- Employee Resource Groups: Take part in employee-led groups that are centered around fostering a diverse and inclusive workplace through events, dialogue and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership.
- Paid Time Off: On top of our flexible PTO, Fetch observes 9 paid holidays, as well as our year-end week-long break.
- Robust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule.
- Calvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more!
- Flexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from anywhere in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles)
About Fetch
Software & SaaS909 employeesFounded 2013
Fetch is a U.S. consumer rewards app and outcomes-based advertising platform serving shoppers and brands.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Fetch
Consumer Public Relations Senior Manager
, USA
Discovered 3 days agoFull-time
Senior ML Operations Engineer II - Ads Platform
, USA
Discovered 3 days agoFull-time
Staff Software Engineer, Audiences
, USA
Discovered 4 days agoFull-time
Senior Data Scientist II, Core & Agentic Automation
, CAN
Discovered 4 days agoFull-time
Staff Data Scientist, Rewarded Apps
, CAN
Discovered 5 days agoFull-time
Ad Operations Coordinator
, USA
Discovered 5 days agoFull-time
Principal Software Engineer, Data Products
, USA
Discovered 5 days agoFull-time
Senior ML Operations Engineer II - Monetization
, USA
Discovered 5 days agoFull-time