AI Engineer
Role Overview
Raised a $15M Series A led by Standard Capital, with backing from Y Combinator, FCVC, Transpose Platform, and Dropbox Ventures.
Our customer base has grown 4× year to date.
A team of 17 and growing.
Based in person in San Francisco at 650 5th Street.
Full Job Posting
What you’ll do
Fine-tune models for long-running agents, including agents that investigate whether test failures come from infrastructure, recent code changes, or test authorship.
Build and curate training and evaluation datasets from production workflows, using failure patterns to identify opportunities for improvement.
Develop LLM evaluations and observability that connect offline results with production behavior and help the team verify improvements.
Tune prompts, context, and model behavior to improve task success, reliability, latency, token usage, and cost.
Design repeatable experiments and help publish benchmarks that demonstrate our core agents’ performance on testing and browser workflows.
Own technical projects from problem definition through delivery, working with the engineering team to make practical tradeoffs and ship improvements.
You’re perfect for this role if...
You have relevant software engineering experience, typically 3+ years, including hands-on contributions to AI-powered capabilities used in production.
You have hands-on experience with supervised fine-tuning, including preparing training data and evaluating model behavior.
You’ve worked with prompt and context tuning, evaluation datasets, and LLM observability to diagnose failures and measure improvements.
You have strong engineering fundamentals, code efficiently, and know what you’re great at and where you have room to grow.
You use AI agents to build software, breaking down work, tracking progress, checking results, and redirecting agents when they get stuck.
You’re comfortable with autonomy and ambiguity. You can turn an unclear problem into a plan, manage your own work, and follow through to delivery.
We’re interested in the depth of your contributions and what you’ve shipped. Your experience may come from owning an AI capability within a larger product.
Nice to haves
Experience with reinforcement learning fine-tuning, RL environments, or LLM-as-judge evaluators.
Experience with long-running agents, recovery from failed steps, or coordinating multiple agents.
Experience publishing benchmarks or running rigorous experiments on AI systems.
About us
Raised a $15M Series A led by Standard Capital, with backing from Y Combinator, FCVC, Transpose Platform, and Dropbox Ventures.
Our customer base has grown 4× year to date.
A team of 17 and growing.
Based in person in San Francisco at 650 5th Street.
Benefits
- Medical, dental, and vision insurance.
- 401(k).
- Unlimited PTO and sick days, plus company-observed US holidays.
- Daily lunch and a fully stocked kitchen.
- Commuter benefits.
- Team offsites and events.
Qualifications
- Experience with reinforcement learning fine-tuning, RL environments, or LLM-as-judge evaluators.
- Experience with long-running agents, recovery from failed steps, or coordinating multiple agents.
Responsibilities
- Fine-tune models for long-running agents, including agents that investigate whether test failures come from infrastructure, recent code changes, or test authorship.
- Build and curate training and evaluation datasets from production workflows, using failure patterns to identify opportunities for improvement.
- Develop LLM evaluations and observability that connect offline results with production behavior and help the team verify improvements.
- Tune prompts, context, and model behavior to improve task success, reliability, latency, token usage, and cost.
- Design repeatable experiments and help publish benchmarks that demonstrate our core agents’ performance on testing and browser workflows.
- Own technical projects from problem definition through delivery, working with the engineering team to make practical tradeoffs and ship improvements.
- You have relevant software engineering experience, typically 3+ years, including hands-on contributions to AI-powered capabilities used in production.
- You have hands-on experience with supervised fine-tuning, including preparing training data and evaluating model behavior.
- You’ve worked with prompt and context tuning, evaluation datasets, and LLM observability to diagnose failures and measure improvements.
- You have strong engineering fundamentals, code efficiently, and know what you’re great at and where you have room to grow.
- You use AI agents to build software, breaking down work, tracking progress, checking results, and redirecting agents when they get stuck.
- You’re comfortable with autonomy and ambiguity. You can turn an unclear problem into a plan, manage your own work, and follow through to delivery.
- We’re interested in the depth of your contributions and what you’ve shipped. Your experience may come from owning an AI capability within a larger product.
- Experience with reinforcement learning fine-tuning, RL environments, or LLM-as-judge evaluators.
- Experience with long-running agents, recovery from failed steps, or coordinating multiple agents.
- Experience publishing benchmarks or running rigorous experiments on AI systems.
About Momentic
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Momentic
Founding Sales Development Representative
San Francisco, USA
Solutions Engineer
San Francisco, USA
Founding Sales Development Representative
San Francisco, USA
Founding Engineer (Front End)
San Francisco, USA
Founding Engineer (AI)
San Francisco, USA
Head of Growth
San Francisco, USA