Senior AI Engineer

What You'll Do:

Minimum Requirements:

Qualifications

• Experience with observability for AI systems: logging, tracing, dashboards, and quality monitoring

Responsibilities

• Own AI Workflows End to End
• Design agentic and LLM-based systems that combine extraction, retrieval, reasoning, and tool use to complete multi-step work, with clear human-in-the-loop points
• Write the design docs, make the build-vs-buy and model choices, and set the success metrics for each workflow you own
• Build Reliable AI in Production
• Build production-grade LLM pipelines: prompt and context design, structured outputs, validation, fallbacks, and confidence scoring
• Integrate AI services cleanly with Jeeves's backend: clear API contracts, retries, graceful degradation, and per-customer data isolation
• Manage cost and latency across models, choosing the right model for each step
• Measure, Monitor, and Improve
• Build evaluation sets and automated evals for every workflow you own, and use them to catch regressions when prompts, models, or data change
• Instrument AI components with logging, tracing, dashboards, and alerts so you know when quality drops before a customer does
• Keep an audit trail of AI decisions that meets the standards of a regulated financial product
• Raise the Bar for AI Engineering at Jeeves
• Set patterns, shared tooling, and practices that other engineers can build on
• Show the wider team how to use coding agents and AI tools well in day-to-day engineering
• Review peers' AI system designs and share what you learn openly
• 7+ years of professional software engineering experience, including at least 2 years building and operating LLM or AI-powered systems in production
• A track record of owning a large system or workflow end to end, from scoping and design through launch and iteration, with limited direction
• Hands-on experience shipping LLM-powered applications with APIs such as Anthropic, OpenAI, or similar, including structured outputs, error handling, and evaluation
• Experience designing agentic or multi-step AI workflows (tool use, orchestration, human review steps)
• Strong proficiency in Python, plus solid backend fundamentals: REST APIs, PostgreSQL or similar relational databases, async patterns, and a major cloud provider (AWS, GCP, or Azure)
• Experience with observability for AI systems: logging, tracing, dashboards, and quality monitoring
• Professional fluency in English, written and spoken