Applied AI Lead - Evaluation & Measurement

Job Summary

Workplace

• Hybrid position open to candidates residing in or near Beavercreek, Ohio; Frederick, Maryland; or Fort Walton Beach, Florida.

Primary Accountabilities

• Design pre-launch evaluations with baselines, acceptance criteria, error budgets, and human-review points.
• Gate launches against documented evaluation standards.
• Build measurement pipelines and records to track actual capability delivery.
• Connect measurements directly to systems that hold ground-truth data.
• Set trace and telemetry standards for diagnosable and auditable deployed systems.
• Deliver monthly and quarterly forecast-versus-actual results for Finance.
• Run intake records so every AI idea is captured through the established process.
• Embed acceptance standards and value measurement logic into the proprietary AI platform.

Required Qualifications

• Have 5+ years in data, ML, or test engineering, including 3+ years in evaluation or model validation and recent LLM evaluation work.
• Demonstrate evaluation rigor through examples of stopping a launch and clearing a questioned capability.
• Have built a measurement practice with baselines, pipelines, and leadership-used dashboards.
• Bring data and measurement instrumentation skills using pipelines, dashboards, records, and source-of-record data.
• Demonstrate professional integrity under pressure when evidence does not support a claim.
• Hold a bachelor's degree and 5 years of experience, or an equivalent combination of education and experience.
• Have proficient technical expertise and strong interpersonal, leadership, negotiation, communication, and writing skills.
• U.S. citizenship is required; an active DOD clearance or ability to obtain one after starting is required.

Preferred Qualifications

• Experience connecting metrics to operational systems of record.
• Statistical literacy for error budgets, confidence bounds, and defensible claims.
• Exposure to cost accounting, benefits realization, or audit.
• Defense cost-accounting context.
• Active or prior security clearance.

Benefits

• Benefits include medical, dental, vision, health savings account contributions, telemedicine, life and disability insurance, legal insurance, and a 401(k) plan.
• Flexible schedules include a 9/80 program, vacation, health or emergency leave, paid parental leave, and community service hours.