Lead AI Engineer - (NPW)

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential. Our seasoned professionals deliver services based on Milestone’s best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Technical direction and ownership

Own the end-to-end technical approach for enabling Module 3 within APEX: the retrieval and grounding strategy, the prompt architecture, the evaluation framework, and the sequencing of implementation across sections.

Make and document consequential engineering decisions within the configuration and prompting envelope, and defend them to the Amgen platform team, the AI&D organisation and senior business stakeholders.

Set and hold the engineering standards for the pod — version control discipline, prompt and configuration change management, evaluation coverage, and documentation quality.

Assess technical feasibility during requirements definition and push back, with reasoning, on requirements that will not hold up under a language model — before they are committed to.

Estimate and sequence the work, and communicate clearly what can and cannot be achieved within the pilot window.

Act as the escalation point on technical risk: identify what could derail the pilot, quantify it, and raise it while it is still cheap to address.

Architecture — retrieval, grounding and evaluation

Design the retrieval and grounding approach: chunking and indexing strategy, embedding selection and retrieval tuning, reranking, source attribution, and the guardrails that prevent unsupported or overstated claims reaching a payer-facing document.

Design the prompt architecture at document level — how sections decompose, how context is assembled per section from a very large source corpus, and how coherence is maintained across a 150–200 page output.

Establish the evaluation framework: golden sets, rubric-based scoring, human review loops, and regression checks that run on every prompt and configuration change.

Instrument and evidence the improvement delivered, so the pilot outcome can be assessed against the POC baseline rather than asserted.

Hands-on implementation

Implement the enhancements already validated through proof-of-concept work, extending existing APEX functionality rather than rebuilding it.

Build the automation, data preparation and evaluation harnesses the pod depends on.

Configure AI-enabled application behaviour including human-in-the-loop checkpoints, and integrate enterprise GenAI capabilities available in the Amgen environment rather than introducing unsanctioned tooling.

Leadership and collaboration

Direct and mentor the Senior AI Engineer; review their work, pair on hard problems, and keep the two workstreams architecturally coherent.

Partner closely with the Business Systems Analyst so that requirements arrive implementable and technical constraints reach the business early.

Work directly with HEOR subject-matter experts in workshops and review sessions, explaining model behaviour, constraints and trade-offs in terms a non-technical audience can act on.

Contribute to weekly progress and status reporting, at a level of candour that surfaces problems early.

Own the handover: leave the configuration, prompt assets, evaluation framework and known limitations documented such that the Amgen platform team can maintain and extend the solution without the pod.

Required qualifications

Experience. 10+ years in software, data or AI engineering, including a minimum of 4 years hands-on delivery of production or pilot-grade generative AI solutions — sustained delivery, not experimentation.

Technical leadership. Proven experience as technical lead on a client-facing or cross-organisational engagement, owning the solution approach and answering for it. This is a requirement, not a preference.

Retrieval and grounding architecture. Deep RAG experience at document scale — chunking and indexing strategy, embedding selection, retrieval tuning, reranking, grounding, and citation or source-attribution mechanics. Able to reason about why a retrieval approach is failing, not only that it is.

Evaluation rigour. Demonstrated ability to design and operate an evaluation framework: golden sets, rubric-based scoring, LLM-as-judge with its limitations understood, human review loops, and regression discipline on prompt changes.

Prompt engineering depth. System and task prompt design, few-shot and template patterns, context window management, chaining and decomposition of long tasks, structured output enforcement — and the judgement to know when prompting is the wrong instrument.

Platform extension. Experience configuring and extending AI-enabled applications, working within an existing product’s extension model rather than building from scratch.

Enterprise tooling. Direct hands-on exposure to one or more of OpenAI / Azure OpenAI, GitHub Copilot, Databricks, or comparable enterprise AI development environments.

Failure-mode fluency. Deep practical familiarity with hallucination, non-determinism, prompt drift, context loss over long documents and silent quality regression — and the engineering practices that contain each.

Engineering craft. Strong Python or equivalent. Git discipline. Comfort in an agile, iterative cadence with evolving requirements.

Communication and independence. Strong written and verbal English, sufficient to present technical reasoning to senior stakeholders in the US and Europe and to disagree constructively. A track record of owning technical outcomes without close supervision.

Preferred qualifications

Experience with AI-assisted long-form document authoring, structured content authoring, or evidence synthesis at scale.

Prior delivery in life sciences, pharmaceutical or biotech — especially medical writing, regulatory authoring, HEOR or market access content.

Familiarity with agentic orchestration, knowledge graphs, or multi-step reasoning architectures.

Exposure to regulated or GxP-adjacent environments and the traceability and audit expectations that come with them.

Experience integrating GenAI output into downstream systems — SharePoint, document management, or workflow orchestration such as Smartsheet, Power Automate or UiPath.

Experience defining AI engineering standards or practices for a team, beyond a single project.

Model fine-tuning, distillation or evaluation-harness tooling experience.

At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for our employees but is also critical to our continued success. Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees. We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.