LLM Ops Engineer

Overview

• ****Who We Are****
• Healthcare needs a better rhythm: one that keeps care continuous and deeply human.
• Heidi is building an AI Care Partner that works alongside clinicians to make that possible.
• We’re a team of doctors, engineers, designers, researchers, and creatives building tools that help clinicians stay focused on what matters most: their patients.
• In just 18 months, Heidi has given back more than 18 million hours to healthcare professionals — supporting 73 million patient visits in 116 countries.
• Today, more than two million patient visits each week are powered by Heidi worldwide.
• Backed by nearly $100 million in funding, we’re growing in the US, UK, Canada, and Europe, partnering with leading health systems including the NHS, Beth Israel Lahey Health, and Monash Health.
• ****What you’ll do****
• ***LLM platform on Kubernetes***
• Design, deploy and maintain AWS/EKS infrastructure running GPU-backed model workloads
• Manage GPU node pools, tune autoscaling for inference traffic patterns, and own the full model serving lifecycle from container builds to production rollouts
• Write and maintain infrastructure as code in Terraform
• ***Model evaluation and quality***
• Build tooling that measures whether models are performing — clinically accurate, latency-appropriate, cost-efficient
• Design offline evaluation harnesses, automated regression tests, and dashboards that surface regressions before they reach clinicians
• ***Cost and performance***
• Own GPU utilization, quantization, request batching, model routing, and spot/on-demand node strategy
• Work closely with the Models Team on fine-tuning workflows and model selection tradeoffs
• ***Cross-functional delivery***
• Collaborate with product engineers and clinicians on prompt engineering, context window management, and model data pipelines
• Close the feedback loop between infrastructure quality and real-world clinical performance
• ***Observability***
• Instrument token usage, latency P99s, GPU memory pressure, hallucination rates, and error classes
• Define alerting thresholds and build self-serve model health tooling so the team is not relying on Slack threads to know something is wrong
• ****What we’re looking for****
• Strong AWS and Kubernetes experience, with hands-on depth in EKS, GPU workload scheduling, and IAM patterns that do not cut corners on healthcare data requirements
• Practical LLMOps experience: model serving frameworks (vLLM, TGI or similar), prompt versioning, model registry management, A/B deployment, shadow traffic, rollback strategies
• Comfort with Python and enough ML context to hold a real conversation about fine-tuning, RLHF, RAG architectures, and evaluation methodology
• Infrastructure-as-code fluency in Terraform
• Experience building evaluation frameworks for generative models — not just accuracy metrics, but latency, cost, and output safety
• A bias toward observable, auditable systems — especially in a regulated industry context
• Strong engineering habits: small PRs, meaningful code review, test coverage, and a low tolerance for tech debt
• Willingness to get close to the clinical domain: you do not need to be a clinician, but you need to care about the consequences of model failures in healthcare
• ***Bonus***
• Experience working with HIPAA, the Australian Privacy Act, or similar healthcare data requirements
• Prior work on latency-sensitive inference pipelines for consumer or clinical products
• Familiarity with agent orchestration frameworks (LangChain, LangGraph, or similar)
• Experience with Karpenter or cluster autoscaler tuning for bursty GPU workloads
• ****The way we work****
• Build to Last
• We design for safety and reliability so clinicians, patients, and our teams can trust what we build every day.

• Own Your Practice

• Move Fast, Stay Steady

• Make Others Better

• Honest feedback, steady support, and shared growth keep our teams improving together.
• Why you will flourish with us
• Flexible hybrid working environment, with 3 days in the office.
• A generous personal development budget of $500 per annum
• Learn from some of the best engineers and creatives, joining a diverse team
• Become an owner, with shares (equity) in the company, if Heidi wins, we all win
• The rare chance to create a global impact as you immerse yourself in one of Australia’s leading healthtech startups
• If you have an impact quickly, the opportunity to fast track your startup career!
• Heidi is dedicated to creating an equitable, inclusive, and supportive work environment that brings people together from diverse backgrounds, experiences, and perspectives.
• Our strength is in our differences.
• We're proud to be an equal opportunity employer and welcome all applicants as we're committed to promoting a culture of opportunity for all.