Principal AI Ops Engineer

Role overview

Responsibilities

• Design, operate, and optimise GPU inference and model-serving infrastructure for production AI and LLM workloads.
• Optimise latency, throughput, batching, quantisation, autoscaling, and infrastructure cost.
• Build release pipelines for models, prompts, and agent configurations, including canaries, regression gates, and rollbacks.
• Establish observability across model, retrieval, and orchestration layers for tracing, latency, cost, and quality.
• Define SLOs for availability, latency, and AI quality; maintain alerting and incident response processes.
• Own GPU and AI infrastructure capacity planning and cost optimisation.
• Build secure, scalable Kubernetes environments and Infrastructure-as-Code patterns.
• Develop reusable deployment tooling and operating standards for engineering teams.
• Lead production incidents, load testing, and root-cause analysis.
• Provide technical leadership and establish standards for AI operations at scale.

Requirements

• Staff, Principal, or equivalent senior IC experience operating production ML or LLM systems at scale.
• Deep hands-on GPU inference and model-serving experience; examples include vLLM, TGI, TensorRT-LLM, or similar.
• Strong understanding of batching, quantisation, latency, throughput, autoscaling, and serving trade-offs.
• AI/LLM observability experience including tracing, quality monitoring, drift, and regression detection.
• Reliability engineering experience across SLOs, incident response, capacity planning, and post-mortems.
• Strong Python skills and experience building production automation and infrastructure tooling.
• Deep Kubernetes, Docker, and Infrastructure-as-Code experience across cloud environments; Terraform is ideal.
• Observability tools may include Langfuse, LangSmith, Arize Phoenix, Grafana, or Prometheus.
• Experience integrating AI evaluation and regression tests into CI/CD and production release processes.
• Cloud infrastructure experience; Azure is ideal. Production security, data residency, or compliance experience is also sought.
• GPU/AI cost optimisation and FinOps experience is advantageous.
• Hands-on approach with ability to set standards while remaining close to production engineering.

Workplace

• Abu Dhabi, United Arab Emirates; source metadata identifies the workplace type as onsite.