Lead Inference Engineer

About Aleroy

About the Role

What You Will Own

• Own and improve the inference serving stack, including prefill/decode disaggregation, continuous batching, KV-cache management, cache-aware routing, and speculative decoding.
• Improve throughput, latency, memory efficiency, and serving cost while maintaining model-quality standards.
• Identify and eliminate GPU performance bottlenecks using CUDA, Triton, profiling, kernel optimisation, and precision strategies.
• Drive efficient serving of large MoE models, including expert parallelism, all-to-all communication, and expert-load balancing.
• Implement and evaluate quantization and other model optimisation techniques while measuring accuracy against full-precision references.
• Own rigorous, reproducible benchmarking across serving performance, quality, and cost.
• Build and improve instrumentation to understand system behaviour, identify bottlenecks, and validate improvements.
• Make technical decisions around routing, scheduling, caching, and workload distribution across diverse hardware.
• Define the AI serving and measurement roadmap and prioritise performance problems and infrastructure improvements.
• Establish technical standards around inference performance, benchmarking, reproducibility, testing, and production readiness.
• Guide and mentor engineers across serving, metrics, and testing while remaining deeply involved in implementation.

About You

• Strong hands-on experience building, optimising, or operating LLM inference-serving systems in production.
• Deep understanding of inference stacks such as vLLM, SGLang, TensorRT-LLM, or similar technologies.
• Strong understanding of paged attention, continuous batching, disaggregated serving, KV-cache management, and modern inference architectures.
• Strong GPU performance engineering experience using CUDA or Triton.
• Strong understanding of GPU performance characteristics including memory bandwidth, compute utilisation, and precisions such as FP8 and FP4.
• Experience serving large Mixture-of-Experts models, including expert parallelism, all-to-all communication, and expert-load balancing.
• Hands-on experience with model quantization and measuring quality against full-precision references.
• Strong benchmarking experience including load generation, latency percentiles, throughput-at-SLO, and reproducibility.
• Strong understanding of distributed systems, particularly routing and scheduling workloads across varied hardware.
• Demonstrated technical leadership experience leading an inference, serving, ML infrastructure, or similarly complex engineering initiative.
• English level: Advanced (C1).

Preferred Qualifications

• Experience with speculative decoding and understanding its behaviour under production load is a strong plus.
• Open-source contributions to inference or serving frameworks are a strong plus.
• Experience serving reasoning models, long-context models, or agentic multi-turn workloads is a plus.
• Experience with energy-aware serving, including throughput-per-watt or power-constrained environments, is a plus.
• Experience with confidential computing or data-residency-aware serving is a plus.
• Production experience with model compression techniques such as structured sparsity, activation sparsity, or KV compression is a plus.

What We Offer

• Real ownership of the backend that powers the Aleroy inference platform.
• Influence over backend architecture and product direction.
• Hands-on work on hard problems in distributed systems, real-time streaming, and LLM serving.
• A small, senior team with fast execution and low bureaucracy.
• Competitive salary with performance-based incentives.
• 24 days annual leave plus public holidays.
• Health insurance.
• Modern office in Yas Creative Hub.