Senior Inference Engineer

About Aleroy

About the Role

What You Will Own

• Inference Serving: prefill/decode disaggregation, continuous batching, KV-cache management, cache-aware routing, speculative decoding.
• Serving Performance: improve throughput, latency, memory efficiency, and serving cost.
• GPU Performance Engineering: eliminate bottlenecks using CUDA, Triton, profiling, kernel and memory optimisation.
• Mixture-of-Experts Serving: expert parallelism, all-to-all communication, expert-load balancing, distributed execution.
• Quantization & Model Optimisation: implement and evaluate techniques while measuring accuracy.
• Benchmarking & Measurement: rigorous, reproducible benchmarking across performance, quality, and cost.
• Performance Instrumentation: build instrumentation to understand system behaviour and validate improvements.
• Distributed Serving: decisions around routing, scheduling, caching, and workload distribution.

About You

• Strong hands-on experience building, optimising, or operating LLM inference-serving systems in production.
• Deep understanding of inference stacks such as vLLM, SGLang, TensorRT-LLM, or similar.
• Strong understanding of paged attention, continuous batching, disaggregated serving, and KV-cache management.
• Strong GPU performance engineering experience using CUDA or Triton.
• Understanding of GPU performance characteristics and modern GPU precisions such as FP8 and FP4.
• Experience serving large Mixture-of-Experts models.
• Hands-on experience with model quantization and quality measurement.
• Strong benchmarking experience including load generation, latency percentiles, and reproducibility.
• Strong understanding of distributed systems, routing, and scheduling.
• Demonstrated technical leadership experience.
• English level: Advanced (C1).

Nice to Have

• Experience with speculative decoding under production load.
• Open-source contributions to inference or serving frameworks.
• Experience serving reasoning models, long-context models, or agentic multi-turn workloads.
• Experience with energy-aware serving.
• Experience with confidential computing or data-residency-aware serving.
• Production experience with model compression techniques.

What We Offer

• Real ownership of the backend that powers Aleroy inference platform.
• Influence over backend architecture and product direction.
• Hands-on work on hard problems in distributed systems and LLM serving.
• Small, senior team with fast execution and low bureaucracy.
• Competitive salary with performance-based incentives.
• 24 days annual leave plus public holidays.
• Health insurance.
• Modern office in Yas Creative Hub.