Principal Software Engineer - AI Foundations

Role overview

Responsibilities

• Design and operate high-throughput, low-latency LLM serving across GPU and CPU fleets.
• Build a GenAI Gateway and inference API layer with access control, routing, quotas, policy hooks, and observability.
• Develop intelligent model routing across multiple model backends with configurable policies and guardrails.
• Optimize GPU serving, memory management, KV caching, model acceleration, quantization, and throughput.
• Design disaggregated serving and distributed inference architectures to improve utilization and tail latency.
• Create secure production code, reusable platform components, SDK integrations, and operational tooling.
• Establish SLOs and SLAs, capacity planning, incident response, benchmarking, and performance and cost observability.
• Architect governed agentic AI engineering workflows with validation, security, resiliency, and reuse guardrails.

Required qualifications

• Formal software engineering training or certification and 7+ years of applied experience with large-scale platforms and services.
• Expert proficiency in one or more programming languages, including Python, Java, Scala, or Go.
• Experience leading agentic AI-enabled development practices with human-in-the-loop validation and traceability.
• Understanding of responsible AI, security, resiliency, data sensitivity, and risk-based governance.
• Experience operating high-scale inference and distributed systems with multi-tenancy and tail-latency controls.
• Understanding of GPU-serving fundamentals and production efficiency optimization.
• Experience with model-serving stacks, rollout strategies, shadow traffic, and performance benchmarking.
• Cloud-native experience with containers, orchestration, infrastructure as code, observability, and SLOs.
• Strong communication skills and ability to influence senior technical leaders and executives.

Preferred qualifications

• Deep experience with LLM inference optimization, including speculative decoding, KV-cache management, paged attention, optimized sampling, or continuous batching.
• Experience with quantization and compression toolchains and production quality and performance trade-offs.
• Experience designing disaggregated serving architectures and operating distributed inference at scale.
• Experience building model routing and governance layers with fallbacks, circuit breakers, tenant controls, and cost-aware routing.
• Strong performance engineering, profiling, GPU profiling, and bottleneck analysis experience.
• Experience with enterprise-scale multi-tenant platforms, reusable frameworks, and developer self-service.
• Security-by-design experience for ML and LLM systems, including access control, data handling, and supply chain controls.

Compensation

• The listed base salary range is USD 204,250 to USD 285,000 yearly.
• Eligible roles may also receive discretionary incentive compensation and benefits based on eligibility.

About JPMorganChase