Lead Engineer - HPC Operations

Overview

Responsibilities

• will extend to collaborating with multidisciplinary teams, leading complex projects, implementing cutting-edge technologies, and providing mentorship to operations engineers.
• Your key responsibilities
• Oversee the daily operational management of HPC infrastructure, including compute, storage, networking, and scheduler components (e.g., Slurm, Kubernetes, etc.).
• Drive efforts to optimize the efficiency and performance of HPC systems, ensuring maximum resource utilization and minimizing downtime.
• Serve as the primary technical escalation point for L2 support teams, ensuring rapid and effective resolution of incidents and service requests.
• Continuously monitor system health, performance, and resource utilization using advanced monitoring tools (e.g., Prometheus, Grafana, DCGM).
• Manage user environments for AI/ML workloads, including container orchestration (e.g., Docker, Kubernetes) and workflow tools (e.g., MLflow, Kubeflow).
• Define and enforce job scheduling policies, priorities, and partitions within Slurm and/or Kubernetes environments to ensure resource fairness, efficiency, and workload optimization.
• Lead root cause analysis (RCA) of operational issues, contributing to post-mortem documentation and driving continuous improvement initiatives.
• Provide mentorship and technical guidance to junior engineers, fostering skills development and knowledge sharing across teams. Participate in on-call rotation as necessary.
• Ensure adherence to security and operational policies, assisting in audits and maintaining documentation for change and incident management processes.
• What we’re looking for
• (a) Required skills / qualifications
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field.
• Minimum of 8 years of experience in HPC operations, systems engineering, or DevOps roles, with at least 2 years in a leadership or ownership capacity.
• Advanced expertise in configuring, optimizing, and maintaining complex HPC environments, including hardware, software, and storage systems.
• Hands-on experience managing Slurm clusters and/or Kubernetes-based environments for AI/ML workloads.
• In-depth knowledge of GPU resource management, workload schedulers, and performance tuning for AI/ML workloads.
• Proficiency with monitoring and observability frameworks such as Prometheus, Grafana, and DCGM.
• Strong scripting and automation skills, including Python, Bash, Ansible, and Terraform.
• Solid understanding of Linux (RHEL/CentOS/Ubuntu), networking technologies (RDMA, InfiniBand, RoCE), and storage solutions (NFS, Lustre, Ceph).