Supercompute Operations Engineering

ABOUT THE ROLE:

RESPONSIBILITIES:

• Own monitoring architecture, alerting, signal quality, and automation of monitoring across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling).
• Own compute hardware change incorporation - physical, firmware, software.
• Be part of 24/7 iincident response, mitigation, and control authority - end to end - from customer impact back to the facility.
• Own day-to-day site sustainment operations, including resolving issues impacting rack-level compute and facility infrastructure.
• Disposition daily issues across both Site Operations (rack compute) and Facility Operations (power and cooling).
• Lead acceptance and deployment testing campaigns to bring new data halls fully online.
• Develop production automation for new datahall bring-up, automation behind alerting and automated incident response.
• Develop and maintain standard operating procedures and planning for facility and site technicians. Includes change / upgrade incorporation planning on both rack level and facility.
• Own and document the live, changing state of datahall operation as new halls come online.
• Enable Site Operations and Facility Operations technicians to do their day to day jobs more efficiently and safely to increase GPU uptime.
• Create predictive and responsive automation to alerts and incidents to enable your response to be faster and more impactful.
• Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing).
• Truly root cause systemic and design failure classes across hardware and integrated facility and operations systems: turn fleet data (MTBF/AFR, repeat-RMA trends) into proactive alerting, standard responses, vendor RCA inputs, and supplier corrective actions.

BASIC QUALIFICATIONS:

• Bachelor's degree in Mechanical, Electrical, or Computer Engineering or a related scientific discipline and 2+ years of experience in data center, critical facilities, power plant, or large-scale compute operations; or 6+ years of equivalent experience in those environments.

PREFERRED SKILLS AND EXPERIENCE:

• Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
• Experience in datacenter site reliability, systems engineering, plant operations, or large-scale production operations.
• Experience running acceptance test campaigns for integrated products.
• Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.
• Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
• Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
• Proven large-scale incident command experience and calm technical leadership on a bridge, whether from an SRE/NOC bridge or a plant control room.
• Experience writing and operating playbooks, runbooks, or standard operating procedures, methods of procedure, emergency operations procedures.
• Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar).
• Ability to work collaboratively with cross-functional, cross-country, multi-disciplinary teams.
• SpaceXSI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.