Manager, Supercomputer Site Reliability Engineering - Memphis
Role Overview
SpaceXSI operates multi-campus, hyperscale GPU infrastructure where cluster availability directly determines training throughput, customer handover commitments, and capital efficiency of the fleet. The Manager of Supercomputer Site Reliability Engineering will be responsible for both compute and infrastructure performance, and will be responsible for significantly increasing the SRE team size. On the compute side, focus will be put on continuous site monitoring, cross-discipline incident command, deep hardware root-cause analysis (RCA), and emphasis on software systems that orchestrate repair velocity. Additional focus will be on physical infrastructure reliability as well. Understanding system margins and resiliency will be key.
Site Reliability engineering manager owns >99% compute uptime outcome, drives unplanned facilities compute losses to 0, ensures efficient allocation of > 3GW site power, upgrade robust fault-free megapack failover systems capable of supporting all halls for >4 hours (160 MWhr per hall), drives robust hardware inventory systems and repair velocity to eliminate hardware chokepoints.
Key Skills for This Role
Full Job Posting
ABOUT THE ROLE:
SpaceXSI operates multi-campus, hyperscale GPU infrastructure where cluster availability directly determines training throughput, customer handover commitments, and capital efficiency of the fleet. The Manager of Supercomputer Site Reliability Engineering will be responsible for both compute and infrastructure performance, and will be responsible for significantly increasing the SRE team size. On the compute side, focus will be put on continuous site monitoring, cross-discipline incident command, deep hardware root-cause analysis (RCA), and emphasis on software systems that orchestrate repair velocity. Additional focus will be on physical infrastructure reliability as well. Understanding system margins and resiliency will be key.
Site Reliability engineering manager owns >99% compute uptime outcome, drives unplanned facilities compute losses to 0, ensures efficient allocation of > 3GW site power, upgrade robust fault-free megapack failover systems capable of supporting all halls for >4 hours (160 MWhr per hall), drives robust hardware inventory systems and repair velocity to eliminate hardware chokepoints.
RESPONSIBILITIES:
- Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions.
- Lead integrated Supercompute systems engineering reliability team responsible for reactive and increasingly proactive cluster and facilities health and reliance initiatives
- Build out the site reliability engineering sub-teams: compute hardware failure analysis, software tools, cluster monitoring, and integrated site reliability engineering
- Drive operational superiority
- Define SRE organizational structure and develop culture of 24/7 operational excellence
- Physics limit anomaly response times and robust proactive health processes
- Define on-call responsibilities, incident management protocols, leadership structure, and communication discipline
- Develop training schemes for tough, competent, and disciplined monitoring and response teams
- Establish clear operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners
- Own vendor oversight and accountability mechanisms
- Drive mean time to repair toward physics limits
- Define and continuously improve operational procedures and analysis documentation mechanisms with strong emphasis on intelligent automation
- Lead robustification of documentation workflow
- Ensure efficient signal to noise in execution
- Define operational targets, health indicators, alerting standards, and incident severity definitions that facilitate measurable team performance insights and accountability mechanisms
- Cultivate engineering excellence
- Drive high technical standard for hardware root cause analysis and remediation teams ensuring industry-leading hardware failure diagnostic expertise
- Integrate with platform engineering teams to develop sophisticated hardware and system understanding to drive high operational reliability
- Drive repeat hardware issues to a minimum and lead automation efforts to eliminate manual overhead for unavoidable remediation needs
BASIC QUALIFICATIONS:
- Bachelor’s degree and 3+ years of experience working in a large scale operations with 2+ years leading people leaders of technical teams or 10+ years of experience working in a large scale operations with 5+ years leading people leaders of technical teams.
- Willingness to travel frequently to data center locations to support operations across sites.
- Physical capability to handle data center tasks, including lifting up to 50 lbs unassisted, standing for long periods, and occasional ladder use.
- Must be willing to work extended hours and/or weekends as needed.
- SpaceXSI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.
PREFERRED SKILLS AND EXPERIENCE:
- Proven ability to lead large, multi-site, 24/7 operations organizations in fast-paced, high-responsibility settings.
- Deep expertise in server hardware, cluster reliability, and data center technologies, from deployment through lifecycle management.
- Experience supporting compute-heavy environments like AI, machine learning, or high-performance computing at scale.
- A track record of owning uptime, Service Level Agreements, or reliability metrics for large compute clusters.
- Experience leading site reliability engineering or equivalent reliability-focused teams, including root cause analysis and procedure ownership.
- Strong analytical skills and the ability to explain technical concepts clearly to diverse audiences, from technicians to executive and tenant partners.
- A history of partnering with vendors at scale, driving mean time to repair down, and scaling operations across multiple sites.
- Familiarity with tooling and automation (e.g., Jira, Python, Bash) used to monitor cluster health and improve team efficiency.
- Enthusiasm for SpaceXAI's mission to accelerate human discovery and unravel the universe.
- Ability to thrive in a dynamic, mission-focused environment with on-call ownership of cluster-impacting events.
About SpaceXSI
Artificial intelligence research and development company.
Visit company websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at SpaceXSI
Sr. Manager, Power Plant Operations - Memphis
, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Supercompute Operations Engineering
, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Regulatory Legal Counsel
London, GBR
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Supervisor, Fleet Maintenance
, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Megapack Field Technician - Memphis
, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Supercompute Operations Specialist - Memphis
Memphis, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Software Engineer, Observability (Supercompute)
Memphis, USA
SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this pos
Client Partner (Dubai)
, UAE
SpaceXSI is seeking an experienced Client Partner for its X Ads Sales MENA team to grow advertising revenue through enterprise and agency relationships. Candidates need 10+ years of digital media sales experience with ME
Sr. Manager, Power Plant Operations - Memphis
, USA
Supercompute Operations Engineering
, USA
Regulatory Legal Counsel
London, GBR
Supervisor, Fleet Maintenance
, USA
Megapack Field Technician - Memphis
, USA
Supercompute Operations Specialist - Memphis
Memphis, USA
Software Engineer, Observability (Supercompute)
Memphis, USA
Client Partner (Dubai)
, UAE