greenhouse
Site Reliability Engineer
xAI
Southaven, USA
Mid · 5+ years experience
Onsite
Discovered 1 weeks ago
bashgrafanajavakubernetesprometheuspython
Role Overview
As a Site Reliability Engineer focused on campus reliability, you will design what the campus watches and trusts, technically command cross-discipline SEVs, and build the guardrails that make the next incident smaller.
You are the connective tissue across compute, network, storage, power, and cooling on one of the largest AI training campuses in the world.
This role demands calm incident leadership, fleet-scale observability judgment, and the ability to drive reliability work across software and facility boundaries.
We are looking for candidates from power plants, nuclear power plants, data centers, or people who currently work or have worked in facilities like SpaceXSI with power, compute, cooling, and related plant systems.
Key Skills for This Role
bashgrafanajavakubernetesprometheuspython
Full Job Posting
RESPONSIBILITIES
- Own monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure.
- Provide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., PagerDuty), and timeline and severity hygiene.
- Run blameless postmortems and drive corrective actions to closed, not filed.
- Gate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process.
- Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing).
- Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs.
- Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects).
- Define error budgets and availability objectives at campus and service boundaries as adopted by the business.
- Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus.
- PROFILE SIGNALS:
- Experience in power plants, nuclear power plants, data centers, or facilities like SpaceXSI — current or prior — spanning power, compute, cooling, or related plant systems.
- Proven large-scale incident command experience and calm technical leadership on a bridge. Nuclear control-room, shift supervisor / Senior Reactor Operator (SRO), Shift Technical Advisor, or Navy nuclear experience counts as incident command.
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality. Treats alert noise as a design failure, not an operator failure.
- Cross-team facilitation and systems engineering depth across software and facility boundaries.
- SUCCESS MEASURED BY:
- MTTD / MTTR trend for SEV-class events
- % of SEVs with a blameless postmortem and closed action items
- Alert actionable ratio
- Recurrence rate of incident classes
- Monitoring coverage of critical dependencies across compute, network, storage, power, and cooling
- GPU fleet availability lost to infrastructure events
BASIC QUALIFICATIONS
- Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field (or equivalent experience).
- 5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations, preferably in power plants, nuclear power plants, high-performance computing, or data center environments.
- Proven large-scale incident command experience and calm technical leadership on a bridge, whether from an SRE/NOC bridge or a plant control room.
- Demonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.
- Hands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
- Experience working across at least two of: compute, network, storage, power, and cooling / facilities telemetry.
- Experience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control room, or NOC partner.
- Proficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar). Not required to be expert in all of them.
- Excellent problem-solving skills with a data-driven approach to reliability engineering.
- Ability to work collaboratively with cross-functional teams, including NOC, data center operations, hardware failure analysis, and infrastructure engineering.
PREFERRED SKILLS AND EXPERIENCE
- Experience in power plants, nuclear power plants, or other high-consequence industrial control environments (ICS/OT).
- Current or prior work in data centers or facilities like SpaceXSI spanning power, compute, cooling, and related plant systems, including CDU / direct liquid cooling.
- Experience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.
- Fleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.
- Reliability engineering methods: MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, and vendor RCA/RMA partnership.
- Experience with CMMS (e.g., Maximo) and commissioning / integrated systems testing (IST) and operational turnover.
- Hands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.
- Experience running game days, dependency mapping, and closed-loop corrective action programs.
- Familiarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.
- Prior work in a fast-paced startup or tech company like SpaceXSI.
- SpaceXSI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.
About xAI
Technology1,001-5,000Founded 2023
xAI is an artificial intelligence company founded by Elon Musk that develops AI systems to accelerate scientific discovery. Its main products include the Grok AI chatbot and the Colossus supercomputer, one of the world's largest AI training clusters.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at xAI
Construction Superintendent, Electrical - Memphis
Southaven, USA
MidFull-time
Discovered 1 weeks agoView →
Regulatory Legal Counsel
, GBR
Discovered Today
Senior Analyst, Safety Operations
Austin, USA
Discovered Today
Security Engineer - GRC Federal Frameworks
Austin, USA
Discovered 3 days ago
Supervisor, Critical Facility Operations
Southaven, USA
Discovered 1 weeks ago
Senior Facilities Operations Technician - Memphis
Southaven, USA
Discovered 1 weeks ago
Facilities Operations Technician - Memphis
Southaven, USA
Discovered 1 weeks ago
Construction Superintendent, Electrical - Memphis
Southaven, USA
Discovered 1 weeks agoFull-time
AI Tutor - Yoruba
, USA
Discovered 1 weeks ago