{bc}
Apply
greenhouse

Platform Reliability Engineer

Firmus Technologies
Melbourne, AUS
Mid
Onsite
Discovered 1 weeks ago
AnsiblePythonCommunicationKubernetesServiceNow

Full Job Posting

Key Responsibilities

  • Triage, diagnose and resolve incidents across the stack, including Kubernetes clusters and workloads, storage, virtualisation, network connectivity and the underlying Linux hosts, owning each incident from escalation through to resolution.
  • Operate the multi-tenant control and management plane that Firmus' AI and infrastructure services depend on, including the tenancy, quota and access controls that support separation between tenant workloads and data.
  • Operate the virtualisation platform that hosts the management plane, and the shared services that run on it, to their declared service levels.
  • Operate Firmus' exabyte-scale distributed and high-performance filesystems and S3-compatible object storage to their declared service levels, handling capacity, performance and availability incidents and escalating to specialist engineers and vendors (for example VAST or WEKA) with clear evidence when further investigation is needed.
  • Operate the GPU compute fleet in production: node health and readiness, GPU and node fault detection and handling, firmware and driver currency to the supported baselines, remediation and return to service, and support for cluster provisioning and validation activities.
  • Reduce toil by converting repeat fixes into runbooks and automated remediation delivered as controlled code, contributing to the team's automation and remediation tooling.
  • Diagnose performance problems across the data path between compute and storage, gathering the evidence needed to resolve them and working with network and storage specialists where required.
  • Deliver change as code through the agreed change process, with peer review, testing, staged rollout where applicable, and a rollback plan.
  • Use the observability stack (for example Grafana, Prometheus, Loki and Tempo) to track service level indicators against their objectives, query metrics, logs and traces, correlate signals across services and diagnose faults, and improve dashboards and alerting so that each alert is actionable and tied to a runbook.
  • Execute the patching cycle and urgent vulnerability remediation across the services the team operates.
  • Manage vendor support cases across the hardware and software portfolio, raising and tracking cases under the relevant support agreements, supplying logs and diagnostic evidence, and following cases through to agreed response and resolution times.
  • Take part in technical recovery during major incidents and in blameless post-incident reviews, feed recurring issues back to the engineering teams as clear requirements, share the follow-the-sun on-call roster, and keep incident tickets, operational procedures and runbooks accurate and current so that others can execute them successfully.

Required Skills

  • Solid experience operating production infrastructure in a 24/7 environment using site reliability engineering practices, including service level objectives, toil reduction, owning incidents from escalation to resolution, on call participation, post-incident review and writing runbooks that others can execute successfully.
  • Strong working knowledge of Linux systems administration and troubleshooting, including storage, host networking, processes, memory and I/O, and a systematic approach to isolating faults.
  • Working experience with scale-out, parallel or enterprise storage (for example VAST, WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance and capacity problems using evidence rather than assumption.
  • Familiarity with a virtualisation platform (for example Proxmox, VMware or KVM).
  • Working experience using observability tooling to diagnose faults (for example Grafana, Prometheus, Loki, Tempo or Elasticsearch), including writing queries across metrics and logs (such as PromQL and LogQL) and correlating signals across services.
  • Working experience with infrastructure-as-code and GitOps practices (for example OpenTofu or Terraform, Ansible, Argo CD), and writing scripts or small tools for operational automation in a language such as Python or Go.
  • Familiarity with server hardware operation and support, including working with hardware and software vendors to raise support cases and supply diagnostic evidence.
  • Familiarity with cyber security principles and how they apply to operating production infrastructure.
  • Clear communication skills, with the ability to write incident updates, ticket notes and escalations that others can act on, and a working style suited to a team that shares accountability for a broad portfolio across regions.
  • Preferred Experience
  • Experience in infrastructure operations for GPU or HPC environments and the demands of large-scale distributed training and inference, including GPU health and fault signals (for example DCGM health checks, NCCL tests and XID errors).
  • Experience in multi-tenant service provider, cloud or colocation environments with contractual response commitments and customer-facing reporting.
  • Working knowledge of networking fundamentals, including TCP/IP, DNS, routing, VLANs, load balancing and firewalls, and the ability to trace a connectivity fault across hosts, switches and services.
  • Knowledge of high-performance fabrics and data-path technologies, such as InfiniBand, RoCE, RDMA and GPUDirect Storage, and DPU or SmartNIC-based networking.
  • Familiarity with bare-metal lifecycle and provisioning automation (for example NVIDIA Infra Controller, MAAS, Ironic or Metal3), and standard operating environments with known-good-state enforcement.
  • Hands-on experience operating and troubleshooting Kubernetes, including workloads, networking, storage and cluster health, and multi-tenant and GPU-enabled Kubernetes patterns.
  • Experience with ITSM and incident management tooling (for example Jira Service Management, ServiceNow or PagerDuty).
  • Experience working under ISO 27001 or SOC 2 controls.
  • Relevant certification, such as Certified Kubernetes Administrator (CKA), a Linux certification (for example RHCSA or LFCS), or a vendor certification.
  • A Bachelor's degree in computer science, engineering or a related discipline, or an equivalent combination of relevant experience and training.
  • Location & Reporting
  • Location: Based in Australia or Singapore, with travel to Australian AI Factory sites as required.
  • On-call: The function runs 24/7. First line monitoring and first response sit with the operations centre. This role shares the after-hours escalation roster for its domain with the other senior engineers in the function.
  • Reporting to: Reports to the Head of AI FactoryOS Operations while the function is being established, working under broad direction with a high degree of autonomy and direct access to the decision makers. As the function reaches its planned structure, the role will report to the Infrastructure Operations Manager, with the Head of AI FactoryOS Operations remaining accountable for the function. The scope, level and remit of the role do not change under either arrangement.
  • Employment Basis
  • Permanent full-time
  • Diversity
  • At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.
  • Join us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.

Qualifications

  • Solid experience operating production infrastructure in a 24/7 environment using site reliability engineering practices, including service level objectives, toil reduction, owning incidents from escalation to resolution, on call participation, post-incident review and writing runbooks that others can execute successfully.
  • Strong working knowledge of Linux systems administration and troubleshooting, including storage, host networking, processes, memory and I/O, and a systematic approach to isolating faults.
  • Working experience with scale-out, parallel or enterprise storage (for example VAST, WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance and capacity problems using evidence rather than assumption.
  • Familiarity with a virtualisation platform (for example Proxmox, VMware or KVM).
  • Working experience using observability tooling to diagnose faults (for example Grafana, Prometheus, Loki, Tempo or Elasticsearch), including writing queries across metrics and logs (such as PromQL and LogQL) and correlating signals across services.
  • Working experience with infrastructure-as-code and GitOps practices (for example OpenTofu or Terraform, Ansible, Argo CD), and writing scripts or small tools for operational automation in a language such as Python or Go.
  • Familiarity with server hardware operation and support, including working with hardware and software vendors to raise support cases and supply diagnostic evidence.
  • Familiarity with cyber security principles and how they apply to operating production infrastructure.
  • Clear communication skills, with the ability to write incident updates, ticket notes and escalations that others can act on, and a working style suited to a team that shares accountability for a broad portfolio across regions.
  • Experience in infrastructure operations for GPU or HPC environments and the demands of large-scale distributed training and inference, including GPU health and fault signals (for example DCGM health checks, NCCL tests and XID errors).
  • Experience in multi-tenant service provider, cloud or colocation environments with contractual response commitments and customer-facing reporting.
  • Working knowledge of networking fundamentals, including TCP/IP, DNS, routing, VLANs, load balancing and firewalls, and the ability to trace a connectivity fault across hosts, switches and services.
  • Knowledge of high-performance fabrics and data-path technologies, such as InfiniBand, RoCE, RDMA and GPUDirect Storage, and DPU or SmartNIC-based networking.
  • Familiarity with bare-metal lifecycle and provisioning automation (for example NVIDIA Infra Controller, MAAS, Ironic or Metal3), and standard operating environments with known-good-state enforcement.
  • Hands-on experience operating and troubleshooting Kubernetes, including workloads, networking, storage and cluster health, and multi-tenant and GPU-enabled Kubernetes patterns.
  • Experience with ITSM and incident management tooling (for example Jira Service Management, ServiceNow or PagerDuty).
  • Experience working under ISO 27001 or SOC 2 controls.
  • Relevant certification, such as Certified Kubernetes Administrator (CKA), a Linux certification (for example RHCSA or LFCS), or a vendor certification.
  • A Bachelor's degree in computer science, engineering or a related discipline, or an equivalent combination of relevant experience and training.
  • Location & Reporting
  • Location: Based in Australia or Singapore, with travel to Australian AI Factory sites as required.
  • On-call: The function runs 24/7. First line monitoring and first response sit with the operations centre. This role shares the after-hours escalation roster for its domain with the other senior engineers in the function.
  • Reporting to: Reports to the Head of AI FactoryOS Operations while the function is being established, working under broad direction with a high degree of autonomy and direct access to the decision makers. As the function reaches its planned structure, the role will report to the Infrastructure Operations Manager, with the Head of AI FactoryOS Operations remaining accountable for the function. The scope, level and remit of the role do not change under either arrangement.
  • Employment Basis
  • Permanent full-time
  • Diversity
  • At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.
  • Join us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.

Preferred Qualifications

  • Preferred Experience

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Firmus Technologies

Systems Administrator

Sydney, AUS

Discovered 3 days ago

Head of AI Factory Commissioning

Sydney, AUS

Discovered 4 days ago

Platform Reliability Engineering Manager

Sydney, AUS

Discovered 5 days ago

Platform Security Operations Manager

Sydney, AUS

Discovered 1 weeks ago

Office Administrator Team Assistant

Launceston, AUS

Discovered 1 weeks ago

Project Manager - FIFO to Tasmania

Sydney, AUS

Discovered 1 weeks agoFull-time

Network Engineer

Sydney, AUS

Discovered 1 weeks ago