Principal Systems Engineer
Full Job Posting
Responsibilities
- Qualifications / Experience:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
- 8+ years of experience in infrastructure engineering, cloud operations, site reliability engineering, production operations, or a related technical discipline.
- Expert-level Linux administration skills, particularly with Ubuntu and Oracle Linux in large-scale production environments.
- Proven ability to troubleshoot complex infrastructure issues across hardware, operating systems, networking, drivers, firmware, and service dependencies.
- Strong understanding of distributed systems and how compute, networking, storage, and application services interact to affect reliability and customer experience.
- Hands-on experience with automation, scripting, and configuration management using tools such as Python, shell scripting, Ansible, and Terraform.
- Strong knowledge of incident management, service management, root cause analysis, and operational improvement.
- Experience with infrastructure provisioning, configuration, monitoring, maintenance, and host lifecycle management.
- Solid understanding of the OSI model, TCP/IP, HTTP, and structured troubleshooting across multiple infrastructure layers.
- Demonstrated ability to serve as a senior technical escalation point and independently drive complex issues to resolution.
- Demonstrated success using operational metrics—such as mean time to restore service (MTTR), ticket age, incident volume, recurring issue rates, automation coverage, and service level performance—to drive measurable improvements.
- Strong planning and organizational skills, including prioritizing work, managing dependencies, tracking execution, and assessing operational risks.
- Clear written and verbal communication skills, with the ability to explain technical issues, customer impact, risks, and recommendations to technical and business stakeholders.
- Infrastructure/Networking:
- Operate and improve Linux-based compute infrastructure to support reliable, scalable AI, GPU, and inference workloads.
- Apply strong networking knowledge across TCP/IP, IPv4/IPv6, DNS, DHCP, VPNs, and SSL/TLS to diagnose connectivity and service issues.
- Troubleshoot communication failures between hosts, nodes, and distributed services, identifying dependencies and potential points of failure.
- Use Ansible, Terraform, Python, and shell scripting to standardize infrastructure configuration and automate operational tasks.
- Support datacentre operations, including host provisioning, maintenance, hardware troubleshooting, repair coordination, and decommissioning.
- Collaborate with infrastructure and networking teams to improve availability, capacity, configuration consistency, and operational readiness.
- Incident Management:
- Lead the technical response and coordination of critical infrastructure incidents, from initial triage through service restoration, recovery validation, and handoff.
- Assess customer and workload impact, clearly distinguishing confirmed issues from symptoms still under investigation.
- Coordinate across the Network Operations Centre (NOC), networking, datacentre operations, GPU and interconnect engineering, hardware repair, and service delivery teams to drive timely resolution.
- Provide calm, clear communication during high-severity incidents, keeping customers and stakeholders informed of impact, progress, risks, and next steps.
- Maintain accurate incident records, including timelines, accountable owners, mitigation actions, decisions, and escalation paths.
- Validate recovery with customers and technical owners before completing incident closure or transitioning to follow-up work.
- Lead root cause analysis and post-incident reviews, ensuring corrective and preventive actions are assigned and tracked to completion.
- Use telemetry, operational trends, and support ticket data to identify recurring issues, prioritize repairs, and strengthen incident response.
- Technical Leadership:
- Provide hands-on technical leadership for compute and inference infrastructure as a senior individual contributor.
- Guide engineers through complex troubleshooting, infrastructure improvements, and operational decisions.
- Mentor team members, share technical knowledge, and contribute to hiring and onboarding.
- Set high standards for engineering quality, ownership, documentation, and operational discipline.
- Drive the adoption of automation, observability, reliable incident response, and continuous improvement.
- Contribute to architecture and design reviews, balancing reliability, scalability, maintainability, cost, and business priorities.
- Lead technical initiatives across teams, aligning stakeholders on priorities, responsibilities, and delivery plans.
- Engineering and Automation:
- Develop and maintain automation that improves reliability, reduces repetitive manual work, and accelerates incident diagnosis and recovery.
- Build reusable scripts, operational tools, and workflows using Python, shell scripting, Ansible, Terraform, or similar technologies.
- Standardize infrastructure provisioning, configuration, validation, maintenance, and recovery procedures.
- Create and maintain clear runbooks and technical documentation to support consistent execution and knowledge sharing.
- Partner with engineering teams to turn recurring operational problems into sustainable engineering improvements.
- Support safe infrastructure changes through testing, staged deployment, validation, and rollback planning.
- Cross-Functional & Stakeholder Engagement:
- Partner with Product, backend service engineering, datacentre operations, networking, and customer support teams to deliver reliable infrastructure services.
- Serve as a technical escalation point for complex production issues and critical infrastructure decisions.
- Translate customer requirements and operational challenges into clear technical priorities and improvement plans.
- Communicate infrastructure health, service risks, dependencies, and delivery progress to relevant stakeholders.
- Coordinate service readiness and operational handoffs, ensuring teams have the monitoring, documentation, and support procedures needed for production.
- Collaborate effectively with globally distributed teams to maintain clear ownership and timely execution.
- Performance Optimization:
- Benchmark and tune infrastructure to improve performance, reliability, scalability, and resource efficiency.
- Analyze GPU utilization and identify opportunities to improve capacity use and workload efficiency.
- Investigate performance bottlenecks across compute, memory, storage, networking, and service dependencies.
- Improve performance consistency and resource isolation in multi-tenant environments.
- Partner with service and workload owners to diagnose customer performance issues and validate improvements.
- Use performance data to recommend configuration changes, capacity adjustments, and engineering priorities.
- Monitoring and Telemetry:
- Build and maintain monitoring, dashboards, and telemetry that provide clear visibility into infrastructure health and customer experience.
- Use Prometheus, Grafana, and related tools to track availability, performance, capacity, and service reliability.
- Define actionable alerts and improve alert quality to support early detection and efficient troubleshooting.
- Correlate metrics, logs, events, and support ticket data to identify failures and emerging operational risks.
- Establish and track service level indicators and objectives, partnering with service owners to address reliability gaps.
- Improve monitoring coverage and diagnostic information across GPU hosts, networking, hardware, and supporting services.
- Preferred / Nice to Have:
- Experience operating large-scale GPU fleets, AI/ML infrastructure, inference platforms, or high-performance computing (HPC) environments.
- Advanced experience with NVIDIA and AMD GPU ecosystems, including GPU diagnostics, performance analysis, and platform troubleshooting.
- Knowledge of high-performance networking technologies such as InfiniBand, RDMA over Converged Ethernet (RoCE), and Remote Direct Memory Access (RDMA).
- Experience with Kubernetes, containers, and workload orchestration in production environments.
- Familiarity with CUDA, NCCL, Slurm, or similar AI and HPC technologies.
- Experience building or scaling GPU cloud platforms, AI factories, or large-scale inference infrastructure.
- Experience providing technical leadership in 24×7 production operations with globally distributed stakeholders.
- Experience supporting multi-region infrastructure deployments or working with hyperscalers, cloud providers, or AI-focused technology companies.
- Experience improving operations through AIOps, event correlation, triage automation, alert noise reduction, and standardized runbooks.
- Experience with patching, upgrades, staged rollouts, change management, operational risk assessment, audit readiness, and compliance requirements.
Qualifications
- Hands-on experience with automation, scripting, and configuration management using tools such as Python, shell scripting, Ansible, and Terraform.
- Experience with infrastructure provisioning, configuration, monitoring, maintenance, and host lifecycle management.
- Experience with Kubernetes, containers, and workload orchestration in production environments.
- Familiarity with CUDA, NCCL, Slurm, or similar AI and HPC technologies.
- Experience with patching, upgrades, staged rollouts, change management, operational risk assessment, audit readiness, and compliance requirements.
About Oracle
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Oracle
Senior Platform Software Engineer
Bengaluru, IND
Owns moderately complex components within platform services or SDKs; leads team-level improvements to integration frameworks and developer tooling. Performs deep debugging across a bounded set of distributed services, dr
Senior Platform Software Engineer
Bengaluru, IND
Owns moderately complex components within platform services or SDKs; leads team-level improvements to integration frameworks and developer tooling. Performs deep debugging across a bounded set of distributed services, dr
Lead Principal Platform Software Engineer
Bengaluru, IND
Drives cross-group platform initiatives (e.g., identity, config, API gateways) with significant impact on reliability, performance, and compliance across the group. Anticipates and resolves ambiguous distributed-systems
Principal Platform Software Engineer
Bengaluru, IND
Leads platform projects crossing multiple teams, evolving runtimes or middleware patterns for interoperability and multi-tenant scale. Diagnoses and resolves complex cross-service issues; enforces versioning, compatibili
Senior Platform Software Engineer
Bengaluru, IND
Owns moderately complex components within platform services or SDKs; leads team-level improvements to integration frameworks and developer tooling. Performs deep debugging across a bounded set of distributed services, dr
Principal Platform Software Engineer
Bengaluru, IND
Leads platform projects crossing multiple teams, evolving runtimes or middleware patterns for interoperability and multi-tenant scale. Diagnoses and resolves complex cross-service issues; enforces versioning, compatibili
Senior Manager, Platform Software Engineering
Bengaluru, IND
Leads one or more platform teams; defines tactical and key strategic elements for middleware evolution serving multiple consumers. Accountable for performance, cost control, and reliability outcomes; escalations for dist
Senior Platform Software Engineer
Bengaluru, IND
Owns moderately complex components within platform services or SDKs; leads team-level improvements to integration frameworks and developer tooling. Performs deep debugging across a bounded set of distributed services, dr
Senior Platform Software Engineer
Bengaluru, IND
Senior Platform Software Engineer
Bengaluru, IND
Lead Principal Platform Software Engineer
Bengaluru, IND
Principal Platform Software Engineer
Bengaluru, IND
Senior Platform Software Engineer
Bengaluru, IND
Principal Platform Software Engineer
Bengaluru, IND
Senior Manager, Platform Software Engineering
Bengaluru, IND
Senior Platform Software Engineer
Bengaluru, IND