Staff Software Engineer - Open Source Data Platform (ODP)

RESPONSIBILITIES

• Own the end-to-end architecture of major platform components, driving design decisions that impact scalability, reliability, and performance.
• Architect and implement next-generation distributed data storage, processing, and query optimization systems.
• Partner with product, engineering leadership, and stakeholders to define technical roadmaps and translate business requirements into scalable solutions.
• Lead contributions to Apache projects and open-source communities, establishing Acceldata's presence and technical reputation.
• Identify and resolve complex performance bottlenecks across distributed systems, optimizing for cost, latency, and throughput.
• Mentor and guide Senior Software Engineers, fostering technical excellence and a culture of continuous learning.
• Establish and enforce coding standards, perform thorough code reviews, and ensure architectural consistency across the platform.
• Evaluate and recommend emerging technologies, tools, and frameworks that align with platform goals.
• Lead root cause analysis for critical production issues and drive systemic improvements to prevent recurrence.
• Work across diverse environments: Bare Metals, VM, Kubernetes, and multi-cloud architectures.

REQUIREMENTS

• 9+ years of hands-on software development experience with at least 5 years focused on distributed systems or big data platforms.
• Expert-level proficiency in Java or Scala, with strong Python skills. Experience with systems languages is a plus.
• Deep understanding of distributed computing principles, including consensus algorithms, data partitioning, replication, fault tolerance, and consistency models.
• Extensive hands-on experience with multiple components of the big data stack (Hadoop, Spark, Hive, Trino, Kafka, Flink, NiFi or similar).
• Expert-level Linux proficiency, including kernel tuning, performance analysis, and debugging at the system level.
• Strong experience with Kubernetes, Docker, and cloud platforms (AWS, GCP, or Azure).
• Deep expertise with Maven, Gradle, or SBT; experienced in CI/CD pipelines, GitHub, and artifact management (Nexus).
• Proven ability to diagnose and resolve complex issues, including memory leaks, deadlocks, GC tuning, and distributed system failures.
• Strong understanding of architectural patterns for scalable, fault-tolerant distributed systems.
• Excellent written and verbal communication skills; ability to articulate complex technical concepts to diverse audiences, including senior leadership.
• Demonstrated ability to work effectively with cross-functional teams, external contributors, and the open-source community.
• History of significant contributions or committer status in Apache projects (e.g., Spark, Kafka, Hive, Iceberg, Flink, etc.).
• Experience with modern table formats (Apache Iceberg, Delta Lake, Hudi) and data lake architectures.
• Experience developing or optimizing query engines, query planners, or execution frameworks.
• Familiarity with data governance, security, lineage, and compliance frameworks.
• Experience leading large-scale data platform migrations or Hadoop distribution upgrades.
• Published technical blogs, papers, or conference presentations on distributed systems topics.
• Experience leading or maintaining open-source projects.

PREFERRED SKILLS