Lead Engineer- Platform & Data

What You'll Do

• Architect and manage the lakehouse. Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization.
• Own the analytical query layer. Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling, caching, and resource isolation.
• Optimize storage formats. Make deliberate choices about Parquet file layout, including row group sizing, compression codecs, encoding, and predicate pushdown, so data stays cheap to store and fast to query.
• Build and maintain ETL/ELT pipelines. Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability.
• Run event-driven autoscaling on Kubernetes. Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics, so you balance cost against latency.
• Manage infrastructure as code. Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
• Tune self-hosted open-source warehouses. Diagnose bottlenecks across CPU, memory, I/O, and network. Tune configurations, run benchmarks, and drive measurable gains in query latency and cost efficiency.
• Champion reliability. Define SLOs, build monitoring and alerting (Prometheus, Grafana, or similar), participate in on-call, and lead incident retrospectives.
• Mentor and influence. Review designs and code, mentor engineers, and help set technical direction for the data platform.
• What We Need:
• 5+ years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems
• Hands-on expertise with Apache Iceberg / Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization.
• Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, Trino, or similar distributed data warehouses.
• Proficiency in building reliable batch and streaming pipelines using Apache Spark and Flink, with strong fundamentals in data quality, idempotency, fault tolerance, and observability.
• Experience operating production-grade, multi-region cloud platforms across AWS, Azure, or OCI, with a focus on scalability, reliability, performance, and cost efficiency.
• Prior experience in mentoring engineers and leading projects while driving platform initiatives end-to-end.