Senior Data Engineer

Role overview

Data platform and pipelines

• Leads enterprise-scale batch, micro-batch, and real-time streaming data ingestion pipeline design, development, and operation.
• Engineers and manages the Databricks Lakehouse, including Delta Lake optimization, performance tuning, and platform reliability.
• Implements and governs Unity Catalog security controls, lineage, access management, and compliance.
• Builds and maintains dbt transformation frameworks, testing standards, documentation, and CI/CD integration.

DataOps, streaming, and platform operations

• Drives automated testing, monitoring, observability, incident management, and platform support through DataOps practices.
• Manages Confluent/Kafka streaming and CDC capabilities for resilient, scalable real-time data movement.
• Partners with AI Platform Engineering on feature pipelines, vector data services, and AI-powered data products.
• Optimizes cloud spend across Databricks, Azure Data Factory, Confluent, and storage services.
• Ensures data security, privacy, governance, and regulatory compliance.

Team leadership

• Leads, coaches, and develops a high-performing Data Engineering team and establishes engineering standards.

Experience and technical qualifications

• Requires 10+ years of Data Engineering experience, including at least three years in technical leadership.
• Requires Databricks expertise, including Delta Lake, Unity Catalog, Databricks Workflows, PySpark, and Spark SQL.
• Requires Azure Data Factory experience and deep Confluent/Kafka, CDC, schema management, and event-driven architecture experience.
• Requires knowledge of dbt, automated testing, DataOps, CI/CD, platform reliability, data quality, observability, and incident response.
• Requires understanding of cloud cost optimization, FinOps, and platform governance.
• Requires ability to collaborate with Data Architects and Data Modelers and translate architecture into scalable engineering solutions.

Education and certifications

• Requires a bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Databricks Certified Data Engineer Professional and/or Azure Data Engineer certifications are highly desirable.