Lead Data Engineer

Role overview

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines with Databricks and PySpark; build batch and real-time ingestion frameworks.
• Develop data models and transformations with SQL and Python; use Delta Lake for data management, quality, and performance.
• Implement orchestration workflows with Azure Data Factory, Airflow, or Databricks Workflows.
• Collaborate with technical and business stakeholders to understand requirements and deliver data solutions.
• Optimize Databricks jobs and Spark applications for performance, scale, and cost; apply governance, security, and compliance practices.
• Monitor and troubleshoot production pipelines, support critical issues, and contribute to reviews, documentation, and knowledge sharing.
• Lead and manage a data engineering team, set strategy, and ensure delivery of reliable, scalable data infrastructure.

Requirements and qualifications

• Strong or expert Spark/PySpark skills and hands-on pipeline development experience.
• Experience with ingestion patterns, Azure orchestration tools, and ADF or Airflow; SQL and denormalized data modeling for big data systems.
• Expertise in Databricks, PySpark, Python, SQL, and Delta Lake; cloud experience with Azure or AWS.
• Experience building enterprise-scale data solutions for analytics, reporting, and machine learning workloads.
• The grade-specific description calls for team leadership, data engineering strategy, and effective solution delivery.

Technical skills

• Technical skills include Spark/PySpark, Jupyter Notebooks, Colab or Databricks, SQL, denormalized data modeling, and Azure orchestration.
• Azure Data Factory, Airflow, and Databricks Workflows are listed as orchestration options.

Compensation

• The stated base compensation range is $150,405 to $175,908; the description does not specify a pay cadence.

Benefits

• Capgemini describes a benefits package for regular full-time employees; eligibility and available benefits depend on local policy.
• Potential benefits include paid time off, healthcare coverage, retirement savings plans, insurance, and employee assistance programs.