Lead Databricks Data Engineer

Job Description: Lead Databricks Data Engineer

Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Employment Type: Full-Time

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

Key Responsibilities Data Engineering & Architecture

• Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
• Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
• Define enterprise data models, ingestion frameworks, and transformation strategies.
• Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
• Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
• Drive cloud data platform modernization and migration initiatives.

Technical Leadership

• Serve as the technical lead for Databricks and Data Engineering programs.
• Establish engineering best practices, coding standards, and architecture guidelines.
• Conduct architecture reviews, code reviews, and solution assessments.
• Mentor and guide Data Engineers, Technical Leads, and junior team members.
• Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.

Cloud & Platform Engineering

• Design cloud-native data solutions on Azure, AWS, or GCP.
• Implement secure, scalable, and resilient data architectures.
• Optimize costs, performance, reliability, and availability of data platforms.
• Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
• Define and implement monitoring, observability, and operational excellence practices.

Governance & Data Quality

• Establish data governance, lineage, cataloging, and security controls.
• Ensure compliance with organizational and regulatory requirements.
• Implement data quality frameworks and monitoring solutions.
• Collaborate with security and compliance teams to enforce data protection standards.

Stakeholder & Delivery Management

• Partner with business leaders to understand strategic data requirements.
• Translate business requirements into scalable technical solutions.
• Drive delivery planning, risk management, and resource coordination.
• Present architecture decisions and technical recommendations to leadership teams.

Required Technical Skills Databricks

• Databricks Lakehouse Platform
• Databricks Workflows
• Delta Lake
• Unity Catalog
• Structured Streaming
• Databricks SQL
• Delta Live Tables (DLT)
• Databricks Asset Bundles

Big Data Technologies

• Apache Spark
• PySpark
• Spark SQL
• Kafka
• Real-Time Streaming Architectures
• Data Lake & Lakehouse Architectures

Cloud Platforms (Strong Experience in One or More) Azure

• Azure Databricks
• Azure Data Factory (ADF)
• ADLS Gen2
• Azure Synapse Analytics
• Azure Key Vault
• Event Hub

AWS

• S3
• EMR
• Glue
• Redshift
• Lambda
• IAM

GCP

• BigQuery
• Cloud Storage
• Dataflow
• Pub/Sub
• Composer

Programming

• Python
• SQL
• PySpark
• Scala (Preferred)
• Shell Scripting

DevOps & Automation

• Git
• Azure DevOps
• GitHub Actions
• Jenkins
• Terraform
• Docker
• Kubernetes

Data Warehousing

• Dimensional Modeling
• Star & Snowflake Schema
• Data Vault
• Master Data Management
• Data Governance Frameworks

Required Experience

• 12-15 years of overall IT experience.
• Minimum 6+ years of hands-on Databricks experience.
• 7+ years of experience in Spark/PySpark-based data engineering.
• Experience designing enterprise-scale Data Lake and Lakehouse solutions.
• Strong experience leading large data transformation and modernization programs.
• Experience handling structured, semi-structured, and unstructured data processing.
• Experience supporting analytics, AI/ML, and reporting ecosystems.
• Proven experience managing technical teams and driving architectural decisions.

Preferred Qualifications

• Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
• Databricks Certified Data Engineer Professional.
• Azure Data Engineer Associate.
• AWS Data Analytics Specialty.
• Google Professional Data Engineer Certification.
• TOGAF or Cloud Architecture certifications are a plus.

Nice to Have

• MLOps and AI Engineering exposure.
• Snowflake and Databricks integration experience.
• Microsoft Fabric exposure.
• Data Mesh implementation experience.
• Data Governance tools such as Collibra, Alation, or Informatica.
• Expertise in Data Quality frameworks such as Great Expectations or Deequ.

Key Competencies

• Solution Architecture
• Technical Leadership
• Stakeholder Management
• Program Delivery
• Data Strategy & Governance
• Team Mentoring & Coaching
• Problem Solving & Decision Making
• Communication & Executive Presentation Skills
• Performance Optimization
• Innovation & Continuous Improvement

Technology Classification

Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

Qualifications

Graduate with 4 years BS or Professional Degree. 15 - 18 Years

Responsibilities

Management of DB Systems, Performance Tuning, Security, Compliance Standards,