Databricks - Senior Engineer

Role Overview

Mandatory Skills

• PySpark.
• Apache Kafka.
• Databricks Workflows.
• Delta Lake on Databricks.

Key Responsibilities

• Design scalable data engineering solutions using PySpark and distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions for enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Design event-driven data architectures using Apache Kafka or Amazon Kinesis.
• Define streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices.
• Apply Lakehouse architecture, data observability, and platform engineering standards.
• Improve data product quality, discoverability, usability, documentation, and trusted consumption.
• Review implementations, troubleshoot complex platform issues, mentor team members, and support delivery.

Behavioral Competencies

• Demonstrate ownership while driving data engineering excellence.
• Collaborate effectively with teams and business stakeholders.
• Promote quality-focused engineering through validation, optimization, and continuous improvement.
• Apply analytical thinking to complex data engineering and platform challenges.
• Adapt to evolving technologies, data ecosystems, and business requirements.
• Communicate delivery status, risks, dependencies, and improvement opportunities effectively.

Benefits

• Benefits are designed to support financial, health, and well-being needs.

Requirements

• Demonstrated mandatory skills in PySpark, Apache Kafka, Databricks Workflows, and Delta Lake on Databricks.
• Ability to design scalable data engineering solutions using PySpark and distributed data processing frameworks.
• Ability to design and implement high-performance batch and streaming data pipelines.
• Ability to design event-driven data architectures and workflow orchestration solutions.
• Strong analytical thinking for complex data engineering and platform challenges.
• Ability to collaborate effectively with technical teams and business stakeholders.
• Ability to communicate delivery status, risks, dependencies, and improvement opportunities effectively.
• Experience with Snowflake or Delta Lake on Databricks solutions.
• Experience with Apache Airflow or Databricks Workflows for orchestration.
• Experience with Apache Kafka or Amazon Kinesis for event-driven architectures.