Python Databricks - Senior Engineer

Mandatory skills

• PySpark, Apache Kafka, Databricks Workflows, and Delta Lake on Databricks are listed as mandatory.

Data engineering and architecture

• Design scalable data engineering solutions using PySpark and distributed processing frameworks.
• Define ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions for enterprise platforms.
• Lead implementation of high-performance batch and streaming pipelines.
• Design event-driven architectures using Apache Kafka or Amazon Kinesis.
• Define streaming standards, integration frameworks, and scalable processing patterns.

Orchestration and platform quality

• Architect workflow orchestration using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices.
• Apply Lakehouse architecture, data observability, and platform engineering standards to improve scalability, reliability, and operational visibility.

Data products and delivery

• Improve data product quality, discoverability, usability, documentation, and trusted analytical consumption.
• Promote responsible AI-assisted engineering for productivity, testing, documentation, and engineering quality.
• Review pipeline designs and implementations against engineering, scalability, and performance standards.
• Troubleshoot data processing, workflow, and streaming issues through root cause analysis.
• Mentor team members and collaborate with teams and stakeholders on end-to-end data platform delivery.

Behavioral competencies

• Demonstrate ownership and drive data engineering excellence.
• Collaborate with teams and business stakeholders to support delivery.
• Promote quality-focused engineering through validation, optimization, and continuous improvement.
• Apply analytical thinking to complex data engineering and platform challenges.

About Iris Software