Senior Data Engineer

Role Summary

Key Responsibilities

• · Design, build and maintain data pipelines from APIs, databases, SaaS applications, files and spreadsheets into Databricks.
• · Develop scalable data models and Delta Lake architecture for reporting, analytics and future AI use cases.
• · Manage Databricks Workflows, Lakeflow or Delta Live Tables, SQL Warehouses and related processing jobs.
• · Administer Databricks workspaces, Unity Catalog, permissions, service principals.
• · Support development, test and production environments, including deployment, monitoring, incident resolution and recovery.
• · Monitor platform performance, usage and costs, and implement appropriate security and operational controls.
• · Implement data quality checks covering completeness, accuracy, timeliness, consistency, duplicates and reconciliation with source systems.
• · Ensure dashboards use governed, reliable and well-documented data.
• · Establish data ownership, common definitions, lineage, access policies and governance standards with business functions.
• · Build monitoring and reporting to identify pipeline failures and data quality issues.
• · Integrate governed Databricks data with Power BI and other approved analytical tools.
• · Work with Finance, Delivery, Operations, HR and PMO to translate requirements into sustainable data solutions.
• · Document data pipelines, controls, dependencies and operating procedures.
• · Coordinate external partners where specialist support is required.

Qualifications

• · Significant experience as a Data Engineer, including ownership of production data pipelines.
• · Strong hands-on experience with Databricks, Delta Lake, Python, SQL and PySpark.
• · Experience integrating data from APIs, business applications, databases and files.
• · Experience with Unity Catalog, access management, data governance and data quality controls.
• · Experience administering Databricks workspaces, compute resources and SQL Warehouses.
• · Experience with Azure data services, Power BI, Git and CI/CD practices.
• · Understanding of environment management, monitoring, troubleshooting and cost control.
• · Ability to work directly with non-technical stakeholders and explain data issues clearly.
• · Strong documentation, prioritisation and problem-solving skills.

Certifications (Preferred)