Lead Data Engineer (Databricks)

River Water Quality Programme Warrington (Hybrid – 3 days onsite per week)

At United Utilities , we’re committed to protecting the environment that millions of people depend on every day. As part of our Digital Services team, you’ll play a pivotal role in one of our most important and high-impact initiatives: near real time monitoring of river water quality .

We’re looking for a Lead Data Engineer to take technical ownership of modern, scalable data platforms that turn complex environmental and telemetry data into trusted insights for regulators, scientists, and operational teams.

As the Lead Data Engineer on our River Water Quality Programme, you’ll design, build, and optimise Databricks-based data pipelines that enable timely, reliable and well-governed data across the organisation.

You’ll:

Design and develop scalable batch and streaming data pipelines using Databricks, Spark and Delta Lake

Implement medallion architecture (bronze/silver/gold) to ensure data quality, lineage and efficient consumption

Optimise Spark jobs and workflows to meet near real-time monitoring SLAs

Build and maintain streaming pipelines using Structured Streaming and Delta Live Tables

Ensure strong data governance , including schema enforcement, validation rules and ACID compliance

Orchestrate and automate pipelines using Databricks Jobs, APIs or cloud-native schedulers

Collaborate with data scientists, analysts and environmental specialists to deliver high-quality, well-documented datasets

Monitor, troubleshoot and continuously improve production pipelines for reliability, performance and cost efficiency

Champion best practices across CI/CD, testing, version control and documentation

Act as a technical leader and mentor , raising standards across the data engineering community

Deep expertise in Databricks (Spark / PySpark, Delta Lake, Unity Catalog, Workflows)

Strong experience building ETL/ELT pipelines , both batch and streaming

Advanced Python and SQL skills

Hands-on experience with cloud platforms (Azure, AWS or GCP)

Solid understanding of data modelling and modern architectures

Strong knowledge of data governance, security and compliance

Experience with orchestration tools such as Airflow

Familiarity with CI/CD pipelines, Git workflows and automated testing

Exposure to real-time, IoT or environmental telemetry data

We believe in rewarding our people properly:

Up to 10.5% annual bonus

Outstanding pension – up to 21% combined contribution

Private healthcare

Share buy scheme

Wellbeing benefits and support programmes

You’ll be working on data that truly matters — protecting rivers, supporting environmental compliance, and helping shape the future of sustainable water management across the North West.

If you’re passionate about modern data engineering, enjoy technical leadership, and want your work to have real-world environmental impact, we’d love to hear from you .