Lead Data Engineer
Role Overview
Serve as senior technical authority for the data platform, setting architectural direction and engineering standards.
Lead hands-on migration from SQL Server to an AWS lakehouse and design real-time streaming architecture supporting automation, BI and analytics.
Key Skills for This Role
Full Job Posting
About Betfred
Betfred's Technology department develops its proprietary platform and products for the betting and gaming industry.
Role overview
Serve as senior technical authority for the data platform, setting architectural direction and engineering standards.
Lead hands-on migration from SQL Server to an AWS lakehouse and design real-time streaming architecture supporting automation, BI and analytics.
Responsibilities
- Own architecture and standards for batch and streaming pipelines using AWS Glue, Lambda, S3 and Redshift.
- Lead SQL Server-to-AWS data warehouse migration, including target modelling, cutover, reconciliation and risk management.
- Design streaming architecture with Amazon MSK or Apache Kafka and PySpark Structured Streaming; evaluate Flink where suitable.
- Define data quality, validation and observability frameworks and enforce them through design and code review.
- Govern lakehouse and warehouse design for consistency, scalability and cost efficiency.
- Own Terraform infrastructure-as-code and CI/CD standards.
- Monitor and optimise AWS spend and manage performance-cost trade-offs.
- Lead root-cause analysis and permanent fixes for complex data issues and performance bottlenecks.
- Support machine learning and advanced analytics deployments with data science and engineering colleagues.
- Mentor engineers, contribute to hiring assessment and represent data engineering in cross-functional planning.
- Embed security and privacy in customer data modelling, access and retention.
Essential experience and skills
- Deep hands-on AWS data-service expertise across Glue, Lambda, S3, Redshift and EMR.
- Advanced Python, SQL and PySpark skills; strong data modelling and data warehousing knowledge.
- Production streaming architecture experience with Apache Kafka or Amazon MSK and Spark Structured Streaming.
- Open table format experience such as Apache Iceberg or Delta Lake, including partitioning and schema evolution.
- Pipeline orchestration and infrastructure-as-code experience with Terraform.
- Technical leadership on a significant data platform or migration programme.
- Experience mentoring engineers, diagnosing complex issues and communicating across technical and non-technical audiences.
- Data security and privacy best practices and Agile development experience.
Desirable experience
- Apache Flink for stateful stream processing is desirable.
- On-premises data migration to cloud and regulated-industry experience are desirable or advantageous.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career