Senior Data Engineer

Skills, Knowledge & Expertise

• Hands-on and knowledge of programming languages particularly Python
• Practical experience with major cloud providers including GCP, Azure, or AWS
• Knowledge of core software engineering concepts like DRY and SOLID
• Proficiency in version control systems, specifically Git or similar tools
• Demonstrated application of TDD and diverse testing methodologies, such as data quality, integration, and unit testing
• Ability to leverage CI/CD pipelines to maintain code quality and facilitate deployments
• Knowledge of Data Engineering fundamentals, including Batch/Streaming and ETL/ELT
• Familiarity with various data architectures, such as Lakes and Warehouses
• Hands-on experience with data cleansing and transformation within unified, standardised schemas
• Analytics & Platform Frameworks
• Orchestration like Airflow or AWS Glue
• Database Technologies
• Data processing libraries & Schema Management tools like Pandas and Pydantic
• Storage Formats: Familiarity with data types such as JSON and Parquet
• Experience of IaC tools, particularly Terraform
• Understanding of implementing logging, monitoring and alerting for enterprise level pipelines
• Understanding of access controls, policies and approaches
• Enthusiasm for learning and self-development and sharing that within a team and organisation
• Knowing how to gather basic requirements from clients and users on a data project and turn those into demonstrable outcomes
• Knowledge of how to document and diagram systems being built
• Great communication skills to explain and relate concepts to stakeholders of different levels
• Able to help coordinate with Product and Delivery managers
• Working at a technology consultancy
• Working with Docker and virtual environments
• Experience with specialised Data Platforms focused on specific domains like Geospatial data
• Experience in working with a team of engineers using a variety of techniques such as pair programming or mob programming.
• Working with data scientists to productionise advanced data deliverables, such as machine learning models
• Working knowledge of statistics
• Working with multidisciplinary digital and technology teams
• Working within the public sector
• Demonstrate a commercial mindset when on projects to grow accounts organically with senior stakeholders by identifying problems and opportunities