Data Engineer

Job Purpose

Key Responsibilities

• Define, design, and develop data ingestion frameworks for large-scale data ingestion, storage, and management across structured, semi-structured, unstructured and streaming data sources.
• Design, develop, and maintain scalable batch and streaming data pipelines.
• Design and implement workflows and pipelines using tools such as Apache NiFi, Kafka, Spark, Flink, and Airflow.
• Design and implement data models supporting analytical and operational workloads.
• Collaborate with internal product teams and third-party service providers on system integrations and platform enhancements.
• Participate in sprint planning, development, and delivery activities.
• Deploy and support data workloads in cloud and containerized environments.
• Monitor, troubleshoot, and optimize data platform performance and reliability.
• Ensure solutions are scalable, observable, secure, and operationally manageable.

Essential Skills & Experience

• Strong experience designing, building, and operating data pipelines in production environments.
• Experience working with modern Data Lake or Lakehouse architectures, including Delta Lake, Iceberg, or equivalent technologies.
• Solid understanding of distributed data architectures, data modelling, OLAP design, and change data capture (CDC), including tools such as Debezium.
• Experience with large-scale storage and messaging platforms, including MinIO/S3 and Kafka.
• Experience integrating with enterprise systems through REST APIs, messaging platforms, and event-driven architectures.
• Proficient in batch and real-time data processing, with familiarity in data formats such as Avro, Parquet, and ORC.
• Strong scripting or programming skills.
• Experience delivering enterprise-scale analytics and reporting platforms, strong understanding of infrastructure and container-based deployments, and the ability to advise on emerging data technologies and best practices.
• Familiarity with monitoring and observability tools such as Grafana, Prometheus, ELK/OpenSearch, Dynatrace, or equivalent.
• Experience working in Agile software development environments.
• Experience with Azure Databricks, Azure Data Lake Storage, Azure Event Hubs, and Azure Synapse Analytics.
• Understanding of data governance, data lineage, access controls, and data security best practices.

Education And Qualifications

• Bachelor's Degree in Computer Science, Software Engineering, Information Technology, Data Engineering, or a related discipline.
• Relevant professional certifications are advantageous.
• 5–8 years of professional experience as a Data Engineer, Big Data Engineer, or Data Platform Engineer