Senior Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Manage and optimize data lake platforms, implement cloud-native solutions, and ensure data governance while utilizing tools like Apache NiFi and Spark.
Key Skills for This Role
Full Job Posting
Key Responsibilities
- Take ownership of the existing enterprise data lake platform, ensuring scalability, reliability, and performance.
- Lead the design, architecture, and implementation of cloud-native data lake solutions and integrations.
- Manage and optimize data ingestion pipelines on Oracle OCI, using tools such as Apache NiFi, Kafka, Batch Processing of data, Data captures, and or CSV.
- Design and implement pipelines for network data ingestion and file formats (e.g., Parquet, Avro, OCR, etc.), ensuring efficient storage, processing, and retrieval.
- Build, configure, and tune query engines such as Trino (Presto), Spark, and Hive for efficient analytics and reporting.
- Implement and maintain metadata management, data governance, and security frameworks.
- Monitor and troubleshoot system performance, ensuring SLAs are met for ingestion, processing, and query workloads.
- Automate platform deployment, monitoring, and maintenance with Infrastructure-as-Code (Terraform, CloudFormation, etc.).
- Collaborate with data engineers, analysts, and business teams to understand data requirements and deliver solutions that maximize data accessibility and usability.
- Keep the data platform up to date with the latest open-source and cloud-agnostic technologies, implementing upgrades and enhancements where needed.
- 5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI).
- Strong expertise in:
- Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others
- Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink.
- Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3).
- Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset.
- Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc. including ingestion, parsing, and analytics within a data lake.
- Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas).
- Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments.
- Hands-on experience with cloud security best practices, including IAM, encryption, and network security.
- Strong proficiency in Python or Java for data engineering and automation tasks.
- Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training.
Preferred Skills
- Exposure to machine learning workflows integrated with data lakes.
- Experience with real-time streaming data pipelines.
- Familiarity with containerization and orchestration (Docker, Kubernetes).
- Knowledge of cost optimization strategies in cloud-based data platforms.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Emaratech
Senior UX Designer
United Arab Emirates, UAE
We are looking for a Senior UX Designer. The ideal candidate will be responsible for creating intuitive, engaging, and accessible digital experiences while collaborating closely...
Performance Testing Engineer (DevOps)
Dubai, UAE
Integrate performance testing in CI/CD, automate scripts, analyze bottlenecks, and collaborate with teams to optimize system performance and manage test data.
Manager - Internal Audit - Noqodi
UAE, UAE
The role entails establishing and maintaining an independent Audit Function to provide assurance and consultation services. Responsibilities include planning and executing internal audits, analysing evidence, and providi
IT Customer Support Executive
Dubai, UAE
Ensure proper computer operation by resolving end user requests, performing maintenance, and documenting processes; requires IT diploma and strong problem-solving skills.
Information Security Risk & Compliance Manager
Dubai, UAE
Oversee governance, policy, and compliance for information security, manage PCI DSS and CBUAE compliance, and conduct risk assessments.
System Administrator
United Arab Emirates, UAE
Responsible for data center operations, system administration, power management, and compliance, requiring expertise in virtualization, monitoring tools, and automation.
Senior Product Owner
Dubai, UAE
Responsible for product vision execution, collaborating with technical teams, producing user stories, and leading sprint planning in an Agile environment.
Administration - Information Security (UAE National)
Dubai, UAE
Responsible for conducting vulnerability assessments, analyzing results, recommending remediation strategies, and collaborating with security teams to enhance information security.
Senior UX Designer
United Arab Emirates, UAE
Performance Testing Engineer (DevOps)
Dubai, UAE
Manager - Internal Audit - Noqodi
UAE, UAE
IT Customer Support Executive
Dubai, UAE
Information Security Risk & Compliance Manager
Dubai, UAE
System Administrator
United Arab Emirates, UAE
Senior Product Owner
Dubai, UAE
Administration - Information Security (UAE National)
Dubai, UAE