Senior Data Engineer

About the Role

Key Responsibilities

• Take ownership of the existing enterprise data lake platform, ensuring scalability, reliability, and performance.
• Lead the design, architecture, and implementation of cloud-native data lake solutions and integrations.
• Manage and optimize data ingestion pipelines on Oracle OCI, using tools such as Apache NiFi, Kafka, Batch Processing of data, Data captures, and or CSV.
• Design and implement pipelines for network data ingestion and file formats (e.g., Parquet, Avro, OCR, etc.), ensuring efficient storage, processing, and retrieval.
• Build, configure, and tune query engines such as Trino (Presto), Spark, and Hive for efficient analytics and reporting.
• Implement and maintain metadata management, data governance, and security frameworks.
• Monitor and troubleshoot system performance, ensuring SLAs are met for ingestion, processing, and query workloads.
• Automate platform deployment, monitoring, and maintenance with Infrastructure-as-Code (Terraform, CloudFormation, etc.).
• Collaborate with data engineers, analysts, and business teams to understand data requirements and deliver solutions that maximize data accessibility and usability.
• Keep the data platform up to date with the latest open-source and cloud-agnostic technologies, implementing upgrades and enhancements where needed.

Requirements

• 5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI).
• Strong expertise in:
• Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others
• Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink.
• Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3).
• Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset.
• Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc. including ingestion, parsing, and analytics within a data lake.
• Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas).
• Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments.
• Hands-on experience with cloud security best practices, including IAM, encryption, and network security.
• Strong proficiency in Python or Java for data engineering and automation tasks.
• Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training.

Preferred Skills

• Exposure to machine learning workflows integrated with data lakes.
• Experience with real-time streaming data pipelines.
• Familiarity with containerization and orchestration (Docker, Kubernetes).
• Knowledge of cost optimization strategies in cloud-based data platforms.