Senior Data Engineer

About The Role

Key Responsibilities

• Take ownership of the existing enterprise data lake platform, ensuring scalability, reliability, and performance
• Lead the design, architecture, and implementation of cloud-native data lake solutions and integrations
• Manage and optimize data ingestion pipelines on Oracle OCI, using tools such as Apache NiFi, Kafka, Batch Processing of data, Data captures, and or CSV
• Design and implement pipelines for network data ingestion and file formats (e.g., Parquet, Avro, OCR, etc.), ensuring efficient storage, processing, and retrieval
• Build, configure, and tune query engines such as Trino (Presto), Spark, and Hive for efficient analytics and reporting
• Implement and maintain metadata management, data governance, and security frameworks
• Monitor and troubleshoot system performance, ensuring SLAs are met for ingestion, processing, and query workloads
• Automate platform deployment, monitoring, and maintenance with Infrastructure-as-Code (Terraform, CloudFormation, etc.)
• Collaborate with data engineers, analysts, and business teams to understand data requirements and deliver solutions that maximize data accessibility and usability
• Keep the data platform up to date with the latest open-source and cloud-agnostic technologies, implementing upgrades and enhancements where needed

Requirements

• 5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI).
• Strong expertise in:
• Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others
• Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink
• Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3)
• Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset
• Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc. including ingestion, parsing, and analytics within a data lake
• Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas)
• Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments
• Hands-on experience with cloud security best practices, including IAM, encryption, and network security
• Strong proficiency in Python or Java for data engineering and automation tasks
• Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training

Preferred Skills

• Exposure to machine learning workflows integrated with data lakes
• Experience with real-time streaming data pipelines
• Familiarity with containerization and orchestration (Docker, Kubernetes)
• Knowledge of cost optimization strategies in cloud-based data platforms