Senior Big Data Engineer
Skills
About This Role
Responsibilities
- Design, implement, and optimize data pipelines for batch and real-time data processing using Cloudera (Hadoop, Hive, Spark, Impala) and Informatica (PowerCenter, Cloud Data Integration)
- Build data extraction, transformation, and loading (ETL) workflows using Informatica PowerCenter for large-scale data integration from source systems (e.g., relational databases, flat files, APIs) into Cloudera Data Lake or data warehouse environments.
- Implement Spark jobs on Cloudera for distributed data processing and optimization of data workflows.
- Leverage Informatica for orchestrating ETL workflows, including data extraction, cleansing, transformation, and loading into data repositories (HDFS, Hive, SQL databases, etc.).
- Optimize the Informatica workflows to minimize runtime, ensure smooth data integration, and maintain high data quality.
- Utilize Hadoop and Spark on Cloudera to process large datasets and implement data transformations using MapReduce, Spark SQL, and PySpark.
- Leverage Impala for low-latency SQL queries on Hadoop, ensuring real-time access to processed data.
- Implement partitioning, bucketing, and indexing strategies in Hive and HBase to improve query performance on large datasets.
- Implement and enforce data quality rules within Informatica workflows, ensuring that all transformations meet the required standards for completeness, consistency, and accuracy.
- Ensure compliance with data governance and security protocols (e.g., encryption, masking, access control) in accordance with industry best practices.
- Automation and Scheduling: Automate ETL workflows using Informatica Server, integrating with Airflow, Nifi or other workflow orchestration tools for scheduling and monitoring jobs.
- Utilize Cloudera Navigator for monitoring and auditing data processes within the Hadoop ecosystem.
- Perform regular tuning of the ETL pipelines, data flows, and SQL queries to ensure optimal performance.
Qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field.
- 6+ years of experience in the same field.
- Proven experience with the Cloudera Distribution of Hadoop (CDH), including expertise in HDFS, Hive, Impala, Spark, and HBase.
- Strong hands-on experience with Informatica PowerCenter (ETL), EDC, IDQ, B2B, and Axon.
- Deep understanding of ETL best practices, data pipelines, and distributed computing technologies such as Spark, MapReduce, PySpark, and Hadoop ecosystem components.
- Advanced SQL skills for data manipulation, aggregation, optimization, and reporting across relational and non-relational data stores (e.g., SQL Server, MySQL, PostgreSQL, Hive, Impala).
- Experience in Python and SQL.
- Strong background in data warehousing principles and data modeling, including dimensional modeling (star schema, snowflake schema) and OLAP/OLTP considerations.
Your resume, rewritten
for this exact role.
Sign up free — Base Career tailors your CV to this job description in 60 seconds.
01 / 05
Resume Tailored to This Job
Your keywords, structure, and story — rewritten to match this exact role and pass ATS filters.
Free · No card · 60 seconds
02 / 05
Cover Letter for This Role, Done
Job-specific cover letters written in Gulf professional tone — ready in seconds, not hours.
Free · No card · 60 seconds
03 / 05
See How Well You Fit This Role
AI match score with clear reasons — know your fit before investing time in the application.
Free · No card · 60 seconds
04 / 05
Use Autofill When You Apply
Autofill any application form on Workday, LinkedIn, Bayt, Greenhouse — with your tailored content.
Free · No card · 60 seconds
05 / 05
Track It. Follow Up at the Right Time.
Visual pipeline for every application with AI-timed follow-up reminders so nothing slips.
Free · No card · 60 seconds
2.2K+
Cover Letters & Follow-ups
1.8K+
Resumes Tailored
190.5K+
Jobs Tracked
Trusted by professionals at
Stop applying blindly.
Start getting hired.
Base Career automates the hardest parts of job searching — apply smarter, not harder.
AI Resume in 60s
Your resume rewritten for this exact role using the job description as the brief.
ATS-Optimized
Get past automated screening filters with the right keywords matched to each job.
Application Tracker
Track every job, follow-up, and interview in one visual kanban board.
Free plan · No credit card required