Senior Data Engineer
Key Skills for This Role
Full Job Posting
Experience and ownership
- 10+ years of hands-on data engineering experience with production pipeline ownership.
- At least one platform or system must have been owned end to end.
Transformation, modeling, and orchestration
- dbt experience includes models, tests, macros, incremental models, and project structure.
- Data modeling includes dimensional modeling, star schemas, and medallion architecture.
- Apache Airflow experience includes DAGs, dependencies, triggers, sensors, parameterized runs, failure handling, and scheduling.
- Production workflow orchestration and monitoring are part of the role.
Ingestion, storage, and databases
- Develop batch ingestion and ETL/ELT pipelines using full, incremental, upsert/merge, and CDC load patterns.
- Ingest from SFTP, REST APIs, relational databases, and files; address data quality, validation, and schema evolution.
- Use Dremio or an equivalent lakehouse query engine, and Apache Iceberg or an equivalent open table format.
- Work with S3-compatible object storage and tune partitions, table layouts, and query performance.
- Use Python, advanced SQL, and PostgreSQL and/or SQL Server.
Non-relational database experience
- Hands-on experience with at least two non-relational database families is required, not just exposure.
- Families listed include key-value/in-memory stores; document databases; wide-column stores; and search/analytics engines.
- Work includes reading from these stores, modeling their data into a lakehouse, and reasoning about consistency and performance.
Formats and distributed processing
- Apache Parquet is required; ORC, Avro, JSON, JSON Lines, CSV, and XML are also listed.
- Handle encoding, delimiters, quoting, compression codecs, nested data, and evolving schemas.
- Use Apache Spark/PySpark or an equivalent distributed engine; understand parallelism, shuffles, memory tuning, and distributed-job costs.
Scripting, security, and connectivity
- Use Bash/shell scripting and the Linux command line for script-based deployment and operations.
- Manage SFTP/SSH keys and automated transfers, including verification, archiving, and retries.
- Understand TLS certificate verification; never disable it in production.
- Troubleshoot DNS, firewalls, proxies, timeouts, and service networking in Kubernetes/OpenShift.
- Use YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests.
Responsibilities
- Own production data pipelines and at least one data platform or system end to end.
- Develop data transformations and models, including dbt models, tests, macros, and incremental models.
- Design, schedule, monitor, and troubleshoot Apache Airflow workflows and production pipeline failures.
- Build batch ingestion and ETL/ELT pipelines from SFTP, REST APIs, relational databases, and files.
- Implement full and incremental loads, upserts/merges, and change data capture patterns.
- Ingest and model data from non-relational stores into the lakehouse, accounting for their consistency and performance characteristics.
- Handle data quality, validation, schema inference, drift, and schema evolution.
- Work with object storage, lakehouse query engines, and open table formats; tune partitioning, table layouts, and query performance.
- Develop with Python and SQL and optimize queries and distributed processing jobs.
- Manage secure file transfers, certificates, configuration, and connectivity troubleshooting across deployment environments.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at DICETEK LLC
Infrastructure Engineer (OpenShift / VMware)
, UAE
Job Summary We are looking for an experienced Infrastructure Engineer with strong hands-on expertise in Red Hat OpenShift, VMware virtualization technologies, Linux and Windows Server administration, and infrastructure m
Trade-Based Money Laundering (TBML) Specialist
, UAE
Job Summary We are seeking an experienced TBML Specialist with strong expertise in identifying, investigating, and mitigating Trade-Based Money Laundering risks within trade finance transactions. The ideal candidate shou
System Analyst- Data Center
, UAE
Minimum Experience At least 6 years’ experience in operations out of which at least 2 years in the banking environment. Working knowledge and experience of 24/7/365 EOD operations, NOC and Data Center operations . Mini
Senior Manager – AI Security Product Owner
, UAE
Knowledge, Skills & Experience 12+ years of experience in Cybersecurity, AI/ML Security, Product Ownership, or related domains. Strong expertise in: AI/ML security, governance, and risk management Cloud security (AWS
AI Security Engineer
, UAE
Minimum Experience At least 4 years of experience in cybersecurity, cloud security, application security, security engineering, DevSecOps or security operations, including hands-on experience securing Generative AI, mach
Liferay Full Stack Developer
, UAE
Role Summary Design, develop, integrate, secure, deploy, and support scalable enterprise portal solutions on Liferay DXP across backend services, frontend experiences, APIs, data persistence, and platform operations. Key
Sanctions High Risk Advisory- Senior Analyst
, UAE
Experience: Ideally 5+ years in FCC/AML, with at least 3 years directly in sanctions within a bank or major financial institution. Hands-on exposure to sanctions investigations, payment screening/escalations, corresponde
Infrastructure Engineer (OpenShift / VMware)
, UAE
Trade-Based Money Laundering (TBML) Specialist
, UAE
System Analyst- Data Center
, UAE
Senior Manager – AI Security Product Owner
, UAE
AI Security Engineer
, UAE
Liferay Full Stack Developer
, UAE
Sanctions High Risk Advisory- Senior Analyst
, UAE