Senior Data Engineer

Role and experience

Transformation, orchestration, and ingestion

• dbt experience includes models, tests, macros, incremental models, and project structure.
• Data modeling includes dimensional modeling, star schema, and medallion architecture; data warehousing and lakehouse concepts are relevant.
• Apache Airflow experience includes DAGs, dependencies, triggers, sensors, parameterized runs, failure handling, and scheduling.
• Work includes batch ingestion and ETL/ELT, full and incremental loads, upsert/merge, CDC, and ingestion from SFTP, APIs, databases, and files.
• Data quality, validation, and schema evolution handling are included.

Lakehouse, databases, and distributed processing

• Dremio or an equivalent lakehouse query engine; Apache Iceberg or an equivalent open table format; S3-compatible object storage.
• Partitioning, table layout, and query performance tuning; Python, advanced SQL, and PostgreSQL and/or SQL Server.
• Hands-on work with at least two non-relational database families: key-value/in-memory, document, wide-column, or search/analytics stores.
• Apache Parquet is required; experience may also include ORC, Avro, JSON, JSON Lines, CSV, XML, and compression codecs.
• Handle nested and evolving schemas, schema inference, drift detection, and column additions, deletions, or type changes.
• Apache Spark/PySpark or an equivalent distributed engine; understand parallelism, shuffles, memory tuning, and distributed job cost.

Scripting and connectivity

• Use Bash/shell and Linux command-line tools for script-based deployment and operations.
• Work with SFTP/SSH key management, automated transfers, file handoffs, verification, archiving, and retries.
• Understand TLS certificates and secure connections; certificate verification must not be disabled in production.
• Troubleshoot DNS, firewalls, proxies, timeouts, and service-to-service networking in Kubernetes/OpenShift.
• Use YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests.

Responsibilities

• Own production data engineering platforms or systems end to end.
• Develop and operate batch ingestion and ETL/ELT pipelines from SFTP, REST APIs, relational databases, and files.
• Build data transformations and models, including dbt models, tests, macros, and incremental models.
• Design and monitor orchestration workflows, including Airflow DAGs, dependencies, scheduling, and failure handling.
• Load, model, and extract data from non-relational stores into a lakehouse, accounting for consistency and performance.
• Manage lakehouse query, storage, and table formats, including partitioning, table layout, and query performance.
• Validate data and handle nested or semi-structured formats, schema inference, drift, and schema evolution.
• Develop pipeline automation and scripts using Python, shell, Linux, YAML, and related deployment tooling.
• Operate distributed processing jobs and tune parallelism, joins, shuffles, memory use, and cost.
• Maintain secure automated file transfers and troubleshoot service connectivity and secure connections.

Requirements

• 10+ years of hands-on data engineering experience, including production pipeline ownership.
• At least one platform or system owned end to end.
• Hands-on experience with at least two non-relational database families, including reading from them and modeling their data into a lakehouse.
• Experience with batch ingestion, ETL/ELT, multiple load patterns, and multi-source data ingestion.
• Experience handling data quality, validation, evolving schemas, and semi-structured data.
• Experience with data warehousing/lakehouse concepts, partitioning, table layout, and query performance tuning.
• Experience with distributed processing concepts, including parallelism, shuffles, memory tuning, and job cost.
• Ability to troubleshoot connectivity involving DNS, firewalls, proxies, timeouts, and service networking in Kubernetes/OpenShift.
• Experience with secure SFTP/SSH transfers, key management, TLS certificate verification, and pipeline configuration.