Senior Data Engineer

Role overview

Transformation, modeling, and orchestration

• dbt experience across models, tests, macros, incremental models, and project structure.
• Data modeling experience, including dimensional modeling, star schemas, and medallion architecture.
• Apache Airflow DAG design, scheduling, dependencies, triggers, sensors, failure handling, and production monitoring.

Ingestion and pipeline operations

• Develop batch ingestion and ETL/ELT pipelines, including full and incremental loads, upserts/merges, and CDC.
• Ingest data from SFTP, REST APIs, relational databases, and files; implement data validation and quality checks.
• Handle schema inference, drift, and schema evolution, including nested and semi-structured data.

Lakehouse, storage, and distributed processing

• Experience with Dremio or an equivalent lakehouse query engine, such as Trino, Presto, Starburst, or Athena.
• Experience with Apache Iceberg or an equivalent open table format, such as Delta Lake or Apache Hudi.
• Work with S3-compatible object storage, partitioning, table layout, and query performance tuning.
• Experience with Apache Spark/PySpark or an equivalent distributed engine, including joins, skew, parallelism, shuffles, and memory tuning.

Non-relational databases

• Hands-on experience with at least two non-relational database families is required, beyond exposure alone.
• Families include key-value/in-memory stores, document databases, wide-column stores, and search/analytics engines.
• Use these stores as a data engineer: read from them, model their data into the lakehouse, and reason about consistency and performance.

Programming and data formats

• Python for pipeline development, automation, and scripting; advanced SQL for query optimization and performance tuning.
• PostgreSQL and/or SQL Server experience with relational databases.
• Apache Parquet is required; other listed formats include ORC, Avro, JSON, JSON Lines, CSV, and XML.
• Handle compression codecs such as Snappy, gzip, and zstd, and nested or evolving schemas.

Scripting, security, and connectivity

• Use Bash/shell scripting and Linux command-line tools for script-based deployment and operations.
• Manage SFTP/SSH keys and automated file handoffs, including verification, result files, archiving, and retries.
• Understand TLS certificates and secure connections; certificate verification must not be disabled in production.
• Troubleshoot DNS, firewalls, proxies, timeouts, and service networking in Kubernetes/OpenShift; configure pipelines with YAML.