Senior Data Engineer

Experience and ownership

Transformation and modeling

• dbt models, tests, macros, incremental models, and project structure.
• Dimensional modeling, star schema, medallion architecture, and data warehouse or lakehouse concepts.
• Apache Airflow DAG design, dependencies, triggers, sensors, parameterized runs, failure handling, and scheduling.

Ingestion and production pipelines

• Build batch ingestion and ETL/ELT pipelines; handle full and incremental loads, upserts/merges, and CDC.
• Ingest from SFTP, REST APIs, relational databases, and files; monitor production workflows.
• Support data quality, validation, and schema evolution.

Lakehouse, storage, and databases

• Dremio or an equivalent lakehouse query engine such as Trino, Presto, Starburst, or Athena.
• Apache Iceberg or an equivalent open table format such as Delta Lake or Apache Hudi.
• S3-compatible object storage such as AWS S3, MinIO, or Dell ECS; partitioning, table layout, and query tuning.
• PostgreSQL and/or SQL Server; advanced SQL, query optimization, and performance tuning.

Non-relational database experience

• Hands-on experience with at least two database families is required, beyond exposure alone.
• Families listed are key-value/in-memory stores; document databases; wide-column stores; and search/analytics engines.
• The engineer must read from these stores, model data into the lakehouse, and reason about consistency and performance.

Formats and distributed processing

• Apache Parquet is required; other formats listed include ORC, Avro, JSON, JSON Lines, CSV, and XML.
• Understand encoding, delimiters, quoting, compression codecs, nested schemas, and schema changes.
• Apache Spark/PySpark or an equivalent distributed engine; understand parallelism, shuffles, memory tuning, and distributed job costs.

Scripting and connectivity

• Python for pipeline development, automation, and scripting.
• Bash/shell scripting and Linux command line for script-based deployment and operations.
• SFTP/SSH key management, automated transfers, file handoffs, verification, archiving, and retries.
• TLS certificate verification, connectivity troubleshooting, and DNS, firewall, proxy, timeout, and service networking issues in Kubernetes/OpenShift.
• YAML and configuration for pipelines, dbt, Airflow, and Kubernetes manifests.