Senior Data Engineer

Position overview

Technology stack

• The stack includes Python, SQL, Apache Spark, PySpark, Spark SQL, Apache Airflow, Hive, Presto/Trino, Snowflake, AWS S3, EMR, and Glue.
• Other listed tools include Git, Jenkins, Power BI, Looker, and AI-assisted tools such as GitHub Copilot, Cursor, ChatGPT, and Claude or equivalent.

Responsibilities

• Use AI tools to accelerate coding, debugging, refactoring, test generation, documentation, and troubleshooting; evaluate generated work for quality and security.
• Design and optimize large-scale batch and near-real-time Spark pipelines and ETL/ELT processing across distributed platforms.
• Build Airflow workflows with DAGs, scheduling, dependencies, retries, and failure recovery; implement data quality checks, monitoring, observability, and alerting.
• Maintain reliable, reusable pipelines supporting incremental processing and backfills; troubleshoot production issues and improve performance and cost efficiency.
• Develop cloud-native AWS data infrastructure and automate provisioning, deployment, and operations with infrastructure as code and CI/CD.
• Apply testing, version control, code review, and software engineering practices; own solutions from design through deployment and production support.
• Partner with product managers, data scientists, analysts, and business stakeholders to translate business needs into scalable technical solutions.
• Design data models and curated datasets for compliance detection, product usage analytics, and business reporting; promote engineering standards and communicate technical trade-offs.

Minimum qualifications

• 4–7 years of relevant experience in data engineering, big data systems, or software engineering, building and operating production-grade data platforms.
• At least 3 years of hands-on Apache Spark experience, including PySpark, Spark SQL, and DataFrame APIs.
• At least 3 years of strong Python and SQL programming experience, with ability to write clean, modular, maintainable, and testable code.
• Strong understanding of distributed computing fundamentals, including parallel processing, partitioning, shuffling, data skew, fault tolerance, and Spark performance tuning.
• Hands-on experience with Apache Airflow for building, scheduling, and orchestrating complex data pipelines.
• Strong knowledge of ETL/ELT design patterns, data modeling, SQL optimization, and large-scale data processing.