Software Engineer Consultant I

Role overview

Responsibilities

• Build Spark batch and streaming pipelines and optimize ETL/ELT ingestion, transformation, validation, and curation.
• Develop solutions using Microsoft Fabric, OneLake, lakehouse, data warehouse, and semantic model capabilities; design data models and processing architectures.
• Deliver reusable analytics-ready datasets with partner teams and stakeholders.
• Apply code quality, peer review, automated testing, CI/CD, documentation, governance, lineage, metadata, security, privacy, and compliance practices.
• Optimize distributed workload performance, scalability, reliability, and cost; develop reusable frameworks and engineering patterns.
• Implement deployment automation and infrastructure-as-code practices; troubleshoot integration, performance, and operational issues.
• Maintain technical documentation, architecture diagrams, runbooks, and implementation guides; participate in design sessions and architecture reviews.
• Mentor data engineers, support production operations and incident resolution, and contribute to Agile delivery, DataOps, automation, and continuous improvement.

Required skills and experience

• Strong data engineering and big data experience; deep Apache Spark expertise, including PySpark or Spark SQL.
• Experience with Microsoft Fabric, OneLake, lakehouse architecture, data warehouses, and semantic models.
• Strong SQL and data modeling skills; experience building and optimizing ETL/ELT pipelines.
• Knowledge of batch and streaming patterns, CI/CD, Git-based development, testing, and deployment automation.
• Understanding of data governance, quality, security, and privacy controls, and experience in cloud analytics and data platforms.
• Strong problem-solving, troubleshooting, and performance-tuning capabilities; ability to mentor junior and mid-level data engineers.