Senior Data Engineer - Microsoft Data & AI

End-to-End Platform Architecture

Lead the design and build of end-to-end data platforms spanning ingestion, cleaning, preparation, transformation, analytics, and activation, using Azure Data Factory, Microsoft Fabric (Lakehouse, Warehouse, OneLake), and/or Azure Databricks with PySpark, Python, and SQL.

Architect Medallion (Bronze, Silver, Gold) lakehouse patterns using OneLake/Delta Lake, balancing performance, reliability, and cost.

Own data modeling for client engagements - dimensional, normalized, and semantic layers - so that data is trustworthy, reusable, and analytics-ready across domains.

Design batch and streaming ingestion patterns, including Fabric Data Factory/pipelines, Azure Databricks Structured Streaming, and event-driven architectures (e.g., Event Hubs).

Identify and resolve architectural risk and technical debt, and optimize pipelines for performance and cost.

APIs, Data Agents & Activation

Design and build APIs and data services that activate governed data for client analytics, reporting, and downstream applications.

Design MCP (Model Context Protocol) and API-based integrations that let front-line business systems, Fabric/Copilot Data Agents, and other agentic AI interfaces securely query and exchange data with the platform.

Partner with AI/Architecture teams to expose enterprise data to Data Agents and Generative AI applications (e.g., Fabric Data Agents, Azure AI Foundry) in a governed, reusable way.

Implement data governance patterns using Microsoft Purview and/or Unity Catalog, including access control, metadata, and lineage.

Support production operations and incident response for critical pipelines and activation services on client engagements.

Technical Leadership & Client Engagement

Mentor and provide technical direction to Data Engineers, reviewing code, data models, and designs to raise delivery quality.

Act as a trusted technical resource to client stakeholders, explaining architecture trade-offs and design decisions in clear terms.

Contribute to reusable frameworks, accelerators, and engineering standards used across engagements.

Support presales activities where needed, including technical input on estimates, proposals, and solution design.

Evaluate new capabilities across Fabric and Azure Databricks and recommend where they add value for client platforms.

Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, Statistics, or a related technical discipline.

5-8 years of experience in data engineering, with strong ownership of production data pipelines on Azure.

Expert-level proficiency in Python, SQL, and PySpark for building distributed data pipelines at scale.

Deep hands-on experience with the Microsoft Azure Data & AI stack (Azure Data Factory, Microsoft Fabric, Synapse, ADLS Gen2) and/or Azure Databricks, including Delta Lake/OneLake and Medallion architecture patterns.

Strong background in data modeling, schema design, and enterprise data architecture across the full lifecycle from ingestion to activation.

Experience designing APIs and MCP-based integrations that exchange data with front-line systems, Data Agents, and other agentic AI interfaces.

Experience with orchestration and CI/CD (Fabric pipelines, Azure Data Factory, Databricks Workflows, Azure DevOps).

Strong knowledge of data governance and security tooling, including Microsoft Purview and/or Unity Catalog.

Demonstrated ability to mentor engineers and lead technical design discussions on client-facing engagements.

Strong problem-solving skills and ability to troubleshoot complex, large-scale data issues independently.

Excellent communication skills, with the ability to explain technical trade-offs to both technical and client audiences.

Relevant Microsoft certifications (e.g., Azure Data Engineer Associate, Fabric Analytics Engineer Associate) and/or Databricks certifications preferred.