Senior Data Engineer – Azure Databricks | AI & Agentic Lakehouse

Opportunity and work arrangement

Lakehouse engineering and integration

• Lead design and implementation of an enterprise Azure Databricks lakehouse and establish Bronze, Silver, and Gold Medallion layers using Delta Lake.
• Define data-modeling standards, data contracts, reusable frameworks, and scalable ingestion and transformation pipelines.
• Develop pipelines using Azure Data Factory, Databricks Workflows, PySpark, and SQL; integrate ERP, legacy systems, SharePoint, business applications, and REST APIs.
• Automate data cleansing, validation, transformation, and migration; optimize performance, scalability, reliability, and cost.
• Implement Unity Catalog governance, access controls, lineage, classification, data quality, exception handling, quarantine, and auditability.
• Design AI agents for discovery, profiling, onboarding, schema inference, quality-rule generation, anomaly detection, entity resolution, metadata enrichment, and intelligent cleansing.
• Enable governed agent access through SQL endpoints, vector indexes, and MCP-style services; integrate Azure OpenAI and related Azure AI services.
• Make technical decisions, set engineering practices, review code, guide engineers and delivery partners, and implement Git-based CI/CD.
• Optimize Databricks costs through cluster policies and compute selection; collaborate with stakeholders to deliver milestones.

Initial delivery expectations

• Assess existing data sources, pipelines, and data-quality challenges; define the target lakehouse architecture and implementation blueprint.
• Establish Databricks Medallion layers, Unity Catalog, and ingestion foundations; build pipelines for priority business domains.
• Implement data-quality and governance controls; prototype AI-assisted profiling, quality-rule generation, and onboarding workflows.
• Demonstrate working deliverables and document the technical implementation.

Required qualifications

• At least 7 years of professional data engineering experience, including senior-level technical ownership.
• Proven experience designing and implementing enterprise lakehouses or modern data platforms.
• Hands-on expertise with Azure Databricks, Delta Lake, Unity Catalog, and Medallion Architecture; advanced Python, PySpark, and SQL proficiency.
• Experience with Databricks Workflows, Delta Live Tables/Lakeflow, production pipelines, Azure Data Factory, ADLS Gen2, Azure Key Vault, and Azure networking.
• Experience with governance, access controls, lineage, data-quality frameworks, ERP or legacy data integration, and cleansing.
• Practical exposure to LLM-powered workflows, Azure OpenAI, or AI-driven automation.
• Experience implementing CI/CD with Azure DevOps or Git-based pipelines.
• Strong problem-solving, architecture design, communication, and technical leadership skills.

Preferred qualifications

• Databricks Certified Data Engineer Professional certification is preferred.
• Knowledge graphs, semantic layers, enterprise metadata management, Azure AI Search, vector databases, and RAG are preferred areas of experience.
• Experience developing AI agents, MCP integrations, or intelligent data services is preferred.
• Enterprise digital transformation or complex data migration project experience is preferred.

About Efforts Solutions IT