Senior Databricks Specialist / Data Architect

Role overview

Responsibilities

• Design and implement enterprise-scale Azure Databricks solutions and maintain platform architecture and standards.
• Design multi-source ingestion and integration pipelines; develop scalable ETL/ELT with Databricks and Spark.
• Implement Medallion Architecture (Bronze, Silver, Gold) where appropriate; optimize Delta Lake tables and pipelines.
• Build data models and transformations for analytical and operational reporting; integrate Dynamics 365, Dataverse, and other enterprise applications.
• Develop and optimize batch and near-real-time data processing.
• Implement data quality, validation, monitoring, lineage, governance, security, and access controls, including use of Unity Catalog.
• Optimize Spark jobs, clusters, SQL queries, Delta tables, and pipelines for performance and cost.
• Support Power BI and other reporting platforms with optimized data models and services.
• Troubleshoot complex engineering and performance issues; establish development, testing, and production standards.
• Support CI/CD and DevOps; conduct technical reviews, mentor data engineers, and address security, availability, scalability, and disaster recovery requirements.

Qualifications

• At least eight years of data engineering or data architecture experience, including at least four years of hands-on Azure Databricks experience.
• Strong hands-on Apache Spark, PySpark, and Delta Lake experience; experience designing enterprise platforms and pipelines.
• Knowledge of Medallion Architecture; experience with Unity Catalog, data governance, access control, and lineage.
• Strong SQL and large-scale data processing experience; experience with Azure data services, Azure Data Factory, and/or Azure Data Lake Storage Gen2.
• Experience integrating Databricks with APIs, enterprise applications, CRM systems, and other sources.
• Experience with data and dimensional modeling, data quality, governance, performance optimization, and Databricks cost optimization.
• Experience with CI/CD, Git, Azure DevOps, deployment automation, and large-volume enterprise datasets; ability to troubleshoot complex production issues.

Preferred certifications

• Preferred certifications include Databricks Certified Data Engineer Professional or Associate, Microsoft Certified: Azure Data Engineer Associate, and other relevant Databricks or Azure certifications.