Sr. Databricks Platform Engineer

The Role

What You'Ll Be Doing

• Unity Catalog governance — design and implement the catalog/schema structure, privilege model, data ownership, and lineage for a platform serving multiple business units and external partners
• Delta Sharing — architect the provider/recipient model for external data onboarding; manage refresh strategy, access scoping, and cross-platform sharing (Databricks-to-Databricks, Databricks-to-Snowflake)
• Identity & access integration — Microsoft Entra ID integration with Unity Catalog; service principal design; RBAC across environment tiers
• Pipeline modernisation — refactor existing Python pipelines into a production-grade Databricks medallion architecture (bronze/silver/gold)
• Platform engineering — workspace structure, environment promotion, CI/CD, cost optimisation, and capacity planning
• Architecture contribution — this is a live engagement with an existing recommendation in place; we want someone who will challenge and improve it, not just implement it
• What you should have:

Must-Have

• Hands-on Unity Catalog implementation in a production environment — not workshop-level exposure. You should be able to talk through catalog structure, privilege model, and lineage from real deployments.
• Real Delta Sharing experience — provider/recipient architecture, external data onboarding, operational challenges you've actually hit and solved
• Strong Python and PySpark, with production pipeline migration experience (not just greenfield builds)
• Microsoft Entra ID integration experience — service principals, RBAC, scoped access
• Track record of making platform architecture decisions yourself, not just executing someone else's design
• 7+ years data platform engineering, 3+ years hands-on Azure Databricks
• Nice to Have
• Databricks Genie evaluation/implementation experience

Azure Data Factory Orchestration