Copy of Lead Databricks Engineer (Senior Software Engineer II)
Role Overview
The Data Platform Lead is technical owner and steward of Nielsen’s corporate Databricks Lakehouse.
The role defines platform strategy, architecture, governance, and operational maturity, including AI-native capabilities and AI-assisted engineering practices.
Key Skills for This Role
Full Job Posting
Role mandate
The Data Platform Lead is technical owner and steward of Nielsen’s corporate Databricks Lakehouse.
The role defines platform strategy, architecture, governance, and operational maturity, including AI-native capabilities and AI-assisted engineering practices.
Core goals and responsibilities
- Own Lakehouse architecture, workspace topology, cluster policies, compute strategy, and multi-cloud deployment patterns across AWS, Azure, or GCP.
- Design and enforce Unity Catalog governance, including catalogs, schemas, access controls, lineage, metadata, and audit logging.
- Set standards for Delta Lake, Delta Live Tables pipelines, medallion architecture, and performance optimization.
- Work with CloudOps and DevOps on infrastructure as code, CI/CD, monitoring, and platform reliability, with a stated 99.99% availability target.
- Manage platform cost transparency and optimization, including right-sizing, workload isolation, chargeback or showback, and budget alerts.
- Enable engineering, analytics, data science, and business teams through platform capabilities and self-service patterns.
- Govern deployment of AI use cases and native AI features within access-control boundaries.
- Lead a Databricks community of practice, publish standards, run enablement sessions, and remove delivery obstacles.
Experience and expertise
- 8–10+ years of experience in data engineering, data platform architecture, or distributed systems, including 3+ years architecting and operating Databricks environments at scale.
- Hands-on experience building, managing, running, and scaling a production-grade Databricks platform end to end.
- Experience with Unity Catalog, Delta Lake, Delta Live Tables, Databricks Workflows, Databricks SQL, and cluster or compute policy design.
- Experience architecting feature stores, vector search or indexes, and GenAI-ready pipelines into a platform, and using AI-assisted or agentic engineering practices.
- Deep hands-on expertise with Apache Spark for large-scale batch and structured-streaming processing and performance tuning.
Platform technologies and practices
- Databricks compute includes job clusters, SQL warehouses, serverless, and multi-cloud deployment patterns.
- Platform standards include Z-ordering, liquid clustering, partitioning, and file compaction.
- Automation references Terraform, Databricks Asset Bundles, CI/CD pipelines, and monitoring.
- Governed AI examples include Databricks Genie, AI/BI Dashboards, and Mosaic AI.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at The Nielsen Company
Senior Product Manager II (multi-country quantitative research)
Bengaluru, IND
•Define product requirements, work with engineering teams, and deliver solutions that meet user needs. Scope includes driving transformational growth across our Sports Research and Consulting products and solutions, as w
Sales Lead, Solutions Consulting (Pre-Sales)
New York City, USA
Serves as a senior strategic partner to prospects and clients, leading complex, high-value consultative sales engagements to close new business and expand Nielsen’s relationships with a diverse set of accounts Requires s
Bilingual Spanish Field Sales Representative
San Francisco, USA
As a Field Sales Representative, you'll be the face of Nielsen, recruiting households to participate in Nielsen ratings. Our field sales representatives spend 75% or more of their week in the field driving to and recruit
Senior Financial Planning & Analysis Partner I (MDG Finance)
Mumbai, IND
About the Role As a Senior Analyst in Master Data Governance within our FP&A organization, you will serve as the critical bridge between finance operations, commercial/supply chain teams, system architecture, and strateg
Senior Legal Counsel, Privacy and Data Protection
, GBR
This position can be done remotely within the UK. A central-London office is also available for hybrid work if preferred. You will be part of a friendly, international team. Job Summary The Senior Legal Counsel, Privacy
Senior Product Manager, Audience Measurement & Media Data
, USA
The Senior Product Manager, Audience Measurement & Media Data will lead a critical area of measurement innovation across streaming, broadcast, and cable television. This role will shape the products, platforms, and data
Senior Product Manager I
Bengaluru, IND
The Senior Product Manager I will help shape and advance reporting capabilities for Nielsen's Ads and Outcomes measurement portfolio. These products help clients understand advertising performance, audience delivery, and
Senior Legal Counsel, Privacy and Data Protection
, GBR
This position can be done remotely within the UK. A central-London office is also available for hybrid work if preferred. You will be part of a friendly, international team. Job Summary The Senior Legal Counsel, Privacy
Senior Product Manager II (multi-country quantitative research)
Bengaluru, IND
Sales Lead, Solutions Consulting (Pre-Sales)
New York City, USA
Bilingual Spanish Field Sales Representative
San Francisco, USA
Senior Financial Planning & Analysis Partner I (MDG Finance)
Mumbai, IND
Senior Legal Counsel, Privacy and Data Protection
, GBR
Senior Product Manager, Audience Measurement & Media Data
, USA
Senior Product Manager I
Bengaluru, IND
Senior Legal Counsel, Privacy and Data Protection
, GBR