Lead Databricks Engineer (Senior Software Engineer II)
Role Overview
Technical owner and steward of Nielsen's Databricks Lakehouse for Corporate functions.
Leads platform strategy, architecture, governance, reliability, and cost maturity.
Builds AI-native platform capabilities and uses AI-assisted engineering practices to scale platform delivery.
Key Skills for This Role
Full Job Posting
Role overview
Technical owner and steward of Nielsen's Databricks Lakehouse for Corporate functions.
Leads platform strategy, architecture, governance, reliability, and cost maturity.
Builds AI-native platform capabilities and uses AI-assisted engineering practices to scale platform delivery.
Platform ownership and governance
- Own Lakehouse architecture, workspace topology, cluster policies, compute strategy, and multi-cloud deployment patterns across AWS, Azure, or GCP.
- Design and enforce Unity Catalog governance, including catalogs, schemas, access controls, lineage, metadata management, and audit logging.
- Define standards for Delta Lake, Delta Live Tables, medallion architecture, and performance tuning.
- Industrialize platform reliability through infrastructure as code, CI/CD, and monitoring, targeting 99.99% availability.
- Manage cost transparency and optimization through right-sizing, workload isolation, chargeback/showback, and budget alerts.
- Partner with engineering, analytics, data science, and business stakeholders on platform capabilities and self-service patterns.
- Govern Databricks AI features and use cases within access-control and governance boundaries.
- Lead a developer community of practice, publish standards, run enablement sessions, and remove delivery friction.
Experience and technical qualifications
- 8–10+ years of experience in data engineering, data platform architecture, or distributed systems.
- 3+ years architecting and operating Databricks environments at scale.
- Hands-on experience building and operating production Databricks platforms, including Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL, and cluster/compute policies.
- Experience architecting feature stores, vector search/indexes, and GenAI-ready pipelines.
- Experience with AI-assisted and agentic engineering practices.
- Deep hands-on Apache Spark expertise for batch and structured-streaming processing and performance tuning.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at The Nielsen Company
Senior Product Manager II (multi-country quantitative research)
Bengaluru, IND
•Define product requirements, work with engineering teams, and deliver solutions that meet user needs. Scope includes driving transformational growth across our Sports Research and Consulting products and solutions, as w
Sales Lead, Solutions Consulting (Pre-Sales)
New York City, USA
Serves as a senior strategic partner to prospects and clients, leading complex, high-value consultative sales engagements to close new business and expand Nielsen’s relationships with a diverse set of accounts Requires s
Bilingual Spanish Field Sales Representative
San Francisco, USA
As a Field Sales Representative, you'll be the face of Nielsen, recruiting households to participate in Nielsen ratings. Our field sales representatives spend 75% or more of their week in the field driving to and recruit
Senior Financial Planning & Analysis Partner I (MDG Finance)
Mumbai, IND
About the Role As a Senior Analyst in Master Data Governance within our FP&A organization, you will serve as the critical bridge between finance operations, commercial/supply chain teams, system architecture, and strateg
Senior Legal Counsel, Privacy and Data Protection
, GBR
This position can be done remotely within the UK. A central-London office is also available for hybrid work if preferred. You will be part of a friendly, international team. Job Summary The Senior Legal Counsel, Privacy
Senior Product Manager, Audience Measurement & Media Data
, USA
The Senior Product Manager, Audience Measurement & Media Data will lead a critical area of measurement innovation across streaming, broadcast, and cable television. This role will shape the products, platforms, and data
Senior Product Manager I
Bengaluru, IND
The Senior Product Manager I will help shape and advance reporting capabilities for Nielsen's Ads and Outcomes measurement portfolio. These products help clients understand advertising performance, audience delivery, and
Senior Legal Counsel, Privacy and Data Protection
, GBR
This position can be done remotely within the UK. A central-London office is also available for hybrid work if preferred. You will be part of a friendly, international team. Job Summary The Senior Legal Counsel, Privacy
Senior Product Manager II (multi-country quantitative research)
Bengaluru, IND
Sales Lead, Solutions Consulting (Pre-Sales)
New York City, USA
Bilingual Spanish Field Sales Representative
San Francisco, USA
Senior Financial Planning & Analysis Partner I (MDG Finance)
Mumbai, IND
Senior Legal Counsel, Privacy and Data Protection
, GBR
Senior Product Manager, Audience Measurement & Media Data
, USA
Senior Product Manager I
Bengaluru, IND
Senior Legal Counsel, Privacy and Data Protection
, GBR