{bc}
Apply
workday

Senior Data Engineer

Illumina
Bengaluru, IND
Full-time
Senior · 3–5 years experience
Onsite
Discovered 3 weeks ago
awsdatabricksgithub-actionspythonsqlterraform

Full Job Posting

Key Responsibilities

  • Platform ownership
  • Own workspace- and account-level architecture: Unity Catalog, compute policies, Lakeflow Jobs and shared compute infrastructure, identity and service principals, secrets management, audit logging, system tables, storage integration, platform upgrades and environment promotion across development, test and production.
  • Design, build and review infrastructure-as-code, CI/CD and observability capabilities for the platform, establishing reusable standards and automation for the wider engineering team.
  • Own platform reliability: availability, capacity, incident management, root-cause analysis, and DR/BCP; define and meet platform SLOs for shared platform services.
  • Run Databricks FinOps: consumption monitoring, cost optimisation, budget and anomaly management, and chargeback/showback with transparent unit costs for consuming teams.
  • Implement platform security and governance controls: identity federation/SCIM, service principals, network controls, secrets, RBAC/ABAC, governed tags, row/column security and masking, PII handling, lineage and audit.
  • Enable and operate the platform capabilities required for AI/ML workloads — MLflow, Model Serving, vector search and GenAI infrastructure patterns — partnering with analytics and AI teams on workload architecture, governance and cost controls.
  • Build and maintain reference implementations and paved-road patterns (PySpark, SQL, Delta Lake, Auto Loader) that show engineering teams how to use the platform; enable consuming teams rather than owning delivery of their data products.
  • Treat the platform as an internal product: understand consuming-team needs, define paved-road capabilities and self-service patterns, track adoption and reliability, and continuously reduce developer friction.
  • Own the platform roadmap: upgrades, new capabilities, deprecations.
  • Engineering standards
  • Contribute to the team engineering handbook and architecture decisions; lead code reviews on platform and foundation work.
  • Take part in architecture and engineering forums, communicating trade-offs clearly to peers and global stakeholders.
  • Maintain reusable patterns and automation used across the team.
  • Establish AI-native engineering practice for the team: how AI assistants are used for infrastructure code, test generation, runbooks, code review and troubleshooting, and where human review is mandatory.
  • Mentoring and collaboration
  • Mentor engineers through onboarding, pairing and code review.
  • Provide secondary coverage for an adjacent platform area as part of the team's cross-skilling model.

Required Qualifications

  • 3–5 years of professional data platform or data engineering experience, including at least 2 years administering and operating Databricks in production as an enterprise platform (account and workspace administration, Unity Catalog, compute policies, environment promotion), not only developing pipelines on it.
  • Demonstrated ability to design, build and review infrastructure-as-code (Terraform or equivalent) and CI/CD for data platforms (GitHub Actions, AWS CodePipeline or similar).
  • Strong Python and PySpark; advanced SQL.
  • Experience running platform operations: monitoring and alerting, incident management, runbooks, capacity and performance management.
  • Experience with Databricks cost management and FinOps practices.
  • Platform security and compliance experience: identity, service principals, secrets, network controls, audit logging, RBAC and PII handling, with an understanding of controls such as SOX.
  • Solid understanding of distributed systems and system design for large-scale data processing.
  • Strong AWS fundamentals: S3, IAM, VPC and networking, KMS, and how Databricks integrates with them.
  • Demonstrated effective use of AI-assisted engineering tools, or a strong ability and willingness to adopt them, across coding, IaC, testing, documentation and troubleshooting, with sound judgement around verification, security and human review. Specific tools are not a screening criterion.
  • Strong written and verbal communication skills; able to work effectively with global stakeholders and lead technical discussion.
  • Bachelor's degree in Computer Science, Information Systems, Engineering or a related field, or equivalent demonstrable experience.
  • Databricks Certified Data Engineer Professional and/or AWS Certified Solutions Architect.
  • Experience operating platforms in compliance-driven environments (SOX, GxP / 21 CFR Part 11 or similar): change control, audit evidence, access reviews, segregation of duties.
  • Experience supporting, onboarding or mentoring other engineers.
  • Experience taking over or migrating an existing platform estate.
  • Experience working in a global delivery model with distributed stakeholders.

Preferred Qualifications

  • Databricks Certified Data Engineer Professional and/or AWS Certified Solutions Architect.
  • Experience operating platforms in compliance-driven environments (SOX, GxP / 21 CFR Part 11 or similar): change control, audit evidence, access reviews, segregation of duties.
  • Experience supporting, onboarding or mentoring other engineers.
  • Experience taking over or migrating an existing platform estate.
  • Experience working in a global delivery model with distributed stakeholders.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Illumina

Apprentice(Finance)

Bengaluru, IND

Discovered YesterdayFull-time

Sr Staff, NPI Sourcing Manager

, USA

Discovered 3 days agoFull-time

Material Coordinator 3

, USA

Discovered 3 days agoFull-time

Senior Bioinformatics Engineer

, USA

Discovered 4 days agoFull-time

Electrical Engineer 2

, USA

Discovered 5 days agoFull-time

Sr. Manager, Bioinformatics

, USA

Discovered 5 days agoFull-time

Field Service Engineer 2 - Atlanta, GA OR Birmingham, AL

, USA

Discovered 1 weeks agoFull-time

Staff Program Manager, Technical

, USA

Discovered 1 weeks agoFull-time