Senior Data Engineer

We are seeking an experienced Senior Data Engineer to lead the design, development, and delivery of scalable enterprise data solutions.

This role will build and optimize batch and real-time data pipelines, reusable integration frameworks, and governed data platforms that support analytics, AI, and self-service data access.

The ideal candidate has deep expertise in Databricks, Apache Spark, AWS, data modeling, governance, and production operations.

The role also provides technical leadership, defines engineering standards, mentors other engineers, and partners with architecture, business, analytics, data science, and DevOps teams.

in manufacturing, biotechnology, pharmaceutical, life sciences, or another regulated industry is preferred.

Roles and Responsibilities

Lead the design and development of scalable batch and real-time ETL/ELT pipelines.

Own complex data solutions from requirements and design through deployment and production support.

Build cloud-based lakehouse and data-platform solutions using Databricks and AWS.

Develop reusable, metadata-driven data integration frameworks.

Integrate structured, semi-structured, and unstructured data from enterprise, manufacturing, API, and third-party sources.

Optimize Spark workloads, Databricks compute, SQL queries, partitioning, storage, and caching for performance and cost.

Implement workflow orchestration, monitoring, alerting, data-quality controls, and recovery processes.

Develop CI/CD pipelines and automated testing for data solutions.

Implement metadata management, lineage, cataloging, governance, RBAC, and data-security controls.

Define data models, data contracts, integration patterns, and reusable engineering standards.

Lead architecture reviews, code reviews, troubleshooting, and root-cause analysis.

Mentor engineers and provide technical guidance across delivery teams.

Collaborate with architects, analysts, data scientists, product teams, and DevOps teams.

Support estimation, sprint planning, technical roadmaps, and delivery-risk management.

Evaluate emerging technologies and recommend solutions based on scalability, security, maintainability, and business value.

Participate in operational support, including occasional off-hours support.

One of the following:

Master’s degree in Computer Science, Engineering, Information Technology, Data Science, or a related field and at least 7 years of relevant experience.

Bachelor’s degree in a related field and at least 9 years of relevant experience.

Advanced hands-on experience with Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Python, and SQL.

designing and operating production-grade batch and streaming pipelines.

Strong understanding of distributed computing, lakehouse architecture, data warehousing, and data integration.

with Databricks Workflows or comparable orchestration tools.

Strong experience with AWS data, compute, storage, security, and monitoring services.

with Spark performance tuning, cluster optimization, partitioning, and cost management.

with Git, CI/CD, automated testing, monitoring, and production deployment.

implementing data quality, metadata management, lineage, governance, and access controls.

Strong understanding of RBAC, least privilege, encryption, auditability, and regulated-data requirements.

leading technical design, code reviews, and complex production implementations.

Ability to define reusable patterns, engineering standards, and development best practices.

Strong communication, collaboration, mentoring, and problem-solving skills.

working in Agile or Scaled Agile delivery environments.

Experience in biotechnology, pharmaceutical, life sciences, manufacturing, or another regulated industry.

with Unity Catalog, data products, Data Fabric, Data Mesh, or similar enterprise data architectures.

building APIs and secure data services.

with relational, NoSQL, analytical, operational, or vector databases.

with OLAP and OLTP data modeling and performance tuning.

with Kafka, Kinesis, or other streaming technologies.

supporting AI and Generative AI solutions, including RAG, embeddings, vector search, and governed enterprise data access.

Familiarity with AI-assisted development tools such as GitHub Copilot, OpenAI Codex, or equivalent platforms.

AWS Certified Data Engineer or another relevant AWS certification.

Databricks Certified Data Engineer Associate or Professional.

Scaled Agile Framework certification.

Excellent analytical and troubleshooting skills.

Strong written, verbal, presentation, and stakeholder-communication skills.

High degree of ownership, initiative, and self-motivation.

Ability to manage multiple priorities in a fast-paced environment.

Ability to work effectively with global and virtual teams.

Strong attention to detail and commitment to engineering quality.

Ability to influence technical decisions and drive work to completion.