Lead Data Engineer
Role Overview
As a Lead Data Engineer at Prevalent AI, you will lead the design and delivery of advanced data engineering projects for our clients.
You will write code, review the team's code, and stay accountable for how the solution performs in production.
The work involves large data volumes, complex joins and transformations, and tight processing windows.
Clients usually come to us when their pipelines are slow, expensive or difficult to change.
Working out why, and then fixing it, is a large part of the role.
This role suits engineers who know Spark internals well enough to read the logs and explain what is going wrong, who ask the right questions about the data before they write code, and who can run a small team to a delivery date.
Key Skills for This Role
Full Job Posting
Skills & Experience
- Must Have:
- 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily.
- Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour.
- Experience tuning long-running jobs down to an SLA and handling data skew, with specific examples of how a job was diagnosed and what changed.
- Hands-on experience with time-series data at scale: event-time semantics, sliding and tumbling windows, and as-of or point-in-time joins across billions of rows.
- Strong Delta Lake or Iceberg experience, covering partitioning, compaction, sort keys, data pruning and versioning.
- Able to read Spark logs, the Spark UI and the history server and work out why a job is slow or failing.
- Experience running Spark on self-managed clusters, not only on managed platforms.
- Strong Python, with experience building reusable, configuration-driven frameworks rather than one-off scripts.
- Able to lead a small team through code reviews, mentoring and unblocking, and hold a delivery date.
- Good to Have
- Trade surveillance, capital markets or FX experience, and familiarity with trade, order and market data.
- KDB/q exposure, or experience migrating off a vendor surveillance platform.
- Experience moving Spark workloads to object storage such as MinIO or S3, and running Spark on Kubernetes.
- CI/CD, automated testing and observability for data pipelines.
- Exposure to regulatory or compliance-driven delivery in a banking environment.
Qualifications
- 10+ years in data engineering, with at least 5 years hands-on PySpark on large-volume workloads, and still writing code daily.
- Deep understanding of Spark internals: execution model, stages and tasks, shuffle, partitioning, broadcast versus sort-merge joins, adaptive query execution, and memory and spill behaviour.
- Experience tuning long-running jobs down to an SLA and handling data skew, with specific examples of how a job was diagnosed and what changed.
- Hands-on experience with time-series data at scale: event-time semantics, sliding and tumbling windows, and as-of or point-in-time joins across billions of rows.
- Strong Delta Lake or Iceberg experience, covering partitioning, compaction, sort keys, data pruning and versioning.
- Able to read Spark logs, the Spark UI and the history server and work out why a job is slow or failing.
- Experience running Spark on self-managed clusters, not only on managed platforms.
- Strong Python, with experience building reusable, configuration-driven frameworks rather than one-off scripts.
- Able to lead a small team through code reviews, mentoring and unblocking, and hold a delivery date.
- Good to Have
- Trade surveillance, capital markets or FX experience, and familiarity with trade, order and market data.
- KDB/q exposure, or experience migrating off a vendor surveillance platform.
- Experience moving Spark workloads to object storage such as MinIO or S3, and running Spark on Kubernetes.
- CI/CD, automated testing and observability for data pipelines.
- Exposure to regulatory or compliance-driven delivery in a banking environment.
About Prevalent AI
Prevalent AI is a private cybersecurity software company providing enterprises a sovereign data fabric for exposure management.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Prevalent AI
Product Manager - Exposure Management
Kochi, IND
Prevalent AI is seeking a Product Manager to own the strategy, roadmap, and growth of its Exposure Management application. The role combines cybersecurity domain expertise, AI-powered product delivery, client engagement,
Talent Acquisition and People Partner
Kochi, IND
VP Data Science
Kochi, IND
Sales Director - US
, USA
Product Manager - Exposure Management
Kochi, IND
Product Manager - Exposure Management
Kochi, IND
Growth Marketing Manager
London, GBR