Senior Data Engineer
Role Overview
We are looking for a hands-on Senior Data Engineer with strong expertise in Python, PySpark and Apache Spark to develop, troubleshoot and optimise production-grade distributed data-processing workloads.
This is a
Spark engineering role, not a general data engineering or orchestration role.
The successful candidate must be able to write efficient PySpark code, understand Spark execution behaviour, diagnose complex performance bottlenecks and independently own engineering problems through implementation, testing and production delivery.
Key Skills for This Role
Full Job Posting
Skills and Experience
- Must Have
- Experience: 5–7 years of relevant software or data engineering experience, including substantial hands-on development.
- Python and PySpark: Strong programming skills and practical experience developing, debugging and optimising production-grade PySpark applications. Spark development must be a core engineering skill, not merely exposure through prebuilt pipelines or orchestration tools.
- Apache Spark: Strong understanding of distributed processing, execution internals, shuffle behaviour, partitioning and parallelism.
- Performance Optimisation: Practical understanding of data skew and mitigation, AQE, broadcast hash and sort-merge joins, shuffle optimisation, and diagnosing bottlenecks using Spark UI, History Server or execution plans.
- PySpark Coding: Ability to independently solve data-processing problems involving joins, aggregations, window functions and deduplication using DataFrame APIs and Spark SQL.
- Python Fundamentals: Strong command of data structures, functions, exception handling, debugging and maintainable code.
- SQL: Strong SQL skills, including joins, aggregations, window functions and query optimisation fundamentals.
- Good to Have
- Spark Structured Streaming and incremental or event-driven processing.
- Kafka or similar streaming platforms.
- Delta Lake, Apache Iceberg or other lakehouse technologies.
- Experience with Spark environments such as Databricks or Amazon EMR.
Qualifications
- Must Have
- Experience: 5–7 years of relevant software or data engineering experience, including substantial hands-on development.
- Python and PySpark: Strong programming skills and practical experience developing, debugging and optimising production-grade PySpark applications. Spark development must be a core engineering skill, not merely exposure through prebuilt pipelines or orchestration tools.
- Apache Spark: Strong understanding of distributed processing, execution internals, shuffle behaviour, partitioning and parallelism.
- Performance Optimisation: Practical understanding of data skew and mitigation, AQE, broadcast hash and sort-merge joins, shuffle optimisation, and diagnosing bottlenecks using Spark UI, History Server or execution plans.
- PySpark Coding: Ability to independently solve data-processing problems involving joins, aggregations, window functions and deduplication using DataFrame APIs and Spark SQL.
- Python Fundamentals: Strong command of data structures, functions, exception handling, debugging and maintainable code.
- SQL: Strong SQL skills, including joins, aggregations, window functions and query optimisation fundamentals.
- Good to Have
- Spark Structured Streaming and incremental or event-driven processing.
- Kafka or similar streaming platforms.
- Delta Lake, Apache Iceberg or other lakehouse technologies.
- Experience with Spark environments such as Databricks or Amazon EMR.
- 5–7 years of relevant software or data engineering experience, including substantial hands-on development.
About Prevalent AI
Provides an AI-driven cybersecurity platform for enterprise exposure management.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Prevalent AI
Business Analyst (Contractor)
, IND
Finance Manager
Kochi, IND
Data Engineer
Kochi, IND
Information Security Director
Kochi, IND
Talent Acquisition and People Partner
Kochi, IND
Lead Data Engineer
Kochi, IND
VP Data Science
Kochi, IND
