{bc}
Company Site

Data Engineer

Prevalent AI
Kochi, IND
Full-time
Mid · 3–5 years experience
Onsite
Discovered 3 weeks ago
databricksdelta-lakeicebergkafkapythonspark

Full Job Posting

Skills and Experience

  • Must Have
  • Experience: 3–5 years of relevant software or data engineering experience, including substantial hands-on development.
  • Python and PySpark: Strong programming skills and practical experience developing, debugging and optimising production-grade PySpark applications. Spark development must be a core engineering skill, not merely exposure through prebuilt pipelines or orchestration tools.
  • Apache Spark: Strong understanding of distributed processing, execution internals, shuffle behaviour, partitioning and parallelism.
  • Performance Optimisation: Practical understanding of data skew and mitigation, AQE, broadcast hash and sort-merge joins, shuffle optimisation, and diagnosing bottlenecks using Spark UI, History Server or execution plans.
  • PySpark Coding: Ability to independently solve data-processing problems involving joins, aggregations, window functions and deduplication using DataFrame APIs and Spark SQL.
  • Python Fundamentals: Strong command of data structures, functions, exception handling, debugging and maintainable code.
  • SQL: Strong SQL skills, including joins, aggregations, window functions and query optimisation fundamentals.
  • Good to Have
  • Spark Structured Streaming and incremental or event-driven processing.
  • Kafka or similar streaming platforms.
  • Delta Lake, Apache Iceberg or other lakehouse technologies.
  • Experience with Spark environments such as Databricks or Amazon EMR.

Qualifications

  • Must Have
  • Experience: 3–5 years of relevant software or data engineering experience, including substantial hands-on development.
  • Python and PySpark: Strong programming skills and practical experience developing, debugging and optimising production-grade PySpark applications. Spark development must be a core engineering skill, not merely exposure through prebuilt pipelines or orchestration tools.
  • Apache Spark: Strong understanding of distributed processing, execution internals, shuffle behaviour, partitioning and parallelism.
  • Performance Optimisation: Practical understanding of data skew and mitigation, AQE, broadcast hash and sort-merge joins, shuffle optimisation, and diagnosing bottlenecks using Spark UI, History Server or execution plans.
  • PySpark Coding: Ability to independently solve data-processing problems involving joins, aggregations, window functions and deduplication using DataFrame APIs and Spark SQL.
  • Python Fundamentals: Strong command of data structures, functions, exception handling, debugging and maintainable code.
  • SQL: Strong SQL skills, including joins, aggregations, window functions and query optimisation fundamentals.
  • Good to Have
  • Spark Structured Streaming and incremental or event-driven processing.
  • Kafka or similar streaming platforms.
  • Delta Lake, Apache Iceberg or other lakehouse technologies.
  • Experience with Spark environments such as Databricks or Amazon EMR.
  • 3–5 years of relevant software or data engineering experience, including substantial hands-on development.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Prevalent AI