Big Data - Senior Engineer

Job Description

Mandatory Skills

• Required technologies include Sqoop, Hadoop, HDFS, YARN, Azure Databricks, HBase, Impala, Hive, Scala, and Apache Hudi.
• Additional skills include PySpark, Spark, SQL, Data Lakehouse technologies, and Big Data engineering practices.
• The posting also lists Cloudera among mandatory competencies.

Key Responsibilities

• Design scalable Big Data solutions using Spark, Hadoop, Azure Databricks, and modern data platforms.
• Define data processing architecture, transformation strategies, and engineering standards.
• Lead distributed data processing pipelines using Spark with Scala or PySpark and Hadoop technologies.
• Design and optimize SQL and Hive-based processing solutions.
• Architect Azure Databricks solutions for large-scale data engineering and analytics workloads.
• Design and implement Lakehouse solutions using Apache Hudi or Apache Iceberg.
• Establish ingestion, transformation, validation, and reconciliation frameworks.
• Drive performance tuning across Spark, Databricks, Hive, and Hadoop environments.
• Review solutions, troubleshoot complex platform issues, and perform root cause analysis.
• Mentor team members and collaborate on end-to-end data platform delivery.

Behavioral Competencies

• Demonstrate ownership and drive Big Data engineering excellence.
• Collaborate effectively with teams and business stakeholders.
• Apply analytical thinking to complex data engineering and platform challenges.
• Adapt to evolving technologies, data ecosystems, and business requirements.
• Communicate delivery status, risks, dependencies, and improvement opportunities clearly.
• Maintain attention to detail across architecture, processing, testing, and implementation.
• Support knowledge sharing and continuous improvement in engineering practices.

About Iris Software

Perks and Benefits

• Iris states that its benefits support the financial, health, and well-being needs of employees.