Big Data Engineer

Overview

About The Role

Key Responsibilities

• *Defining and developing services and solutions*
• Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.
• Define, design, and run robust data pipelines/batch jobs in a production environment.
• Architecting highly scalable, highly concurrent, and low latency systems
• *Maintain, support, and enhance current systems.*
• Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.
• Contribute feedback to improve the quality, readability, and testability of the code base within your team.
• Mentor and train other developers in a non-line management capacity.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.
• *Collaborating with Internal and external stakeholders*
• Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.
• Work with third-party and other internal providers to support a variety of integrations.
• As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.
• Proactively advise on best practices.

To Be The Right Fit, You'll Need

• Degree in Computer Science, Software Engineering or related preferred
• Minimum of 5 years experience in a Big Data
• Follow Clean Code/Solid principles, adhere and use TDD/BDD.
• Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.
• Highly Proficient in either Functional Java or Scala is a must
• Knowledge of Azure Big Data/Analytics services – ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks
• Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs
• Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs
• In-depth knowledge of Hadoop technology ecosystem – HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet
• Experience debugging a complex multi-server service.
• In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)
• Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.
• Knowledge of relational and non-relational database systems
• Understanding continuous integration and delivery.
• Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).
• Knowledge of IDE Intellij or Eclipse.
• Knowledge of build tools (One of SBT, Gradle, Maven).
• An ability to communicate technical concepts to a non-technical audience.
• Experience working knowledge of unix-like operating systems such as Linux and/or Mac OS X.
• Knowledge of the git version control system.
• Ability to quickly research and learn new programming tools and techniques.