Senior Data Engineer - Pyspark

Job Description

• Good understanding of source systems
• data structures,
• data retentions policies and data partitioning for modeling raw data vault structures.
• Drive data reusability, reduce duplicity of data, minimizing adverse impact to performance.
• Develop
• data architecture
• vision and standards in conjunction with business users and stewards.
• Create data exchange standards to ensure reusability and a decoupled architecture.
• Create standards for
• data archival
• and purging.
• Identify and document critical data elements across source system of records.
• Work with data engineers and source system experts to define the standards and principles.

Job Dimensions

• Master or Bachelor’s degree in computer science, information systems management or related field.
• More than 5+ years of experience in information technology, with
• 3+ years spent in data engineering, architecture
• and technology solutions definitions and implementations.
• Extensive experience in banking services or related domain.
• Expertise in

Data Architecture, Data Strategy

Data Management & Analytics

• solutions.
• Experience in transforming traditional
• Data Warehousing approaches to Big Data based approaches
• and proven track record of managing risks and data security.
• Expertise with
• DW Dimensional modeling techniques, Star & Snowflake schemas
• , modeling slowly changing dimensions and role playing dimensions, dimensional hierarchies, and data classification.
• Experiences in cloud native principals, designs and deployments.
• Extensive experience working with and
• enhancing Continuous Integration (CI) and Continuous Development (CD) environments
• .
• Expertise in
• Data Quality, Data Profiling, Data Governance, Data Security, Metadata Management, and Data Archival.
• Demonstrated ability to work in a fast paced and changing environment with short deadlines, interruptions, and multiple tasks/projects occurring simultaneously.
• Must be able to work independently and have skills in planning, strategy, estimation, scheduling, Strong problem solving, influencing, communication, and presentation skills, self-starter.
• Experience with
• data processing frameworks and platforms (Hadoop, MongoDB, Tez, Hive, Spark etc.).
• Hands-on experience in work with
• bigdata platforms and related programming languages (e.g.
• Linux, Python, PySpark, GIT, Jenkins)