Senior Data Engineer

Responsibilities

Your role

• will encompass a range of activities focused on data pipeline development, data quality, and cross-functional collaboration.
• **Data Pipeline Architecture and Development:**
• Design, construct, install, test, and maintain highly scalable data pipelines with a focus on machine learning models and analytics.
• **Data Integration:**
• Work closely with data scientists, ML engineers, and stakeholders to ensure that data is accessible, consistent, and reliable for ongoing projects.
• **API and Data Services:**
• Develop and maintain APIs for data access and manipulation, and integrate with external data services as needed.
• **Data Storage:**
• Manage and optimize data storage solutions for both structured and unstructured data, where structured data includes relational databases and unstructured data includes Text, Image, Audio and Video, Search Engines like Elasticsearch, and NoSQL databases, to support the requirements of machine learning models.
• Understand data engines and structure to effectively design solutions for transactional, analytics, and search purposes.
• **Data Quality and Governance:**
• Implement processes to monitor data quality and ensure production data is always accurate and available for key stakeholders.
• **Collaboration and Support:**
• Collaborate with ML engineers to assist in data-related technical issues and provide architectural guidance and solutions.
• **Security and Compliance:**
• Ensure compliance with data security and privacy policies.
• **Documentation:**
• Maintain clear and up-to-date documentation, including data dictionaries, metadata, and architectural diagrams.

Skills and Attributes for Success

• Bachelor's degree in Computer Science, Engineering, Mathematics, or a related field; or equivalent work experience
• 8+ years of experience in a Data Engineering role
• Proficiency in programming languages including Python, Java, and Scala; experience managing large-scale data at Terabyte to Petabyte scale
• Hands-on experience with big data technologies like Hadoop, Spark, and Flink
• Familiarity with machine learning frameworks such as TensorFlow, PyTorch, or similar
• Strong understanding of data warehousing concepts, ETL processes, and data modeling
• Experience with API development and integration with data services
• Experience with cloud platforms like Azure/AWS/GCP, VMware and other
• Knowledge of DevOps, CI/CD methods, and containerization technologies like Docker or Kubernetes
• Cloud & Infrastructure: Hands-on experience with AWS services (S3, Redshift, Glue, Lambda) as the primary cloud platform, with a strong focus on security and scalability
• BI & Visualization: Experience with business intelligence tools such as Metabase for data visualization, reporting, and enabling self-serve analytics
• Data Governance & Cataloging: Familiarity with metadata management and data catalog tools (e.g., OpenMetadata) to support data discoverability and governance initiatives
• Experience with real-time / streaming data processing
• Good to have fintech experience

Technical Stack

• **Programming Languages**

: Python, Java, Scala, SQL, Bash

• **Big Data Technologies**

: Hadoop, Spark, Flink

• **Unstructured Data**

: Text, Image, Audio & Video

• **Databases**

: MySQL, PostgreSQL, MongoDB, Cassandra, HBase, Redis

• **Cloud Platforms**
• : Azur, GCP, AWS and similar
• **API Development**

: RESTful APIs, GraphQL, OpenA

• **PIData Services**
• : Kafka, Rabbit
• **MQContainers**
• : Docker, Kubernetes