Data Engineer

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines for ingesting, transforming, and loading data from multiple sources.
• Build and optimize scalable data processing solutions using Azure Databricks and Apache Spark.
• Develop and manage workflows using Azure Data Factory for orchestrating enterprise data movement and transformation.
• Create, maintain, and enhance SQL Server Integration Services (SSIS) packages for on-premises and hybrid data integration requirements.
• Develop optimized SQL queries, stored procedures, views, and database objects to support reporting and analytics.
• Collaborate with business teams to understand data requirements and translate them into technical solutions.
• Ensure data quality through validation, cleansing, reconciliation, and monitoring processes.
• Optimize pipeline performance, troubleshoot bottlenecks, and implement best practices for scalability and reliability.
• Integrate structured and semi-structured data from multiple enterprise systems.
• Monitor production data pipelines, resolve failures, and implement proactive monitoring mechanisms.
• Participate in code reviews, documentation, and knowledge-sharing initiatives.
• Follow data governance, security, compliance, and best practices throughout the data lifecycle.
• Support migration of legacy ETL processes to modern Azure-based data platforms where applicable.
• 5–8 years of experience in Data Engineering or ETL Development.
• Strong hands-on expertise in Azure Databricks.
• Extensive experience with Azure Data Factory (ADF).
• Proficiency in SQL Server Integration Services (SSIS).
• Strong SQL programming skills with experience in query optimization and performance tuning.
• Experience developing scalable ETL/ELT pipelines.
• Good understanding of Azure Data Lake Storage and cloud-based data architectures.
• Experience with Apache Spark using PySpark or Spark SQL.
• Strong knowledge of relational databases and data warehouse concepts.
• Familiarity with data modeling, data transformation, and data integration techniques.
• Experience with source control systems such as Git.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience with Azure Synapse Analytics.
• Knowledge of Azure SQL Database or SQL Server administration.
• Familiarity with Delta Lake architecture.
• Experience with CI/CD pipelines for Azure data solutions.
• Exposure to Power BI or other business intelligence tools.
• Understanding of DevOps practices for data engineering.
• Experience working with REST APIs and data ingestion from external systems.
• Knowledge of data governance, security, and compliance standards.
• Familiarity with Agile/Scrum development methodologies.

Must-Have Skills

Good-to-Have Skills

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Microsoft Azure Data Engineer certification is an added advantage.
• Excellent communication and stakeholder management skills.
• Ability to work independently while collaborating effectively within cross-functional teams.
• Strong commitment to delivering high-quality, scalable, and maintainable data engineering solutions.

Qualifications

• Key Responsibilities
• Design, develop, and maintain robust ETL/ELT pipelines for ingesting, transforming, and loading data from multiple sources.
• Build and optimize scalable data processing solutions using Azure Databricks and Apache Spark.
• Develop and manage workflows using Azure Data Factory for orchestrating enterprise data movement and transformation.
• Create, maintain, and enhance SQL Server Integration Services (SSIS) packages for on-premises and hybrid data integration requirements.
• Develop optimized SQL queries, stored procedures, views, and database objects to support reporting and analytics.
• Collaborate with business teams to understand data requirements and translate them into technical solutions.
• Ensure data quality through validation, cleansing, reconciliation, and monitoring processes.
• Optimize pipeline performance, troubleshoot bottlenecks, and implement best practices for scalability and reliability.
• Integrate structured and semi-structured data from multiple enterprise systems.
• Monitor production data pipelines, resolve failures, and implement proactive monitoring mechanisms.
• Participate in code reviews, documentation, and knowledge-sharing initiatives.
• Follow data governance, security, compliance, and best practices throughout the data lifecycle.
• Support migration of legacy ETL processes to modern Azure-based data platforms where applicable.
• Must-Have Skills
• 5–8 years of experience in Data Engineering or ETL Development.
• Strong hands-on expertise in Azure Databricks.
• Extensive experience with Azure Data Factory (ADF).
• Proficiency in SQL Server Integration Services (SSIS).
• Strong SQL programming skills with experience in query optimization and performance tuning.
• Experience developing scalable ETL/ELT pipelines.
• Good understanding of Azure Data Lake Storage and cloud-based data architectures.
• Experience with Apache Spark using PySpark or Spark SQL.
• Strong knowledge of relational databases and data warehouse concepts.
• Familiarity with data modeling, data transformation, and data integration techniques.
• Experience with source control systems such as Git.
• Strong analytical, troubleshooting, and problem-solving skills.
• Good-to-Have Skills
• Experience with Azure Synapse Analytics.
• Knowledge of Azure SQL Database or SQL Server administration.
• Familiarity with Delta Lake architecture.
• Experience with CI/CD pipelines for Azure data solutions.
• Exposure to Power BI or other business intelligence tools.
• Understanding of DevOps practices for data engineering.
• Experience working with REST APIs and data ingestion from external systems.
• Knowledge of data governance, security, and compliance standards.
• Familiarity with Agile/Scrum development methodologies.