Data QA Engineer

Project description

The ETL/Data QA Engineer will be responsible for ensuring the accuracy, reliability, and quality of data processed through the organisation's data warehousing and integration platforms. Operating within established testing and governance frameworks, the engineer will validate ETL processes, conduct detailed source-to-target data verification, and contribute to continuous improvements in data quality practices. This role demands strong hands-on experience with Informatica PowerCenter, SQL, Python, and AI-driven automation, as well as the ability to collaborate effectively with cross-functional teams.

Responsibilities

• Validate ETL mappings, workflows, and transformations to ensure compliance with business and technical requirements.
• Perform functional and non-functional testing for data validation between source and target DBs as per the business requirements using relevant Azure components such as Azure data bricks and Azure data factories.
• Execute ETL test cycles, including data-quality checks for both full and incremental data loads.
• Conduct smoke, integration, regression, and other tests on products under development and released
• Collaborate with developers, business analysts, and users to understand testing needs and identify opportunities for automation and testing.
• Lead defect triage sessions, facilitating timely resolution and communication among development, business, and QA teams.
• Utilize Informatica PowerCenter for ETL validation, debugging, and workflow monitoring.
• Apply ETL and data-warehousing concepts, demonstrating a strong understanding of principles, architecture, and best practices.
• Use Unix/Linux commands for job monitoring, log analysis, and operational support.
• Leverage Python and AI expertise to support automation, data validation, and advanced testing scenarios.
• Analyze and document test data, results, and recommendations to aid decision-making.
• Support integrated testing by creating and maintaining test data, training data, and automated scripts.

SKILLS

Must have

• 5+ years’ experience in data automation using Python, PySpark.
• Hands on experience in building customized Automation framework using Python, PySpark and SQL.
• Extensive experience with ETL concepts & technologies, Data migration testing, Data warehouse testing, and BI Reporting.
• Hands on experience in working with CSV and JSON using PySpark.
• High level of Proficiency in writing complex SQL queries in Azure data bricks from Business requirements to read data from source DB hosted in Azure and compare it with the target DB.
• Proficiency in relational database models, SQL queries, XML data models.
• Extensive experience with software test automation tools and working in fast-paced Agile environments.
• Experience with Azure Data Factory for pipeline execution and analysis.
• Experience with Azure Databricks to access data from Azure Data Lakes.
• Experience with Azure Notebooks
• Experience with Cosmos DB
• Experience in understanding in version control, CI/CD solutions using tools such as Azure DevOps.
• Experience in the Banking domain is highly desirable
• Bachelor's Degree in IT, Business, Computer Science, or a related field
• Good knowledge of testing lifecycle processes
• Experience with Atlassian tools (Jira and Confluence)
• Good communication and presentation skills

Nice to have

Experience in the finance industry is highly desirable.

Responsibilities

• • Validate ETL mappings, workflows, and transformations to ensure compliance with business and technical requirements. • Perform functional and non-functional testing for data validation between source and target DBs as per the business requirements using relevant Azure components such as Azure data bricks and Azure data factories. • Execute ETL test cycles, including data-quality checks for both full and incremental data loads. • Conduct smoke, integration, regression, and other tests on products under development and released • Collaborate with developers, business analysts, and users to understand testing needs and identify opportunities for automation and testing. • Lead defect triage sessions, facilitating timely resolution and communication among development, business, and QA teams. • Utilize Informatica PowerCenter for ETL validation, debugging, and workflow monitoring. • Apply ETL and data-warehousing concepts, demonstrating a strong understanding of principles, architecture, and best practices. • Use Unix/Linux commands for job monitoring, log analysis, and operational support. • Leverage Python and AI expertise to support automation, data validation, and advanced testing scenarios. • Analyze and document test data, results, and recommendations to aid decision-making. • Support integrated testing by creating and maintaining test data, training data, and automated scripts.

Skills

• • 5+ years’ experience in data automation using Python, PySpark. • Hands on experience in building customized Automation framework using Python, PySpark and SQL. • Extensive experience with ETL concepts & technologies, Data migration testing, Data warehouse testing, and BI Reporting. • Hands on experience in working with CSV and JSON using PySpark. • High level of Proficiency in writing complex SQL queries in Azure data bricks from Business requirements to read data from source DB hosted in Azure and compare it with the target DB. • Proficiency in relational database models, SQL queries, XML data models. • Extensive experience with software test automation tools and working in fast-paced Agile environments. • Experience with Azure Data Factory for pipeline execution and analysis. • Experience with Azure Databricks to access data from Azure Data Lakes. • Experience with Azure Notebooks • Experience with Cosmos DB • Experience in understanding in version control, CI/CD solutions using tools such as Azure DevOps. • Experience in the Banking domain is highly desirable • Bachelor's Degree in IT, Business, Computer Science, or a related field • Good knowledge of testing lifecycle processes • Experience with Atlassian tools (Jira and Confluence) • Good communication and presentation skills