Data Quality Engineer (Databricks)

Overview

Location: Abudhabi - UAE - Onsite

Experience: 5 to 7 Years

Role Overview

Key Responsibilities

• **Databricks Platform Configuration and Administration*** Configure and manage the Databricks environment supporting enterprise data quality operations.
• Establish and maintain:
• + Compute clusters.
• + PySpark notebook frameworks.
• + Delta Lake storage structures.
• + Unity Catalog integration.
• Optimise platform performance for large-scale profiling and rule execution across all in-scope datasets and CDEs.
• Implement development, testing, and production deployment standards for data quality assets.
• **Data Profiling and Quality Assessment*** Design and develop AI-assisted profiling notebooks using PySpark.
• Perform baseline data quality assessments across the six quality dimensions:
• + Completeness.
• + Accuracy.
• + Consistency.
• + Validity.
• + Timeliness.
• + Uniqueness.
• Capture and analyse:
• + Null value rates.
• + Duplicate records.
• + Invalid values.
• + Format violations.
• + Outliers.
• + Schema drift.
• Produce quality profiling outputs for all prioritised CDEs and datasets.
• **Data Quality Rule Factory Development*** Design and implement a reusable Data Quality Rule Factory.
• Build parameterised PySpark-based rule templates capable of supporting large-scale rule deployment.
• Enable automated generation and management of approximately 6,730 data quality rules without manual rule-by-rule development.
• Ensure rules are reusable, configurable, and maintainable across multiple datasets and domains.
• **Data Quality Controls and Pipeline Integration*** Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines.
• Embed quality controls within Bronze, Silver, and Gold processing stages.
• Implement automated quality gates preventing data progression where defined thresholds are not met.
• Maintain rule traceability and execution history for audit and governance purposes.
• **Data Cleansing and Quality Improvement*** Develop automated remediation and cleansing pipelines using PySpark.
• Implement:
• + Standardisation routines.
• + Data enrichment processes.
• + Deduplication logic.
• + Schema harmonisation controls.
• Deploy machine learning models managed through MLflow for:
• + Anomaly detection.
• + Exact duplicate detection.
• + Fuzzy matching and duplicate identification.
• Ensure all AI and ML recommendations are explainable, auditable, and routed through human-in-the-loop validation processes where required.
• **Exception Management and Reprocessing*** Design and manage exception handling processes for failed quality records.
• Implement quarantine Delta Lake tables serving as the Failed Record Register.
• Capture and maintain:
• + Failure reason.
• + Associated CDE.
• + Rule reference.
• + Processing timestamp.
• + Resolution status.
• Develop reprocessing workflows to support correction and controlled re-ingestion of remediated records.
• **Data Quality Metrics and Reporting*** Develop Delta Lake metric aggregation structures supporting enterprise quality reporting.
• Calculate and publish:
• + Data Quality Index (DQI) scores.
• + Dimension-level quality metrics.
• + Rule pass/fail rates.
• + Dataset compliance scores.
• + SLA adherence indicators.
• Provide curated outputs to support Power BI quality dashboards and executive reporting.
• **Monitoring, Alerting and Predictive Quality Management*** Configure automated quality monitoring and alerting mechanisms.
• Implement threshold-based notifications using:
• + Databricks SQL Alerts.
• + Azure Monitor integrations.
• Develop predictive risk scoring models to identify datasets at risk of future quality degradation.
• Support proactive quality management and operational intervention activities.
• **Root Cause Analysis and Remediation Support*** Apply Databricks machine learning and pattern analysis techniques to profiling and rule execution outputs.
• Support AI-assisted root cause analysis across established remediation categories.
• Identify recurring quality issues, systemic defects, and process breakdowns.
• Produce prioritised remediation datasets for business and operational stakeholders.
• Export remediation outputs to Power BI and Excel to support:
• + Remediation Tiering Matrix.
• + Prioritisation Scoring Models.
• + Governance reporting processes.
• **Governance, Compliance and Stakeholder Engagement*** Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with authoritative data definitions and metadata standards.
• Support DDA and DGE governance processes by producing required quality artefacts and evidence.
• Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes.
• Participate in quality reviews, governance forums, and stakeholder workshops.
• **Required Skills and Experience*** Strong experience designing and implementing enterprise Data Quality frameworks.
• Advanced Databricks engineering experience.
• Strong PySpark development skills.
• Experience with:
• + Delta Lake.
• + Unity Catalog.
• + Databricks Workflows and Jobs.
• + Databricks SQL.
• Experience building scalable data validation and quality rule frameworks.
• Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
• Experience using MLflow for model management and deployment.
• Strong understanding of data governance, metadata management, and data lifecycle processes.
• Experience integrating data quality metrics into reporting platforms such as Power BI.
• Knowledge of cloud-based data engineering and modern lakehouse architectures.
• **Deliverables*** Configured Databricks Data Quality environment.
• Enterprise Data Quality Rule Factory.
• AI-assisted profiling notebooks and baseline assessment outputs.
• Automated data quality validation and gating processes.
• Data cleansing and remediation pipelines.
• Failed Record Register and reprocessing workflows.
• Data Quality metric aggregation tables.
• DQI reporting feeds and dashboard datasets.
• Predictive quality monitoring and alerting solutions.
• Root Cause Analysis and remediation support datasets.
• Governance, audit, and compliance artefacts supporting DDA and DGE reviews.
• dmGrkjT88v