Data Engineer

Role Overview

Databricks Engineering

• Design, develop, and maintain Databricks pipelines using Delta Lake and Databricks SQL.
• Build Databricks Workflows for orchestration, scheduling, monitoring, and dependency management.
• Implement reusable ingestion and transformation frameworks with production error handling, logging, monitoring, and recovery.
• Optimize workloads for performance, scalability, reliability, and cost efficiency.

REST API Integration

• Develop reusable REST API ingestion frameworks supporting authentication, pagination, rate limits, retries, incremental extraction, and error handling.
• Transform API responses into structured datasets for downstream analytics.
• Troubleshoot connectivity, data availability, schema changes, and ingestion failures.

Data Modeling and Python

• Design dimensional models, fact tables, dimension tables, and relationships for reporting and analytics.
• Write and optimize advanced SQL for transformation, validation, aggregation, and analytical processing.
• Develop reusable Python libraries and utilities for ingestion, transformation, validation, automation, testing, and operations.

Quality and Governance

• Implement data quality checks for completeness, accuracy, consistency, uniqueness, validity, reconciliation, thresholds, alerts, and exceptions.
• Configure Unity Catalog access controls, permissions, metadata, documentation, governance, security, discoverability, and lineage.

Production Operations

• Deploy and support Databricks pipelines in production environments.
• Monitor execution, performance, failures, and data quality, and perform root-cause analysis.
• Apply CI/CD and deployment practices where applicable and collaborate with architects, analysts, DevOps, product owners, and business stakeholders.

Required Experience

• Production Data Engineering experience with Databricks, Delta Lake, Databricks SQL, Databricks Workflows, Unity Catalog, REST APIs, advanced SQL, and Python.
• Experience with dimensional data modeling, data quality frameworks, ingestion, transformation, orchestration, and production operations.
• Understanding of enterprise data security, access controls, governance, scalable solutions, and maintainable data engineering practices.

Preferred Experience

• Medallion Architecture, Delta Live Tables, Azure Data Factory, Azure Data Lake Storage, Azure Functions, or Azure Synapse experience is strongly preferred.
• Experience with adapter or canonical schemas, engineering data sources, AI coding-assistant telemetry, Git, Azure DevOps, CI/CD, or data observability is strongly preferred.