Data Product Manager

Role Overview

Data Pipeline Engineering

• Architect end-to-end data products with Medallion Architecture using Amazon S3 and Delta Lake.
• Design production batch and real-time streaming pipelines using PySpark, SQL, and Delta Live Tables.
• Integrate Databricks with Amazon Kinesis or Managed Streaming for Apache Kafka.
• Optimize Lakehouse storage with Liquid Clustering, Z-Ordering, and data compaction.
• Apply unit, integration, and negative testing, along with model monitoring, alerting, and pipeline observability.

AWS Security and Governance

• Implement governance and role-based access control for AWS-hosted data assets using Unity Catalog.
• Configure security roles and personas, and use AWS Secrets Manager to protect pipeline credentials and API keys.
• Optimize cloud spend through Databricks cluster management, EC2 instance selection, and DBU tracking.

DevOps and Engineering Excellence

• Use Azure DevOps Git integration, automated unit testing, and CI/CD pipelines such as GitHub Actions and Databricks DAB.
• Orchestrate workflows with Databricks Workflows or Apache Airflow.
• Deploy infrastructure as code with Terraform for Databricks workspaces and AWS data resources.
• Mentor mid-level engineers, establish coding standards, and conduct peer code reviews.

Required Skills and Qualifications

• Requires 6+ years in data engineering, including 4+ years focused on Databricks in AWS.
• Requires expert Python with PySpark and complex analytical SQL.
• Requires deep expertise in Delta Lake, Amazon S3, Spark optimization, EC2, IAM, Secrets Manager, and VPC fundamentals.
• Requires Terraform experience for data-platform provisioning and automated deployment tools.
• Requires technical design leadership, cost-efficient cloud decision-making, and clear documentation skills.

Preferred Certification

• Databricks Certified Data Engineer Associate or Professional certification is optional.