Lead Software Engineer - Data Engg. - Databricks / Snowflake

Role overview

Responsibilities

• Execute end-to-end data-driven software solutions, from design and development through troubleshooting.
• Build an enterprise data-pipeline control plane for pipeline definition, scheduling, deployment, governance, and runtime management.
• Develop self-service APIs/SDKs, templates, and configuration-driven onboarding with guardrails and centralized metadata.
• Build scalable pipelines using Python, PySpark, SQL, and Databricks on AWS; develop fact and dimension models.
• Provide standardized data quality, security, lineage, and observability through logs, metrics, traces, dashboards, alerting, runbooks, and automated remediation.
• Lead and participate in the SDLC and production support to improve platform and pipeline reliability.
• Shape data-management strategy with stakeholders and document data flows, logic, and transformation rules.
• Mentor engineers and lead communities of practice promoting modern engineering practices.
• Drive responsible adoption of approved AI-assisted engineering tools, with secure coding, peer review, automated tests, and validation standards.

Required qualifications

• Formal training or certification in software engineering concepts and 5+ years of applied experience.
• Large-scale data management and ETL/ELT experience; strong SQL, Python, and PySpark with performance tuning and query optimization.
• Hands-on Databricks/Spark and cloud data lake experience integrated with AWS services.
• Experience building platform services/control planes, APIs, configuration-driven systems, and version-compatible services.
• Knowledge of data quality, security by design, lineage/auditability, IAM/least privilege, and secrets management.
• Production engineering skills in observability, monitoring, incident response, and always-on service operations.
• CI/CD and release engineering proficiency, including automated testing and safe deployments/rollbacks.
• Experience leading approved AI-assisted development and validating output for correctness, performance, and security.
• Knowledge of responsible AI use, data sensitivity, secure handling, resiliency, and security; coaching engineers on compliant adoption.

Preferred qualifications

• Orchestration frameworks such as Databricks Workflows/Jobs, Airflow, or Step Functions; DAGs, replays, and backfills.
• Data governance integrations such as Unity Catalog concepts.
• Infrastructure as Code such as Terraform or CloudFormation, and developer-platform enablement.
• FinOps/cost controls, data formats such as Parquet, JSON, CSV, Avro, or Delta Lake, regulatory reporting, financial data aggregation, or Databricks/AWS certifications.