{bc}
Apply
company_site

Sr Data Engineer - Databricks

Koch
Atlanta, USA
Senior
Onsite
Discovered 3 weeks ago

Full Job Posting

  • Design, build, deploy, and support Databricks-native data engineering solutions, including Delta Lake pipelines, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader ingestion pipelines, Unity Catalog resources, and CI/CD automation
  • Develop reliable, governed data platforms and pipelines that improve the accuracy, consistency, accessibility, and trustworthiness of enterprise data assets
  • Create automated tools, reusable templates, and self-service patterns that accelerate data development, ingestion, feature engineering, and consumption
  • Operate and improve the platform through performance monitoring, incident resolution, migrations, upgrades, observability, and cost optimization aligned with long-term scalability and resilience goals
  • Partner with product owners, operations teams, architects, and delivery teams to identify common use cases, prioritize platform enhancements, and expand adoption
  • Integrate Databricks pipelines with enterprise governance, metadata, lineage, classification, certification, ownership, and data-quality capabilities
  • Develop and curate governed data assets for AI and advanced analytics use cases, including semantic models, APIs, Genie Spaces, and AI agents
  • Implement validation rules, monitoring, and remediation processes that improve the fitness and reliability of critical data assets
  • Experience designing, building, and supporting production-grade Databricks, Delta Lake, and Apache Spark pipelines for batch and streaming workloads
  • Experience developing Databricks Jobs (i.e. Lakeflow Declarative Pipelines, Structured Streaming) and multi-task workflows, including scheduling, parameters, dependencies, retries, notifications, and operational monitoring
  • Experience implementing Databricks infrastructure and environment-aware deployment automation using Terraform, Databricks Asset Bundles, or other Infrastructure as Code practices, Git, and CI/CD workflows
  • Experience implementing data governance and access controls with Unity Catalog, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions
  • Experience using Python and Spark diagnostic tools, including job logs, Spark UI, query plans, partitioning, join optimization, and Adaptive Query Execution, to troubleshoot and optimize production data workloads
  • Must have legal authorization to work permanently in the United States for any employer without requiring a visa transfer or visa sponsorship
  • Experience building production ingestion pipelines with Databricks Auto Loader
  • Advanced Python experience applied to data engineering, automation, reusable libraries, and platform capabilities
  • Experience with open table formats such as Delta Lake, Apache Hudi, or Apache Iceberg
  • Experience developing governed data assets for AI, semantic modeling, APIs, advanced analytics, Genie Spaces, or AI agents
  • Experience with AWS, Microsoft Azure, or Google Cloud and native storage services such as Amazon S3, Azure Data Lake Storage Gen2, OneLake, or Google Cloud Storage

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Koch

Instrument Engineer

Enid, USA

Discovered Today

End User Data Specialist

Green Bay, USA

Discovered 2 days ago

Solutions Architect

Lyons, USA

Discovered 2 days ago

Engineering Lab Technician

Hudson, USA

Discovered 2 days ago

Production Supervisor (1st Shift)

Georgetown, USA

Discovered 2 days ago

Billing Coordinator

Paragould, USA

Discovered 2 days ago

Senior Infrastructure Delivery Engineer

Lyons, USA

Discovered 2 days ago

Operations Manager (Manufacturing)

Georgetown, USA

Discovered 2 days ago