Data Engineer

The role

What you'll do

What you'll bring

Nice to have

• Familiarity with the Databricks ecosystem.
• Experience with streaming pipelines and logical replication, such as replicating Postgres into Databricks.
• Experience building RAG pipelines.
• AWS infrastructure experience.
• Platform administration experience: roles, permissions, access controls, and visibility management.
• Basic programming skills in any language (Python is a plus).
• Location
• San Francisco. We're an in-person team - 5 days in the office. At this stage, proximity genuinely accelerates product quality and team cohesion.

Location

• San Francisco. We're an in-person team - 5 days in the office. At this stage, proximity genuinely accelerates product quality and team cohesion.

Qualifications

• Hands-on experience building, deploying, and maintaining production Spark pipelines.
• Deep understanding of how concepts like clustering, shuffles, different join strategies, and batch versus stream processing create tradeoffs in performance, freshness, and cost.
• Experience with reliability, monitoring, and observability, and the ability to explain how you diagnosed a real pipeline failure or bottleneck.
• Strong SQL.
• Comfort working as an individual contributor, directly with product teams, and through ambiguity.

Responsibilities

• Build, deploy, and operate production Spark data pipelines on Databricks.
• Partner with product teams to turn analytics-product needs into pipeline requirements, including data quality and freshness expectations.
• Set up monitoring and observability so pipeline reliability is measured, not assumed.
• Diagnose performance and cost problems and fix their root causes.
• Recommend pragmatic standards for running pipelines as our data and platform scale.