Data Engineer Lead

About Capital Group

Role Overview

What You Will Do

• Set the data engineering strategy and roadmap for CSGT, including the Lakehouse architecture on Databricks and AWS.
• Design and build ingestion, transformation, and serving pipelines with Databricks, PySpark, Delta Lake, dbt, and Airflow, and create reusable patterns and frameworks.
• Analyze new structured and unstructured datasets at the business-capability level and fit them into the platform's data domains and subject areas.
• Own delivery of complex, cross-team data initiatives from requirements through production support, including estimates, work breakdown, sequencing, dependencies, and cost.
• Lead the continued evolution of AI-first engineering: turn business outcomes into well-defined specifications and direct agents to plan, build, test, and document changes.
• Build reusable agent workflows, skills, and tool integrations for data engineering work such as profiling, source-to-target mapping, pipeline and test generation, schema-change analysis, and incident investigation.
• Make data understandable and reliable for AI through curated Unity Catalog metadata, lineage, business definitions, semantic models, and access controls.
• Define the testing strategy across all layers of the platform, including performance, stability, and availability, and review and approve quality metrics before release.
• Partner with investment professionals and product managers to develop a shared product vision and ownership of business outcomes.
• Raise the engineering bar: lead design and code reviews, direct the day-to-day work of engineers on your initiatives, and guide them through complex data and performance issues.

Required Qualifications

• 10+ years of experience in data or software engineering, including technical leadership of complex production data platforms delivered across multiple teams.
• Strong hands-on Python and SQL skills, sound software design judgment, and a deep understanding of distributed data processing, query performance, and automated testing.
• Production experience with Databricks on AWS, including PySpark, Delta Lake, Unity Catalog, Databricks Jobs, Databricks SQL, and Databricks Asset Bundles.
• Orchestrated production pipelines with Apache Airflow (including Astronomer) and built tested transformations with dbt.
• Strong data modeling and governance experience, including dimensional and time-series models, slowly changing dimensions, bi-temporal history, data contracts, lineage, and semantic metadata.
• Implemented data quality, observability, and CI/CD for data platforms using tools such as Deequ, dbt tests, Lakehouse Monitoring, Datadog, Terraform, and Harness.
• Experience preparing governed data for AI through natural-language-to-SQL tools such as Databricks Genie, semantic metadata, or other governed data-access patterns.
• Use AI coding agents well beyond code completion: writing specifications, supplying context, running tests, and reviewing generated changes through source control.
• Evaluate AI-generated output with representative test cases, regression tests, execution traces, and human review.
• Understand prompt injection, sensitive data handling, and least-privilege access, and can design approval boundaries and audit trails for agents working against enterprise systems.
• Lead architecture discussions, influence without formal authority, develop other engineers, and explain technical choices and trade-offs clearly.
• Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.

Preferred Qualifications

• Experience with investment management data such as portfolios, positions, returns, exposures, benchmarks, and attribution, or with multi-asset portfolio construction.
• Experience building LLM applications or agent workflows that call tools and APIs, including context management, retrieval, state, and error handling.
• Familiarity with Model Context Protocol (MCP).
• Experience with PostgreSQL, SQL Server, or Lakebase, or with modernizing legacy data platforms onto a Lakehouse.

Compensation

• Southern California Base Salary Range: $201,683-$322,693.
• New York Base Salary Range: $213,795-$342,072.
• Eligible for an individual annual performance bonus, plus Capital's annual profitability bonus plus a retirement plan where Capital contributes 15% of eligible earnings.