Lead Data Engineer
Role Overview
Lead the data platform team for a large U.S. enterprise client with more than 1,000 locations and analytics and machine learning capabilities being built from the ground up.
Own the lakehouse and MLOps foundation, from governed pipelines and reusable data products to model deployment and production monitoring.
This is a hands-on technical leadership and business consulting role involving code, engineering standards, reviews, mentoring, architecture decisions, and business outcomes.
Key Skills for This Role
Full Job Posting
About Neurogent
Neurogent.ai builds agentic AI solutions for workflow automation and customer experience in banking, healthcare, insurance, and financial services.
Its solutions include AI agents, chatbots, and voice assistants using architectures such as RAG, LangGraph, and agentic AI.
Role overview
Lead the data platform team for a large U.S. enterprise client with more than 1,000 locations and analytics and machine learning capabilities being built from the ground up.
Own the lakehouse and MLOps foundation, from governed pipelines and reusable data products to model deployment and production monitoring.
This is a hands-on technical leadership and business consulting role involving code, engineering standards, reviews, mentoring, architecture decisions, and business outcomes.
Data platform and business consulting
- Set platform architecture, technical direction, engineering standards, delivery priorities, and team direction.
- Build medallion pipelines and integrate operational, CRM, HRIS/payroll, web, finance, and other enterprise data sources.
- Create reliable ETL/ELT pipelines, reusable feature and prediction tables, and multi-year historical data; optimize platform performance and costs.
- Understand business problems, identify process automation and improvement opportunities, challenge assumptions, assess ROI, and develop data-backed business cases.
- Translate objectives into technical requirements, influence stakeholders, and measure business outcomes.
Governance, CI/CD, and data quality
- Separate sandbox, development, and production environments; establish catalog governance, RBAC, secrets management, least-privilege access, lineage, metadata, and security controls.
- Implement CI/CD, Git workflows, automated tests, environment configuration, secure secrets handling, and controlled production promotion.
- Own orchestration for scheduled, event-triggered, and manual workflows, including visibility, retries, recovery, and SLAs.
- Implement data-quality checks and alerts for schema changes, null spikes, unexpected ranges, duplicates, and other quality issues; make failures actionable.
MLOps, analytics, and leadership
- Build MLOps support for data science teams, including experiment tracking, model registry and versioning, deployment, rollback, monitoring, drift detection, and reproducible workflows.
- Build batch inference and support real-time or low-latency inference as needed; productionize data scientist models.
- Deliver trusted data to Power BI or equivalent tools, support downstream APIs and applications, and work with teams to meet business needs.
- Mentor engineers, conduct code and architecture reviews, establish engineering patterns, document architectures and runbooks, and proactively resolve risks.
Required experience
- At least six years in data engineering, data platforms, or related roles, including team leadership or end-to-end platform ownership.
- Hands-on experience with a modern lakehouse or cloud data platform such as Databricks, Snowflake, Microsoft Fabric, BigQuery, or equivalent.
- Strong Python and SQL skills; production ETL/ELT experience with incremental loads, schema evolution, backfills, data modeling, and reconciliation.
- Experience with Spark or another distributed processing engine, CI/CD and testing for data and ML, orchestration, and data-quality tooling.
- Working knowledge of a major cloud platform; Azure is preferred, and AWS or GCP are acceptable.
- Experience with infrastructure as code, cloud security, and compute, performance, and cost optimization.
- Experience working directly with senior client stakeholders and presenting, documenting, and defending architecture recommendations.
- Business consulting and operations experience, including process improvement, business cases, stakeholder influence, and measurable outcomes.
- Ability to translate business objectives into architecture and delivery outcomes; strong written and verbal communication; ability to work with remote, cross-functional, U.S.-based teams.
- Comfort using AI-assisted engineering tools such as Claude Code, Cursor, GitHub Copilot, or similar.
Nice to have
- Founder, co-founder, founding engineer, or early-stage engineering experience.
- Databricks experience, including Delta Lake, Databricks Workflows, Unity Catalog, or MLflow; platform migration experience.
- ML/AI experience in feature engineering, model training, deployment, monitoring, retraining, or production use cases such as churn, propensity, forecasting, and recommendations.
- Streaming, real-time processing, low-latency model serving, R-based data science, Power BI, or equivalent BI experience.
- Data, AI, or consulting project delivery for U.S. clients, or services, consulting, or client-facing engineering experience.
- Relevant Databricks, Snowflake, Azure Data Engineer, AWS, or GCP certifications.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career