Lead Data Engineer

About Two Circles

Role Overview

Key Responsibilities

• Provide technical leadership for the Data Engineering team, helping establish technical direction, engineering standards, and best practices while remaining hands-on with the platform.
• Mentor and support engineers in their technical development, providing guidance on system design, troubleshooting, engineering practices, and complex technical challenges.
• Work as part of a cross-functional product team to deliver a high-quality SaaS based data platform.
• Lead and facilitate technical design and architecture reviews, ensuring major technical decisions consider scalability, reliability, maintainability, security, cost, and operational impact.
• Design and implement data pipelines with a clear focus on data quality and reliability.
• Partner with Product Management and stakeholders, as well as technical and non-technical team members, to deliver the vision, roadmap, and data strategy in addition to innovative client solutions.
• Function as an advocate for development best practices including technical design reviews, implementing test plans, test driven development, monitoring/alerting, peer code reviews, and documentation.
• Contribute to the DevOps culture and participate in ownership of designs through production operations.
• Support the ongoing maintenance and operations of the data platform.

Ideal Tech Stack

• .Net / C#
• Python
• AWS or similar
• Snowflake
• SQL
• Airflow

Requirements - Experience

• 8+ years of software engineering experience with a strong data focus, including designing, building, and operating production data systems in fast-paced environments.
• Demonstrated experience providing technical leadership within an engineering team, including guiding technical direction, leading design discussions, and supporting engineers through complex technical decisions.
• Experience designing and evolving data architectures and large-scale data pipelines, including making and communicating architectural decisions and trade-offs.
• Experience mentoring engineers and helping raise engineering standards and technical capabilities across a team.
• Experience leading technical initiatives that span multiple engineers, systems, or teams, from technical discovery and design through implementation and production operation.
• Strong experience writing, debugging, optimizing, and reviewing database queries and data-processing workloads.
• Strong experience with object-oriented programming and software engineering practices, preferably using Python and C#/.NET.
• Strong experience building and operating cloud infrastructure and services, preferably AWS.
• Experience with modern data platforms and orchestration technologies such as Snowflake and Airflow.
• Experience establishing or improving engineering practices such as automated testing, CI/CD, observability, monitoring and alerting, code reviews, documentation, and production readiness.
• Experience identifying opportunities for automation and implementing solutions that reduce repetitive operational work, improve developer experience, and increase platform reliability.
• Experience collaborating with Product Management and technical and non-technical stakeholders and communicating complex technical topics, risks, and trade-offs clearly.

Technical Skills

• Strong experience designing, building, and operating data pipelines using orchestration and batch-processing technologies such as Apache Airflow.
• Strong proficiency with Snowflake and MS SQL Server, including data modeling, query optimization, performance troubleshooting, and production operations.
• Proficiency in Python and C#/.NET. Both are required to be successful in this role.
• Strong experience designing and integrating with APIs, including considerations such as scalability, rate limiting, retries, idempotency, error handling, and observability.
• Strong experience with containerization technologies such as Docker and deploying containerized workloads in cloud environments.
• Experience with monitoring and observability tools such as New Relic, including defining appropriate metrics, alerts, and operational monitoring for data pipelines and services.
• Experience designing effective automated testing strategies, including unit, integration, and data-quality testing, using frameworks such as PyTest.
• Strong understanding of batch, event-driven, and streaming architectures, with the ability to evaluate their trade-offs and select appropriate patterns based on system and business requirements.
• Strong understanding of distributed systems concepts relevant to data platforms, including concurrency, messaging, failure handling, retries, idempotency, consistency, and scalability.
• Ability to evaluate existing architectures, identify technical risks and limitations, and propose pragmatic improvements or modernization strategies.
• Ability to define and communicate technical patterns and engineering standards that can be consistently applied across the team.
• Ability to communicate architecture and complex technical decisions clearly through design documents, diagrams, technical discussions, and reviews.