Data Engineering Operations Lead

Team and role overview

Core responsibilities

• Own daily data engineering operations, including ingestion workflows, pipeline monitoring, and incident resolution.
• Monitor pipeline health; triage and resolve failures, bottlenecks, and data quality issues across ingestion layers.
• Organize the daily workload, assign tasks, track progress, and remove blockers.
• Maintain and improve pipelines moving data into lakes, warehouses, and real-time streaming systems.
• Lead escalation, root-cause analysis, and permanent fixes for pipeline failures, SLA breaches, and data anomalies.
• Set operational standards for runbooks, alerting, on-call procedures, and incident management.
• Coordinate dependencies and pipeline status with upstream providers and downstream consumers.
• Lead daily stand-ups; mentor junior engineers; collaborate with data scientists, analysts, and product partners.
• Improve reliability, observability, efficiency, governance, and data quality monitoring.
• Design, build, test, and deploy scalable data solutions; lead technical delivery and oversee work to completion.

Experience and education

• Bachelor’s degree in Computer Science, Engineering, Statistics, or a related field.
• 10 years of data engineering experience, including at least 3 years in lead roles.
• 6 or more years of big data technology experience, such as Spark, Hive, Hadoop, or Databricks.
• Proven experience leading, mentoring, and supporting junior team members; ability to lead technical work autonomously and manage workload and delivery.

Essential technical skills

• Advanced Apache Spark (PySpark), including tuning and performance optimization; Python and Pandas proficiency; Scala or Java knowledge is desirable.
• Working knowledge of Apache Hive and strong SQL skills, including T-SQL, SQL Server, SSMS, and GCP BigQuery.
• GCP pipeline and ETL expertise: BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer, Cloud Functions, and Dataproc (Spark).
• Batch and real-time streaming, ETL, incident resolution, pipeline recovery, and Apache Airflow workflow development.
• Git source control; CI/CD concepts and design for data pipelines; Delta Lake, common data formats, and Lakehouse architecture.
• Software engineering principles including OOP, design patterns, SDLC, Agile, TDD, and performance optimization.

Desirable and preferred skills

• Desirable: logical and physical data modeling, data warehouse and data mart design, and relevant certifications such as Google Cloud Professional Data Engineer.
• Kafka, R and Sparklyr, and Harness experience are pluses.
• Knowledge of MLOps, AI/ML lifecycle management, and MLflow is desirable.
• AI engineering fundamentals and GCP certifications are listed as preferred development skills.

About TransUnion