{bc}
Apply
successfactors

Senior Data Engineer

NTT DATA Services
Bengaluru, IND
Senior · 10+ years experience
Onsite
Discovered 4 days ago
airflowgithadoophiveicebergkubernetes

Full Job Posting

Experience

  • 10+ years of Data Engineering experience, including:
  • 5+ years hands-on Apache Spark/PySpark.
  • Strong Hadoop/Hive experience.
  • Experience leading Data Engineering teams.
  • Large-scale data migration/data modernization experience.
  • Strong data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred.
  • Python
  • PySpark
  • Apache Spark / Spark SQL
  • Hadoop / HDFS
  • Hive / HiveQL / Hive Metastore
  • SQL
  • Apache Iceberg
  • Data modelling
  • Source-to-target mapping
  • Parquet
  • Linux
  • Git
  • Large-volume ETL/data migration
  • Data validation/reconciliation
  • Spark performance tuning
  • logical and physical data modelling,
  • schema/data-type mapping,
  • entity relationships,
  • keys and referential relationships,
  • dimensional/relational modelling concepts,
  • leading Data Engineering teams,
  • technical estimation/planning,
  • detailed solution design,
  • code/design reviews,
  • mentoring engineers,
  • production troubleshooting,
  • root-cause analysis,
  • communicating technical issues with architects/customer teams.
  • Lead a team of Data Engineers through assessment, development, migration, validation and production execution.
  • Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.
  • Own technical estimates, engineering backlog, work allocation and technical delivery.
  • Conduct design and code reviews and enforce engineering standards.
  • Mentor Data Engineers and provide hands-on technical guidance.
  • Act as the primary engineering escalation point for complex migration and production issues.
  • Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.
  • Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions. Hadoop-to-Iceberg Migration
  • Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.
  • Harden and extend the reusable, configuration-driven migration framework developed during the POC.
  • Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.
  • Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.
  • Design execution approaches for small, medium, large and very-large tables.
  • Optimize parallel migration across tables and partitions.
  • Implement checkpointing, restart/recovery, idempotency and safe retries.
  • Lead technical execution and troubleshooting of migration waves. Data Modelling & Schema Engineering
  • Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.
  • Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.
  • Perform source-to-target data mapping and schema/data-type mapping.
  • Define and implement table structures consistent with approved Iceberg standards.
  • Analyze: o entities and relationships, o primary/foreign-key relationships, o data types, o partition keys, o business keys, o reference/cross-reference structures.
  • Identify schema incompatibilities and recommend remediation.
  • Work with the Data Architect on partitioning, file-layout and model-design decisions.
  • Ensure migrated models preserve required business relationships and downstream compatibility.
  • Support future Bronze/Silver/Gold modelling requirements where applicable. Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards. Spark/PySpark Engineering & Performance
  • Develop and review production-grade Python/PySpark components.
  • Tune Spark jobs for PB-scale migration.
  • Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.
  • Optimize table/partition/chunk parallelism.
  • Tune workloads to meet migration-throughput objectives.
  • Identify bottlenecks across: Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation
  • Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework. Apache Iceberg Engineering
  • Implement approved Iceberg table standards.

Key Responsibilities

  • Technical Leadership
  • Lead a team of Data Engineers through assessment, development, migration, validation and production execution.
  • Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.
  • Own technical estimates, engineering backlog, work allocation and technical delivery.
  • Conduct design and code reviews and enforce engineering standards.
  • Mentor Data Engineers and provide hands-on technical guidance.
  • Act as the primary engineering escalation point for complex migration and production issues.
  • Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.
  • Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions. Hadoop-to-Iceberg Migration
  • Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.
  • Harden and extend the reusable, configuration-driven migration framework developed during the POC.
  • Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.
  • Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.
  • Design execution approaches for small, medium, large and very-large tables.
  • Optimize parallel migration across tables and partitions.
  • Implement checkpointing, restart/recovery, idempotency and safe retries.
  • Lead technical execution and troubleshooting of migration waves. Data Modelling & Schema Engineering
  • Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.
  • Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.
  • Perform source-to-target data mapping and schema/data-type mapping.
  • Define and implement table structures consistent with approved Iceberg standards.
  • Analyze: o entities and relationships, o primary/foreign-key relationships, o data types, o partition keys, o business keys, o reference/cross-reference structures.
  • Identify schema incompatibilities and recommend remediation.
  • Work with the Data Architect on partitioning, file-layout and model-design decisions.
  • Ensure migrated models preserve required business relationships and downstream compatibility.
  • Support future Bronze/Silver/Gold modelling requirements where applicable. Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards. Spark/PySpark Engineering & Performance
  • Develop and review production-grade Python/PySpark components.
  • Tune Spark jobs for PB-scale migration.
  • Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.
  • Optimize table/partition/chunk parallelism.
  • Tune workloads to meet migration-throughput objectives.
  • Identify bottlenecks across: Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation
  • Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework. Apache Iceberg Engineering
  • Implement approved Iceberg table standards.
  • Develop/configure Iceberg tables and Spark→Iceberg processing.
  • Implement: o partitioning, o schema evolution, o snapshots, o file sizing, o compaction, o metadata maintenance, o small-file remediation.
  • Troubleshoot Iceberg catalog/write/commit issues.
  • Optimize Iceberg table performance and storage characteristics. Cyber-3 / Tokenization
  • Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.
  • Integrate tokenization into Spark migration pipelines.
  • Implement batching, retry, timeout and recovery handling.
  • Troubleshoot tokenization failures/performance bottlenecks.
  • Measure impact of tokenization on end-to-end migration throughput.
  • Support Cyber/tokenization validation based on PNC-defined rules. Migration Controls & Reconciliation
  • Lead implementation of PNC Data Movement Controls, including: o checkpointing, o idempotency, o control totals, o restart/recovery, o retry, o audit logging, o exception handling.
  • Build reusable source-to-target reconciliation capabilities covering: o row counts, o schema validation, o partition completeness, o control totals, o aggregates, o hash/checksum where appropriate, o null/duplicate validation, o tokenization validation.
  • Isolate discrepancies at table/partition level.
  • Enable selective reprocessing rather than full-table reruns.
  • Produce technical evidence required for migration acceptance.

Mandatory Skills

  • Strong hands-on expertise in:
  • Distributed-system troubleshooting. Data Modelling Strong experience with:
  • large-scale analytical data models. Technical Leadership Demonstrated experience:

Preferred Skills

  • Apache Polaris/Nessie/Glue.
  • OpenShift/Kubernetes/Spark Operator.
  • S3-compatible object storage.
  • NetApp StorageGRID/ONTAP.
  • TaaS/Protegrity or equivalent tokenization technology.
  • Trino.
  • PostgreSQL.
  • Control-M/Airflow.
  • CI/CD for Spark/Python applications.
  • Banking/financial-services experience.
  • PB-scale data migration.
  • Bronze/Silver/Gold modelling and implementation.
  • HiveQL→Spark SQL modernization.
  • Strong data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred.
  • Strong data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred. Key Responsibilities Technical Leadership

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at NTT DATA Services

Sales Executive III

New York City, USA

Discovered 2 days ago

QA Tester - Hybrid

Rockville, USA

Discovered 2 days ago

SAP ABAP Technical

Bengaluru, IND

Discovered 2 days ago

Facility Security Officer (FSO)

Herndon, USA

Discovered 2 days ago

Corporate Counsel for Commercial contracts - Remote in the US

Plano, USA

Discovered 2 days ago

Tech Consultant - Infrastructure Patch & Unix Engineer

Jacksonville, USA

Discovered 2 days ago

Unix Engineer - Infrastructure Patch

Jacksonville, USA

Discovered 2 days ago

Spring Migration Engineer

Austin, USA

Discovered 2 days ago