{bc}
Apply
successfactors

Data Engineer Advisor

NTT DATA Services
Bengaluru, IND
Senior · 10+ years experience
Onsite
Discovered 3 days ago
airflowgithadoophiveicebergkubernetes

Full Job Posting

Experience

  • 10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark. ________________________________ Key Responsibilities Migration Framework Engineering
  • Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code.
  • Build reusable Hive metadata/schema discovery and migration-configuration components.
  • Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history.
  • Implement PNC-defined Data Movement Controls including: o checkpointing, o restart/recovery, o idempotency, o retries, o control totals, o audit logging, o exception handling. PB-Scale Migration Execution
  • Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale.
  • Design migration patterns for small, medium, large and very-large tables.
  • Tune parallel Spark execution across tables, partitions and chunks.
  • Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables.
  • Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks.
  • Support migration-wave execution and production issue resolution. Cyber / Tokenization Integration
  • Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines.
  • Implement secure processing of sensitive fields.
  • Handle service failures, retries, throttling and restart scenarios.
  • Measure and optimize tokenization impact on end-to-end migration throughput.
  • Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules. Apache Iceberg Engineering
  • Create/configure Iceberg tables using approved target standards.
  • Implement appropriate partitioning and file-sizing strategies.
  • Optimize compaction and small-file handling.
  • Work with Iceberg snapshots, schema evolution and metadata.
  • Troubleshoot Iceberg write/commit/catalog issues.
  • Support table maintenance and performance optimization. Data Validation & Reconciliation
  • Develop automated source-target validation covering: o row counts, o schema conformity, o partition completeness, o control totals, o aggregates/checksums, o null/duplicate checks, o tokenization validation where applicable.
  • Isolate reconciliation failures at table/partition level.
  • Implement selective retry/reprocessing of failed data.
  • Produce auditable migration evidence. Production Engineering
  • Monitor and troubleshoot distributed Spark jobs.
  • Analyze Spark UI, executor logs and resource utilization.
  • Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues.
  • Build operational telemetry covering migration status, throughput, retries and failures.
  • Participate in code reviews, testing, CI/CD and engineering standards.
  • Mentor mid-level/junior Data Engineers. ________________________________ Mandatory Skills Strong hands-on experience with:
  • Python
  • PySpark
  • Apache Spark / Spark SQL
  • Hadoop / HDFS
  • Hive / HiveQL / Hive Metastore
  • SQL
  • Apache Iceberg
  • Parquet
  • Linux
  • Git
  • distributed data processing
  • large-volume ETL/data migration
  • source-target reconciliation
  • Spark performance tuning. Practical Iceberg experience should include:
  • table creation/read/write,
  • partitioning,
  • snapshots,
  • schema evolution,
  • file sizing,
  • compaction,
  • Spark integration. Strong understanding of:
  • parallelism,
  • checkpoint/restart,
  • idempotency,
  • retry/recovery,
  • large-table processing,
  • small-file problems,
  • distributed-system troubleshooting. ________________________________ Preferred Skills
  • Apache Polaris/Nessie/Glue or another Iceberg catalog.
  • Kubernetes/OpenShift/Spark Operator.
  • S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred.
  • TaaS/Protegrity or equivalent tokenization/masking technology.
  • PostgreSQL for migration configuration/control/audit metadata.
  • Trino.
  • Control-M/Airflow or equivalent enterprise scheduler.
  • CI/CD for Python/Spark applications.
  • Banking/financial-services experience.
  • PB-scale migration experience.
  • HiveQL→Spark SQL modernization experience.
  • Bronze/Silver/Gold lakehouse experience.

Preferred Qualifications

  • distributed-system troubleshooting. ________________________________ Preferred Skills
  • S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at NTT DATA Services

Sales Executive III

New York City, USA

Discovered Yesterday

QA Tester - Hybrid

Rockville, USA

Discovered Yesterday

SAP ABAP Technical

Bengaluru, IND

Discovered Yesterday

Facility Security Officer (FSO)

Herndon, USA

Discovered Yesterday

Corporate Counsel for Commercial contracts - Remote in the US

Plano, USA

Discovered Yesterday

Tech Consultant - Infrastructure Patch & Unix Engineer

Jacksonville, USA

Discovered Yesterday

Unix Engineer - Infrastructure Patch

Jacksonville, USA

Discovered Yesterday

Spring Migration Engineer

Austin, USA

Discovered Yesterday