Data Platform Modernization Engineer - Junior (Databricks)

Role Overview

Day-to-Day Job Duties

• Analyze Informatica workflows, mappings, and AWS Glue jobs to understand source-to-target mappings and transformation logic.
• Develop Databricks notebooks and data pipelines to migrate existing ETL workloads to the Databricks Lakehouse.
• Create data transformations using Python, PySpark, Spark SQL, and Databricks SQL.
• Implement Bronze, Silver, and Gold layer processing using established project architecture and coding patterns.
• Build batch and incremental pipelines using Delta Lake and watermark or control mechanisms.
• Develop and maintain Databricks Jobs, Workflows, and Lakeflow Declarative Pipelines as required.
• Use project frameworks for logging, auditing, data quality, error handling, and notifications.
• Convert Informatica transformations, mappings, lookups, and business rules into equivalent Databricks implementations.
• Support migration of AWS Glue jobs and database-based processing into Databricks.
• Perform testing, reconciliation, validation, deployment, monitoring, stabilization, and cutover activities.

Collaboration and Delivery

• Investigate and resolve data, SQL, PySpark, and pipeline execution issues.
• Support integration testing, regression testing, user acceptance testing, and production validation.
• Work with senior developers and technical leads to resolve technical issues and dependencies.
• Follow coding, performance, security, and development standards and participate in code reviews.
• Prepare technical documentation and support knowledge transfer for migrated workloads.

Basic Qualifications

• 3–6 years of experience in data engineering, ETL development, or application and data platform development.
• Hands-on experience with Databricks and Apache Spark.
• Strong development skills in Python or PySpark and SQL.
• Experience with Databricks notebooks, Spark SQL or Databricks SQL, Delta Lake, and Databricks Jobs or Workflows.
• Experience developing batch or incremental ETL/ELT pipelines.
• Experience with one or more ETL technologies such as Informatica PowerCenter, AWS Glue, SSIS, or similar.
• Experience with relational databases, SQL-based processing, data validation, troubleshooting, and unit testing.
• Basic understanding of layered data architectures and ability to follow established development frameworks and coding standards.
• Bachelor’s degree in Computer Science, Information Technology, Engineering, or equivalent experience.

Nice to Have

• Databricks certification.
• Experience with Lakeflow Declarative Pipelines or Delta Live Tables and Unity Catalog.
• Experience with Databricks Asset Bundles and CI/CD.
• Experience with AWS S3, Glue, Lambda, or Redshift.
• Experience with Auto Loader, streaming ingestion, CDC, or SCD Type 1 and Type 2 processing.
• Experience with data quality, audit, error-handling, performance optimization, or cloud data modernization projects.

About NTT DATA