Data Platform Modernization Engineer - Junior (Databricks)

Role overview

Day-to-day job duties

• Analyze Informatica workflows, mappings, and AWS Glue jobs to understand source-to-target mappings and transformation logic.
• Develop Databricks notebooks, data pipelines, and transformations using Python, PySpark, Spark SQL, and Databricks SQL.
• Implement Bronze, Silver, and Gold processing layers and batch or incremental pipelines using Delta Lake.
• Maintain Databricks Jobs, Workflows, and Lakeflow Declarative Pipelines as required.
• Convert Informatica transformations, mappings, lookups, and business rules into equivalent Databricks implementations.
• Support AWS Glue and database-processing migrations into Databricks.
• Perform unit testing, source-to-target reconciliation, integration testing, regression testing, UAT, and production validation.
• Investigate data, SQL, PySpark, and pipeline execution issues.
• Work with senior developers and technical leads on technical issues and dependencies.
• Follow coding, performance, security, and development standards and participate in code reviews.
• Prepare technical documentation, support knowledge transfer, and assist with deployment, monitoring, stabilization, and cutover.

Basic qualifications

• 3–6 years of experience in data engineering, ETL development, or application/data platform development.
• Hands-on experience with Databricks and Apache Spark.
• Strong development skills in Python or PySpark and SQL.
• Experience with Databricks notebooks, Spark SQL or Databricks SQL, Delta Lake, and Databricks Jobs or Workflows.
• Experience developing batch or incremental ETL/ELT pipelines.
• Experience with Informatica PowerCenter, AWS Glue, SSIS, or a similar ETL technology.
• Experience with relational databases, SQL-based processing, data transformation, source-to-target mapping, unit testing, validation, and troubleshooting.
• Basic understanding of layered data architectures such as Bronze, Silver, and Gold.
• Ability to follow established development frameworks and coding standards.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent experience.

Preferred qualifications

• Databricks certification is a nice-to-have qualification.
• Experience with Lakeflow Declarative Pipelines or Delta Live Tables, Unity Catalog, Databricks Asset Bundles, or CI/CD is beneficial.
• Experience with AWS S3, Glue, Redshift, Lambda, Auto Loader, streaming ingestion, CDC, or SCD processing is beneficial.
• Experience with data quality, audit, error-handling, and Spark or Delta performance optimization is beneficial.
• Experience in cloud data modernization or migration projects is beneficial.

About NTT DATA