Data Platform Modernization Engineer - Junior (Databricks)

Role Overview

Day-to-Day Duties

• Analyze Informatica workflows, mappings, and AWS Glue jobs to understand source-to-target mappings and transformation logic.
• Develop Databricks notebooks and data pipelines to migrate existing ETL workloads to the Databricks Lakehouse.
• Develop transformations using Python, PySpark, Spark SQL, and Databricks SQL.
• Implement Bronze, Silver, and Gold layer processing using established architecture and coding patterns.
• Develop batch and incremental pipelines using Delta Lake and watermark or control mechanisms.
• Develop and maintain Databricks Jobs, Workflows, and Lakeflow Declarative Pipelines as required.
• Reuse frameworks for logging, auditing, data quality, error handling, and notifications.
• Convert Informatica transformations, mappings, lookups, and business rules into Databricks implementations.
• Support migration of AWS Glue jobs and database-based processing into Databricks.
• Perform unit testing, source-to-target reconciliation, integration testing, regression testing, UAT, and production validation.
• Investigate data, SQL, PySpark, and pipeline execution issues.
• Work with senior developers and technical leads to resolve issues and dependencies.

Engineering Practices

• Follow established coding, performance, security, and development standards.
• Participate in code reviews and incorporate review feedback.
• Prepare technical documentation and support knowledge transfer for migrated workloads.
• Support deployment, monitoring, production stabilization, and cutover activities.

Basic Qualifications

• 3–6 years of experience in data engineering, ETL development, or application/data platform development.
• Hands-on experience with Databricks and Apache Spark.
• Strong development skills in Python, PySpark, and SQL.
• Experience with Databricks notebooks, Spark SQL or Databricks SQL, Delta Lake, and Databricks Jobs or Workflows.
• Experience developing batch or incremental ETL/ELT pipelines.
• Experience with one or more ETL technologies such as Informatica PowerCenter, AWS Glue, SSIS, or similar.
• Experience with relational databases and SQL-based data processing.
• Understanding of data transformation, source-to-target mapping, and ETL development.
• Experience with unit testing, data validation, and troubleshooting.
• Basic understanding of Bronze, Silver, Gold, or other layered data architectures.
• Ability to follow established development frameworks and coding standards.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent experience.

Preferred Skills

• Databricks certification.
• Lakeflow Declarative Pipelines or Delta Live Tables.
• Unity Catalog.
• Databricks Asset Bundles and CI/CD.
• AWS S3, Glue, Lambda, and Redshift.
• Auto Loader or streaming ingestion.
• CDC and SCD Type 1 or Type 2 processing.
• Data quality, audit, and error-handling frameworks.
• Spark or Delta performance optimization.
• Experience with cloud data modernization or migration projects.

About NTT DATA