Senior Data Engineer

Role Overview

Data Engineering and Integration

• Build and productionize modular, scalable ELT/ETL pipelines and data infrastructure using organizational data sources.
• Implement ingestion and curation pipelines that provide integrated, business-centric data for BI, reporting, and downstream systems.
• Clean, prepare, and optimize datasets while applying lineage and quality controls throughout data integration.
• Develop performant data pipelines and models at scale using Python, Spark, and SQL, consuming sources such as XML, CSV, JSON, and REST APIs.
• Orchestrate pipeline execution to meet latency expectations, manage dependencies, keep datasets current, and minimize disruption.

Security, Analytics, and Operations

• Coordinate with data architecture, infrastructure, and cyber teams to secure data in transit and at rest.
• Support BI analysts with dimensional data models and aggregation optimization for visualization and reporting.
• Provide production support for ingestion, transformation, pipeline performance, data accuracy, and integrity issues.
• Monitor production solutions, troubleshoot issues, and provide Tier 2 support for team-produced datasets.
• Implement role-based access to data products and automate routine work to reduce toil.

Delivery and Governance

• Collaborate with data architects, analysts, scientists, engineers, modelers, and solution architects on pipelines feeding the data marketplace.
• Contribute to feature and user-story backlog development within Agile Scrum and Kanban work management.
• Build the data catalog and maintain metadata for enterprise datasets.
• Document built pipelines and data products, use source control, and participate in peer code review.
• Use CI/CD and DevOps pipelines to automate infrastructure and code delivery, release management, and versioning.
• Write and perform automated unit and regression tests and assist with user acceptance and system integration testing.

Technology Stack

• Microsoft tools include Azure Data Factory, Azure Data Lake, Azure SQL databases, Azure Data Warehouse, Azure Synapse Analytics, Azure Databricks, Collibra, and Power BI.
• The role uses Python, PySpark, SparkSQL, SQL, and data sources including XML, CSV, JSON, and REST APIs.

Qualifications

• Four-year university education in computer science, computer/software engineering, or another relevant program in data engineering, data analysis, AI, or machine learning.
• More than four years of experience building data pipelines and data-processing frameworks using Python, PySpark, SparkSQL, and SQL.
• Experience as a Data Engineer building pipelines and with Azure Data Factory, ADLS, Synapse Analytics, and Databricks.
• Experience building pipelines for data lakehouses and data warehouses.
• Understanding of data structures, data-processing frameworks, data governance, and data quality principles.
• Effective communication skills for translating technical details to non-technical stakeholders.
• Commitment to integrity, corporate values, and the OPG Code of Business Conduct.

Salary Information

• The posting lists a salary range of $2,751.92–$3,227.88 but does not state the compensation cadence.

Schedule and Travel

• Day shift.
• Travel required: 10%.

About OPG