.NET Data Engineer

Responsibilities

• Design repeatable processes to import data from CSV files and external APIs into SQL Server tables, then transform and cleanse it into production tables.
• Write and maintain SQL stored procedures (SPs) for data migration and transformation, using parameters and variables so the same procedure can be reused across data sources, environments, or business units (LOBs).
• Ensure stored procedures and.NET import tools are safely re-runnable (idempotent) handling duplicates, missing data, and partial failures without corrupting data.
• Build and maintain.NET services that integrate with third-party APIs (e.g., Salesforce) for data enrichment and synchronization.
• Handle authentication flows (OAuth client credentials, Bearer tokens) and design request/response models that map cleanly to C# classes.
• Design and build.NET / C# applications — Azure Functions (isolated worker model) and console utilities — that call external APIs, process data in bulk, and write results to SQL Server.
• Use dependency injection, configuration binding (appsettings.json, User Secrets, Azure App Configuration, Key Vault), and structured logging as standard patterns across all new services.
• Implement asynchronous, concurrent processing (async/await, SemaphoreSlim-based bounded concurrency) to handle bulk data volumes efficiently without overwhelming downstream APIs.
• Build resilient HTTP client code: authentication (OAuth/Bearer tokens), retry logic with backoff for transient failures, and proper error handling so a single bad record does not break an entire batch run.
• Work with Entity Framework Core and LINQ queries for reading and writing application data in SQL Server.
• Write and maintain unit and integration tests for data-processing logic.
• Strong experience in React.js, JavaScript/TypeScript, and front-end best practices, with the ability to build responsive, scalable, and high-performance user interfaces.
• Develop and maintain web applications using React.js; build reusable UI components and front-end libraries for future use.
• Collaborate with backend developers to deliver high-quality, end-to-end solutions.
• Validate data accuracy after each import/migration run (row counts, key-field checks, spot-checks against source data).
• Use Git for version control (branching, pull requests) and Azure DevOps YAML pipelines for CI/CD (build, test, multi-environment deployment).
• Document data flows, mapping logic, and known data quality issues for the rest of the team.
• Build and maintain KPI queries/views and reporting datasets to support business and operational dashboards.
• Work with stakeholders to translate reporting requirements into reliable, well-documented SQL views or stored procedures.

Requirements

• Data Import, ETL & SQL
• Design repeatable processes to import data from CSV files and external APIs into SQL Server tables, then transform and cleanse it into production tables.
• Write and maintain SQL stored procedures (SPs) for data migration and transformation, using parameters and variables so the same procedure can be reused across data sources, environments, or business units (LOBs).
• Ensure stored procedures and.NET import tools are safely re-runnable (idempotent) handling duplicates, missing data, and partial failures without corrupting data.
• API & Third-Party Integrations
• Build and maintain.NET services that integrate with third-party APIs (e.g., Salesforce) for data enrichment and synchronization.
• Handle authentication flows (OAuth client credentials, Bearer tokens) and design request/response models that map cleanly to C# classes.
• .NET Application Development
• Design and build.NET / C# applications — Azure Functions (isolated worker model) and console utilities — that call external APIs, process data in bulk, and write results to SQL Server.
• Use dependency injection, configuration binding (appsettings.json, User Secrets, Azure App Configuration, Key Vault), and structured logging as standard patterns across all new services.
• Implement asynchronous, concurrent processing (async/await, SemaphoreSlim-based bounded concurrency) to handle bulk data volumes efficiently without overwhelming downstream APIs.
• Build resilient HTTP client code: authentication (OAuth/Bearer tokens), retry logic with backoff for transient failures, and proper error handling so a single bad record does not break an entire batch run.
• Work with Entity Framework Core and LINQ queries for reading and writing application data in SQL Server.
• Write and maintain unit and integration tests for data-processing logic.
• React – Front End
• Strong experience in React.js, JavaScript/TypeScript, and front-end best practices, with the ability to build responsive, scalable, and high-performance user interfaces.
• Develop and maintain web applications using React.js; build reusable UI components and front-end libraries for future use.
• Collaborate with backend developers to deliver high-quality, end-to-end solutions.
• Data Quality, DevOps & Documentation
• Validate data accuracy after each import/migration run (row counts, key-field checks, spot-checks against source data).
• Use Git for version control (branching, pull requests) and Azure DevOps YAML pipelines for CI/CD (build, test, multi-environment deployment).
• Document data flows, mapping logic, and known data quality issues for the rest of the team.
• Reporting & KPIs
• Build and maintain KPI queries/views and reporting datasets to support business and operational dashboards.
• Work with stakeholders to translate reporting requirements into reliable, well-documented SQL views or stored procedures.

Full Job Posting