Data Engineer - Python & Microservices

Overview

Key Responsibilities

• Design, develop, and maintain scalable backend applications and web frameworks using Python.
• Build robust RESTful APIs and microservices to serve enterprise datasets and business applications.
• Develop and optimize data transformation pipelines to process, clean, enrich, and prepare data for analytics and business consumption.
• Design and implement scalable data access layers integrating multiple relational and NoSQL databases.
• Collaborate with frontend developers to integrate user-facing components with backend services.
• Ensure applications are secure, highly available, performant, and maintainable.
• Design and maintain database schemas and data models aligned with business requirements.
• Implement ETL processes for extracting, transforming, and loading large-scale datasets.
• Optimize database performance, query execution, and data retrieval mechanisms.
• Troubleshoot production issues and implement scalable solutions for long-term platform stability.
• Participate in code reviews, testing activities, technical design discussions, and engineering best practices.
• Work closely with business stakeholders, architects, and engineering teams within Agile delivery environments.

Python Development

• Strong hands-on experience in Core Python development.
• Expertise in building backend services and APIs using Python.
• Strong understanding of object-oriented programming principles and software design patterns.
• Experience writing clean, reusable, maintainable, and production-quality code.

Web Frameworks & Api Development

• Strong experience with one or more Python frameworks including: FastAPI Flask Django Pyramid
• Experience with: RESTful API design and development API versioning and documentation Routing and middleware implementation Authentication and authorization Framework testing and debugging

Microservices Architecture

• Strong experience designing and developing microservices-based applications.
• Deep understanding of distributed systems architecture.
• Experience building scalable and resilient service-oriented platforms.
• Knowledge of service communication patterns and API orchestration.
• Experience handling concurrent requests and high-volume transaction processing.

Asynchronous Programming

• Strong understanding of asyncio and asynchronous programming concepts.
• Experience handling concurrent connections and non-blocking operations.
• Knowledge of performance optimization for I/O-intensive applications.

Security & Api Protection

• Strong understanding of API and microservices security best practices.
• Experience implementing: OAuth 2.0 JWT Authentication API Gateway integrations Encryption and secure communication protocols
• Experience implementing authentication, authorization, and access control mechanisms.

Database & Orm Expertise

• Strong experience with relational and NoSQL databases.
• Experience designing database schemas and optimizing database performance.
• Strong proficiency with: SQL PostgreSQL MySQL MongoDB Redis
• Hands-on experience with ORMs including: SQLAlchemy Django ORM
• Ability to: Define models Perform CRUD operations Write optimized queries Handle transactions Manage database migrations

Data Engineering & Transformation

• Experience building and maintaining ETL pipelines.
• Strong understanding of data transformation, enrichment, aggregation, and cleansing techniques.
• Experience ensuring data quality, consistency, and integrity.
• Ability to support analytical and data-driven applications through scalable processing frameworks.

Devops & Ci And Cd

• Experience building and maintaining CI/CD pipelines.
• Experience deploying and supporting production-grade applications.
• Understanding of infrastructure automation and modern software delivery practices.
• Experience with release management and deployment workflows.

Big Data & Data Technologies

• Exposure to technologies such as: Hadoop Apache Spark Apache Kafka
• Understanding of: Data Warehousing concepts Distributed data processing Enterprise data architectures

Version Control & Development Practices

• Strong experience with Git version control.
• Experience with: Branching strategies Code reviews Pull Requests Merge conflict resolution Collaborative development workflows

Testing & Debugging

• Experience with: Unit Testing Integration Testing Logging Monitoring Performance Troubleshooting Production Support
• Nice to Have
• Experience with AWS, Azure, or GCP.
• Experience with Docker and Kubernetes.
• Exposure to cloud-native architectures.
• Knowledge of data governance, compliance, privacy, and security standards.
• Experience supporting large-scale enterprise data platforms.

Required Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management capabilities.
• Strong collaboration and teamwork skills.
• Ability to work effectively in Agile/Scrum environments.
• Strong ownership mindset with attention to detail.
• Ability to document solutions and communicate technical concepts clearly.