Data Engineer - Python & Microservices

Overview

Key Responsibilities

• Design, develop, and maintain scalable backend applications and web frameworks using Python.
• Build robust RESTful APIs and microservices to serve enterprise datasets and business applications.
• Develop and optimize data transformation pipelines to process, clean, enrich, and prepare data for analytics and business consumption.
• Design and implement scalable data access layers integrating multiple relational and NoSQL databases.
• Collaborate with frontend developers to integrate user-facing components with backend services.
• Ensure applications are secure, highly available, performant, and maintainable.
• Design and maintain database schemas and data models aligned with business requirements.
• Implement ETL processes for extracting, transforming, and loading large-scale datasets.
• Optimize database performance, query execution, and data retrieval mechanisms.
• Troubleshoot production issues and implement scalable solutions for long-term platform stability.
• Participate in code reviews, testing activities, technical design discussions, and engineering best practices.
• Work closely with business stakeholders, architects, and engineering teams within Agile delivery environments.

Python Development

• Strong hands-on experience in Core Python development.
• Expertise in building backend services and APIs using Python.
• Strong understanding of object-oriented programming principles and software design patterns.
• Experience writing clean, reusable, maintainable, and production-quality code.

Web Frameworks & API Development

• Strong experience with one or more Python frameworks including:
• FastAPI
• Flask
• Django
• Pyramid

Experience

• with:
• RESTful API design and development
• API versioning and documentation
• Routing and middleware implementation
• Authentication and authorization
• Framework testing and debugging

Microservices Architecture

• Strong experience designing and developing microservices-based applications.
• Deep understanding of distributed systems architecture.
• Experience building scalable and resilient service-oriented platforms.
• Knowledge of service communication patterns and API orchestration.
• Experience handling concurrent requests and high-volume transaction processing.

Asynchronous Programming

• Strong understanding of asyncio and asynchronous programming concepts.
• Experience handling concurrent connections and non-blocking operations.
• Knowledge of performance optimization for I/O-intensive applications.

Security & API Protection

• Strong understanding of API and microservices security best practices.
• Experience implementing:
• OAuth 2.0
• JWT Authentication
• API Gateway integrations
• Encryption and secure communication protocols
• + Experience implementing authentication, authorization, and access control mechanisms.

Database & ORM Expertise

• Strong experience with relational and NoSQL databases.
• Experience designing database schemas and optimizing database performance.
• Strong proficiency with:
• SQL
• PostgreSQL
• MySQL
• MongoDB
• Redis
• Hands-on experience with ORMs including:
• SQLAlchemy
• Django ORM
• Ability to:
• Define models
• Perform CRUD operations
• Write optimized queries
• Handle transactions
• Manage database migrations

Data Engineering & Transformation

• Experience building and maintaining ETL pipelines.
• Strong understanding of data transformation, enrichment, aggregation, and cleansing techniques.
• Experience ensuring data quality, consistency, and integrity.
• Ability to support analytical and data-driven applications through scalable processing frameworks.

DevOps & CI/CD

• Experience building and maintaining CI/CD pipelines.
• Experience deploying and supporting production-grade applications.
• Understanding of infrastructure automation and modern software delivery practices.
• Experience with release management and deployment workflows.

Big Data & Data Technologies

• Exposure to technologies such as:
• Hadoop
• Apache Spark
• Apache Kafka
• Understanding of:
• Data Warehousing concepts
• Distributed data processing
• Enterprise data architectures

Version Control & Development Practices

• Strong experience with Git version control.
• Experience with:
• Branching strategies
• Code reviews
• Pull Requests
• Merge conflict resolution
• Collaborative development workflows

Testing & Debugging

• Experience with:
• Unit Testing
• Integration Testing
• Logging
• Monitoring
• Performance Troubleshooting
• Production Support
• Nice to Have
• Experience with AWS, Azure, or GCP.
• Experience with Docker and Kubernetes.
• Exposure to cloud-native architectures.
• Knowledge of data governance, compliance, privacy, and security standards.
• Experience supporting large-scale enterprise data platforms.

Required Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management capabilities.
• Strong collaboration and teamwork skills.
• Ability to work effectively in Agile/Scrum environments.
• Strong ownership mindset with attention to detail.
• Ability to document solutions and communicate technical concepts clearly.

Education