Senior Software Engineer (Python)

Overview

Accountabilities

• In this role, you will be responsible for designing and maintaining large-scale data ingestion and processing systems that support search and classification capabilities.
• Design and build distributed web crawling and data extraction systems capable of operating at scale in complex environments.
• Develop robust data pipelines to extract, process, and normalize data from web pages, APIs, PDFs, and other document formats.
• Build and maintain systems for unifying heterogeneous data into structured, consistent schemas for downstream use.
• Implement preprocessing and transformation logic to support ML/NLP models, classification systems, and search indexing.
• Develop APIs and services that expose structured data through ElasticSearch/OpenSearch.
• Collaborate with ML and data science teams to integrate classification models into production pipelines.
• Automate workflows using tools such as Apache Airflow and deploy scalable systems using Kubernetes and AWS.
• Optimize and scale data processing pipelines using distributed computing frameworks such as Spark (EMR).

Requirements

• This role requires strong backend engineering expertise, with deep experience in Python and large-scale data systems.
• 5+ years of professional experience in Python development, particularly in web scraping and data pipeline systems at scale.
• Strong experience working with REST APIs and processing structured and unstructured data formats (including PDFs and OCR tools like Tesseract or PyMuPDF).
• Solid understanding of search and data technologies such as ElasticSearch/OpenSearch and relational or NoSQL databases.
• Hands-on experience with distributed processing frameworks such as Apache Airflow and Spark (EMR or equivalent).
• Strong problem-solving skills, especially in handling anti-scraping mechanisms, scaling challenges, and data complexity.
• Experience working in cloud environments such as AWS or GCP.
• Good understanding of system design principles for scalable and resilient backend systems.
• Familiarity with Kubernetes and containerized deployments is a plus.
• Exposure to ML/NLP concepts, LLMs, or frameworks such as spaCy, Hugging Face, TensorFlow, or PyTorch is an advantage.

Benefits

• Competitive salary aligned with senior-level engineering expertise.
• Fully remote flexibility with distributed team collaboration.
• Comprehensive health coverage including medical, dental, vision, and prescription plans.
• Paid time off, parental leave, and medical leave for employees and family care.
• Retirement savings plan with employer matching contributions.
• Flexible spending accounts (FSA) and health savings accounts (HSA).
• Wellness support, including mental health resources and monthly wellness allowances.
• Home office setup support after one year of employment.
• Continuous learning and professional development support.
• Inclusive, collaborative work culture with strong engineering ownership and impact.

How Jobgether Works

Data Privacy Notice