Data Engineer — ML Training Data Pipeline
Key Skills for This Role
Full Job Posting
What We Expect
Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets
Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch)
Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format)
Implement smart deduplication and sampling to balance training distribution
Design identity-aware train/test splits that measure true generalization
Build data validation gates to detect schema drift and format anomalies
Create a continuous pipeline that auto-processes new production traces for retraining
Requirements
- Experience: 6+ years data engineering focused on ML data pipelines
- Python: Strong — pandas, pyarrow, JSONL processing at scale
- ML Data Libraries: HuggingFace Datasets, Arrow-based storage
- Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting
- Deduplication: Content hashing, identity-based grouping strategies
- AWS: S3, EC2, batch processing workflows
- Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright.
Benefits
- Comprehensive Medical Coverage: Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
- Robust Protection Plans: Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
- Retirement Benefits: PF and Gratuity provided as per standard government regulations.
- Flexible Work Options: Enjoy hybrid work arrangements & flexible working hours.
- Generous Leave Policy: 21 days of annual leave, in addition to 10 company-declared holidays.
- Employee Well-being Spaces: Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
About DATAECONOMY
Cloud-first data and AI consultancy helping enterprises modernize data platforms, build AI solutions, and improve business decisions.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at DATAECONOMY
Business Analyst (Investment banking)
, IND
DATAECONOMY is seeking a Business Analyst to deliver data-driven solutions across investment banking and wealth management platforms. The role requires banking-domain business analysis, hands-on SQL, investment-products
Java Full Stack Developer
Hyderabad, IND
DATAECONOMY is seeking an experienced Java Full Stack Developer to build and maintain enterprise applications across backend, frontend, cloud, and deployment layers. The role requires strong Java, Spring Boot, Spring Sec
Business Analyst (Investment banking)
, IND
PLSQL Developer
Hyderabad, IND
Business Analyst (Investment banking)
Pune, IND
Tableau Server Admin
Pune, IND
FDE - Agentic Ai
Hyderabad, IND
ML / Fine-Tuning Engineer
Hyderabad, IND
Java Full Stack Developer
Hyderabad, IND