Data Engineer (m/f/d)

Role Overview

Key Responsibilities

• On-vehicle data recording pipeline Design and manage high-throughput recording systems, including topic selection, multi-GB/s write pipelines, and efficient data formats (MCAP/rosbag2). Oversee on-platform storage and ensure reliable data transfer to cloud environments with integrity checks. Ensure timestamp accuracy and synchronization across recorded data.
• Data lake architecture Design and maintain scalable storage solutions across S3, FSx/Lustre, and GCS. Define data organization, regional placement, caching strategies, retention policies, data lineage, and cost optimization.
• Dataset pipeline development Build pipelines that convert raw sensor data into structured, training-ready datasets. Ensure accurate time alignment across modalities, including ego-pose, calibration metadata, and scenario tagging.
• Versioning and dataset management Implement robust dataset versioning and discovery processes. Evaluate and deploy tools such as DVC, LakeFS, Deep Lake, and FiftyOne, ensuring datasets are reproducible, traceable, and easily accessible.
• Dataset format design Contribute to defining efficient on-disk dataset formats, focusing on write performance and optimized I/O for large-scale training workloads.
• Annotation workflows Develop and manage annotation pipelines, including defining vendor handoff formats, ingesting labeled data, performing quality control, handling schema evolution, and supporting iterative dataset improvements.

Required Experience

• 5+ years of experience building production-grade data infrastructure, ideally involving large-scale multimodal or sensor data (e.g., robotics, autonomous systems, geospatial, or scientific domains)
• Strong proficiency in Python, with the ability to work with C++ for ROS2 and pipeline-related tooling
• Hands-on experience with cloud storage and distributed systems (S3, GCS, FSx, Lustre), including performance and cost optimization
• Experience with dataset versioning and ML data tools such as DVC, LakeFS, Deep Lake, FiftyOne, or similar platforms

Preferred Qualifications

• Background in autonomous systems or mobile platforms, particularly in complex or unstructured environments
• Experience working with large-scale annotation workflows and external labeling providers
• Familiarity with distributed training approaches (e.g., DDP, FSDP) to support efficient collaboration with machine learning infrastructure
• Data Engineer in Abu Dhabi, United Arab Emirates