Lead Data Engineer

Overview

Overview

Responsibilities

• Design, build, and optimize scalable data pipelines for AI/LLM workloads, including vectorization and embedding processing.
• Develop and maintain ETL/ELT workflows for structured, unstructured, and streaming data.
• Create and manage vector database indexing and similarity search pipelines using tools like FAISS, Pinecone, Weaviate, Qdrant, Chroma.
• Build retrieval systems for RAG, semantic search, and enterprise knowledge retrieval.
• Develop robust, reusable data orchestration pipelines using Airflow, Spark, or similar tools.
• Architect and manage data pipelines across Azure (primary), AWS, and GCP environments.
• Integrate and optimize storage and processing across SQL, NoSQL, and vector databases.
• Contribute to the design and implementation of event-driven architectures.
• Collaborate with AI teams to enable embedding generation, LLM integration, and model-serving pipelines.
• Ensure end-to-end data quality, monitoring, reliability, and observability.
• Lead or participate in system design for large-scale, distributed data and AI systems.

Programming & Data

• Strong expertise in Python for data processing, APIs, automation, or distributed workloads.
• Strong proficiency in SQL and knowledge of NoSQL databases (MongoDB, DynamoDB, Cosmos DB, etc.).
• Experience with vector databases, such as: FAISS, Pinecone, Weaviate, Qdrant, Chroma.
• Strong knowledge of data modeling, pipeline development, and ETL/ELT frameworks.

Ai And Llm Infrastructure

• Solid understanding of vectorization, embeddings, and similarity search techniques.
• Familiarity with LLMs, embedding models, and RAG pipeline concepts.
• Experience integrating embedding-generation pipelines via Hugging Face, OpenAI, or other model providers.

Cloud & Distributed Systems

• Proficiency with Azure (primary), and familiarity with AWS and GCP.
• Experience with Docker and containerized development.
• Understanding of Kubernetes is a strong plus.

Orchestration & Big Data

• Expertise in Apache Airflow for scheduling and orchestration.
• Experience with Apache Spark or equivalent distributed processing frameworks.

Architecture & Engineering Fundamentals

• Strong system design fundamentals for scalable and distributed systems.
• Knowledge of event-driven architecture and modern data platforms.
• Strong understanding of DevOps, CI/CD, version control, and observability best practices.

Qualifications

• 8+ years of progressive experience in data engineering, distributed systems, or AI/ML data infrastructure
• Experience building RAG pipelines in production.
• Knowledge of graph databases or hybrid search systems.
• Understanding of model deployment, inference optimization, and caching techniques for LLM workloads.
• Familiarity with data governance, IAM, and security patterns across cloud ecosystems

What We Look For

What Working At Inception Offers