Lead/Sr AI Engineer with Azure Data Bricks || Immediate Joiner

Role Overview

Primary Responsibilities

• Design and build end-to-end data and AI pipelines using Azure Databricks.
• Develop ETL and ELT workflows with Python, PySpark, and SQL.
• Implement CI/CD pipelines for Databricks deployments, including jobs, notebooks, and workflows.
• Integrate Databricks with Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
• Optimize jobs for performance, cost, and reliability, and build reusable modular code.
• Implement logging, monitoring, error handling, and production pipeline optimization.
• Move machine learning and generative AI models from experimentation to production with data scientists and platform teams.

AI and Databricks Capabilities

• Develop and deploy ML and generative AI models, including LLMs, embeddings, RAG pipelines, computer vision, and predictive analytics.
• Fine-tune LLMs with LoRA or QLoRA and integrate Azure OpenAI or Hugging Face models.
• Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
• Use Databricks Unity Catalog permissions, OBO tokens, and service principals for Databricks Agents, Genie Spaces, machine learning, and foundation models.
• Build and deploy Databricks Apps using Visual Studio Code.
• Use FastAPI, React, chat interfaces, Claude models or APIs, prompt engineering, graph-based dependency analysis, and Claude Vision APIs.
• Understand token economics, AI safety guardrails, retrieval-augmented generation, hierarchical chunking, and caching mechanisms.

Required Qualifications

• Strong hands-on experience with Azure Databricks, Python, PySpark, SQL, and Azure services.
• Experience with ADLS Gen2, Azure Data Factory, Key Vault, IAM concepts, CI/CD, environment promotion, and Azure DevOps.
• Strong understanding of the ML lifecycle and MLOps best practices.
• Experience deploying models using MLflow or a similar framework.

Preferred Qualifications

• Experience with ML and generative AI workloads on Databricks, RAG, embeddings, inference pipelines, or production monitoring and cost optimization.
• Experience with Terraform, ARM, Bicep, Databricks Asset Bundles, Airflow, or Azure Data Factory orchestration.
• Knowledge of LangChain or similar AI application development frameworks.
• Experience with Azure Machine Learning or Azure Cognitive Services.