Lead/Sr AI Engineer with Azure Data Bricks || Immediate Joiner

Role Overview

Key Responsibilities

• Design end-to-end data and AI pipelines using Azure Databricks.
• Develop ETL/ELT workflows with Python, PySpark, and SQL.
• Implement CI/CD for Databricks jobs, notebooks, and workflows.
• Integrate Databricks with Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
• Optimize workloads for performance, cost, and reliability.
• Move ML and generative AI models from experimentation into production.
• Build LLM, embedding, RAG, NLP, computer vision, and predictive analytics solutions.
• Fine-tune LLMs using LoRA or QLoRA and integrate Azure OpenAI or Hugging Face models.
• Implement vector retrieval using FAISS or Azure Cognitive Search.
• Implement responsible AI practices, including bias detection and model governance.
• Conduct scenario-based testing and align code changes and acceptance criteria with stakeholders.
• Build Databricks Apps using Visual Studio Code and send custom notifications through APIs and Teams webhooks.

Required Skills

• Azure Databricks experience with jobs, workflows, clusters, and preferably Unity Catalog.
• Python expertise with a heavy PySpark focus.
• SQL expertise including complex joins, window functions, and analytical queries.
• Azure knowledge including ADLS Gen2, ADF, Key Vault, and IAM concepts.
• Pipeline orchestration, CI/CD, and environment-promotion experience.
• Azure DevOps experience.
• Strong understanding of the ML lifecycle and MLOps best practices.
• Model deployment experience using MLflow or a similar framework.

Additional Technical Scope

• Understand Unity Catalog permissions, OBO tokens, and service principals for Databricks agents, Genie Spaces, ML, and foundation models.
• Build minimal FastAPI and React single-page applications with chat interfaces and Claude APIs.
• Apply prompt engineering using system and user prompts.
• Develop Databricks-specific RAG using AI Search indexes and hierarchical chunking.
• Implement basic AI safety guardrails and understand caching mechanisms.

Strong Advantages

• Experience with ML and generative AI workloads on Databricks.
• Experience with RAG, embeddings, or inference pipelines.
• Knowledge of Terraform, ARM, or Bicep.
• Experience with Databricks Asset Bundles.
• Experience with Airflow or ADF orchestration.
• Knowledge of LangChain or similar AI application development frameworks.
• Experience with Azure Machine Learning or Azure Cognitive Services.