Sr Databricks Architect

Role overview

Platform architecture and delivery

• Design optimized Bronze, Silver, and Gold Lakehouse architecture for performance, cost, and downstream consumption.
• Build production pipelines using PySpark, Databricks SQL, Auto Loader, and Delta Live Tables.
• Develop batch and event-driven ingestion with SLA management, data quality, and lineage tracking.
• Define data product domains, ownership, contracts, metadata standards, and discoverability.
• Lead Unity Catalog governance, permissions, lineage, security, and cross-workspace controls.
• Own AWS workspace integration, including S3, IAM, VPC networking, and secure credentials.
• Drive Git-based DevOps and CI/CD for infrastructure and pipeline deployments.
• Tune Spark workloads with Photon, Serverless compute, and cluster configuration.
• Build Power BI dashboards, DAX measures, and data models for stakeholders.
• Lead architecture reviews, performance tuning, code quality, and technical roadmap decisions.

AI and GenAI responsibilities

• Implement Databricks AI/BI Genie spaces for natural-language business querying.
• Design RAG pipelines using Databricks Vector Search and Delta Lake.
• Productionize ML models and AI agents using MLflow, Feature Store, and Mosaic AI Model Serving.
• Support AI use-case discovery, assess data readiness, and scope RAG or agentic workflows.
• Implement LLMOps monitoring, evaluation loops, and CI/CD for AI model assets.
• Apply Unity Catalog governance to AI models, vector indexes, access controls, and audit trails.

Technical skills

• Databricks, PySpark, SQL, Python, Delta Lake, Auto Loader, Delta Live Tables, Workflows, and SQL Warehouses.
• Unity Catalog, medallion architecture, Data Mesh principles, data products, contracts, versioning, and SLAs.
• AWS services include S3, IAM, VPC, Glue, Lambda, Step Functions, and CloudWatch.
• Terraform, Git, Databricks CLI/SDK, Repos, CI/CD, Photon, Serverless compute, and Spark performance tuning.
• Power BI, DAX, DirectQuery/Import, and data models.
• AI/ML tools include Mosaic AI, MLflow, Feature Store, Model Serving, Vector Search, RAG, AI/BI Genie, and LLM integration.

Qualifications

• Bachelor's or master's degree in Computer Science, Engineering, or a related field.
• Header states 14+ years of experience; a later qualification states 8–12 years in data engineering and cloud architecture.
• At least 4 years of hands-on Databricks experience at production scale.
• Experience delivering cloud Lakehouse data products; AWS is preferred.
• Experience leading technical RFP tracks, scoping complex projects, and estimating effort.
• 1–2 years of exposure to productionizing AI/ML solutions on Databricks is a strong advantage.
• Strong ownership, communication, engineering leadership, mentoring, and ability to translate architecture into business value.
• Pragmatic AI mindset focused on data readiness, automation, performance, and reusable platforms.

Work arrangement

• Work mode is hybrid.
• Job locations listed in the description are Bangalore and Chennai.

About CGI