Data Engineer
Key Skills for This Role
Full Job Posting
"Position Overview We are seeking an experienced Databricks Data Engineer to design and enhance enterprise document-processing and data-extraction solutions. The role will focus on ingesting structured and unstructured content, extracting business and policy information using LLM-based approaches, improving extraction accuracy, and making validated information available to downstream applications through an ontology-driven data layer. The engineer will join a small, hands-on team working on document intelligence solutions supporting brokerage and underwriting workflows. A major focus will be redesigning existing pipelines to improve accuracy, performance, scalability, and maintainability while supporting a broader migration from Microsoft Fabric to Databricks. Key Responsibilities • Design, build, and optimize Databricks data pipelines for document ingestion, classification, extraction, validation, and downstream consumption. • Process information from multiple document and communication formats, including: o PDF documents o Emails and attachments o Excel/XLSX files o Insurance applications o Statements of Values (SOVs) o Other structured and unstructured documents • Improve existing extraction pipelines where LLMs produce false positives, incorrect metadata, or inaccurate field values. • Develop validation and quality-control approaches that improve the reliability of extracted information. • Implement document classification and content-aware extraction strategies. • Handle source-specific extraction rules, such as extracting particular attributes from an email rather than from attached documents. • Design pipelines that transform extracted information into consistent, structured JSON outputs for downstream applications. • Contribute to the design and implementation of a custom ontology layer that represents key insurance business entities and relationships. • Map extracted information into the ontology so applications can consume standardized business concepts rather than raw document fields. • Address performance challenges associated with processing large or complex Excel/XLSX and SOV files. • Optimize pipeline scalability and concurrency for production workloads. • Work with PostgreSQL/Lakebase-based persistence for final structured outputs. • Collaborate closely with brokerage, underwriting, AI, and data engineering teams to connect extracted application data with downstream underwriting processes. Required Qualifications • Strong hands-on experience with Databricks and production data engineering. • Strong proficiency in Python, PySpark, and SQL. • Experience building scalable ingestion and transformation pipelines on the Databricks platform. • Experience processing unstructured and semi-structured data, particularly PDFs, emails, Excel files, JSON, and business documents. • Experience integrating LLMs or AI models into data-processing pipelines. • Understanding of prompt-based and structured data extraction from documents. • Experience implementing data validation, reconciliation, or quality controls for AI-generated outputs. • Strong understanding of data modeling, metadata, schemas, entity relationships, and semantic/ontology concepts. • Experience designing APIs, JSON structures, or data products for downstream application consumption. • Experience troubleshooting performance, scalability, and concurrency issues in production data pipelines. • Experience working with cloud-based data architectures. Minimum 7+ years of experience in data engineering, software development, or production data pipeline development. Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines."
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at NTT DATA Services
Helpdesk Associate - ITIL
Bengaluru, IND
Technology Consultant - End User Computing (EUC) Engineer
Pune, IND
ERP Specialist Advisor
Bengaluru, IND
Systems Integration Advisor
Mumbai, IND
Systems Integration Senior Analyst
Bengaluru, IND
Systems Integration Specialist Advisor
Bengaluru, IND
Email Security Engineer
Noida, IND