Data Engineer Consultant II
Role Overview
Allstate’s Data & Analytics Technology organization is seeking a Data Engineer to design, build, and operate scalable, reliable, and high performing data pipelines that support enterprise analytics, reporting, and advanced data use cases.
In this role, you will focus on building robust batch and streaming data solutions using Apache Spark and modern cloud data platforms.
Key Skills for This Role
Full Job Posting
Key Responsibilities
- Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud‑native data technologies.
- Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analytics‑ready datasets.
- Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads.
- Build and manage data processing workloads within modern Lakehouse platforms, including Microsoft Fabric / One Lake (preferred).
- Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes.
- Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasets.
- Manage and evolve data schemas while handling schema drift and upstream source changes.
- Develop reusable frameworks, libraries, and standardized patterns to improve data engineering productivity and consistency.
- Implement CI/CD pipelines for data workloads to enable automated testing, deployment, and rollback.
- Monitor data pipelines and jobs, troubleshoot failures, and resolve performance or data quality issues.
- Partner with analytics engineers, BI developers, and data scientists to understand data requirements and deliver curated datasets.
- Collaborate with platform, security, and governance teams to ensure data security, compliance, and proper access controls.
- Contribute to Agile delivery processes, including sprint planning, design reviews, and continuous improvement initiatives.
Required Qualifications
- Strong experience as a Data Engineer building and operating production data pipelines.
- Hands‑on experience with Apache Spark for large‑scale data processing.
- Proficiency in Python , SQL , and data transformation best practices.
- Experience with cloud‑based data platforms and storage (e.g., Data Lakes, Lakehouse architectures).
- Familiarity with Microsoft Fabric , One Lake , or similar analytics platforms (strong plus).
- Experience designing and optimizing data models for analytical workloads.
- Understanding of distributed data processing concepts, performance tuning, and fault tolerance.
- Experience with CI/CD, version control, and infrastructure‑as‑code concepts.
- Strong problem‑solving skills and ability to troubleshoot complex data issues.
- Excellent communication skills and ability to collaborate across technical and non‑technical teams.
- 4+ years of experience in data engineering or equivalent role (preferred).
Preferred / Nice‑to‑Have Skills
- Experience with real‑time or event‑driven data processing.
- Familiarity with data governance, metadata management, and data quality frameworks.
- Exposure to orchestration tools and workflow management systems.
- Experience supporting analytical, reporting, or machine learning use cases.
Supervisory Responsibilities
- This job does not have supervisory responsibilities.
About Allstate
Joining our team isn’t just a job — it’s an opportunity. One that takes your skills and pushes them to the next level. One that encourages you to challenge the status quo. One where you can shape the future of protection while supporting causes that mean the most to you. Joining our team means being part of something bigger – a winning team making a meaningful impact.
The Allstate Corporation is one of the largest publicly held insurance providers in the United States. Ranked No. 84 in the 2023 Fortune 500 list of the largest United States corporations by total revenue, The Allstate Corporation owns and operates 18 companies in the United States, Canada, Northern Ireland, and India. Allstate India Private Limited, also known as Allstate India, is a subsidiary of The Allstate Corporation. The India talent center was set up in 2012 and operates under the corporation's Good Hands promise. As it innovates operations and technology, Allstate India has evolved beyond its technology functions to be the critical strategic business services arm of the corporation. With offices in Bengaluru and Pune, the company offers expertise to the parent organization’s business areas including technology and innovation, accounting and imaging services, policy administration, transformation solution design and support services, transformation of property liability service design, global operations and integration, and training and transition.
Learn more about Allstate India here .
Keys to Success
At Allstate, it’s all about teamwork, flexibility, and thinking ahead. We all contribute to the bigger picture, combining unique ideas to design innovative, more affordable protection solutions for customers.
We look for candidates with these skills to help us achieve that goal:
Learning agility Quickly adapt to new situations, continually build new skills, experiment, and embrace new ways of doing things
Customer centricity Deliver exceptional experience with a customer-first mindset and human-centered design
Digital literacy Discover and apply emerging digital technology tools, data and insights.
Results-oriented Start with measurable outcomes and drive results with speed
Inclusive leadership Integrate diverse viewpoints into decision-making processes to enhance creativity and innovation
Together, we’re all working toward Our Shared Purpose , using our strengths to make a real difference for our people, our customers, our company, and the world around us.
You’re in Good Hands® is more than a promise we make to our customers. It’s a promise we make to our employees, too.
We want you to love where you work. That starts with the freedom to be yourself. Our workplace flexibility and focus on individuality means everyone is seen, heard and respected.
When you join us, you’ll have the opportunity to push your skills to the next level with access to development programs to support your career aspirations – whatever that means for you. Because as you learn and grow, so do we.
Working here also means getting the chance to make a real impact in your community. We have been driving change for over 90 years, but the mark we leave on the world can be even greater when we work together.
Benefits
- Experience the benefits that make Allstate a great place to work.
- *Benefits vary based on position.
- Health
- Medical, dental and vision coverage
- HSAs and FSAs
- Wellbeing programs
- Free therapy sessions
- Retirement
- 401(k) plan
- Pension plan
- Free financial counseling
- Work-life balance
- Paid time off
- Work flexibility
- Career development
- Talent shares
- Tuition reimbursement
- Learning opportunities
- For a full description of Allstate’s benefits visit our benefits page
A workplace for everyone
- We are all stronger when everyone feels safe, respected, and heard. By bringing together diverse perspectives, values and ideas, we can give our employees, customers and communities a better experience.
- View inclusive initiatives
About Allstate
A leading provider of insurance and protection solutions.
Visit company websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Allstate
Senior Insider Threat Investigator
, USA
Lead Threat Hunter
, USA
Senior Subrogation Attorney, Remote (Los Angeles, CA)
, USA
Represented Bodily Injury Adjuster - PST/MST/CST - Remote
, USA
Senior Trial Attorney, Remote (Sacramento / San Francisco, CA)
, USA
Lead Engineer - Classic Products-2
, USA
Lead Threat Hunter
, USA
Senior Insider Threat Investigator
, USA