Python with Spark Developer (5.1-7 years)-Chennai
Key Skills for This Role
Full Job Posting
Responsibilities
- Direct Responsibilities
- Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
- Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
- Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
- Unit‐test, integrate‐test, and review code.
- Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
- Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
- Document data lineage, pipeline design, and operational run‐books in Confluence/SharePoint.
- Keep up to date with latest technologies and trends and provide input, expertise and recommendations.
- Contributing Responsibilities
- Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement.
- Participate in Agile ceremonies (sprint planning, daily standups, retrospectives) and help groom the backlogs.
- Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data‐engineering patterns.
- Evaluate emerging technologies and deliver proof‐of‐concepts for CEFS.
- Technical & Behavioral Competencies
- Resourceful to quickly understand complexities involved and provide the way forward.
- Good experience in technical analysis of n-tier applications with multiple integrations using object oriented, APIs & Microservices approaches.
- Strong knowledge about design patterns and development principles.
- Inclination and prior experience of working across SQL, Python and ETL.
- Strong Hands-on experience - Python (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle.
- PySpark - DataFrames, Spark SQL, Structured Streaming, performance tuning (partitioning, caching, broadcast joins).
- Advanced SQL – complex queries, stored procedures, query optimization.
- Good Knowledge and experience to use Python packages such as Pandas, NumPy, etc. for cleaning up of Data, Data Wrangling, Analysis of Data and Visualization of Data.
- Good experience in development and maintenance of code/scripts in both functional and technical specifications of all applications component, bug fixing and production support.
- Good knowledge on Linux/Unix environment (basic commands, shell scripting, etc.), testing phases, documentation and new framework.
- Some experience of working with build tools like Maven & DevOps tools like Bitbucket, Jenkins.
- Knowledge of Agile, Scrum, DevOps.
- Development experience in Data Engineering environment.
- Ability & willingness to learn & work on diverse technologies (languages, frameworks, and tools).
- Self-motivated, good interpersonal skills and inclination to constantly upgrade new technologies and frameworks.
- Good communication and co-ordination skills.
- Specific Qualifications:
- Good to have knowledge of front-end technologies preferably Flask.
- Skills Referential (Required knowledge, skills and abilities)
Technical Skills:
- Python
- Pyspark
- SQL
- ETL
- Behavioral Skills:
- Ability to synthetize / simplify
- Ability to collaborate / Teamwork
- Attention to detail / rigor
- Ability to deliver / Results driven
- Education Level: Bachelor’s degree or equivalent
- Location: Chennai
About Capco
Global management and technology consultancy dedicated to the financial services and energy industries, operating as a Wipro company.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Capco
React.JS Developer
Bengaluru, IND
Job Title: Senior Front End Developer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Be
Java Full Stack Developer(2-5 yrs)
Chennai, IND
About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 b
AWS Support Engineer
Bengaluru, IND
Job Title: AWS Support Engineer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Com
Technical Writer
Toronto, CAN
Capco is seeking an experienced Technical Writer to support addressing regulatory findings identified by the Office of the Superintendent of Financial Institutions (OSFI). In this contract role, you will work closely wit
Java Core
Chennai, IND
About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 b
Bench/ Resource/ Workforce Manager
Bengaluru, IND
Job Title: Workforce Management/ Resource Management/ Bench Manager Location: Bangalore About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the
Reg reporting Ops - Nomura - Mumbai
Mumbai, IND
React.JS Developer
Bengaluru, IND
Java Full Stack Developer(2-5 yrs)
Chennai, IND
ETRM Developer
, USA
AWS Support Engineer
Bengaluru, IND
Technical Writer
Toronto, CAN
Java Core
Chennai, IND
Bench/ Resource/ Workforce Manager
Bengaluru, IND