greenhouse
Senior Data Engineer
Great Waters Federal
Remote, USA
Full-time
Senior
Remote
Discovered 3 weeks ago
DatabricksApache SparkPySparkPandasPythonR
Key Skills for This Role
DatabricksApache SparkPySparkPandasPythonR
Full Job Posting
Key Responsibilities
- Analyze and collect data from various sources, including relational databases, APIs, external data providers, and real-time streaming sources.
- Design and implement efficient, scalable data pipelines to cleanse, transform, and aggregate data for downstream reporting and analytics.
- Build and maintain Databricks solutions using medallion architecture, including Bronze, Silver, and Gold layers, to create structured, reliable, and progressively refined datasets.
- Develop source-to-target mapping documentation for all data pipelines and conduct thorough unit testing to validate data accuracy and pipeline logic.
- Write advanced SQL to create aggregations across complex datasets and analyze data for anomalies, quality issues, and inconsistencies.
- Implement real-time and near-real-time data ingestion using AWS Database Migration Service (DMS) and other AWS-native services.
- Develop data-processing logic using Python and/or R, leveraging Spark and related packages such as PySpark and Pandas for large-scale data transformation.
- Manage source code, versioning, and deployments using GitLab, and support automated build and release processes through CI/CD pipelines.
- Collaborate with cross-functional teams in an Agile project environment, participating in sprint planning, stand-ups, and iterative delivery.
- Ensure that data engineering practices align with FISMA High and multi-tenant security, governance, and compliance requirements.
- Leverage AI automation tools to support data engineering pipeline development, testing, and validation and to accelerate delivery.
Required Qualifications
- Proven, hands-on experience as a Data Engineer or Databricks Developer building production-grade data pipelines.
- Strong working knowledge of Databricks on AWS, including E2 architecture, cluster configuration, job orchestration, and workspace management.
- Hands-on experience with Apache Spark, PySpark, and Pandas for large-scale, distributed data processing.
- Solid programming skills in Python; working knowledge of R is a plus.
- Experience with Databricks Auto Loader for scalable, incremental file ingestion.
- Practical experience with AWS Database Migration Service (DMS) for change data capture and real-time data replication.
- Strong experience with Delta Lake and Delta tables, including schema evolution, time travel, and optimization techniques such as OPTIMIZE, Z-ORDER, and VACUUM.
- Experience with Amazon RDS and other relational database sources used in data extraction and integration workflows.
- Advanced SQL skills, including complex joins, window functions, aggregations, and performance tuning.
- Solid understanding of medallion architecture and modern data lakehouse design principles.
- Experience with GitLab and CI/CD pipelines for automated testing, build, and deployment of data engineering code.
- Experience working in Agile/Scrum project environments.
Preferred Skills
- Broader knowledge of AWS services beyond DMS and RDS, including S3, Glue, Lambda, Step Functions, CloudWatch, and IAM.
- Experience operating within FISMA High, FedRAMP, or other regulated, multi-tenant compliance environments.
- Familiarity with data quality frameworks, anomaly detection techniques, and automated data validation.
- Exposure to infrastructure-as-code tools such as Terraform or CloudFormation for provisioning data platform resources.
- Knowledge of streaming technologies such as Kafka, Kinesis, or Spark Structured Streaming.
- Equal Opportunity Statement: Seneca Holdings provides equal employment opportunities to all employees and applicants without regard to race, color, religion, sex/gender, sexual orientation, national origin, age, disability, marital status, genetic information and/or predisposing genetic characteristics, victim of domestic violence status, veteran status, or other protected class status. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leave of absence, compensation and training. The Company also prohibits retaliation against any employee who exercises his or her rights under applicable anti-discrimination laws. Notwithstanding the foregoing, the Company does give hiring preference to Seneca or Native individuals. Veterans with expertise in these areas are highly encouraged to apply.
About Great Waters Federal
Parent group profile
Seneca Holdings
Holding Companies & Conglomerates1300 employeesFounded 2009
Tribally owned investment holding company that builds federal-contracting, environmental, travel, and sports businesses for the Seneca Nation.
Visit parent group websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer