{bc}
Apply
workday

Data Platform Engineer

Shift4
Toronto, CAN
Full-time
Mid · 5+ years experience
Onsite
Discovered 1 weeks ago
ansibleawscloudwatchdatadogdockerecs

Full Job Posting

Responsibilities

  • Operate, tune, and scale the ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake: task sizing, autoscaling, consumer concurrency, batching, retries, and dead-letter handling
  • Own pipeline reliability: define SLOs for freshness and lag, build alerting and dashboards, lead incident response and root-cause analysis for pipeline failures
  • Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services
  • Continuously benchmark and tune throughput versus cost across ECS, SQS, and Snowflake ingestion (Snowpipe/COPY, stages)
  • Onboard new sources and consumers to the platform in partnership with application, data, and analytics teams
  • Apply data-quality and security controls appropriate for PCI-compliant or payment-handling environments: encryption, secrets handling, least-privilege access, audit trails
  • Proactively explore and apply AI tools to improve efficiency across the pipeline lifecycle (development, testing, runbooks, and documentation)

Qualifications

  • 5+ years in platform, infrastructure, or DevOps roles supporting data-intensive systems in production
  • Deep hands-on AWS experience: ECS/Fargate (task sizing, autoscaling, deployments), SQS (throughput, DLQs, visibility timeouts, batching), S3, IAM, CloudWatch, and Aurora/RDS from the operator side
  • Proven track record tuning message-driven workloads: consumer concurrency, backpressure, retries, idempotency, throughput versus cost
  • Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY, stages, load monitoring)
  • Infrastructure as code with Terraform/Ansible, containerization with Docker, and CI/CD pipelines
  • Observability and alerting for pipeline health and lag using Prometheus/Grafana, CloudWatch, Datadog, or the Elastic Stack
  • Enough Python to read, debug, and instrument worker code
  • Passion for automation, performance, and clean system design

Nice to have

  • CDC tooling (Debezium, AWS DMS) and MySQL Binlog replication
  • Message brokers beyond SQS: Apache Kafka/Confluent, RabbitMQ
  • Elastic Stack or time-series database (Prometheus, InfluxDB) administration
  • Experience in a PCI or otherwise regulated environment
  • Payments or fintech domain knowledge
  • We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Shift4

Print Production Designer

Toronto, CAN

Discovered 2 weeks agoFull-time

Payment Specialist

Norway, USA

Discovered 4 weeks agoFull-time

Solution Specialist Payments

Vienna, USA

Discovered 4 weeks agoFull-time

Managing Director Denmark

Denmark, USA

Discovered 4 weeks agoFull-time

Refund Operator

Amsterdam, USA

Discovered 4 weeks agoPart-time

Pre-Launch, Sr Associate

Tallmadge, USA

Discovered 1 months agoFull-time

Solution Specialist- SC

, USA

Discovered 1 months agoFull-time

Solution Specialist

, USA

Discovered 1 months agoFull-time