Senior Data Engineer

Overview

Why This Matters

Why Deriv

• We鈥檙e in production, not planning.
• Natural language interfaces querying the data warehouse directly鈥攍eaders ask questions in English, answers come back in seconds
• Continuous KPI monitoring with anomaly detection surfacing problems before stakeholders notice
• Dozens of fraud detection models running against production data, continuously
• 400+ users on our internal workflow orchestration platform, fed by the data infrastructure you鈥檒l help build and scale
• We share openly.
• Deriv芒聼篓ed芒聼漏 is where we write about what we鈥檙e shipping, what breaks, and what we figured out the hard way.
• You鈥檒l join a transformation that鈥檚 underway, not one waiting for approval.

What You鈥檒l Do

• Your work cuts across trading and product analytics, compliance and regulatory reporting, AI/ML feature pipelines, and business intelligence.
• Your placement depends on team needs and your strengths鈥攜our impact won鈥檛 be limited to one domain.
• Build pipelines that are governed by design
• Design and build ETL/ELT pipelines across batch and real-time workloads using AI-assisted development鈥攔educing build time without cutting reliability
• Bake in observability from the start: freshness checks, completeness monitoring, schema drift detection, lineage tracking, and alerting. Not retrofitted after launch
• Implement automated data quality checks and anomaly detection into every pipeline鈥攇overnance built in, not bolted on
• Own data accuracy before anyone has to ask
• Identify and resolve data issues before they surface to analysts or stakeholders
• Define and maintain data contracts: SLAs, SLOs, schema agreements, and producer-consumer alignment
• Handle PII, access control, and auditability correctly in a regulated financial environment
• Make the platform better than you found it
• Optimise warehouse performance and cost鈥攓uery efficiency, partitioning, clustering, orchestration reliability
• Build data models designed to scale beyond the immediate use case: dimensional modelling, semantic layers, reusable abstractions
• Spot gaps in the data platform and address them without waiting to be assigned
• Raise the bar for the team
• Partner with analysts, product, finance, and compliance teams to translate requirements into reliable, governed data products
• Peer-review pipeline code and push quality standards higher across the team
• Help onboard new engineers鈥攕hare context, catch mistakes early, make others productive faster

Who You Are

• You build pipelines, not just queries
• 6+ years in data engineering. You know that writing SQL is the easy part. The hard part is making sure data arrives on time, matches the schema you promised, and doesn鈥檛 silently break when upstream changes. You think about data quality before someone files a bug.
• You know the cloud data stack cold
• GCP, BigQuery, Airflow, Python鈥攐r the equivalent at comparable scale. You鈥檝e built and maintained real pipelines across batch and streaming workloads. You鈥檝e shipped with dbt or Dataform and understand why transformation layers matter. You use AI coding assistants daily as part of your workflow, not as a novelty.
• You think in contracts, not assumptions
• Data modelling techniques like Kimball star schema, Data Vault, or Medallion architecture aren鈥檛 buzzwords to you鈥攖hey鈥檙e tools you pick based on the problem. You understand that a pipeline without a data contract is a pipeline waiting to fail. Even better if you鈥檝e implemented data contract frameworks or schema registries at scale.
• You fix root causes, not symptoms
• When data is wrong, you don鈥檛 patch the dashboard. You trace the problem to its source, fix it systematically, and make sure it doesn鈥檛 recur. You explain what happened clearly鈥攖o engineers and to non-technical stakeholders who need to trust the numbers again.
• You make the people around you better
• You share learnings actively, create reusable resources, and give feedback that helps others ship better code. When a junior engineer is stuck, you unblock them. When a stakeholder鈥檚 requirements are vague, you ask the right questions and turn ambiguity into a spec.

You鈥檝e Seen What Regulated Data Looks Like

• A fintech, trading, or compliance-heavy background is a strong plus. You know the difference between handling PII in a textbook and handling it when auditors are checking your work. Experience with warehouse cost optimisation and containerised data platforms rounds out the picture.

• Cloud & Warehouse: GCP, BigQuery

• Orchestration: Airflow

• Languages: Python, SQL

• Transformation: dbt / Dataform

• Streaming: Kafka, Pub And Sub

• Practices: CI/CD, version-controlled pipelines, peer review, AI-assisted development

The Honest Reality