Data Engineer - Commodities

About Millennium

Commodities Technology team

Responsibilities

• Build end-to-end Python and SQL ETL workflows for vendor and internal commodities data.
• Maintain standardized data models, schemas, and metadata for discoverable and reusable datasets.
• Operate scheduled pipelines with Airflow or similar orchestration tools.
• Implement validation, reconciliation, and anomaly detection to protect data quality.
• Use AI for schema inference, schema drift management, and scalable ingestion.
• Apply AI-enabled data quality, observability, and documentation tools to monitor data health and produce lineage.
• Maintain production-ready code and repeatable deployments with Git, GitHub Actions, PyTest, and automated testing.
• Work with portfolio managers, researchers, analysts, and data strategists to refine datasets and documentation.

What you bring

• At least 5 years in data engineering, analytics engineering, or a similar ETL-focused role.
• Strong Python and SQL skills and experience with large datasets and complex transformations.
• Hands-on Airflow or comparable orchestration experience.
• Experience with Git, CI/CD such as GitHub Actions, and automated tests such as PyTest.
• Strong attention to data quality and documentation; ability to identify edge cases and protect data integrity.
• Able to work independently, communicate across technical and non-technical groups, and manage concurrent initiatives.
• Commodities market and data knowledge is preferred.
• Data warehousing, columnar storage, analytical databases, catalog or governance tools, or financial services, trading, or research experience is preferred.