Data Engineer - Commodities

What You'll Be Doing

  • Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources.
  • Build and maintain standardized data models, schemas, and metadata to make commodities datasets easy to understand, discover, and reuse.
  • Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery.
  • Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency.
  • Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development.
  • Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation.
  • Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices.
  • Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs.

Requirements

  • 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines.
  • Strong Python and SQL skills, including experience working with large datasets and complex data transformations.
  • Hands-on experience with Airflow or comparable workflow orchestration tools.
  • Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest.
  • Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity.
  • Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives.

Nice to Haves

  • Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows.
  • Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools.
  • Experience in financial services, trading, or research-driven environments.
AI Summary ✨

Similar jobs

  • Data Platform Engineer

    ClickHouse4 locations2 days ago
    NewRemoteRegistered sponsor
  • Senior Data Engineer I - Global KYC & Onboarding

    WiseGreater London, UK2 days ago
    Registered sponsor
    ≈ €105k – €133k / year base
  • Research Engineer, Gemini Omni Data Infrastructure, DeepMind

    GoogleLondon, UK3 days ago
    Registered sponsor
  • Senior Data Engineer II - Real Time Data Engineering- Java

    WiseGreater London, UK5 days ago
    Registered sponsor
    ≈ €126k – €156k / year base
  • See all jobs in UK →