Senior Data Engineer II - Real Time Data Engineering

Requirements

  • Proven experience building, deploying, and operating production-grade, low-latency stream processing pipelines using Apache Kafka or related technologies (Kafka Streams, Flink, Spark Streaming).
  • Deep hands-on expertise in handling idempotency, duplicate events, out-of-order logs, late-arriving data, and exactly-once processing semantics in financial or mission-critical domains.
  • Strong knowledge of windowing techniques such as tumbling, sliding, and session windows, along with stateful stream processing and high-throughput real-time aggregation.
  • Advanced experience in streaming data modeling for real-time consumer patterns and downstream analytical stores, including event schemas, state-store schemas, and schema evolution with Protobuf or Avro.
  • Skill in defensive engineering practices, designing systems with dead-letter queues, backpressure management, circuit breakers, and state-store recovery.

What you'll be doing

  • Designing and building foundations of a real-time liquidity observability stack and core streaming data pipelines powering global financial operations.
  • Architecting high-throughput Kafka and Flink streaming pipelines to process global bank transactions and balance demand versus supply intraday.
  • Building low-latency streaming pipelines to deliver instant, global liquidity visibility across currencies, jurisdictions, and nostro accounts.
  • Fueling machine learning forecasting models and routing high-value transactions with fine-grained event controls.
  • Bulletproofing event architecture against out-of-order data, network partitions, and schema shifts.
  • Leveraging a Medallion architecture with sub-second OLAP querying to provide Treasury Ops with precise liquidity visibility.
  • Collaborating closely with Product Managers, Data Scientists, Quantitative Analysts, and Treasury Operations in autonomous, cross-functional teams.
  • Driving clarity and quality via clear communication, design reviews, and writing well-tested, maintainable code.
AI Summary ✨

Similar jobs

  • 4 days ago
  • Senior Data Engineer II - Scalable Growth

    Wise·Greater London, UK

    ≈ €103k – €171k / yearbase

    7 days ago
  • Senior Data Engineer

    Zego·London, UK

    9 days ago
  • See all jobs in UK →