Software Engineer II, DataProc

What you'll be doing

  • Design, build, and operate the services and libraries for data orchestration and workflow platform (Airflow, TaskService) used company-wide.
  • Develop, scale, and optimize distributed data processing jobs on Spark running across Amazon EMR, Kubernetes, and Databricks.
  • Collaborate with engineering teams to migrate and modernize big data workloads, such as moving jobs onto Databricks or Spark-on-Kubernetes.
  • Enhance platform reliability and observability through metrics, alerting, tracing (OpenTelemetry), and on-call tooling.
  • Build and maintain internal tools and lightweight UIs to help engineers monitor, debug, and operate data pipelines.
  • Own projects end-to-end, from design and code review through deployment, rollout, and production support.
  • Debug and resolve complex issues in large-scale, distributed production systems.
  • Contribute to the technical roadmap and define best practices for data pipeline development company-wide.

Requirements

  • Strong software engineering fundamentals with production experience in Python, Scala, or Java.
  • Experience building or operating distributed data processing systems (e.g., Spark, Databricks, Hadoop) and/or workflow orchestration tools (e.g., Airflow).
  • Comfortable working with cloud infrastructure (AWS, Kubernetes) and CI/CD pipelines.
  • Solid grounding in data structures, algorithms, and distributed systems design.
  • Strong debugging and troubleshooting skills for large-scale production systems.
  • Some experience with or interest in building internal tools, dashboards, or lightweight UIs.
  • Effective communication skills and ability to collaborate with engineering teams across the company.
  • Pragmatic, ownership-driven approach to building and operating software.
  • 2+ years of software development/engineering experience.
  • Bachelor’s or Master’s degree in computer science, relevant engineering field, or equivalent experience.

Nice to haves

  • Experience with SQL and data warehousing technologies (e.g., Snowflake, Vertica).
  • Knowledge of Kubernetes operators and cluster administration.
  • Familiarity with observability tooling such as OpenTelemetry, Grafana, and Prometheus.
  • Experience carrying production on-call responsibilities for critical systems.
  • Exposure to machine learning pipelines or ad-tech / real-time bidding systems.

Perks and benefits

  • Global team with diverse backgrounds, experiences, and perspectives fostering an inclusive and supportive environment.
  • Commitment to equal opportunity employment and creating an inclusive hiring experience.
  • Support for accommodation requests or accessibility needs during the hiring process.
AI Summary ✨

Similar jobs

  • Software Engineer II, Data Platform (Hybrid, London)

    CrowdStrikeGreater London, UKa day ago
    NewRegistered sponsor
  • Software Engineer II - Data Engineering

    JPMorgan ChaseGreater London, UKa day ago
    New
  • Sr Lead Software Engineer

    JPMorgan ChaseCity of London, UKa day ago
    New
  • See all jobs in UK →