Requirements:

Strong engineering experience in large-scale distributed training or HPC systems.
Experience with multi-node cluster orchestration (Slurm, Ray, Kubernetes, or similar).
Comfort debugging performance issues across CUDA/NCCL, networking, IO, and data pipelines.
Experience working with containerized environments (Docker, Singularity/Apptainer).
A track record of building tools that increase developer velocity for ML teams.
Excellent judgment around trade-offs: performance vs complexity, research velocity vs maintainability.
Strong collaboration skills — you’ll work closely with infra, research, and deployment teams.

Nice to Haves:

Experience with training LLMs or other large transformer architectures.
Contributions to ML frameworks (PyTorch, JAX, DeepSpeed, Megatron, xFormers, etc.).
Familiarity with evaluation and serving frameworks (vLLM, TensorRT-LLM, custom KV caches).
Experience with data pipeline optimization, sharded datasets, or caching strategies.
Background in performance engineering, profiling, or low-level systems.

Build and own the training framework responsible for large-scale LLM training.
Design distributed training abstractions.
Improve training throughput and stability on multi-node clusters.
Develop and maintain tooling for monitoring, logging, debugging, and developer ergonomics.
Collaborate closely with infra teams to support high-performance training.
Investigate and resolve performance bottlenecks across the ML systems stack.
Build systems that ensure reproducible, debuggable, large-scale runs.

A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.
Full health and dental benefits, including a separate budget for mental health.
RRSP matching, 401K, Pension Scheme.
100% Parental Leave top-up for up to 6 months, for either parent.
Annual enrichment benefits.
Education & learning stipend for conferences, courses, and coaching.
6 weeks of paid vacation.
Budget for traveling to other offices if you are remote, plus an annual company offsite.