HPC Infrastructure Engineer - GPU Clusters

Requirements

  • Have run large-scale Linux server or GPU environments in production and enjoy both building and operating
  • Know the NVIDIA stack well — drivers, CUDA, NCCL, DCGM — or have deep systems experience and learn hardware stacks fast
  • Are comfortable with bare-metal environments, server hardware, and high-speed networking
  • Write solid automation in Python and/or Bash, with IaC tools like Ansible or Terraform
  • Are happy digging into noisy data (metrics, logs, PromQL) to find what’s actually wrong
  • Like owning real scope end to end and being the person others rely on
  • Don't consider any task above or beneath you — datacenter trips included

Nice to have

  • Experience supporting ML training workloads from the infra side (distributed training failure modes, checkpointing patterns)
  • Experience evaluating and working with GPU cloud providers
  • Parallel filesystems (WEKA, VAST, etc) or large-scale object storage
  • BMC/IPMI/Redfish automation, PXE provisioning at scale
  • Power and cooling awareness for dense GPU deployments

What you'll be doing

  • Operate and improve our GPU fleet end to end: provisioning, scheduling, monitoring, upgrades, capacity planning
  • Build automation that keeps the fleet healthy without human intervention — node health checks, automated draining and remediation, burn-in pipelines for new capacity
  • Own the stack beneath the training code: OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, high-speed networking (InfiniBand/RoCE)
  • Run and tune job scheduling (Slurm or similar) so researchers get compute fairly and fast
  • Build and maintain high-performance storage for datasets and checkpoints
  • Hunt down performance problems: stragglers, degraded links, thermal issues, flaky GPUs — and fix the class of problem, not just the instance
  • Evaluate rented GPU capacity: benchmark it, validate it, hold providers to their SLAs
  • Hands-on hardware work when it’s needed: racking, cabling, diagnostics, coordinating with datacenter staff and vendors
  • Keep clusters secure by default: access control, network isolation, secrets

Perks and benefits

  • Innovative culture: Be part of a generational opportunity to define the trajectory of AI, surrounded by a team pushing boundaries
  • Growth paths: Dynamic team with countless opportunities to drive impact beyond immediate role
  • Learning & development: Annual discretionary stipend for professional development
  • Social travel: Annual discretionary stipend to meet up with colleagues
  • Annual company offsite: Entire team meets yearly in a new location
  • Co-working: Monthly co-working stipend if not near main hubs
AI Summary ✨

Similar jobs

  • Staff Software Engineer (SRE)

    PhantomRemote EMEA5 days ago
    Remote EMEA
  • Staff Platform Engineer - EU

    Ashby16 locations14 days ago
    Remote EMEA
    €154k – €188k / year base
  • See all jobs in Bulgaria →