Senior Systems Software Engineer - HPC and AI Storage
Senior Systems Software Engineer - HPC and AI Storage
What you'll be doing:
Engineer and integrate storage systems for large-scale HPC and AI clusters, including Lustre and NFS, ensuring smooth interaction with compute nodes, networks, and storage hardware.
Examine performance and reliability issues throughout the I/O stack, managing application behavior across filesystem clients, Linux kernel paths, network protocols, servers, and storage devices.
Develop software fixes and performance improvements, collaborating with NVIDIA engineering teams, storage partners, and open-source maintainers through review, testing, and deployment.
Characterize real application I/O patterns such as training-data access, checkpoint/restart, metadata operations, and concurrent workloads, translating insights into software and system design improvements.
Build reproducible benchmarks, diagnostic tools, and regression tests to measure efficiency, latency, metadata performance, scalability, and behavior under contention.
Lead storage qualification and commissioning by validating functionality, data integrity, failure recovery, and sustained performance at cluster scale.
Define and implement storage placement, caching, and data lifecycle policies based on workload requirements, capacity, and operational evidence.
Requirements:
Deep engineering experience with Lustre, NFS, or comparable shared storage systems, including implementation-level debugging and Linux cluster integration.
Strong C programming and Linux debugging skills with experience in kernel or user-space storage software.
Understanding of filesystem and distributed-system behavior including metadata operations, caching and consistency, locking, concurrency, client/server communication, and recovery.
Experience analyzing application access patterns and tuning filesystem or storage-stack behavior for performance improvements.
Ability to take complex performance or correctness problems from symptoms through reproduction, root-cause analysis, implementation, and validation.
Proficiency in Python and shell scripting to develop maintainable diagnostics, test automation, and analysis tools.
Ability to independently lead technical efforts, influence design decisions, and communicate findings clearly.
BS or MS in Computer Science, Engineering, Mathematics, Physics, or equivalent experience, with 12+ years of relevant work experience.
Nice to haves:
Contributions to Lustre, Linux filesystems, NFS implementations, or other storage projects including features, performance improvements, bug fixes, or reviews.
Expertise in Lustre metadata and client I/O, Linux VFS and page cache, NFS protocol internals, or distributed locking and recovery.
Experience with hierarchical storage management, policy engines, or migration between filesystems and object storage, including data recall and recovery.
Experience with RDMA storage transports, GPUDirect Storage, or optimizing data loading and checkpointing for GPU workloads.
Experience with storage security and multi-tenancy such as authentication, identity mapping, permissions, and encryption.