Senior Engineer, NCX

Requirements:

• BS, MS, or Ph.D. in Computer Science, Computer/Electrical Engineering, or a related technical field, or equivalent experience.
• 8+ years of experience in infrastructure engineering, Site Reliability Engineering, DevOps, cloud platform engineering, systems engineering, or similar roles supporting large-scale production environments.
• Strong experience operating Linux-based distributed systems and cloud infrastructure in production.
• Deep understanding of Kubernetes, containers, cluster scheduling, and the operational lifecycle of large multi-node environments.
• Strong understanding of production observability, including metrics, logging, alerting, dashboards, health checks, and operations guided by service level agreements.
• Experience crafting automation for infrastructure lifecycle management, failure detection, remediation, upgrades, and configuration management.
• Strong networking fundamentals and experience troubleshooting complex distributed systems across compute, network, and storage layers.
• Programming and automation experience using Python, Go, shell scripting, or similar languages.

Nice to Haves:

• Experience managing extensive GPU or accelerated computing infrastructure that supports AI training and inference workloads.
• Experience with NVIDIA technologies including DGX/HGX systems, CUDA, NVLink/NVSwitch, NVIDIA networking, InfiniBand, RoCE, GPU Operator, Network Operator, or related NVIDIA infrastructure software.
• Proven experience collaborating with NVIDIA Cloud Partners, hyperscale cloud providers, managed AI clouds, or extensive service-provider infrastructure and operating SLOs for large-scale compute infrastructure and using operational data to improve availability, performance, and fleet efficiency.
• Extensive knowledge of infrastructure observability tools including Prometheus, Grafana, OpenTelemetry, Alertmanager, and scalable telemetry pipelines and translating reference architectures or infrastructure requirements into repeatable production operating models across multiple customer or partner environments.
• Knowledge of failure modes related to large distributed AI workloads and the infrastructure features necessary to consistently support extended training and production inference.

What you'll be doing:

• Lead NCP Day 2 operational readiness efforts. Collaborate directly with NVIDIA Cloud Partners to set up the systems, procedures, automation, and operational methods necessary to consistently manage NVIDIA accelerated infrastructure following initial deployment and activation.
• Build continuous infrastructure validation. Develop and implement methods to continuously validate GPU, CPU, storage, and network health. Do this across large-scale AI clusters to identify degraded infrastructure before it impacts critical training or inference workloads.
• Establish observability and operational telemetry. Help NCPs implement comprehensive telemetry, monitoring, alerting, dashboards, and operational signals across compute, GPU, InfiniBand/RoCE networking, storage, Kubernetes, and AI workloads.
• Develop automated detection and remediation. Build workflows to detect, isolate, drain, repair, validate, and return unhealthy infrastructure to service while minimizing disruption to customer workloads.
• Refine fleet lifecycle administration. Implement scalable strategies for managing sizable GPU fleets, including NVIDIA driver and firmware lifecycle administration, Kubernetes node maintenance, OS patching, configuration management, upgrades, and configuration drift identification.
• Operationalize NVIDIA reference architectures. Translate NVIDIA NCP requirements and reference architectures into production operating practices, validation criteria, runbooks, automation, and measurable operational standards.
• Define operational health and readiness. Develop health signals, SLOs, important metrics, acceptance criteria, and ongoing validation mechanisms that provide NVIDIA and NCPs with clear insight into infrastructure reliability and service readiness.
• Build reusable operational frameworks. Develop tooling, automation, implementation guides, runbooks, operational playbooks, and reference implementations that can be applied consistently across multiple NCP environments.

Perks and Benefits:

• Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.
• For Poland: The base salary range is 292,500 PLN - 507,000 PLN for Level 4, and 375,000 PLN - 650,000 PLN for Level 5.