Senior Deep Learning Engineer, Accuracy Evaluation

Requirements:

• BS, MS, or PhD in Computer Science, Machine Learning, Statistics, or a related field

• 6+ years of hands-on experience with LLMs, designing and running evaluations for large language models or multimodal AI systems, including experience with agentic, multi-turn, or reasoning-heavy settings

• Strong statistical foundations: experimental design, significance testing, regression analysis, and the ability to distinguish signal from noise in benchmark results at scale

• Proven experience building evaluation infrastructure (pipelines, benchmark harnesses, reproducible CI systems) not just consuming existing benchmarks

• Clear, precise communicator who can translate quantitative evaluation results into decisions for researchers, product teams, and senior leadership

What you'll be doing:

• Design and build decision-grade evaluation environments for NVIDIA's frontier models spanning reasoning, multimodal, long-context, and agentic systems, producing auditable accuracy signals that gate every major model release

• Research and develop novel evaluation methodologies for emerging model families and capability domains (low-precision numerics, multi-turn agentic tasks, code generation) where established benchmarks don't yet exist or don't generalize

• Build and operate the evaluation infrastructure and pipelines including benchmark environments, regression CI systems, and statistical analysis tooling, used by model, product, and applied research teams across NVIDIA

• Partner with model research, training, and customer teams to translate evaluation signals into concrete decisions: release go/no-go, training iteration direction, and competitive positioning against external frontier models

Nice to haves:

• Deep familiarity with open-source evaluation frameworks

• Experience designing evaluations for agentic systems: tool use, multi-turn reasoning, environment-based benchmarks (SWE-bench, GAIA, WebArena-style), or interactive evaluation settings

• Track record of publishing or contributing to evaluation research, new benchmark design, methodology papers, or reproducibility analyses that shaped how the field measures model capability

• Experience measuring model accuracy under low-precision inference (FP8, INT4, quantization-aware settings) and understanding how calibration and sparsity interact with benchmark results

• Comfort running large-scale workloads on HPC/Slurm clusters, including reproducible experiment management (MLflow, W&B) and compute cost optimization across hundreds of benchmark runs

Perks and benefits:

• Base salary determined based on your location, experience, and the pay of employees in similar positions

• Opportunity to work with world-class software engineers and partners in a fast-growing company at the forefront of the AI revolution

• Access to the most powerful enterprise-grade GPU clusters capable of hundreds of PetaFLOPS

• Gain early access to unreleased hardware and make a direct impact on NVIDIA's roadmap and the broader AI landscape