Research Lead, Evaluations

Requirements

  • Proven experience designing, running, and maintaining large and complex evaluation suites for frontier AI systems, involving both human and automated approaches.
  • Strong research background with experience in designing and implementing evaluations for AI research; proficient in Python and agentic coding.
  • Experience as a people manager with at least one year of active team management.
  • Strong understanding of statistics including variance, confidence intervals, and significance testing; ability to critically assess numerical results.
  • Excellent communication skills to convey evaluation results and their limitations clearly to project leads and leadership.
  • Ability to think from first principles to derive new evaluations for emerging capabilities.

What You'll Be Doing

  • Defining what to evaluate by creating, maintaining, and extending the evaluations roadmap in collaboration with product and research leads.
  • Defining how to evaluate by implementing the evals roadmap, building new benchmarks and objective functions including automated, AI-based, and human evaluations.
  • Minimizing noise in the evaluation pipeline to ensure reproducible, trustworthy, and accurate measurement of relevant factors.
  • Identifying and communicating limitations and blind spots of evaluations to support project teams in selecting comprehensive evaluation sets.
  • Monitoring critical training runs independently to detect regressions and emerging capabilities early and communicate findings clearly.
  • Managing and growing the evaluation team, defining hiring roadmaps, and supporting team members' technical and career development.

Perks and Benefits

  • Competitive compensation of EUR 255,000 - 295,000 plus equity, depending on profile.
  • Hybrid work arrangement requiring presence in the Freiburg office at least 2 days per week or one full week every other week.
  • Work in a cutting-edge research lab competing at the forefront of AI and visual intelligence development.
AI Summary ✨

Similar jobs

  • Senior Software Engineer — Autonomous Driving

    NVIDIAGermanya day ago
    NewRemote
    €112.2k – €194.5k / year base
  • Senior Data Scientist, Campaigns

    BoltBerlin, Germany7 days ago
  • See all jobs in Germany →