Proven experience designing, running, and maintaining large and complex evaluation suites for frontier AI systems, involving both human and automated approaches.
Strong research background with experience in designing and implementing evaluations for AI research; proficient in Python and agentic coding.
Experience as a people manager with at least one year of active team management.
Strong understanding of statistics including variance, confidence intervals, and significance testing; ability to critically assess numerical results.
Excellent communication skills to convey evaluation results and their limitations clearly to project leads and leadership.
Ability to think from first principles to derive new evaluations for emerging capabilities.
What You'll Be Doing
Defining what to evaluate by creating, maintaining, and extending the evaluations roadmap in collaboration with product and research leads.
Defining how to evaluate by implementing the evals roadmap, building new benchmarks and objective functions including automated, AI-based, and human evaluations.
Minimizing noise in the evaluation pipeline to ensure reproducible, trustworthy, and accurate measurement of relevant factors.
Identifying and communicating limitations and blind spots of evaluations to support project teams in selecting comprehensive evaluation sets.
Monitoring critical training runs independently to detect regressions and emerging capabilities early and communicate findings clearly.
Managing and growing the evaluation team, defining hiring roadmaps, and supporting team members' technical and career development.
Perks and Benefits
Competitive compensation of EUR 255,000 - 295,000 plus equity, depending on profile.
Hybrid work arrangement requiring presence in the Freiburg office at least 2 days per week or one full week every other week.
Work in a cutting-edge research lab competing at the forefront of AI and visual intelligence development.