Requirements

• Advanced degree (MS/PhD) in a relevant field or demonstrated equivalent practical experience
• Practical experience with evaluating generative models, including diffusion or flow-matching based image or video models
• Very strong Python and software engineering skills with reliable and readable pipelines
• Working knowledge of PyTorch and modern ML infrastructure and practices
• Strong grasp of statistics, including sample sizes, inter-annotator agreement, and confidence levels, with healthy skepticism
• Practical experience building and operating LLM-as-judge systems, understanding their limitations
• Familiarity with modern benchmarking practices and standard datasets and benchmarks
• Inclination to analyze actual outputs beyond aggregate scores

Nice to Haves

• Experience with human evaluation, study design and execution via panels, and analysis of annotator performance
• Experience where evaluation meets training (reward models, preference data, RLHF, DPO or similar)
• Experience evaluating agentic or multi-step systems with attribution challenges
• Developed internal tools widely adopted by teams
• Public contributions to the field such as papers or open source work

What You'll Be Doing

• Building and maintaining the evaluation stack for model checkpoints, including automated benchmarking and human evaluation panels
• Designing evaluation dimensions covering prompt adherence, editing fidelity, aesthetics, text rendering, consistency, and safety
• Assessing the evaluation stack for agreement between human and automated evaluations, stability over time, and improvements
• Continuously benchmarking FLUX models against frontier competitors and analyzing relative strengths and weaknesses
• Identifying quality shortcomings in model outputs not captured by aggregate metrics
• Collaborating with RL and finetuning researchers to define target metrics and verify improvements in new checkpoints
• Designing optimal use of captured signals through APIs and proposing new signals with input from research, data engineering, and legal teams
• Maintaining metric effectiveness over their lifecycle and anticipating issues such as gaming and overfitting

Perks and Benefits

• Opportunity to work at a frontier research lab driving foundational technologies in image and video generation
• Hybrid work model with at least 2 days per week in Freiburg office or one full week every other week, with travel costs covered
• Compensation ranging from €150K to €220K base salary plus equity, depending on experience
• Collaborative and value-driven culture prioritizing scientific excellence, low ego, boldness, and kindness