Requirements

• 2+ years of experience conducting empirical research with large language models or AI systems
• Strong experience with AI coding agents, including usage or design/development
• Experience with LLM-as-a-judge setups or AI monitoring broadly
• Experience designing and running experiments, analyzing results, and iterating based on empirical findings
• Strong Python programming skills
• Demonstrated ability to work independently on open-ended research problems

Nice to Haves

• Familiarity with AI safety concepts, particularly agent-related risks
• Familiarity with computer security, including security testing and secure system design
• Experience fine-tuning language models or working with smaller open-source models
• Previous experience building developer tools or monitoring systems
• Publications or contributions to AI safety or ML research
• Experience with production log systems or production log analysis

What You'll Be Doing

• Design and implement control protocols and test them on real-world production systems at scale
• Design and conduct experiments to test monitor effectiveness across various failure modes and agent behaviors
• Build and maintain evaluation frameworks and high-quality datasets for monitoring progress
• Iterate on monitoring approaches balancing accuracy and computational efficiency
• Contribute newly discovered coding agent failure modes to a shared failure mode library
• Stay current with research on AI safety, agent failures, detection methodologies, and coding security vulnerabilities
• Fine-tune open-source models for efficient monitors in high-volume production environments
• Design and develop agentic monitoring systems to autonomously investigate logs for failure modes
• Incorporate adversarial findings from red-teaming campaigns into monitor design and evaluation

Perks and Benefits

• Market competitive salary and equity
• Flexible work hours and schedule
• Unlimited vacation and sick leave
• Up to 6 months of paid parental leave
• Comprehensive health, dental, and vision insurance
• Retirement savings plans with competitive employer matching
• Provided lunch, dinner, and snacks on workdays
• Paid work trips including staff retreats, business trips, and relevant conferences
• Yearly $1,000 USD professional development budget
• Relocation support and visa fee coverage if applicable