Requirements

Bachelor’s degree or equivalent practical experience.
2 years of experience with software development in one or more programming languages, or 1 year with an advanced degree.
1 year of experience with end-to-end Machine Learning (model deployment, evaluation, data processing, debugging).
Experience with Generative AI techniques or concepts, including LLMs, Multi-Modal, Large Vision Models, language modeling, and computer vision.
Experience in Generative AI.
Experience using Python.

Nice to haves

Research, experiment, and prototype novel methods to analyze model behavior and failure patterns.
Conduct empirical investigations into model outputs, agent trajectories, and error patterns to guide benchmark design and evaluation trustworthiness.
Develop and refine benchmark suites for complex agentic reasoning across domains like code, math, and specialized fields, ensuring high quality and no data contamination.
Create and maintain pipelines that convert benchmark execution logs, agent traces, and artifacts into reliable evaluation datasets and training data.
Collaborate with research scientists, senior engineers, and cross-functional teams to turn research insights into practical evaluation infrastructure.
Work on projects critical to Google’s needs, with opportunities to switch teams and projects as the business evolves.
Invent, prototype, and build next-generation techniques and pipelines for analyzing model behavior and evaluating complex reasoning.