Intermediate Machine Learning Engineer, AI Framework

Requirements

• Proven experience designing and implementing LLM evaluation systems.
• Strong understanding of ML model architectures, including public vs. private implementations.
• Expertise in ML evaluation metrics and dataset management.
• Demonstrated ability to build production-grade ML infrastructure.
• Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, ELI5).
• Excellent problem-solving skills with an engineering mindset.
• Ability to collaborate in an asynchronous, remote-first environment.

What You'll Be Doing

• Design and implement technical evaluators for LLM assessment.
• Contribute to evaluation infrastructure consolidation efforts.
• Build scalable evaluation pipelines and frameworks.
• Develop and manage datasets and evaluation metrics.
• Collaborate with feature teams to integrate validation solutions.
• Optimize performance across ML evaluation systems.
• Support improvements to GitLab’s AI-powered tools through validation.
• Ensure all solutions align with GitLab’s infrastructure and security protocols.

Nice to Haves

• Familiarity with open-source development and contribution is a plus.

Perks and Benefits

• Benefits to support your health, finances, and well-being
• All remote, asynchronous work environment
• Flexible Paid Time Off
• Team Member Resource Groups
• Equity Compensation & Employee Stock Purchase Plan
• Growth and development budget 
• Parental leave 
• Home office support