Senior Software Engineer – Annotation & Evaluation (AI Platform)

Requirements:

• You have 6+ years of software engineering experience, including 2+ years working with ML/AI products or platforms.
• You have experience building backend or data infrastructure systems, especially for model evaluation, benchmarking, or labeling workflows.
• You’re comfortable working across the stack: from orchestrating large-scale data processing pipelines to building UI integrations for labeling or eval dashboards.
• You have a solid understanding of ML concepts, including model performance metrics, prompt evaluation, and trust & safety concerns.
• You’re fluent in Python or Go and familiar with cloud-native development patterns.

What You'll Be Doing:

• Design and build systems that support automated evaluation of AI models, including LLMs and agents, using production-like telemetry and scenarios.
• Lead efforts to develop benchmark suites, evaluation pipelines, and model comparison tooling with built-in trust & safety metrics.
• Build and maintain integrations with labeling systems (e.g., Label Studio) and coordinate with external/internal annotation workflows.
• Collaborate closely with Applied AI and Bits AI teams to enable fast iteration, reproducible experimentation, and interpretable evaluations.
• Develop data pipelines that feed metrics, results, and alerts into our observability stack to track model behavior at scale.
• Drive best practices around safe model deployment by building systems for bias checks, hallucination detection, and human-in-the-loop review.

Nice to Haves:

• Bonus points: experience with open-source evaluation frameworks (e.g., lm-eval-harness), vector DBs, or human feedback loops.

Perks and Benefits:

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training, and career pathing
• Intradepartmental mentor and buddy program for in-house networking
• An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks, our internal panel discussions
• Free, global mental health benefits for employees and dependents age 6+
• Competitive global benefits