Requirements

A Product-minded engineer who ships AI to production
5+ years experience with backend systems and microservices performance: tracing, latency breakdowns, concurrency, and resiliency patterns
Proficient in a modern programming language; strong API/service design; production ops (monitoring, alerting, on-call rotation)
Proven experience delivering LLM/agent features to production
Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics
End-to-end AI implementation owner: Understands the end-to-end LLM product lifecycle
Fluent with offline/online evals for AI systems

What you'll be doing

Build AI-driven deployment gates: Design and ship decision systems that evaluate customer deployments using CI/CD context and Datadog telemetry, producing safe, explainable allow/block outcomes
Own evals and rollout: Define precision, recall, and trust metrics; build offline and online evals; validate changes in shadow mode; and safely promote improvements to enforcement
Design for robustness and safety: Implement conservative defaults, guardrails, fallbacks, and human-in-the-loop paths so gates behave predictably under noisy or incomplete data
Partner closely with Product: Work hand-in-hand with the Product Manager to translate customer problems, adoption signals, and roadmap goals into concrete technical decisions and iterations
Integrate across the Datadog platform: Partner with internal AI teams building the Faulty Deployment Detection pipeline, as well as teams working on LLMs and AI agents
Own production systems: Build and operate reliable backend services that run in the critical path of customer deployments, and be on-call for those services

Experience with Continuous Delivery tools (i.e. ArgoCD / Argo Rollouts, Spinnaker, Octopus Deploy)
Exposure to planning/agent frameworks, tool-use orchestration, RAG, and retrieval/indexing for observability data

New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
Continuous professional development, product training, and career pathing
Intradepartmental mentor and buddy program for in-house networking
An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
Access to Inclusion Talks, our Internal panel discussions
Free, global mental health benefits for employees and dependents age 6+
Competitive global benefits