Software Engineer – Serving (AI Platform)

Requirements:

• You have 3+ years of experience in backend or infrastructure engineering roles, ideally with exposure to ML workflows.
• You're comfortable working with distributed systems and enjoy building services that are reliable and observable.
• You have experience with Python, Go, or another backend-oriented language.
• You're curious about how inference infrastructure works, and excited to learn about model serving and GPU optimization.

Nice to Haves:

• Bonus points: familiarity with Ray, Kubernetes, or LLM-serving architectures.

What You'll Be Doing:

• Help build and maintain distributed serving infrastructure for ML models running on Ray clusters.
• Collaborate with scientists and engineers to deploy models and debug performance issues in production.
• Contribute to rollout mechanisms, traffic controls, and observability tooling for deployed inference endpoints.
• Implement tools for tracking and analyzing inference metrics such as latency, throughput, and token usage.
• Participate in design discussions, code reviews, and on-call rotations as part of the AI Platform team.

Perks and Benefits:

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training, and career pathing
• Intradepartmental mentor and buddy program for in-house networking
• An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks, our internal panel discussions
• Free, global mental health benefits for employees and dependents age 6+
• Competitive global benefits