Requirements

• 4+ years of professional software engineering experience in Python building production-grade systems, with hands-on strength in either CI/CD pipeline engineering or AI evaluation methodology, plus working familiarity with the other
• 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates
• 1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing)
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience

Nice to Haves

• Experience with AI evaluation and observability frameworks such as LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix
• Strong knowledge of schema validation libraries and API contract testing (JSON Schema, Pydantic, OpenAPI)
• Familiarity with Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses
• Experience with static code analysis, linters, and test automation tooling to enforce code contracts
• Understanding of AI security and governance, including LLM failure modes, prompt injection, data sanitization, and RBAC boundaries
• Strong engineering discipline in test coverage, dataset versioning, evaluator calibration, and regression prevention
• Ability to lead technical communication, own problem domains end-to-end, collaborate across teams, and produce clear documentation and architecture designs

What You'll Be Doing

• Building tiered continuous evaluation pipelines separating instant static/schema checks from multi-turn trajectory and LLM judging eval suites for release gating
• Architecting and maintaining CI/CD build pipelines, skill registry versioning, and runner environments to enforce automated pass/fail release gates on pull requests
• Developing synthetic data generation pipelines and automated red-teaming tools for edge-case evaluation based on API schemas and domain specs
• Curating versioned benchmark datasets and executing continuous regression testing across model upgrades and security checks
• Ingesting real-world production traces and failure logs to continuously improve test coverage
• Bridging AI skill development with production platform engineering using modern evaluation frameworks alongside CI/CD pipelines

Perks and Benefits

• Competitive base salary range (€96,000 EUR - €144,000 EUR for Ireland)
• Eligibility for Workday Bonus Plan or role-specific commissions/bonuses
• Annual refresh stock grants awarded by Workday Inc.
• Flexible work approach combining remote and in-person time, requiring at least 50% in-office or field time each quarter
• Supportive company culture rooted in integrity, empathy, and shared enthusiasm
• Commitment to reasonable accommodations for qualified individuals and inclusive workforce policies