Requirements

Experience implementing, training, testing, and evaluating Machine Learning systems
Strong Python knowledge, with familiarity in OOP principles being a big plus
Professional exposure to Generative AI / LLM solutions such as prompt engineering, evaluation, LLM-powered workflows, or Agentic AI
Knowledge and experience developing and evaluating GenAI / LLM solutions, including automated evaluation harnesses and prompt engineering
Experience with statistical analysis, experiment design, and failure mode analysis in ML or LLM systems
Strong product mindset and ability to work independently in a cross-functional, cross-team environment
Good communication skills to effectively engage with non-technical individuals and SMEs
Strong problem-solving skills with the ability to refine problem statements and devise solutions

Nice to Haves

Familiarity with automating operational processes using Large Language Models, AI agents, or multi-step AI workflows
Experience implementing fine-tuning, reinforcement learning alignment, and evaluation techniques within an LLM training pipeline
Familiarity with agentic frameworks such as LangGraph or similar
Experience designing tool-using agents, orchestration workflows, or human-in-the-loop AI systems
Willingness to review many historical operational cases in detail
Working knowledge of Java

Lead development and deployment of AI models to augment operational workflows, including automation of case summaries, red flag generation, risk classifications, and agentic workflows
Establish labeling taxonomies and guidelines with EDD SMEs, construct evaluation datasets, and implement automated evaluation harnesses
Audit prompts and run structured experiments to optimize prompt design, reduce hallucinations, formatting errors, and drift
Design and develop agentic AI workflows that reason across multiple steps, utilize tools, and support complex operational processes
Design shadow testing and parallel execution deployments, and implement post-deployment monitoring for data and output drift
Develop and deploy production-ready Python services, balancing quick value delivery with code quality
Design human-in-the-loop AI systems that provide recommendations while ensuring final decision-making rests with human operators
Analyze upstream data to develop strategies that reduce manual workload by deflecting financial crime attempts before they reach operations
Provide technical leadership, set direction, mentor teammates, and guide complex technical and architectural decisions