Requirements

• Experience delivering applied ML systems beyond prototypes, including evaluation, product integration, and post-release improvement
• Strong programming and engineering fundamentals, especially expertise in Python, familiarity with SQL, and knowledge of Spark and cloud data environments like AWS or Databricks
• Practical experience in information retrieval, semantic search, natural language processing, clustering, or LLM-based applications
• Skills in evaluation and experimentation: constructing datasets, selecting metrics, comparing against baselines, and investigating offline vs. customer outcomes
• Sound data judgement regarding issues like data leakage, selection bias, imbalanced data, sparse labels, and overfitting
• Production awareness including consideration of latency, inference cost, observability, and changing data, with ability to collaborate on diagnosing and improving production behavior
• Clear communication and collaboration skills to explain technical trade-offs and work cross-functionally

Nice to haves

• Building retrieval-augmented generation systems involving chunking, embeddings, hybrid search, and reranking
• Experience clustering or discovering themes in large unstructured text collections with traceable evidence
• Designing human evaluation, annotation, or relevance judgement for subjective outputs
• Using implicit or explicit user feedback to improve ML ranking, recommendations, or other behaviors
• Handling cold-start scenarios, limited labeled data, or heterogeneous customer datasets
• Running online experiments or controlled releases and analyzing offline vs. production discrepancies
• Model fine-tuning, selection, or inference optimization with evidence-based evaluation of complexity trade-offs
• Building reusable evaluation tooling, regression checks, or monitoring systems for multi-stage ML and LLM pipelines

What you'll be doing

• Turn product problems into testable ML hypotheses and design experiments against clear baselines
• Build representative evaluation datasets combining automated checks and human judgment
• Develop and improve retrieval, clustering, and language-model pipelines and integrate with customer workflows
• Balance quality, latency, cost, and operational complexity with evidence-based trade-offs
• Evaluate performance across customer segments and dataset sizes including sparse and imbalanced data
• Support production deployment with regression checks, monitoring, controlled rollouts, and failure diagnostics
• Collaborate with product, design, and engineering to interpret feedback and set priorities
• Reuse shared AI infrastructure and contribute to evaluation methods and tooling across Product Collection

Perks and benefits

• Flexible work locations: office, home, or hybrid options to support personal priorities
• Opportunity to work remotely from any country with a legal Atlassian entity