What You'll Be Doing

• Own competitive benchmarking initiatives across search, vector and hybrid retrieval, and analytical and observability workloads.
• Build and maintain automated benchmark harnesses, datasets, query sets, and load profiles reflecting real-world usage, including concurrent ingestion and querying.
• Design fair system comparisons with documented versions, configurations, tuning decisions, resource budgets, and functional differences.
• Evaluate latency distributions, throughput, resource consumption, and cost efficiency; measure retrieval performance at comparable recall or relevance targets.
• Adapt and extend workload-appropriate tools such as Elasticsearch Rally, ClickBench, and vector search benchmark frameworks.
• Investigate performance differences via profiling, system metrics, and controlled experiments; collaborate with engineering to find causes and validate improvements.
• Evaluate self-managed and managed cloud deployments, including serverless offerings, accounting for scaling behavior and service constraints.
• Work with engineering and content partners to publish reproducible benchmarks and clear technical reports explaining methodology, findings, limitations, and tradeoffs.
• Partner with engineering and product management to prioritize benchmark coverage and translate findings into actionable product improvements.

Requirements

• Solid software engineering skills with experience building maintainable automation and analysis tools in Python or similar languages.
• Hands-on experience evaluating and diagnosing performance in distributed systems, databases, or search infrastructure.
• Depth in at least one relevant area: search, vector retrieval, analytical databases, or observability systems, with ability to learn new systems.
• Experience designing representative datasets and workloads, including query patterns, concurrency, and ingestion behavior.
• Sound experimental judgment, including repeated measurements, variability analysis, warm-up and cache effects, and distinguishing meaningful changes from noise.
• Experience building reproducible test environments and documenting configurations, procedures, and results.
• Clear technical writing and communication skills for explaining complex findings and limitations.
• Strong collaboration skills with ability to turn performance evidence into engineering recommendations.

Nice to Haves

• Experience with Elasticsearch, Lucene, Rally, ClickBench, or vector search benchmark frameworks.
• Experience measuring retrieval quality along with latency, throughput, and resource usage.
• Familiarity with cloud performance and cost analysis, including autoscaling and serverless systems.
• Public technical writing, open-source contributions, or published benchmark studies.

Perks and Benefits

• Competitive pay based on role responsibilities rather than previous salary.
• Health coverage for employees and families in many locations.
• Flexible work locations and schedules for many roles.
• Generous vacation allowance each year.
• Company matches up to $2000 (or local equivalent) for financial donations and volunteer service.
• Up to 40 hours annually to use for volunteer projects.
• At least 16 weeks of parental leave.
• Commitment to diversity, inclusion, and accommodation for applicants with disabilities.